Black Box Recursive Translations for Molecular Optimization

1 INTRODUCTION . Automated molecular design using generative models offers the promise of rapidly discovering new compounds with desirable properties . Chemical space is large , discrete , and unstructured , which together , present important challenges to the success of any molecular optimization campaign . Approximately 108 compounds have been synthesized ( Kim et al. , 2015 ) while the range of potential drug-like candidates is estimated to between 1023 and 1080 ( Polishchuk et al. , 2013 ) . Consequently , new methods for intelligent search are paramount . A recently introduced paradigm for compound generation treats molecular optimization as a translation task where the goal is to map an input compound to a target compound with favorable properties ( Jin et al. , 2019b ) . This framework has presented impressive results for constrained molecular property optimization where generated compounds are restricted to be structurally similar to the source molecule . We extend this framework to unconstrained molecular optimization by treating inference , vis-à-vis decoding strategies , as a first-class citizen . We observe that generated molecules can be repeatedly fed back into the model to generate even better compounds . This finding is invariant to the choice of translation model , making this a ” black box ” algorithm . This invariance is particularly attractive considering the recent emphasis on new molecular representations ( Gómez-Bombarelli et al. , 2018 ; Jin et al. , 2018 ; Dai et al. , 2018 ; Li et al. , 2018 ; Kusner et al. , 2017 ; Krenn et al. , 2019 ) . Using our simple drop-in replacement , our method can leverage these recently introduced molecular representations in a translation setting for better optimization . We introduce Black Box Recursive Translation ( BBRT ) , a new inference method for molecular property optimization . Surprisingly , by applying BBRT to well-known sequence- and graph-based models in the literature , we can produce new state-of-the-art results on property optimization benchmark tasks . Through an exhaustive exploration of various decoding strategies , we demonstrate the empirical benefits of using BBRT . We introduce simple ranking methods to decide which outputs are fed back into the model and find ranking to be an appealing approach to secondary property optimization . Finally , we demonstrate how BBRT is an extensible tool for interpretable and user-centric molecular design applications . 2 RELATED WORK . De novo molecular design . Recent work has focused on learning new molecular representations including graphs ( You et al. , 2018b ; Li et al. , 2018 ) , grammars ( Kusner et al. , 2017 ; Dai et al. , 2018 ) , trees ( Jin et al. , 2018 ) , and sequences ( Gómez-Bombarelli et al. , 2018 ; Krenn et al. , 2019 ) . Provided with a molecular representation , latent variable models ( Kusner et al. , 2017 ; Dai et al. , 2018 ; Jin et al. , 2018 ) ) , Markov chains ( Seff et al. , 2019 ) , or autoregressive models ( You et al. , 2018b ) have been developed to learn distributions over molecular data . Molecular optimization has been approached with reinforcement learning ( Popova et al. , 2018 ; Zhou et al. , 2019 ) and optimization in continuous , learned latent spaces ( Gómez-Bombarelli et al. , 2018 ) . For sequences more generally , Mueller et al . ( 2017 ) perform constrained optimization in a latent space to improve the sentiment of source sentences . We build on recent work introducing a third paradigm for design , focusing on molecular optimization as a translation problem ( Jin et al. , 2019b ) , where molecules are optimized by translating from a source graph to a target graph . While retaining high similarity , the target graph improves on the source graph ’ s biochemical properties . With many ways to modify a molecule , Jin et al . ( 2019b ) use stochastic hidden states coupled with a left-to-right greedy decoder to generate multi-modal outputs . We extend the translation framework from similarity-constrained molecular optimization to the unconstrained setting of finding the best scoring molecules according to a given biochemical property . We show that while their translation model is not fundamentally limited to constrained optimization , their inference method restricts the framework ’ s application to more general problems . Matched molecular pair ( MMP ) analysis . MMP analysis is a popular cheminformatics framework for analyzing structure-property relationships ( Kramer et al. , 2014 ; Turk et al. , 2017 ) . MMPs are extracted by mining databases of measured chemical properties and identifying pairs of chemical structures that share the same core and differ only by a small , well-defined structural difference ; for example , where a methyl group is replaced by an isopropyl group ( Tyrchan & Evertsson , 2017 ) . The central hypothesis underlying MMPs is that the chemical properties of a series of closely related structures can be described by piecewise-independent contributions that various structural adducts make to the properties of the core . MMP analysis has become a mainstream tool for interpreting the complex landscape of structureactivity relationships via simple , local perturbations that can be learnt and potentially transferred across drug design projects ( Kubinyi , 1988 ) . This framework serves as the chemistry analogue to an interpretability technique in machine learning called local interpretable model-agnostic explanations ( LIME ) ( Ribeiro et al. , 2016 ) . Both MMP and LIME learn local surrogate models to explain individual predictions . We view molecular translation as a learned MMP analysis . While Jin et al . ( 2019b ) use neural networks to learn a single high-dimensional MMP step , we extend this framework to infer a sequence of MMPs , extending the reach of this framework to problems beyond constrained optimization . Translation models . Machine translation models composed of end-to-end neural networks ( Sutskever et al. , 2014 ) have enjoyed significant success as a general-purpose modeling tool for many applications including dialogue generation ( Vinyals & Le , 2015 ) and image captioning ( Vinyals et al. , 2015 ) . We focus on a recently introduced application of translation modeling , one of molecular optimization ( Jin et al. , 2019b ) . The standard approach to inference – approximately decoding the most likely sequence under the model – involves a left-to-right greedy search , which is known to be highly suboptimal , producing generic sequences exhibiting low diversity ( Holtzman et al. , 2019 ) . Recent work propose diverse beam search ( Li & Jurafsky , 2016 ; Vijayakumar et al. , 2018 ; Kulikov et al. , 2018 ) , sampling methods geared towards open-ended tasks ( Fan et al. , 2018 ; Holtzman et al. , 2019 ) , and reinforcement learning for post-hoc secondary objective optimization ( Wiseman et al. , 2018 ; Shen et al. , 2016 ; Bahdanau et al. , 2017 ) . Motivated by molecular optimization as a translation task , we develop Black Box Recursive Translation ( BBRT ) . We show BBRT generates samples with better molecular properties than its non-recursive peers for both deterministic and stochastic decoding strategies . 3 MOLECULAR OPTIMIZATION AS A TRANSLATION PROBLEM .. For illustrative purposes , we describe the translation framework and the corresponding inference method for sequences . We emphasize that our inference method is a black box , which means it is invariant to specific architectural and representational choices . Background . Our goal is to optimize the chemical properties of molecules using a sequencebased molecular representation . We are given ( x , y ) ∈ ( X , Y ) as a sequence pair , where x = ( x1 , x2 , . . . , xm ) is the source sequence with m tokens and y = ( y1 , y2 , . . . , yn ) is the target sequence with n tokens , and X and Y are the source and target domains respectively . We focus on problems where the source and target domains are identical . By construction , our training pairs ( x , y ) have high chemical similarity , which helps the model learn local edits to x. Additionally , y always scores higher than x on the property to be optimized . These properties are specified beforehand when constructing training pairs . A single task will typically optimize a single property such as potency , toxicity , or lipophilic efficiency . A sequence to sequence ( Seq2Seq ) model ( Sutskever et al. , 2014 ) learns parameters θ that estimate a conditional probability model P ( y|x ; θ ) , where θ is estimated by maximizing the log-likelihood : L ( θ ) = ∑ ( x , y ) ∈ ( X , Y ) logP ( y|x , θ ) ( 1 ) The conditional probability is typically factorized according to the chain rule : P ( y|x ; θ ) =∏n t=1 P ( yt|y < t , x , θ ) . These models use an encoder-decoder architecture where the input and output are both parameterized by recurrent neural networks ( RNNs ) . The encoder reads the source sequence x and generates a sequence of hidden representations . The decoder estimates the conditional probability of each target token given the source representations and its preceding tokens . The attention mechanism ( Bahdanau et al. , 2014 ) helps with token generation by focusing on tokenspecific source representations . 4 BLACK BOX RECURSIVE TRANSLATION . 4.1 CURRENT INFERENCE METHODS . For translation models , the inference task is to compute y∗ = argmax p ( y|x , θ ) . Because the search space of potential outputs is large , in practice we can only explore a limited number of sequences . Given a fixed computational budget , likely sequences are typically generated with heuristics . Decoding methods can be classified as deterministic or stochastic . We now describe both classes of decoding strategies in detail . For this section , we follow the notation described in Welleck et al . ( 2019 ) . Deterministic Decoding . Two popular deterministic methods include greedy search and beam search ( Graves , 2012 ; Boulanger-Lewandowski et al. , 2013 ) . Greedy search performs a single leftto-right pass , selecting the most likely token at each time step : yt = argmax p ( yt|y < t , x , θ ) . While this method is efficient , it leads to suboptimal generation as it does not take into account the future when selecting tokens . Beam search is a generalization of greedy search and maintains a set of k hypotheses at each timestep where each hypothesis is a partially decoded sequence . While in machine translation beam search variants are the preferred method , for more open-ended tasks , beam search fails to generate diverse candidates . Recent work has explored diverse beam search ( Li & Jurafsky , 2016 ; Vijayakumar et al. , 2018 ; Kulikov et al. , 2018 ) . These methods address the reduction of number of duplicate sequences to varying extents , thereby increasing the entropy of generated sequences . Stochastic Decoding . A separate class of decoding methods sample from the model at generation time , yt ∼ q ( yt|y < t , x , pθ ) . This method has shown to be effective at generating diverse samples and can better explore target design spaces . We consider a top-k sampler ( Fan et al. , 2018 ) , which restricts sampling to the k most-probable tokens at time-step t. This corresponds to restricting sampling to a subset of the vocabulary U ⊂ V . U is the subset of V that maximizes∑ y∈U pθ ( yt|yy < t , x ) : q ( yt|y < t , x , pθ ) = { pθ ( yt|y < t , x ) Z yt ∈ U 0 otherwise

Summary: This paper presents a translation-based method for molecular property optimization. It uses a sequence- or graph-based encoder/decoder framework to produce molecules with (hopefully) improved properties, then feeds a subset of these molecules back into the encoder/decoder to generate a new set of molecules; this process is repeated for a fixed number of iterations to arrive at a final set of "optimized" molecules. The method is agnostic to the form of the encoder/decoder; the emphasis is on the iterative approach. Additionally, this approach enables visualization of "molecular" traces that can reveal pathways between molecules that follow relationships similar to matched molecular pairs. The work extends related work in translation-based property optimization [6]. The paper is well-written and generally easy to read.