Black Box Recursive Translations for Molecular Optimization
Authors: Farhan Damani
Vishnu Sresht
Stephen Ra
Venue:  (2020)
Keywords: molecules
chemistry
drug design
generative models
application
translation

Abstract: Machine learning algorithms for generating molecular structures offer a promising new approach to drug discovery. We cast molecular optimization as a translation problem, where the goal is to map an input compound to a target compound with improved biochemical properties. Remarkably, we observe that when generated molecules are iteratively fed back into the translator, molecular compound attributes improve with each step. We show that this finding is invariant to the choice of translation model, making this a "black box" algorithm. We call this method Black Box Recursive Translation (BBRT), a new inference method for molecular property optimization. This simple, powerful technique operates strictly on the inputs and outputs of any translation model. We obtain new state-of-the-art results for molecular property optimization tasks using our simple drop-in replacement with well-known sequence and graph-based models. Our method provides a significant boost in performance relative to its non-recursive peers with just a simple "``for" loop. Further, BBRT is highly interpretable, allowing users to map the evolution of newly discovered compounds from known starting points. 

Under review as a conference paper at ICLR 2020
BLACKBoL RECURSIVE TRANSLATIONS FOR MOLEC- ULAR OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Machine learning algorithms for generating molecular structures offer promis- ing new approach to drug discovery. We cast molecular optimization as a transla- tion problem, where the goal map an input compound target compound with improved biochemical properties. Remarkably, we observe that when gener- ated molecules are iteratively fed back into the translator, molecular compound at- tributes improve with each step. We show that this finding is invariant to the choice of translation model, making this "black box" algorithm. We call this method Black Box Recursive Translation (BBRT), new inference method for molecular property optimization. This simple, powerful technique operates strictly on the in- puts and outputs of any translation model. We obtain new state-of-the-aut results for molecular property optimization tasks using our simple drop-in replacement with well-known sequence and graph-basee bash- models. Our method provides sig- nificant boost in performance relative to its non-recursive peers with just a simple "for" loop. Further, BBRT is highly interpretable, allowing lution of newly discovered compounds from known starting points.
to map the evo-
INTRODUCTION
Automated molecular design using generative models offers the promise of rapidly discovering new compounds with desirable properties. Chemical space is large, discrete, and unstructured, which together, present important challenges to the success of any molecular optimization campaign. Ap- proximately 10 compounds have been synthesized (Kim et al. 2015) while the range of potential drug-like candidates estimated to between 1023 and 10 (Polishchuk et 2013). Consequently, new methods for intelligent search are paramount.
A recently introduced paradigm for compound generation treats molecular optimization as a transla- tion task where the goal is map an input compound to target compound with favorable properties [Jin et al.] 2019b). This framework has presented impressive results for constrained molecular prop- erty optimization where generated compounds restricted to be structurally similar to the source molecule.
We extend this framework to unconstrained molecular optimization by treating inference, vis-à-vis decoding strategies, as a first-class citizen. We observe that generated molecules can be repeatedly fed back into the model generate even better compounds. This finding invariant to the choice of translation model, making this a "black box" algorithm. This invariance is particularly attrac- tive considering the recent emphasis on new molecular representations (Cmmez-Bommarrlli al., 2018 Jin et al. 2018 Dai et al.) 2018 Li et al. 2018 Kusner et al.) [017]] Krenn et al., 2019). Using our simple drop-in replacement, our method can leverage these recently introduced molecular representations in translation setting for better optimization.
We introduce Black Box Recursive Translation (BBRT), a new inference method for molecular property optimization. Surprisingly, by applying BBRT to well-known sequence- and braed- models in the literature, we can produce new state-of-the-art results on property optimization bench- mark tasks. Through an exhaustive exploration of various decoding strategies, we demonstrate the empirical benefits of using BBRT. We introduce simple ranking methods to decide which outputs are fed back into the model and find ranking to be appealing approach to secondary property opti- mization. Finally, we demonstrate how BBRT is an extensible tool for interpretable and user-centric molecular design applications.

---

Under review as a conference paper at ICLR 2020
Score
Sample
Top
Black Box Translator
Figure/ Black Box Recursive Translation (BBRT).
2
RELATED WORK
De nove molecular design. Recent work has focused on learning new molecular representations in-
cluding graphs ( You al. 2018b] et al. 2018), grammars (Kusner et al. [017] et al.
2018).
trees (Jin et al., 2018), and sequences (Gómez- Bomnnez-ommaarlli al. 2018; Krenn et al. 2019
Pro-
vided with a molecular representation, latent variable models (Kusner et al. 201 7: Dai et al. 2018
Jin et al. 2018)), Markov chains (Seff et al. 2019), autoregressive models You et al. 8018b)
have been developed to learn distributions over molecular data. Molecular optimization has been approached with reinforcement learning (Popova et al.) 2018 Zhou et al., (019)) and optimization in continuous, learned latent spaces (Gömez-Bombarell et al. 2018). For sequences more generally, Mueller al. (2017) perform constrained optimization in latent space improve the sentiment of source sentences.
We build on recent work introducing a third paradigm for design, focusing on molecular optimiza- tion as translation problem (Jin et al. (2019)), where molecules are optimized by translating from a source graph to target graph. While retaining high similarity, the target graph improves on the source graph's biochemical properties. With many ways to modify a molecule, Jin et al. (2019b) use stochastic hidden states coupled with a left-to-right greedy decoder to generate multi-modal outputs. We extend the translation framework from constrained setting of finding the best scoring molecules according to a given biochemical property. We show that while their translation model is not fundamentally limited to constrained optimization, their inference method restricts the framework's application to more general problems.
molecular optimization to the un-
Matched molecular pair (MMP) analysis. MMP analysis is popular cheminformatic, framework for analyzing structure-propent relationships (Kramer et al.) 2014, Turk et al., (017). MMPs are extracted by mining databases of measured chemical properties and identifying pairs of chemical structures that share the same core and differ only by a small, well-defined structural difference; for example, where methyl group replaced by isopropyl group (Tyrchan & Evertsson (017).
The central hypothesis underlying MMPs is that the chemical properties of a series of closely related structures can be described by piccewise-indepeneen contributions that various structural adducts make to the properties of the core.
MMP analysis has become mainstream tool for interpreting the complex landscape of structure- activity relationships via simple, local perturbations that can be learnt and potentially transferred across drug design projects (Kubinyi 1988). This framework serves as the chemistry analogue to an interpretability technique in machine learning called local interpretable model-agnostic explana- tions (LIMH) (Ribeiro et al., (016). Both MMP and LIME learn local surrogate models to explain individual predictions.
We view molecular translation as a learned MMP analysis. While Jin et al. (2019b) use neural networks to learn single high-dimensional MMP step, we extend this framework to infer sequence of MMPs, extending the reach of this framework to problems beyond constrained optimization.
Translation models. Machine translation models composed of end-to-end neural networks (Sutskever et al. 2014) have enjoyed significant success as a general- modeling tool for many applications including dialogue generation (Vinyals & Le 2015) and image captioning (Vinyals et al. 2015). We focus on a recently introduced application of translation modeling, one of molecular optimization (Jin et al. 2019b).
The standard approach to inference approximately decoding the most likely sequence under the model involves a left-to-right greedy search, which is known to be highly suboptimal, producing
2

---

Under review as a conference paper at ICLR 2020
generic sequences exhibiting low diversity (Holtmman et al. 2019). Recent work propose diverse
beam search (Li & Jurafsky 2016 Vijayakumar et al. 2018: Kulikov et al. 2018). sampling meth-
ods geared towards open-ended tasks (Fan et al.) 2018 Holtzman et al. 2019). and reinforcement
learning for post-hoe secondary objective optimization (Wiseman et al. 2018 Shen et al. 2016) Bahdanau et al., 2017). Motivated by molecular optimization
translation task, develop Black Box Recursive Translation (BBRT). We show BBRT generates samples with better molecular properties than its non-recursive peers for both deterministic and stochastic decoding strategies.
MOLECULAR OPTIMIZATION AS TRANSLATION PROBLEM.
For illustrative purposes, we describe the translation framework and the corresponding inference method for sequences. We emphasize that our inference method a black box, which means is invariant to specific architectural and representational choices.
Background. Our goal is to optimize the chemical properties of molecules using a sequence- based molecular representation. We are given (x, y) (X )) as sequence pair, where (I1, Fmm) is the source sequence with m tokens and = (v1 V2...., Vn) is the target se- quence with tokens, and X' and y' are the source and target domains respectively. We focus problems where the source and target domains are identical. By construction, our training pairs (ar, y) have high chemical similarity, which helps the model learn local edits x. Additionally, y always scores higher than x on the property to be optimized. These properties are specified before- hand when constructing training pairs. A single task will typically optimize single property such as potency, toxicity, or lipophilic efficiency.
sequence sequence (Seq2Seq) model (Sutskever et al., 2014) learns parameters that estimate a conditional probability model P(y|æ; 0), where @ is estimated by maximizing the log-likelihood:
L(0)-
Σ
log P(y|z,))
(1)
(x.)))(()))))
The conditional probability is typically factorized according to the chain rule: P(yla: 0) II P(yllye,, 0). These models use an encoder-decoder architecture where the input and out- put are both parameterized by recurrent neural networks (RNNs). The encoder reads the source sequence x and generates a sequence of hidden representations. The decoder estimates the con- ditional probability of each target token given the source representations and its preceding tokens. The attention mechanism (Bahdanau et al. 2014) helps with token generation by focusing on token- specific source representations.
4
BLACK Box RECURSIVE TRANSLATION
4.1 CURRENT INFERENCE METHODS
For translation models, the inference task is to compute y" = arg max 0). Because the search space of potential outputs is large, in practice can only explore limited number of sequences. Given a fixed computational budget, likely sequences are typically generated with heuristics. De- coding methods can be classified as deterministic or stochastic. We now describe both classes of decoding strategies in detail. For this section, we follow the notation described in Welleck et al. 2019).
Deterministic Decoding. Two popular deterministic methods include greedy search and beam et al. (013). Greedy search performs a single left- to-right pass, selecting the most likely token at each time step: Y/ arg max p(1a Veer, 2r, 0). While this method efficient, leads to suboptimal generation as does not take into account the future when selecting tokens.
search (Graves) 2012, Gouulager-Lewwndowwe
Beam search is generalization of greedy search and maintains a set of & hypotheses at each time- step where each hypothesis is a partially decoded sequence. While in machine translation beam search variants are the preferred method, for more open-ended tasks, beam search fails to generate diverse candidates. Recent work has explored diverse beam search (Li & Jurafsky 2016 Vijayaku- mar et al.) 2018; Kulikov et al. 2018). These methods address the reduction of number of duplicate sequences varying extents, thereby increasing the entropy of generated sequences.
3

---

Under review as a conference paper at ICLR 2020
Stochastic Decoding, A separate class of decoding methods sample from the model at generation time, V ~ (89eWee,, z, po). This method has shown to be effective at generating diverse sam- ples and can better explore target design spaces. We consider top-k sampler (Fan et al.] 2018), which restricts sampling to the k most-probable tokens at time-step t. This corresponds to re- stricting sampling subset of the vocabulary U c V. U is the subset of V that maximizes E yEU Po Po(yelyyee,.)
polste (ece.a))
V/ EU
((deleer p,) =
otherwise
4.2 RECURSIVE INFERENCE
We are given (x. E (X, Y) as a sequence pair where by construction (x, v) has high chemi- cal similarity and y scores higher on prespecified property compared æ. At test time, interested in recursively inferring new sequences. Let denote random sequence for recursive
iteration i. Let {y,-) k==1 )= be set of K outputs generated from po [z) when 0. We use scoring function to compute the best of K outputs denoted as ÿ.- For i > 0, we infer K outputs from po (y 11) This process is repeated for iterations.
Scoring functions. In principle, all K outputs at iteration can be independently conditioned on 1. This procedure scales exponentially with respect
generate new candidates for iteration to space and time as a function of n iterations. Therefore, we introduce suite of simple ranking strategies score K output sequences to decide which output becomes the iteration's source sequence. We consider a likelihood based ranking as well as several chemistry-speeiffe metrics further described the experiments. Designing well-informed scoring functions can help calibrate the distributional statistics of generated sequences, aid in multi- optimization, and provide interpretable sequences of iteratively optimal translations.
Ensemble outputs. After recursive iterations, we ensemble the generated outputs (yo, .......,........................ yn Jk
and score the sequences on a desired objective. For property optimization, we return the arg max. In principle, BBRT is not limited to ensembling recursive outputs from single model. Different modeling choices and molecular representations have different inductive biases, which influence the diversity of generated compounds. BBRT can capitalize these differences by providing coherent method to aggregate results.
5 EXPERIMENTS
We apply BBRT to solve unconstrained and multi-property optimization problems. To highlight the generality of our method, we apply recursive translation to both sequence and graph-based trans- lation models. We show that BBRT generates state-of-the-art results molecular property optimization using already published modeling approaches. Next we describe how recursive in- ference lends itself to interpretability through the generation of molecular traces, allowing practition- ers to map the evolution of discovered compounds from known starting points through a sequence of local structural changes. At any point in a molecular trace, users may introduce "break point" to consider alternative translations thereby personally evaluating the trade-off's between conflicting design objectives. Finally, we apply BBRT the problem of secondary property optimization by ranking.
Models. We apply our inference method sequence and graph-based molecular representations. For sequences, we use the recently introduced SELFIES molecular representation (Krenn et al.) 2019) a sequence-based representation for semantically constrained graphs. Empirically, this method generates high percentage of valid compounds (Krenn et al., 2019). Using SELFIES, we develop a Seq2Seq model with an encoder-decoder framework. The encoder and decoder are both parameterized by RNNs with Long Short- Memory (LSTM) cells (Hochreiter & Schmid- huber 1997). The encoder is a 2-layer bidirectional RNN and the decoder is a 1-layer unidirectional forward RNN. We also attention (Bahdanau al. 2014) for decoding. The hidden representa- tions are probabbbaiiiittcc and are optimized to minimize standard cross-ents cross-entropy loss with teacher forcing. Decoding is performed using deterministic and stochastic decoding strategies described in
4

---

Under review as a conference paper at ICLR 2020
Penalized logP
QED
Method
1st
2nd
3rd
1st
2nd
3rd
ZINC-250K
4.52
4.30
4.23
0.948
0.948
0.948
ORGAN
3.63
3.49
3.44
0.896
0.824
0.820
JT-VAE
5.30
4.93
4.49
0.925
0.911
0.910
GCPN
7.98
7.85
7.80
0.948
0.947
0.946
JTNN
5.97
4.96
4.71
0.948
0.948
0,948
Seq2Seq
4.65
4.53
4.49
0.948
0.948
0.948
BBRT-JTNN
10.13
10.10
9.91
0.948
0.948
0.948
BBRT-SeqQSeq
6.74
6.47
6.42
0.948
0.948
0.948
Table Top property scores on penalized logP and QED tasks.
Section 4.1]
For graphs, we use tree-based molecular representation (Jin et al. (018)) with the exact architecture described in Jin et al. (2019b). Decoding is performed using probabilistic extension with latent variables described in Jin et al. (2019b) we sample from the prior times followed by left-to-right greedy decoding.
Data. We construct training data by sampling molecular pairs (X, Y) with molecular similarity sim sim(X, Y) > and property improvement o(Y) > 8(X) for given property 8. Constructing training pairs with a similarity constraint can help avoid degenerate mappings. In contrast to et al. (2019)), only enforce the similarity constraint for the construction of training pairs. Similarity constraints are not enforced at test time. Molecular similarity is measured by computing Tanimoto similarity with Morgan fingerprints (Rogers Hahn 2010). All models were trained the open- source ZINC dataset (Irwin et al. 2012). We use the ZINC-250K subset, as described in Gómez- Bombarelli et al. (2018).
Properties. For all experiments, we focus on optimizing two well-known drug properties of molecules. First, we optimize the water-octanol partition coefficient (logP). Similar Din et al. 2018); Kusner et al.) (2017); Gomee-Bombarelli et al. (2018), we consider penalized logP score that incorporates ring size and synthetic accessibility. The penalized logP score uses dataset nor- malization score described in You et al. (2018a). Following Jiin et al. (2019b) we extracted 99K translation pairs for training using a similarity constraint 0.4. Second, we optimize quantitative estimate of drug likeness (QED) (Bickerton al. 2012). Following Jin al. (019b), construct training pairs where the source compound has a QED score within the range [0.7 0.8) and the target compound has QED score within the range [0.9 1.0]. While Jin et al. (019b) evaluates QED performance based on a closed set translation task, we evaluate this model for unconstrained opti- mization. We extracted training of 88K molecule pairs. We report the details how these properties were computed in Appendix A
Scoring functions. Here we describe scoring functions that are used to rank K outputs for recursive iteration i. The highest scoring sequence is used as the next iteration's 's source compound.
Penalized logP: Choose the compound with the maximum logP value. This is useful when optimizing logP as primary or secondary property.
QED: Choose the compound with the maximum QED value. This is useful when optimiz- ing QED as primary or secondary property.
Max Delta Sim: Choose the compound with the highest chemical similarity the previous iteration's 's source compound. This is useful for interpretable, molecular traces by creating sequence of translations with local edits.
Max Init Sim: Choose the compound with the highest similarity to the initial seed com- pound. This useful for input-constrained optimization.
Min Mol Wt: Choose the compound with the minimum molecular weight. This is useful for rectifying a molecular generation artifact where models optimize logP by simply adding functional groups, thus increasing the molecular weight (Figure
5

---

Under review as a conference paper at ICLR 2020
Top 100 log?
Seg2fom
ITMM
Top 100 tairiig B2ST (Th
Nomooloo Prequees Frequetcy
4.
15 Penalizee log?
3.0
6.5
66
14
Penalized logP
Diversity
Figure 2: Left and Center: Top 100 logP generated compounds under BBRT-SeQ2Seq, BBRT-JTNN, and their non non-recursivv counterparts. Right: Diversity of top 100 generated compounds under both BBRT models and the top 100 compounds from the training data.
Penalized logp
QED
BBRT-Seq2Seq
lag? -MM
logp
CLU
BBRT-JTNN
logp
logf
OED - .48
GED
Figure 3: Top scoring compounds for properties logP and QED under BBRT-Seq2Seq and BBRT- JTNN.
Diversity is computed as follows. Let Y be a set of translated compounds. Consistent with the literature (Jin al. 2019a) we report diversity as
DIV(Y) = |K(YY
E-
Σ Σ 1 1- 8(v.2/)
(2)
NEY y'EY''
8 is the Tanimoto similarity computed on the Morgan fingerprints of y and y'.
5.1 MOLECULE GENERATION RESULTS
Setup. For property optimization. the goal is generate compounds with the highest possible pe- nalized logP and QED values. For notation, we denote BBRT applied to model X as 'BBRT-X'. We consider BBRT-ITNN and BBRT-Seq2Seq under 3 decoding strategies and 4 scoring functions. For the logP task, we seed our translations with 900 maximally diverse compounds with an average pairwise diversity of 0.94 + 0.04 relative to 0.86 0.04, which is the average pairwise diversity of the training data. Maximally diverse compounds were computed using the MaxMin algorithm Ashton et al. 2002) Morgan fingerprints. We found seeding our translations with diverse com- pounds helped BBRT generate higher scoring compounds (Supplement Fig. 10 For both BBRT applications, we sampled 100 complete sequences from a top-2 and from a top-5 sampler and then aggregated these outputs with beam search using 20 beams outputting 20 sequences.
Baselines. We compare our method with the following state-of ofee-e--rrrt-r---- baselines. Junction Tree VAE (JT-VAE: Jin et al. 2018) combines a graph-based representation with latent variables for molecular graph generation. Molecular optimization is performed using Bayesian Optimization on the learned latent space to search for compounds with optimized property scores. JT-VAE has been shown to outperform other methods including Character VAE (Gómez Bombarelli et al. 2018), Grammar VAE VAE (Dai et al. 2018). We also compare against two Generative Adversarial
(Kusner et al. 2017), and inforcement learning molecular generation methods:
Networks (ORGAN; Guimaraes al. 2017) uses SMILES strings (Weininger 1988), text-based

---

Under review as a conference paper at ICLR 2020
Mean log
loge
Max Pairotin
Top
Top
logh:
Mox Parroise fimm
And Top 1812
Max Soed Sim, BBRT
And Top 8B3T
Min Mei Wt. HURT
BBRT Seq2Seq: Penalized logP
BBRT Seq2Seq: QED
0.9-
Mean Pairwise Diversity
0.8
0.7
0.6
Bear
And Top
Rnd Top
15
20
25
Iteration
Figure 4: Ablation experiments using BBRT-Seq2Seq. A. Left: Mean logP from 900 translations as function of recursive iteration for three decoding strategies. Dotted lines denote non-recursive counterparts. Right: Mean logP as function of recursive iteration for scoring functions. B. Left: Diversity of generated outputs across recursive iterations for logP translation. Right: Diversity of generated outputs across recursive iterations for QED translation.
molecular representation, and the Graph Convolutional Policy Network (GCPN; You al. 8a), uses graphs.
We also compare against non-recursive variants of the Seq2Seq and Variational Junction- Tree Encoder-Decoder (JTNN; Jín et al. (019b) models. Seq2Seq is trained on the SELFIES repre- sentation (Krenn et al. 2019). For a fair comparison, we admit similar computational budgets to these baselines. For Seq2Seq we include deterministic and stochastic decoding baseline. For the deterministic baseline, we use beam search with 20 beams and compute the 20 most probable se- quences under the model. For the stochastic baseline, sample 6600 times from top-5 sampler. For details on the sampler, we refer readers to Soccton 4.1 For JTNN, we use the reported GitHub implementation from et al. (2019b). There is not obvious stochastic and deterministic parallel considering their method is probabilistic. Therefore, we focus on comparing to fair computational budget by sampling 6600 times from the prior distribution followed by greedy decoding. For fair comparisons to the recursive application, the same corresponding sampling strategies are used, with 100 samples per iteration.
Results. Following reporting practices in the literature, we report the top 3 property scores found by each model. Table summarizes these results. The top 3 property scores found in ZINC-250K are included for comparison. For logP optimization, BBRT-ITNN significantly outperforms all base- line models including JTNN, Seq2Seq. and BBRT-Seq2Seq, BBRT-SeQ2Seq outperforms Seq2Seq, highlighting the benefits of recursive inference for both molecular representations. For QED prop- erty optimization, the two translation models and the BBRT variants all find the same top 3 property state-of-it of-the-art result for QED optimization.
scores, which is
In Figure 2 we report the top 100 logP compounds generated by both BBRT applications relative to its ron-recursvve counterparts and observe significant improvements in logP from using BBRT. We also report a diversity measure the generated candidates for both BBRT models and the top 100 logP compounds in the training data. The JTNN variant produces logP compounds that more

---

Under review as a conference paper at ICLR 2020
diverse than the compounds in the training data, while the compounds generated by Seq2Seq less diverse.
Figure 3] visualizes the top 2 discovered compounds by BBRT-JTNN and BBRT-Seq2Seq under both properties. For logP, while BBRT-JTNN produces compounds with higher property values, BBRT-Seq2Sey's top 2 generated compounds have richer molecular vocabulary. BBRT-Seq2Seq generates compounds with heterocycles and linkages while BBRT-JTNN generates a chain of lin- ear hydrocarbons, which resemble the top reported compounds in GCPN (You et al. 2018b), an alternative graph-based representation. The stark differences in the vocabulary from the top scoring compounds generated by the sequence- and graph- based representations highlight the importance of using flexible frameworks that can ensemble results across molecular representations.
Differences between logP and QED. For logP, BBRT provides a 27% improvement over state-of- the-art for property optimization, while for QED, despite recursive methods outperforming ORGAN, JT-VAE and GCPN, the corresponding non-recursive techniques eechiques-Seq?Seey and JTNN baselines perform just as well as with and without BBRT. We argue this might be result of these models reaching an upper bound for QED values, motivating the importance of the community moving to new metrics in the future (Korovina et al. 2019).
5.2 EMPIRICAL PROPERTIES OF RECURSIVE TRANSLATION.
We perform a sequence of ablation experiments to better understand the effect of various BBRT design choices on performance. We highlight the variability in average logP from translated outputs at each iteration with different decoding strategies (Figure left) and scoring functions (Figure AA right).
On the importance of stochastic decoding. For non-recursive and recursive translation models, stochastic decoding methods outperformed deterministic methods on average logP scores (Fig- ure 4A left) and average pairwise diversity (Figure (48) for generated compounds
function of recursive iteration. Non-greedy search strategies are not common practice in de novo molecular design (Gómez-Bombarelli et al. 2018 Kusner al., 2017 Jin et al. 2019b). While recent work emphasizes novel network architectures and generating diverse compounds using latent variables (Gómez-Bombarelli et al. 2018 Kusner et al.) 2017 Jin al.] 2018), we identify an important design choice that typically has been underemphasizer This trend has also been observed in the natural language processing (NLP) literature where researchers have recently highlighted the impor- tance of well-informed search techniques (Kulikov et al. 2018).
Regardless the decoding strategy, we observed improvements in mean logP with iterations (Figure A when using BBRT. When optimizing for logP, logP scoring function quickly discovers the best scoring compounds while secondary scoring functions improve logP at a slower rate and do not converge to the same scores (Figure AA right). This trade-off highlights the role of conflicting molccular design objectives.
For Figure AA the standard deviation typically decreased with iteration number Property values concentrate to a certain range. With property values concentrating, it is reasonable to question whether BBRT produces compounds with less diversity. In Figure 4B show average pairwise diversity of translated outputs per recursive iteration across 3 decoding strategies and observe decay in diversity for logP. For the best performing decoding strategy, the top-5 sampler, diversity decays from about 0.86 after single translation to about 0.78 after n = 25 recursive translations. This decay may be product of the data- higher logP values tend to be less diverse than a random set of compounds. For QED (Figure 4 right), we observe limited decay. Differences in decay rate might be attributed to task variability, one being explorative and the other interpolative.
5.3 INTERPRETABLE, USER-CENTRIC OPTIMIZATION
For project teams in drug design, the number of generated suggestions can be a practical challenge to prioritization of experiments. We found that interpretable paths of optimization facilitate user adoption.
Molecular traces. BBRT generates sequence of iterative, local edits from initial compound to an optimized one. We call these optimization paths a molecular trace (Figure5A)) We use the Min Delta Sim scoring function to generate traces that have the minimum possible structural changes,
8

---

Under review as a conference paper at ICLR 2020
legf
OED
Mol Wt.
Rot. Bonds
Sim.
A
Imput
3,146
0.566
262.08
1.0
KM
XEI- 960
3,503
0.531
339,99
0.439
MEA270
1R 2540
2.227
234.00*
227
ARV-ESI
31
2,312
0.767'
272.12
0:311
474473
015287
3.17
0.531
339.99
0,771*
Figure 5: A. Generated molecular trace by ranking intermediate outputs by the maximum pairwise Tanimoto similarity. B. An example molecular breakpoint. Alternative translations are considered from compound (2) each with its own design trade-offs.
while still improving logP. While some graph-based approaches (Jin et al. 2018 You et al. 18b Kearnes et al. 2019) can return valid, intermediate graph states that are capable of being interro- gated, we liken our molecular traces to a sequence of Free-Wilson analysis (Free & Wilson 1964) steps towards optimal molecules. Each step represents a local model built using molecular sub- graphs with the biological activity of molecules being described by linear summations of activity contributions of specific subgraphs. Consequently, this approach provides interpretability within the chemical space spanned by the subgraphs (Eriksson et al. 2014).
Molecular breakpoints. Molecular design requires choosing between conflicting objectives. For example, while increased logP is correlated with poor oral drug-like properties and rapid clearance from the body (Ryckmans et al., 2200)), increasing QED might translate to a compound that is structurally dissimilar from the seed compound, which could result in an inactive compound against a target. Our method allows users to "debug" any step of the translation process and consider alternative steps, similar to breakpoints in computer programs. In Figure BB, we show an example from an BBRT-Seq2Seq model trained to optimize logP. Here, we revisit the translation from step (2) step (3) in Figure by considering four alternatives picked from 100 stochastically decoded compounds. These alternatives require evaluating the trade-offs between logP, QED, molecular weight, the number of rotational bonds, and chemical similarity with compound (2).
5.4 IMPROVING SECONDARY PROPERTIES BY RANKING
We consider secondary property optimization by ranking recursive outputs using a scoring function. This function decides what compound is propagated to the next recursive step. We apply BBRT to Seq2Seq modeling (BBRT-SeQ2Seq) and use the trained QED translator described [Section 5. The inference task is optimize QED and logP the primary and secondary properties respectively. We compare scoring outputs based on QED and logP. In Figure we compute the average logP per recursive iteration for set of translated compounds across three decoding strategies. Dotted lines optimize logP as the scoring function while the solid lines optimize QED. For both scoring functions, we report the maximum logP value generated. For all decoding strategies, average logP reaches higher values under scoring by logP relative to scoring by QED. In Figure (B. we plot average QED values using the same setup and observe that optimizing logP still significantly improves the QED values of generated compounds. This method also discovers the same maximum QED value as scoring by QED. This improvement, however, has trade-offs in the limit for average QED values generated. After = 15 recursive iterations the averuge QED values of the generated compounds under a logP scoring function converge to lower values relative to QED values for compounds scored by QED for all three decoding strategies. We repeat this experiment with JTNN and show similar effects (Figure 8)
Secondary property optimization by ranking extends to variables that are minimum loosely posi- tively correlated. For QED optimization, the average logP value for unoptimized QED compounds is 0.30 while for optimized QED compounds the average logP value is 0. 1..4. Additionally, QED compounds in the target range [0.9 1.0] in the training data had a positive correlation between its logP and QED values (Spearman rank correlation; p 0.07 P 0.026). This correlation did

---

Under review as a conference paper at ICLR 2020
logp
QED
beam Logs
losP
Beam, QRO
Beam CED
Red Top Loc?
Ard Tog Logg
Red Top 020
And Tog 2. OED
And Top Log G5D
Red Tog 5, loop
Rtr Top 5, ocD
llrration
Figure 6: Applying BBRT to multi-property optimization. QED is the primary target and logP the secondary property. A: Average logP as function of recursive iteration for three decoding strategies with primary and secondary property scoring functions. B: Average QED as a function recursive iteration for three decoding strategies with primary and secondary property scoring functions.
not hold for QED compounds in the range [0.7 0.8] unoptimized QED compounds (p 0.007, P < 0.8).
FUTURE WORK
We develop BBRT for molecular optimization. BBRT is simple algorithm that feeds the output of translation models back into the same model for additional optimization. We apply BBRT to well-known models in the literature and produce new state-of-the-ar results for property optimiza- tion tasks. We describe molecular traces and user-centric optimization with molecular breakpoints. Finally, we show how BBRT can be used for multi-property optimization. For future work, we will extend BBRT to consider multiple translation paths simultaneously. Moreover, as BBRT is limited by the construction of labeled training pairs, we plan to extend translation models low-resource settings, where property annotations expensive collect.
REFERENCES
Mark Ashton, John Barnard, Florence Casset, Michael Charlton, Geoffrey Downs, Dominique Gorse, John Holliday, Roger Lahana, and Peter Willett. Identification of diverse database subsets using Property-Basee and Fragment-Band Based molecular descriptions. Quant. Struct, -ct. Relat., December 2002.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning Align and Translate. arXiv.org, September 2014.
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An Actor-Critic Algorithm for Sequence Prediction. Interna- tional Conference on Learning Representations, 2017.
Richard Bickerton, Gaia Paolini, Jérémy Besnard, Sorel Muresan, and Andrew Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90, 2012.
Nicolas Boulanger- Lewandowski, Yoshua Bengio, and Pascal Vincent. Audio Chord Recognition with Recurrent Neural Networks, ISMIR, 2013.
Hanjun Dai, Yingtao Tian, Bo Dai, Steven Skiena, and Song. Syntax- variational autoen- coder for structured data. In International Conference Learning Representations, 2018.
Mats Eriksson, Hongming Chen, Lars Carlsson, J. Willem M. Nissink, John G. Cumming, and Ingemar Nilsson. Beyond the Scope of Free-Wilson Analysis. 2: Can Distance Encoded R-Group Fingerprints Provide Interpretable Nonlinear Models? Journal of Chemical Information and Modeling, 54(4):1 2014. ISSN 1549-9596. doi: 10.1021/ci500055.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical Neural Story Generation. ACL, pp- 889-898, 2018.
10

---

Under review as a conference paper at ICLR 2020
Spencer M Free and James W Wilson. A Mathematical Contribution to Structure-Activity Studies. Journal of Medicinal Chemistry, 7:395-399, July 1964.
Rafael Gómez-Bombarrlli. Jennifer Wei, David Duvenaud, José Miguel Hernández-Lobato, Timothy D Hirzel,
Benjamín Baaichez-Lenggling Dennis Sheberla, Jorge Ryan P Adams, and Alán Sapuru-Guzie Automatic Chemical Design Using a Data-Driven Con- tinuous Representation of Molecules. ACS Central Science, 4(2):668 276, January 2018.
Alex Graves. Sequence Transduction with Recurrent Neural Networks. arXiv.org, November 2012.
Gabriel Lima Guimaraes, Benjamin Sanccee--engeelngg Carlos Outeiral, Pedro Luis Cunha Farias, and Alán Aspuru-Guzai Guzik. Objective-Reinfont Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation Models. arXiv.org, May 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, November 1997. ISSN 0899-7667.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. The Curious Case of Neural Text De- generation. arXiv.org, April 2019.
John J Irwin, Teague Sterling. Michael M Mysinger, Erin S Bolstad, and Ryan G Coleman. ZINC: A Free Tool to Discover Chemistry for Biology. Journal of Chemical Information and Modeling, 52(7):1757-7688, 2012.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junetion tree variational autoencoder for molecular graph generation. In International Conference on Machine Learning, pp. 2328 2337,, 2018.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Multi-resolution Autoregressive Graph-to- Graph Translation for Molecules. arXiv.org, June 2019a.
Wengong Jin, Kevin Yang, Regina Barzilay, and Tommi Jaakkola. Learning multimodal graph-to- graph translation for molecule optimization. In International Conference Learning Represen- tations, 2019b.
Steven Kearnes, Li Li, and Patrick Riley. Decoding Molecular Graph Embeddings with Reinforce- ment Learning. arXiv.org, April 2019.
Sunghwan Kim, Paul A Thiessen, Evan E Bolton, Jie Chen, Gang Fu, Asta Gindulyte, Lianyi Han, Jane He. Siqian He, Benjamin A Shoemaker, et al. Pubchem substance and compound databases. Nucleic acids research, 44(D1):D1202-D12215 2015.
Ksenia Korovina, Sailun Xu, Kirthevasan Kandasamy, Willie Neiswanger, Barnabas Poczos, Jeff Schneider, and Eric Xing. ChemBO: Bayesian Optimization of Small Organic Molecules with Synthesizable Recommendations. arXiv.org, August 2019.
Christian Kramer, Julian E Fuchs, Steven Whitebread, Peter Gedeck, and Klaus R Liedl. Matched Molecular Pair Analysis: Significance and the Impact of Experimental Uncertainty. Journal of Medicinal Chemistry, 57(9):386-33822 April 2014.
Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alán Aspuru- Guzik. SELFIES: a robust representation of semantically constrained graphs with an example application in chemistry. arXiv.org. May 2019.
Hugo Kubinyi. Free wilson analysis. theory, applications and its relationship to hansch analysis. Quantitative Structure-ctivity Relationships, 123:132 1988.
Ilya Kulikov, Alexander H Miller, Kyunghyun Cho, and Jason Weston. Importance of a Search Strategy in Neural Dialogue Modelling. arXiv.org, November 2018.
Matt J Kusner, Brooks Paige, and José Miguel Hernández-Lobato. Grammar variational autoen- coder. In International Conference on Machine Learning, pp. 1945-1954, 2017.
Jiwei Li and Dan Jurafsky. Mutual Information and Diverse Decoding Improve Neural Machine Translation. arXiv.org, January 2016.
11

---

Under review as a conference paper at ICLR 2020
Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. Learning Deep Generative Models of Graphs. arXiv.org, March 2018
Jonas Mueller, David Gifford, and Tommi Jaakkola. Sequence to better sequence: continuous re- vision of combinatorial structures. International Conference on Machine Learning, pp. 2536- 2544, 2017.
Pavel G Polishchuk, Timur I Madzhidov, and Alexandre Vamek. Estimation of the size of drug- like chemical space based on gdb-17 data. Journal of computer-aided molecular design, 27(8): 675-679, 2013.
Mariya Popova, Olexandr Isayev, and Alexander Tropsha. Deep reinforcement learning for de novo drug design. Science Advances, 4(7))eaap7885, July 2018.
Marco Tulio Ribeiro. Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining KDD 16, 2016.
David Rogers and Mathew Hahn, Extendedd-connectivty fingerprints. Journal of chemical informa- fion and modeling, 50(5):742-754, 2010.
Thomas Ryckmans, Martin P. Edwards, Val A. Horne, Ana Monica Correia, Dafydd R. Owen, Lisa R. Thompson, Isabelle Tran, Michelle F. Tutt, and Tim Young. Rapid assessment of novel series of selective CB2 agonists using parallel synthesis protocols: A Lipophilic Efficiency (LipE) analysis. Bioorganic and Medicinal Chemistry Letters, 2009, ISSN 0960894X. doi: 10. 10.101 bmcl.2009.055. 00.006...
Ari Seff,, Wenda Zhou, Farhan Damani, Abigail Doyle, and Ryan P Adams. Discrete Object Gener- ation with Reversible Inductive Construction. arXiv.org, July 2019.
Shiqi Shen, Yong Cheng, Zhongjun He, Wei Hc, Hua Wu, Maosong Sun, and Yang Liu. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1683-1692, 2016.
lyya Sutskever, Oriol Vinyals, and Quoe V. Le. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems- Volume 2, NIPS' 14, pp. 3104-3112, Cambridge, MA, USA, 2014. MIT Press.
Samo Turk, Benjamin Merget, Friedrich Rippmann, and Simone Fulle. Coupling Matched Molec- ular Pairs with Machine Learning for Virtual Compound Optimization. Journal of Chemical Information and Modeling, 37112)33079-3855 dec 2017. ISSN 1549-9596. doi: 10.1021/acs. jeim.7600298.
Christian Tyrchan and Emma Evertsson. Matched Molecular Pair Analysis in Short: Algorithms, Applications and Limitations. Computational and Structural Biotechnology Journal, 15:86-90, jan 2017. ISSN 20010370. doi: 10.1016fj.shh.2266..222233 12.0000
Ashwin Vijayakumar, Michael Cogswell, Ramprasaath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse Beam Search for Improved Description of Complex Scenes. Thirty-Second AAAI Conference on Artificial Intelligence, April 2018.
Oriol Vinyals and Quoe Le. A Neural Conversational Model. arXiv.org June 2015.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3156-3164 2015.
David Weininger. Smiles, chemical language and information system. introduction method- ology and encoding rules. Journal of chemical information and computer sciences, 28(1):31-30) 1988.
Sean Welleck, Kianté Brantley, Hal Daumé lü,, and Kyunghyun Cho. Non-monotonic sequential text generation. In International Conference on Machine Learning, pp. 6716-6726, 2019.
12

---

Under review as a conference paper at ICLR 2020
Sam Wiseman, Stuart Shieber, and Alexander Rush. Learning neural templates for text generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3174-13-1 2018.
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional pol- icy network for goal-directed molecular graph generation. In Advances in Neural Information Processing Systems, pp. 6410-6421, 2018a.
Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. Graphrnn: Generat- ing realistic graphs with deep auto-regressive models. In International Conference on Machine Learning, pp. 5694-5703 2018b.
Zhenpeng Zhou, Steven Kearne,, Li Li, Richard N Zare, and Patrick Riley. Optimization of Molecules via Deep Reinforcement Learning. Scientific reports, 9(1):10752, July 2019.
A RECURSIVE PENALIZED LOGP EXPERIMENTS
Training details. For the Seq2Seq model, the hidden state dimension is 500. We use 2 layer bidirectional RNN encoder and layer unidirectional decoder with attention (Bahdanau et al. 2017). The model was trained using an Adam optimizer for 20 epochs with learning rate 0.001. For the graph-based model, we used the implementation from Jnn et al. (2019b), which can downloaded from: https:: http://gttttt.tt.. com www.a//wengnggggge www.weonnnnnnnnnnnnnnneeeeeeeeee jinliclrl1nnieenoo wwtprr19--rraahggggg graph2graph
Property calculation. Penalized logP is calculated using the implementation from You et al. (0) Their implementation utilizes RDKit to compute clogP and synthetic accessibility scores.
QED scores are also computed using RDKit.
5
3-
Mean logp
2
logp
1
Max Pairwise Sim
Max Seed
0-
Mol Wt
logP, Rec, Inf.
Max Pairwise Sim, Rec. Inf.
-1
Max Seed Sim, Rec. Inf.
Min Mol Wt. Rec. Inf.
-2
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Iteration
Figure 7: Comparison of scoring functions for BBRT-JTNN. Y-axis mean logP from 900 transla- tions as function of recursive iteration. Dotted lines denote non-recursive counterparts. 'Rec. Inf.' synonymous with BBRT-JTNN.
13

---

Under review as a conference paper at ICLR 2020
0.96
logP
0.90
0.88
Max, Log? Max, QED
Max: LogP
QED
Greedy,
Gready, LogP
Gready, QED
Cready, OED
0.0
0.5
1.0
1.5
2.0
2,5
3.0
3.5
4.0
0.0
0.5
1.2
13
20
2.5
3.0
3.5
4.0
Iteration
Figure 8: Applying BBRT-JTNN to secondary property optimization. QED the primary target and logP is the secondary property. Left: Average logP function of recursive iteration under two scoring functions-QED and logP. Right: Average QED as a function of recursive iteration for same scoring functions.
6.00
6.00
5.75
5.75
Nax Loge
Logl
5.25
5.00
max dvv min dic avg db.
4.50
medium
4.25
righ
Figure 9: A. Applying BBRT-Seq2Seq to three seed sequence sets with 100 samples each. The MaxMin algorithm
used to select samples with varying levels of diversity (max: 0.94, avg: 0.86, and min: 0.78). For each input sequence, we sampled 100 times from a top5 sampler and ranked samples using logP. Standard error reported by averaging over 10 BBRT-Seq2Seq runs with different seed sets. B. Applying BBRT-SeQ2Seq to three seed sequence sets with 100 samples each and varying logP values (low: logP values < 1, medium: values between -1 and 1, high: values 1). Standard error is reported by averaging over 10 runs each with randomly chosen set of seed sequences.
14

---

Under review as a conference paper at ICLR 2020
sim
sim 0.27
4A46
(2) 1.585
(3) 3.198
(4) 4,241
sim 0.40
sim 0.70
4,882
(7) 4,806
4.)4.
(5) 4.350
0.32
sim 0.19
sim 0.65
(9)4466
(10) 4.798
(11)
Figure 10: A molecular trace from optimizing logP using a logP scoring function.
0.6
softmax_randtopp logp
softmax_randtonn maxdeltasim
softmax_ Seftmaxxrandtop5, logp
softmax_randtoon maxdeltasim
0.5
Edit Distance
0.4
0.3
0.2
0
10
15
20
25
Iteration
Comparing pairwise Levenshtein edit distances for generated sequences after running BBRT under two different scoring functions (logP and maximum pairwise Tanimoto similarity) and two different decoding strategies (top 2 and top 5 sampling). Standard errors reported from population 900 sequences per iteration.
Figure 11:
15


Reviews:
[{"review_id": "ry1VC8LqB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a translation-based method for molecular property optimization. It uses a sequence- or graph-based encoder/decoder framework to produce molecules with (hopefully) improved properties, then feeds a subset of these molecules back into the encoder/decoder to generate a new set of molecules; this process is repeated for a fixed number of iterations to arrive at a final set of \"optimized\" molecules. The method is agnostic to the form of the encoder/decoder; the emphasis is on the iterative approach. Additionally, this approach enables visualization of \"molecular\" traces that can reveal pathways between molecules that follow relationships similar to matched molecular pairs. The work extends related work in translation-based property optimization [6]. The paper is well-written and generally easy to read.\n\nThe method is evaluated on two tasks, logP and QED. Both of these are computed properties that have known issues (see the discussion in [3]), but I understand that these properties are used in many publications and are thus easy to compare. The method presented here performs similar to others on a QED task. They claim superior results on the logP task, but I have concerns about the fairness of the comparison since logP can be exploited by very simple models if there are no limits on the size of the generated molecules (or, similarly, the number of tokens/generative steps allowed for each molecule). Additionally, the authors claim to perform multi-objective optimization but do not actually do this.\n\nThe iterative nature of this method is very interesting. However, my concerns about the types of experiments and comparisons that were done (see below for more details) are big enough that I cannot approve this paper in its current form. Weak reject.\n\nSpecific notes (starting with page number):\n\n- 1: \"potential druggable candidates\" does not make sense; compounds are not \"druggable\" (their targets are), although they may be \"drug-like\".\n- 2: Consider citing Kramer et al.'s seminal work on matched molecular pairs [1].\n- 3: Please explain what it means for y to \"paraphrase\" x?\n- 5: For your logP experiments, you need to be more clear about how you are comparing to other models. You are guaranteed to get to higher logP values if you can generate larger molecules (more tokens) than the baselines, since logP is essentially linear in the number of carbons. Are you doing something to limit the number of tokens you can generate in each iteration? Or why should I believe these comparisons are fair?\n- 5: In Table 1, note that some literature uses a \"normalized\" penalized logP, while others use the formula directly without a dataset-specific normalization (which can appear to give better results). Can you confirm which you are using here and whether the baseline models are the same?\n- 5: The results in Table 1 would be more compelling if they were not divorced from their starting points. Please include information about the similarity of these molecules to the starting molecule as well as the property delta. Consider an approach like Jin et. al [2], where results were specifically categorized by similarity constraints.\n- 5: \"All models were trained on the open-source ZINC dataset.\" What subset of ZINC are you using?\n- 5: The supplementary figure showing that logP is broken is missing?\n- 5: \"Consistent with the literature we report diversity as...\". Please cite some literature that you are consistent with.\n- 6: \"we sample 100 times from a top-2...\"; does this mean you are doing 100 iterations? Sampling 100 times from the same top-2 sampler doesn't really make sense, but I'm not entirely sure what you are describing here.\n- 7: Figure 4 says these are \"ablation\" experiments. What exactly are you ablating?\n- 8: You state that better performance on logP and similar performance on QED is not known in the literature. In fact, the MolDQN paper [3] calls this out explicitly (and also contains a discussion of bounded vs. unbounded logP).\n- 8: \"Recent RL methods focus on molecular construction and are therefore not well-suited for the generation of molecular traces\"; I disagree with this. RL methods that can start from a predefined graph have the ability to move between compounds, possibly in a way that is orthogonal to traditional similarity-based exploration (see the discussion of \"MDP edit distance\" in [4]). Also note that one of the key features of graph-based generators like [2] and [5] is that all of the intermediate states are valid, so you could do similar molecular traces for interpretability (although your differences are more like MMPs with functional group-level deltas).\n- 9: \"synthetic chemists can carry out the individual steps of a molecular trace...\"; in general this is not true. The known medicinal chemistry transformations are a relatively small set of operations, and your molecular traces are unlikely to capture them in any systematic way. Please avoid making claims like this unless you can back them up with experimental evidence or comparisons to models explicitly trained for synthetic route planning.\n- 9: These experiments are not multi-property optimization. Measuring the value of a secondary property while optimizing a primary property is not the same as optimizing them both simultaneously. The latter requires some strategy for incorporating both property values into the decision function, such as scalarizing (see \"multi-objective optimization\" on Wikipedia).\n\nReferences:\n\n[1] Kramer, C. et al. Learning Medicinal Chemistry Absorption, Distribution, Metabolism, Excretion, and Toxicity (ADMET) Rules from Cross-Company Matched Molecular Pairs Analysis (MMPA). J. Med. Chem. 61, 3277–3292 (2018).\n[2] Jin, W., Barzilay, R. & Jaakkola, T. Junction Tree Variational Autoencoder for Molecular Graph Generation. arXiv [cs.LG] (2018).\n[3] Zhou, Z., Kearnes, S., Li, L., Zare, R. N. & Riley, P. Optimization of Molecules via Deep Reinforcement Learning. Sci. Rep. 9, 10752 (2019).\n[4] Kearnes, S., Li, L. & Riley, P. Decoding Molecular Graph Embeddings with Reinforcement Learning. arXiv [cs.LG] (2019).\n[5] You, J., Liu, B., Ying, R., Pande, V. & Leskovec, J. Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation. arXiv [cs.LG] (2018).\n[6] Jin, W., Yang, K., Barzilay, R. & Jaakkola, T. Learning Multimodal Graph-to-Graph Translation for Molecular Optimization. arXiv [cs.LG] (2018).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "r1liUowJtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors frame molecule optimization as a sequence-to-sequence problem where a source molecule is translated to a target molecule with improved properties. The authors extend existing methods for improving molecules by applying them recursively over multiple rounds, and show that it is beneficial for optimizing logP but not QED. An advantage over existing methods is that the trajectory of optimized molecules is interpretable. Altogether, I find the paper borderline: it is clearly written but the methodological contribution is incremental, some citations to related work missing, and some parts of the results section are weak. Detailed comments below.\n\nMajor comment\n============\n1. Framing optimizing as a sequence to sequence problem is not new. As described in the related work section, the BBRT is closely related to Jin et al. However, it is not clearly described what the major improvement over Jin et al is. Please clarify ‘their inference method restricts the framework’s application to more general problems.’.  The method is also closely related to Zou et al (https://www.nature.com/articles/s41598-019-47148-x) and Mueller et al (http://proceedings.mlr.press/v70/mueller17a.html), which are not cited in the text. Zou et al used RL to learn to optimize molecules by mutating existing molecules. Mueller et al used Seq2Seq to optimize the sentiment of sentences. Please cite these papers and discuss why BBRT is better.\n\n2. Please compare to ChemBO (http://arxiv.org/abs/1908.01425). The current baselines are one-shot in that they are proposing a batch of molecules once without using the acquired target function label to propose subsequent batches. ChemBO optimizes a target function such as logP over multiple rounds similar to recursive BBRT approach, and should therefore be included as a baseline. Another suitable baseline would be performing BO in the latent space by applying Gomez et Bombarelli recursively (embed molecule; optimize GP in embedding space; decode molecule; iterate).\n\n3. The method names (Graph2Graph, Seq2Seq, R-Graph2Graph, R-Seq2Seq, BBRT-JTNN, …) are not defined in section 5.1-baselines, and used inconsistently. Is JTNN the same as Graph2Graph and does BBRT mean recursive (R-)? This makes is hard to follow the results section.\n\n4. Section 5.1: How does the performance depends on the initial seed of sequences? How sensitive is it i) to the choice of the diversity cutoff, and ii) to the target value of the initial molecules? \n\n5. Fig 4a, right: Is is expected that logP increases fastest when using it as a scoring function. Please show instead QED vs. the number of iterations. QED combines several molecular properties, including logP, and is therefore more suited for quantifying drug likeness.\n\n6. ‘Differences between logP and QED.’ I do not understand this section. Please clarify the goal of an explorative vs interpolative task? Are molecules with the highest QED in the training dataset? Motivate why BBRT does not achieve a higher QED in table 1?\n\n7. ‘The distinction in the vocabulary highlights the usefulness …’. Is the conclusion that representing molecules as sequences is better than representing them as graphs? This would contradict several recent papers on graph-based representations. You are only comparing the top molecules. Is there a significant difference in the complexity between the top 100 (for example) molecules?\n\n8. Section 5.3 is verbose and can shortened to a few sentences saying that applying edits to molecules recursively makes the model interpretable. How do traces look like when logP is used a  selection criteria? How does the trace of the best molecule shown in figure 3 look like? What is the average edit distance between molecules are and intermediate molecules valid? Are transitions plausible?\n\n9. Section 5.4: You are optimizing a single objective (e.g. logP) while reporting in parallel a second objective (e.g. QED). This is not multi-objective optimization, where multiple objectives are optimized in parallel. Optimizing a single objective while reporting a second objective can be also done with methods other than BBRT. Please clarify the take-away message of this paragraph or remove it.\n\n\nMinor comments\n=============\n10. Introduction: ‘discrete and unstructured’. Why unstructured? I would say that molecules are structured--they must follow a certain grammar to be valid.\n\n11. Introduction: ‘treating inference as a first class citizen’ is unclear since ‘inference’ is undefined. Either remove this sentence or clarify.\n\n12. Please discuss that BBRT is limited by the need of a labeled dataset for constructing training pairs.\n\n13. Section 5.1, ‘Similar computational budget’. How did you quantify the computation budget?\n\n14. Section 5.1, ‘In Fig 2, we report’. Do you mean Fig 3? Same as with ‘Fig 3’ in the following paragraph.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Skl3e32vcB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a novel approach to generating molecules using Black Box Recurrent Translation.\nThe authors uses existing machine-translation inspired schemes to generate new, similar, molecules with better properties according to some measure.\nThen, the recursion takes the top K best molecules and runs another iteration to generate even better molecules, ad infinitum.\nThe authors use the newly introduced SELFIES strings as vocabulary for generation.\nThe authors also analyze the decoding strategy, and how the process generates interpretable molecular traces.\nRelating to wetlab work, having a molecular trace available from the recursive translation scheme is valuable for drug-sythesis.\nThe authors also show that this technique can optimize multiple properties at once.\n\nI am leaning towards an accept for this paper, since not only does the technique presented seem general, the authors does in depth analysis into the model and how it affects drug discovery.\n- Recursive black box translation seems to be widely applicable to new models.\n- The model seems to reach a significantly better state of the art on the metrics proposed.\n- None of the baselines seem to use SELFIES as the string of choice.\n  This means it's difficult to tell how much the \"Blackbox recursive\" part of the algorithm adds to the model.\n  An ablation experiment without BBRT might inform us of how much of the benefit is due to the molecule representation (Fig 4A reports the mean, but it would be good to have the same metric as Table 1).\n- The authors provide an in depth discussion about how having molecular traces would hhelp in drug design.\n  This makes the tool seem more widely appealing and useful.\n\nA few questions would clear up the strengths of the paper:\n- Is there a connection to the backtranslation work in Lample 2018? (Phrase-Based & Neural Unsupervised Machine Translation)\n  It seems like a similar idea - except in this domain, the target language and source language are the same.\n- How can there be multiple scoring functions? \n  Were they combined in one run, or were these separately optimized runs? Are these only used in Figure 4?\n- Why would beam search do less well than stochastic? \n  Is it because during recursive translation, the beam search variants have low diversity?\n  Then, training with stochastic decoding and generation with a beam search should do even better, right?\n  This would highlight that the advantage of stochastic decoding is really online in the context of recursive translation, not generally.\n- What is the point of Fig 4A right? Why do we expect that maximizing non-logP properties will increase mean logP?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1eRIAw2YB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper builds on existing translation models developed for molecular optimization, making an iterative use of sequence to sequence or graph to graph translation models by wrapping them in a meta-procedure. The primary contribution is really just to apply the translation models iteratively, i.e., feeding translation outputs from the models back in as inputs for retranslation. A few strategies are introduced to score / rank candidates before they are chosen for retranslation. The overall idea is very simple, and is likely to work in some basic cases where the property has a natural \"additive\" nature, e.g., logP that you can improve by adding functional groups. This is recognized but not really controlled in the paper except for selecting for input similarity before retranslating. Moreover, I don't think that you really ever want to just maximize logP for any drug so this particular task is a bit artificial in the first place. Other properties are not additive in the same sense, e.g., drug likeness or QED, and the method doesn't appear to improve it (though, to be fair, there may be a ceiling effect for QED in particular). \n\nOne of the main ways that one can control the final output in the iterated translation process is by judiciously selecting or ranking candidates for retranslation. The authors use essentially the score from the model itself, similarity to input, and some basic chemistry metrics to do that. Wouldn't it be much better to train a separate ranking method to guide the iterative steps? \n\nThe empirical results are clean though not convincing (see the logP discussion above). Additional properties should be included to demonstrate that the method might actually have some practical value, i.e., generalize beyond additive logP. Multi-property optimization would be one possible setting since de novo models have a hard time to reach intersections of different property constraints. Abstractly, one could imagine that an iterative, successively guided approach could work well. The proposed approach in the paper is somewhat undeveloped. It merely uses a translation model for the primary property, and ranks candidates by the other. This is unlikely to get you to any challenging intersections. Also, since logP was always one of the properties effectiveness in this regard is not really demonstrated either. A slightly more sophisticated approach might use relaxed, separately trained ranking models in intermediate steps, successively tightened towards the intersection as the iteration progresses. E.g.,\n\nBrookes et al., Design by adaptive sampling, arXiv:1810.03714\n\nThe paper is clearly written but for such a simple method one would need really convincing results and experiments. Maybe better as a workshop submission?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a simple method for improving molecular optimization with a learned model. The method operates by repeatedly feeding generated molecules back through an encoder decoder pair trained to maximize a desired property. Reviewers liked the simplicity of the method, and found it interesting but ultimately there were concerns about the metrics used to evaluate the method. Reviewers 3 and 4 both noted issues with the log P (and penalized log P) metric, noting that it is possible to artificially increase both metrics in a way that isn't useful in practice. During the discussion phase, Reviewer 4 constructed a specific example where simply adding long carbon chains to a molecule would yield a linear increase the penalized log P metric, and noted that the "best molecules" found by the method in Figure 3 also have extremely long carbon chains (long carbon chains are not generally desirable for drug discovery). 
I recommend the authors resubmit after finding a better way to evaluate that their method generates molecules with more useful properties for drug discovery.
Decision: Reject