Fragment-Based Sequential Translation for Molecular Optimization
Authors: Benson Chen
Xiang Fu
Tommi S. Jaakkola
Regina Barzilay
Venue: ICLR 2022 Submitted (2022)
Keywords: molecular optimization
molecular generation
drug discovery
reinforcement learning

Abstract: Searching for novel molecular compounds with desired properties is an important problem in drug discovery. Many existing frameworks generate molecules one atom at a time. We instead propose a flexible editing paradigm that generates molecules using learned molecular fragments---meaningful substructures of molecules. To do so, we train a variational autoencoder (VAE) to encode molecular fragments in a coherent latent space, which we then utilize as a vocabulary for editing molecules to explore the complex chemical property space. Equipped with the learned fragment vocabulary, we propose Fragment-based Sequential Translation (FaST), which learns a reinforcement learning (RL) policy to iteratively translate model-discovered molecules into increasingly novel molecules while satisfying desired properties. Empirical evaluation shows that FaST significantly improves over state-of-the-art methods on benchmark single/multi-objective molecular optimization tasks. 

Under review as a conference paper at ICLR 2022
FRAGMENT-BASED SEQUENTIAL TRANSLATION FOR MOLECULAR OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Searching for novel molecular compounds with desired properties is an impor- tant problem in drug discovery. Many existing frameworks generate molecules one atom at time. We instead propose a flexible editing paradigm that gener- substructures of
ates molecules using learned molecular molecules. To do so, we train variational autoencoder (VAE) to encode molec- ular fragments in a coherent latent space, which we then utilize as a vocabulary for editing molecules to explore the complex chemical property space. Equipped with the learned fragment vocabulary. we propose Fragment-bass based Sequential Translation (FaST), which learns a reinforcement learning (RL) policy to iter- atively translate model-discovered molecules into increasingly novel molecules while satisfying desired properties. Empirical evaluation shows that FaST sig- nificantly improves over state-of-the--ar methods on benchmark single/multi- objective molecular optimization tasks.
geggments-meainngg
INTRODUCTION
Molecular optimization is a challenging task that is pivotal to drug discovery applications. Part of the challenge stems from the difficulty of exploration in the molecular space: not only are there phys- ical constraints on molecules (molecular strings/graphs have to obey specific chemical principles). molecular property landscapes are also very complex and difficult characterize: small changes in the molecular space can lead to large deviations in the property space.
Recent fragment-based molecular generative models have shown significant empirical advantages (Jin et al., 2019a; Podda et al., 2020; Xie et al., 2021) over atom-by-atom generative models in molecular optimization. However, they generally operate over fixed of fragments which limit the generative capabilities of these models. Shifting away from previous frameworks, we learn distribution of molecular fragments using vector-quantized variational autoencoders (VQ-VAB) (van den Oord et al., 2017). Our method builds molecular graphs through the addition and deletion of molecular fragments from the learned distributional fragment vocabulary, enabling the generative model to span a much larger chemical space than models with a fixed fragment vocabulary. Con- sidering atomic edits primitive actions, the idea of using fragments can be thought of options (Sutton et al., 1999; Stolle & Precup, 2002) as temporal abstraction to simplify the search problem.
We further introduce novel sequential translation scheme designed for fragment- based molecular optimization. We start the molecular search by translating from known active molecules and store the discovered molecules new potential initialization states for subsequent searches; we incorporate a delete action in our model, enabling our method to backtrack to good molecular states. Previous works optimize molecules either by generating from seratch molecules, which is inefficient in finding high-quality molecules and often discovering molecules lacking novelty/diversity. Our proposed framework addresses these deficiencies since method is (1) very efficient in finding molecules that satisfy property constraints as the model stay close to the nigh-property-scomm chemical manifold; and (2) able to produce highly novel molecules with our flexible learned fragment vocabulary and a sequence of fragment-based editing.
single translation from known
Combining the advantage of distributional fragment vocabulary and the sequential translation scheme, we propose Fragment-based Sequential Translation (FaST), which is realized by an RL policy that proposes fragment addition/deletion generate molecules under various objectives such as property constraints, novelty constraints, and diversity constraints. The main contribution of this paper includes:
given molecule. Our proposed method can

---

Under review as a conference paper at ICLR 2022
1. We demonstrate way to learn distributional molecular fragment vocabulary through VQ- VAE and the effectiveness of the learned vocabulary in molecular optimization.
2. We propose a novel molecular search scheme of sequential translation, which gradually improves the quality and novelty of generation through backtracking and stored frontier.
3. We implement RL policy combining the fragment vocabulary and the sequential translation scheme that significantly outperforms state-of-the-art methods in benchmark single/multi-objectee molecular optimization tasks.
RELATED WORK
Molecular generation and optimization. Early works molecular optimization build genera- tive models on both SMILES/SELFFES string (Gómez-Bombaree et al., 2018; Segler et al., 2018 Kang & Cho, 2018; Krenn et al., 2020; Nigam al., 2021b; Shin al., 2021), and molecular graphs (Simonovsky & Komodakis, 2018; Liu et al., 2018; Ma et al., 2018; De Cao & Kipf, 2018; Samanta al., 2020; Mercado et al., 2021) and generate molecules sharacter-by-charae Jin et al. (2018) generates graphs as junction trees by considering the vocabulary as the set of atoms or predefined rings from the data; Jin et al. (2020) use the same atom+ring vocabulary generate molecules by augmenting extracted rationales of molecules.
node-by-me
Generating molecules with fixed molecular fragment vocabulary is a well-established idea in traditional drug design (Erlanson, 2011), and has been recently explored through deep learning models (Podda et al., 2020; Xie et al., 2021: Kong et al., 2021; Fu et al., 2021; 2020; Leguy et al., 2020), outperforming previous atom-level models. Recent yyntheszabbilityawarr models can also generate single-step reaction (Bradshaw et al., 2019) and molecule synthesis graphs (Bradshaw et al., 2020) based on a fixed reactant pool. However, the fixed fragment vocabularies used by these models, which are typically small and predefined a priori, limit the chemical space spanned by the models. In our work, we utilize a learned molecular fragment vocabulary, which is obtained by training a VQ- IQ-VAE on a large set of fragments extracted from ChEMBL (Gaulton et al., 2012). By sampling fragments from the learned distribution, our model can span a much larger chemical space than methods using a fixed vocabulary (visualized in Figure 3a, Figure 3b).
Sequential generation of molecules. Guimaraes et al. (2017); Olivecrona et al. (2017); You et al. (2018); Zhou et al. (2019) frame the molecular optimization problem as a reinforcement learning problem, but they generate on the atom/character level and from scratch each time, reducing the efficiency of the search algorithm. Jin al. (2019h) 1b)) uses graph-to-graph translation model for property optimization. However, it requires a large number of translation pairs to train, which often involves expert human knowledge and is expensive to obtain. Others have used geneticlevolutionary algorithms to tackle this problem (Nigam et al., 2020; 2021a), which performs random mutations on chemical strings. Although these methods use learned discriminators to prune sub-optimal molecules, the random mutation process can become inefficient in searching for good molecules under complex property constraints. Xie et al. (2021); Fu et al. (2021) applies Markov Chain Monte Carlo (MCMC) sampling through editing molecules, while Kong et al. (2021) uses Bayesian op- timization the latent space. While there are extensive studies in exploration strategies for RI. (Pathak et al., 2017; Burda et al., 2019; Ecoffet et al., 2021), diversity/novelty driven molecular gen- eration is under-explored. We train novelty/diversity-aaaa molecules that retain desired properties. Our method initializes searches from model-discovered molecules, which greatly improves the efficiency and diversity of the generated molecules. Our ablation in Section show that our proposed RL framework works well with the learned fragment vocabulary, while simpler search strategy is not able to utilize this powerful editing paradigm.
RL policy to search for novel, diverse
PRELIMINARIES
Message Passing Neural Networks (MPNN) Molecules are represented as directed graphs, where the atoms are the nodes and the bonds are the edges of the graph. More formally, let x (V, E) denote directed graph where v, V are the atoms, and eij E E are the edges of the graph. The network maintains hidden states he,, for each directed edge, where / is the layer index. At each step, the hidden representations aggregate information from neighboring nodes and edges, and captures a larger neighborhood of atoms. Iteratively, the hidden states are updated as:
2

---

limited generative span under fixed Under review as a conference paper at ICLR 2022
Ress
f(hhayi
E
Here, I is parameterized by RNN cells (e.g. LSTM cells (Hochreiter Schmidhuber, 1997) or GRU cells (Chung et al., 2014)), and (v;) is the set of neighbors of v, After T steps of message- passing, the final node embeddings he; are obtained by summing their respective incoming edge embeddings:
he. = (eL
Σ
(2)
(N
The final node embeddings are then summed to get a graph embedding representation hr Eey he.-
Vector-Quantised Variational Autoencoders (VQ-VAE) To learn useful representations of frag- ments, we employ the VQ-VAE architecture (van den Oord et al., 2017), which maps molecule fragment graphs to a discrete latent space through using categorical distributions for the prior and posterior. The VQ- VAE defines dictionary of & embedding elements, (2,...,....s)) E R Given an input x (here the graph for a molecular fragment), let z (x) E RR be the output of the encoder (a MPNN in case). We define I be the same dimension for both encoder output embeddings ze (x) and dictionary embeddings 8s, because input 2g (x) is computed by finding the l2 nearest neighbor dictionary elements for each row of (x)):
2g (x); = Sk; where k = arg min (xr,, - 8yll2 for i == =1:= I,...,d
(3)
This embedding scheme allows us to represent each molecular fragment using a length d vector, where each entry takes value from {1.... k} that corresponds to the dictionary embedding index for that row. The combinatorial vocabulary defined by the VQ-VAE has the capacity to represent ha distinct molecular fragments, which lifts the constraints fragment vocabulary.
Since the discretization step does allow for gradient flow, gradients are passed through the net- work through approximating the gradient from the dictionary embeddings to the encoder embed- dings. Additionally. there is commitment loss that encourages the encoder output embeddings that are similar to those in the dictionary (hence commitment). The total loss of the VAE is the following:
L= =o1gp((120(())
[sg[zc(>)]] s5112 + 3 Σ [1=e(z); sg[su] 1
(4)
Where Sty is the closest dictionary element sy for the ze (z),. Additionally, 8 is hyperparameter that controls for contribution of the commitment term, and ag represents the doop-gradient operator.
METHODS
Molecular Optimization. The goal of molecular optimization is to generate set of high-quality molccules (Constrained set) which satisfy or optimize a set of properties P. High novelty and diversity (detailed in Section 5) also desired for de novo generation applications. We model the molecular optimization problem as a Markov decision process (MDP), defined by the 5-tuple {S, (S,A,p p, Po). where the state space S is the set of all possible molecular graphs. As an overview, our method introduces novel designs over the action space A and the transition model (Section 4.1) by utilizing a distributional fragment vocabulary, learned by a VQ-VAE. We define the reward and initial state distribution, r and Pu (Section 4.2) accordingly for specified tasks and to implement the proposed sequential translation generation scheme. An illustration of our model is in Figure 1.
4.1 LEARNING DISTRIBUTIONAL FRAGMENT VOCABULARY
Molecular Fragments are extracted from molecules in the ChEMBL database (Gaulton et al., 2012). For each molecule, randomly sample fragments by extracting subgraphs that contain
3

---

Under review as a conference paper at ICLR 2022
Step 1
Encoder: B(S
SELFIES Decoder:
D(2)
Step 2
Sample fogmert
Add fregeent
F: Frontier
Sample x-8 IUC
Inttial
C: Discover Set
Sample de ction edge and delete fregmeent
DELETS:
Grow with new molecules
Figure 1: Overview of Fragment- based Sequential Translation (FaST). FaST is trained in two-step fashion. In the first step, we train a VQ-VAE that embeds molecular fragments. the second step, train a search policy that uses the learned latent space starts an episode by sampling a molecule from the frontier set F, which consists of an initial set of starting molecules (Z), and good molecules discovered by the policy (C). The molecule is encoded by an MPNN, which is then used to predict either an Add or Delete action. When the Add action is selected, the model predicts and samples a fragment to attach to that atom. When the Delete action is selected, the model samples directed edge, indicating the molecular fragment to be deleted.
action space. The search policy
as the attachment point and subsequently predicts
ten fewer atoms that have single bond attachment the rest of the molecule. We then a VQ- to encode these fragments into a meaningful latent space. The use of molecular frag- ments simplifies the search problem, while the variable-sized fragment distribution maintains the reachability of most molecular compounds. Because our search algorithm ultimately uses the latent representation of the molecules as the action space, we find that using VQ-VAE with a categorical prior instead of the typical Gaussian prior makes RL training stable and provides good performance gains (Tang & Agrawal, 2020; Grill et al., 2020). The training instability under a normal VAE with Gaussian prior and continuous latents causes failture of the RL training. Our ablation study also shows that the fragment samples from VQ-VAE more diverse than the samples from continuous VAE (Section 6).
Encoder/Decoder We MPNN encoders for any graph inputs, which include both fragments for the VQ-VAE, as well as molecular states during policy learning. The graph models are especially suitable for describing actions the molecular state, they explicitly parametrize the represen- tations of each atom and bond. Meanwhile, the decoder architecture is a recurrent network that decodes SELFIES representation of molecule. We choose recurrent network for the decoder because we do not need the full complexity of graph decoder. Due to the construction scheme, the fragments are rooted trees, and all have single attachment point. As our fragments small in molecular size (S 10 atoms), the string grammar is simple to learn, and we find the SELFIES decoder works well empirically (see Appendix FF for more details).
Adding and deleting fragments as actions. At each step of the MDP, the policy network first takes the current molecular graph input and produces Bernoulli distribution whether add or delete a fragment. Equipped with the fragment VQ-VAE, we define the Add and Delete actions at the fragment-level:
Fragment Addition. The addition action is characterized by (1) probability distribu- tion over the atoms of the molecule: (1d a MLP((h.)), where is the softmax operator. (2) Conditioned on the graph embedding h2 and the attachment point atom Vodd sampled from Padd, we predict a d-channel categorical distribution Pfregment a [MLP(( hz])] E ddxk where each row of Pfragment sums 1. We can then sam- ple the discrete categorical latent Zadd E (1,..., kddd from Pfragment- The fragment add
4

---

Under review as a conference paper at ICLR 2022
is then obtained by deocoding Zadd through the learned frozen fragment decoder. We then assemble the decoded fragment with the current molecular graph by attaching the fragment to the predicted attachment point bodd- Note that the attachment point over the fragment is indicated through the generated SELFIES string.
Fragment Deletion. The deletion action acts over the directed edges of the molecule. A probability distribution over deletable edges is computed with a MLP: (ea)(Ssg) = a (MIP(he::) ) One edge is then sampled and deleted; since the edges are directed, the directionality specifies the the molecule to keep and the fragment to be deleted.
With the action space A defined above, the transition model for the MDP is simply p(s'|s, 1 if applying the addition//elettin action to the molecule results in the molecule s', and p(s' |s, a)
otherwise. The tragment-based action space powerful and suitable for policy learn- ing as it (1) is powered by the enormous distributional vocabulary learned by the fragment VQ- VAE, thus spans diverse set of editing operations over molecular graphs: (2) exploits the meaningful la- tent representation of fragments since the representation of similar fragments are grouped together. These advantages greatly simplify the molecular search problem. We terminate an episode when the molecule fails to satisfy the desired property or when the episode exceeds ten steps.
4.2 DISCOVER NOVEL MOLECULES THROUGH SEQUENTIAL TRANSLATION
We propose sequential translation that incrementally grows the set of discovered novel molecules and use the model-discovent molecules as starting points for further search episodes. This regime of starting exploration from states reached in previous episodes was also explored under the setting of RI. from image inputs (Ecoffet et al., 2021). More concretely, we implement sequential translation with a reinforcement learning policy that operates under the fragment-based action space defined in Section 4.1, while using a moving initial state distribution Po. which is a distribution over molecules the frontier set F = ICC. By starting new search episodes from the frontier set- the union of the initial set and good molecules that are discovered by the RL policy, we achieve efficient search in the chemical space by staying close to the high-quality subspace and achieve novel molecule generation through sequence of fragment- based editing operations to the known molecules. Our proposed search algorithm is detailed in Algorithm
Discover novel molecules and expand the frontier. Our method explores the chemical space with a property-aware and novelty/diversit--wwaa reinforcement learning policy that proposes ad- dition/deletion modifications to the molecular state at every environment step to optimize for the reward r. We gradually expand the discovered set C by adding qualified molecules found in the RL exploration within the MDP. A molecule is qualified if: (1) satisfies the desired properties measured by property scores
Cp(c) = II 1 (score,)()) thresholdp)
PEP
where P the set of desired properties and threshold, is the score threshold for satisfying property p. A molecule x satisfying all desired properties has Cr(x)) = and Cr(ar) = 0 otherwise.
(2) x is novel/diverse compared molecules currently in the frontier F, measured by fingerprint similarity (detailed in Section 5):
CND(#) = {max s sim(x, i) < thresholdmec (Imean(simm,, g)) < thresholda.) (6) BEG
ler
Where s m denotes fingerprint similarity, thresholdnor and thresho dai are predefined sim- ilarity thresholds for novelty and diversity, I and C are the initial set of good molecules and model discovered molecules as defined in previous sections. A molecule that satisfies novelty/diversity criterion has CND (x)
and CND(z) = 0 otherwise.
We use reward of +1 for a transition that results in a molecule qualified for the set C, and discour- age the model from producing invalid molecules by adding a reward of -0.1 for transition that produces an invalid molecular graph !
r(z,a) Cr((x + al)- Cxd(2> + a)) 0.1 1(x + invalid)
where (ar +- a] denotes the molecule resulting from editing x with the fragment addition/deletion action validityy is checked by the chemistry software RDKit.
5

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Molecular Optimization through Fragment-based Sequential Translation (FaST)
1: Input N the desired number of discovered new molecules
2: Input Z the initial set of molecules
3: Input D the pretrained fragment decoder of VQ-VAE
4: Input Cr 5 {0. }} returns if the input satisfies desired properties
Equation (5)
Input CND : S + {0, }} returns the input æ satisfies novelty/diversity criterion
Equation (6)
6: Let C = be the discovered set of molecules
7: Let F = IUG C be the frontier where search is initialized from
8: Let = 0 be the number of episodes
9: while [C] S N do
10: Let L++1
11:
Update (Zu, t) E F according to Equation (8)
12:
Sample initial molecule x = (V, E) from Pinit a(UCB(aa, V)) E F
13:
Let step =
14:
while Cp(x) & step do
I 10 in experiments Encode with (PNN( to get node representation ha; Vo V and graph representation h₂
15:
16:
Sample action type a E (ADD, DELETE) from Paction o [MI.
17:
if
ADD then
18:
Sample from (v)
MLP(().) Vo
19:
Sample fragment encoding Radd from Pfragment MLP(( Boodal)
Section 4.1
20:
Decode fragment y ((zada)
Add fragment y to molecule: x <- y
22:
else
23:
Sample from (aa(e) = o MLLP(he, E Let y be the fragment designated by e, delete fragment +- x
> Section 4.1
24:
25:
if (x) =1 & CND(x) = then
26:
C (U
27:
I-C
28:
Let step + step +1
Initialize search episodes from promising candidates. To bias the initial state distribution po to favor molecules that can derive more novel high-quality molecules, we keep an upper-confidence- bound (UCB) score for each initial molecule in the frontier F. We record the number of times we initiate a search Nxx, t) from a molecule æ E F, and the number of molecules qualified for adding C that is found an episode strating from x: R(x, t). Here Evepo (xx) is the total number of search episodes. The UCB score of the initial molecule m calculated by:
V log(( +1)
UCB(x, R(x, + N(a,t) 8)
(8)
(xx
The probability of molecule in the initialization set being sampled the starting point of new episode is then computed by softmax over the UCB scores: (rnat L 1) Exer ((r.))
(BrUU)((rA)))
To summarize, FaST learns policy that (1) choose good initial molecules to start search episodes; delete subgraph from given state (a molecule in our case); (3) choose what to add through predicting a fragment latent embedding, or what to delete through predicting a directed edge, and remove part of the molecular graph accordingly.
(2) choose add fragment
Although we present our method in this section under the most realistic multi-objective optimization task settings (with experiments in Section 5), method is easily extendable other problem settings by modifying the constraints Cp, CND. and the reward function accordingly. For example, Appendix B for the application of method the standard constrained penalized log P task and Section 5 for multi-objective molecular optimization under different novelty/diversity metrics.
EXPERIMENTS
Datasets. We use benchmark datasets for molecular optimization, which aims to generate ligand molecules for inhibition of two proteins: glycogen synthase kinase-3 beta (GSK33) and c-Jun N- terminal kinase 3 (JNK3). Following previous work (Jin et al., 2020; Xie et al., 2021; Nigam et al., 2021a), we adopt the same strategy of using random forest trained on these datasets the oracle property predictor, and incorporate the additional factors, quantitative estimate of drug-likeliness (QED) (Bickerton al., 2012) and synthetic accessibility (SA) (Ertl & Schuffenhauer, 2009)

---

Under review as a conference paper at ICLR 2022
FAST IP3R
FaSteSac (12341
4000
Teticruls (200))
MASS
3000
2000
Number
1000
200
400
600
800
1000
Number steps (k)
Figure 2: Comparison of the sample complexity of different methods. The x-axis is the number of molecules searched trrough and the y-axis is the number of discovered molecules, where the target is to obtain set of 5,000 molecules that achieves SR 1, Nov 1 and Div = .7. Our method (FaST) achieves the best sample complexity with 71k molecules visited. Fast+50k and MARS+50k are their respective models trained with the same fixed fragment vocabulary extracted from ChEMBL.
optimization objectives. Single property optimization is often a flawed task, because the generator can overfit to the pretrained predictor and generate unsynthesizable compounds.
Evaluation metrics. Following previous works, evaluate our generative model on three target metries, success, novelty and diversity. 5,000 molecules are generated by the model, and the metric scores are computed as follows: Success rate (SR) measures the proportion of generated molecules that fit the desired properties. Novelty (Nov) measures how different the generated molecules are compared the set of actives in the dataset (range [0, 1)), and Diversity (Div) measures how dif- ferent the generated molecules are compared to each other (range [0, 1). PM is the product of the three metrics above (PM SR- Nov Div).
Implementation details. We construct the initial set of molecules for our search algorithm from the rationales extracted from Jin et al. (2020). These rationales obtained through sampling process on the active molecules that tries to minimize the size of the rationale subgraph, while maintaining their inhibitory properties. Rationales for multi-property tasks (GSK3,++JKK3) are extracted by combining the rationales for single-property tasks. Initializing generation with subgraphs is com- monly done in molecular generative models such as Shi et al. (2020) and Kong et al. (2021). We train the RL policy using the Proximal Policy Optimization (PPO, Schulman et al. 2017) algorithm. We find the RL training robust despite both the reward function r and the initial state distribution Pn are non-stationary (i.e., changing during RL training). Hyperparameters used for producing the results in Section 5 and molecule samples from FaST are included in Appendix and Appendix E.
Baseline methods. Rationale-RL (Jin et al., 2020) extracts rationales of the active molecules and then uses RI train completion model that add atoms the rationale in sequential manner to generate molecules satisfying the desired properties. GA+D & JANUS (Nigam et al., 2020; 2021a) two genetic algorithms that use random mutations of SELFIES strings to generate promising molecular candidates; JANUS leverages a two-pronged approach, accounting for mutations to- wards both exploration and exploitation. MARS (Xie et al., 2021) uses Markov Chain Monte Carlo (MCMC) sampling to iterative build new molecules by adding or removing fragments, and the model is trained to fit the distribution of the active molecules. To provide a fair comparison against baselines that do not use rationales, we additionally include a baseline MARS+ Rationale that initialize the MARS algorithm with the same starting initial rationale set used in Rationale-RI. and our method. where possible, we use the numbers from the original corresponding paper.
The evaluation metrics shown in Table FaST significantly outperforms all baselines on all tasks including both single-property and multi-property optimization. On the most challenging task, 30% in the product of the three evaluation metrics. Our model is able to efficiently search for molecules that stay within the constrained property space, and discover novel and diverse molecules by sequentially translating known and discovered active molecules. The MARS+Rationale model, which uses the same rationales the initialization for their search algorithm, does not perform
Performance.
FaST improves upon the previous best model by over

---

Under review as a conference paper at ICLR 2022
Table 1:
FaST outperforms all baselines on both pingle-property and multi-property optimization.
Error bars indicates one standard deviation, obtained from averaging 5 random seeds.
GSK38
GSK3E+QED+SA
Model
SR
Nov
Div
PM
SR
Nov
Div
PM
Rationale-RL
1.00
.334
.888
.474
.699
.402
.893
.251
GA+D
.846
1.00)
.714
.600
.891
1.00
.628
.608
JANUS
1.00
.829
.884
.732
MARS
1.00
.840
.718
.600 (+: .04
.995
.950
.719
.680 (#: .03)
M
.995
.804
.746
.597 .07)
.981
.800
.807
.632 (+ .07)
FaST
1.00
1.00
.905
.95 (1 .000)
1.00
1.00
.861
.811 (1 .001)
JNK3
JNKS+QED+SA
Model
SR
Nov
Div
PM
SR
Nov
Div
PM
Rationale-RL.
1.00
.462
.862
.400
.623
.376
.865
.203
GA+D
.288
.983
.726
.380
.857
.998
.504
.431
JANUS
1.00
.426
.895
.381
MARS
.888
.889
.748
.660 (1: .04)
.913
.948
.779
.674 (#: .2)
MARSeRationale
.976
.843
.780
.642 (1: .04)
.634
.779
.787
.386 (#: .08)
FaST
1.00
1.00
.905
.955 .001)
1.00
.866
.856
.741 (#: .001)
GSK33+JNK3
GSK33+INK3+
Model
SR
Nov
Div
PM
SR
Nov
Div
PM
Rationale-RL.
1.00
.973
.824
.800
.750
.555
.706
.294
GA+D
.847
1.00
.424
.360
.857
1.00
.363
.311
JANUS
1.00
.778
.875
.681
1.00
.326
.821
.268
MARS
.995
.753
.691
.520 (+: .08)
.923
.824
.719
.547 (+ .05)
MARS-Rationale
9966
.843
.780
.642 (+ .04)
.654
.687
.724
.321 (+ .09
FaST
1.00
1.00
.863
.863 .001)
1.00
1.00
.716
.716 CE .0T1)
well compared to the original implementation, which initializes each search with simple "C-C" molecule.
Sample complexity comparison given performance thresholds. Another comparison scheme is to let a model keep generating molecules until achieves a good candidate set under certain per- formance thresholds. Under this evaluation protocol, all models will have the same very similar performance in SR, Nov, and Div. The metric of interest will then be the sample complexity of the algorithm: how many molecules it requires to visit/generate
obtain good candidate set. This places every model under the same regime, allowing each model to generate molecules in novelty/diversity-awaaa
setting. We compare FaST Rationale-RL. and MARS under this setting in Figure 2, where we impose SR=1, Nov=1, Div=.7 for the candidate set. FaST on average searched through 71k molecules in total to gather the 5k proposal set, while Rational-RI. and MARS need to search through 205k and 759k molecules to obtain their corresponding proposal sets. Being a pretrained generative model, Rationale-RL. has steeper slope initially but then slows down to find more good molecules. The flexibility of our learned vocabulary and the RL search strategy lead to the superior performance of FaST, which we further verify through ablation study in Section 6.
Optimize for different novelty/diversity metrics. The Morgan fingerprints used for similarity comparison contain certain inductive biases. Under different applications, different novelty/diversity metrics may be of interest. To demonstrate the viability of our model under any metrics, we train FaST using Atom Pairs (AP) fingerprints (Carhart et al., 1985) the CSK3/+JNK3+EED4SA task. The results, and discussion of the different fingerprint methods, are reported Appendix C. We find that (1) FaST can still find high-quality molecules that are novel and diverse, while the baseline methods suffer a low novelty; (2) FaST trained with AP fingerprints still attains good performance when evaluated under Morgan fingerprints but the reverse is not true. This result shows that Novelty/diversity under AP fingerprints is stricter criterion to satisfy and necessitates noveltyldiversit--waarre during optimization.
Penalized logP Maximization. To demonstrate the wide applicability of our method to any molec- ular optimization task, also include our results the standard constrained penalized logP opti- mization task in Appendix B. We show that our model significantly outperforms all baselines in this task under different constraint levels. We also provide insight the task itself: while this task has
8

---

Under review as a conference paper at ICLR 2022
RSST
FAST
(a)
(b)
Figure 3: (a, b) plots the t-SNE embedding of fragments from generated compounds of our model vs. Rational-RL and MARS. The visualization shows that that our model produces a much more diverse set of fragments, which a proxy for functional groups appearing in generated molecules.
been studied in many previous works, the task, as it is currently defined, not entirely chemically meaningful. Additionally, drawback of this task is that model can achieve high performance by simply generating large molecules. detailed discussion can be found Appendix B.
ABLATION AND ANALYSIS
Diversity of generation. In addition to the fingerprint diversity metrics presented in Section 5, we also examine functional group diversity. We extract all unique molecular fragments of the 5,000 molecules generated for (SK339+NK3+QFE4SA task for each model, and produce t-SNE visual- extracted 1.7k unique fragments
ization of these fragments in Figure 3a and Figure 3b. In total, from our model outputs vs only 1.1k unique fragments for Rational-RI. and 500 unique fragments from MARS. The visualization shows that the fragments in the molecules generated by model spans a much larger chemical space. This confirms the advantages of using a learned vocabulary, compared using fixed set of fragments, we able to utilize much more diverse set of chemical subgraphs. Sampled trajectories (Figure 6) and molecules (Figure 7) are included in Ap- pendix E.
Benefit of distributional vocabulary. To investigate the benefit of using distributional vocabulary, instead of using the pretrained VQ-VAE, also train model using fixed vocabulary of frag- ments, which consists of roughly 50k unique fragments (the same set used to pretrain the VQ-VAE). Figure compares the performance of the models. On average, the model with fixed fragments took 122k steps, while with VQ-VAE only took 71k steps to find a set 5,000 good molecules (72% improvement). We further analyze the benefit of using discrete latents with VQ-VAE rather than continuous latents with a Gaussian prior VAE in Appendix A.
Importance of RL search. We also demonstrate the importance of our RL search policy compared to previous sampling methods. To do so, we run MARS with both the 50k fixed fragments and our VQ-VAE. Figure 2 shows the performance of using the 50k fixed fragment vocabulary compared to the original MARS model which uses a small 1k vocabulary. When the vocabulary is large, MARS exhibits very poor sample complexity. Additionally, sampling strategy proposed in MARS, but this model was altogether unable to successfully generate good candidate molecules. Therefore, we that when the vocabulary is more complex, need better search strategy, highlighting the importance of our RL algorithm.
also implemented
VQ-VAR with the
CONCLUSION
We propose a new framework for molecular optimization, which leverages a learned vocabulary of molecular fragments to search the chemical space efficiently. We demonstrate that Fragment-based Sequential Translation (FaST), which adaptively grows a set of promising molecular candidates, can generate high-quality, novel, and diverse molecules single-property and multi-property op- timization tasks. Ablation study shows that all components of our proposed method contribute its superior performance. The learning of flexible vocabulary complementary module to other research in fragment- based drug design. Incorporating FaST to more practical drug discovery pipelines while taking synthesis paths in mind is an exciting avenue for future work.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Rim Assouel, Mohamed Ahmed, Marwin H Segler, Amir Saffari, and Yoshua Bengio. De- factor: Differentiable edge factorization- based probabilistic graph generation. arXiv preprint w1X1v:1111.0976. 2018. 15
Richard Bickerton, Gaia Paolini, Jérémy Besnard, Sorel Muresan, and Andrew Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90-98, 2012. 6
John Bradshaw, Brooks Paige, Matt J. Kusner, Marwin H. S. Segler, and José Miguel Hernández- Lobato. A model to search for synthesizable molecules. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp- 7935-7947, 2019. URL htwpt: pttttts://prrceeeeee http::///pcceeddeee neur Abstract
httppaapr/221111 2/209//hsh/// - html. 2
John Bradshaw, Brooks Paige, Matt J. Kusner, Marwin H. S. Segler, and José Miguel Hernández- molecule synthesis dags.
Lobato. Barking up the right tree: approach to search In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: http::s://pprceeding..... newr.nnn.s..uuuuee.e neurips. ccc wtl.c//pppae paper/2020/hashti 4 4cc05b35c2 937 c5bd c366f htmll
Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. In 7th International Conference Learning Representations, ICLR 2019, New Or- leans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL http:p//////eeeeeeeeeeeeeee forum? 1 1d=H1 1JJnR5 2
Alice Capecchi, Daniel Probst, and Jean-Louis Reymond. One molecular fingerprint to rule them all: drugs, biomolecules, and the metabolome. Journal of cheminformatics, 12(1):1-15, 2020. 16
Raymond E Carhart, Dennis H Smith, and R Venkataraghavan. Atom pairs as molecular features in atructure-activitty studies: definition and applications. Journal of Chemical Information and Computer Sciences, 24(2):64-73, 1985. 8, 15
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks sequence modeling. arXiv preprint arXiv: 2014.
Nicola De Cao and Thomas Kipf. Molgan: An implicit generative model for small molecular graphs. arXiv preprint arXiv: /1933, 2018. 2
Joseph L Durant, Burton A Leland, Douglas Henry, and James G Nourse. Reoptimization of mdl keys for use in drug discovery. Journal of chemical information and computer sciences, 42(6): 1273- 2002. 15
Adrien Ecoffet, Joost Huizinga. Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580-588) 2021. 2,
Daniel A Erlanson. Introduction to fragment- drug discovery. Fragment-based drug discovery and X-ray crystallography, pp. 1-32, 2011.
Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. Journal of cheminfor- matics, 1(1):1-11, 2009. 6
Tianfan Fu, Cao Xiao, and Jimeng Sun. Core: Automatic molecule optimization using copy & refine strategy. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp- 638-645, 2020. 2
10

---

Under review as a conference paper at ICLR 2022
Tianfan Fu, Cao Xiao, Xinhao Li, Lucas M. Glass, and Jimeng Sun. MIMOSA: multi-constan molecule sampling for molecule optimization. In Thirty-Fifthh AAAI Conference on Artificial Intelligence, AAAl 2021. Thirty- Conference telligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelli- gence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 125-133. AAAI Press, 2021. URL https: //jss///j/s/////////////// aaai. org/ index.ixx.. php/ AAAI/ SaAAAAarriilll. wwt.cceeeeveeeiie
Innovative Applications of Artificial In-
Anna Gaulton, Louisa J Bellis, Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani. et al. Chembl: a large-scale bioactivity database for drug discovery. Nucleic acids research, 40(D1):D110) 1110-1007,,,,,- 2012. 2, 3
Rafael BómezrBombarelli, Jennifer Wei, David Duvenaud, José Miguel Hernández-Lobate Benjamín Sánchez-Lengelingg Dennis Sheberla, Jorge aguilera-Iparraguinn Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using data-driven contin- uous representation of molecules. ACS central science, 4(2):268-276. 2018. 2
Jean-Bastien Grill, Florent Altché, Yunhao Tang. Thomas Hubert, Michal Valko, Ioannis Antonoglou, and Rémi Munos. Monte-carlo tree search as regularized policy optimization. In International Conference Machine Learning, pp. 3769-3778, PMLR, 2020. 4
Gabriel Lima Guimaraes, Benjamin Sanchez-Lengeling, Carlos Outeiral, Pedro Luis Cunha Farias, and Alán Aspuru-Gurs Guzik. Objective-reinforeed generative adversarial networks (organ) for se- quence generation models. 2017. 2
Paul CD Hawkins, A Geoffrey Skillman, Gregory L Warren, Benjamin A Ellingson, and Matthew Stahl. Conformer generation with omega: algorithm and validation using high quality structures from the protein databank and cambridge structural database. Journal of chemical information and modeling, 50(4):572-58) 2010. 15
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997. 3
John J Irwin, Teague Sterling, Michael M Mysinger, Erin S Bolstad, and Ryan G Coleman. Zine: a free tool to discover chemistry for biology. Journal of chemical information and modeling, 52 (7):1757-1768, 2012. 14
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. International conference on machine learning, pp. 2323-2332. PMLR, 2018. 2, 15
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hierarchical graph-to-grape translation for molecules. arXiv preprint wrXiv:1907.12220 2019a.
Wengong Jin, Kevin Yang. Regina Barzilay, and Tommi Jaakkola. Learning multimodal graph-to- graph translation for molecule optimization. In International Conference on Learning Represen- fations, 2019b. 2, 14, 15
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Multi-objective molecule generation using interpretable substructures. In International Conference on Machine Learning, pp. 4849-4859. PMLR, 2020. 2, 6.7
Seokho Kang and Kyunghyun Cho. Conditional molecular design with deep generative models. Journal of chemical information and modeling, 59(1):43-52, 2018. 2
Xiangzhe Kong. Zhixing Tan, and Yang Liu. Graphpiece: Efficiently generating high-quality molec- ular graph with substructures. arXiv preprint 100,:99800.5908, 2021. 2, 7, 14, 15
Mario Krenn, Florian Häse, Kumaa Nigam, Pascal Friederich, and Alan Aspuru- Guzzkk Self- referencing embedded strings (selfies): A 100% robust molecular string representation. Machine Learning: Science and Technology, (4))0445024 2020. 2, 17
Jules Leguy, Thomas Cauchy, Marta Glavatskikh, Béatrice Duval, and Benoit Da Mota. Evomol: a flexible and interpretable evolutionary algorithm for unbiased de novo molecular generation. Journal of cheminformatics, 12(1):1-19, 2020.

---

Under review as a conference paper at ICLR 2022
Christopher A Lipinski, Franco Lombardo, Beryl W Dominy, and Paul J Feeney. Experimental and computational approaches to estimate solubility and permeability in drug discovery and develop- ment settings. Advanced drug delivery reviews, 23(1-3):3-25 1997. 15
Qi Liu, Miltiadis Allamanis, Mare Brockschmidt, and Alexander L Gaunt. Constrained graph vari- ational autoencoders for molecule design. arXiv preprint arXiv: 09076, 2018. 2
Tengfei Ma, Jie Chen, and Cao Xiao. Constrained generation of semantically valid graphs via regularizing variational autoencoders. arXiv preprint /809,0060, 2018. 2
Rocío Mercado, Tobias Rastemo, Edvard Lindelöf, Günter Klambauer, Ola Engkvist, Hongming Chen, and Esben Jannik Bjerrum. Graph networks for molecular design. Machine Learning: Science and Technology, 2(2):025023, 2021. 2021. 2
AkshatKumar Nigam, Pascal Friederich, Mario Krenn, and Alán Aspuru- Guzik. Augmenting ge- netic algorithms with deep neural networks for exploring the chemical space. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. 2, 7, 14, 15
AkshatKumar Nigam, Robert Pollice, and Alan Aspuru-Guzik, Janus: Parallel tempered genetic algorithm guided by deep neural networks for inverse molecular design. 2021a. 2, 6,
AkshatKumar Nigam, Robert Pollice, Mario Krenn, Gabriel dos Passos Gomes, and Alan Aspuru- Guzik. Beyond generative models: superfast traversal, optimization, novelty, exploration and discovery (stoned) algorithm for molecules using selfies. Chemical science, 2021b. 2
Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. Journal of cheminformatics, 9(1):1-14, 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2787.
PMLR, 2017. 2
Marco Podda, Davide Bacciu, and Alessio Micheli. A deep generative model for fragment-based molecule generation. In International Conference on Artificial Intelligence and Statistics, pp- 2240 2250.. PMLR, 2020. 1,2
David Rogers and Mathew Hahn. coeneeddcommectivity fingerprints. Journal of chemical informa- tion and modeling, 50(5):742-754 2010. 14
Bidisha Samanta, Abir De, Gourhari Jana, Vicenç Gómez, Pratim Kumar Chattaraj, Niloy Ganguly, and Manuel Gomez-Rodriguez Nevae: A deep generative model for molecular graphs. Journal of machine learning research. 2020 Apr; 21 (114); 1-33, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. 2017. 7
Marwin HS Segler, Thierry Kogej, Christian Tyrchan, and Mark P Waller. Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS central science, 4(1): 120-131, 2018. 2
Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graphaf: a fowedbaed autoregressive model for molecular graph generation. arXiv preprint arXiv:2001.09382 2020. 7, 14, 15
Bonggun Shin, Sungsoo Park, JinYeong Bak, and Joyce C Ho. Controlled molecule generator for optimizing multiple chemical properties. In Proceedings of the Conference on Health, Inference, and Learning, pp. 146-153, 2021. 2
Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. In International conference on artificial neural networks, pp. 412-422. Springer, 2018. 2
Martin Stolle and Doina Precup. Learning options in reinforcement learning. In International Symposium on abstraction, reformulation, and approximation, pp. 212-223. Springer, 2002.
12

---

Under review as a conference paper at ICLR 2022
Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction reinforcement learning. Artificial Intelligence, 112(1):111-11,, 1999.
Yunhao Tang and Shipra Agrawal. Discretizing continuous action space for on-policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5981-5988, 2020. 4
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. pp. 6306-6315, 2017. 1,3
Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. (MARS): Markov molecular sampling for multi-objective drug discovery. In International Conference on Learning Representations, 2021. 1, 2,
Jiaxuan You, Bowen Liu, Zhitao Ying. Vijay Pande, and Jure Leskovec. Graph convolutional pol- icy network for goal-directed molecular graph generation. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Pro- cessing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 6412-6422, 2018. 2, 14, 15
Zhenpeng Zhou, Steven Kearnes, Li Li, Richard N Zare, and Patrick Riley. Optimization of molecules via deep reinforcement learning. Scientific reports, 9(1):1-10, 2019, 2, 15
13

---

Under review as a conference paper at ICLR 2022
OH
VAE Samplees VO VAE Samples
(a)
(b)
(c)
Figure 4: (a) t-SNE of fragments sampled from trained VAE and VQ-.AE. The fragments sampled from the VAE are tightly clustered, showing much less diversity compared to the fragments sampled from the VQ-VAE. (b,c) Random samples from the VAE (b) and the VQ-VAE (c). "A" denotes the attachment point for the fragment. We see that the samples from the VAE are relatively simple. Meanwhile, the samples from the VQ-VAE are more diverse.
A VOCABULARY LEARNING THROUGH VQ- VAE
To evaluate the benefits of VQ- VAE over a typical VAE trained with Gaussian priors, we train both models, and look the distribution of fragments. Figure 4a compares the t-SNE distributions the two models, where we sample 2,000 fragments from each model. The VAE model has tight clusters, while the VQ-VAE model exhibits a much more diverse set of fragments. We visualize random samples from VAE and VQ-VAE Figure 4, where we see that the samples from VAE are relatively simple and generic fragments, while samples from the VQ- VAE demonstrate diverse patterns. This is because the more generic fragments appear more frequently in real molecules, and a Gaussian prior over the fragment latent space would favor these fragments.
B CONSTRAINED PENALIZED LOGP TASK
To demonstrate the general applicability of our model for any molecular optimization task, we also run our model on another constrained optimization task, here optimizing for penalized octanol-water partition coefficients (logP) scores ZINC (Irwin al., 2012) molecules. The penalized logP score is the logP score penalized by synthetic accessibility and ring size. We use the exact computation You al. (2018), where the components of the penalized logP score are normalized across the entire 250k ZINC training set. The generated molecules are constrained to have similar Morgan fingerprints (Rogers Hahn, 2010) as the original molecules.
Following the same setup as previous work (Jin et al., 2019b; You et al., 2018; Shi et al., 2020; Nigam et al., 2020; Kong al., 2021), we try to optimize the 800 test molecules from ZINC with the lowest penalized logP scores (the initial set Z). Specifically, the task is to translate these molecules into new molecules with the Tanimoto similarity of the fingerprints constrained within 8 E {.4, .6). This task aims for optimizing a certain quantity (instead of satisfying property constraints) and is translation task (need stay close to original molecules rather than finding novel ones). To run FaST on this task, we apply the following changes to the reward function, the qualification criterion, and the episode termination criterion, of FAST. We denote score(s (x) be the penalized logP scoring function, and s im(---) to be the Tanimoto similarity between two molecules:
reward score(z,) score(x;) for any transition from molecule + Zz
C, the discovered set contains all explored molecules that satisfy Equation (5), where the threshold is given by the input parameter 5
We terminate episode when the number of steps exceeds 10.
For each molecule add to G, keep track of its original parent (the molecule from the 800 test molecules). After training, for each of the 800 test molecules, we take the set of translated molecules in G, and select the with the highest property score.
14

---

Under review as a conference paper at ICLR 2022
Table Results on the constrained penalized logP task. FaST significantly outperform all baselines.
ò == 0.4
0.6
Method
Improvement
Success
Improvement
Success
JT-VAB (Jin al., 2018)
0.84 1.45
83.6 %
0.21 0.71
46.4
GCPN (You et al., 2018)
2.49 = 1.30
100.0 %
0.79 + 0.63
100.0 %
DEFactor (Assouel et al., 2018)
3,41 1.67
85.9 %
1.55 1.19
72.6
MolDQN (Zhou et al., 2019)
3,37 1.62
100 %
1.86 1.21
100 %
GraphAF (Shi et al., 2020)
3.74 1.25
100 %
1.95 0.99
100 %
GP-VAE (Kong et al., 2021)
4.19 1.30
98.9 %
2.25 1.12
90.3 %
VJTNN (Jin al., 2019b)
3.55 1.67
2.33 1.17
GA+D (Nigam et al., 2020)
5,93 = 1.14
100 %
3,44 + 1.09
99.8 %
FaST (Ours)
18.09 8.72
100
8.98 6.31
96.9
Tanimoto Similarity: 0.65 penalized log P: 7.86
Figure 5: Sample translation of our model for the constrained penalized logP task (8 = 0.6). The model generates molecule with repeating aromatic rings; though not realistic, this molecule achieves a high score, while having close Tanimoto similarity using Morgan fingerprints.
Results are shown Table 2: our method greatly outperforms the other baselines, but we point out few flaws intrinsic the task. Because the similarity is computed through Morgan fingerprint, which are hashes of substructures, repeatedly adding aromatic rings can often not change the fingerprint by lot. Nevertheless, adding aromatic rings will linearly increase penalized logP score, which allows trivial solutions to produce high scores for this task (see Figure 5). This phenomenon is noted by Nigam et (2020), but they add regularizer constrain the generated compounds to look similar to the reference molecules. Due to the mentioned issues, we believe this task can be reformulated. For instance, one could use different fingerprint method so that the fingerprint similarity easily exploited (see AP (Carhart et al., 1985), MACCS (Durant et al., 2002), or ROCS (Hawkins al., 2010)), size constraints should be incorporated. Nevertheless, provide our results for comparison to other molecular generation methods.
so
In general, the task of optimizing (increasing) the penalized logP scores is entirely meaningful. According to Lipinski's 's rule of five (Lipinski et al., 1997), which are widely established rules to evaluate the druglikeness of molecules, the logP score should be lower than 5. So unbounded optimization of logP has little practical usability. Perhaps better task would be to optimize for all 5 rules in Linpinski's rule of five which includes constraints involving the number of hydrogen bond donors/acceptors and molecular mass.
C DIFFERENT NOVELTY/DIVERSITY METRICS
FaST is capable of optimizing for different noveltyldiversity metrics. In this section, compute the novelty/diversity metrics using atom-pair (AP) fingerprints (Carhart et al., 1985). While Morgan fingerprints have successfully been applied many molecular tasks such as drug screening, it has
15

---

Under review as a conference paper at ICLR 2022
some failure modes (Capecchi et al., 2020). Namely, Morgan fingerprints is often not informative about the size or the shape of the molecules. These properties are better captured in AP finger- prints, AP fingerprints account for all atom pairs, including their pairwise distances. We rum the same experiment on the GSK3 S task described in Section 5, but change the fin- gerprint from Morgan to AP for the novelty/diversity metries. The results shown in Table 3 with comparison to baselines. We observe that our method outperform baselines by a greater margin, especially in the novelty metric. This is not surprising because our model can explicitly optimize for any similarity metric, while the baseline methods are not povelty/diversity-awaaa Interestingly, we find that optimizing for AP fingerprints also yields molecules that score high under Morgan fingerprints for this task (but the converse is not true).
during training.
Table 3: Results the CSK394NNK++QE4SA task using AP fingerprints instead of Morgan fingerprints for novelty/diversity computation.
Success (SR)
Novelty (Nov)
Diversity (Div)
Method
Morgan
AP
Morgan
AP
Morgan
AP
Rationale-RI.
.750
.750
.555
.023
.706
.630
MARS
.923
.733
.824
.077
.719
.644
FaST (Morgan)
1.00
1.00
1.00
.555
.716
.674
FaST (AP)
1.00
1.00
.987
.867
.675
.719
D IMPLEMENTATION DETAILS
Table 4: Hyperparameters for the VQ-VAE.
VQ-VAE Param
Value
hidden size
200
MPNN depth
4
MPNN output size (d)
10
# Dictionary elements (k)
10
Dictionary latent size (7)
10
batch size
32
dictionary loss coef commitment loss coef
1.0
1.0
learning rate # epochs
le-4
10
Table Hyperparameters for training the RL Agent.
Param Name
Value
learning rate
2e-4
0.999
XGAE 0.95
batch size
64
PPO Epoch
3
Agent
param clip
0.2
value loss coef
0.5
entropy loss coef
0.01
le-5
max grad norm
0.5
hidden size
1024
Actor
hidden depth
1
hidden size
256
Critic
hidden depth
1
16

---

Under review as a conference paper at ICLR 2022
Add
Delete
(.61), (.84), (2.9)
65K3p (.67), INK3 (63)), QED (.81), SA (2.8)
GSK3p JNK3) (.66),
Delete
Delete
GsKa (.86), JNG (47), QED [,57], SA (2.5)
GSKB (85), JNK3 (.51), (70), SA (2.5)
GSKBB (.22), INK3 (:56), QED (.72), SA (2.4)
rum
Delete
65K38 (185), INK3 (57),, QED (77)) SA 12.0)
GSK3P (778), INK3 (.54), QED (69%) SA (3.2)
GSK3P (182), JNK3 (:58), QFD (63), SA (3.5)
Add
Delete
GEKaE (.82), INKE [-48), QED (.79), SA [2.4]
GSK3R (.53), JNK3 (533), QED (.80), SA (2:3)
(l2?) JNMS
CgD (999) Ss (6.2
Add
Delete
G5K3D (.6),, JNIE (.55), QED (81), SA (3.7)
(.63), (.66),
GSКзр (62), JNKG (550,, QED (.67), SA (3.9)
Figure 6: Samples of FaST the GSK38+INK++QEDSSA task. Red indicates the atoms and bonds that are eligible for addition/deletion, while green indicates the selected atom or bond.
E SAMPLE TRAJECTORIES AND OPTIMIZED MOLECULES FROM FAST
We provide example molecular optimization trajectories of model the GSK3/HNNK++OED4SA task in Figure 6.
Additional generated moles for the GSK3/+INK3+OED+SA task shown in Figure 7
F DECODER ARCHITECTURE: SELFIES vs SMILES
For VAE, decode into string representations of fragments for several reasons. (1) Graph decoders are complex and requires lot hand engineered rules to work. fact, many graph decoders require their own vocabularies, which makes the decoder architecture (2) Our fragments are small, less than 10 atoms, so a simpler representation the fragments in string form is appropriate. Additionally, because SELFIES strings have a more robust grammar compared to SMILES strings, and has been shown through many past experiments (Krenn al., 2020). Empirically, SELFIES strings gives hetter reconstruction accuracy compared to using SMILES strings for the decoder in the VAE architecture.
complex.
choose to SELFIES strings over SMILES strings,
in Figure 8 that
17

---

Under review as a conference paper at ICLR 2022
Figure Randomly sampled molecules generated by FaST on the GSK33+NNK++EE+SA task.
18

---

Under review as a conference paper at ICLR 2022
Reconstruction Accuracy
0.6
0.4
0.2
SMILES
SELFIES
0.0
100
200
300
400
500
Number of steps (k)
Figure 8: Comparison of reconstruction accuracy when using SMILES string vs SELFIES string.
The decoder trained with SELFIES grammar has better reconstruction accuracy (96% vs 86%).
19


Reviews:
[{"review_id": "xL3BsmvGiM5", "reviewer": "Reviewer_eP4U", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- VQ-VAE is a natural model for encoding discrete objects and produces more diverse structures, which is confirmed by t-SNE plots.\n- The authors use Upper Confidence Bound (UCB) to sample compounds effectively and accelerate the search of compounds with optimized properties.\n- There is an interesting discussion on the usability of logP optimization and the flaws of measuring compound similarity using molecular fingerprints.\n\nCons:\n- The text structure is very similar to the MARS paper, and there is an apparent similarity between these two methods, although there are some significant changes.\n- One of the similarities is the main results table, which includes metrics that do not correctly emphasize the superiority of one method above others. The success rate (SR), which should be the main objective, is equal (almost) 1 for many models, so it is difficult to say if one method is significantly better than others.\n- Besides SR, other metrics describe the novelty and diversity of the generated compounds. These metrics are directly optimized by the proposed model. Are other methods in the comparison trained so that these metrics are maximized? If not, then it is not surprising that FaST wins in this competition. Also, the last metric, PM, is the product of all the other metrics, so suboptimal diversity or novelty scores of other models negatively impact the scoring.\n- To confirm the usefulness of the model, a different comparison should be proposed, or more biological targets should be added. One idea would be to create a benchmark that uses uncertainty prediction models. This way, the novel compounds that cannot be reliably predicted by the activity models would be discarded.\n- In Section 4, some claims are not supported experimentally in the paper: \n  - \"we find that using a VQ-VAE with a categorical prior instead of the typical Gaussian prior makes RL training stable and provides good performance gains\" - the FaST variant with the Gaussian prior should be included in the results table,\n  - \"we find the SELFIES decoder works well empirically\" - this could be further discussed in the ablation studies.\n- More examples of generated compounds would be welcome.\n\nOther comments:\n- In the paragraph about VQ-VAE in Section 3, should it not be $k^d$ instead of $d^k$ distinct fragments?\n- typos: \"specify the the molecule to keep\"", "summary_of_the_paper": "The paper presents a new way of molecular optimization that is based on chemical fragments. The authors employ VQ-VAE to encode a fragment library in a coherent latent space. They argue that this method produces more diverse rationales than a Gaussian VAE. The proposed generative method uses reinforcement learning (PPO) to select modifications of the chemical structure: a fragment decoded by VQ-VAE is added at a chosen attachment point, or a part of the compound is removed. The model retains only modifications that fulfill initial constraints regarding the novelty, diversity, and chemical properties of a generated molecule measured by formerly trained predictive models. The experiments show significantly better performance of the proposed FaST model compared to a diverse set of baseline models.", "summary_of_the_review": "Based on the above comments, I currently recommend rejecting this paper. The novelty of the work lies in incremental changes to existing methods, and I think the modifications are substantial. However, the experiments, in my opinion, do not demonstrate well the impact of these changes.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "fDMZ4gNTvgl", "reviewer": "Reviewer_UFDm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n### Fragment generation model\n Fragment-based generation with a fragment generation model is novel and seems to be a reasonable generative model of molecules. It successfully inherits the benefit of the existing fragment-based generative model while avoiding from the limited generative capability due to the fixed fragment dictionary. I also like the authors validate this approach empirically in Section 6.\n\n### Realistic benchmark task\n I like the authors working on a realistic benchmark task, in addition to the traditional penalized logP optimization task.\n\n## Weaknesses\nI have concerns on the reinforcement learning module and experiments.\n\n### Non-stationary environment\nAs the authors state in the \"implementation details\" paragraph, the environment is non-stationary, i.e., both the initial state distribution and the reward function change during training. This non-stationarity is generally not favorable, and should be avoided if possible, because I don't think RL on non-stationary environments is not well understood. I would avoid the non-stationarity by devising the definition of the environment.\n\n### No theoretical justification\nThe authors employ the UCB score to define the initial state distribution, but I don't see any theoretical background to justify it.\n\n### Reward design\nThe reward is designed so that the agent can discover molecules with desirable properties, while being novel and diverse enough. On the other hand, the agent aims to maximize the cumulative reward. I don't see a reasonable interpretation of the cumulative reward, and I would like the authors to discuss it.\n\n### Potentially unfair experimental design?\nAs the authors state in \"optimize for different novelty/diversity metrics\" paragraph, the baseline models are not novelty/diversity-aware during training. I am not very sure this is a fair comparison, because the existing RL-based method can be novelty/diversity-aware by introducing the proposed reward design (Eq. 7). In my opinion, the main contribution of this paper is the generative model of fragments, and it would be more valuable (to me) if the experiment is designed so that it highlights the benefit of the generative model of fragments by for example using the same RL component across different methods.", "summary_of_the_paper": "The present paper aims to discover novel molecules with desirable properties. One of the main contributions is to compose a molecule fragment-by-fragment, where each fragment is drawn not from a fixed fragment dictionary but from a NN-based pre-trained fragment generator. Other contributions are related to the optimization module using RL techniques; introducing fragment deletion as an action, sequential translation, novelty- and similarity-aware reward design, and initial state distribution.\nThe proposed method is evaluated by ligand generation tasks, where generated molecules are evaluated in terms of success rate, novelty, and diversity. ", "summary_of_the_review": "My recommendation is weak accept.\n\nI like the idea of a generative model of fragments, and the analysis on its benefit in Section 6. However, I am not very convinced by the RL component, because of three concerns raised above. Therefore, I can't aggressively recommend to accept this paper.\n\nMy humble suggestion is to set the generative model of fragments as the main contribution and the RL components as minor ones (or even re-using the existing RL component), because the generative model of fragments itself seems to be novel enough (at least to me) to be presented to the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "XYAm9fd2-Fw", "reviewer": "Reviewer_KpDT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The proposed model outperforms other baselines in the multi-objective molecules optimization benchmark. \n2. Applying VQ-VAE into molecular fragment vocab learning is a properly motivated hypothesis and empirically well-validated. \n3. MolDQN[1] and CMG[2] are other multi-objective optimization models. Better to include them at least in the related work.\n\n[1] Zhou, Zhenpeng, et al. \"Optimization of molecules via deep reinforcement learning.\" Scientific Reports 9.1 (2019): 1-10.\n[2] Shin, Bonggun, et al. \"Controlled molecule generator for optimizing multiple chemical properties.\" Proceedings of the Conference on Health, Inference, and Learning. 2021.\n\n\nWeakness:\n1. In the Encoder/Decoder section on page 4, the authors claimed as \"we ﬁnd the SELFIES decoder works well empirically\". It's not convincing why they choose to use the SELFIES decoder. It'd be good if it's well-motivated and empirically shown. \n2. Minor typo: In eq (6), the outer parenthesis of the diversity may be before the \"<\" sign?\n3. The overall framework is novel, however, it relies on many known methods.", "summary_of_the_paper": "This paper proposes a new reinforcement learning method for molecular optimization. They train VQ-VAE to learn the vocabulary of molecular fragments which enables the optimizer to efficiently search the chemical space. The optimizer consisted of general reinforcement learning and MPNN, and devised sampling schemes to improve performance. By integrating evaluation criteria as a reward function, the proposed model can additionally improve the performance in terms of those criteria. The experiments show not only its superior performance but shorter optimization time.\n", "summary_of_the_review": " Overall it is a well-written paper with good motivation, appropriate method, and supportive experiments. The overall framework is novel, however, it relies on many existing methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UQLYfEoFK0v", "reviewer": "Reviewer_MhuZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Combing variational auto-encoders and RL policy for fragment-based molecular optimization gives a good practical performance, but the novelty contribution is marginal. Their method of encoding fragments into latent space is not particularly novel and has been applied in other works for similar tasks (e.g. [1][2][3]). Fragment-based drug design is also a very common approach that has been widely explored for a long while from the cheminformatics community, and there is nothing unique about the fragmentation or the definition of molecular editing action in this work (For example, there are three editing actions defined in [4] and seven in [5] ). The diversity-aware reinforcement learning methods are also common in NLP community [6]. FaST introduces a similarity regularization in the reward function to increase the diversity, which makes a little sense but makes it easy for the model to fool your defined metrics (the difference between AP-based and Morgan-based similarity calculations isn't that great). Such a high level of Nov. and SR. is not proof that your model really works, but rather more like some leakage has occurred, as you did with Table 2 in Appendix B (this experiment actually makes a lot of sense, as the logP task has always been considered less suitable for evaluating molecular optimization). \n\nMinor\n1.\tIt would be better if the authors provide a more intuitive and clear illustration of the motivation of introducing each module in the model (e.g. the VQ-VAE).\n2.\tFigure 1 is confusing, why does the arrow at the bottom point to a blank space?\n\n\n\n[1] Bradshaw, John, et al. \"Barking up the right tree: an approach to search over molecule synthesis dags.\" arXiv preprint arXiv:2012.11522 (2020).\n\n[2] Bradshaw, John, et al. \"A model to search for synthesizable molecules.\" arXiv preprint arXiv:1906.05221 (2019).\n\n[3] Fu, Tianfan, Cao Xiao, and Jimeng Sun. \"Core: Automatic molecule optimization using copy & refine strategy.\" Proceedings of the AAAI Conference on Artificial Intelligence. 2020.\n\n[4] Fu, Tianfan, et al. \"MIMOSA: Multi-constraint Molecule Sampling for Molecule Optimization.\" arXiv preprint arXiv:2010.02318 (2020).\n\n[5] Leguy, Jules, et al. \"EvoMol: a flexible and interpretable evolutionary algorithm for unbiased de novo molecular generation.\" Journal of cheminformatics 12.1 (2020): 1-19.\n\n[6] Shi, Zhan, et al. \"Toward diverse text generation with inverse reinforcement learning.\" arXiv preprint arXiv:1804.11258 (2018).\n", "summary_of_the_paper": "This paper proposes to perform molecular optimization by sequential translation. A reinforcement learning policy is employed to increase the diversity of the generated molecules. The proposed method is shown to be able to generate high-quality molecules on single-property and multi-property optimization tasks. The paper is well written and easy to follow.", "summary_of_the_review": "In general, I do not see too much theoretical or technical contribution. I would request the authors to carefully position their work with the related work mentioned above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agree that the paper is well-motivated and the proposed method is somewhat interesting and well-experimented. However, reviewers feel that the paper relies on many existing methods and does not appear to be novel enough.
Decision: Reject