Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Generation
Authors: Soojung Yang
Doyeong Hwang
Seul Lee
Seongok Ryu
Sung Ju Hwang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Drug discovery
Molecule generation

Abstract: Recently, utilizing reinforcement learning (RL) to generate molecules with desired properties has been highlighted as a promising strategy for drug design. Molecular docking program -- a physical simulation that estimates protein-small molecule binding affinity -- can be an ideal reward scoring function for RL, as it is a straightforward proxy of the therapeutic potential. Still, two imminent challenges exist for this task. First, the models often fail to generate chemically realistic and pharmacochemically acceptable molecules. Second, the docking score optimization is a difficult exploration problem that involves many local optima and less smooth surface with respect to molecular structure. To tackle these challenges, we propose a novel RL framework that generates pharmacochemically acceptable molecules with large docking scores. 
Our method -- Fragment-based generative RL with Explorative Experience replay for Drug design (FREED) -- constrains the generated molecules to a realistic and qualified chemical space and effectively explores the space to find drugs by coupling our fragment-based generation method and a novel error-prioritized experience replay (PER). We also show that our model performs well on both de novo and scaffold-based schemes. Our model produces molecules of higher quality compared to existing methods while achieving state-of-the-art performance on two of three targets in terms of the docking scores of the generated molecules. We further show with ablation studies that our method, predictive error-PER (FREED(PE)), significantly improves the model performance.

Hit and Lead Discovery with Explorative RL and F Fragment-based Molecule Generation
Soojung Yang
Doyeong Hwang f AITRICS
AITRICS soojungydmit.edt
deserbeegle11@@mai1.com
Seul Lee KAIST cllenlee7390ggail..com com
Seongok Ryu
Sung Ju Hwang
AITRICS
AITRICS, KAIST
seongokryu@gaauux.oou co. co.kr
s sjhwangs2@kaist.commm ac kr
Abstract
Recently, utilizing reinforcement learning (RL) to generate molecules with de- sired properties has been highlighted as promising strategy for drug design. A molecular docking program molecule binding affinity can be an ideal reward scoring function for RL, as it is a straightforward proxy of the therapeutic potential. Still, two imminent challenges exist for this task. First, the models often fail to generate chemically realistic and pharmacochemically acceptable molecules. Second, the docking score optimization is difficult exploration problem that involves many local optima and less smooth surfaces with respect to molecular structure. To tackle these challenges, we propose novel RL framework that generates pharmacochemically acceptable molecules with large docking scores. Our method Fragment- based generative RL with Ex- plorative Experience replay for Drug design (FREED) constrains the generated molecules to a realistic and qualified chemical space and effectively explores the space to find drugs by coupling our Tragment-based generation method and novel error-rioortiize experience replay (PER). We also show that our model performs well on both de novo and scaffold-based schemes. Our model produces molecules of higher quality compared to existing methods while achieving state-of-the art performance on two of three targets in terms of the docking scores of the gener- ated molecules. We further show with ablation studies that our method, predictive error-PER (FRE significantly improves the model performance.
physical simulation that estimates protein-small small
1 Introduction
Searching for "hits", the molecules with desired therapeutic potentials, is a critical task in drug discovery. Instead of screening library of countless potential candidates brute-force manner, designing drugs with sample-efficient generative models has been highlighted as promising strategy, While many generative models for drug design are trained on the distribution of known active compounds [1-3], such models tend to produce molecules that are similar to that of the training dataset [4], which discourages finding novel molecules.
In this light, reinforcement learning (RL) has been increasingly used for goal-directed molecular design, thanks to its exploration ability. Previous models have been assessed with relatively simple
Currently MIT. Cuurently LG Al Research.
CCurently Galux inc.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(Fragment-bassde basdd) Ganaration Step
(Pradictive Error-]
KEY CONCEPT
Prioritized Experience Replay
wighto SMac
Experissos replay butfer
compling bo piloeity
5. Ry Su
Ganerd redeccll
Mexn shas ralicaee
S
St-1
Docking simulation
S., Sr, Sprss Pe.
upcode
(Entropy-rggulaized]
prediche arar
Findlng realistic hit moleculas via fragment- besed generation tion and effective exploration
T
oxperience
Figure I: Overview of our generative drug discovery method. To find realistic 'hit' molecules that have high docking scores, we combine fragment- Tragment-based generation method with SAC and PER. This figure illustrates our version of PER where the priority of the experience is defined by predictive error.
objectives, such as cLogP and QED, or estimated bioactivity scores predicted by auxiliary models [5-8]. However, high scores in those simple properties of molecules guarantee neither drug-liken likenesss nor therapeutic potential, emphasizing the necessity of more relevant design objectives in generative tasks [9, 10]. The molecular docking method is physical simulation that estimates the protein- small molecule binding affinity, a key measure of drug efficacy. As docking simulations are a more straightforward proxy of therapeutic potential, coupling RL with docking simulations would be promising strategy. While the simplistic scores (e.g., cLogP, QED) are computed by a sum of local fragments' scores and are not function global molecular structure, making the optimization tasks relatively simple, docking score optimization is more demanding exploration problem for RL agents. change in docking score is nonlinear with the molecule's local structural changes, and significant variance exists among the structures of high-scoring molecules, meaning that there exists many local optima [9].
In addition, many previous RL models often suffer from generating unreal and inappropriate struc- tures. Docking score maximization alone is not sufficient to qualify molecules as drug candidates, as the molecules should satisfy strong structural constraints such as chemical realisticness and pharma- cochemical suitability. In other words, small molecule drug candidates should have enough steric stability to arrive at the target organ in the intended form (chemical realisticness), and they should not include any seriously reactive toxic substructure (pharmacochemical suitability). The low quality of generated molecules can arise from a single improper addition of atoms and bonds, which would deteriorate the entire sanity of the structure. Since such 'mistakes' easy occur, implicitly guiding the model (e.g.. jointly training the model with QED optimization) cannot completely prevent the mistakes. Thus, explicitly restricting the generation space within realistic and qualified molecules by generating molecules as combination of appropriate fragments can be a promising solution [1, 11, 12].
However, such a strong constraint in generation space would make the optimization problem of docking score even harder and render higher probability of over-fitting few solutions, urging need for better exploration for RL agents. In this respect, we introduce a new framework. Fragment- based generative RL with an Explorative Experience replay for Drug design (FREED), which encourages effective exploration while only allowing the generation of qualified molecules.
Our model generates molecules by attaching a chemically realistic and pharmacochemically ac- ceptable fragment unit on a given state of molecules at each step'. We enforce the model to form new bond only the attachment sites that considered as appropriate at the fragment library preparation step. These strategies enable us to utilize medicinal chemistry prior knowledge and successfully constrain the molecule generation within the chemical space eligible for drug design. We also explore several explorative algorithms based on curiosity-driven learning and prioritized
'In this work, we widely define the term "chemically realistic molecule" a stable molecule and narrowly define it as molecule that an assembly of fragments appear in the ZINC data. Also, we widely define the term "inappropriate molecule/fragment" or "pharmacochemically inacceptable molecule/fragmennt as a molecule that has nonspecific toxicity or reactivity. and narrowly define it as molecule that cannot pass through the three medicinal chemistry filters, which are Glaxo, PAINS, SureChEMBL filters.

---

experience replay (PER) [13]. We devise an effective novel PER method that defines priority as the novelty of the experiences estimated by the predictive error or uncertainty of the auxiliary reward pre- dictor's outcome. With this method, we aim to avoid the lack of robustness of previous methods and encourage the exploration of diverse solutions. We provide an overall illustration of our framework in Figure 1.
Our main contributions can be summarized as follows:
We propose novel RI. framework that can be readily utilized to design qualified molecules of high therapeutic potential.
Our fragment-basee based generation method including sannectivity-preseevvee fragmentation and augmentation allows our model to leverage Chemical prior knowledge.
We propose novel explorative algorithms based on PER and show that they significantly improve the model performance.
2 Related Works
SMILES- and atom-based generation methods. SMILES-basee methods [14] are infea- sible for scaffold-based tasks' since molecular structures can substantially change through the sequential extension of the SMILES strings. Also, as explained in the Introduction, atom-based gen- eration methods such as You et al.' GCPN [6] inherently suffer from unrcalistic generated molecules. Thus, we focus our discussion on motif-based generation methods.
Mosed- molecular generation methods. A number of previous works [L. 11, 12] have in- vestigated similar motif-based molecule generation based on the variational autoencoders (VAE). JT-VAE and HierVAE [1, 11] decompose and reconstruct molecules into tree structure of motifs. These models might not be compatible with the scaffold-basee generation, since a latent vector from their encoders depends on motif-wise decomposition order which is irrelevant information for docking score that may bias the subsequent generation [12]. Maziarz et al. [12] also propose a VAF-based model which encodes a given scaffold with graph neural networks (GNNs) and decodes the motif-adding actions to produce an extended molecule. While Maziarz et al.'s motif-adding actions resemble our generation steps, we additionally introduce connectivity preeeevin fragmentation and augmentation procedure which helps our model generate molecules of better quality.
Coupling an RL policy network with fragment-based generation holds general advantages compared to VAE-based methods. For example, RL models do not need to be trained on reconstruction tasks which might restrict the diversity of generated molecules. Our work is one of the earliest applications of RL with fragment- baased molecular generation method. While Ståhl et al.'s DeepFMPO [15] is also an application of RL with a fragment- based molecular generation method, DeepFMPO is designed to introduce only slight modifications to the given template molecules, which would make inappropriate for the de novo drug design. Moreover, while DeepFMPO's s generation procedure cannot change the connectivity of the fragments of the given template, our method is free to explore various connectivity.
Docking as a reward function of RL. Studies on docking score optimization task has started very recently. Jeon et al. [16] developed MORLD, atom-based generative model guided by MolDQN algorithm [17]. Olivecrona et al. [10] and Thomas et al. [18] utilized [EINVENT[14], a simplified molecular-input line-entry system (SMILES) model generative model guided by improved REINFORCE algorithm [19] to generate hit molecules.
RL algorithms for hard-exploration problems. Our view that, on a high level, there are two main approaches to achieve efficient exploration. The first one is introduce the "curiosity" or exploration bonus as intrinsic reward [20-22] for loss optimization. Bellemare et al. [23] first exploration bonus
proposed a pseudo-count-basee curiosity as distance between the true next state feature vector and the predicted estimate of the next state feature vector. Thiede al. [25] brought curiosity- learning into the molecular generation task. In Thiede et al., curiosity is defined as a distance between the true reward of the current state and the predicted estimate of the current state reward. However, these cuuriosity-drivnn intrinsic reward-based models sometimes fail to solve complex problems [26]. The failures are explained as a result of the RL agent's 's detachment derailment from the optimal solutions [27].
intrinsic reward. Pathak et al. [24] defined
A drug design scheme where the drug candidates are designed from the given scaffold.
3

---

(a)
Goaph entedkigg
Note ambedding
Chemically realistk fraraentilbrary
Action
Augmented fragment
Artion 1
Q
Action
Possible attachment
Neat state molecute S
Current state melecula St
(b)
Graph
Grach
Graph ECFP
Tyttt
Tactt
Action
Action 2
Action
Figure 2: Overview of our generation method (a) and policy network (b). Small colored squares represent a graph embedding of molecule, node embedding of each attachment sitem and ECFP representation of each fragment. The graph or node embeddings and ECFP representations from the chosen actions are autoregressively passed onto the next action's 's policy network.
The other approach of solving hard exploration problems is sample-efficien use of experiences [28, 13]. Prioritized experience replay (PER) method introduced in Schaul et al. [13] samples experiences that can give more information to the RL agent and thus have more 'surprisal' defined by the temporal-differee (SII.) introduced in Oh et al. [29] samples only the 'good' experiences where the actual reward is larger than the agent's 's value estimate (i.c., estimate from the value function or Q function). However, prioritized sampling based on the agent's value estimate susceptible to outliers and spikes, which may lead to destabilizing the agent itself [30]. Moreover, our explorative algorithm aims to preserve sufficient diversity and find many possible solutions instead finding the most efficient route to single solution. Thus, we modify the formulation of priority using the estimates for sample novelty.
(TD) error - in higher probability. a similar sense, self-imitation learning
Methods
3.1 Generation method
Generation steps. The key concept of our method to generate high-quality molecules that bind well to a given target protein. In order to achieve this goal, we devise a fragment-based generation method in which molecules are designed as an assemble of chemically realistic and pharmacochemically acceptable fragments. Given molecular state, our model chooses "where to attach a new fragment", "what fragment to attach", and "where on a new fragment to form a new chemical bond" in each step". Note that the action "where to form new bond" makes our model compatible with docking score optimization since the scores depend on the dine-dimessional arrangement of molecular substructures.
Preserving fragment connectivity information in molecule generation. Another important fea- ture of our method is harnessing the predefined connectivity information of the fragments and initial molecules. This feature allows our model to leverage chemists' expert knowledge in several ways. The connectivity information arises in the fragmentation procedure, in which we follow CReM [31] when the algorithm decomposes any arbitrary molecules into fragments while preserving the fragments' attachment sites shown in Figure 3(a).
In the GNN embedding phase, the attachment sites are considered as nodes like any other atoms in the molecule, while an atom type is exclusively assigned to the attachment sites. We also keep tracking the indices of the attachment sites as the states so that our policy can choose the next attachment site where new fragment should be added (Action 1). Similarly, we keep the indices of the attachment sites the fragments and use them throughout the training so that our policy can choose the attachment site from the fragment side (Action 3).
Start and end points of new bonds are restricted the attachment sites of given fragments and molecules. This restriction contributes to the chemical realisticness since the stability of the molecule
*Our model is designed to finish the episodes after four steps (in de novo cases) or two steps (in scaffold-bass cases). At the end of the episode, the model substitute all the remaining attachment sites with hydrogen atoms.

---

bond
NH
Pyrrol
Pyrroilm not anrmatic undaalle
2-medhyloyyrle amrrati, atabie
aomenis stacle
Figure Advantages of information is preserved as attachment sites in the fragmentation procedure. (b) Attaching new bond to the nitrogen of a pyrrole ring would result unstable molecule, breaking the aromaticity. Thus, pyrrole rings within an existing molecule are connected to other fragments by its carbon atoms, and preserving such an information would help us construct chemically realistic molecules.
fragmentation. (a) Substructure connectivity
depends on where the fragment is attached as illustrated in Figure 3(b). Also, we can utilize our prior knowledge of protein-ligand interaction by rationally assigning the attachment sites of the initial molecule (scaffold), which have been widely harnessed by medicinal chemists usual scaffold-based scenario.
Policy network. Our generation method is coupled with the policy networks which guide the model generate hit molecules. We provide overall illustration of policy network in Figure 2. Our state embedding network and policy network are designed as Markov models so that generation steps take any arbitrary molecule as a current state and predict the next state, making the model more plausible for a scaffold-based generation.
In our framework, each state vector 8; represents molecular structure at the (-th step, and each step is a sequence three actions that define an augmentation fragment. Following the method from Hwang et al. [32]. the current state molecule is represented consists of a node feature matrix, an adjacency matrix, and an attachment site masking vector. A graph convolutional network (GCN) [33] -based network provides node embeddings, H, which then aum-aggregated to produce a graph embedding, Ng. Note that the same GCN encodes the current state molecular graph (for Action 1) and the fragments (for Action 3).
undirected graph G' which
Our policy network is an autoregressive process where Action depends on Action I & 2 and Action 2 depends on Action 1.
act -Tall (2) 21st MI(hy: Hum)
n012 -Tac12 2nd 2nnd MI(= ECFP(h)gu))
(2)
p
nc13 - (2dnd), 2 MI(2 Ucand)
(3)
where Han refers to the node embeddings of attachment sites. For the first step of our policy network, Taet takes the multiplicative interactions (MI) [34] of the node embedding of each attachment site and the rows of the graph embedding of the molecule, 21st E -2sst, as inputs and predicts which attachment site should be chosen as Action Since graph embeddings and node embeddings are defined in a heterogeneous space, we apply MI to fuse those two sources of information.
Given Action Faa2 takes the MI of 2|11 the row of Z under index actl, and the RDKit [35] ECFP (Extended Connectivity Fingerprint) representation each candidate fragment, hga.s as inputs, and predicts which fragment should be chosen as Action 2. By taking Zist as one of the inputs, Tact2 reflects on the context of the current state and Action 1.
Finally, given Action and 2, Taet3 takes the MI of 2nd and the node embeddings of the chosen fragment's 's attachment sites, Ucand, as inputs and predicts which fragment attachment site should be chosen as Action 3. Each of the three policy networks Tattl Tact2 and тa consists three fully-connected layers with ReLU activations followed by softmax layer to predict the probability of each action. In order to make gradient flow possible while sampling discrete actions from the probabilities, we implement the Gumbel-softmax reparamcterization trick [36].
5

---

3.2 Explorative RL for the discovery of novel molecules
Soft actor-critic. We employ soft actor-critic (SAC), an off off-policy actor-critic RL algorithm based on maximum entropy reinforcement learning [37, 38], which is known to explore the space of state- action pairs more effectively than predecessors. Assuming that our docking score optimization task requires more effective exploration than simplistic tasks, we chose SAC as our RI. baseline algorithm.
n arg max
Σ BE Feer,ae)rp,, ar) + (HH(((((+)))]
(4)
x
E E 1 (a log π (aa ss] Σ IaTT(at o0 (s))
(5)
x (log π (a (s)) - log π (a. (8,, a.t + log ग (aa st, ar" aaa)))
SAC aims attain optimal policy that satisfies (4) where a is the temperature parameter balancing between exploration and exploitation of the agent, (-(((-);)) is entropy of action probabilities given 8t, and fr is state-action transition distributions created from æ. As we define act act2 and act3 autoregressively, the entropy regularization term defined in [39] Es (a log x (a|s)] is decomposed into (5).
Explorative algorithms. To encourage exploration, we prioritize novel experiences during sam- pling batches for RL updates. We regard an experience as a novel experience if the agent has not visited the state before. Defining priority estimate function in the state space and not in the state-action space has been introduced for the molecular generative task in Thiede et al. [25]. For novel states, the reward estimator would yield high predictive eπrоr high variance (Bayesian uncertainty). In this regard, we train an auxiliary reward predictor consisting of a graph encoder and fully- layers that estimate a given state's reward (docking score). Then, we the predictor's predictive error (1.2 distance) or Bayesian uncertainty as priority estimate. We name the former method PER(PE) and the latter method PER(BU). The use of predictive error as novelty estimate has been introduced for curiosity-driven learning [25], but our work is the first to apply this to PER.
For PER(BU), we follow Kendall et al. [40] to obtain the Bayesian uncertainty of the prediction. We train the reward predictor network to estimate the reward's mean and variance (aleatoric uncertainty). Every layer in the network is MC dropout [41] layer so that the predictor provide the epistemic uncertainty. We add aleatoric and epistemic uncertainty to obtain the total uncertainty of the estimate.
The reward predictor is optimized for every docking step, and we only optimize it based on final state transitions since docking scores are only computed for the final states. The reward predictor predicts the reward of any state, both intermediate and final. For PER(PE), when we compute the priority of a transition including an intermediate state, we use Q value as a substitute for the true docking score. After updating the policy network and Q function with loss multiplied by importance sampling (IS) weight, we recalculate and update the priority values of the transitions in the replay buffer.
The following experiments section shows that PER with our priority estimate functions performs better than the previous methods.
Results and Analysis
4.1 Quantitative metrics
In this section, we introduce quantitative metric scores we used to assess our model. For every metric, repeated every experiment five times with five different random seeds and reported the mean and the standard deviation of the scores. Also, we calculated the scores when 3,000 molecules were generated and used to update the model during training.
Quality score. We report three widely used pharmacochemical filter scores Glaxo [42]. SureChEMBL [43], PAINS [44] - as quality scores. The quality scores are defined as a ratio of accepted, valid molecules to total generated molecules, the filters reject the compounds that contain functional groups inappropriate for drugs (i.e., toxic, reactive groups). The higher the quality scores, the higher the probability that the molecule will be acceptable drug. We also report the ratio of

---

valid molecules to total generated molecules (validity) and the ratio of unique molecules among valid generated molecules (uniqueness).
Hit ratio. We define hit ratio as a ratio of unique hit molecules to total generated molecules. We report the hit ratio compare the model's ability produce as many unique hit molecules in a given length of iterations, where we define hits as molecules whose docking scores are greater than the median of known active molecules' docking scores.
Top 5% score. We report the average score of the top 5%-scored generated molecules to compare the model's ability to produce molecules with better docking scores.
4.2 Quantitative performance benchmark
In this section, we compare the quality of the generated molecules and the model performance with three baseline models, MORLD [16], REINVENT [14], and HierVAE [11]. Our model, FREED(PE). our fragment-bass
generation method coupled with SAC and PER(PE). MORLD and REINVENT are the models utilized for docking score optimization tasks in previous works [16, 10, 18]. HierVAE a strong non-RL fragment-
molecular generative model. Since ViE is distributional learning method, we train HierVAE in two schemes 'one-time" training the known active molecules (HierVAB V) and "active learning (AL)' training where we train the model once on the known actives and twice the top-scoring molecules from the generated molecules (HierVAE(AL)). In both schemes, we initialized the models with the model pre-trained on ChEMBL. We trained the models for three carefully chosen protein targets fa7, parp1, and 5htlb. The choice of protein targets and the specifics of experimental settings are described in Appendix. For the experiments in this section, we use the small fragment library that includes 66 pharmacochemically acceptable fragments.
Table 1: Quality scores of the models. We trained our model and three baseline models with the target fa7 and computed quality scores of the first 3,000 molecules generated during training for each model. The two baseline models REINVENT and MORLD that are jointly trained to maximize filter scores noted as 'REINVENT w/ filter' and 'MORLD w/ filter'. Standard deviation is given in brackets.
Glaxo
SureChEMBL.
PAINS
validity
uniqueness
MORLD
0.561 (.009)
0.131 (.013)
0.805 (.013)
1.000 (.000)
1.000 (.000)
MORLD w/ filter
0.578 (.010)
0.145 (.018)
0.816 (.008)
1.000 (.000)
1.000 (.001)
REINVENT
0.773 (.023)
0.667 (.030)
0.769 (.022)
0.813 (.024)
0.988 (.008)
REINVENT w/ filter
0.832 (.034)
0.747 (.040)
0.842 (.034)
0.872 (.028)
0.990 (.007)
HiE
0.899 (.027)
0.748 (.024)
0.975 (.006)
1.000 (.000)
0.138 (.006)
HierVAE(AL)
0.975 (.004)
0.795 (.007)
0.893 (.011)
1.000 (.000)
0.131 (.003)
Ours: FREED(PE)
0.996 (.001)
0.808 (.049)
0.991 (.002)
1.000 (.000) 0.723 (.135)
Quality scores of generated molecules. In real-world drug design, the generated molecules should be acceptable by pharmacochemical criteria. We excluded the fragments that are considered inap- propriate by pharmacochemical filters to guarantee the quality of the generated molecules. Such an explicit constraint cannot be applied to atom-based (MORLD) or SMILFSS-baed (REINVENT) methods.
As shown in Table 1, our model mostly generated acceptable molecules while the other models showed the poor rate of acceptable molecules, confirming our approach's 's advantage in drug design. We also investigated whether one could improve the quality of the molecules generated from the baselines by using a multi-objective reward function. MORLD and REINVENT optimized in multi-objective with both docking scores and the quality scores, denoted as 'MORLD w/ filter' and REEIVVNT w/ filter', show improved quality scores compared to single-objective MORLD and REINVENT. However, the multi-objective reward method was not as effective as our fragment-based approach, strengthening our claim that the explicit constraints are the effective strategy. Such trend was consistent for all three targets. See Appendix for visual inspection of the quality score results.
7

---

fa7
parpl
5ht1b
(a)
0.35
6.11
REBATNT
0.61
EORMIT
HOLS
HNHAN Hande:)1)
KLMIINT
(b)
-10
Figure 4: Hit ratio and top 5% score of model FREED(PE), REINVENT, MORLD, Hier- VAE, and Hier VAE(AL). Standard deviation is given as error bars. Higher hit ratios and greater negative value of the top 5% scores indicate better performance.
HierVAE showed high quality scores, the HierVAE fragment library itself had very few problematic substructures (See Appendix A.5 for details). Such a result substantiates the advantage of the explicit fragment-han based approach. However, HierVAE showed low uniqueness, possibly due to the small size of training data for fine-tuning (~1,200 known active molecules) and active learning (~1,500 generated high-scoring molecules).
We provide quality scores our model (FREED(PE) trained with the small library and the large library for all three protein targets in Table 2 and Table 3 of Appendix A.1, respectively. A significant increase uniqueness was observed when we used the large library, which implies that the low uniqueness of our model is due to the small size of the fragment library. Thus, believe con- structing a fragment library that large enough to guarantee high uniqueness while only including pharmacochemically acceptable fragments will be the best strategy in production.
Docking score optimization result. Figure 4 shows the hit ratios and top 5% scores of the gen- erative models. Our model outperforms the other generative models MORLD, REINVENT, and HierVAE terms of both hit ratio and top 5% score, except for the hit ratio of the 5htlb case. Such results show that model's 's performance is superior or least competitive to existing baselines, while our model exhibits many more practical advantages such as generating acceptable molecules, integrating chemist's expert knowledge by srmnectivtyy-preeeving fragmentation and augmentation, and the feasibility both de novo and scaffold-basee generation.
4.3 Ablation studies: explorative algorithms
We also perform ablation studies of our algorithm to investigate the effect of our explorative algorithms. We used the larger library of 91 unfiltered fragments, as this section assesses the effect of the algorithms on the model's performance regardless of the pharmacochemical acceptability.
In Figure 5, we observe that all SAC models with explorative algorithms performed better than the vanilla SAC model, while the PPO model showed the worst performance. FREED with both PE and BU outperformed curiosity-driven models with PE and BU, showing the effectiveness of our methods in our task compared to curiosity-driven explorations. Moreover, our predictive error-PER method outperformed the TD error-PER PER method. We conjecture that such a result is due to 1) novelty- based experience prioritization encourages better exploration leveraging an auxiliary priority predictor network makes PER training more robust than internal value estimate functions (Q function). We provide the significance analysis (one-tail paired t-test) the result Table of Appendix A.I. 4.4 Case study on drug design
In this section, we show the practicality of our framework on novo and scaffold-hased drug design. We test FREED(PE) with our large fragment library which includes 91 fragments.

---

parp1
3.25
k.00
8mh
800
SAC
SAC
--1
---1
-30
-15
100
302/11 01.03
19:04:1
Figure 5: Hit ratio and top 5% score ablation studies. Models can be categorized by whether they use (PER, curiosity-driven exploration(curioo) and whether they use (predictive error from predictor(PE), Bayesian UnCertainty(BU), and TD error from agent(TD)) priority or intrinsic reward. Standard deviation is given as error bars.
fa7
parpl
5htlb
nevo
Doocdnng Score
Docodnng Score
Docling
-30
PREECYPE)
Random
Aetiv
Inactiv
FREEDSPE)
Sandom
Active
Iracthue
PREEDAPE)
Random
Active
Inactive
Dredkins Sear
Broking Soote
PREEDVPE)
Random
Aclive
Inscivv
PREEDIPE) Rendom
Aftive
Inachhe
PREEDUPE) Random
Active
(RaCTive
Figure Docking score distribution of the generated molecules. Duplicate molecules were removed after gathering 3,000 molecules each from five random seed experiments. "Random" molecules are generated by our fragment-based generation method without training the policy net- work. "FREED(PE)" molecules are generated by the fragment-based generation method while training the policy network. We also plot known "Active" and "Inactive" molecules from DUD-E (fa7, parp 1) ChEMBL (5htlb) datasets for comparison. Colored horizontal lines indicate the median of the corresponding distribution. (a) de novo scenario (b) scaffold-baseh scenario
De novo scenario. Figure 6 (a) shows the distribution the generated molecules before ("random") and after ("FREED(PE)") optimizing the policy network. Our model was able to effectively generate molecules that have higher docking scores compared the known active molecules. Figure (i) shows the structure of each target's 's optimized molecules.
Scaffold-based scenario. We validate our model on scaffold-bard scenario. where we attempt to improve docking scores by adding fragments to an initial scaffold molecule. Figure 6 (b) shows the distribution of the optimized molecules before ("random") and after ("FREED(PH)") training the policy network, with scaffold of each target as initial molecule.
Figure 6 (b) highlights our model's ability to optimize given scaffold to have higher binding affinity with the target. Surprisingly, in Figure (b), even the molecules randomly optimized with fragment-based generation algorithm show high docking scores when given the proper scaffold.

---

de novo (a) ta7
(b) parp1
(c) Sht1b
Docking m-11 scone 3.50
HI Docking acom
Hit Dooking scera -119
(i) scaffold- based
(a)
8.78
Hil teoallodd
Load Dodiingecom -133
Hil (sealfed)) Dodesgacore 101
Lead Docsngssore 5A 2.54
Hil (ooafMe)
Lead Dooking scers SA.oow 3.42
Dockingscore -1.9
Dociiig ecore
Figure 7: Generated samples and their docking scores with our method, for de novo (i) and scaffold-based scenario (ii). For each target, of the high-scoring generated molecules is dis- played with the initial molecule (benzene ring or scaffold). The purple line highlights the fragments augmented by the model in scaffold- based generation. Numbers below the compounds the docking scores and SA scores.
This result implies the importance of scaffold in hit discovery and highlights our generative method's ability to to span the chemical space around the scaffold effectively.
Figure 7 (ii) shows the structure of each target's 's scaffold and corresponding optimized molecules. We can see that the scaffold structures are well preserved generated lead molecules. We provide an analysis of 3D docking poses of the scaffolds and generated lead molecules in Figure 2 and Figure of Appendix A.
It is notable that our framework does not have to be specific for de novo scaffold- scenarios, except for the initial molecule and number of fragments to be added. Since our model is fully Markovian, whether the initial molecule is benzene ring or scaffold does not affect the model's training.
Chemical realisticness of generated molecules. Figure we report the SA (synthetic acces- sibility) score of the molecules, which is widely used metric that estimates ease of synthesis by penalizing the presence of non-standard structural features. The SA score distribution of the catalogue molecules of commercial compound providers has its mode around 3.0 [45]. Accordingly, we can assume our generated molecules as reasonably synthesizable and thus chemically realistic.
Conclusion
In this work, developed FREED, novel RL framework for real-world drug design that couples fragment-basse -based molecular generation strategy with a highly explorative RL algorithm to generate qualified hit molecules. Our model generates pharmacochemically acceptable molecules with high docking scores, significantly outperforming previous docking RI. approaches. Our code is released at https://gi tttpggtthhbh..e/ thub. com/ AITRICS/ FFREED.
Limitations and future work. While our method does not explicitly account for the synthesizability of generated molecules, ours. It would be able to combine our method and forward synthesis- method by substituting attachment site bond formation actions with chemical reactions. In this way, we explicitly take synthesizability into account while providing the appropriate model inductive bias for docking score optimization. We leave such improvement as future work.
believe forward synthesis-based methods [46] can be complementary to
Negative societal impacts. used maliciously, our framework can be utilized to generate harmful compounds such as biochemical weapons. Thus, conscious use of the AI model is required.
10

---

Acknowledgments and Disclosure of Funding
We thank anonymous reviewers, Seung Hwan Hong, and Sihyun Yu for providing helpful feedback and suggestions. This work was supported by the National Research Foundation of Korea (NRF) grant funded by the project NRF2019M33ESD400696..
References
[1] W. Jin, R. Barzilay, and T. Jaakkola, "Junction variational autoencoder for molecular graph generation," in International Conference on Machine Learning. PMLR, 2018, pp. 2323-2332.
[2] A. Zhavoronkov, Y. A. Ivanenkov, A. Aliper, M. S. Veselov, V. A. Aladinskiy, A. V. Aladinskaya, A. Terentiev, D. A. Polykovskiy, M. D. Kuznetsov, A. Asadulaev et al., "Deep learning enables rapid identification of potent ddrl kinase inhibitors," Nature biotechnology, vol. 37, no. 9, pp. 1038-10401 2019.
[3] N. De Cao and T. Kipf, "Molgan: An implicit generative model for small molecular graphs." arxiv preprint arxiv:1805 1973, 2018.
[4] W. P. Walters and M. Murcko, "Assessing the impact of generative ai on medicinal chemistry," Nature biotechnology. vol. 38, no. 2, pp. 143-145, 2020.
[5] Z. Zhou, S. Kearnes, L.. Li, R. N. Zare, and P. Riley, "Optimization of molecules via deep reinforcement learning," Scientific reports, vol. 9, no. 1, pp. 1-10, 2019.
[6] J. You, B. Liu, R. Ying, V. Pande, and J. Leskovec, "Graph convolutional policy network for goal-directed molecular graph generation," arXiv preprint arXiv: 1806.0246..3 2018.
[7] M. Popova, O. Isayev, and A. Tropsha, "Deep reinforcement learning for de novo drug design." Science advances, vol. 4, no. 7, p. eaap7885, 2018.
[8] T. Blaschke, O. Engkvist, J. Bajorath, and H. Chen, "Memory-assisted reinforcement learning for diverse molecular de novo design," Journal of cheminformatics, vol. 12, no. pp. 1-17, 2020.
[9] W. C. Coley, "Defining and exploring chemical spaces," Trends in Chemistry, vol. 3, pp. 133- 145, 2021.
[10] Cieplinski, T. Danel, S. Podlewska, and Jastrzębski, "We should at least be able to design molecules that dock well," arXiv preprint Skki::2006.1655511 2020.
[11] W. Jin, R. Barzilay, and T. Jaakkola, "Hierarchical generation of molecular graphs using structural motifs," Proceedings of the 37th International Conference on Machine Learning, PMLR, no. 119, pp. 4839-48444 2020.
[12] K. Maziarz, H. Jackson-Flus Flux, P. Cameron, F. Sirockin, N. Schneider, N. Stiefl, and M. Brockschmidt, "Learning to extend molecular scaffolds with structural motifs," arXiv preprint
2021.
[13] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, "Prioritized experience replay," arXiv preprint 1.105555. 2015.
[14] M. Olivecrona, Blaschke. O. Engkvist, and H. Chen, "Molecular de-novo design through deep reinforcement leaminn." Journal of Cheminformatiic, vol. 9, no. 48, 2017.
[15] N. Ståhl, G. Falkman, A. Karlsson, G. Mathiason, and J. Boström, "Deep reinforcement learning for multiparameter optimization in de novo drug design," " J. Chem. Inf. Model., vol. 59, no. pp. 3166-3176, 2019.
[16] W. Jeon and D. Kim, "Autonomous molecule generation using reinforcement learning and docking to develop potential novel inhibitors." Scientific Reports, vol. 10, no. 22104, 2020.
[17] 7. Zhou, Kearnes, I.. Li, N. Zare, and P. Riley, "Optimization of molecules via deep reinforcement learning," Scientific Reports, no. 10752, 2019.

---

[18] M. Thomas, R. Smith, N. M. O' O'Boyle, C. de Graaf, and A. Bender, "Comparison of structure- and ligand-bar based scoring functions for deep generative models: a gper case study." ChemRxiv preprint, 2021.
[19] R. J. Williams, "Simple statistical gradient-folowingg algorithms for reinoreemet learning," Machine Learning, vol. 8, pp. 229-256, 1992.
[20] R. M. Ryan and E. L. Deci, "Intrinsic and extrinsic motivations: Classic definitions and new directions," Contemporary educational psychology, vol. 25, no. 1, pp. 54-67, 2000.
[21] P.-Y. Oudeyer and F. Kaplan, "What is intrinsic motivation? a typology of computational approaches," Frontiers in neurorobotics, vol. 1, p. 6, 2009.
[22] A. G. Barto, "Intrinsic motivation and reinforcement learning," in Intrinsically motivated learning in natural and artificial systems. Springer, 2013, pp. 17-47.
[23] M. G. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos, "Unifying count-based exploration and intrinsic motivation," arXiv preprint arXiv: 01868, 2016.
[24] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, "Curiosity-driven exploration by self- supervised prediction," in International Conference on Machine Learning. PMLR, 2017, pp. 2778-2787.
[25] L. A. Thiede, M. Krenn, A. Nigam, and A. Aspuru-Guzik, "Curiosity in exploring chem- ical space: Intrinsic rewards for deep molecular reinforcement learning," arXiv preprint arXiv:2012.11293, 2020.
[26] A. A. Taiga, W. Fedus, M. C. Machado, A. Courville, and M. G. Bellemare, "Benchmark- ing bonus-based exploration methods on the arcade learning environment," arXiv preprint 1908 02388, 2019.
[27] A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune, "Go-explore: a new approach for hard-exploration problems," arXiv preprint arXiv: 10995, 2019.
[28] T. L. Paine, C. Gulcehre, B. Shahriari, M. Denil, M. Hoffmann, H. Soyer, R. Tanburn, S. Kaptur- owski, N. Rabinowitz, D. Williams et al., "Making efficient use of demonstrations to solve hard exploration problems," arXiv preprint
2019.
[29] Oh, Y. Guo, S. Singh, and H. Lee, "Self-imitation leaming." in International Conference on Machine Learning. PMLR, 2018, pp. 3878-3887.
[30] S. Lee, J. Lee, and I. Hasuo, "Predictive per: Balancing priority and diversity towards stable deep reinforcement learning"" arXiv preprint arXiv:2011.13093 2020.
[31] P. Polishchuk, "Crem: chemically reasonable mutations framework for structure generation," Journal of Cheminformatics, vol. 12, no. 28, 2020.
[32] D. Hwang, S. Yang, Y. Kwon, K. H. Lee, G. Lee, H. Jo, S. Yoon, and S. Ryu, "Comprehensive study on molecular supervised learning with graph neural networks." Journal of Chemical Information and Modeling. vol. 60, no. 12, pp. 5936-5945, 2020.
[33] N. Kipf and M. Welling, "Semi 'Spmiisupe classification with graph convolutional networks." arXiv preprint arXiv: SkKi:1609.020000 2016.
[34] S. M. Jayakumar, W. M. Czarnecki, J. Menick, J. Schwarz, J. Rae, S. Osindero, Y. W. Teh, T. Harley, and R. Pascanu, "Multiplicative interactions and where to find them," in International Conference on Learning Representations, 2019.
[35] G. Landrum, "Rdkit: Open-source cheminformatics software," 2016. [Online]. Available: http://hhw...ommmiiidddddeeeeeee
2016_099 09_4
[36] E. Jang, S. Gu, and B. Poole, "Categorical reparameterization with gumbel-softmax," arXiv preprint arXiv:1611.01144 2016.
12

---

[37] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, "Soft actor-critic: Off- maximum entropy deep reinforcement learning with a stochastic actor," in International Conference on Machine Learning. PMIR, 2018, pp. 1861-1870.
[38] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel et al., "Soft actor-critic algorithms and applications," arXiv preprint arXiv:1812.05905, 2018.
[39] P. Christodoulou, "Soft actor-critic for discrete action settings." arXiv preprint
WwXiv:1910007207 2019.
[40] A. Kendall and Y. Gal, "What uncertainties do we need in bayesian deep learning for computer vision?" arXiv preprint (7Xix:1703.0497) 2017.
[41] Y. Gal and Z. Ghahramani, "Dropout as a bayesian approximation: Representing model uncer- tainty in deep learning," in international conference on machine learning. PMLR, 2016, pp. 1050-1059.
[42] S. Lane, D. S. Eggleston. K. A. Brinded, J. C. Hollerton, N. L. Taylor, and S. A. Readshaw, "Defining and maintaining high quality screening collection: the gsk experience," Drug discovery today, vol. 11, no. 5-6, pp. 267-272, 2006.
[43] G. Papadatos, M. Davies, N. Dedman, J. Chambers, A. Gaulton, J. Siddle, R. Koks, S. A. Irvine, J. Pettersson, N. Goncharoff el al., "Surechembl: large-scale, chemically annotated patent document database," Nucleic acids research, vol. 44, no. D1, pp. 11220-D1228, 2016.
[44] J. B. Baell and G. A. Holloway, "New substructure filters for removal of pan assay interference compounds (pains) from screening libraries and for their exclusion in bioassays." " Journal of medicinal chemistry, vol. 53, no. 7, pp. 2719-2740, 2010.
[45] P. Ertl and A. Schuffenhauer, "Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions," Journal of Cheminformatics, vol. I, no. 8, 2009.
[46] S. K. Gottipati, B. Sattarov, S. Niu, Y. Pathak, H. Wei, S. Liu, Y. Bengio al., "Learning to navigate the synthetically accessible chemical space using reinforcement learning," 2020.
13


Reviews:
[{"review_id": "gVWHoDCn7uk", "reviewer": "Reviewer_dE1i", "summary": "The paper introduces a new generative method for designing molecules. The method is based on chemical fragments (motifs) that are attached to the generated compound autoregressively using reinforcement learning. This method, contrarily to many other fragment-based models, does not use an encoded latent space. The goal of the presented approach is to design bioactive molecules that are \"realistic\", which is achieved by utilising a precomputed fragment library of druglike and stable chemical structures. The bioactivity is guided by the docking-score rewards, and 3 different realisticness measures are employed to assess the generated compounds. The results show strong performance with regard to both these optimization objectives: binding affinity from molecular docking and pharmacochemical filters.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The originality of this paper lies in a new combination of the used machine-learning techniques to achieve an ambitious goal of designing realistic compounds that can be developed as potential drugs. While reinforcement learning and fragment-based drug discovery are not new techniques in the development of drug candidates, their combination is not yet well-explored in the literature. However, I believe **the statement in line 85 that this work is \"the first application of RL with fragment-based molecular generation method\" may not be true**. There is at least one paper in which RL (the actor-critic model) is used to optimize molecules; [1] use RL in fragment-based molecular optimization to replace compound fragments. Other works introducing RL-based generative models also suggest that these methods can work on substructures instead of single atoms (e.g. [2]), but they do not support their claims with experimental evidence. The authors creatively solve the problems that arise from the definition of generative process, which is in their case a 3-step procedure repeated iteratively. For example they use multiplicative interaction to mix different information sources and the gumbel-softmax reparametrization trick to allow for sampling discrete actions. The focus in the paper is on the exploration of chemical space, and the solid comparison of different techniques, such as employing the soft actor-critic model or curiosity-driven exploration, is the novelty of this work.\n\nThe experimental setup is properly described, and all hyperparameters are included in the supplemental material. The error intervals are provided over 5 repetitions of model training. The proposed model is compared against two strong generative methods, and the selection of target proteins seems also reasonable - each target belongs to a different protein family. **The addition of at least one model that works on fragments (maybe a VAE model or a genetic algorithm) would diversify this benchmark.** The authors use three different quality scores for assessing pharmacochemical utility of the generated compound. However, the experiment comparing these scores between the methods is not convincing to me. The \"inappropriate\"  fragments were discarded from the fragment library, so these results are obviously higher for the authors' method. The other methods, REINVENT and MORLD, were not artificially encouraged to generate correct structures, including realistic rings. For the fairness of this comparison, **a column with the FREED(PE) results achieved using the larger fragment library should be added**, and the reward objective should be the same for all the methods (maybe only with small adjustments required by these methods). Figure 7 shows interesting results of the molecule generation using both *de novo* and scaffold-based scenarios (and also the choice of the used scaffolds is explained in the supplemental material). The authors write: \"We can see that optimized molecules are chemically realistic.\" The molecules in the figure in fact look similar to what I have seen before, but I am not a trained medicinal chemists (and neither is the majority of the NeurIPS audience), so **a quantitative measure would be helpful here to support this claim**, e.g. QED or other continuous measure related to the quality filters. Also, these compounds seem huge, often with 5 or more rings. I would love to see **the results of the molecular docking (3D view of the binding site)**, especially for the scaffold-based scenario. If the optimized hits are twice the size of the original compound, do they still fit the binding pocket? Do they have the same interactions, or do they rotate in the pocket?\n\nThe paper is clearly written, but **some concepts could be explained in more detail**, or at least there should be a reference to the supplemental material. For example, terms like \"chemically realistic\" or \"inappropriate fragments\" appear throughout the text. They could be defined more precisely, which is to some extent done in the supplemental material. The information about what this \"realisticness\" constitutes is missing. Also, **the information about how the fragment library is built is missing in the text**, which may confuse readers (the reference to the SI should be enough). Apart from these minor issues, the text flow is logical and clear. The formatting of figures and tables can also be improved if the paper is accepted (although this did not impact my rating):\n- The text in the figures is too small. The rule of thumb is to match the text size of the caption for best readability.\n- The carbonyl group in Figure 7 (ii b) is cut.\n- In the caption of Figure 7 there is missing information about the meaning of the numbers below the compounds. Are they docking scores?\n- In the hit-to-lead examples in Figure 7, the orientation of the scaffold can be aligned for better readability. If RDKit was used to create this drawing, then this function can be used: `AllChem.GenerateDepictionMatching2DStructure`.\n- The tables typically have a horizontal line at the bottom (`\\bottomrule`) as in the formatting instructions for NeurIPS.\n- If the authors decide to keep the checklist, the instructions block of the checklist should be removed.\n- I also would recommend the authors to check the text one more time to remove typos, e.g. in line 123 \"molecules that binds\" or line 134 \"we follows\".\n\nIn computer-aided drug design, the quality of generated compounds is still a big problem. The authors show a method that aims to solve this issue, so the presented work is very relevant to the current research. The paper also tackles the problem of the chemical space exploration. The related work section describes current approaches, including recently published articles on using molecular docking as a reward function to discover bioactive compounds. I think that the significance of this paper is high, in particular if the code is made public as marked in the checklist.\n\nBased on all the above, I recommend this paper to be accepted if the authors are willing to address the comments in the final version.\n\n[1] Ståhl, Niclas, et al. \"Deep reinforcement learning for multiparameter optimization in de novo drug design.\" Journal of chemical information and modeling 59.7 (2019): 3166-3176.\n\n[2] You, Jiaxuan, et al. \"Graph convolutional policy network for goal-directed molecular graph generation.\" arXiv preprint arXiv:1806.02473 (2018).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and societal impacts were described in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "YjxevD381kP", "reviewer": "Reviewer_EZqQ", "summary": "A reinforcement learning method is proposed for drug design. In this work, small molecules are presented as fragments which can attach together to form a molecule.  Docking score and physicochemical filters are combined in the reward function.  Key feature in the method is the use of soft actor-critic in drug design. Specifically, a prioritized sampling-based approach with auxiliary priority predictor network is used to provide predictive error and Bayesian uncertainty as priority for experience.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed framework named FREED integrates fragment-based molecular representation, docking scores, and soft actor-critic method for drug design. Overall, the originality of this work is mild. The major contribution is from the sophisticated design of the reinforcement learning algorithm. See details below.\n\n1. The use of this type of fragmentation method in reinforcement learning is not new. It is highly similar to the work in [Niclas Ståhl et al. Deep Reinforcement Learning for Multiparameter Optimization in de novo Drug Design. J. Chem. Inf. Model. 2019, 59, 3166-3176].\n2. The priority predictor uses a graph network and fully connected layers to estimate docking score. The estimation of docking scores is critical, but its performance (e.g. in terms of R^2) is not reported.\n3. There is no comparison with non-RL drug design / molecular optimization methods. \n4. I suggest significance analysis should be done for the results presented on Figure 5. Otherwise, it is hard to justify the performance of the proposed method.\n5. A very small number (<100) of fragments are used as the fragment library. This could extremely limit the search space. How many fragments were initially generated in your fragmentation of the zinc data? I guess the number could be more than a few thousand. \n6. In computational chemistry, docking scores are calculated based on force field, which is slow. This may partially explain why a small number of fragments were used and small numbers of molecules were generated. Thus time analysis should be provided and the scalability should be discussed. \n7. Grammatical errors were seen in multiple places, e.g. approach is cannot -> approach cannot. I also suggest the authors carefully check each reference iterm. Case errors can be found in many places, e.g. Journal of cheminformatics -> Journal of Cheminformatics.\nMinor: In Figure 4, the authors need to indicate target for each panel. Also, the minus signs in lower panels are confusing. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Properly addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "WUY6cxVA9bQ", "reviewer": "Reviewer_sXy7", "summary": "This paper proposes a framework for optimizing over drug-like molecules, composed of a generative model and an RL algorithm. The former builds upon recent work in graph-based generation, adding several useful extensions; the latter is an off-policy RL method with a specialized form of experience replay. The paper contains an evaluation of the method on docking score optimization, showing that the generated molecules score highly while being chemically relevant. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### ===== Detailed pros: ===== \n\n1. The problem setup is highly relevant to real-world drug discovery:\n- The property being optimized is docking score, which is better than toy-ish tasks commonly used in other works, such as logp optimization.\n- By relying on a pre-computed (and then post-filtered) set of motifs, the model is designed to stay very close to the space of reasonable molecules. This is crucial to ensure that the molecules that come out are of any use: powerful and highly unconstrained generative models [1,2] tend to get exceptionally high scores on any (smooth enough) scoring function, but then show low quality quantitatively [3], trigger \"blind spots\" of learned property predictors [4], or just tend to produce structures that chemists consider nonsensical.\n- The paper also includes a scaffold-based evaluation, which is relevant to realistic drug design projects.\n\n2. The generative model proposed in this work extends existing fragment-based methods with several interesting modifications, which could be transferred to improve prior works as well:\n- Using fingerprints to compute motif representations, which could be useful for [3,5,6].\n- Explicitly considering attachment points, including attachment points on the initial scaffold. This is very relevant, and (to my knowledge) underexplored in existing work.\n- Filtering the fragment vocabulary to exclude groups that, albeit common, are likely to lead to molecules of low quality. As the authors note, one could even constrain the set to be objective-specific (potentially using something similar to [7]). This gives yet another useful place where expert knowledge can be plugged in. \n\n3. The quantitative and qualitative evaluations are (for the most part) nice and informative:\n- The authors consider three different quality metrics, which gives a better coverage of the different ways in which a compound can be \"bad\".\n- The ablation on the different RL algorithms that can be paired with the proposed generative model (Figure 5) is rather convincing.\n- The qualitative study (Section 4.4) is interesting; in particular, the method the authors use to mine good scaffolds (described in paragraph \"Scaffold detection\" in the Appendix) sounds natural and useful.\n\n### ===== Detailed cons: =====\n\n1. While the ideas/figures are nice, the text itself left me with a multitude of small-ish clarification questions:\n- Little is said about how the motif vocabulary is extracted. In particular, I wondered if two motifs with the same graph structure but different attachment points are considered different or if you perform a \"set sum\" on the attachment point sets. Close inspection of Figure 6 in the Appendix suggests it's the former (i.e. they are treated as different), but that should be made clearer.\n- A quite interesting detail about your method is whether the scaffold/fragment attachment points are explicitly present in the graph (either as nodes or as flags on nodes), as that means the model can learn to condition on them. Again, I think the answer is that they are explicit separate nodes in the graph, which I infer from Table 3 in the Appendix, but even there it's not said explicitly.\n- There is no explanation for why the Gumbel Softmax trick is needed, given that the parameters are trained using RL. The \"Gumbel softmax\" paragraph in the Appendix says it's because of the auto-regressive formulation of the actions. It seems that an alternative approach would be to consider the action selected in the last step as part of the environment – would that make sense?\n- Is the same GNN used to encode the current partial graph (for Action 1) and the newly selected fragment (for Action 3)?\n- I am confused about Equations 4-5. Shouldn't sum and expectation in Equation 4 be the other way around? Also, the part of Equation 5 after the \"=\" sign seems to be multiplying two terms, where one looks like a rewrite of the other?\n- I'm having troubles parsing the paragraph in lines 200-206. In RL terms, is it that \"priority estimator\" = \"policy\", but \"estimator\" = \"value function\"?\n- When you remove some fragments from your vocab, is it by simply interpreting them as full molecules and running the filters? Some filters (those that aren't based on exact substructure match but e.g. on SMARTS) may fire for a graph but not fire for its supergraph. In other words, I'm wondering if it's possible that some of the fragments you remove could give rise to full molecules that would pass the quality filters.\n- Caption of Table 1 says that you compute the quality scores for the first 3000 generated molecules. However, it's more interesting to look at optimized molecules, i.e. some number of best ones; are the results similar there?\n- Why does the study in Section 4.3 suddenly switch to the larger fragment vocabulary?\n- It's surprising that for the de novo case, you don't use an empty partial graph, and rather a benzene ring. I don't see a technical reason why starting with an empty graph wouldn't be possible - is it that you simply didn't want to deal with an empty partial graph in your code?\n\n2. It would be easier to contextualise this work if it also showed results on the Guacamol optimization benchmarks [8] and quality results obtained by using [9]. As SotA on Guacamol has been claimed by highly unconstrained models (which typically also get low quality), I wouldn't expect FREED to beat or even reach SotA in terms of raw score, as it's highly constrained by design. However, it would still be interesting to see how well it can optimize the scoring functions in Guacamol, and whether the optimized molecules indeed show very high quality. While Guacamol tasks may be a bit more synthetic than docking scores, they are still much more realistic than e.g. simple logP optimization, and hence worthwhile to compare on.\n\n3. From page 3 onward, the text gets a bit rough, with many typos and grammar errors. While these technically don't impede understanding (the issues that do, I mentioned in the point (1) above), it makes the reading experience not as pleasant as it could be. Not all of these would necessarily have to be fixed for publication, but fixing at least some would put the paper in a better shape:\n- a/an/the are missing in many places, e.g. \"a\" before \"more\" on line 36, \"a\" before \"fragment\" on line 86, \"an\" before \"RL\" on line 96, \"a\"/\"the\" before \"fragmentation\" on line 134, etc.\n- Some prepositions are off, e.g. \"compatible for\" -> \"compatible with\" on line 89, \"compatible to\" -> \"compatible with the\" on line 106, etc. \n- Some places use plural form when singular should be used, e.g. \"generations\" on line 89, \"follows\" on line 134, etc.\n- Finally, there are problems around citations: many are missing a space before them, [6] on line 30 is rendered as [6?], and citation [9] on line 77 says \"Olivecrona et al\", but it's actually \"Cieplinski et al\".\n\n### ===== Recommendation: =====\n\nOverall, I think the paper is quite interesting conceptually, and I like many of the ideas introduced. On the other hand, many important details are missing or unclear, and the text needs some revising. For now I'm marking this as a \"weak reject\", but if the authors clarify the issues I highlighted and update the text, I will consider increasing my rating.\n\n### =====  Other comments: =====\n\n- Figures 2 & 3 are very nice.\n- Do I understand correctly that FREED doesn't use any form of pretraining at all, and starts completely from scratch for every optimization task?\n- I noticed that FREED is using a plain sum aggregator to compute the graph embedding. Possibly using a more powerful aggregator (e.g. a multi-headed weighted sum or [10]) could improve results?\n- z^1st_act1 confused me a bit, until I realised you're using act1 as an index here. Maybe you could clarify this or just verbosely say \"the element under index act1\".\n- Line 232 says \"n=66\" without explanation; the fact that this is the number of fragments in the vocab is introduced later.\n- \"Markovic\" -> \"Markovian\"\n\n### =====  Some rephrasing suggestions: =====\n\n- Lines 108-109: \"RL algorithms on hard exploration problem\" -> \"RL algorithms for hard exploration problems\"; \"We view that there are two main approaches to achieve efficient exploration on a high level.\" -> \"Our view is that, on a high level, there are two main approaches to achieve efficient exploration\".\n- Line 118: \"approach is cannot\" -> \"approach cannot\"\n- Line 152: \"is a multi-set information for the entire graph\" -> \"encodes the entire graph as a multi-set of nodes\"\n- Line 159: \"two information\" -> \"two sources of information\"\n- Line 162 says pi_act2 takes Z^1s as an input: technically, it only takes one row (vector) of this matrix?\n- Line 174: \"policy such that\" -> \"policy that\"\n- First sentence of Section 6 could be reorganized.\n\n### ===== References =====\n\n[1] Reinforced Molecular Optimization with Neighborhood-Controlled Grammars \n\n[2] Efficient multi-objective molecular optimization in a continuous latent space \n\n[3] Learning to Extend Molecular Scaffolds with Structural Motifs \n\n[4] On Failure Modes of Molecule Generators and Optimizers \n\n[5] Junction Tree Variational Autoencoder for Molecular Graph Generation \n\n[6] Hierarchical Generation of Molecular Graphs using Structural Motifs \n\n[7] Multi-Objective Molecule Generation using Interpretable Substructures \n\n[8] GuacaMol: Benchmarking Models for de Novo Molecular Design \n\n[9] https://github.com/PatWalters/rd_filters \n\n[10] Principal Neighbourhood Aggregation for Graph Nets \n\n### ===== Update 24/08/2021 =====\n\nTo reflect the progress of the (still ongoing!) discussion with the authors and the other reviewers, I now raise both my score and confidence level by 1.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I believe limitations and societal impact are discussed in sufficient depth.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "BMmxzdsUmW", "reviewer": "Reviewer_BZA3", "summary": "This paper proposes a new RL algorithm for molecule design (called FREED). The RL algorithm builds molecules up through fragments; at each step the algorithm expands the current molecule state through picking three actions autoregressively: (i) which attachment site of the current molecule should be used, (ii) which new fragment should be attached to the current molecule, and (iii) which part of the picked fragment should connect to the current molecule at the just-picked attachment site. \n\nThe authors:\n* show how this RL algorithm can find better molecules (judged using three different docking tasks) compared to previous RL algorithms using atom-by-atom or SMILES-based construction techniques (Figure 4).\n* show that by controlling the fragment vocabulary used they are able to control the quality of the molecules the method produces (judged using pharmacochemical filter scores -- see Table 1).\n* develop a prioritized experience replay (PER) buffer based on a MC dropout model and show through an ablation study (Figure 5) that this is critical to the good performance of their approach. \n* show that their RL algorithm can be used in a scaffold-based scenario, to further improve promising scaffolds (Figure 6), rather than creating molecules only from scratch.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Review Summary\nOverall I found the paper interesting and the experimental results seem to validate the proposed approach. Code is provided to allow the method to be built upon in the future. A more minor criticism regarding clarity, was that I found it harder to work out how the models utilized by the PER actually worked (see Cl1); however, I believe this could be easily fixed prior to publication. Therefore, I have currently gone with a higher score.\n\n## Strengths\nS1. The authors present strong empirical results both in terms of the quality of the molecules they generate and the scores that these molecules obtain. The proposed method works much better than previous RL-based approaches for this task.  \nS2. The new PER (prioritized experience replay) methods appear to work well; they are shown in the ablation studies to be much better than the previous (TD) method (Figure 5). This could be interesting to RL practitioners in general and lead to future extensions.  \nS3. It is nice that the same model (after being trained) can be used for both de novo and scaffold-based approaches (i.e. it can be used to invent a new molecule from scratch or improve an existing one).\n\n## Quality\nAs far as I am aware the work seemed technically sound. The claims of being able to find better and higher \"quality\" molecules than previous methods are backed up empirically (although it would have been nice if they had compared to some of the other methods they mention -- see L1). The ablation study (section 4.3) is nice in breaking down the improvements of various aspects of their model.\n\n\n## Originality/Significance\n(O1) I found the authors' approach new and interesting. While fragment based approaches have been used before in deep generative models (e.g. [1,10]), pairing this idea with the RL algorithm proposed appears to work very well and significantly improves upon older methods. It is nice to see the authors evaluating their methods on docking scores, rather than the more simplistic metrics used by previous work.  \n\n(O2) In general the authors do a reasonable job of describing related work, given the space constraints (Section 2). However, if possible they want to mention:\n* some of the other previous RL based molecular optimization techniques, e.g.  \n>Segler, M. H. S., Kogej, T., Tyrchan, C. and Waller, M. P. (2018) ‘Generating Focused Molecule Libraries for Drug Discovery with Recurrent Neural Networks’, ACS central science, 4(1), pp. 120–131.\n\n* other recent ML work using fragments (in non-RL settings), e.g.    \n> Podda, M., Bacciu, D. and Micheli, A. (2020) ‘A Deep Generative Model for Fragment-Based Molecule Generation’, in Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics. PMLR, pp. 2240–2250.  \n> Xie, Y., Shi, C., Zhou, H., Yang, Y., Zhang, W., Yu, Y. and Li, L. (2021) ‘MARS: Markov Molecular Sampling for Multi-objective Drug Discovery’, in International Conference on Learning Representations. Available at: https://openreview.net/forum?id=kHSu4ebxFXY.\n\n* or some of the older fragment-based approaches from the chemoinformatics literature, see e.g.  \n> Hartenfeller, M. and Schneider, G. (2011) ‘Enabling future drug discovery by de novo design’, WIREs Computational Molecular Science, 1(5), pp. 742–759.\n\n\n## Clarity\nI thought the paper was generally well structured and the figures were helpful for explaining the approach.\n\n(Cl1) However, I found the description of the explorative algorithms in Section 3.2 harder to follow. I was not quite sure about the differences between the Bayesian uncertainty and predictive error approaches. Are both these approaches only trained on the structures for which you have docking data? Are the networks for the two approaches the same (apart from the addition of dropout)? \n\n(Cl2) I was impressed by the experiment in Appendix A.1, investigating whether one could improve the quality of the molecules generated from the baselines by using a multiobjective reward function (helping to show that while the quality could improve, this could adversely affected the docking score, _and_ still was not as effective as the fragment-based approach proposed in the paper). However, I did not see any mention of this result in the main paper. It might be nice to more explicitly link to the appendix in the main paper so that the reader is aware of these extra details. (Another place this might be helpful is for the scaffold section lines 281-295 -- when initially reading this section I was wondering where the initial scaffolds came from and it would have been nice to have an explicit link to Figure 2 of the Appendix.)\n\n\n\n## Minor Comments\n\n(M1) The authors mention in the first paragraph of the introduction the importance of sample efficiency. However, this aspect of the approach seems relatively underexplored in the experiments section. How many docking scores do the different approaches require? How long does the docking take? My understanding is that docking scores are only computed at the end of the episode (i.e. rather than at every state). How does this affect the number of samples required versus the speed of convergence? The authors mention in the appendix that a faster form of docking is used (lines 105-110). How does this affect the fidelity of the results?\n\n(M2) I'm a little confused as to why the authors only take the first 3000 samples when analysing the quality of the molecules generated (Table 1). Why not the best 3000 or the final 3000? My understanding is that the first 4000 molecules for FREED are sampled randomly anyway rather than optimized (lines 97-99 of Appendix) so do these molecules have good scores? \n\n(M3) The authors mention in the appendix that they consider only episodes with a maximum length of 4 actions (Appendix line 86). How does this limit the size of the molecules produced? Does the algorithm have the ability to stop earlier? Were initial states other than benzene rings considered?\n\n(M4) I was wondering what happened to the wildcard nodes at the end of the episode. Do they get converted into another element or are these nodes simply removed from the graph?\n\n(M5) The authors consider fragment vocabularies which contain 66-91 members. This seems rather small. I believe previous ML fragment-based approaches have vocabularies of size 300+  [1,10]. The authors themselves mention that the larger library enables them to generate more unique molecules (lines 241-247) but I would be interested about how this quantitatively affects the quality scores.\n\n(M6) Other **very** minor comments:\n* The authors could perhaps explicitly write out the form for multiplicative interaction (MI) in the Appendix (lines 32-37) to save readers having to go look it up.\n* Figure 5 both the abbreviations BE and BU are used for Bayesian uncertainty.\n* line 157: the graph embedding of the molecule is denoted as $Z^{1st}$, is this a typo? I believe $h_g$ is meant here instead?\n* line 134: \"follows\" -> \"follow\" to conjugate correctly with \"we\" (I think also the word \"the\" is missing before the word \"fragmentation\" earlier in the sentence).\n* line 30: one of the references has broken.\n* line 294: RE: \"Markovic\". I believe \"Markovian\" is the more common expression...? (I apologize if both are used).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "## Limitations\n\n(L1) Aside from the comments I made about clarity in the box above, the other area in which I think this paper could be improved is the experimental baselines used. The authors compare to two previous RL based approaches, one based on SMILES, and one atom-based. I think it would have been interesting to also compare to some of the previous ML approaches based on fragments mentioned in Section 2.    \n\n(L2) It is nice that the authors discuss the limitation of their work wrt synthesizability in Section 5 and discuss potential future work in this direction.\n\n## Societal impact\nThe authors include an explicit broader impact section in their paper (Section 6) and discuss potential negative impacts.\n", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: In the end all reviewers were happy to accept this paper: I vote to accept. The reviewers had very detailed feedback about new experiments and about openly presenting results. The reason they voted to accept is because of the engagement by the authors with their concerns and the changes the authors promised in the camera ready version. Please carefully implement these suggested changes, it will make an already good paper an excellent contribution to the conference.
Decision: Accept (Poster)