Hit and Lead Discovery with Explorative RL and Fragment-based Molecule Generation
Authors: Soojung Yang
Doyeong Hwang
Seul Lee
Seongok Ryu
Sung Ju Hwang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Drug discovery
Molecule generation

Abstract: Recently, utilizing reinforcement learning (RL) to generate molecules with desired properties has been highlighted as a promising strategy for drug design. Molecular docking program -- a physical simulation that estimates protein-small molecule binding affinity -- can be an ideal reward scoring function for RL, as it is a straightforward proxy of the therapeutic potential. Still, two imminent challenges exist for this task. First, the models often fail to generate chemically realistic and pharmacochemically acceptable molecules. Second, the docking score optimization is a difficult exploration problem that involves many local optima and less smooth surface with respect to molecular structure. To tackle these challenges, we propose a novel RL framework that generates pharmacochemically acceptable molecules with large docking scores. 
Our method -- Fragment-based generative RL with Explorative Experience replay for Drug design (FREED) -- constrains the generated molecules to a realistic and qualified chemical space and effectively explores the space to find drugs by coupling our fragment-based generation method and a novel error-prioritized experience replay (PER). We also show that our model performs well on both de novo and scaffold-based schemes. Our model produces molecules of higher quality compared to existing methods while achieving state-of-the-art performance on two of three targets in terms of the docking scores of the generated molecules. We further show with ablation studies that our method, predictive error-PER (FREED(PE)), significantly improves the model performance.

Hit and Lead Discovery with Explorative RL and F Fragment-based Molecule Generation
Soojung Yang
Doyeong Hwang f AITRICS
AITRICS soojungydmit.edt
deserbeegle11@@mai1.com
Seul Lee KAIST cllenlee7390ggail..com com
Seongok Ryu
Sung Ju Hwang
AITRICS
AITRICS, KAIST
seongokryu@gaauux.oou co. co.kr
s sjhwangs2@kaist.commm ac kr
Abstract
Recently, utilizing reinforcement learning (RL) to generate molecules with de- sired properties has been highlighted as promising strategy for drug design. A molecular docking program molecule binding affinity can be an ideal reward scoring function for RL, as it is a straightforward proxy of the therapeutic potential. Still, two imminent challenges exist for this task. First, the models often fail to generate chemically realistic and pharmacochemically acceptable molecules. Second, the docking score optimization is difficult exploration problem that involves many local optima and less smooth surfaces with respect to molecular structure. To tackle these challenges, we propose novel RL framework that generates pharmacochemically acceptable molecules with large docking scores. Our method Fragment- based generative RL with Ex- plorative Experience replay for Drug design (FREED) constrains the generated molecules to a realistic and qualified chemical space and effectively explores the space to find drugs by coupling our Tragment-based generation method and novel error-rioortiize experience replay (PER). We also show that our model performs well on both de novo and scaffold-based schemes. Our model produces molecules of higher quality compared to existing methods while achieving state-of-the art performance on two of three targets in terms of the docking scores of the gener- ated molecules. We further show with ablation studies that our method, predictive error-PER (FRE significantly improves the model performance.
physical simulation that estimates protein-small small
1 Introduction
Searching for "hits", the molecules with desired therapeutic potentials, is a critical task in drug discovery. Instead of screening library of countless potential candidates brute-force manner, designing drugs with sample-efficient generative models has been highlighted as promising strategy, While many generative models for drug design are trained on the distribution of known active compounds [1-3], such models tend to produce molecules that are similar to that of the training dataset [4], which discourages finding novel molecules.
In this light, reinforcement learning (RL) has been increasingly used for goal-directed molecular design, thanks to its exploration ability. Previous models have been assessed with relatively simple
Currently MIT. Cuurently LG Al Research.
CCurently Galux inc.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(Fragment-bassde basdd) Ganaration Step
(Pradictive Error-]
KEY CONCEPT
Prioritized Experience Replay
wighto SMac
Experissos replay butfer
compling bo piloeity
5. Ry Su
Ganerd redeccll
Mexn shas ralicaee
S
St-1
Docking simulation
S., Sr, Sprss Pe.
upcode
(Entropy-rggulaized]
prediche arar
Findlng realistic hit moleculas via fragment- besed generation tion and effective exploration
T
oxperience
Figure I: Overview of our generative drug discovery method. To find realistic 'hit' molecules that have high docking scores, we combine fragment- Tragment-based generation method with SAC and PER. This figure illustrates our version of PER where the priority of the experience is defined by predictive error.
objectives, such as cLogP and QED, or estimated bioactivity scores predicted by auxiliary models [5-8]. However, high scores in those simple properties of molecules guarantee neither drug-like likenesss nor therapeutic potential, emphasizing the necessity of more relevant design objectives in generative tasks [9, 10]. The molecular docking method is physical simulation that estimates the protein- small molecule binding affinity, a key measure of drug efficacy. As docking simulations are a more straightforward proxy of therapeutic potential, coupling RL with docking simulations would be promising strategy. While the simplistic scores (e.g., cLogP, QED) are computed by a sum of local fragments' scores and are not function global molecular structure, making the optimization tasks relatively simple, docking score optimization is more demanding exploration problem for RL agents. change in docking score is nonlinear with the molecule's local structural changes, and significant variance exists among the structures of high-scoring molecules, meaning that there exists many local optima [9].
In addition, many previous RL models often suffer from generating unreal and inappropriate struc- tures. Docking score maximization alone is not sufficient to qualify molecules as drug candidates, as the molecules should satisfy strong structural constraints such as chemical realisticness and pharma- cochemical suitability. In other words, small molecule drug candidates should have enough steric stability to arrive at the target organ in the intended form (chemical realisticness), and they should not include any seriously reactive toxic substructure (pharmacochemical suitability). The low quality of generated molecules can arise from a single improper addition of atoms and bonds, which would deteriorate the entire sanity of the structure. Since such 'mistakes' easy occur, implicitly guiding the model (e.g.. jointly training the model with QED optimization) cannot completely prevent the mistakes. Thus, explicitly restricting the generation space within realistic and qualified molecules by generating molecules as combination of appropriate fragments can be a promising solution [1, 11, 12].
However, such a strong constraint in generation space would make the optimization problem of docking score even harder and render higher probability of over-fitting few solutions, urging need for better exploration for RL agents. In this respect, we introduce a new framework. Fragment- based generative RL with an Explorative Experience replay for Drug design (FREED), which encourages effective exploration while only allowing the generation of qualified molecules.
Our model generates molecules by attaching a chemically realistic and pharmacochemically ac- ceptable fragment unit on a given state of molecules at each step'. We enforce the model to form new bond only the attachment sites that considered as appropriate at the fragment library preparation step. These strategies enable us to utilize medicinal chemistry prior knowledge and successfully constrain the molecule generation within the chemical space eligible for drug design. We also explore several explorative algorithms based on curiosity-driven learning and prioritized
'In this work, we widely define the term "chemically realistic molecule" a stable molecule and narrowly define it as molecule that an assembly of fragments appear in the ZINC data. Also, we widely define the term "inappropriate molecule/fragment" or "pharmacochemically inacceptable molecule/fragmennt as a molecule that has nonspecific toxicity or reactivity. and narrowly define it as molecule that cannot pass through the three medicinal chemistry filters, which are Glaxo, PAINS, SureChEMBL filters.

---

experience replay (PER) [13]. We devise an effective novel PER method that defines priority as the novelty of the experiences estimated by the predictive error or uncertainty of the auxiliary reward pre- dictor's outcome. With this method, we aim to avoid the lack of robustness of previous methods and encourage the exploration of diverse solutions. We provide an overall illustration of our framework in Figure 1.
Our main contributions can be summarized as follows:
We propose novel RI. framework that can be readily utilized to design qualified molecules of high therapeutic potential.
Our fragment-basee based generation method including sannectivity-preseevvee fragmentation and augmentation allows our model to leverage Chemical prior knowledge.
We propose novel explorative algorithms based on PER and show that they significantly improve the model performance.
2 Related Works
SMILES- and atom-based generation methods. SMILES-basee methods [14] are infea- sible for scaffold-based tasks' since molecular structures can substantially change through the sequential extension of the SMILES strings. Also, as explained in the Introduction, atom-based gen- eration methods such as You et al.' GCPN [6] inherently suffer from unrealistic generated molecules. Thus, we focus our discussion on motif-based generation methods.
Mosed- molecular generation methods. A number of previous works [L. 11, 12] have in- vestigated similar motif-based molecule generation based on the variational autoencoders (VAE). JT-VAE and HierVAE [1, 11] decompose and reconstruct molecules into tree structure of motifs. These models might not be compatible with the scaffold-basee generation, since a latent vector from their encoders depends on motif-wise decomposition order which is irrelevant information for docking score that may bias the subsequent generation [12]. Maziarz et al. [12] also propose a VAF-based model which encodes a given scaffold with graph neural networks (GNNs) and decodes the motif-adding actions to produce an extended molecule. While Maziarz et al.'s motif-adding actions resemble our generation steps, we additionally introduce connectivity preeeevin fragmentation and augmentation procedure which helps our model generate molecules of better quality.
Coupling an RL policy network with fragment-based generation holds general advantages compared to VAE-based methods. For example, RL models do not need to be trained on reconstruction tasks which might restrict the diversity of generated molecules. Our work is one of the earliest applications of RL with fragment- baased molecular generation method. While Ståhl et al.'s DeepFMPO [15] is also an application of RL with a fragment- based molecular generation method, DeepFMPO is designed to introduce only slight modifications to the given template molecules, which would make inappropriate for the de novo drug design. Moreover, while DeepFMPO's s generation procedure cannot change the connectivity of the fragments of the given template, our method is free to explore various connectivity.
Docking as a reward function of RL. Studies on docking score optimization task has started very recently. Jeon et al. [16] developed MORLD, atom-based generative model guided by MolDQN algorithm [17]. Olivecrona et al. [10] and Thomas et al. [18] utilized [EINVENT[14], a simplified molecular-input line-entry system (SMILES) model generative model guided by improved REINFORCE algorithm [19] to generate hit molecules.
RL algorithms for hard-exploration problems. Our view that, on a high level, there are two main approaches to achieve efficient exploration. The first one is introduce the "curiosity" or exploration bonus as intrinsic reward [20-22] for loss optimization. Bellemare et al. [23] first exploration bonus
proposed a pseudo-count-basee curiosity as distance between the true next state feature vector and the predicted estimate of the next state feature vector. Thiede al. [25] brought curiosity- learning into the molecular generation task. In Thiede et al., curiosity is defined as a distance between the true reward of the current state and the predicted estimate of the current state reward. However, these cuuriosity-drivnn intrinsic reward-based models sometimes fail to solve complex problems [26]. The failures are explained as a result of the RL agent's 's detachment derailment from the optimal solutions [27].
intrinsic reward. Pathak et al. [24] defined
A drug design scheme where the drug candidates are designed from the given scaffold.
3

---

(a)
Goaph entedkigg
Note ambedding
Chemically realistk fraraentilbrary
Action
Augmented fragment
Artion 1
Q
Action
Possible attachment
Neat state molecute S
Current state melecula St
(b)
Graph
Grach
Graph ECFP
Tyttt
Tactt
Action
Action 2
Action
Figure 2: Overview of our generation method (a) and policy network (b). Small colored squares represent a graph embedding of molecule, node embedding of each attachment sitem and ECFP representation of each fragment. The graph or node embeddings and ECFP representations from the chosen actions are autoregressively passed onto the next action's 's policy network.
The other approach of solving hard exploration problems is sample-efficien use of experiences [28, 13]. Prioritized experience replay (PER) method introduced in Schaul et al. [13] samples experiences that can give more information to the RL agent and thus have more 'surprisal' defined by the temporal-differee (SII.) introduced in Oh et al. [29] samples only the 'good' experiences where the actual reward is larger than the agent's 's value estimate (i.c., estimate from the value function or Q function). However, prioritized sampling based on the agent's value estimate susceptible to outliers and spikes, which may lead to destabilizing the agent itself [30]. Moreover, our explorative algorithm aims to preserve sufficient diversity and find many possible solutions instead finding the most efficient route to single solution. Thus, we modify the formulation of priority using the estimates for sample novelty.
(TD) error - in higher probability. a similar sense, self-imitation learning
Methods
3.1 Generation method
Generation steps. The key concept of our method to generate high-quality molecules that bind well to a given target protein. In order to achieve this goal, we devise a fragment-based generation method in which molecules are designed as an assemble of chemically realistic and pharmacochemically acceptable fragments. Given molecular state, our model chooses "where to attach a new fragment", "what fragment to attach", and "where on a new fragment to form a new chemical bond" in each step". Note that the action "where to form new bond" makes our model compatible with docking score optimization since the scores depend on the dine-dimessional arrangement of molecular substructures.
Preserving fragment connectivity information in molecule generation. Another important fea- ture of our method is harnessing the predefined connectivity information of the fragments and initial molecules. This feature allows our model to leverage chemists' expert knowledge in several ways. The connectivity information arises in the fragmentation procedure, in which we follow CReM [31] when the algorithm decomposes any arbitrary molecules into fragments while preserving the fragments' attachment sites shown in Figure 3(a).
In the GNN embedding phase, the attachment sites are considered as nodes like any other atoms in the molecule, while an atom type is exclusively assigned to the attachment sites. We also keep tracking the indices of the attachment sites as the states so that our policy can choose the next attachment site where new fragment should be added (Action 1). Similarly, we keep the indices of the attachment sites the fragments and use them throughout the training so that our policy can choose the attachment site from the fragment side (Action 3).
Start and end points of new bonds are restricted the attachment sites of given fragments and molecules. This restriction contributes to the chemical realisticness since the stability of the molecule
*Our model is designed to finish the episodes after four steps (in de novo cases) or two steps (in scaffold-bass cases). At the end of the episode, the model substitute all the remaining attachment sites with hydrogen atoms.

---

bond
NH
Pyrrol
Pyrroilm not anrmatic undaalle
2-medhyloyyrle amrrati, atabie
aomenis stacle
Figure Advantages of information is preserved as attachment sites in the fragmentation procedure. (b) Attaching new bond to the nitrogen of a pyrrole ring would result unstable molecule, breaking the aromaticity. Thus, pyrrole rings within an existing molecule are connected to other fragments by its carbon atoms, and preserving such an information would help us construct chemically realistic molecules.
fragmentation. (a) Substructure connectivity
depends on where the fragment is attached as illustrated in Figure 3(b). Also, we can utilize our prior knowledge of protein-ligand interaction by rationally assigning the attachment sites of the initial molecule (scaffold), which have been widely harnessed by medicinal chemists usual scaffold-based scenario.
Policy network. Our generation method is coupled with the policy networks which guide the model generate hit molecules. We provide overall illustration of policy network in Figure 2. Our state embedding network and policy network are designed as Markov models so that generation steps take any arbitrary molecule as a current state and predict the next state, making the model more plausible for a scaffold-based generation.
In our framework, each state vector 8; represents molecular structure at the (-th step, and each step is a sequence three actions that define an augmentation fragment. Following the method from Hwang et al. [32]. the current state molecule is represented consists of a node feature matrix, an adjacency matrix, and an attachment site masking vector. A graph convolutional network (GCN) [33] -based network provides node embeddings, H, which then aum-aggregated to produce a graph embedding, Ng. Note that the same GCN encodes the current state molecular graph (for Action 1) and the fragments (for Action 3).
undirected graph G' which
Our policy network is an autoregressive process where Action depends on Action I & 2 and Action 2 depends on Action 1.
act -Tall (2) 21st MI(hy: Hum)
n012 -Tac12 2nd 2nnd MI(= ECFP(h)gu))
(2)
p
nc13 - (2dnd), 2 MI(2 Ucand)
(3)
where Han refers to the node embeddings of attachment sites. For the first step of our policy network, Taet takes the multiplicative interactions (MI) [34] of the node embedding of each attachment site and the rows of the graph embedding of the molecule, 21st E -2sst, as inputs and predicts which attachment site should be chosen as Action Since graph embeddings and node embeddings are defined in a heterogeneous space, we apply MI to fuse those two sources of information.
Given Action Faa2 takes the MI of 2|11 the row of Z under index actl, and the RDKit [35] ECFP (Extended Connectivity Fingerprint) representation each candidate fragment, hga.s as inputs, and predicts which fragment should be chosen as Action 2. By taking Zist as one of the inputs, Tact2 reflects on the context of the current state and Action 1.
Finally, given Action and 2, Taet3 takes the MI of 2nd and the node embeddings of the chosen fragment's 's attachment sites, Ucand, as inputs and predicts which fragment attachment site should be chosen as Action 3. Each of the three policy networks Tattl Tact2 and тa consists three fully-connected layers with ReLU activations followed by softmax layer to predict the probability of each action. In order to make gradient flow possible while sampling discrete actions from the probabilities, we implement the Gumbel-softmax reparameterization trick [36].
5

---

3.2 Explorative RL for the discovery of novel molecules
Soft actor-critic. We employ soft actor-critic (SAC), an off off-policy actor-critic RL algorithm based on maximum entropy reinforcement learning [37, 38], which is known to explore the space of state- action pairs more effectively than predecessors. Assuming that our docking score optimization task requires more effective exploration than simplistic tasks, we chose SAC as our RI. baseline algorithm.
n arg max
Σ BE Feer,ae)rp,, ar) + (HH(((((+)))]
(4)
x
E E 1 (a log π (ma ss] Σ (at(at o0 (s))
(5)
x (log π (a (s)) - log π (a. (s,, a.t + log ग (aa st, ar" aaa)))
SAC aims attain optimal policy that satisfies (4) where a is the temperature parameter balancing between exploration and exploitation of the agent, (-(((-);)) is entropy of action probabilities given 8t, and fr is state-action transition distributions created from æ. As we define act act2 and act3 autoregressively, the entropy regularization term defined in [39] Es (a log x (a|s)] is decomposed into (5).
Explorative algorithms. To encourage exploration, we prioritize novel experiences during sam- pling batches for RL updates. We regard an experience as a novel experience if the agent has not visited the state before. Defining priority estimate function in the state space and not in the state-action space has been introduced for the molecular generative task in Thiede et al. [25]. For novel states, the reward estimator would yield high predictive eπrоr high variance (Bayesian uncertainty). In this regard, we train an auxiliary reward predictor consisting of a graph encoder and fully- layers that estimate a given state's reward (docking score). Then, we the predictor's predictive error (1.2 distance) or Bayesian uncertainty as priority estimate. We name the former method PER(PE) and the latter method PER(BU). The use of predictive error as novelty estimate has been introduced for curiosity-driven learning [25], but our work is the first to apply this to PER.
For PER(BU), we follow Kendall et al. [40] to obtain the Bayesian uncertainty of the prediction. We train the reward predictor network to estimate the reward's mean and variance (aleatoric uncertainty). Every layer in the network is MC dropout [41] layer so that the predictor provide the epistemic uncertainty. We add aleatoric and epistemic uncertainty to obtain the total uncertainty of the estimate.
The reward predictor is optimized for every docking step, and we only optimize it based on final state transitions since docking scores are only computed for the final states. The reward predictor predicts the reward of any state, both intermediate and final. For PER(PE), when we compute the priority of a transition including an intermediate state, we use Q value as a substitute for the true docking score. After updating the policy network and Q function with loss multiplied by importance sampling (IS) weight, we recalculate and update the priority values of the transitions in the replay buffer.
The following experiments section shows that PER with our priority estimate functions performs better than the previous methods.
Results and Analysis
4.1 Quantitative metrics
In this section, we introduce quantitative metric scores we used to assess our model. For every metric, repeated every experiment five times with five different random seeds and reported the mean and the standard deviation of the scores. Also, we calculated the scores when 3,000 molecules were generated and used to update the model during training.
Quality score. We report three widely used pharmacochemical filter scores Glaxo [42]. SureChEMBL [43], PAINS [44] - as quality scores. The quality scores are defined as a ratio of accepted, valid molecules to total generated molecules, the filters reject the compounds that contain functional groups inappropriate for drugs (i.e., toxic, reactive groups). The higher the quality scores, the higher the probability that the molecule will be acceptable drug. We also report the ratio of

---

valid molecules to total generated molecules (validity) and the ratio of unique molecules among valid generated molecules (uniqueness).
Hit ratio. We define hit ratio as a ratio of unique hit molecules to total generated molecules. We report the hit ratio compare the model's ability produce as many unique hit molecules in a given length of iterations, where we define hits as molecules whose docking scores are greater than the median of known active molecules' docking scores.
Top 5% score. We report the average score of the top 5%-scored generated molecules to compare the model's ability to produce molecules with better docking scores.
4.2 Quantitative performance benchmark
In this section, we compare the quality of the generated molecules and the model performance with three baseline models, MORLD [16], REINVENT [14], and HierVAE [11]. Our model, FREED(PE). our fragment-bass
generation method coupled with SAC and PER(PE). MORLD and REINVENT are the models utilized for docking score optimization tasks in previous works [16, 10, 18]. HierVAE a strong non-RL fragment-
molecular generative model. Since ViE is distributional learning method, we train HierVAE in two schemes 'one-time" training the known active molecules (HierVAB V) and "active learning (AL)' training where we train the model once on the known actives and twice the top-scoring molecules from the generated molecules (HierVAE(AL)). In both schemes, we initialized the models with the model pre-trained on ChEMBL. We trained the models for three carefully chosen protein targets fa7, parp1, and 5htlb. The choice of protein targets and the specifics of experimental settings are described in Appendix. For the experiments in this section, we use the small fragment library that includes 66 pharmacochemically acceptable fragments.
Table 1: Quality scores of the models. We trained our model and three baseline models with the target fa7 and computed quality scores of the first 3,000 molecules generated during training for each model. The two baseline models REINVENT and MORLD that are jointly trained to maximize filter scores noted as 'REINVENT w/ filter' and 'MORLD w/ filter'. Standard deviation is given in brackets.
Glaxo
SureChEMBL.
PAINS
validity
uniqueness
MORLD
0.561 (.009)
0.131 (.013)
0.805 (.013)
1.000 (.000)
1.000 (.000)
MORLD w/ filter
0.578 (.010)
0.145 (.018)
0.816 (.008)
1.000 (.000)
1.000 (.001)
REINVENT
0.773 (.023)
0.667 (.030)
0.769 (.022)
0.813 (.024)
0.988 (.008)
REINVENT w/ filter
0.832 (.034)
0.747 (.040)
0.842 (.034)
0.872 (.028)
0.990 (.007)
HiE
0.899 (.027)
0.748 (.024)
0.975 (.006)
1.000 (.000)
0.138 (.006)
HierVAE(AL)
0.975 (.004)
0.795 (.007)
0.893 (.011)
1.000 (.000)
0.131 (.003)
Ours: FREED(PE)
0.996 (.001)
0.808 (.049)
0.991 (.002)
1.000 (.000) 0.723 (.135)
Quality scores of generated molecules. In real-world drug design, the generated molecules should be acceptable by pharmacochemical criteria. We excluded the fragments that are considered inap- propriate by pharmacochemical filters to guarantee the quality of the generated molecules. Such an explicit constraint cannot be applied to atom-based (MORLD) or SMILFSS-baed (REINVENT) methods.
As shown in Table 1, our model mostly generated acceptable molecules while the other models showed the poor rate of acceptable molecules, confirming our approach's 's advantage in drug design. We also investigated whether one could improve the quality of the molecules generated from the baselines by using a multi-objective reward function. MORLD and REINVENT optimized in multi-objective with both docking scores and the quality scores, denoted as 'MORLD w/ filter' and REEIVVNT w/ filter', show improved quality scores compared to single-objective MORLD and REINVENT. However, the multi-objective reward method was not as effective as our fragment-based approach, strengthening our claim that the explicit constraints are the effective strategy. Such trend was consistent for all three targets. See Appendix for visual inspection of the quality score results.
7

---

fa7
parpl
5ht1b
(a)
0.35
6.11
REBATNT
0.61
EORMIT
HOLS
HNHAN Hande:)1)
KLMIINT
(b)
-10
Figure 4: Hit ratio and top 5% score of model FREED(PE), REINVENT, MORLD, Hier- VAE, and Hier VAE(AL). Standard deviation is given as error bars. Higher hit ratios and greater negative value of the top 5% scores indicate better performance.
HierVAE showed high quality scores, the HierVAE fragment library itself had very few problematic substructures (See Appendix A.5 for details). Such a result substantiates the advantage of the explicit fragment-han based approach. However, HierVAE showed low uniqueness, possibly due to the small size of training data for fine-tuning (~1,200 known active molecules) and active learning (~1,500 generated high-scoring molecules).
We provide quality scores our model (FREED(PE) trained with the small library and the large library for all three protein targets in Table 2 and Table 3 of Appendix A.1, respectively. A significant increase uniqueness was observed when we used the large library, which implies that the low uniqueness of our model is due to the small size of the fragment library. Thus, believe con- structing a fragment library that large enough to guarantee high uniqueness while only including pharmacochemically acceptable fragments will be the best strategy in production.
Docking score optimization result. Figure 4 shows the hit ratios and top 5% scores of the gen- erative models. Our model outperforms the other generative models MORLD, REINVENT, and HierVAE terms of both hit ratio and top 5% score, except for the hit ratio of the 5htlb case. Such results show that model's 's performance is superior or least competitive to existing baselines, while our model exhibits many more practical advantages such as generating acceptable molecules, integrating chemist's expert knowledge by srmnectivtyy-preeeving fragmentation and augmentation, and the feasibility both de novo and scaffold-basee generation.
4.3 Ablation studies: explorative algorithms
We also perform ablation studies of our algorithm to investigate the effect of our explorative algorithms. We used the larger library of 91 unfiltered fragments, as this section assesses the effect of the algorithms on the model's performance regardless of the pharmacochemical acceptability.
In Figure 5, we observe that all SAC models with explorative algorithms performed better than the vanilla SAC model, while the PPO model showed the worst performance. FREED with both PE and BU outperformed curiosity-drive in our task compared to curiosity-driven explorations. Moreover, our predictive error-PER method outperformed the TD error-PER PER method. We conjecture that such a result is due to 1) novelty- based experience prioritization encourages better exploration leveraging an auxiliary priority predictor network makes PER training more robust than internal value estimate functions (Q function). We provide the significance analysis (one-tail paired t-test) the result Table of Appendix A.I. 4.4 Case study on drug design
models with PE and BU, showing the effectiveness of our methods
In this section, we show the practicality of our framework on novo and scaffold-based drug design. We test FREED(PE) with our large fragment library which includes 91 fragments.

---

parp1
3.25
k.00
8mh
800
SAC
SAC
--1
---1
-30
-15
100
302/11 01.03
19:04:1
Figure 5: Hit ratio and top 5% score ablation studies. Models can be categorized by whether they use (PER, curiosity-driven exploration(curioo) and whether they use (predictive error from predictor(PE), Bayesian UnCertainty(BU), and TD error from agent(TD)) priority or intrinsic reward. Standard deviation is given as error bars.
fa7
parpl
5htlb
nevo
Doocdnng Score
Docodnng Score
Docling
-30
PREECYPE)
Random
Aetiv
Inactiv
FREEDSPE)
Sandom
Active
Iracthue
PREEDAPE)
Random
Active
Inactive
Dredkins Sear
Broking Soote
PREEDVPE)
Random
Aclive
Inscivv
PREEDIPE) Rendom
Aftive
Inachhe
PREEDUPE) Random
Active
(RaCTive
Figure Docking score distribution of the generated molecules. Duplicate molecules were removed after gathering 3,000 molecules each from five random seed experiments. "Random" molecules are generated by our fragment-based generation method without training the policy net- work. "FREED(PE)" molecules are generated by the fragment-based generation method while training the policy network. We also plot known "Active" and "Inactive" molecules from DUD-E (fa7, parp 1) ChEMBL (5htlb) datasets for comparison. Colored horizontal lines indicate the median of the corresponding distribution. (a) de novo scenario (b) scaffold-baseh scenario
De novo scenario. Figure 6 (a) shows the distribution the generated molecules before ("random") and after ("FREED(PE)") optimizing the policy network. Our model was able to effectively generate molecules that have higher docking scores compared the known active molecules. Figure (i) shows the structure of each target's 's optimized molecules.
Scaffold-based scenario. We validate our model on scaffold-bard scenario. where we attempt to improve docking scores by adding fragments to an initial scaffold molecule. Figure 6 (b) shows the distribution of the optimized molecules before ("random") and after ("FREED(PH)") training the policy network, with scaffold of each target as initial molecule.
Figure 6 (b) highlights our model's ability to optimize given scaffold to have higher binding affinity with the target. Surprisingly, in Figure (b), even the molecules randomly optimized with fragment-based generation algorithm show high docking scores when given the proper scaffold.

---

de novo (a) ta7
(b) parp1
(c) Sht1b
Docking m-11 scone 3.50
HI Docking acom
Hit Dooking scera -119
(i)
(a)
8.78
scaffold- based
Hil teoallodd
Load Dodiingecom -133
Hil (sealfed)) Dodesgacore 101
Lead Docsngssore 5A 2.54
Hil (ooafMe)
Lead Dooking scers SA.oow 3.42
Dockingscore -1.9
Dociiin ecore
Figure 7: Generated samples and their docking scores with our method, for de novo (i) and scaffold-based scenario (ii). For each target, of the high-scoring generated molecules is dis- played with the initial molecule (benzene ring or scaffold). The purple line highlights the fragments augmented by the model in scaffold- based generation. Numbers below the compounds the docking scores and SA scores.
This result implies the importance of scaffold in hit discovery and highlights our generative method's ability to to span the chemical space around the scaffold effectively.
Figure 7 (ii) shows the structure of each target's 's scaffold and corresponding optimized molecules. We can see that the scaffold structures are well preserved generated lead molecules. We provide an analysis of 3D docking poses of the scaffolds and generated lead molecules in Figure 2 and Figure of Appendix A.
It is notable that our framework does not have to be specific for de novo scaffold- scenarios, except for the initial molecule and number of fragments to be added. Since our model is fully Markovian, whether the initial molecule is benzene ring or scaffold does not affect the model's training.
Chemical realisticness of generated molecules. Figure we report the SA (synthetic acces- sibility) score of the molecules, which is widely used metric that estimates ease of synthesis by penalizing the presence of non-standard structural features. The SA score distribution of the catalogue molecules of commercial compound providers has its mode around 3.0 [45]. Accordingly, we can assume our generated molecules as reasonably synthesizable and thus chemically realistic.
Conclusion
In this work, developed FREED, novel RL framework for real-world drug design that couples fragment-basse -based molecular generation strategy with a highly explorative RL algorithm to generate qualified hit molecules. Our model generates pharmacochemically acceptable molecules with high docking scores, significantly outperforming previous docking RI. approaches. Our code is released at https://gi tttpggtthhbu..e/ thub. com/ AITRICS/ FFREED.
Limitations and future work. While our method does not explicitly account for the synthesizability of generated molecules, ours. It would be able to combine our method and forward synthesis- method by substituting attachment site bond formation actions with chemical reactions. In this way, we explicitly take synthesizability into account while providing the appropriate model inductive bias for docking score optimization. We leave such improvement as future work.
believe forward synthesis-based methods [46] can be complementary to
Negative societal impacts. used maliciously, our framework can be utilized to generate harmful compounds such as biochemical weapons. Thus, conscious use of the AI model is required.
10

---

Acknowledgments and Disclosure of Funding
We thank anonymous reviewers, Seung Hwan Hong, and Sihyun Yu for providing helpful feedback and suggestions. This work was supported by the National Research Foundation of Korea (NRF) grant funded by the project NRF2019M33ESD400696..
References
[1] W. Jin, R. Barzilay, and T. Jaakkola, "Junction variational autoencoder for molecular graph generation," in International Conference on Machine Learning. PMLR, 2018, pp. 2323-2332.
[2] A. Zhavoronkov, Y. A. Ivanenkov, A. Aliper, M. S. Veselov, V. A. Aladinskiy, A. V. Aladinskaya, A. Terentiev, D. A. Polykovskiy, M. D. Kuznetsov, A. Asadulaev et al., "Deep learning enables rapid identification of potent ddrl kinase inhibitors," Nature biotechnology, vol. 37, no. 9, pp. 1038-10401 2019.
[3] N. De Cao and T. Kipf, "Molgan: An implicit generative model for small molecular graphs." arxiv preprint arxiv:1805 1973, 2018.
[4] W. P. Walters and M. Murcko, "Assessing the impact of generative ai on medicinal chemistry," Nature biotechnology. vol. 38, no. 2, pp. 143-145, 2020.
[5] Z. Zhou, S. Kearnes, L.. Li, R. N. Zare, and P. Riley, "Optimization of molecules via deep reinforcement learning," Scientific reports, vol. 9, no. 1, pp. 1-10, 2019.
[6] J. You, B. Liu, R. Ying, V. Pande, and J. Leskovec, "Graph convolutional policy network for goal-directed molecular graph generation," arXiv preprint arXiv: 1806.0246.. 2018.
[7] M. Popova, O. Isayev, and A. Tropsha, "Deep reinforcement learning for de novo drug design." Science advances, vol. 4, no. 7, p. eaap7885, 2018.
[8] T. Blaschke, O. Engkvist, J. Bajorath, and H. Chen, "Memory-assisted reinforcement learning for diverse molecular de novo design," Journal of cheminformatics, vol. 12, no. pp. 1-17, 2020.
[9] W. C. Coley, "Defining and exploring chemical spaces," Trends in Chemistry, vol. 3, pp. 133- 145, 2021.
[10] Cieplinski, T. Danel, S. Podlewska, and Jastrzębski, "We should at least be able to design molecules that dock well," arXiv preprint Skki::2006.1655511 2020.
[11] W. Jin, R. Barzilay, and T. Jaakkola, "Hierarchical generation of molecular graphs using structural motifs," Proceedings of the 37th International Conference on Machine Learning, PMLR, no. 119, pp. 4839-48444 2020.
[12] K. Maziarz, H. Jackson-Flus Flux, P. Cameron, F. Sirockin, N. Schneider, N. Stiefl, and M. Brockschmidt, "Learning to extend molecular scaffolds with structural motifs," arXiv preprint
2021.
[13] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, "Prioritized experience replay," arXiv preprint 15 2015.
[14] M. Olivecrona, Blaschke. O. Engkvist, and H. Chen, "Molecular de-novo design through deep reinforcement leaminn." Journal of Cheminformatiic, vol. 9, no. 48, 2017.
[15] N. Ståhl, G. Falkman, A. Karlsson, G. Mathiason, and J. Boström, "Deep reinforcement learning for multiparameter optimization in de novo drug design," " J. Chem. Inf. Model., vol. 59, no. pp. 3166-3176, 2019.
[16] W. Jeon and D. Kim, "Autonomous molecule generation using reinforcement learning and docking to develop potential novel inhibitors." Scientific Reports, vol. 10, no. 22104, 2020.
[17] 7. Zhou, Kearnes, I.. Li, N. Zare, and P. Riley, "Optimization of molecules via deep reinforcement learning," Scientific Reports, no. 10752, 2019.

---

[18] M. Thomas, R. Smith, N. M. O' O'Boyle, C. de Graaf, and A. Bender, "Comparison of structure- and ligand- based scoring functions for deep generative models: a gper case study." ChemRxiv preprint, 2021.
[19] R. J. Williams, "Simple statistical gradient-folowingg algorithms for reinoreemet learning," Machine Learning, vol. 8, pp. 229-256, 1992.
[20] R. M. Ryan and E. L. Deci, "Intrinsic and extrinsic motivations: Classic definitions and new directions," Contemporary educational psychology, vol. 25, no. 1, pp. 54-67, 2000.
[21] P.-Y. Oudeyer and F. Kaplan, "What is intrinsic motivation? a typology of computational approaches," Frontiers in neurorobotics, vol. 1, p. 6, 2009.
[22] A. G. Barto, "Intrinsic motivation and reinforcement learning," in Intrinsically motivated learning in natural and artificial systems. Springer, 2013, pp. 17-47.
[23] M. G. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos, "Unifying count-based exploration and intrinsic motivation," arXiv preprint arXiv: 01868, 2016.
[24] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, "Curiosity-driven exploration by self- supervised prediction," in International Conference on Machine Learning. PMLR, 2017, pp. 2778-2787.
[25] L. A. Thiede, M. Krenn, A. Nigam, and A. Aspuru-Guzik, "Curiosity in exploring chem- ical space: Intrinsic rewards for deep molecular reinforcement learning," arXiv preprint arXiv:2012.11293, 2020.
[26] A. A. Taiga, W. Fedus, M. C. Machado, A. Courville, and M. G. Bellemare, "Benchmark- ing bonus-based exploration methods on the arcade learning environment," arXiv preprint 1908 02388, 2019.
[27] A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune, "Go-explore: a new approach for hard-exploration problems," arXiv preprint arXiv: 10995, 2019.
[28] T. L. Paine, C. Gulcehre, B. Shahriari, M. Denil, M. Hoffmann, H. Soyer, R. Tanburn, S. Kaptur- owski, N. Rabinowitz, D. Williams et al., "Making efficient use of demonstrations to solve hard exploration problems," arXiv preprint
2019.
[29] Oh, Y. Guo, S. Singh, and H. Lee, "Self-imitation leaming." in International Conference on Machine Learning. PMLR, 2018, pp. 3878-3887.
[30] S. Lee, J. Lee, and I. Hasuo, "Predictive per: Balancing priority and diversity towards stable deep reinforcement learning"" arXiv preprint arXiv:2011.13093 2020.
[31] P. Polishchuk, "Crem: chemically reasonable mutations framework for structure generation," Journal of Cheminformatics, vol. 12, no. 28, 2020.
[32] D. Hwang, S. Yang, Y. Kwon, K. H. Lee, G. Lee, H. Jo, S. Yoon, and S. Ryu, "Comprehensive study on molecular supervised learning with graph neural networks." Journal of Chemical Information and Modeling. vol. 60, no. 12, pp. 5936-5945, 2020.
[33] N. Kipf and M. Welling, "Semi 'Spmiisupe classification with graph convolutional networks." arXiv preprint NxXiv:660.00200) 2016.
[34] S. M. Jayakumar, W. M. Czarnecki, J. Menick, J. Schwarz, J. Rae, S. Osindero, Y. W. Teh, T. Harley, and R. Pascanu, "Multiplicative interactions and where to find them," in International Conference on Learning Representations, 2019.
[35] G. Landrum, "Rdkit: Open-source cheminformatics software," 2016. [Online]. Available: http://hhh...ommmiiiidddceeeeeee
2016_09. 09_4
[36] E. Jang, S. Gu, and B. Poole, "Categorical reparameterization with gumbel-softmax," arXiv preprint arXiv:1611.01144 2016.
12

---

[37] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, "Soft actor-critic: Off- maximum entropy deep reinforcement learning with a stochastic actor," in International Conference on Machine Learning. PMIR, 2018, pp. 1861-1870.
[38] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel et al., "Soft actor-critic algorithms and applications," arXiv preprint arXiv:1812.05905, 2018.
[39] P. Christodoulou, "Soft actor-critic for discrete action settings." arXiv preprint
WwXiv:1910007207 2019.
[40] A. Kendall and Y. Gal, "What uncertainties do we need in bayesian deep learning for computer vision?" arXiv preprint (7Xix:1703.0497) 2017.
[41] Y. Gal and Z. Ghahramani, "Dropout as a bayesian approximation: Representing model uncer- tainty in deep learning," in international conference on machine learning. PMLR, 2016, pp. 1050-1059.
[42] S. Lane, D. S. Eggleston. K. A. Brinded, J. C. Hollerton, N. L. Taylor, and S. A. Readshaw, "Defining and maintaining high quality screening collection: the gsk experience," Drug discovery today, vol. 11, no. 5-6, pp. 267-272, 2006.
[43] G. Papadatos, M. Davies, N. Dedman, J. Chambers, A. Gaulton, J. Siddle, R. Koks, S. A. Irvine, J. Pettersson, N. Goncharoff el al., "Surechembl: large-scale, chemically annotated patent document database," Nucleic acids research, vol. 44, no. D1, pp. 11220-D1228, 2016.
[44] J. B. Baell and G. A. Holloway, "New substructure filters for removal of pan assay interference compounds (pains) from screening libraries and for their exclusion in bioassays." " Journal of medicinal chemistry, vol. 53, no. 7, pp. 2719-2740, 2010.
[45] P. Ertl and A. Schuffenhauer, "Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions," Journal of Cheminformatics, vol. I, no. 8, 2009.
[46] S. K. Gottipati, B. Sattarov, S. Niu, Y. Pathak, H. Wei, S. Liu, Y. Bengio al., "Learning to navigate the synthetically accessible chemical space using reinforcement learning," 2020.
13


Reviews:
[{"review_id": "vmb-scxCxNG", "reviewer": "Reviewer_USDx", "summary": "The authors present a reinforcement learning framework for the design of molecules active against specific proteins. The novelty in the authors methodology is to attach molecular fragments (as opposed to single atoms) at each step and optimise the algorithm with respect to the molecular docking score. Attaching fragments allows for the generation of pharmacochemically-valid molecules (those which are suitable drug-candidates) by only attaching pharmacochemically-valid attachments. Optimising with respect to the docking score allows the algorithm to attain a good proxy of the binding affinity of a given molecule to a given protein.\n\nGiven a current state (the current molecule), the framework designs molecules in three steps: \n\n1) The optimal attachment site is identified on the current molecule. \n2) Given the choice of site, the optimal molecular fragment is chosen from a set of predefined fragments.\n3) Given the choice of site and fragment, the optimal attachment site on the fragment is identified.\n\nThe experimental results indicate that the proposed approach generates many valid molecules in comparison to baseline approaches.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nI believe the work possesses originality in the way the authors have designed their framework. The novelty in the authors methodology is to attach molecular fragments (as opposed to single atoms) at each step and optimise the algorithm with respect to the molecular docking score. Attaching fragments allows for the generation of pharmacochemically-valid molecules (those which are suitable drug-candidates) by only attaching pharmacochemically-valid attachments. Optimising with respect to the docking score allows the algorithm to attain a good proxy of the binding affinity of a given molecule to a given protein.\n\nQuality:\nWhilst the results look promising, I think there are areas for improvement. The standard of writing could be improved as it can be confusing at times. The interpretation of the results seems shallow. \n\nFor instance, why ignore the very large standard deviations in figure 4? The authors state (lines 250-256) \"Our model outperforms the other generative models MORLD and REINVENT in terms of both hit ratio and top 5% score, except for the hit ratio of the 5ht1b case. Such results show that our model’s performance is superior or at least competitive to existing base lines...\". On average, this does happen. Yet, the large variability indicates that it is not always the case. This deserves further investigation, surely? Either through more random restarts or at least a plausible explanation of why this variability occurs. On the topic of figure 4, it does not indicate which plots correspond to which proteins. Also, I think a description of how to read the top 5% plots would be helpful (I can only assume the greater negative scores indicate better performance?). \n\nThe interpretation of Figure 7 is that the molecules look \"chemically realistic\". I find this phrase somewhat misleading since, to me (I'm not a chemist), any two-dimensional molecular graph looks chemically realistic. What makes these specific structures look realistic? I understand the authors want to show off the molecules they have generated (rightly so!) but they must be careful with their interpretations.\n\nClarity:\nThis is mentioned in the previous section but the standard of writing could be improved. The work seems a little rushed, which is a shame since it diverts the attention from the nice results.\n\nSignificance:\nMachine learning applied to drug-design is currently a fairly hot topic, so this paper is of obvious significance.\n\nGeneral remarks:\nThe paper makes a nice contribution through the use of fragment-based drug design and the results mainly back this up (see my comment regarding  Figure 4). However, the work seems more like a first draft than a submission-ready paper. For this reason, I would have to reject.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Sections 5 & 6 cover this sufficiently.", "review_text": "", "extra_scores": {"ethical_concerns": "Section 6 covers this sufficiently.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Xx85gmwf1VP", "reviewer": "Reviewer_TmJ6", "summary": "The paper proposes a RL framework to generate molecules by assembling molecular fragments, such that the ending result has optimized docking scores. The RL model uses docking score as a reward function; an autoregressive policy network which chooses among three possible actions to modify an existing molecule (source attachment site, fragment to attach, target attachment site); a Soft Actor-Critic algorithm to learn the optimal policy, and explores different Prioritized Experience Replay strategies to produce diverse molecules. In the experiments, the authors show that the model can produce molecules with enhanced docking scores for some reference proteins, both in a de novo scenario and scaffold-based scenario. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main idea behind the paper is not novel (using docking scores as a reward function), but the proposed solution is interesting in principle. However, the paper is not written well; there are inconsistencies in notation (e.g. h_g is used for a graph representation, but then ECFP(h_g_{cand}) is used, while ECFP should take as input a molecule, not a representation), confusingly defined symbols (e.g. z_{1st}^{act1}), undefined acronyms (TD, BU, PPO), misspelled acronyms (BE instead of BU), which made reading this paper very difficult. Furthermore, the English language is really faltering sometimes. First and foremost, I suggest that the paper undergoes serious proofreading, otherwise I will be forced to confirm my (temporary) rejection.\n\nOn the technical side, I have several concerns: \n- on page 6, the authors affirm that the model has been trained on 3 different proteins. However, the results reported in Table 1 concern only one of them (fa7). I checked the supplementary material (not very carefully, I reckon) and there is no sign of the evaluation on those proteins. Why is that so?\n- on Figure 7b)  (and in Figure 7a) for protein 5htb1) the results basically say that the proposed method is no more effective than attaching fragments randomly to generate higher docking molecules, which makes me wonder if using PER strategies is effective at all;\n- how are bond types handled by the GCN? Judging from line 150, the GCN doesn't handle edge types, which is quite unlikely.\n- What are the novelty rates (i.e. the ratio of generated molecules that are not in the training set) of the model? Why you didn't report them?\n- Similarly, the authors use several techniques to improve the diversity when generating molecules. So I would have expected that the chemical diversity of the 3000 sample was measured, when in fact it wasn't.\n- When does the generative process stop? I couldn't see it explained on the paper. Perhaps I can find it in the supplementary material? It's information that I would like to read in the paper itself, though.\n\n__Overall judgement__\n\nRight now, I recommend rejection, given the technical doubts but most of all given the poor writing quality. I am open to revising my score, if the authors will put effort into making this paper readable and comprehensible (in which case, I will also go through the supplementary material more carefully).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are discussed to some extent. One that I find critical for these types of models is computational time. It takes a long time to compute accurate docking simulations. In this paper, the authors use a docking program that is known to produce faster but less reliable docking estimations. However, no one would use this program in a practical scenario; hence, it should be properly discussed by the authors what is the computational burden to train the model for its usage in the real world, not just for experimenting.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "JM9OSk-YmB1", "reviewer": "Reviewer_bYsg", "summary": "This paper is concerned about optimizing molecules by assembling fragments using reinforcement learning. The main idea to generate \"drug-like\" molecules is to generate a molecule by combining chemically realistic and pharmacochemically acceptable fragments, while restricting the attachment sites of the fragments. This generation process coupled with the explorative RL algorithm is the proposed method.\n\nThe effectiveness of the proposed method is investigated by comparing it with two existing methods. The experimental results suggest that (i) the molecules generated by the proposed method are more likely to pass the pharmacochemical filters than the existing methods, (ii) the proposed method achieves higher or comparable hit ratio and top 5% scores than the existing methods, (iii) the explorative RL algorithm used in the proposed method is suitable to this task than the other RL methods.\n\nThe authors also provide a case study on drug design in both de novo scenario and scaffold-based one.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Summary\n- I like the authors work on realistic problems rather than synthetic ones such as logP optimization, formally considered in the literature.\n- Technically, the proposed method seems to be minor contributions to the literature. For example, the fragment-based generation with connectivity preservation has been introduced in the paper \"Molecular Hypergraph Grammar with Its Application to Molecular Optimization\" by Kajino (ICML-19), and a similar generation process is coupled with RL in the paper \"Reinforced Molecular Optimization with Neighborhood-Controlled Grammars\" by Xu et al. (NeurIPS-20). \n- There are two major concerns; see below.\n\n# On the hypothesis\nAn important hypothesis of the present work is that attaching a chemically realistic and pharmacochemically acceptable fragment units leads to realistic and acceptable molecules. The filters used in the experiments seem to be based on the same hypothesis, but I am curious whether this hypothesis is widely accepted or not. If this hypothesis is just a necessary condition for a molecule to be acceptable, it should be noted as a limitation of this work.\n\n# Comparison to the existing methods\nIn order to validate the proposed method, it is necessary to empirically compare the proposed method with the existing methods, at least those mentioned below.\n- In l.93-95, the authors state that their connectivity-preserving fragmentation and augmentation procedure is beneficial, but this is not obvious and should be empirically confirmed.\n- In l.99-107, the authors state that the forward synthesis methods will not be able to handle docking score optimization, but this is also not obvious and should be empirically confirmed.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The fragment-based generation could be limited as compared to the atom-wise generation in that the generated molecules are less diverse; this is because the generated molecules are restricted to the fragment library. So, I would recommend the authors to investigate the diversity of generated molecules.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "HSeVl94Obra", "reviewer": "Reviewer_ADzq", "summary": "The paper presents a reinforcement learning approach in which molecule fragmetns are combined together in order to construct realistic and pharmacologically relevant molecules. The approach relies on considerable use of domain knowledge, since the library of fragments that are used to generate the molecules are themselves realistic and pharmacologically relevant and they do not exhibit undesired properties, e.g. toxicity or low reactivity. The overall goal of the method presented in the paper is to learn to generate molecules that will dock to given proteins. With respect to the RL problem the reward that is used is based on the docking score as well as on a pharmacochemical filter scores. This is a hard-exploration learning problem due to very sparse rewards, which according to my understanding are available once all the molecule has been generated. To address the hard-exploration problem the paper relies on a number of variants of SAC which make use of prioritise replay. The paper compares against a couple of baselines with favorable results. My impression is that the main advantage of the method with respect to the baselines is that the library of fragments over which it is operating is carefully selected towards producing molecules that are pharmacoligically relevant and acceptable, and less the algorithmic contributions, e.g prioritised replay. \n\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the paper is in  general easy to ready, I had issues on a number of occasions. For example the I found the discussion of the prioritised reaply methods that the paper used confusing and unclear. I believe I also missed a precise definition of the reward that it was used, in the appendix it is described as \"docking score reward + pharmacochemical filter reward\" but what are the filters that are used? \n\nIn the experiments the paper compares against two baselines, for which not many details are given except the fact that they are generative models, though in the description in the introduction it seems that they are rather reinforcement learning algorithms. I could not help but wonder whether a quantitative comparison with proper generative models, many of which the paper discusses, would not be possible? Many of these generative models have excellent performance for the tasks on which they have been used/benchmarked (and they are also readily available for conditional generation), though I am not sure any of them has been tried on the docking generation problem. If we were to take a conditional generation approach on the docking docking problem I guess the task would be something like given a particular target learn the distribution of the molecules that dock to it. I am not sure whether this is easy to do, since there would be issues even with the generation of proper training sets. In any case I am not sure that one can so easily dismiss these models, and with respect to that aspect I have the feeling that the set of baselines is not complete, though I admit benchmarking these methods on the docking problem is not obvious. \n\n\nlines 96-98: Is there evidence out there that learning to reconstruct might harm diversity? Why having a decoder that is able to reconstruct will have less chances to produce diverse molecules? and by the way the same argument can also be given for the method presented in this paper, i.e. isnt't the fact that the approach is based on fragments limit the diversity of the generated molecules, since these can only be combinations of existing fragments? which in fact seems to be supported by the experiments. \n\nline 145: I am not sure how should I interpret the \"predict a next state\"? should it be simply action prediction?\n\nActions: \n* lines 154-155: What is the action space? The first action selects the docking point in the so far generated molecule, the possibilities can be very different depending on the so far generated molecule, how this is handled? or in other words what is the structure of the output of $\\pi_{\\text{act1}}$? similar question for second and third action selection. \nRelate to this, in the appendix, line 86 states that the maximum number of actions in an episode is four? does that mean that there only four transitions, i.e. all episodes have a length of four? is there no possibility that after fours steps we might end up with an incomplete molecule?\n\nline 169: for my undestanding, all rewards are zero except the last one which is given by the docking score and the pharmacological filter rewards?\n\nlines 173-174: eq 5: Isn't the expectation missing the $\\pi(a|s)$ term ? Also the expectation is approximated by the sample based estimate but it is not equal to it. \nTo avoid ambigiuty would be useful to have the two sums, the one that comes from the expecation and the one over the actions for the entropy. \n\n\nHard exploration part: \n* lines 187-193: To address the hard-exploration the paper couples SAC with prioritized replay where the priority of a state is either the predictive error of a reward prediction network\n or the uncertainty of that reward as estimated by a neural net with dropout. Since the priority is based on reward uncertainty shouldn't this be computed based on state-action \npairs, instead of states? I am not sure I understand what is the reward of a state. By the way the auxiliary network that estimates reward is independent of the critic of SAC? In the appendix \nlines 53-61 (curiosity driven learning) there is a discussion of reward prediction, though only the loss is given, $\\hat{y_i}$ is the estimated reward of some given state? and the true reward\nof that state will be almost always 0? unless the molecule is complete, in which case it will be the docking score plus the pharma filters?\n\n* lines 194-199: I have to say that here I got confused, I am not sure I see how 1) and 2) relate do they provide two different ways to prioritise experience replay? one seems to focus on states only, \nwhile in 2) the focus seems to be transitions? By the way a transition is the quadruple (s,a,r,s')? and is the priority of a given transition based on the reward prediction or on the docking score estimation? both are mentioned in the paragraph, and I thought the docking score is only given for complete molecules but the paragraph makes reference to docking score estimation for a given state. \n\nExperiments:\n* baselines: a couple of sentences that describe the type of generative approach that REINVENT and MORLD follow would have been appreciated.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes they have. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "8-FQFuIjdfk", "reviewer": "Reviewer_5BYY", "summary": "The authors propose a novel approach based on reinforcement learning (RL) to accelerate the discovery of pharmacologically relevant inhibitors. One of the key limitations of existing methods is that many molecules produced by generative models based are chemical invalid. In this work, the authors circumvent this problem by allowing the agent to modify an existing ligand (state) with a set of actions that result in a new chemically valid molecule (next state). Through this introduction of domain knowledge, the authors achieve higher sample efficiency. However, the experiments show that this inadvertently reduces the diversity of generated molecules.\n\nOverall, the approach seems reasonable, and experimental results impressive. However, the manuscript lacks methodological novelty, and thus, might not be suited for this venue. Further, the presentation (language, precision, and figures) requires work. For this reason, I vote for rejection.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Abstract\n- L7: define pharmacochemically acceptable more precisely\n- L8: \"the docking score optimization is a difficult exploration problem that involves many local optima and less smooth surface\". This statement is not clear\n- L13: What is \"qualified chemical space\"?\n\n# Introduction\n- L30: is there is a citation missing?\n- Similar to the abstract, the introduction requires more precision.\n\n# Related Work\n- Good overview of existing methods but no mention of RL approaches that generate 3D structures, such as:\n    - Meldgaard, S. A.; Mortensen, H. L.; Jørgensen, M. S.; Hammer, B. Structure Prediction of Surface Reconstructions by Deep Reinforcement Learning. J. Phys.: Condens. Matter 2020, 32 (40), 404005. https://doi.org/10.1088/1361-648X/ab94f2.\n    - Simm, G. N. C.; Pinsler, R.; Hernández-Lobato, J. M. Reinforcement Learning for Molecular Design Guided by Quantum Mechanics. In Proceedings of the 37th international conference on machine learning; III, H. D., Singh, A., Eds.; Proceedings of machine learning research; PMLR, 2020; Vol. 119, pp 8959–8969.\n    - Christiansen, M.-P. V.; Mortensen, H. L.; Meldgaard, S. A.; Hammer, B. Gaussian Representation for Image Recognition and Reinforcement Learning of Atomistic Structure. J. Chem. Phys. 2020, 153 (4), 044107. https://doi.org/10.1063/5.0015571.\n    - Simm, G. N. C.; Pinsler, R.; Csányi, G.; Hernández-Lobato, J. M. Symmetry-Aware Actor-Critic for 3D Molecular Design; ICLR 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. The authors need to revise the manuscript to improve readability and precision. Currently, the manuscript is not ready for publication.\n1. The proposed method seems reasonable, but the novelty on the machine learning side is very limited. The contributions are worthwhile but most probably not relevant to the target audience of this venue.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission has its merits but, even after carefully checking the author rebuttal, the reviewers unanimously agree that the submission is not strong enough for publication at NeurIPS. The main reasons for that are the lack of novelty, clarity, and supporting evidence.

Decision: Accept (Poster)