Reinforced Genetic Algorithm for Structure-based Drug Design
Authors: Tianfan Fu
Wenhao Gao
Connor W. Coley
Jimeng Sun
Venue: NeurIPS 2022 Accept (2022)
Keywords: molecule generation
molecule optimization
drug design

Abstract: Structure-based drug design (SBDD) aims to discover drug candidates by finding molecules (ligands) that bind tightly to a disease-related protein (targets), which is the primary approach to computer-aided drug discovery. Recently, applying deep generative models for three-dimensional (3D) molecular design conditioned on protein pockets to solve SBDD has attracted much attention, but their formulation as probabilistic modeling often leads to unsatisfactory optimization performance. On the other hand, traditional combinatorial optimization methods such as genetic algorithms (GA) have demonstrated state-of-the-art performance in various molecular optimization tasks. However, they do not utilize protein target structure to inform design steps but rely on a random-walk-like exploration, which leads to unstable performance and no knowledge transfer between different tasks despite the similar binding physics. To achieve a more stable and efficient SBDD, we propose Reinforced Genetic Algorithm (RGA) that uses neural models to prioritize the profitable design steps and suppress random-walk behavior. The neural models take the 3D structure of the targets and ligands as inputs and are pre-trained using native complex structures to utilize the knowledge of the shared binding physics from different targets and then fine-tuned during optimization. We conduct thorough empirical studies on optimizing binding affinity to various disease targets and show that RGA outperforms the baselines in terms of docking scores and is more robust to random initializations. The ablation study also indicates that the training on different targets helps improve the performance by leveraging the shared underlying physics of the binding processes. 
The code is available at https://github.com/futianfan/reinforced-genetic-algorithm.

Reinforced Genetic Algorithm for Structure-based Drug Design
Tianfan Fu'", Wenhao Gao²*, Connor W. Coley Jimeng Sun
Department of Computational Science and Engineering, Georgia Institute of Technology,
Department of Chemical Engineering, Massachusetts Institute of Technology.
& Department of Flectrical Engineering and Computer Science, Massachusetts Institute of Technology. Department of Computer Science, University of Illinois at Urbana-Champainn
5 Carle Illinois College of Medicine, University of Illinois at Urbana-Champaign, "Equal Contributions
tfu420gatech. edu, {whgao, ccoley) @mit edut j jimengeilling www.eiiiiniiiiiiiimmiiiiiimiieel edu
Abstract
Structure-based drug design (SBDD) aims to discover drug candidates by finding molecules (ligands) that bind tightly to a disease-related protein (targets), which is the primary approach to computer-aided drug discovery. Recently, applying deep generative models for thrce-dimensional (3D) molecular design conditioned on protein pockets to solve SBDD has attracted much attention, but their formulation probabilistic modeling often leads to unsatisfactory optimization performance. On the other hand, traditional combinatorial optimization methods such as genetic algorithms (GA) have demonstrated state-of-ther ofethe-art performance in various molec- ular optimization tasks. However, they do not utilize protein target structure to inform design steps but rely unstable performance and no knowledge transfer between different tasks despite the similar binding physics. To achieve more stable and efficient SBDD, we propose Reinforced Genetic Algorithm (RGA) that uses neural models to prioritize the profitable design steps and suppress random-wall behavior. The neural mod- els take the 3D structure of the targets and ligands as inputs and are pre-trained using native complex structures utilize the knowledge of the shared binding physics from different targets and then fine-tuned during optimization. We con- duet thorough empirical studies on optimizing binding affinity to various disease targets and show that RGA outperforms the baselines in terms of docking scores and is more robust to random initializations. The ablation study also indicates that the training on different targets helps improve the performance by leveraging the shared underlying physics the binding processes. The code is available https: /::::://ttttt com/ fut funianffii an/ www.rreinorrrrreee senetic-algors thm.
Handom-walk-lide exploration, which leads
Introduction
Rapid drug discovery that requires less time and cost is of significant interest in pharmaceutical science. Structure-based drug design (SBDD) [ that leverages the three-dimensional (3D) structures of the disease-related proteins to design drug candidates is one primary approach to accelerate the drug discovery processes with physical simulation and data-driven modeling. According to the lock and key model (2), the molecules that bind tighter disease target are more likely to expose bioactivity against the disease, which has been verified experimentally B. As AlphaFold2 has provided accurate predictions to most human proteins 4 55, SBDD has tremendous opportunity to discover new drugs for new targets that we cannot model before 6
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

SBDD could be formulated as an optimization problem where the objective function is the binding affinity estimated by simulations such docking 2] The most widely used design method is virtual screening, which exhaustively investigates every molecule in library and ranks them. Lyu et al. successfully discovered new chemotypes for AmpC 3-lactamase and the D4 dopamine receptor by studying hundreds of millions of molecules with docking simulation 77. However, the number of the drug-like molecules is large as estimated to be 10 [1]. and is computationally prohibitive to screen all of the possible molecules. Though machine learning approaches have been developed to is still challenging to screen large enough chemical space within the
accelerate screening g [9]
foresecable future.
Instead of naively screening a library, designing drug candidates with generative models has been highlighted as a promising strategy, exemplified by [IO] [.] This class of methods models the problem as the generation of ligands conditioned on the protein pockets. However, as generative models are trained to learn the distribution of known active compounds, they tend to produce molecules similar to training data [12. which discourages finding novel molecules and leads to unsatisfactory optimization performance.
more straightforward solution is combinatorial optimization algorithm that searches the implicitly defined discrete chemical space. As shown in multiple standard molecule optimization bench- combinatorial optimization methods, especially genetic algorithms (GA) [66 17]. often perform better than deep generative models. The key to superior performance is GA's action definition. Specifically, in each generation (iteration). GA maintains a population of possible candi- dates (a.k.a. parents) and conducts the crossover between two candidates and mutation from single candidate to generate new offspring. These two types of actions, crossover and mutation, enable global and local traversal over the chemical space, allowing thorough exploration and superior optimization performance.
marks [3
However, most GAs select mutation and crossover operations randomly [16], leading to significant variance between independent runs. Especially in SBDD, when the oracle functions are expensive molecular simulations, it is resource-consumiigg to ensure stability by running multiple times. Further, most current combinatorial methods are designed for purposs molecular optimization and simply use docking simulation as an oracle. It is challenging to leverage the structure of proteins in these methods, and we need to start from scratch whenever we change a protein target, even though the physics of leads to unnecessary exploration steps and, thus, demands for many more oracle calls, which require expensive and unnecessary simulations (18).
interaction is shared. Ignoring the shared information across tasks
To overcome these issues in the GA method, we propose Reinforced Genetic Algorithm (RGA), which attempts to reformulate an evolutionary process as a Markov decision process and uses neural networks to make informed decisions and suppress the random-walk behavior. Specifically, we utilize an E(3)-equivariant neural network [19] to choose parents and mutation types based on the 3D structure of the ligands and proteins. The networks are pre-trained with various native complex structures to utilize the knowledge of the shared binding physics between different targets and then fine-tuned with a reinforcement learning algorithm during optimizations. We test RGA's performance with various disease-related targets, including the main protease of SARS-CoV-2.
The main contributions of this paper can be summarized as follows:
We propose an evolutionary Markov decision process (EMDP) that reformulates an evolutionary process as a Markov decision process, where the state is population of molecules instead of single molecule (Section)
We show the first successful attempt to use neural model to guide the crossover and mutation operations in a genetic algorithm to suppress aadom-walk behavior and explore the chemical space intelligently (Section 3.3).
We present a structure-based de novo drug design algorithm that outperforms baseline methods consistently through thorough empirical studies on optimizing binding affinity by leveraging the underlying binding physics (Section 4

---

2
Related Works
We will discuss the related works on methods of drug design and discuss the advantage of the proposed method over the existing works.
General Molecular Design. Molecular generation methods offer a promising direction for the automated design of molecules with desired pharmaceutical properties such synthesis accessibility and drug-likelines Based on how to generate or search molecules, these approaches can be categorized into two types, (1) deep generative models (DGMs) imitate the molecular data distribution, including variational autoencoder (VAE) 220 generative adversarial network (GAN) [22] 23), normalizing flow model 24 251. energy based model [26]: and (2) combinatorial optimization
methods directly search over the discrete chemical space, including genetic algorithm (GA) [T6] 27
28 . reinforcement learning approaches (RL)
29
30
31
32 33]. Bayesian optimization (BO)
Markov chain Monte Carlo (MCMC) [35 36
37
and gradient ascent [38 39
General molecular design algorithms often use general black-box oracle functions, and some only tested with trivial or self-designed oracles. For example, using penalized octanol-water partition coefficient (LogP) the oracle function, it grows monotonically with the number of carbons, and thus there exists a trivial policy to optimize LogP. These oracles do not reflect the challenges real drug discovery, and those algorithms have limited value for pharmaceutical discovery. Recent works are optimizing docking scores to simulate a more realistic discovery scenario [40] 41 [8] 42] same our work. However, they still ignoring the information in the given protein structures that could potentially accelerate the design process. However, the extension to leveraging the structural knowledge is nontrivial.
Structure- based Drug Design. Structure- based drug design (SBDD) could utilize the structural information guide the design of molecules, which are potentially more efficient in drug discovery tasks but poses additional challenges of how to leverage the structures. Since early 1990s, various SBDD algorithms have been proposed, mostly based on combinatorial optimization algorithms such as tree search [43
[44] 45 and evolutionary algorithms [46] 47). Those methods typically optimize the ligands in the pockets according a physical model characterizing the binding affinity. For example, RASSE 4 used a fieee-fieddllike scoring function 488 to evaluate the partial solutions within a tree search. However, obtaining a fast and accurate model to quantify binding free energy itself is still an unsolved challenge.
Recently, generative modeling of 3D molecules conditioned on protein targets is attracting more attention [10 11]. Similar to DGMs in general molecular design, those methods learn the atom's 's compositional and spatial distribution of native structure of protein-ligand complexes with neural models and design new ligands by complete the complex structure given targets. Deep generative models are end-to-end and data-driven thus surpass the necessity of understanding the physies of interaction. However, as the training objective is to learn the distribution of known active compounds, the models tend produce molecules close the training set [12), which is undesired in terms of patentability and leads to unsatisfactory optimization performance.
3 Method
In this paper, we focus on structure-based drug design. The goal is design drug molecules (a.k.a. ligands) that could bind tightly with the disease-related proteins (a.k.a. targets). Given the 3D structures of the target proteins, including binding site information, docking is popular computational method for assessing the binding affinity, which can be roughly retrieved as the free energy changes during the binding processes. We present a variant of genetic algorithm that is guided by reinforcement learning and docking oracle. Next, we will first describe the general evolutionary process used genetic algorithms (Section 3.T): Then we will present how to model this evolutionary process as a Markov decision process (MDP) where RI. framework can be constructed (Section (.2); After that, we describe the detailed implementation of this MDP framework using multiple policy networks (Section 3.3
3

---

3.1 Evolutionary Process
In this section, we introduce the primary setting of the evolutionary processes. With both optimization performance and synthetic accessibility taken into account [49 14], we follow the action settings in Autogrow 4.0 [7]. It demonstrated superior performance over other GA variants in the empirical validation of structure-based drug design [17], and its mutation actions originated from chemical reactions so that the designed molecules are more likely to be synthesizable. Specifically, an evolutionary processes starts by randomly sampling a population of drug candidates from a library. In each generation (iteration), carries out (i) crossover between parents selected from the last generation, and (ii) mutation on single child to obtain the offspring pool. An illustration of both crossover and mutation operations available in Appendix. Note that we only adopted the action settings from Autogrow 4.0, without using other tricks such as elitism.
Crossover, also called recombination, combines the structure of two parents generate new children. Following Autogrow 4.0 [17), we select two parents from the last generation and search for the largest common substructure shared between them. Then we generate two children by randomly switching their decorating moieties, i.c., the side chains attached to the common substructure.
Mutation operates single parent molccule and modifies its structure slightly. Following Autogrow 4.0 [7]. we adopt transformations based on chemical reactions. Unlike naively defined atom atom- editing actions, mutation steps based on chemical reactions could ensure all modification is reasonable in reality, leading to a larger probability of designing synthesizable molecules. We included two types of chemical reactions: uni-molecular reactions, which only require one reactant, and bi-molecular reactions, which require two reactants. While uni-molecular reactions could be directly applied to the parent, we sample a purchasable compound to react with the parent when conducting bi-molecular reaction. In both cases, the parent serves as one reactant, and we use the main product as the child
molecule. We use the chemical reactions from 17.. which was originally from 47 50].
Evolution. At the t-th generation (iteration), given population of molecules denoted as S(9,, we generate an offspring pool denoted as @(4) by applying crossover and mutation operations. Then we filter out the ones with undesirable physical and chemical properties (e.g., poor solubility, high toxicity) in the offspring pool and select the most promising K to form the next generation pool (S(++1)).
3.2 Evolutionary Markov Decision Process
Next we propose the evolutionary Markov decision process (EMDP) that formulates an evolutionary process of genetic algorithm as a Markov decision process (MDP). The primary purpose is to utilize reinforcement learning algorithms to train networks inform the decision steps to replace random selections. Taking generation
state, Markov property that requires ((S(t+))g()).... ,8())
P(S(++)) ()((t)) is naturally satisfied by the evolutionary process described above, where ((t) denotes the state at the t-th generation, which is the population of ligands. We use X to denote a ligand. We elaborate essential components for Markov decision process as follows, and the EMDP pipeline is
illustrated in Figure 0
State Space. We define the population at the t-step generation, 8(0, in the evolutionary process as the state at the t-step in an EMDP. A state includes population of candidate molecules (i.e., ligand, denoted X) and their 3D poses docked to the target, fully observable to the RL agent. At the beginning of the EMDP, we randomly select population of candidate molecules and use docking simulation to yield their 3D poses as the initial state.
Action Space. The actions in an EMDP are to conduct the two evolutionary steps: crossover and mutation, population. For each evolutionary step, we need two actions to conduct it. Concretely, crossover (X cressverr X xpaeent crossveer Xchiisiver> X child (hissiver) can be divided to two steps: (1) select the first candidate ligand X from the current state (population S(9); (2) conditioned on the 2 from the remaining candidate ligand set ((t) [xpeett and apply crossover (Section 3.1 to them.
first selected candidate x Xeresstees
I select the second candidate ligand X

---

Genetic Algorithm
Pocket (binding site)
Target Protein
Docking Simulation
Random Sample
Crossover
Evaluate Fitness
Parents
Keep Top-K
Random Sample
Random Mutation
Parent
Population
Offspring
Population
Reinforced Genetic Algorithm
Target Protein
Pocket [binding site)
Conditioned on
Docking Simulation
Second Parent
Evaluate Fitness
Parent
Keep
Top-K
Mutation
Parent
Mutation
Policy Net
Policy Net
Type
State (t)
Offspring
State (t+1)
Figure 1: We illustrate one generation (iteration) of GA (top) and RGA pipeline (bottom). Specifically, we train policy networks that take the target and ligand as input to make informed choices on parents and mutation types in RGA.
mutated by S, Xchild X can be divided to two steps: (I) select the candidate ligand X yparent to be mutated from the current state (population S(t)); (2) conditioned on the selected candidate ligand X X select the reaction & from the reaction set R and apply it to X x-prret mulated"
Mutation (X (utaation
As applying the crossover and mutation steps deterministic, the actions in EMDP focus selecting parents and mutation types. Upon finish the action, we could obtain offspring pool, ((1),
State Transition Dynamies. The state transition in an EMDP identical the evolution in an evolutionary process. Once we finish the actions and obtain the offspring pool, Q
{XXchild cchild .. we apply molecular quality filters filter out the ones unlikely be drug and then select the most promising to form the parent set for the next generation (S(++))).
Reward. We define the reward as the binding affinity change (docking score). The actions leading to stronger binding score would be prioritized. As there "episode" concept in EMDP, we treat every step equally.
3.3 Target-Ligann Policy Network
To utilize molecular structures' translational and rotational invariance, adopt equivariance neural networks (ENNs) [9 as the target-ligand policy neural networks to select the actions in both mutation and crossover steps. Each ligand has 3D pose that binds to the target protein, and the complex serves as the input of ENN.
Specifically, we want to model 3D graph V, which can be ligand, target, target-ligand complex. The input feature can be described as vocabulary set V {H, C, ... }) and Z represents 3D coordinates of the atoms. Suppose D E IV Xd is the embedding matrix of all the categories of atoms in vocabulary set V, is randomly initialized and learnable, is the hidden dimension ENN. Each kind of atom corresponds to a row in D. We suppose there are N atoms, and each atom corresponds to node in the 3D graph. Node embeddings at the 1-th layer are denoted (
(A,, 2), where A represents atoms' categories (the
[h.
( Ti where 0, 1...... I, number D DTa, a, d embeds the i-th node, where
of layers in ENN. The initial node embedding h

---

aj is one-hot vector that encode the category of the i-th atom. Coordinate embeddings at the 1-th layer are denoted ( {z; (1) " fi=1. The initial coordinate embeddings ( J. are the real
3D coordinates of all the nodes. The following equation defines the feedforward rules of ENN, for
j 1,... N, f j,
1
0, 1, L 1, we have
N Σ
ig
(1-1) MLP. h
(1)
+ h
1 1 - 2) 2))
E ER",
(1+))
w
ij
(++1)
ER",
E
N
z;
(4+1)
zi
(1)
Σ
(1)
/ MLP3 (w!)) E CR.
h h(+1))
MLP/ (n.") + @v v((())))
E ER".
j=1jzi
by
(L)
E Rd
hy EEN()))
where @ denotes the concatenation of vectors; (-) : R2 + R (MMMPPs>) : R
R; (+) R22d Rd are all two-layer multiple layer perceptrons (MLPs) with Swish acti- vation in the hidden layer [51]. At the I-th layer, Wij represents the message vector for the edge from node i to node ji ()) represents the message vector for node i, z; is the position embeddiin for node (1) ;; h3 is the node embedding for node i. H [h) ..... N
(1)
(L) are the node embeddings of the L-th (last) layer. We aggregate them using sum function as readout function to obtain a representation
of the 3D graph, denoted hy. The whole process is written as hy- = ENN()).
Crossover Policy Network. We design two policy networks for two corresponding actions in a crossover, as mentioned in Section 3.2 (1) the first action in crossover operation is select the first parent ligand X X I from the population 8(t), Similar to the first action in mutation opera- tion, we obtain valid probability distribution over all the available ligands based on target-ligand complex as input feature and ENN as the neural network architecture, the selection probability
exp (MLP(h xe
(1) of the ligand X Crosssver E 8(() is Perossover |
(S()))
where T
Ex(es(s) exp (MLP(hT2x())
and X denotes target and ligand (including 3D pose), respectively, T denotes target-ligand complex. (2) The second action is to select the second parent ligand conditioned on the first par- ent ligand selected in the first action. Specifically, for ligand in the remaining population set, we concatenate the ENN's embedding of the target, first parent ligand Coossver I and the second par- 2 ent ligand X muration and feed it into an MLP to estimate scalar
unnormalized probability. The unnormalized probabilities for all the ligands in the remaining population set are normalized via softmax function, i.e., Perossover ( Xx xrssove
1 s(t)) Softmax MLP(h7 + h x +
X Xxneen ),..,, J X ES( ((X y. Given two parents ligands, crossover finds the largest sub- structure that the two parent compounds share and generates a child by combining their decorating moieties. Thus, the generation of child ligands are deterministic, and the probability of the generated ligands X-hhid cressover is
Perosswerr
X
ehild crossover:
X-hhld
crossover (8(1)) = Peressover ( parent crossover: X parent 2
S()))
(2)
P Penssover
(1)
X
parent 1
(S())
-P -l2l XX
parent 2
X crussowrr S(()).
1
crossaver
crossover
Mutation Policy Network. We design two policy networks for two corresponding actions in mutation, mentioned in Section 3.2 (1) the first action in mutation operation is to select candidate ligand to be mutated from population Sit). It models the 3D target-ligand complex to learn if there is improvement space in the current complex. Formally, we obtain a valid prob- ability distribution over all the available ligands based on target-ligand complex as input fea- ture and ENN as neural architecture, the selection probability of the ligand X pareet E (()) is
exp (MLP TEx
P Pmutationn musation (T))
(X- X-perent
Exresto exp (MLP(bTexx))) where T&X denotes target-ligand complex,
hT&X ENN(T&X) represents the ENN's embedding of target-ligand complex. (2) The sec- ond action is to select the SMARTS reaction from the reaction set conditioned on the selected ligand in the first action. Specifically, for each reaction, we generate the new ligand X X then mutation

---

obtain the embedding of target, first ligand X angtaatoon and the new ligand X X concatenate these three embeddings and feed into an MLP to estimate scalar as unnormalized probability. The unnormalized probabilities for all the reactions are normalized via softmax func-
mutation through ENN,
tion, i.e., Pimutaionn
[ X
mutation:
parent
S(t)) = Softmax MLP + h hymru
h yaai
Dr .ER where
X mutation
mutated by E
X
mulation
R is the reaction set. The probability of the generated ligand X
is
((())
(1)
(S(0) P'mmtatioon (E)
mutationn (()).
parent
Pmuuttton mutation
Pmmttation mutation
Policy Gradient. We leverage policy gradient to train the larget-ligand policy neural network. Specifically, we consider maximizing the expected reward as objective via REINFORCE (2).
max Ex Ex~p(X(s(e) [x)arr((X)]:
(4)
where p(X) is defined Equation
2 and for crossover and mutation, respectively. The whole pipeline is illustrated in Figure [ To provide warm start and leverage the structural information, we pretrain the ENNs on 3D target-ligann binding affinity prediction task, where the inputs are the target-ligand complexes, and the outputs are their binding affinity.
Experiment
In this section, we briefly describe the experimental setup and results. The Appendix includes more details, including software configuration, implementation details, dataset description & processing, hyperparameter tuning, ablation study, and additional experimental results. The code is available at https://github:
-ereeererrddgeiiicccoo algorithm
4.1 Experimental Setup
Docking Simulation. We adopt AutoDock Vina [22 to evaluate the binding affinity. The docking score estimated by AutoDock Vina called Vina score and roughly characterizes the free energy changes of binding processes in kcal/mol. Thus lower Vina score means stronger binding affinity between the ligand and target. We picked various discase-related proteins, including G-protein coupling receptors (GPCRs) and kinases from DUD-E (53] and the SARS-CoV-2 main protease [54] as targets. Please see the Appendix for more information.
Baselines. The baseline methods cover traditional brute-force search methods (Screening), deep generative models (JTVAE and Gen3D), genetic algorithm (GA+D, graph-GA, Autogrow 4.0), reinforcement learning methods (MoIDQN, RationaleRL, REINVENT, GEGL), and MCMC method (MARS). Gen3D and Autogrow 4.0 are structure-based drug design methods, while others are general-purpose molecular design methods. Although methods explicitly utilizing target structures relatively few, we add general-purpor purpose molecular design methods optimizing the same docking oracle scores ours, which is common case, baselines [66] [4]. Concretely, Screening mimics high throughput screening via sampling from ZINC database randomly: JTVAE (Junction Tree Variational Auto-Eno Ento-Encoder) [21] uses Bayesian optimization on the latent space indirectly optimize molecules;
Gen3D 10 is an regressgee generative model that grows 3D structures atom-wise inside the
binding pocket: GA+D 277 represents molecule as SELFIES string [55) and uses genetic algorithm enhanced by discriminator neural network; Graph-GA 16 conduct genetic algorithm on molecular graph representation: Autogrow 4.0 [17 is the -fate-of-the-art genetic algorithm in cruucture-based
drug design; MolDQN (Molecule Deep Q-Network) 31 leverages deep Q-value learning to grow molecules ftom-wwsely; RationaleRL 32 uses rationale (c.g., functional groups or subgraphs) as the building block and policy gradient method to guide the training of graph neural network- based generator; REINVENT (29 represent molecules as SMILES string and uses policy gradient based reinforcement learning methods to guide the training of the RNN generator; GEGL (genetic expert- guided learning) (33] uses LSTM guided by reinforcement learning to imitate the GA exploration; MARS (Markov Molecule Sampling) [36] leverages Markov chain Monte Carlo sampling (MCMC) with adaptive proposal and annealing scheme to search chemical space. To conduct a fair comparison, we limit the number of oracle calls to 1,000 times for each method. All the baselines can be run with one-line code using the software (https: http//ggtttttttt/ com/ wenhao seanhaa-ggoomol... molecular optimization benchmark [15].
practical
7

---

Table 1: The summarized performance of different methods. The mean and standard deviation across targets are reported. Arrows (t. +) indicate the direction of better performance. For each metric, the best method is underlined and the top-3 methods are bolded. RGA-pretrain and RGA-KTT are two variants of RGA that without pretraining and without training on different target proteins, respectively.
Method
TOP- TOP-100L
TOP-101
TOP-14
Novi
Divt
QED1
SAL
screening
+.343
10,4333
+1,,000
0.0
0.858:0005
0.678
2.68920077
MARS
-7.758am012
-8. 8.875
292257.079
100.0gams
0.877
0.709
2.450
MolDQN
-6.287=0.06
-7.043--0487
-7.501=00:00
100.0:a0s
0.877:a.m0
0.170=am2
1.8
GEGL
-9,064=n920
-9,91
-10.45
100.0
0,853+0000
0.643+am14
2.99
REINVENT
-10.181
-1,2234-0.322
-12.010+
100.0
0.857=0001
0.445=a058
2.596
RationaleRL
-22233
-10.8344.856
-11.642+1.102
100.0
0.717
5
2.919
JTVAE
9.291 202
-10,242:08
-10.963-11113
98. 08.0200075
0.867 2:001
0.593
3.222.0116
Gen3D
1.686.11.20
-9.285 1:0.544
-9.83220.324
100.0
0.870
0.701
3,450
GA+D
77.487=0797
-8.305
-8.760=0.996
99.2200115
0.834gomm
0.405
5.02420.64
Graph-GA
-1.5848-..50
-11.702:0.500
-12.302:1.00
100.0=a.s
0.811
0.456=n057
3.503+0.367
Autogrow 4.0
-11.371
-12.2132a623
-12.474:40899
100.0:aos
0.852
0.748 n
2.497:0049
RGA (ours)
-1..867=0.17
-12.564::0257
-12.869
100.0=a.s
0.857=a.20
0.742
2.473
RGA pretrain
-1.1443=0.11
-12,42440
-12.435200s4
100.0:a0s
0.854
0.750=0014
2.49420043
RGA KT
-11.434-0.10>
-12.43720394
-12.50220005
100.0amm
0.853imm
0.738
2.501
Dataset: we randomly select molecules from ZINC [56] database (around 250 thousands drug-like
molecules) as 0-th generation of the genetic algorithms (RGA, Autogrow 4.0, GA+D). ZINC also serves the training data for pretraining the model in JTVAE, REINVENT, RationaleRL, etc. We adopt CrossDocked2022 [57] dataset that contains around 22 million ligand-proted complexes as the training data for pretraining the policy neural networks, mentioned in Section B.3 More descriptions are available in Appendix.
Metrics. The selection of evaluation metrics follows recent works in molecule optimization 21 27 32 (3) and structure-based drug design [17 [0 [4]. For each method, we select top-100 molecules with the best docking scores for evaluation and consider the following metrics: TOP-1/10//00 (average docking score of top- 1/10//0001000 molecules): docking score directly measures the binding affinity between the ligand and target and is the most informative metric in druuttrr-bassd drug design; Novelty (Nov) (% of the generated molecules that are not in training set); Diversity (Div) (average pairwise Tanimoto distance between the Morgan fingerprints): We also evaluate some simple pharmaceutical properties, including quantitative drug-likeness (QED) and synthetic accessibility (SA). QED score indicates drug-likeliness ranging from 0 to 1 (higher the better). SA score ranges from 1 to 10 (lower the better). All the evaluation functions are available at Therapeutics data commons (TDC, http::ttttc/ooomoooooto ai/fct_oveeee//////aaaaa
58
4.2 Results
Stronger Optimization Performance. We summarized the main results of the structure-based drug design in Table []] We evaluate all the methods on all targets and report each metric's mean and standard deviations across all targets. Our result shows RGA achieves the best performance in TOP-100/10/1 scores among all methods we compared. Compared to Autogrow 4.0, RGA'S better performance in docking score demonstrates that the policy networks contribute positively to the chemical space navigation and eventually help discover more potent binding molecules. On the other hand, including longer-rang navigation steps enabled by crossover leads to superior performance than other RL methods (REINVENT, MolDQN, GEGL and RationaleRL) that only focus on local modifications. In addition, we also observed competitive structure quality measured by QED (> 0.7) and ___Score( (< 2.5) Autogrow 4.0 and RGA without involving them as optimization objectives, thanks to the mutation steps originating from chemical reactions. We visualize two designed ligands with optimal affinity for closer inspection in Figure ((a) and 2(b) and find both ligands bind tightly with the targets.
Suppressed Random- Walk Behavior. Especially in SBDD, when the oracle functions are expen- sive molecular simulations, robustness to random seeds is essential for improving the worst-cass performance of algorithms. One of the major issues in traditional GAs is that they have a signifi- cant variance between multiple independent runs as they randomly select parents for crossover and mutation types. To examine this behavior, we run five independent runs for RGA, Autogrow 4.0 and graph-GA (three best baselines, all are GA methods) on all targets and plot the standard deviations between runs in
and 2(d) With policy networks guiding the action steps, we observed that the random-wal walk behavior in Autogrow 4.0 was suppressed in RGA, indicated by the smaller variance.
8

---

(a) Example of 7111, -10.8 kcal/mol (b) Example of 3eml, 13.22 keal/mol
graph-GA
oracle
Autogrow
-10.0
500
RGA
TOP-100 Vina score
1k
-11.0
top
-11.5-
-120
300 500
1000
graph-GA
Autogrow
RGA
# of oracle call
(c) TOP-100 vs oracle call
(d) Score bar for different runs
Figure 2: (a) and (b): Example of ligand poses (generated by RGA) and binding sites target structures. Example 7111: the PDB ID of target 7111, which is SARS-COVE NARS-COV-22201--NCO main protease, the Vina score is -0.88 kcal/mol. Example of 3eml: the PDB ID is 3eml, which is human A2A Adenosine receptor, the Vina score is -13.2 kcal/mol. (c) and (d): studies of suppressed random-wasd walk behavior. (c) reports TOP- 100 docking score as a function of oracle calls. The results the means and standard deviations of independent runs. (d) shows the bars of TOP-100 docking score for various independent runs.
Especially in the later learning phase (after 500 oracle calls), the policy networks are fine-tuned and guide the search more intelligently. This advantage leads to improved worst-case performance and higher probability of successfully identifying bioactive drug candidates with constrained resources.
Knowledge Transfer Between Protein Targets. To verify if RGA benefited from learning the shared physics of ligand-target interaction, we conducted an ablation study whose results are the last two rows of Table ! Specifically, we compare RGA with two variants: (1) RGA- RGA-pretrain that does not pretrain the policy network with all native complex structures in the CrossDocked2020; (2) RGA-KT (knowledge transfer) that fine-tune the networks with data of individual target independently. We find that both strategies positively contribute to RGA on TOP-100/100 docking score. These results demonstrate the policy networks successfully learn the shared physies of agaandtarrge interactions and leverage the knowledge to improve their performance.
5 Conclusion
In this paper, we propose Reinforced Genetic Algorithm (RGA) to tackle the structure-based drug design problem. RGA reformulate the evolutionary process in genetic algorithms as a Markov decision process called evolutionary Markov decision process (EMDP) that the searching processes could benefit from trained neural models. Specifically, we train policy networks to choose the parents to crossover and mutate instead of randomly sampling them. Further, we also leverage the common physics of the ligand-targel interaction and adopt a knowledge-transte transsfr strategy that uses data from other targets to train the networks. Through empirical study, show that RGA has strong and robust optimization performance, consistently outperforming baseline methods in terms of docking score.
Though we adopted mutations originating from chemical reactions and the structural quality metrics seem good, we need emphasize that the designed molecules from RGA not guarantee synthesiz- ability 449 as the crossover operations may break inheriting synthesizability. Directly working

---

synthetic pathways could solve the problem [2, but the extension is not trivial. As for future direction, we expect theoretically analyze the EMDP formulation and the performance of RGA.
Acknowledgments and Disclosure of Funding
T.F. and J.S. were supported by NSF award SCH-2205289, SCH-2014438, IIS-1838042, NIH award R01 1R01NS107291-01. W.G. and C.C. were supported by the Office of Naval Research under grant number N00014-21-1-2195 and the Machine Learning for Pharmaceutical Discovery and Synthesis consortium. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Office of Naval Research.
References
[1] Regine Bohacek, Colin McMartin, and Wayne C Guida. The and practice of structure- hased drug design: a molecular modeling perspective. Medicinal research reviews, 16(1):3-50, 1996.
[2] Ashutosh Tripathi and Vytas A Bankaitis. Molecular docking: From lock and key to combination lock. Journal of molecular medicine and clinical applications, 2(1), 2017.
[3] Assaf Alon, Jiankun Lyu, Joao M Braz, Tia A Tummino, Veronica Craik, Matthew J O'Meara, Chase M Webb, Dmytro S Radchenko, Yurii S Moroz, Xi-Ping Huang, et al. Structures of the a2 receptor enable docking for bioactive ligand discovery. Nature, 600(7890);759-766 2021.
[4] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873):533-88) 2021.
[5] Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, et al. Alphafold protein structure database: Massively expanding the structural coverage protein-sequence space with high-accuracy models. Nucleic acids research, 50(DI):D439-DD D444, 2022.
[6] Feng Ren, Xiao Ding, Min Zheng. Mikhail Korzinkin, Xin Cai, Wei Zhu, Alexey Mantsyzov, Alex Aliper, Vladimir Aladinskiy, Zhongying Cao, et al. Alphafold accelerates artificial intelligence powered drug discovery: Efficient discovery of a novel Cyclin-depended Kinase 20 (CDK20) small molecule inhibitor. arXiv preprini arXiv:2201 009647, 2022.
[7] Jiankun Lyu, Sheng Wang, Trent E Balius, Isha Singh, Anat Levit, Yurii S Moroz, Matthew J O' O'Meara, Tao Che, Enkhjargal Algaa, Kateryna Tolmachova, et al. Ultra-large library docking for discovering new chemotypes. Nature, 566(7773)
2019.
[8] David E Graff, Eugene I Shakhnovich, and Connor W Coley. Accelerating high-throughpus virtual screening through molecular pool-based active learning. Chemical science, 12(22):7866- 7881, 2021.
[9] Francesco Gentile, Jean Charle Yaacoub, James Gleave, Michael Fernandez, Anh-Tien Ton, Fuqiang Ban, Abraham Stern, and Artem Cherkasov. Artificial mnelligence-enablee virtual screening of ultra-large chemical libraries with deep docking. Nature Protocols, pages 1-26, 2022.
[10] Shitong Luo, Jiaqi Guan, Jianzhu Ma, and Jian Peng. A 3D generative model for maneturr-bbaedd drug design. In Thirty-Fifih Conference on Neural Information Processing Systems, 2021.
[11] Yibo Li, Jianfeng Pei, and Luhua Lai. Structure-based de novo drug design using 3D deep generative models. Chemical science, 12(41):13644-13655 2021.
[12] W Patrick Walters and Mark Murcko. Assessing the impact of generative Al on medicinal chemistry, Nature biotechmology, 18325,1 2020.
10

---

[13] Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher. GuacaMol: bench- marking models for de novo molecular design. Journal of chemical information and modeling. 59(3):1096 108, 2019.
[14] Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitmik. Therapeuties Data Commons: machine learning datasets and tasks for therapeutics. NeurlPS Track Dalasets and Benchmarks, 2021.
[15] Wenhao Gao, Tianfan Fu, Jimeng Sun, and Connor W Coley. Sample efficiency matters: A benchmark for practical molecular optimization. NeurIPS Track Datasets and Benchmarks, 2022.
[16] Jan H Jensen. A graph-based genetic algorithm and generative model/Monte Carlo Tree Search for the exploration of chemical space. Chemical science, 10(12):3567-3577 2019.
[17] Jacob O Spiegel and Jacob Durrant. AutoGrow4; an open-source genetic algorithm for de novo drug design and lead optimization. Journal of cheminformatiics 12(1):1-16, 2020.
[18] Austin Tripp, Gregor NC Simm, and José Miguel Hernández-Lobato. A fresh look at de novo molecular design benchmarks. In NeurIPS 2021 Al for Science Workshop, 2021.
[19] Victor Gareia Satorras, Emiel Hoogeboom, and Max Welling. (n) equivariant graph neural networks. Proceedings of the 38th International Conference on Machine Learning. ICML, 139:9323-9333 2021.
[20] Rafael Benjamín Sánchez-Lengellng Dennis Sheberla, Jorge paallera-lprraguirree, Timothy D Hirzel, Ryan Adams, and Alán Aspuru- Automatic chemical design using a data-driven continuous representation of molecules. ACS central science, 2018.
Jennifer Wei, David Duvenaud, José Miguel Beeránddz--Lobate
[21] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. ICMI., 2018.
[22] Gabriel Lima Guimaraes, Benjamin Sanchez-Lengelingg Carlos Outeiral, Pedro Luis Cunha Farias, and Alán Aspuru-Guzik. Objective-Reinforee Generative Adversarial Networks (OR- GAN) for sequence generation models. arXiv preprint urXiv:/705. 10843, 2017.
[23] Nicola De Cao and Thomas Kipf. MolGAN: An implicit generative model for small molecular graphs, 2018.
[24] Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. GraphAF: a flow-based autoregressive model for molecular graph generation. In ICLR, 2020.
[25] Youzhi Luo, Keqiang Yan, and Shuiwang Ji. GraphDF: A discrete flow model for molecular graph generation. Proceedings of the 38th International Conference on Machine Learning, ICML., 139:7192-7203, 2021.
[26] Meng Liu, Keqiang Yan, Bora Oztekin, and Shuiwang Ji. GraphEBM: Molecular graph generation with energy-based models. arXiv preprint arXiv:2102.00556 2021.
[27] AkshatKumar Nigam, Pascal Friederich, Mario Krenn, and Alán Aspuru-Guraa Guzik. Augmenting genetic algorithms with deep neural networks for exploring the chemical space. In ICLR, 2020.
[28] Wenhao Gao, Rocío Mercado, and Connor Coley. Amortized tree generation for bottom-up synthesis planning and synthesizable molecular design. International Conference on Learning Representations, 2022.
[29] M. Olivecrona, T. Blaschke, O. Engkvist, and H. Chen. Molecular de-novo design through deep reinforcement learning. Journal of Cheminformatiic,, 2017.
[30] Jiaxuan You, Bowen Liu, Rex Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. In NIPS, 2018.
[31] Zhenpeng Zhou, Steven Kearnes, Li, Richard N Zare, and Patrick Riley. Optimization of molecules via deep reinforcement learning. Scientific reports, 91): 2019.
11

---

[32] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Multi-objective molecule generation using interpretable substructures. In International Conference on Machine Learning, pages 4849-4855 4859. PMLR, 2020.
[33] Sungsoo Ahn, Junsu Kim, Hankook Lee, and Jinwoo Shin. Guiding deep molecular optimization with genetic exploration. Advances in neural information processing systems, 01112008-1222) 2020.
[34] Ksenia Korovina, Sailun Xu, Kirthevasan Kandasamy, Willie Neiswanger, Barnabas Poczos, Jeff Schneider, and Eric Xing. ChemBO: Bayesian optimization of small organic molecules with synthesizable recommendations. In International Conference on Artificial Intelligence and Statistics, pages 3393-3403. PMLR, 2020.
[35] Tianfan Fu, Cao Xiao, Xinhao Li, Lucas M Glass, and Jimeng Sun. MIMOSA: Multi-constraint molecule sampling for molecule optimization. AAAI, 2020.
[36] Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. MARS: Markov molecular sampling for multi-objective drug discovery. In ICLR, 2021.
[37] Yoshua Bengio, Tristan Deleu, Edward J. Hu, Salem Lahlou, Mo Tiwari, and Emmanuel Bengio. GFlowNet foundations. CoRR, abs/21 1 1.09266, 2021.
[38] Tianfan Fu, Wenhao Gao, Cao Xiao, Jacob Yasonik, Connor W Coley, and Jimeng Sun. Differentiable scaffolding tree for molecular optimization. International Conference on Learning Representations, 2022.
[39] Cynthia Shen, Mario Krenn, Sagi Eppel, and Alan Aspuru-Guzik. Deep molecular dreaming: Inverse machine learning for de-novo molecular design and interpretability with surjective representations. Muchine Learning: Science and Technology, 2021.
[40] Tobiasz Cieplinski, Tomasz Danel, Sabina Podlewska, and Stanislaw Jastrzebski. We should at least be able to design molecules that dock well. arXiv:2006. 2020.
[41] Casper Steinmann and Jan H Jensen. Using genetic algorithm to find molecules with good docking scores. PeerJ Physical Chemistry, 3:e18, 2021.
[42] Soojung Yang, Doyeong Hwang, Seul Lee, Seongok Ryu, and Sung Ju Hwang. Hit and lead discovery with explorative RI. and fragment-based molecule generation. Advances in Newral Information Processing Systems, 34, 2021.
[43] Zhaowen Luo, Renxiao Wang. and Luhua Lai. RASSE: a new method for structure- strueture-based drug design. Journal of chemical information and computer sciences, 1666)118771119, 1996.
[44] Valerie Gillet, Peter Johnson, Pauline Mata, Sandor Sike, and Philip Williams. SPROUT: program for structure generation. Journal of computer-aided molecular design, 122):127 1993.
[45] David Pearlman and Mark Murcko. CONCEPTS: drug suggestion. Journal of computational chemistry, 14(10): 1193,,11,,,,11 1993.
New dynamic algorithm for de novo
[46] Dominique Douguet, Etienne Thoreau, and Gérard Grassy. A genetic algorithm for the auto- mated generation of small organic molecules: drug design using an evolutionary algorithm. Journal of computer-aided molecular design, 1465):449446, 2000.
[47] Jacob D Durrant, Steffen Lindert, and J Andrew McCammon. AutoGrow 3.0: improved algo- rithm for chemically tractable, semi-automated protein inhibitor design. Journal of Molecular Graphics and Modelling, 44:104 14:104-11 112, 2013.
[48] Renxiao Wang, Liang Liu, Luhua Lai, and Youqi Tang. SCORE: A new empirical method for estimating the binding affinity of a protein-ligand complex. Molecular modeling annual, 4(12):379-39. 1998.
[49] Wenhao Gao and Connor Coley. The synthesizability molecules proposed by generative models. Journal of chemical information and modeling, 50(12):5714-572) 2020.
12

---

[50] Markus Hartenfeller, Martin Eberle, Peter Meier, Cristina Nieto-Oberhuber Karl-Heinz Alt- mann, Gisbert Schneider, Edgar Jacoby, and Steffen Renner. A collection of robust organic synthesis reactions for in silico molecule design. Journal of chemical information and modeling. 5111))309933008, 2011.
[51] Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint arXiv: (7X0v00970005944 2017.
[52] Oleg Trott and Arthur J Olson. Autodock Vina: improving the speed and accuracy of docking with new scoring functionn efficient optimization, and multithreading. Journal of computational chemistry, 31(2):55-466) 2010.
[53] Michael M Mysinger, Michael Carchia, John Irwin, and Brian K Shoichet. Directory of useful decoys, enhanced (DUD-E): better ligands and decoys for better benchmarking. Journal of medicinal chemistry, 55(1 (1114)658226594 2012.
[54] Chun-Hui Zhang, Elizabeth A Stone, Maya Deshmukh, Joseph A Ippolito, Mohammad M Ghahremanpour, Julian Tirado-Rives, Krasimir Spasov, Shuo Zhang, Yuka Takeo, Shalley N Kudalkar, et al. Potent noncovalent inhibitors of the main protease of sars-cov-2 from molecular sculpting of the drug perampanel guided by free energy perturbation calculations. ACS central science, 7(3):467-477)) 2021.
[55] Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alan Aspuru-Guzik. Self-refe reff-referencing embedded strings (SELFIES): A 100% robust molecular string representation. Machine Learning: Science and Technology, 14):0455224 2020.
[56] Teague Sterling and John Irwin. ZINC 15-ligand discovery for everyone. Journal of chemical information and modeling, 55(11):2324 2337, 2015.
[57] Paul G Francoeur, Tomohide Masuda, Jocelyn Sunseri, Andrew Jia, Richard B Iovanisci, Ian Snyder, and David R Koes. shree-dimensional convolutional neural networks and a cross- docked data set for structure-basee drug design. Journal of Chemical Information and Modeling. 60(9):4200 4215, 2020.
[58] Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Artificial intelligence foundation for therapeutic science. Nature Chemical Biology, 2022.
[59] John Bradshaw, Brooks Paige, Matt Kusner, Marwin Segler, and José Miguel Hernández- Lobato. Barking up the right tree: an approach to search over molecule synthesis dags. Advances in Neural Information Processing Systems, 33:6852 (6852-6866, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you deseribe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL.)? [Yes] See the supplementary materials.
13

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See the supplementary materials.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)? [Yes] See the supplementary materials.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See the supplementary materials.
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See the supplementary materials.
(b) Did you mention the license of the assets? [Yes] See the supplementary materials.
(c) Did you include any assets either in the supplemental material URL? [Yes]
See the supplementary materials.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals. if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "j8WfJMyCtEA", "reviewer": "Reviewer_KpKV", "summary": "The paper proposes a Reinforced Genetic Algorithm that combines different models for structure-based drug design. The algorithm takes the 3D structure of the targets and ligands as inputs and is pre-trained using native complex structures to utilize the knowledge of the shared binding physics from different targets and then fine-tuned during optimization. Empirical studies show that the proposed method outperforms the baselines and the ablation study suggests the contribution of two training strategies. ", "questions": "I would suggest the author focus on one aspect of the method instead of considering multiple model combinations. This creates confusion on which part is more important than others. ", "limitations": "See above", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The paper proposes a new method pipeline to solve an existing drug design problem.\n\nWeakness: There is very little technical innovation, more like a direct application of existing methods to a specific problem. Regarding the model design, there is no explanation of the rationalities. For example, why did the authors combine the MDP with a EE neural network? There is also no explanation of the theoretical support for convergence or a better performance guarantee. In addition, the method is heavily adapted to one specific problem and can’t be generalized to a wide range of problems. The experiments are not representative either. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ieniULSPQvv", "reviewer": "Reviewer_yDTi", "summary": "This article introduces a reinforced genetic algorithm to tackle the problem of structure-based drug design, one of the key approaches to achieve fast and efficient drug discovery. The authors overcome limitations of current genetic algorithms, which is one of the most efficient screening procedures for binding affinity, by reformulating it as a Markov decision process, using neural networks to make informed decisions and suppress the random-walk behavior. ", "questions": "In the diversity metric, the RGA algorithm introduced in this paper provides somewhat worse results than competing methods and even brute force screening. Could the authors comment on the importance of this metric and whether some improvements could be made in this respect?\n", "limitations": "While this work provides a useful approach to prioritizing promising drug design steps, the pathway to an implementation in real life drug design cycle remains unclear.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This article is well written and provides meaningful theoretical advances to a key pharmaceutical science problem. It provides thorough comparisons with current existing methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Responsible Research Practice (e.g., IRB, documentation, research ethics)"]}, "extra_text": {}}, {"review_id": "cTNHluizJ05", "reviewer": "Reviewer_QPzy", "summary": "This paper proposes a genetic algorithm (mostly based on Autogrow) guided by reinforcement learning. The RL agent in this method can directly perceive the 3D structure of protein targets. In contrast, Autogrow searches molecules in a ``blind’’ way, unable to directly see the 3D structure during sampling.\n\nIn this work, RL is used to guide two processes in the genetic algorithm: crossover and mutation. The crossover agent selects parents and mutation for producing ``children’’ molecules. The mutation agent selects chemical reaction to modify the molecules.\n\nThis paper presented a very comprehensive evaluation of the proposed method and existing SBDD methods. The improvement over of the method over previous ones is evidenced by the comprehensive experiment, though it is not very significant compared to traditional genetic algorithm.", "questions": "See strengths and weakness.", "limitations": "See strengths and weakness.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n(1) Using RL to empower GA-based SBDD is a good idea. It can make the GA process more directed and uses more information such as protein 3D structures.\n\n(2) Extremely comprehensive survey and evaluation of previous SBDD methods. The survey and evaluation cover a wide range of methods including graph-based, 3D-based, and GA-based.\n\n(3) Acceptable performance improvement. Though the improvement in performance is over GA-based algorithms is not that significant, it surpasses GNN-based and 3D-based methods by a very significant margin.\n\nWeakness\n\n(1) Bias in evaluation metric. The algorithm optimizes Vina score, it is not surprising that the Vina score is good in evaluation. Have the author considered using different scoring functions for GA and testing?\n\n(2) Few examples of generation. I did not find any presentation of generated molecules in the main text or in the supplementary material. Examples are important and their comparison to a ``ground truth’’ molecules are important to understand and evaluate what the model generates.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aTDKPxxhmC", "reviewer": "Reviewer_S7k1", "summary": "This authors of  this submission propose to apply structure based drug design information to the selection, crossover, and mutation operations of evolutionary computation for drug discovery. An equivariance neural network is used to represent target-ligand binding complex and used as a base network for crossover and mutation action selections. In these evolutionary operations, substructure similarity was considered in crossover; and mutation was inspired by chemical reactions. This proposed method is a hybrid model combining deep reinforcement learning and evolutionary optimization for drug design. Experimental results show that the proposed method outperform existing benchmarks. Ablation studies show that the incorporation of structural information in crossover and mutation can improve performance in terms of several metrics.  ", "questions": "1. I would like to see the performance of binding affinity regression in ENNs. \n2. I would like to see more discussions of hybrid models, such as RL+EC, DL+EC, etc. \n3. Significance studies, in terms of p-values, should be reported to compare the proposed work with existing (naive) GA algorithms for drug design. \n4. Scalability of this proposed model should be explicitly discussed as well. ", "limitations": "Limitations and potential improvements of the work were slightly discussed in the conclusion section. ", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The consideration of structural information through policy networks for selection, crossover, and mutation are the major contribution of this work from my viewpoint. The combination of reinforcement learning (RL) and evolutionary computation (EC) may be novel for this application. However, the authors should discuss more about existing work that combines RL and EC in general. \n2. Experimental benchmarkings are thorough. Various well-known existing models (modified to integrate docking) were compared. \n\nWeaknesses: \n1. The regression performance of the ENNs was not discussed. The performance of all other operations depends on the reliability of such ENNs networks. Thus, the authors must show their regression results in pretraining ENNs.\n2. Evolutionary operations were largely influenced by Autogrow 4.0. The major difference is that policy networks were employed. The base neural networks used in this work are existing architectures, which weakens the novelty of this work. The signifiance of the result obtained by the proposed work (which is a complicated process) in comparison with other simpler GAs should be further studies, in terms of p-value. \n3. The presentation of this paper could be improved. Line 230: is X a mutation child? a MLP -> an MLP. structured based -> structure based. References should be reformatted. Please pay attention of capitalizations. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept