De novo Drug Design using Reinforcement Learning with Multiple GPT Agents
Authors: Xiuyuan Hu
Guoqing Liu
Yang Zhao
Hao Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: De novo drug design
Molecular generation
Multi-agent reinforcement learning
GPT

Abstract: *De novo* drug design is a pivotal issue in pharmacology and a new area of focus in AI for science research. A central challenge in this field is to generate molecules with specific properties while also producing a wide range of diverse candidates. Although advanced technologies such as transformer models and reinforcement learning have been applied in drug design, their potential has not been fully realized. Therefore, we propose MolRL-MGPT, a reinforcement learning algorithm with multiple GPT agents for drug molecular generation. To promote molecular diversity, we encourage the agents to collaborate in searching for desirable molecules in diverse directions. Our algorithm has shown promising results on the GuacaMol benchmark and exhibits efficacy in designing inhibitors against SARS-CoV-2 protein targets. The codes are available at: https://github.com/HXYfighter/MolRL-MGPT.

De novo Drug Design using Reinforcement Learning with Multiple GPT Agents
Xiuyuan Hu Hul:2; Guoqing Liu?! Yang Zhao', Hao Zhang
Department of Electronic Engineering, Tsinghua University
Microsoft Research Al4Science
buxy22@mails..ssinuueee eereshaaaeeu....m en, guoqingliudmicroooof zhao thao-yangotsinghua. edu. cn, haozhangStsinghua. edu. cn
Abstract
De novo drug design is a pivotal issue in pharmacology and new area of focus in AI for science research. A central challenge in this field generate molecules with specific properties while also producing a wide range of diverse candidates. Although advanced technologies such as transformer models and reinforcement learning have been applied in drug design, their potential has not been fully realized. Therefore, we propose MoIRL-MGPT, reinforcement learning algorithm with multiple GPT agents for drug molecular generation. To promote molecular diversity, we encourage the agents to collaborate in searching for desirable molecules diverse directions. Our algorithm has shown promising results on the GuacaMol benchmark and exhibits efficacy in designing inhibitors against protein targets. The codes are available at: betps::/github. com/HXYfighter/ MoIRL-MGPT
1 Introduction
In recent years, significant strides have been made in computer-aided drug discovery (CADD) thanks the development of various fields, including proteomics, genomics and deep learning 13 66 The conventional drug discovery process is typically cone-consuming and financially demanding, with low success rate. However, advanced machine learning techniques have the potential reverse this predicament, greatly benefiting the economy and society's 's development 58). Currently, major issue in the field of pharmacology goal-directed de novo drug design, which involves generating new drug molecules with specific biochemical properties, such designing compounds with high binding affinity to a designated protein target [15 3 Despite the numerous proposed machine learning algorithms for molecular generation, the chemical space is vast, and the relationship between molecular properties and structures is intricate, making it challenging to obtain satisfactory results in practical applications 42]
Molecular diversity is critical concern in drug design because a diverse set of candidates can provide more choices for downstream screening and avoid drug resistance and unknown side effects [2] However, existing algorithms encounter challenges in designing diverse drug molecules. Many algorithms tend to generate sets of highly similar compounds, which is of little value for the subsequent drug development process.
Over the past several years, generative language models have made remarkable strides in natural language processing, vision, and audio. Among these models, the generative pre-trained transformer (GPT) is particularly notable, which has demonstrated impressive capabilities of language under-
Work was done while Xiuyuan Hu was research intern at Microsoft Research. Corresponding author: Hao Zhang (Ist), Guoqing Liu.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In the field of chemistry, transformer- language models pre-trained on the SMILES (simplified molecular input line entry system) representation of molecules have emerged. Through transfer learning, these models can be adapted to range of tasks, including molecular property prediction, reaction prediction, and molecular optimization [26 27
standing and generation 45 46
Reinforcement learning (RL) has emerged as a promising approach for de novo drug design [39 68 |9 The basic idea is consider molecular property predictors (scoring functions) rewards and train an RL agent to iteratively generate candidate compounds with increasingly high scores. These RL algorithms can explore the vast chemical space remarkably faster than human chemists, but they may have limitations in molecular diversity. Although multi-agent reinforcement learning (MARL) is commonly used technique for promoting diversity in searching problems [34 14]. has not yet been effectively utilized in the field of drug design.
To address the limitations of current approaches, we propose MoIRI-MGPT (Molecular design using Reinforcement Learning with Multiple GPT agents), a novel MARI. framework for de novo drug molecular design that utilizes GPT models as agents. Our approach treats molecular design as a cooperative Markov game, where multiple lightweight GPT agents collaborate to generate high-scoring molecules during the RL process. These agents share identical pre-trained parameters on a molecular SMILES dataset for initialization and have a common optimization objective for specific properties. To enhance the diversity of generated candidates, we incorporate an auxiliary loss function that encourages agents to explore in diverse directions. Our algorithm has demonstrated superior performance compared various baselines on the GuacaMol benchmark. We also apply to resolve the real-world problem of designing candidates against two SARS-CoV-2 protein targets, resulting in potentially desirable candidates with good binding affinity, -ike-liseness, and synthetic accessibility. Moreover, we further validate the effectiveness of our design by comparative and ablation experiments on GNK33, JNK3 and QED maximization tasks.
2
Related works
Machine learning has become a formidable instrument in molecular generation with applications to de novo drug design, aiding scientists in identifying novel molecules that possess the desired properties for drug discovery. The molecular representation is the basis of molecular generation and optimization algorithms, which can be roughly divided into three categories: 1D string, 2D image, and 3D geometry [15..
SMILES is the most commonly used ID representation of molecules, which employs strings of characters to encode a molecule. Although the SMILES cach molecule is not unique, there is only one canonical SMILES, and each SMILES corresponds to maximum of molecule. It is noteworthy that most SMILES strings are invalid; that is, their corresponding structures cannot exist in the real world. Some algorithms and techniques applied in natural language processing (NLP) have been adopted to generate molecules using SMILES representations, such as variational autoencoder (VAE) [22] [16,, recurrent neural network (RNN) 49] generative adversarial network (GAN) (23 and Bayesian optimization (BO) (37]
On the other hand, 2D and 3D molecular representations are more intuitive and have become popular in molecular design in recent years. For 2D molccular graphs with vertices corresponding to atoms and edges corresponding to chemical bonds between atoms, techniques such graph neural network (GNN), genetic algorithm (GA) and flow network for graph data have already been applied to drug 35| 21.. For 3D geometries of compounds, which theoretically contain the most structural information of the molecules, although models including diffusion have been introduced to the chemistry field 220 B3), they cannot currently design candidates with desired biochemical properties as ID/2D de novo drug design approaches.
development 28 30 62
RL-based drug design algorithms Reinforcement learning is the most popular technique for molecule generation. In molecular design, deep neural networks are usually employed as agents in studies that use RL for ID string generation, which are then fine-tuned via customized reward functions 39
59 Likewise, when RL is applied to generating 2D molecular graphs, the states correspond to incomplete representations, and the actions involve adding substructures (such as
atoms, bonds, and rings) to specific positions (65 68 29
63 19
What is more, recent studies

---

have incorporated 3D geometries into the RL process to consider the spatial properties of molecules during their generation [51 52 18.
Transformers for chemical language Transformer is type of deep neural network architecture entirely based on attention mechanism and has been widely applied in the field of NLP [57].
has demonstrated better ability to process long text sequences and parallel computing capabilities than traditional architectures such as RNN. Some works have already applied transformers in the
field of chemistry. For instance, ChemForme (26). MolGPT [4] and [24) have focused on SMILES- based pre-tree pre-trained transformer models. Meanwhile, PROTAC-RL [67]. TamGent [61] and 44 have
concentrated on transformer based drug design against protein targets, while MCMG 59 has used transformer component to enhance the learning capability of the algorithm.
Diversity in drug design Previous research has proposed using reinforcement learning to generate diverse molecules for drug development [8 59 41 However, these studies have not adequately addressed this concern, and multi-agent reinforcement learning is yet to be effectively applied in de novo drug design.
3 Methodology
3.1 Problem definition
The fundamental aspect of the molecular generation problem formulation is scoring function s(x: of molecular properties, also known as an oracle, with the input of x being a molecule and the output being real number. Molecular properties and the number of aromatic rings, as well as chemical properties such as logP and drug-likeness. Additionally, the molecular properties that we often related to biological activity, with binding affinity to protein targets being the most common, which can be estimated using docking software.
include physical properties such as molecular weight
concerned with in real-world drug design
Standardly, the scoring function s(.) is typically constrained within the interval [0, 1] when applied to valid molecular input, where higher score corresponds to better molecular property. Invalid molecules consistently receive a score of -1. Researchers may implement a transformation function t(-) for each molecule property predictor p(z)) to achieve uniformity with the standard form, and create a multi-objective scoring function through a weighted combination of various oracles.
sxx) E [0, [0,1]um {-1}, s(x) t(p(z)) or sxx) Σ wy t((p())),, Σu,= Ew==1 =1
The evaluation of de novo molecule generation algorithm usually requires the assessment of set of generated high-scoring molecules, and a common approach is to report the average score and diversity of the top-k scoring generated molecules, where k N+ is given. widely used metric of molecular diversity is internal diversity (IntDiv) 66:
In) = |A[(IA] -
Σ (2.))EAAA,77
((F))
(2)
where is set of compounds, dr represents the Tanimoto distance cereendedcoonecttvii fingerprint (ECFP) [4] of a molecule
55 and F(x) refers to the
In drug molccular design tasks, we primarily focus on evaluating the final set of generated molccular candidates. Generally, pay little attention to the generation process, including factors like time and computing resource consumption, as these costs are relatively insignificant compared to the conventional drug discovery process.
Hence, we can represent the problem of designing novel drug candidates as a cooperative Markov game consisting of multiple generative model agents. At every iteration (i = 1, 2, ... s), each agent = 1, 2,...,,u n)) generates yn molecules (actions), and the scoring function acts as an environment providing rewards in the form of scores, which in turn are used to update the agents. The game's 's objective is primarily maximize the average of the highest scores of generated molecules and secondarily to improve the diversity of the candidates.

---

3.2 MoIRL-MGPT algorithm
Loss
SMILES
SMILLE Taken Prediction
Layer Normalization
Agent
SMILES
Loss
Agent
SMILES
Feed Forward
Prior
Layer Normalization
Likelihaod
Rowards
Scoring Function
Expérience Replay
Updáte Memory
Masked 8-head Self Attention
Memory
Layer Normalization
Joora
SMILLE Token Embeddding Position Embedding
(a)
(b)
Figure 1: (a) Overview of our MoIRL-MGPT algorithm. (b) The model architecture of the GPT prior model and agents.
As illustrated in (a), the MoIRL-MGPT algorithm consists of iterations which multiple agents updated in reinforcement learning way. Specifically, in each iteration, each agent first samples set of SMILES strings with high likelihood and retrieves several high-scoring molecules in the past iterations from the molecular memory. Then, the scoring function used obtain the reward of each generated SMILES string. The GPT agents conduct this process in order. The loss functions are designed as follows.
Loss function In MoIRL-MGPT, the reward function is defined as the scores predicted by the scoring function. Our primary objective is to increase the average scores of SMILES strings generated by each agent. Furthermore, to prevent the agents from disregarding the knowledge acquired during the pre-training phase, we impose penalty the deviation between the policies and the prior model, which is also employed to initialize all the agents. The loss function of the 1st GPT agent is designed as follows:
L (x; 01) [log P(r) log P(x) P(r) 01 +0110s((5 s(x))]
where O is the parameters of the 1st agent, is generated molecule, aı is coefficient for controlling the term of scores, and mox)l refers to the likelihood of generating x by model. should be noted that typically P(z)Prior P(z)
Additionally, we encourage the agents to explore different directions in the chemical space instead of conducting repetitive searches by introducing a term indicating the deviation between the current agent and previous ones. The loss function of the keth GPT agent designed as follows:
(x; 0x) =L1 (2; Op)
Σ
(x) log P(x) Agent, log P(x) Agent,
= [log P(x) log P(x) +01 sxx))2
(4)
Σ s(x) log P(x) P(z) log P(x) (()nt,))!))
where k: = 1,2,... n, 0₂ the parameters of the kth agent, and 02 coefficient for the encourage- ment term.

---

The selection of coefficients can be adapted according to the specific task at hand. Moreover, we propose implement decreasing schedule for σi such that σi decreases as s(x) increases during the RI. process. This is because we expect the increase in scores to correspond to decrease in loss.
Molecular Memory MoIRL-MGPT utilizes memory consisting of high-scoring molecules with a maximum size of 1000. The memory is updated with every SMILES string sampled, and compounds stored in the memory are sorted based on their scores.
3.3 Implementations
Pre-training on chemical language Instead of the commonly used recurrent neural network (RNN) architecture for generating chemical language in SMILES strings, we adopt transformer-based architecture. As shown in Fig (b), our model a mini version of the GPT model with 8 layers of transformer blocks, each consisting of 8 attention heads. The embedding size is 256, and the maximum length SMILES strings is set to 128. The pre-training approach of our model involved learning molecular structure patterns. Our prior model has only around 6.4M parameters, much less than the GPT-2 model [46].
Two versions of the prior model are trained using unsupervised learning the ChEMBL 36 and 7INC-100M 53 datasets, respectively. The data are preprocessed by removing molecules with ionized structures and SMILES strings longer than 100 characters, which are not typical for small molecule drugs. Additionally, we use SMILES randomization for data augmentation, which has been proven helpful in enhancing generating capacity [3]. Ten training epochs are conducted with a batch size 2048 and a maximum learning rate of 0.001, using a learning rate schedule featuring warm-up and cosine decay. The training ChEMBL (roughly 2 million SMILES) takes around 5 hours using a single NVIDIA A100 GPU.
The pre-trained SMILES models should be objectively evaluated by the valid ratio of the generated molecules, representing the proportion of valid molecules present in the generated SMILES. Generally, this value can exceed 90% 43 Our pre-trained transformer model has achieved valid ratio of 98% generating molecules, indicating its success in capturing the inherent grammar and rules of molecular SMILES strings.
Experience replay Experience replay is a widely employed technique in deep reinforcement learning. The agent or learner records its experience in a memory buffer and randomly samples past experiences, enabling to learn from previous experiences with reduced correlation between consecutive training samples. This technique helps agents avoid overfitting, generalize to unseen situations, and achieve more stable and efficient learning process [32]. In MoIRL-MGPT, we replay the "successful" experiences by randomly sampling 5 molecules from the 25 highest-scoring molecules in the molecular memory and computing the loss on these molecules together with SMILES strings generated in the current iteration.
*Similarity penalization Some previous works for de novo drug design add a penalizing term for high similarity of identical skeletons between new molecules and previously found high-scoring molecules in order to encourage the agent to search in unexplored space rather than repeatedly finding the same similar compounds [9 60). However, experiments indicate that this trick does not work with our algorithm.
Experiments
To demonstrate the performance of MoIRL-MGPT, we conduct three groups of experiments. Firstly, we run our algorithm on the public de novo molecular design benchmark, GuacaMol, and compare with existing advanced methods. Secondly, we apply MoIRL-MGPT to the design of inhibitors against SARS-CoV-2 protein targets, which is a current real-world challenge for human beings. Thirdly, we conduct comparative and ablation experiments on GNK3,9, JNK3 and QED maximization tasks to validate the effectiveness of modules in our design.
5

---

4.1 GuacaMol benchmark
GuacaMol [00] is widely recognized open-source evaluation framework for de novo molecular design algorithms. It contains 20 goal-directed tasks mimicking the drug discovery objectives, corresponding to 20 standard scoring functions descrihed in Sec These tasks cover commonly used objectives in drug design, such as structural, physicochemical. and biochemical properties.
For comparison, we select the following baselines: (1) SMILES GA [64], a genetic algorithm based on SMILES; (2) SMILES LSTM [49], LSTM network generating SMILES strings autoregressively, combined with a hill-climb algorithm for optimization; (3) Graph GA (28], a genetic algorithm with crossovers and mutations performed molecular graphs: (4) Reinvent 39 deep reinforcement
learning framework for training an RNN model generating SMILES; (5) GEGL [ genetic expert- guided learning framework for training RNN for molecular generation. Results of some other baselines are shown in the Appendix.
The prior model for MoIRL-MGPT was pre-trained on the official GuacaMol dataset, which is subset ChEMBL. The haper-parameters are set such that we run each tasks for 5000 tasks (break the score has achieved 1.000) with 4 GPT agents, and the batch size (number of sampled SMILES strings) of each agent 256. The values of coefficients are set to σ = 1000 with a linear decreasing schedule, and 02 0.1. The entire of 20 tasks tasks less than 400 hours to complete when a single NVIDIA A100 GPU.
Table Scores of MoIRL-MGPT and other baselines the GuacaMol benchmark.
SMILES
SMILES LSTM
Graph GA
MolRL-
Tasks
Reinvent
GEGL
GA
MGPT
Celecoxib rediscovery
0.732
1.000
1.000
1.000
1.000
1.000
2. Troglitazone rediscovery
0.515
1.000
1.000
1.000
0.552
1.000
3. Thiothixene rediscovery
0.598
1.000
1.000
1.000
1.000
1.000
4. Aripiprazole similarity
0.834
1.000
1.000
1.000
1.000
1.000
5. Albuterol similarity
0.907
1.000
1.000
1.000
1.000
1.000
6. Mestranol similarity
0.790
1.000
1.000
1.000
1.000
1.000
7. Cn H24
0.829
0.993
0.971
0.999
1.000
1.000
8. CI
0.889
0.879
0.982
0.877
1.000
0.939
9. Median molecules
0.334
0.438
0.406
0.434
0.455
0.449
10. Median molecules 2
0.380
0.422
0.432
0.395
0.437
0.422
11. Osimertinib MPO
0.886
0.907
0.953
0.889
1.000
0.977
12. Fexofenadine MPO
0.931
0.959
0.998
1.000
1.000
1.000
13. Ranolazine MPO
0,881
0.855
0.920
0.895
0.933
0.939
14. Perindopril MPO
0.661
0.808
0.792
0.764
0.833
0.810
15. Amlodipine MPO
0.722
0.894
0.894
0.888
0.905
0.906
16. Sitagliptin MPO
0.689
0.545
0.891
0.539
0.749
0.823
17. Zaleplon MPO
0.413
0.669
0.754
0.590
0.763
0.790
18. Valsartan SMARTS
0.552
0.978
0.990
0.095
1.000
0.997
19. deco hop
0.970
0.996
1.000
0.994
1.000
1.000
20. scaffold hop
0.885
0.998
1.000
0.990
1.000
1.000
Total
14.396
17.340
17.983
16.350
17.627
18.052
As shown in Table I the MoIRL-MGPT algorithm outperforms baselines in 13 molecular design tasks in the GuacaMol benchmark, and its total score 20 tasks also ranks first. These results indicate that MoIRL-MGPT excels in general scenarios of de novo drug design.
4.2 Designing inhibitors against SARS-CoV-2 targets
Molecular docking a computational method used to predict the binding modes of small molecules to protein target. It involves predicting the spatial orientation and binding affinity of the small molecule in the active site identifying potential drug candidates and understanding how they interact with their targets. Autodock most widely used molecular docking software. However, we opt to use Quick Vina 2 12). novel and more efficient alternative.
protein. This information is useful drug discovery since enables
Vina [56) is currently

---

The quantitative binding affinity is named docking score, which is calculated based on the energies of the interaction between the ligand and the receptor, and a lower docking score indicates a more stable and, therefore, more likely binding pose. Typically, docking scores are negative, and desirable docking scores range from -10 to kcal/mol. Therefore, we use reverse sigmoid function as the transformation function of the docking score:
Ldocking (p) 1 + 100.625-(p+-0))
(5)
SARS-CoV-2 (Severe Acute Respiratory Syndrome Coronavirus 2), commonly referred as the novel coronavirus, is a respiratory virus which, in late 2019, emerged as a global pandemic leading to the COVID-19 disease that mainly targets the respiratory system. This disease is severe public health concern that continues to pose a crisis worldwide and requires a comprehensive response to apply our algorithm to the design of inhibitors against protein targets of SARS- CoV-2, which a significant real-world issue. Following [48], we select two targets: PLPro_ J and BdRp_6YYT
mitigate its spread and negative effects. Therefore,
Besides docking scores, we also consider two additional oracles often employed in practical drug design: (1) QED (Quantitative Estimate of Drug-likenesse lilelikeeesss, which quantifies the drug-likeness of molecule based on the concept of desirability of eight molecular properties 77. ranging [o, 1]; (2)
SA (Synthetic Accessibility), which incorporates fragment contributions and a complexity penalty
17 ranging in [1, 10). Their transformation functions are linear:
10 10-P
(p) p,
tsS (p)
9
Moreover, the scoring function for designing inhibitors against SARS-CoV-2 protein targets is linear combination of docking scores, QED scores and SA scores:
Stotal (x) 0.8 Sdocking (x) + 0.1 SQED (x) + 0.1 0.1. (x)
We run the RL process for 1000 iterations on each target with 4 GPT agents, and the batch size of each agent is 128. The whole process for target takes approximately 100 hours single NVIDIA A100 GPU and 64 CPU cores] Details of the generated candidates are as follows.
PLPro 7JIR target PLPro (papain-livg protease) an attractive target for SARS-CoVas since plays fundamental role in cleavage and maturation of viral polyproteins, assembly of the replicase- transcriptase complex, and disruption of host responses. 7JRR a C111S mutant form of the structure of PLPro 40) Three candidate inhibitors against the PLPro__]D 7JIR target generated by MoIRL-MGPT are shown Table 2
Table 2: Candidate inhibitors against the PI.Pro_JJIR target generated by MoIRI-MGPT. The SMILES of the three candidates are: O=C(c wtt.e//ccccctccoooooeeoooooooooa
(2) cl
(3) Qcc.cooooocccccccccclccltcoaoooa
Molecule
docking score (L)
-11.3
-11.1
-11.2
QED score (f)
0.310
0.258 2.729
0.214
SA score (L)
2.530
2.549
http:///ww.rrrb.or/ssttttttttt/r
https://www.rcbb.. http://wwww.rbbbbrrrrssttutttttt
Docking consumes more than 93% of the time, although parallel computing of Quick Vina software is implemented.

---

RdRp 6YYT target RdRp dependepnnd RNA polymerase) works for the replication of genome and the transcription of genes of SARS-CoV-2, and 6YYT is the PDB identification code of its structure [25] Three candidate inhibitors against the RdRp_6YYe target generated by MoIRI.- MGPT are shown in Table
Table 3: Candidate inhibitors against the RdRp_6YYT target generated by MoIRL-MGPT. The
SMILES of the three candidates are:
(2) oellCC//arreeeeeeeeeeoeeoceeeeed
nc3cccoc23)CCNI:
CcceaNNCCCCCDDpeecaaaaaaoocoCcoC 3n2)cc1.
(3) Cclec((
Bose
papro
Molecule
docking score (L)
-12.3
-13.1
-13.2
QED score (1)
0.237
0.253
0.241
SA score (L)
2.772
3.104
2.806
The generated candidates against the PLPro_ PLPR and RdRp_6Y0 _ targets exhibit desirable binding affinities (docking scores) and synthetic accessibility. Although the drug-likeness scores these candidates may not be high, it is reasonable since QED is estimated based on the distribution of existing drug molecules. All these drugs are ineffective inhibiting the two targets of SARS-CoV-2; thus, designing new dissimilar compounds becomes necessary.
4.3 GSK33, JNK3 and QED maximization
To demonstrate the effectiveness of our design, we perform ablation experiments commonly used oracles for simulating real-world drug design with low consumption: GSK33 and JNK3. Their scores estimated by random forests trained EXCAPE-DB dataset (54), which measure the bioactivities of molecules against the Glycogen synthase kinase beta target (GSK3,3) and the c-Jun N-terminal kinase target (JNK3). Previous research has shown that inhibiting these targets can benefit the treatment of Alzheimer's 's Disease (31).
We carry out ablation experiments validate the following settings of MoIRL-MGPT:
1. The optimal number of agents for a fixed total batch size,
2. Whether the loss term that encourages agents to search in different directions (ED) really works,
3. Whether the experience replay (ER) really works,
4. Whether the decreasing schedule σ (DS) really works,
5. Whether the possible technique of similarity penalization (SP) works +
The base algorithm is denoted MoIRL-MGPT, which consists of 4 agents and incorporates the modules ED, ER, DS, without SP. For both GSK33 and JNK3 tasks, we present the average and standard deviation of the mean scores and internal diversities of the 10p-100 high-scoring molecules generated by each strategy.
Besides ablation experiments, we also compare the performance of MoIRL-MGPT with several baselines on GSK33, JNK3 and QED maximization tasks. The baselines are Graph GA (28 Reinvent 39 VAE 30 and GFlowNet [5). More details are provided in the Appendix.
Tadica/ that the algorithm demonstrates the best performance with four agents, and additional agents do not improve the results. Encouraging agents to explore different paths in the chemical
Penalize on the score of molecule if its similarity to one of previously generated candidates is larger than 0.8.

---

Table 4: Results of experiments on GSK333 and JNK3 maximization.
GSK33 top- 100
JNK3 top-100
mean score
IntDiv
mean score
IntDiv
agent
1,000 0.000
0
0.954
0.343
agents
1,000 : 0.000
0.017
0.012
0 2
MoIRL-MGPT
1.000 = 0.000
0.005
0.022 0
8 agents
1.000
0.000
0.369 0.055
w/o ED
1.000
0.285
0.000
0.025
ER
0.964 0.005
0.019
0.008
0.025
w/o DS
0.997 =: 0.001
0.016
0.370 0.014
w/ SP
1,000 + 0.000
0.0021
0.956 0.009
0.365 0.015
GelowNet
0.649 = 0.072
0.237 0.115
0.116
GraphGA
0.919 + 0.016
0.024
0.875 1
0.380
JT-VAE
0.235 = 0.083
0.770
- 0.159 0.040
0.781 0.127
Reinvent
0.965 # 0.011
0,308 + 0.035
0,942 + 0.019
0,368 # 0.021
Table 5: Results of experiments on QED maximization.
QED top-I00 100
mean score
IntDiv
MoIRL-MGPT
0,948 0.000
0.862 0.004
GFlowNet
0.938 : 0.001
0.809 : 0.017
GraphGA
0.928 = 0.001
0.845 = 0.005
JT-VAE
0.921 0.003
0.856 0.012
Reinvent
0.948 0.000
0.658 0.035
space does promote diversity of the generated molecules, and using experience replay and decreasing schedule of o benefits mean scores of candidates. Additionally, the similarity penalization trick does not appear to be effective in MoIRL-MGPT. As consequence, we have verified the effectiveness of our design.
As shown in Table and 5 compared with baselines with competitive mean scores, MoIRL-MGPT performs better in internal diversity.
Conclusion and discussion
In this paper, we present MoIRL-MGPT, a multi-agent reinforcement learning framework for de novo drug molecular design, which adopts transformer models as agents and the fundamental idea is to encourage agents to collaborate to search with different directions in the chemical space. MolRL- MGPT demonstrates superior performance on the GuacaMol benchmark and does well in designing inhibitors against SARS-CoV-2 protein targets.
Admittedly, there exist some possible approaches for further improvements to our algorithm:
Better data source: As we all know, data sources play a decisive role in the performance of MI algorithms in chemistry and biology. Higher quality pre-training data or more annotated data tailored to specific tasks may further improve the performance of MoIRI-MGPT.
The design of the scoring function may also improve the algorithm's performance. For example, in multi-property joint tasks, adjusting the coefficients of terms in the scoring function may be beneficial, and more accurate and fast software for predicting molecular properties also helpful.
Scoring function:
Insights for specific objectives: In practical drug development, with more specialized and in-depth research on each objective, we should utilize the knowledge specific to certain tasks more fully design candidate drug molecules better.
In summary, MoIRL-MGPT is promising and feasible approach for de novo drug design. It provides pharmaceutical researchers with fast and effective method to generate diverse set of molecular structures that meet specified conditions, as long as the scoring functions of these conditions are provided. We believe that MoIPL-MGPT will be of great assistance in drug discovery.

---

References
[1] Sungsoo Ahn, Junsu Kim, Hankook Lee, and Jinwoo Shin. Guiding deep molecular optimization with genetic exploration. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, H. Lin, editors, Advances in Newral Information Processing Systems, volume 33, pages 12008-12021. Curran Associates, Inc., 2020.
[2] Amr Alhossary, Stephanus Danicl Handoko, Yuguang Mu, and Chee- Keong Kwoh. Fast, accurate, and reliable molecular docking with quickvina 2. Bioinformatics, 31(13):2214 (113))2221-22221 221611 2015.
[3] Josep Arús-Pous, Simon Viet Johansson, Oleksii Prykhodkoo Eshen Jannik Bjerrum, Christian Tyrchan, Jean-Louis Reymond, Hongming Chen, and Ola Engkvist. Randomized smiles strings improve the quality of molecular generative models. Journal of cheminformatics, 11(1):1-13, 2019.
[4] Viraj Bagal, Rishal Aggarwal, P. K. Vinod, and U. Deva Priyakumar. Molgpt: Molecular generation using transformer-dccoder model. Journal of Chemical Information and Modeling, 62(9):2064 2076, 2022.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non- -oe-rttrteive diverse candidate generation. Advances in Neural Information Processing Systems, 34:27381-273999 2021.
[6] Mostapha Benhenda. Can ai reproduce observed chemical diversity? bioRxiv, page 292177, 2018.
[7] G Richard Bickerton, Gaia V Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90-98, 2012.
[8] Thomas Blaschke, Josep Arús-Pous, Hongming Chen, Christian Margreitter, Christian Tyrchan, Ola Engkvist, Kostas Papadopoulos, and Atanas Patronov. Reinvent 2.0: An ai tool for de novo drug design. Journal of chemical information and modeling, 2020.
[9] Thomas Blaschke, Ola Engkvist, Jürgen Bajorath, and Hongming Chen. Memory- reinforcement learning for diverse molecular de novo design. Journal chemical information and modeling, 2020.
[10] Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling. 59(3):1096 10000996-100,, 1108, 2019.
[11] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[12] Philip Michael Dean and Richard Lewis. Molecular diversity drug design. Springer, 1999.
[13] Jianyuan Deng, Zhiho Yang, Iwao Ojima, Dimitris Samaras, and Fusheng Wang. Artificial intelligence in drug discovery: applications and techniques. Briefings in Bioinformatics, 23(1 (31):hhbab430, 2022.
[14] Wei Du and Shifei Ding. A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications. Artificial Intelligence Review, 54:3215-3253 2021.
[15] Yuanqi Du, Tianfan Fu, Jimeng Sun, and Shengchao Liu. Molgensurvey: A systematic survey in machine learning models for molecule design. arXiv preprint arXiv:2203. 140000000 2022.
[16] Peter Eckmann, Kunyang Sun, Bo Zhao, Mudong Feng. Michael K Gilson, and Rose Yu. Limo: Latent inceptionism for targeted molecule generation. In International Conference on Machine Learning, PMLR, 2022.
[17] Peter Ert and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. Journal of cheminformatics, 1(1):1-11, 2009.
[18] Tianfan Fu, Wenhao Gao, Connor W Coley, and Jimeng Sun. Reinforced genetic algorithm for structure- based drug design. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems NeurlPS, 2022.
[19] Tianfan Fu, Cao Xiao, Lucas M Glass, and Jimeng Sun. Moler: incorporate molecule-level reward enhance deep generative model for molecule optimization. IEEE transactions on knowledge and data engineering, 34(11):5444 (1:555595--441 2021.
[20] Niklas Gebauer, Michael Gastegger, and Kristof Schütt, Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules. Advances in neural information processing systems, 32, 2019.
10

---

[21] Zijie Geng, Shufang Xie, Yingee Xia, Lijun Wu, Tao Qin, Jie Wang, Yongdong Zhang, Feng Wu, and Tie- Yan Liu. De novo molecular generation via connection-aware motif mining. In International Conference on Learning Representations, 2023.
[22] Rafael Bombazz-o Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato Benjamín Baachezz-Lngelinge Dennis Sheberla, Jorge caaalera-parraguinnee Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules. ACS centrul science, 4(2):268-276) 2018.
[23] Gabriel I Lima Guimaraes, Benjamin Sanchez-I. .engeelin..g Carlos Outeiral, Pedro Luis Cunha Farias, and Alán Aspuru-Guzik, Objective-reinforeed generative adversarial networks (organ) for sequence generation models. arXiv preprint 1.XXiv:1705,1045533 2017.
[24] Jiazhen He, Eva Nittingrr, Christian Tyrchan, Werngard Czechtizky, Atanas Patronov, Eshen Jannik Bjerrum, and Ola Engkvist. Transformer-hasel molecular optimization beyond matched molecular pairs. Journal of cheminformatics, 14(1):18, 2022.
[25] Hauke S Hillen, Goran Kokic, Lueas Farnung, Christian Dienemann, Dimitry Tegunov, and Patrick Cramer. Structure replicating sars-cov-2 polymerase. Nature, 584(7819): 2020.
[26] Ross Irwin, Spyridon Dimitriadis, Jiazhen He, and Esben Jannik Bjerrum. Chemformer: a pre-trained transformer for computational chemistry. Machine Learning: Science and Technology. 311:005502, 2022.
[27] Kevin Maik Jablonka, Philippe Schwaller, and Berend Smit. Is gpt-3 all you need for machine learning for chemistry? In Al for Accelerated Materials Design NeurlPS 2022 Workshop, 2022.
[28] Jan H Jensen. A graph-based genetic algorithm and generative model/monte carlo tree search for the exploration of chemical space. Chemical science, 10 10(12)33577-3557, 2019.
[29] Wengong Jin, Regina Barzilay, and T. Jaakkola. Multi-objective molecule generation using interpretable substructures. In International Conference on Machine Learning, page 4849-4855 PMLR, 2020.
[30] Wengong Jin. Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference on machine learning, pages 2323-2332. PMLR, 2018.
[31] Yibo Li, Liangren Zhang, and Zhenming Liu. Multi-ohjective de novo drug design with conditional graph generative model. Journal of cheminformatics, 10(1):1-24, 2018.
[32] Long-Ji Lin. Self-improving reactive agents based reinforcement learning, planning and teaching. Machine learning, 833:299333,,, 1992.
[33] Youzhi Luo and Shuiwang Ji. An autorerressive flow model for 3d molecular geometry generation from scratch. In International Conference Learning Representations (ICLR), 2022.
[34] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. Advances in Neural Information Processing Systems. 32, 2019.
[35] Krzysztof Maziarz, Henry Jackson-Flux, Pashmina Cameron, Finton Sirockin, Nadine Schneider, Nikolaus Stiefl, Marwin Segler, and Marc Brockschmidt. Learning to extend molecular scaffolds with structural motifs. In International Conference on Learning Representations (ICLR), 2022.
[36] David Mendez, Anna Gaulton, A Patrícia Bento, Jon Chambers, Marleen De Veij, Eloy Félix, María Paula Magariños,, Juan F Mosquera, Prudence Mutowo, Michal Nowotka, et al. Chembl: towards direct deposition of bioassay data. Nucleic acids research, 47(D1)) 2019.
[37] Henry Moss, David Leslic, Daniel Beck, Javier Gonzalez, and Paul Rayson. Boss: Bayesian optimization over string spaces. Advances in neural information processing systems, 33:15476-15486, 2020.
[38] Varnavas Mouchlis, Antreas Afantitis, Angela Serra, Michele Fratello, Anastasios G Papadiamantis. Vassilis Aidinis, Iscult Lynch, Dario Greco, and Georgia Melagraki. Advances in de novo drug design: From conventional to machine learning methods. International journal of molecular sciences, 22(4): 2021.
[39] Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. Journal of Cheminformatics, 9, 2017.
[40] Jerzy Osipiuk, Saara-Ama Azizi, Steve Dvorkin, Michael Endres, Robert Jedrzejczak, Krysten A Jones, Soowon Kang, Rahul S Kathayat, Youngchang Kim, Vladislav G Lisnyak, et al. Structure of papain- like protease from cor-- and its complexes with non- covalent inhibitors. Nature communications. 12(1):743, 2021.
11

---

[41] Tiago Pereira, Maryam Abbasi, Bernardete Ribeiro, and Joel P Arrais, Diversity oriented deep reinforce- ment learning for targeted molecule generation. Journal of cheminformatics, 13(1):1-17, 2022...
[42] Pavel G Polishchuk, Timur I Madzhidov, and Alexandre Vamee. Estimation of the size of drug-like chemical space based on gdb-17 17 data. Journal of computer-aided molecular design. 27(8):675-679, 2013.
[43] Damiil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-I Lengeling. Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, ct al. Molec- ular sets (moses); benchmarking platform for molecular generation models. Frontiers in pharmacology. 11:565644, 2020.
[44] Hao Qian, Cheng Lin, Dengwei Zhao, Shikui Tu, and Lei Xu. Alphadrug: protein target specific novo molecular generation. PNAS Nexus, 144):gac222, 2022.
[45] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[46] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188)9, 2019.
[47] David Rogers and Mathew Hahn. Extennedd-comeectiitty fingerprints. Journal of Chemical Information and Modeling, 50(5):44--54, 2010.
[48] David M Rogers, Rupesh Agarwal, Josh V Vermaas, Micholas Dean Smith, Rajitha Rajeshwar, Connor Cooper, Ada Sedova, Swen Boehm, Matthew Baker, Jens Glaser, et al. Sars-cov2 billion-compound docking. Scientific Data, 10(1):173, 2023.
[49] Marwin HS Segler, Thierry Kogej, Christian Tyrchan, and Mark P Waller. Generating focused molecule lihraries for drug discovery with recurrent neural networks. ACS central science, 4(1):120-131, 2018.
[50] Marwin HS Segler, Mike Preuss, and Mark P Waller. Planning chemical syntheses with deep neural networks and symbolic ai. Nature, 555(7698))6004 610, 2018.
[51] Gregor Simm, Robert Pinsler, and José Miguel Hernández Lobato. Reinforcement learning for molecular design guided by quantum mechanics. In International Conference on Machine Learning, pages 8959-89666 PMLR, 2020.
[52] Gregor N. C. Simm, Robert Pinsler, Gábor Csányi, and José Miguel Ilernández-Lobato. Symmetry-aware actor-critic for 3d molecular design. In 9th International Conference on Learning Reprexentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[53] Teague Sterling and John J. Irwin. Zine 1,5 ligand discovery for everyone. Journal of chemical information and modeling, 15(11):2324 2332222222332222222222 2015.
[54] Jiangming Sun, Nina Jeliazkova, Vladimir Chupakhin, Jose-Felipe Golib-Dzib, Ola Engkvist, Lars Carlsson, Jörg Wegner, Hugo Ceulemans, Ivan Georgiev, Vedrin Jeliazkov, et al. Excape-db: integrated large scale dataset facilitating big data analysis in chemogenomics. Journal of cheminformaties, 9(1):1-9, 2017.
[55] T. T. Tanimoto, An elementary mathematical theory of classification and prediction. IBM Internal Report, 1958.
[56] Oleg Trott and Arthur J Olson. Autodock vina: improving the speed and accuracy of docking with a new scoring function, efficient optimization, and multithreading. Journal of computational chemistry, 31(2):455-46) 461, 2010.
[57] Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[58] Hanchen Wang, Tianfan Fu, Yuangi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak. Shengehao Liu, Peter Van Katwyk, Andreca Deac, et al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972)44--60, 2023.
[59] Jike Wang, Chang Yu Hsich. Mingyang Wang, Xiaorui Wang. Zhenxing Wu, Dejun Jiang. Benben Liao, Xujun Zhang, Bo Yang. Qiaojun He, et al. transformer, knowledge distillation and reinforcement learning. Nature Machine Intelligence, 3(10)9914 922, 2021.
molecular generation based on conditional
12

---

[60] Mingyang Wang, Chang-Yo Yu Hsieh, Jike Wang, Dong Wang, Gaoqi Weng. Chao Shen. Xiaojun Yao. Zhitong Bing. Honglin Li. Dongsheng Cao, al. Relation: A deep generative model for structure-based de novo drug design. Journal of Medicinal Chemistry, 65(13))9478-94 -99922 2022.
[61] Kehan Wu, Yingce Xia, Yang Fan, Pan Deng, Haiguang Liu, Lijun Wu, Shufang Xie, Tong Wang. Tao Qin, Yan Liu. Tailoring molecules for protein pockets: a transformer-bast based generative solution for structured- based drug design. arXiv preprint arXiv:2209 06158, 2022.
[62] Yutong Xie, Chence Shi, Zhou, Yuwei Yang. Weinan Zhang, Yong Yu, and Lei Li. Mars: Markov molecular sampling for multi-objectivv drug discovery. In International Conference on Learning Represen- tations, 2021.
[63] Soojung Yang, Doyeong Hwang, Seul Lee, Seongok Ryu, and Sung Ju Hwang. Hit and lead discovery with explorative rl and fragment- based molecule generation. Advances in Neural Information Processing Systems, 34:7924-7999 7936, 2021.
[64] Naruki Yoshikawa, Kei Terayama. Masato Sumita, Teruki Homma, Kenta Oono, and Koji Tsuda. Population-based de novo molecule generation, using grammatical evolution. Chemistry Letters, 47(11):1431-1444, 2018.
[65] Jiaxuan You. Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network goal-directed molecular graph generation. Advances neural information processing systems, 31, 2018.
[66] Linlin Zhao, Heather L. Ciallella, Lauren M. Aleksuncs, and Hao Zhu. Advancing computer-aiee aided drug discovery (cadd) by big data and data data-driven machine learning modeling. Drug Discovery Today, 25())1122-11638, 2020.
[67] Shuangjia Zheng, Youhai Tan, Zhenyu Wang, Chengtan Li, Zhiqing Zhang, Xu Sang, Hongming Chen, and Yuedong Yang. Accelerated rational protac design via deep learning and molecular simulations. Nature Machine Intelligence, 4(9):739-748, 2022.
[68] Zhenpeng Zhou, Steven M. Kearnes, Li Li, Richard N. Zare, and Patrick F. Riley. Optimization of molecules via deep reinforcement learning. Scientific Reports, 9, 2019.
13

---

A
GuacaMol benchmark
Table 6: More results of the experiments on the GuacaMol benchmark.
Graph MCTS
MolRL-
Tasks
dataset
GFlowNet
MGPT
1. Celecoxib rediscovery
0.505
0.355
0,409
1.000 + 0.000
Troglitazone rediscovery
0.419
0.311
0.211
1.000 0.000
3. Thiothixene rediscovery
0.456
0.311
0,342
1.000 + 0.000
Aripiprazole similarity
0.595
0.380
0,586
1.000 0.000
5. Albuterol similarity
0.719
0.749
0,458
1.000 0.000
6. Mestranol similarity
0.629
0.402
0.396
1.000 0.000
7. H24
0.684
0,410
0.535
1.000 0.000
8. 1o
0.747
0.631
0.224
0.939 0.003
9. Median molecules 1
0.334
0.225
0.218
0.447 0.006
10. Median molecules
0.351
0.170
0.195
0.423 0.004
11. Osimertinib MPO
0.839
0.784
0.792
0.977 0.001
12. Fexofenadine MPO
0.817
0.695
0.715
1.000 + 0.000
13. Ranolazine MPO
0.792
0.616
0.680
0.939 0.000
14. Perindopril MPO
0.575
0.385
0.459
0.809 + 0.005
15. Amlodipine MPO
0.696
0.533
0,430
0.906 0.001
16. Sitagliptin MPO
0.509
0,458
0.042
0.822 + 0.003
17. Zaleplon MPO
0.547
0.488
0.072
0.790 0.008
18. Valsartan SMARTS
0.259
0.040
0.000
0.997 + 0.000
19. deco hop
0.933
0.590
0.587
1.000 0.000
20. scaffold hop
0.738
0.478
0.475
1.000 = 0.000
Total
12.144
9.009
7,826
18.049
0.003
Designing inhibitors against SARS-CoV-2 targets
Table 7: The mean scores and internal diversity of the top- 100 drug candidates against the Pl.Pro_7JJJ 7JJR target generated by MoIRL-MGPT and other baselines.
Methods
Docking score (!)
QED score (1)
SA score ())
IntDiv
JT- VAE
-8.76 0.35
0.795 # 0.038
2.994 0.140
0.836 + 0.032
Get
-9.11 0.21
0.015
2.823 + 0.076
0.825 0.010
GraphGA
-10.83
0,380 0.013
3.638 + 0.162
Reinvent
-10.75 0.05
0.392 # 0.008
2.649 0.035
0,619 = 0.023
0.023
MoIRL-MGPT
-11.02 0.06
0.386 + 0.006
2.550 + 0.047
0.745 + 0.008
Table 8: The mean scores and internal diversity of the top-100 drug candidates against the RdRp_6Y 6YYT target generated by MoIRL-MGPT and other baselines.
Methods JT-VAE
Docking score ({)
QED score (t)
SA score (L)
IntDiv
-8.33 = 0.25
0.71920.019
2.959 + 0.094
0.828 + 0.018
GFlowNet
-8.89 0.16
0.656 + 0.033
2,854 0.061
0.770 + 0.015
GraphGA
-11.26 0.12
0.262 + 0.010
3.520 + 0.049
0.658 + 0.009
Reinvent
-11.30
0.275 + 0.006
2.917 0.035
0.616+0002)
MoIRL-MGPT
-11.84 0.07
0.278 # 0.005
2.894 0.072
0.670 # 0.013
14


Reviews:
[{"review_id": "rCOLpCOlXO", "reviewer": "Reviewer_xd9X", "summary": "This paper proposes a novel multi-agent reinforcement learning algorithm with agents parameterized with a pre-trained GPT architecture for de novo drug design.  The authors propose a modified objective function with an intrinsic reward inspired bonus to encourage diversity between agents and also propose to use a constraint to keep the fine-tuned agents close to the pre-trained agents.  The authors evaluate their algorithm on Guacamol, by generating a number of inhibitors for SARS-CoV2 targets.  They also perform ablations on their method with the GSK3$\\beta$ and JNK3 maximization tasks.", "questions": "1. In the loss function, why does the sum over agents only go up to $k-1$?  Shouldn't the second loss term be something like $\\sum_{j=1}^n s(x) \\left|P_k(x) - P_j(x)\\right|$?  Why compare to only the agents before this index as the ordering seems arbitrary.\n2. Did the authors consider using a method for encouraging diversity which already exists in the multi-agent RL literature?  If so, why did they not use it and did they run any comparisons?", "limitations": "The main and most important limitation of this work is that I do not know whether the proposed method actually is competitive with rival methods due to some missing experiments, baselines, and insufficient reproducibility.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Some strengths of this paper are:\n\n- An intrinsic reward like term added to the agents' loss which encourages diversity.\n- The method performs favorably to the other methods compared against on Guacamol benchmark.\n- It performs comprehensive ablations on the GSK3$\\beta$ and JNK3 maximization tasks", "weaknesses": "My main complaint about this paper is that I am not convinced of the algorithm's superiority over rivaling methods based upon the experiments section.  I feel that the paper has both missed some necessary baseline methods and that the experiments as they stand are insufficient in demonstrating the paper's main claim that its method leads to improved diversity over other methods.  I also have concerns about missing related work in multi-agent RL in which there is already a body of literature on encouraging diversity among agents, as well as other competing methods which have been applied to molecular drug generation such as diffusion models and GFlowNets.  I also have concerns regarding this paper's reproducibility.  I will go over these concerns one by one.\n\n## Experiments section\n1. The main claim of this paper is that its approach leads to superior diversity, but I did not see any experiments _comparing_ the diversity of molecules it generates to molecules generated by other methods.  Indeed, there were experiments looking at the diversity of molecules generated by their method, but it was only their method.  There isn't an indication whether their main claim of improved diversity is true if there is no comparison to other methods.\n2. While the generated molecules do look diverse to my non-chemist eye, I would like to see more generated molecules and critically some measurement of diversity of the generated molecules to be convinced.  Also, these results were on only one seed which is insufficient.  At minimum there should be three seeds, ideally quite a bit more (see [here](https://ai.googleblog.com/2021/11/rliable-towards-reliable-evaluation.html)).\n3. There are missing baselines in the experiments.  The authors should have compared to an existing LLM molecule generation method such as MolGPT, but this was missing from their experiments.  There are other methods for encouraging diversity for molecule design with RL-inspired machinery such as GFlowNets (https://arxiv.org/abs/2106.04399) that should be compared against.  Also, it would have been nice to see a comparison with one of the recent works on diffusion for molecular design (e.g., https://arxiv.org/pdf/2203.17003.pdf, https://arxiv.org/pdf/2305.01140.pdf), or at minimum a compelling reason for why not to compare to these methods.\n4. In the Guacamol experiments it seems experiments were run over one seed.  This is insufficient.\n5. A more minor comment: it's hard to understand the significance of the Guacamol task when the tasks are ordered 1-20 without context of what the tasks actually are.\n\n## Missing references\n1. I mentioned in the last point, but it would have been nice to see some discussion of other methods such as GFlowNets or diffusion models which also try to encourage diversity in molecular design.\n2. There is already a body of literature on encouraging diversity in multi-agent RL, but I did not see references to this literature.  Some representative papers may be https://arxiv.org/abs/2106.02195 and https://openreview.net/forum?id=H-6iczs__Ro.\n\n## Reproducibility\n1. All experiments seem to have been run with one seed.  There is no way to know if the results would hold with more seeds.\n2. There is no listing of the hyperparameters used or the hyperparameter tuning methods used (or values tried if a grid search).\n3. There is no (anonymized) submitted code available to verify or reproduce the authors' claims.\n\n## Clarity issues\n1. In the section explaining the loss function, the indexing used is rather confusing.  E.g., the authors use a loss $L_1$ which seems fixed, then also a loss $L_k$ which seems to index the different agents (so what about when $k=1$?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable review comments! We address your main concerns below, and we sincerely hope that you will reconsider and upgrade your rating:\n\n**Q1 (diversity comparison, in *experiments 1, 2*)**\n\n**A1**: To further demonstrate the advantages in diversity of our approach, we have added several baselines to the experiments on GSK3$\\beta$ and JNK3 maximization (**Table 2 in pdf**)  for comparison. Additionally, we supplement a experiment on QED maximization (**Table 3 in pdf**) with diversity measurements. The results of these experiments indicate that our approach is indeed effective in promoting the molecular diversity in drug design.\n\nWe did not provide diversity measurements in experiments on the GuacaMol benchmark, where each task produces a single score that comprehensively represents the performance of a given method (following the official guidelines). Therefore, the advantages of our approach in terms of diversity have been inherently contained in our higher scores, while reporting molecular diversity in GuacaMol is unreasonable.\n\n**Q2 (missing baselines, in *experiments 3 / missing references 1*)**\n\n**A2**: \n\n1. We have listed the related LLM-based methods of molecular generation in our paper (line 92), but notably none of them are designed for *de novo* drug design tasks. It should be especially discriminated that *de novo* drug design aims to generate molecules with properties beyond the property distribution of the training set, while previous LLM-based methods cannot achieve this  (https://arxiv.org/abs/2203.14500). Specifically, MolGPT, mentioned by the reviewer, is a pre-trained model aiming to learn the existing datasets and generate molecules within the property distribution. Other LLM-based approaches, including ChemFormer, also do not target *de novo* drug design, so we did not include them in our experiments.\n2. You also mention some diffusion-based methods for molecular generation, but at the moment diffusion model works only in 3D molecular generation, which mainly target the structure and quantum properties. In contrast, 1D/2D molecular generation (our aim) focuses on the biochemical properties, which is a different direction of research than 3D molecular generation (https://arxiv.org/abs/2203.14500). Hence, we did not include diffusion models in our experiments.\n3. GFlowNet is a baseline of *de novo* drug design, and we didn't include it in the original paper mainly because of its relatively poor performance. As supplementary, we have included its results in **Table 1, 2, 3 (in pdf)**.\n\n**Q3 (multiple seeds & Guacamol tasks, in *experiments 2, 4, 5 / reproducibility 1*)**: \n\n**A3**: First, we would like to emphasize that in section 4.3, we report the standard deviations across different seeds. In the GuacaMol benchmark experiment, we have also launched multiple runs to verify the robustness of the algorithm. However, due to the high stability of our algorithm, and most of the previous works on GuacaMol did not report standard deviations, so we did not include them in the original paper.\n\nHowever, to address any reservations, we have included the standard deviation values for our algorithm's performance on the GuacaMol task across 5 different seeds in **Table 1 (in pdf)**. Additionally, we have listed the names of the 20 GuacaMol tasks, the specifics of which can be found in the GuacaMol paper.\n\n**Q4 (MARL references, in *missing references 2/questions 2*)**\n\n**A4**: Although, as you've mentioned, previous literature in cooperative multi-agent reinforcement learning has presented some approaches to encourage diversity. However, they can hardly be directly applied to *de novo* drug design because:\n\n1. Typically, the objectives of these techniques are enhancing the behavioral diversity of agents, leading to a collection of diverse agents at the end of the RL process. In contrast, our aim centers on enhancing the diversity of the objects (molecules in our context) that the agents are tasked with searching.\n2. These methods are typically designed for well-defined virtual spaces (like games) or real-world 3D spaces, leveraging specific space characteristics such as trajectories, which are difficult to define and utilize in the chemical space.\n\n**Q5 (in *reproducibility 2, 3*)**\n\n**A5**: Our codes are available at https://anonymous.4open.science/r/MolRL-MGPT-835E. We explain your question in detail in the author rebuttal.\n\n**Q6 (in *clarity*)**\n\n**A6**: We have accurately formulated the loss function in section 3.2. Specifically, Eq. (3) defines the loss function for the first agent, which is also the first term of the loss functions for other agents. Eq. (4) defines the loss functions for all agents, and it's noteworthy that when $k=1$, Eq. (4) simplifies to Eq. (3).\n\n**Q7 (*questions 1*)**\n\n**A7**: For the design of loss functions, a seemingly natural approach is to reward the differences between all agents equally. However, this method exhibits relatively low robustness in experiments, that is, agents are likely to greatly lose the validity of generated SMILES due to great mutual interference. In contrast, our approach (Eq. (4)) organizes agents in a sequence, and each agent is only rewarded for the difference between it and the agents preceding it, which enables agents to get promoted more stably."}, {"review_id": "lgleXOOdzO", "reviewer": "Reviewer_4Wce", "summary": "This paper creates a multi-agent reinforcement learning approach to promote diversity in the search space of small molecules during molecular optimization.  Because of the complicated nature of early stage drug discovery research, the diversity is useful during early work in the identification and validation of small molecules, however previous RL methods did not manage to address diversity in a satisfactory way.  The algorithm is rather simple and employs pre-trained GPT agents that speak the language of molecular SMILES. Importantly, the algorithm manages to outperform previous state of the art methods on a number of public benchmarks.  \n", "questions": "What is the radius and the bitlength in the features in Eq(2).  Is chirality included?\n\nThis may be a limitation of the notation used in the paper and perhaps the authors already tried to do the correct thing in the code (I didn't check), but it appears that equation (4) does not use a standardized SMILES representation of the molecule, so it is possible that the agents will try to learn to produce different textual representations of the same molecule (i.e. each agent could canonicalize in a different way).  Of course the magnitude of such an effect probably depends on the details of the training protocol and on the amount of augmentation with randomized SMILES in the pre-training protocol.  Would it be possible to instead use the canonical and certain randomly sampled representations of the same molecule as the inputs x to equation (4)?  Could the authors test their final agents for their preference towards different textual representations of the same molecules?\n\nThe statement at the end of page 7 is very unclear: did the authors test the listed small molecules in tables 2 and 3 against the SARS CoV-2 protease and polymerase in a laboratory setting and show that they don't work?  If so, it would be very useful to provide a brief description of these experiments in a supplemental material or cite a separate publication after anonymization is no longer a concern.  At a minimum, they should clarify the language at the end of that section.\n\nRegarding performance, did the authors monitor the GPU usage during training and do they achieve the maximal power draw on the GPU during the optimization experiments?\n", "limitations": "This work has no negative societal impacts.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The use of multiple agents in this small molecule optimization setting is original and proves to be very useful.  The explicit diversification by equation (4) ends up creating a stronger model than previous attempts that used RL or other deep learning methodologies, and importantly, it outperforms (even if only barely) the graph Genetic Algorithm, a significantly simpler algorithm that puts other methods to shame in several benchmarks.  The paper demonstrates the increase in diversity of the designs with the increasing number of agents.  The paper is written in a clear fashion, and the main result it significant.  The authors provide code.\n", "weaknesses": "A minor weakness of this method is that the performance of the code is somewhat slow, which is somewhat understandable given the slow individual query of the GPT models and the challenges of the additional RL-related operations.  I agree with the overall attitude of the authors that this particular concern is not very important.  The performance of the graphGA algorithm is nearly identical to that of the new method, despite its extreme simplicity and efficiency.  On the other hand the lack of pre-training is a limitation of graphGA, at least in principle.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your approval and valuable review comments! We address your main concerns below:\n\n**Q1 (in *weaknesses*)**: A minor weakness of this method is that the performance of the code is somewhat slow, which is somewhat understandable given the slow individual query of the GPT models and the challenges of the additional RL-related operations. \n\n**A1**: While our algorithm does require a considerable amount of time to execute, the primary time consumption does not stem from the inference and updating of GPT agents. Instead, the bulk of the time is dedicated to the computation of oracles on CPUs. For instance, in a single docking task for a specific target, the entire reinforcement learning process demands around 100 hours when executed on a single NVIDIA A100 GPU alongside 64 CPU cores, during which more than 95% of the time is cost by the docking process itself, even though parallel computing of Quick Vina software is implemented. Therefore, the issue of slow drug design isn't unique to our algorithm; other methods also struggle with low efficiency when dealing with time-intensive oracles like docking. Furthermore, as you rightly mentioned, the runtime of our algorithm remains significantly shorter when compared to the years that conventional drug discovery processes typically span.\n\n**Q2 (in *weaknesses*)**: The performance of the GraphGA algorithm is nearly identical to that of the new method, despite its extreme simplicity and efficiency. On the other hand the lack of pre-training is a limitation of graphGA, at least in principle.\n\n**A2**: Initially, it's important to acknowledge that although GraphGA does not leverage the deep learning techniques that have gained prominence in recent times, it has been proven to be a competitive approach for molecular design. It ranks second in a comprehensive benchmark test, trailing only behind another SMILES-based reinforcement learning method  (https://openreview.net/forum?id=yCZRdI0Y7G). Nevertheless, as you mentioned, this approach may have approached the zenith of performance attainable through non-deep methods. This implies that for intricate molecular generation tasks, such as docking, GraphGA's performance is likely to be suboptimal, because it cannot effectively utilize the chemical information embedded within molecular datasets through pre-training paradigm.\n\n**Q3 (in *questions*)**: What is the radius and the bitlength in the features in Eq(2). Is chirality included?\n\n**A3**: For the implementation of internal diversity, we adopt the function in the widely-used Therapeutics Data Commons (TDC) package (https://github.com/mims-harvard/TDC/blob/6af2a41679a0699446ad627be8051504548e86fa/tdc/chem_utils/evaluator.py#L99C31-L99C31). Specifically, in the Morgan fingerprints (ECFPs) the radius is 2, the bitlength is 2048, and chirality is not included.\n\n**Q4 (in *questions*)**: It appears that equation (4) does not use a standardized SMILES representation of the molecule, so it is possible that the agents will try to learn to produce different textual representations of the same molecule. Would it be possible to instead use the canonical and certain randomly sampled representations of the same molecule as the inputs x to equation (4)? Could the authors test their final agents for their preference towards different textual representations of the same molecules? \n\n**A4**: We refrain from employing the canonicalized SMILES representation while computing the generation probability in equations (3) and (4). This choice arises from the fact that GPT agents inherently grasp the grammar of SMILES strings rather than focusing on molecular structures. The probability using an equivalent yet non-originally generated SMILES string lacks a direct meaning in reinforcement learning optimization. As a result, it is indeed possible for the agents to produce different SMILES of the same molecule.\n\nNevertheless, when computing property scores, the variability in SMILES representation does not influence the outcome, since oracles take as input either the molecule items in RDKit or molecular fingerprints, both of which uniquely correspond to the molecular structure. Furthermore, in the process of updating the molecular memory, canonicalized SMILES are employed. This ensures that a given molecule does not exist in the memory in multiple forms, preventing duplicate entries.\n\n**Q5 (in *questions*)**: The statement at the end of page 7 is very unclear: did the authors test the listed small molecules in tables 2 and 3 against the SARS CoV-2 protease and polymerase in a laboratory setting and show that they don't work?\n\n**A5**: We sincerely regret any confusion caused. Our approach lacked wet experiments to validate the molecular properties outlined in tables 2 and 3 due to the absence of experimental conditions. The values presented in the tables are the result of in silico oracles. The intention of our statement at the end of page 7 is to emphasize that:\n\nAs the Quantitative Estimate of Drug-Likeness (QED) is calculated through a comparison with the distribution of compounds in the existing drug database – one that lacks molecules designed for SAR-COV-2 – it's important to note that a low QED score doesn't necessarily imply inefficacy against the SAR-COV-2 targets. On the contrary, molecules exhibiting effectiveness might well lie beyond the boundaries of the existing drug compound distribution.\n\n**Q6 (in *questions*)**: Regarding performance, did the authors monitor the GPU usage during training and do they achieve the maximal power draw on the GPU during the optimization experiments?\n\n**A6**: As we explained in **A1**, the predominant portion of our algorithm's execution time is cost by the oracles running on the CPU, despite utilizing 64 CPU cores. Consequently, the GPU operated below its maximal capacity throughout the reinforcement learning process and is barely operating most of the time in the docking tasks."}, {"review_id": "R8sRuqkJK7", "reviewer": "Reviewer_AzpA", "summary": "This paper proposes a method named MolRL-MGPT for drug molecular generation.\nConcretely, GPT-based agents are used to iteratively generate candidate compounds, and a special reward signal is adpoted to encourage agents to explore in diverse directions.\nThe experiments on GuacaMol benchmark show the superiority of the method.\n", "questions": "1.\tThough there are some transformer-based models for chemical language. There still exists a gap between molecules structure and natural language. How the model will perform when the GPT is replaced with graph network?\n2.\tHow to determine the suitable hyperparameters in real-world applications?\n", "limitations": "No potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1.\tAn effective method for De novo Drug Design is proposed.\n2.\tA special reward signal is proposed to promote the diversity of the agents.\n", "weaknesses": "It is not appropriate to call the proposed method a “RL-based” method though the score function can be treated as the “reward”. RL follows Markov decision process and aims to maximize the accumulated return, but it seems that the goal in this paper is to maximize the final score. Meanwhile, there is no RL objective function in this paper. (I suggest to change the RL-related statement to another, and this will not affect my rating.)\n1.\tEq.3 aims to make each agent get a fix improvement which neglect the difficulty of different generation timesteps.\n2.\tEq.4 forcing the different agents obtain different scores, but different score is not equal to the different explore direction.\n3.\tIt seems that the agents are treated as independent units without sharing their experience. Since the different agents cooperate with each other and aim to achieve a common goal, different kinds of correlation need to be considered.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable review comments! We address your main concerns below:\n\n**Q1 (in *weaknesses*)**: It is not appropriate to call the proposed method a “RL-based” method though the score function can be treated as the “reward”. RL follows Markov decision process and aims to maximize the accumulated return, but it seems that the goal in this paper is to maximize the final score. Meanwhile, there is no RL objective function in this paper.\n\n**A1**: I concur with your opinion that our algorithm diverges somewhat from the conventional understanding of \"reinforcement learning\". However, we employ this terminology primarily because, in the context of molecular design for computer-aided drug discovery, \"reinforcement learning\" serves as the widely recognized term for denoting a category of multi-step generation algorithms featuring reward functions (https://www.sciencedirect.com/science/article/pii/S0928098722002093), and our algorithm falls within the scope of this methodological category. Moreover, in section 2 (related works), we allocate a paragraph to provide an overview of \"RL-based drug design algorithms\", which contains various previous works in this category.\n\n**Q2 (in *weaknesses*)**: 1. Eq.3 aims to make each agent get a fix improvement which neglect the difficulty of different generation timesteps. 2. Eq.4 forcing the different agents obtain different scores, but different score is not equal to the different explore direction. 3. It seems that the agents are treated as independent units without sharing their experience. Since the different agents cooperate with each other and aim to achieve a common goal, different kinds of correlation need to be considered.\n\n**A2**: It seems that you are a little confused about our algorithm design, so I will clarify certain aspects of our design in further detail.\n\n1. Eq.3 is the loss function of the first agent in the RL process. Its purpose is to encourage the agent to learn the characteristics of high-scoring molecules as far as possible without deviating too much from the prior parameter. At each RL step, each term of the loss function is recalculated, so the improvement of the agent is not fixed. In addition, during the RL process, it does tend to be progressively more challenging for the agent to improve the molecular score, but this does not contradict the objective of the loss function to improve the molecular score.\n2. Eq.4 is not intended to force different agents to generate molecules with different scores, but rather to motivate all agents to comprehend the characteristics of high-scoring molecules while simultaneously exploring the chemical space in diverse directions. We accomplish the second objective by rewarding the difference in the probability of generating the same SMILES by different agents, which is similar to the idea in Eq.3 of guaranteeing the effectiveness of agents by punishing the difference in the probability of generating the same SMILES for agents and the prior model.\n3. Indeed, we do not treat agents as independent units in our algorithm. This is evident in Eq.4 where we explicitly reward the differences between agents. You also mention some ideas in the design of cooperative multi-agent reinforcement learning algorithms, but they might not be directly applicable to the specific framework of our algorithm design.\n\n**Q3 (in *questions*)**: Though there are some transformer-based models for chemical language, there still exists a gap between molecules structure and natural language. How the model will perform when the GPT is replaced with graph network?\n\n**A3**: This question poses an important inquiry into the effectiveness of distinct molecular representations.  Notably, the Simplified Molecular Input Line Entry System (SMILES) string we employ encapsulates identical structural information as the two-dimensional molecular graph. Hence, the potential of SMILES-based molecular design algorithms parallels that of their 2D molecular graph counterparts.\n\nFurthermore, recent research has demonstrated the superior performance of SMILES-based reinforcement learning algorithms compared to graph-based models in molecular design tasks (https://openreview.net/forum?id=yCZRdI0Y7G). This outcome underscores our confidence that our algorithm using GPT to generate SMILES strings competes robustly with graph-based approaches.\n\nIt is worth acknowledging that both SMILES strings and molecular graphs inherently sacrifice spatial information in comparison to actual three-dimensional molecules. Nonetheless, the existing algorithms for generating *de novo* molecules in 3D have not reached a stage where they can supplant the effectiveness of string-based and graph-based methods. I hold the view that should a significantly advanced 3D generation model emerge, its integration to replace the GPT agent in our algorithm would hold substantial promise as a future direction.\n\n**Q4 (in *questions*)**: How to determine the suitable hyperparameters in real-world applications?\n\n**A4**: In the RL process of MolRL-MGPT, the pivotal hyper-parameters include: the number of agents, the number of RL steps, learning rate, $\\sigma_1$, $\\sigma_2$, the number of samples for experience replay, etc. Among them, our experiments in section 4.3 suggest that 4 is the best number of agents. Regarding the number of RL steps, it's advisable to extend this count as far as practicable, until the agents show no significant progress for many steps. Other hyper-parameters do need to be tuned specifically in each design task (by monitoring the changing curves of the agents throughout the RL process), where the learning rate and $\\sigma_2$ are particularly sensitive. For our experiments, the setting of hyper-parameters can be found in the paper and codes."}]

Author rebuttal:
We would like to express our sincere appreciation to all the reviewers for your valuable feedback on our paper, and we have responded to all your questions (in the corresponding rebuttal sections). We also add some supplementary experimental results in the **pdf**, including more baselines on the GuacaMol benchmark, more baselines on GSK3$\beta$ and JNK3 maximization, new experiments on QED maximization and other details. We promise to present these contents in the final version of the paper.

In addition, we would like to explain the issue raised by reviewer xd9X that **our codes are not available**:

Indeed, in the abstract of my paper, I have included an Anonymous GitHub link to the MolRL-MGPT codebase (https://anonymous.4open.science/r/MolRL-MGPT-835E).   Regrettably, it appears that during your review, the Anonymous GitHub service encountered an interruption, obstructing your reproducing our experiments and potentially tarnishing your impression.   We sincerely apologize for this inconvenience and want to clarify that we do not consider it to be attributable to our actions.   Now the Anonymous Github has been restored, and the timestamp verifies that we initially committed the code in May.  We earnestly beseech you to reevaluate the code, as it contains many details of our experiments, including hyper-parameter settings, etc.

Meta-review: This paper proposes MolRL-MGPT, a new RL algorithm for de-novo drug design.

The authors had a very productive discussion with the reviewers during the rebuttal period and all the reviewers raised their scores. Specifically, reviewer xd9X changed his mind about the paper after the discussion and new experimental results.

I recommend an accept and encourage the authors to incorporate all the clarifications and additional results to the camera-ready version of the paper.
Decision: Accept (poster)