SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We run the RL process for 1000 iterations on each target with 4 GPT agents, and the batch size of each agent is 128. The whole process for one target takes approximately 100 hours on a single NVIDIA A100 GPU and 64 CPU cores. Details of the generated candidates are as follows.
PLPro_7JIR target PLPro (papain-like protease) is an attractive target for SARS-CoV-2 since it plays a fundamental role in cleavage and maturation of viral polyproteins, assembly of the replicase-transcriptase complex, and disruption of host responses. 7JIR is a C111S mutant form of the structure of PLPro [40]. Three candidate inhibitors against the PLPro_7JIR target generated by MolRL-MGPT are shown in Table 2.
(1)  $\mathrm{O} = \mathrm{C}(\mathrm{c}1\mathrm{cccc}(-\mathrm{c}2\mathrm{ccc}3\mathrm{c}(-\mathrm{c}4\mathrm{nc}5\mathrm{cc}(\mathrm{F})\mathrm{c}(\mathrm{F})\mathrm{cc}5[\mathrm{nH}]4)\mathrm{n}[\mathrm{nH}]\mathrm{c}3\mathrm{c}2)\mathrm{c}1)\mathrm{c}1\mathrm{cccc}1;}$
(2) c1(-c2cc(-c3cc4c(cc3)snn4)ccc2)ccc(C(NCc2cc(C=O)Nc3nn[nH]n3)ccc2F)=O)cc1;
(3) c1c(-c2c(C)ccc(C==O)c3cc4cccc4[nH]3)c2)nc(-c2ccnc(-c3cccc(-c4cccc4)c3)c2)cc1.
Table 2: Candidate inhibitors against the PLPro_7JIR target generated by MolRL-MGPT. The SMILES of the three candidates are:
Molecule
docking score (↓)	-11.3	-11.1	-11.2
QED score (↑)	0.310	0.258	0.214
SA score (↓)	2.530	2.729	2.549
<sup>3</sup>https://www.rcsb.org/structure/7JIR
<sup>4</sup>https://www.rcsb.org/structure/6YYT
$^{5}$ Docking consumes more than  $95\%$  of the time, although parallel computing of Quick Vina software is implemented.
RdRp_6YYT target RdRp (RNA-dependent RNA polymerase) works for the replication of genome and the transcription of genes of SARS-CoV-2, and 6YYT is the PDB identification code of its structure [25]. Three candidate inhibitors against the RdRp_6YYT target generated by MolRL-MGPT are shown in Table 3.
(1)  $\mathrm{O = C1CCN(c2nc(-c3cccc(C (= O)N4CCN(c5cc(-c6cccccc6)nc6cccccc56)CC4)c3)cc3cccccc23)CCN1}$  (2)  $\mathrm{O = C1CCN(c2nc(-c3cccc(C (= O)N4CCN(c5cc(-c6nc7cccccc7c (= O)[nH]6)c6cccccc6n5)CC4)c3)}$  nc3cccccc23)CCN1;
(3) Cc1ccc(-c2cc(N3CCN(C=O)c4cccc(-c5nc(N6CCNC(=O)CC6)c6cccccc6n5)c4)CC3)c3cccccc 3n2)cc1.
Table 3: Candidate inhibitors against the RdRp_6YYT target generated by MolRL-MGPT. The SMILES of the three candidates are:
Molecule
docking score (↓)	-12.3	-13.1	-13.2
QED score (↑)	0.237	0.253	0.241
SA score (↓)	2.772	3.104	2.806
The generated candidates against the PLPro_7JIR and RdRp_6YYT targets exhibit desirable binding affinities (docking scores) and synthetic accessibility. Although the drug-likeness scores of these candidates may not be high, it is reasonable since QED is estimated based on the distribution of existing drug molecules. All these drugs are ineffective in inhibiting the two targets of SARS-CoV-2; thus, designing new dissimilar compounds becomes necessary.
4.3 GSK3 $\beta$ , JNK3 and QED maximization
To demonstrate the effectiveness of our design, we perform ablation experiments on commonly used oracles for simulating real-world drug design with low consumption:  $\mathrm{GSK3}\beta$  and JNK3. Their scores are estimated by random forests trained on ExCAPE-DB dataset [54], which measure the bioactivities of molecules against the Glycogen synthase kinase 3 beta target (GSK3 $\beta$ ) and the c-Jun N-terminal kinase 3 target (JNK3). Previous research has shown that inhibiting these targets can benefit the treatment of Alzheimer's Disease [31].
We carry out ablation experiments to validate the following settings of MolRL-MGPT:
1. The optimal number of agents for a fixed total batch size,
2. Whether the loss term that encourages agents to search in different directions (ED) really works,
3. Whether the experience replay (ER) really works,
4. Whether the decreasing schedule of  $\sigma_{1}$  (DS) really works,
5. Whether the possible technique of similarity penalization (SP) works  $^{6}$ .
The base algorithm is denoted as MolRL-MGPT, which consists of 4 agents and incorporates the modules ED, ER, DS, without SP. For both the GSK3 $\beta$  and JNK3 tasks, we present the average and standard deviation of the mean scores and internal diversities of the top-100 high-scoring molecules generated by each strategy.
Besides ablation experiments, we also compare the performance of MolRL-MGPT with several baselines on GSK3 $\beta$ , JNK3 and QED maximization tasks. The baselines are Graph GA [28], Reinvent [39], JT-VAE [30] and GFlowNet [5]. More details are provided in the Appendix.
Table 4 indicates that the algorithm demonstrates the best performance with four agents, and additional agents do not improve the results. Encouraging agents to explore different paths in the chemical
6Penalize on the score of a molecule if its similarity to one of previously generated candidates is larger than 0.8.
Table 4: Results of experiments on GSK3 $\beta$  and JNK3 maximization.
GSK3β top-100	JNK3 top-100
mean score	IntDiv	mean score	IntDiv
1 agent	1.000 ± 0.000	0.318 ± 0.020	0.954 ± 0.012	0.343 ± 0.017
2 agents	1.000 ± 0.000	0.335 ± 0.017	0.960 ± 0.012	0.357 ± 0.028
MolRL-MGPT	1.000 ± 0.000	0.362 ± 0.015	0.961 ± 0.010	0.372 ± 0.025
8 agents	1.000 ± 0.000	0.360 ± 0.020	0.958 ± 0.015	0.369 ± 0.018
w/o ED	1.000 ± 0.000	0.285 ± 0.023	0.961 ± 0.008	0.345 ± 0.025
w/o ER	0.964 ± 0.005	0.332 ± 0.019	0.918 ± 0.008	0.356 ± 0.023
w/o DS	0.997 ± 0.001	0.358 ± 0.016	0.940 ± 0.014	0.370 ± 0.027
w/ SP	1.000 ± 0.000	0.360 ± 0.021	0.956 ± 0.009	0.365 ± 0.015
GFlowNet	0.649 ± 0.072	0.715 ± 0.104	0.437 ± 0.219	0.716 ± 0.145
GraphGA	0.919 ± 0.016	0.365 ± 0.024	0.875 ± 0.025	0.380 ± 0.015
JT-VAE	0.235 ± 0.083	0.770 ± 0.067	0.159 ± 0.040	0.781 ± 0.127
Reinvent	0.965 ± 0.011	0.308 ± 0.035	0.942 ± 0.019	0.368 ± 0.021
Table 5: Results of experiments on QED maximization.
QED top-100
mean score	IntDiv
MolRL-MGPT	0.948 ± 0.000	0.862 ± 0.004
GFlowNet	0.938 ± 0.001	0.809 ± 0.017
GraphGA	0.928 ± 0.001	0.845 ± 0.005
JT-VAE	0.921 ± 0.003	0.856 ± 0.012
Reinvent	0.948 ± 0.000	0.658 ± 0.035
space does promote diversity of the generated molecules, and using experience replay and decreasing schedule of  $\sigma_{1}$  benefits mean scores of candidates. Additionally, the similarity penalization trick does not appear to be effective in MolRL-MGPT. As a consequence, we have verified the effectiveness of our design.
As shown in Table 4 and 5, compared with baselines with competitive mean scores, MolRL-MGPT performs better in internal diversity.
In this paper, we present MolRL-MGPT, a multi-agent reinforcement learning framework for de novo drug molecular design, which adopts transformer models as agents and the fundamental idea is to encourage agents to collaborate to search with different directions in the chemical space. MolRL-MGPT demonstrates superior performance on the GuacaMol benchmark and does well in designing inhibitors against SARS-CoV-2 protein targets.
Admittedly, there exist some possible approaches for further improvements to our algorithm:
- Better data source: As we all know, data sources play a decisive role in the performance of ML algorithms in chemistry and biology. Higher quality pre-training data or more annotated data tailored to specific tasks may further improve the performance of MolRL-MGPT.
- Scoring function: The design of the scoring function may also improve the algorithm's performance. For example, in multi-property joint tasks, adjusting the coefficients of terms in the scoring function may be beneficial, and more accurate and fast software for predicting molecular properties is also helpful.
- Insights for specific objectives: In practical drug development, with more specialized and in-depth research on each objective, we should utilize the knowledge specific to certain tasks more fully to design candidate drug molecules better.
In summary, MolRL-MGPT is a promising and feasible approach for de novo drug design. It provides pharmaceutical researchers with a fast and effective method to generate a diverse set of molecular structures that meet specified conditions, as long as the scoring functions of these conditions are provided. We believe that MolRL-MGPT will be of great assistance in drug discovery.
[60] Mingyang Wang, Chang-Yu Hsieh, Jike Wang, Dong Wang, Gaoqi Weng, Chao Shen, Xiaojun Yao, Zhitong Bing, Honglin Li, Dongsheng Cao, et al. Relation: A deep generative model for structure-based de novo drug design. Journal of Medicinal Chemistry, 65(13):9478-9492, 2022.
[61] Kehan Wu, Yingce Xia, Yang Fan, Pan Deng, Haiguang Liu, Lijun Wu, Shufang Xie, Tong Wang, Tao Qin, and Tie-Yan Liu. Tailoring molecules for protein pockets: a transformer-based generative solution for structured-based drug design. arXiv preprint arXiv:2209.06158, 2022.
[62] Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. Mars: Markov molecular sampling for multi-objective drug discovery. In International Conference on Learning Representations, 2021.
[63] Soojung Yang, Doyeong Hwang, Seul Lee, Seongok Ryu, and Sung Ju Hwang. Hit and lead discovery with explorative rl and fragment-based molecule generation. Advances in Neural Information Processing Systems, 34:7924-7936, 2021.
[64] Naruki Yoshikawa, Kei Terayama, Masato Sumita, Teruki Homma, Kenta Oono, and Koji Tsuda. Population-based de novo molecule generation, using grammatical evolution. Chemistry Letters, 47(11):1431-1434, 2018.
[65] Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. Advances in neural information processing systems, 31, 2018.
[66] Linlin Zhao, Heather L. Ciallella, Lauren M. Aleksunes, and Hao Zhu. Advancing computer-aided drug discovery (cadd) by big data and data-driven machine learning modeling. *Drug Discovery Today*, 25(9):1624–1638, 2020.
[67] Shuangjia Zheng, Youhai Tan, Zhenyu Wang, Chengtao Li, Zhiqing Zhang, Xu Sang, Hongming Chen, and Yuedong Yang. Accelerated rational protac design via deep learning and molecular simulations. Nature Machine Intelligence, 4(9):739-748, 2022.
[68] Zhenpeng Zhou, Steven M. Kearnes, Li Li, Richard N. Zare, and Patrick F. Riley. Optimization of molecules via deep reinforcement learning. *Scientific Reports*, 9, 2019.
A GuacaMol benchmark
Table 6: More results of the experiments on the GuacaMol benchmark.
Tasks	dataset	Graph MCTS	GFlowNet	MolRL-MGPT
1. Celecoxib rediscovery	0.505	0.355	0.409	1.000 ± 0.000
2. Troglitazone rediscovery	0.419	0.311	0.211	1.000 ± 0.000
3. Thiothixene rediscovery	0.456	0.311	0.342	1.000 ± 0.000
4. Aripiprazole similarity	0.595	0.380	0.586	1.000 ± 0.000
5. Albuterol similarity	0.719	0.749	0.458	1.000 ± 0.000
6. Mestranol similarity	0.629	0.402	0.396	1.000 ± 0.000
7. C11H24	0.684	0.410	0.535	1.000 ± 0.000
8. C9H10N2O2PF2Cl	0.747	0.631	0.224	0.939 ± 0.003
9. Median molecules 1	0.334	0.225	0.218	0.447 ± 0.006
10. Median molecules 2	0.351	0.170	0.195	0.423 ± 0.004
11. Osimertinib MPO	0.839	0.784	0.792	0.977 ± 0.001
12. Fexofenadine MPO	0.817	0.695	0.715	1.000 ± 0.000
13. Ranolazine MPO	0.792	0.616	0.680	0.939 ± 0.000
14. Perindopril MPO	0.575	0.385	0.459	0.809 ± 0.005
15. Amlodipine MPO	0.696	0.533	0.430	0.906 ± 0.001
16. Sitagliptin MPO	0.509	0.458	0.042	0.822 ± 0.003
17. Zaleplon MPO	0.547	0.488	0.072	0.790 ± 0.008
18. Valsartan SMARTS	0.259	0.040	0.000	0.997 ± 0.000
19. deco hop	0.933	0.590	0.587	1.000 ± 0.000
20. scaffold hop	0.738	0.478	0.475	1.000 ± 0.000
Total	12.144	9.009	7.826	18.049 ± 0.003
B Designing inhibitors against SARS-CoV-2 targets
Table 7: The mean scores and internal diversity of the top-100 drug candidates against the PLPro_7JIR target generated by MolRL-MGPT and other baselines.
Methods	Docking score (↓)	QED score (↑)	SA score (↓)	IntDiv
JT-VAE	-8.76 ± 0.35	0.795 ± 0.038	2.994 ± 0.140	0.836 ± 0.032
GFlowNet	-9.11 ± 0.21	0.726 ± 0.015	2.823 ± 0.076	0.825 ± 0.010
GraphGA	-10.83 ± 0.08	0.380 ± 0.013	3.638 ± 0.162	0.740 ± 0.017
Reinvent	-10.75 ± 0.05	0.392 ± 0.008	2.649 ± 0.035	0.619 ± 0.023
MolRL-MGPT	-11.02 ± 0.06	0.386 ± 0.006	2.550 ± 0.047	0.745 ± 0.008
Table 8: The mean scores and internal diversity of the top-100 drug candidates against the RdRp_6YYT target generated by MolRL-MGPT and other baselines.
Methods	Docking score (↓)	QED score (↑)	SA score (↓)	IntDiv
JT-VAE	-8.33 ± 0.25	0.719 ± 0.019	2.959 ± 0.094	0.828 ± 0.018
GFlowNet	-8.89 ± 0.16	0.656 ± 0.033	2.854 ± 0.061	0.770 ± 0.015
GraphGA	-11.26 ± 0.12	0.262 ± 0.010	3.520 ± 0.049	0.658 ± 0.009
Reinvent	-11.30 ± 0.04	0.275 ± 0.006	2.917 ± 0.035	0.616 ± 0.021
MolRL-MGPT	-11.84 ± 0.07	0.278 ± 0.005	2.894 ± 0.072	0.670 ± 0.013