3DMolFormer: A Dual-channel Framework for Structure-based Drug Discovery
Authors: Xiuyuan Hu
Guoqing Liu
Can Chen
Yang Zhao
Hao Zhang
Xue Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Structure-based Drug Discovery
Protein-ligand Docking
3D Molecule Generation
Transformer
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Structure-based drug discovery, encompassing the tasks of protein-ligand docking and pocket-aware 3D drug design, represents a core challenge in drug discovery. However, no existing work can deal with both tasks to effectively leverage the duality between them, and current methods for each task are hindered by challenges in modeling 3D information and the limitations of available data. To address these issues, we propose 3DMolFormer, a unified dual-channel transformer-based framework applicable to both docking and 3D drug design tasks, which exploits their duality by utilizing docking functionalities within the drug design process. Specifically, we represent 3D pocket-ligand complexes using parallel sequences of discrete tokens and continuous numbers, and we design a corresponding dual-channel transformer model to handle this format, thereby overcoming the challenges of 3D information modeling. Additionally, we alleviate data limitations through large-scale pre-training on a mixed dataset, followed by supervised and reinforcement learning fine-tuning techniques respectively tailored for the two tasks. Experimental results demonstrate that 3DMolFormer outperforms previous approaches in both protein-ligand docking and pocket-aware 3D drug design, highlighting its promising application in structure-based drug discovery. The code is available at: https://github.com/HXYfighter/3DMolFormer.

Published as a conference paper at ICLR 2025
3DMOLFORMER: A DUAL-CHANNEL FRAMEWORK FOR STRUCTURE-BASED DRUG DISCOVERY
Xiuyuan Hu', Guoqing Liu Can (Sam) Chen Yang Zhao', Hao Zhang'; Xue Liu 'Department of Electronic Engineering, Tsinghua University, "Microsoft Research Al for Science McGiil University, "Mila Quebec Institute
edu. cn, can chenemineaia quebec, zhao wwng@tsingnaaaaa.........aa edu, en, haozhang@ edu. xieliuecs .tm.:is..mmmll..cc.
wwry22emails.tsiime
ABSTRACT
Structure-baas drug discovery, encompassing the tasks protein-ligand docking and pocket-aware 3D drug design, represents core challenge in drug discovery. However, no existing work can deal with both tasks to effectively leverage the duality between them, and current methods for each task are hindered by chal- lenges in modeling 3D information and the limitations of available data. To ad- dress these issues, we propose MoMolForrrr, unified dual-channel transformer- based framework applicable to both docking and 3D drug design tasks, which exploits their duality by utilizing docking functionalities within the drug design process. Specifically, we represent 3D pocket-ligand complexes using parallel se- quences of discrete tokens and continuous numbers, and we design a correspond- ing dual-channel transformer model to handle this format, thereby overcoming the challenges of 3D information modeling. Additionally, we alleviate data limi- tations through large-scale pre-training mixed dataset, followed by supervised and reinforcement learning fine-tuning techniques respectively tailored for the two tasks. Experimental results demonstrate that 3DMolFormer outperforms previ- ous approaches in both protein-ligand docking and pocket-aware 3D drug design, highlighting its promising application in structure- based drug discovery. The code is available at: https: http///ttuuuettt wtt..ccoo/..ooo CoomHXXYfightt /tom://Niiigggter///111Po1P 30wr://MMMoMooooo
INTRODUCTION
In recent years, the application of machine learning in drug discovery has gained significant trac- tion (Mak et al.] 2023). achieving substantial advancements in tasks such as molecular property prediction (Zhang al.
2021 Wieder al. 2020), protein structure predic- Baek et al. 2021 1 Lin et al. (002)). and drug molecular design (Olivecronal al. 2017]] Luo et 2021 Fu al.] 2022 Du et al.) 2022 2024 These developments hold the
2021 Wang et al.
tion (Jumper et al. 2021
promise of dramatically enhancing the efficiency of drug development processes Ollanco-Gonzall
al. 2023). Notably, the transformer architecture, which has seen breakthroughs in natural lan- guage processing (NLP) (Devlin et al. 2019 Brown et al. 2020), has been successfully adapted for molecular representation learning (Zhou al. [003a]] Gao al. 2024a), protein- protein-ligand interaction prediction (Zhao et al.) 2022 Abramson et al. 2024). and molecular generation tasks (Bagal et al.)
2021
al. 2023).
Structure-based drug discovery (SBDD) is one of the most critical strategies in drug discovery prac- tices, relying on theories of drug-receptor interactions to study the complexes formed between pro- tein pockets and small molecule ligands (Van Montfort & Workman 2017). SBDD encompasses two core tasks: (1) protein-ligand binding pose prediction (docking), which involves predicting the 3D binding conformation of ligand given the 3D structure of protein and the 2D representation of the ligand (Yang et al. 2022). and (2) pocket-aware 3D drug design, which entails designing 3D drug molecules that bind well (with low binding energy) to a given pocket target protein
"Corresponding authors: Hao Zhang (1st), Guoqing Liu.

---

Published as a conference paper at ICLR 2025
structure (Zang et al., 2023b; Isert et al. 2023b). These two tasks are inherently dual, and one is predictive, while the other is generative.
However, as of now, the application of machine learning in these two SBDD tasks remains widely recognized as a challenge (Pala & Clark, 2024). The accuracy and generalization of protein-ligand docking methods are still unsatisfactory (Morehead et al., 2024), and pocket-aware 3D drug design approaches have not achieved obvious improvements by explicitly utilizing 3D structural informa- tion compared to 2D methods (Zheng et al. 2024). This predicament can be attributed three primary reasons:
Underutilized duality: Protein-ligand docking and pocket-aware 3D drug design are naturally dual tasks, and improvements in docking performance could directly benefit drug design. How- ever, since these two tasks are different in type (predictive vs. generative), this duality has unfor- tunately not been leveraged by previous machine learning approaches.
Challenges in modeling 3D information: Modeling 3D information is key difficulty in SBDD, as protein sequences and small molecule graphs contain only discrete information, whereas 3D coordinates are continuous values. Merging these two modalities of information has proven chal- lenging (Zhu et al. 2022).
Limited data: Ground-truth data on protein-ligand complexes are scarce. Currently, the largest dataset, PDBbind (Liu et al. 2017), contains fewer than 20,000 complexes, which is insufficient for training a robust machine learning model.
To address these challenges, propose 3DMolFormer, unified wransformer-basee framework for both of the two SBDD tasks. First, to fulfill the input-output causal relationships essential for both docking and 3D drug design, introduce a parallel sequence format to represent 3D complex of a protein pocket and a small molecule ligand, as shown in Figure and which comprises a token sequence for discrete protein atoms and small molecule SMILES, alongside a numerical sequence for 3D coordinates. Subsequently, we construct the 3DMolFormer model based on this parallel sequence, as illustrated in Figure 3] augmenting the GPT architecture (Radford et al. (2019) with a numerical head corresponding to the token head, enabling the model to be directly applied for autoregressive generation of the parallel sequences.
Due to data limitations, we utilize a "pre-training fine-tuning" approach (Quinn et al. 2019) in NLP for 3DMolFormer, as large-scale pre-training helps mitigate these data challenges. During the pre-training phase, the model undergoes large-batch training (Keskar et al. 2017) on a large- scale mixed dataset, which includes data on protein pockets, ligands, and pocket-ligand complexes. A composite loss function is employed for autoregressive training of the parallel sequence, where cross-cntropy loss applics to the token sequence and mean squared error loss applies to the nu- the
merical sequence. For the protein-ligand docking task, we perform supervised fine-tuning ground-truth pocket-ligand complexes, using the mean squared error of the numerical sequences corresponding to the ligand's 3D coordinates the loss function. Moreover, to utilize the duality between the two SBDD tasks, for the pocket-aware drug design task, we apply a regularized max- imum likelihood estimation loss for reinforcement learning fine-tuning, and leverage the weights fine-tuned for docking to generate the 3D coordinates of the small molecules.
Experimental results for protein-ligand docking demonstrate that 3DMolFormer outperforms all search-based and deep-lcarning docking baselines in binding pose prediction accuracy, particu- larly showing reduction in samples with large prediction errors. Results for pocket-aware 3D drug design indicate that through a carefully designed composite reward function, 3DMolFormer can generate drug candidates that meet satisfactory levels of binding affinity (docking score), drug- likeness, and synthesizabilityy during the reinforcement learning process, in particular significantly surpassing existing state-of-the -fth---rt-- baselines in terms of binding affinity and success rates in meet- ing multi-objective criteria. These results reflect the outstanding performance of the 3DMolFormer framework structure- based drug discovery.
In summary, main contributions include:
We propose 3DMolFormer, the first unified framework applicable to both grotein-ligamt docking and pocket-aware 3D drug design.
2

---

Published as a conference paper at ICLR 2025
one amino acid
3D coordinates
Token Sequence
PS
C
0
N
PE
PCS
PCE
Numerical Sequence
1.0
1.0
1.0
10
1.0
1.0
1.0
0.35
1.72
0.48
1.39
Figure 1: The parallel sequence of protein pocket with 3D coordinates.
3D coordinates
Token Sequence
LCS
y
X
LCE
Numerical Sequence
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
-4.73
1.60
5.88
-1.26
2.18
6.93
Figure 2: The parallel sequence of small molecule ligand with 3D coordinates.
We design parallel sequence format for pocket-ligand complexes and establish a dual-channel transformer architecture autoregressively generate this format, effectively addressing the chal- lenges of modeling 3D information in SBDD.
Through large-scale pre-training and respective fine-tuning, 3DMolFormer outperforms various previous baselines in both SBDD tasks.
RELATED WORKS
Molecular Pre-training The success of large-scale pre-training has extended from NLP to the field of drug discovery (Xia et al. [023 Chen et al. (023). Many studies focus on molecular rep- resentation learning, which maps molecular structures to informative embeddings for downstream predictive tasks Yang et al.) (2021a): Fang et al. (2022) Gao et al. (2024a). Several representa- tion learning methods for protein-ligand binding have been proposed, including InteractionGraphi Net (Jiang et al. T02T) and BindNet (Feng et al. 2024a). with Uni-Mol (Zhou et al. 2023a) col- lecting and pre-trap training on extensive 3D datasets of proteins and small molecules, achieving high accuracy in protein-ligand
docking. Furthermore, models such MolGPT (Bagal al.) (021), Chemformer (Irwin et al. 2022), and BindGPT (Zholus et al., 2024) utilize pre-training to enhance molecular distribution learning, enabling applications in generative tasks.
Protein-ligand Docking Protein-ligand docking encompasses three sequential tasks: binding site prediction, binding pose prediction, and binding affinity prediction, with binding pose predic- tion being the most critical in structure-based drug discovery (Zhang al., (023b). Traditional search-based methods typically employ combinatorial optimization techniques to identify the best binding poses (known targeted docking) within given protein pocket, using tools such as AutoDock4 (Morris et al. 2009), AutoDock Vina (Trott & Olson 2010) Eberhardt et al.. (021)) and Smina (Koes al. 2013), which widely used in practical virtual screening. Recently, deep learning approaches have been introduced for this task, exemplified by DeepDock (Méndez-Lucio et al.] (021) and Uni-Mol (Zhou et al.) 2023a). Additionally, various deep learning techniques for blind docking have emerged, which simultaneously predict binding sites and poses (Stärk et al.) 2022 Lu al., 2022 Zhang et al. 2023a Pei al., 2024 Corso al., 2023 2024). However, blind docking methods are primarily hindered by inaccuracies in binding site prediction, making direct comparisons with targeted docking methods less meaningful. Moreover, some end-to-ent approaches that predict binding affinity without 3D poses fail to provide the crucial structural infor- mation required in SBDD (Wang et 2024a).
Pocket-aware 3D Drug Design Drug design is the ultimate goal of molecular design. Cur- rently, most machine learning methods focus on generating ID SMILES strings or 2D molecular graphs (Segler et al. 2018. Eekmann et al. 2022 Lee et al. 2024). with reinforcement learning

---

Published as a conference paper at ICLR 2025
Token Sequence
Numerical Sequence
Docking Input
Docking Output
Token Mode
Numerical Mode
T Tolenn Embedding
Pocket toms
X, y, Z,
Ligand SMILES ES
X, y, Z,
1.0, 1.0, 1.0
Pocket 3D Coords
1.0, 1.0, .. 1.0
Ligand 3D Coords
Drug Design Input
Drug Design Output
Positional Embedding
Supervised Fine-tuning for Docking:
Nx
Transformer
Layers
17K ground truth Poc-Lig Complexes
Pre-training
Token Head
Number
3.2M Pockets
Head
3DM ormorrer
+ 209M Ligands
167K docked
Predicted
Predicted Number
Logits
Poc- Lig Complexes
RL Fine-tuning for Drug Design
Predicted
Token
Figure 3: Overview of 3DMolFormer. The left shows the dual-channel model architecture, the top right illustrates the input and output of the two SBDD tasks in a parallel sequence, and the bottom right outlines the pre-training and fine-tuning process.
being popular paradigm (Olvvecrona al., 2017 You al. 2018 Ahn al. 2020 Jin al., 2020 Simm et al., 2020 Yang et al.) 2021b). b.. However, these approaches can only output dis- crete information about atoms and chemical bonds, lacking the capability to generate 3D coordi- nate values, thus limiting their application in SBDD. In contrast, pocket-aware 3D drug design explicitly utilizes the 3D structures protein targets to generate de novo small molecules with high binding affinity. Various machine learning techniques have been applied to pocket-aware 3D drug design, including genetic algorithms (e.g.. AutoGrow (Spiegel & Durrant (0200). variational au- toencoders (e.g.. liGAN (Ragoza et al. 2022)), autoregressive models (e.g.. AR (Luo et al. (021), Pocker2Mol (Peng et al. 2022). Lingo (Feng et al., (024b)), and flow models (GraphBP (Liu al. ))22)). Recently, diffusion models have achieved state-of oh-the-an art performance in this task, cluding Di (Schneuing et al. 2022). TirgetDiff (Guan et al.) (003a)). and DecompDiff (Guan al. 2023b). Notably, some studies have developed canformmr--bssed 3D drug design models. The XYZ-transformer (Flam-Shepherd & Aspuru-Guzii 2023) directly uses 3D coordinate values (re- taining three decimal places) as tokens, while BindGPT (Zholus et al. (2024) decomposes the integer and decimal parts of coordinates into two tokens to reduce vocabulary size. Token-Mol (Wang et al.) 2024b), on the other hand, employs torsion angles of small molecules instead of coordinate values to shorten sequence lengths. However, these methods represent values using discrete tokens, which disrupts the continuity coordinates.
3DMOLFORMER
3.1 FORMAT POCKET AND LIGAND SEQUENCES WITH 3D COORDINATES
To leverage causal language model for handling 3D protein pockets and small molecules while explicitly separating discrete structural information from continuous spatial coordinates, we design parallel sequence format. This format consists of discrete token sequence Stok and continuous numerical sequence Snum- both of which share the same length and align element-wise The to- ken sequence consists of tokens in predefined vocabulary, while the numerical sequence contains floating-poin values.
As shown in Figure the sequence for a protein pocket goe consists of two parts: the first poe represents an atomic list, and the second spoc_cnord contains 3D coordinate information. The atomic list is encoded in the token sequence, which includes all atoms in the protein pocket except for
4

---

Published as a conference paper at ICLR 2025
hydrogen atoms. Aside from alpha carbon atoms, denoted as 'CA', other atoms represented by their element type, such as 'C', 'O', 'N', and 'S' The sequence of atoms follows the order of the pdb file, where each amino acid begins with ['N', 'CA', 'C', 'O'J followed by the side-chain atoms. The normalized 3D coordinates for each atom in the atomic list are included in the numerical sequence in the same order, with each dimension ('x', "y", "z') occupying a separate position. The length of the 3D coordinate sequence is always three times the length of the atomic list. Moreover, in the token sequence, the start and end of the atomic list are marked by 'PS' and 'PE', while the 3D coordinates are delineated by 'PCS' and 'PCE' at the start and end, respectively. In the numerical sequence, numbers that do not correspond to 3D coordinates are padded with 1.0.
As illustrated in Figure 2 the sequence for small molecule s li is similar to that of the protein pocket, comprising both SMILES string section gllg.smilee and a 3D coordinate section glog.coord After atom-level tokenization (Schwaller et al., 2019), the SMILES string of the small molecule encoded in the token sequence, excluding hydrogen atoms. It is important to note that some tokens may not correspond to atoms, and thus, no 3D coordinates will be associated with them. The normalized 3D coordinates for each atom in the tokenized SMILES string are included in the numerical sequence, with each coordinate dimension ('x', 'y', 'z') occupying a separate position. The length of the 3D coordinate sequence is always three times the number of atoms in the small molecule. In the token sequence, the start and end of the SMILES tokens are marked by 'LS'' and 'LE', while the 3D coordinates of the corresponding atoms are marked by 'LCS' and 'LCE' at the start and end, respectively. the numerical sequence, numbers not corresponding to 3D coordinates are similarly padded with 1.0.
When the sequence of a protein pocket is concatenated with that of small molecule ligand, it forms a pocket-ligand complex sequence along with their 3D coordinates spoc-lig, This sequence format offers three advantages:
It fully encapsulates the structural and 3D coordinate information of both the protein pocket and the small molecule ligand.
Discrete structural information and continuous numerical data are separated into two parallel quences, enabling independent processing of each data type.
The sequence of the pocket-ligand complex maintains causal logic. As depicted in the upper
right of Figure 3 this sequence structure allows autoregressive prediction, which can effectively represent both pocket-ligand docking and pocket-aware drug design tasks.
Specifically, we normalize the coordinates of all pocket-ligand complexes by translating their center of mass to the origin (0, O, 0). Additionally, to ensure numerical stability during training (Quinn) ct al. 2019). we scale the coordinate values by factor q 1 reduce the range of their distribution:
(4/s 1/,
Ze
Re.
q
q
where (rz, U/s 2)) is the original coordinate of the i-th atom, (Ie: yes 2e) is the coordinate of the center of mass, and (a% v/. 2/) refers to the normalized values used in the numerical sequence.
3.2 MODEL ARCHITECTURE
To process the aforementioned parallel sequences, we require an autoregressive language model that can simultaneously take discrete token sequence and a continuous floating-point sequence as input, while predicting both the next token and the next numerical value. Inspired by x Val (Golkar et al.) 2023). we propose dual-channel transformer architecture for 3DMolFormer, as illustrated in the left part of Figure The module handling the token sequence is based on the GPT-2 model (Radford jet al., 2019), featuring identical token embeddings, positional embeddings, multiple transformer layers, and a prediction head for logits. On top of this, we introduce parallel numerical channel at both the input and output stages.
At the input stage, we multiply the embedding of each token in the token sequence with the corre- sponding value in the numerical sequence, using this product as the input to the positional embed- ding. This is why numerical values that lack meaningful information are padded with 1.0. At the output stage, in parallel with the token prediction head, we add a number head to predict the next floating-poine value.
5

---

Published as a conference paper at ICLR 2025
During inference with 3DMolFormer, the outputs are handled in two modes:
Token Mode: In the drug design task, when predicting ligand SMILES tokens, the corresponding numerical output holds no meaningful value and is therefore padded with 1.0.
Numerical Mode: In docking and drug design tasks, once the ligand SMILES is determined, the length of the 3D coordinate sequence and its tokens are also fixed. Therefore, the token output no longer holds meaningful information and is filled with the expected tokens (from ["x', "y, 'z', 'LCS', 'LCE'J. When the position corresponds ['x', 'y", 'z'], the predicted floating-poim values are appended to the input numerical sequence. For tokens corresponding to ['LCS', 'LCE'), the numerical values are also set to 1.0.
3.3 SELF-SUPERVISED PRE-TRAINING
To enable the 3DMolFormer model to learn the general patterns of pocket-ligand complex se- quences, we conduct large-scale pre-training on 3D data, which includes three datasets: approxi- mately 3.2M protein pockets, about 209M small molecule conformations, and around 167K pocket- ligand complexes. The first two datasets were collected by Uni-Mol (Zhou al. 2023a) for large- scale pre-training on 3D protein pockets and small molecules, while the last dataset was generated by CrossDocked2020 (Francoeur et 2020).
In order for the dual-channel autoregressive model to capture both the token sequence format and 3D coordinate patterns of pocket-ligand complexes, we adopt composite loss function for the prediction of the next token and the corresponding numerical value. This loss function incorporates the cross-cntropy (CE) loss for the whole token sequence and the mean squared error (MSE) loss for the numerical sequence corresponding the 3D coordinates:
L($, s) CE Stok) + a MSE $ geoord coord) )
(2)
num
where & represents the sequence predicted by 3DMolFormer, refers to the training data, and a is coefficient that controls the balance between the CE loss and the MSE loss. This composite loss is applied to all of the three types of pre-training data.
Additionally, we employ large-batch training (Keskar et al. 2017) through gradient accumulation, which we found to be crucial for the pre-training stability of 3DMolFormer. For further details on pre-training and syper-parameter settings, please refer Section and Appendix B
3.4 FINE-TUNING
After the large-scale pre-training, we further fine-tune the 3DMolFormer model on two downstream drug discovery tasks: supervised fine-tuning for pocket-ligand docking, and reinforcement learning (RL) fine-tuning for pocket-aware drug design.
3.4.1 SUPERVISED FINE-TUNING FOR PROTEIN-LIGAND BINDING POSE PREDICTION
In the protein-ligand binding pose prediction (docking) task, illustrated in Figure 3 each sample consists of a pocket-ligand complex. The input sequence contains the atoms of the protein pocket and their 3D coordinates, along with the SMILES sequence of the ligand. The output is the 3D coordinates of each atom in the ligand.
The pre-training data for 3DMolFormer already includes about 167K pocket-ligand complexes from CrossDocked2020 (Francoeur et al. 2020); however, these complexes are generated using the dock- ing software Smina Koes al. (2013) 3), which means that the docking performance with this data would not exceed that of Smina. To improve the upper limit of our model's 's docking performance, we fine-tune on the experimentally determined PDBBind dataset [iu et al.) (2017). which contains approximately 17K ground-truth pocket-ligand complexes. Additionally, we employ a task-specific loss function that computes the mean squared error (MSE) loss only for the 3D co- ordinates of the ligand in the context of next numerical value prediction, since the inference process of docking operates entirely in numerical mode:
models trained
Trdccknng (8 coord lig coord MSF
( coord lig coord nuum ppm
(3)

---

Published as a conference paper at ICLR 2025
To mitigate overfitting during supervised fine-tuning. SMILES randomization Arús-Pous et al., 2019) and random rotation of the 3D coordinates of complexes are used as data augmentation strate- gies. For further details on docking fine-tuning, please refer to Section 4.1 and Appendix C
3.4.2 RL FINE-TUNING FOR POCKET-AWARE 3D DRUG DESIGN
In the pocket-aware drug design task, as illustrated in Figure 3 each sample is also a pocket-ligand pair. The input sequence includes the atoms of the protein pocket and their 3D coordinates, while the output consists of the ligand SMILES sequence and the 3D coordinates of its atoms.
Inspired by 1D RL-based molecular generation methods (Olivecrona al., (017). an RL agent with the 3DMolFormer architecture is initialized with the pre-trained weights, and a molecular property scoring function for each protein pocket is designed as the RL reward. Then, the agent iteratively optimized maximize the expected reward of its outputs. Specifically, each RL step, the agent samples a batch of 3D ligands, and the regularized maximum likelihood estimation (MLE) loss Svensson al. 2023) of each ligand is computed and used update the agent:
(4) where glig (glig.smiile smiles and glig.coord) is sample generated by the RL agent, m is the 3D molecule represented by şlix, and R(-) reward function evaluating the property the molecule. trained is the likelihood of the pre-trained 3DMolFormer model for generating the sequence Tagemt (s) is the corresponding likelihood of the agent model, and a is coefficient hyper-parameter to control the importance of the reward. This loss function encourages the agent to generate molecules with higher expected rewards while retaining low deviation from the pre-trained weights.
Lesignn (SUB) (log Tpre-trained tok
alig.mmlles) + (R(m) log Tagent took
(llilssmlles) ))²
It is important to note that to leverage the duality of the two SBDD tasks, the sampling of ligand SMILES utilizes the weights of the RL agent's 's model, which are continuously updated during fine- tuning. In contrast, the generation of atomic 3D coordinates uses the weights from the model fine- tuned for docking, which remains unchanged during this process. For additional details on RL. fine-tuning and hyper- syper--aramettr settings, please refer Section 4.2 and Appendix D
4 EXPERIMENTS
In this section, we present the results of two parts of experiments: pocket-ligand docking and pocket- aware 3D drug design. Through pre-training, the 3DMolFormer model is theoretically capable of being applied to the conformation generation of small molecules. However, as Zhou et al. (2023b) pointed out, the existing benchmarks for conformation generation are wrong; therefore, this experi- ment is not conducted.
Following the configuration the GPT-2 small model (Radford et al. 2019), the 3DMolFormer model with total of 92M parameters has 12 transformer layers, each containing 12 self self-attention heads, and the embedding dimension is 768. The maximum length for the parallel sequences is set to 2048, which covers over 99% of the samples in the training set as well as all samples in the test set for protein-ligand docking.
For pre-training, all samples with coordinate range larger than 40 screened out. Then, we repli- cate each protein pocket five times and each pocket-ligand complex twenty times, mixing them with small molecule conformations, resulting pre-trained on this dataset for only one epoch, using a batch size of 10K implemented by gradient accumulation. The maximal learning rate is set to 5 10-4 with a warmup period of 1% steps followed by cosine decay. An AdamW optimizer Loshchilov & Hutter (2019) with weight decay factor of 0.1 is employed, and the coefficient a the loss function of Eq. (2) is set to 1.0. The pre-training process takes less than 48 hours with 4 A100 80G GPUs. For further details on the selection of ttper-parameters. parameters for pre-training, please refer to Appendix
total of 228M training data samples. 3DMolFormer is
4.1 PROTEIN-LIGANT BINDING POSE PREDICTION
Experiments of protein-ligand binding pose prediction conducted in the targeted and semi- flexible docking scenario, where the protein pocket for binding is specified and fixed, while the ligand conformation is entirely flexible.
7

---

Published as a conference paper at ICLR 2025
Table 1: Experimental results of 3DMolFormer, its variants, and other baselines protein-ligand binding pose prediction, following the results reported in Uni-Mol (Zhou et al. 2023a). (1) (1)
denotes that higher lower value better. The best result in each column bolded.
Methods
%<1.0 (1)
%<2.0A (1)
%<3.0A (1)
%55.0A (1)
Avg. (1)
AutoDock4
21.8
35.4
47.0
64.6
3.53
AutoDock Vina
44.2
64.6
73.7
84.6
2.37
Vinardo
41.8
62.8
69.8
76.8
2.49
Smina
47.4
65.3
74.4
82.1
1.84
Uni-Mol
43.2
80.4
87.0
94.0
1.62
3DMolFormer w/o PT
15.5
57.8
78.1
92.4
2.25
3DMolFormer w/o DA
10.3
51.0
74.9
91.6
2.45
3DMolFormer
43.8
84.9
96.4
98.8
1.29
Data Following Uni-Mol (Zhou et al. 2023a use PDBbind v2020 Liu et al. 2017) as the
training set for supervised fine-tuning on protein-ligand docking and CASF-2016 (Su et al., 2018) the test set, which includes 285 test samples. In addition, apply the same data filtering process as Uni-Mol to remove training samples with high similarity to the protein sequences or molecular structures of the complexes the test set, which results in training set comprising 18,404 ground- truth complexes.
Baselines We select four search-based methods: AutoDock4 (Morris et al. 2009). AutoDock Vina (Trott & Olson 2010 Eberbardt et al.) 2021), Vinardo (Quiroga & Villarreal 2016), and Smina (Koes et al.) 2013), along with Uni-Mol (Zhou et al. (2023a), which currently the state- of-the-an deep learning method for targeted docking, as baselines.
Two variants of 3DMolFormer are established: (I) training a 3DMolFormer model from scratch on the fine-tuning set for protein-ligand docking without pre-training (w/o PT), and (2) fine-tuning the pre-trained 3DMolFormer model without data augmentation (w/o DA).
Ablation Studies
Evaluation The root mean square deviation (RMSD) between the predicted ligand pose and the ground truth is used to assess binding pose accuracy. Specifically, two metrics employed: (1) the percentage of RMSD results that fall below predefined thresholds, with higher percentages indicat- ing better performance, and (2) the average RMSD, where lower values are preferred.
Fine-tuning For supervised fine-tuning for pocket-ligand binding pose prediction, we train the model for 2000 epochs with batch size of 128. The maximum learning rate is set 10-4, with warmup period of 1% of the steps and cosine decay applied thereafter. The training process takes less than 24 hours with A100 80G GPUs.
Results As shown in Table 3DMolFormer outperforms all baselines in both average RMSD and the percentage of predictions with RMSD less than 2.0, 3.0, and 5.0 Á. Notably, it significantly surpasses other methods in the percentages for RMSD below 3.0 and 5.0 A. This indicates that 3DMolFormer is less prone to making "large errors" compared to the basclines, reflecting its robust- ness. However, for the percentage of predictions with RMSD below 1.0 Å, the search-basee method Smina outperforms the deep learning approaches, suggesting that there is still room for improvement the ability of deep learning methods to capture the intricate interactions between protein pockets and ligands. Moreover, the ablation studies demonstrate that the pre-training and data augmentation both play a crucial role in the training of the 3DMolFormer docking model.
It is worth noting that, unlike all baseline methods, 3DMolFormer does not require initialized 3D conformation the ligand as input, indicating that the model has acquired the capability to predict small molecule 3D conformations through pre-training. This feature enhances the usability of 3DMolFormer compared to previous docking approaches.
Additionally, the average time taken by 3DMolFormer to predict binding pose is 0.8 seconds using A100 80G GPU, and this can be significantly accelerated through parallel inference. This suggests that 3DMolFormer has great potential for applications in large-scale virtual screening. For
8

---

Published as a conference paper at ICLR 2025
further details and results of experiments on protein- logdin binding pose prediction, please refer to Appendix C
4.2
POCKET-AWARE 3D DRUG DESIGN
In the experiments for pocket-aware 3D drug design, small molecule ligands and their 3D confor- mations are designed to bind well with a specified pocket on a protein whose structure remains fixed.
Data Following previous works (Peng et al., 2022; Guan et al. 2023a b). we select 100 protein pockets from the CrossDocked2020 (Francoeur al. 2020) dataset that exhibit low similarity (< 30%) to the protein sequences of pocket-ligand complexes used in pre-training, thereby establishing our targets for 3D drug design.
Baselines We compare 3DMolFormer against various baselines for pocket-aware 3D molecular generation, including AR (Luo et al. 202T), liGAN (Ragoza et al. 2022). GraphBP (Liu et al,) 2022). Pocker2Mol (Peng et al.) (0222) TargetDiff (Guan et al.] (023a)) and DecompDiff (Guan ct al. 2023b). Additionally, we report the results of the ligands corresponding to the 100 protein pockets in the CrossDocked2020 dataset for reference.
Evaluation In alignment with previous works, we evaluate 100 3D molecules generated for each protein pocket. Four metries are selected to comprehensively assess the potential of generated molecules in practical drug design: (1) Vina Score, which directly estimates the binding affinity based on the generated 3D molecules; (2) Vina Dock, representing the best possible binding affinity of the molecules estimated by re-docking; (3) QED (Quantitative Estimate of (iug-llieness) (Bick-)
erton et al. 2012); and (4) SA (Synthetic Accessibility) (Ertl & Schuffenhauer 2009 !
We employ
Quick Vina (Alhossary et al. 2015) to estimate the binding affinity, which is an efficient alternative to AutoDock Vina. For all metries, we report their average values across designed drug molecules for all protein pockets. Following Long et al. (2022) and Guan et al. (2023b), we also report the per- centage of designed drug molecules meeting specific criteria: Vina Dock< 88118 QED 0.25, and SA> 0.59. This percentage, referred to the Success Rate, reflects the performance of different methods multi-objective drug design, which is a common scenario in practical drug discovery.
Reward Function For the aforementioned drug design objectives, we formulate a composite ward function for the RI. fine-tuning process (R(m) Eq. 44). First, a reverse sigmoid function (Hu et al. 2023) is applied transform the Vina Dock score into range of [0, 1]. where higher values are preferable:
(Dock(m) = 1/(1 + ( (V))
(5)
where refers to a small molecule.
Next, we utilize a step function for QED and SA, as these properties are auxiliary to the docking score; thus, they only need to exceed certain thresholds rather than aiming for higher values.
ROED (m) = I(QED(m) > 0.25), Rsa(m) I(SA(m) > 0.59),
(6)
where I(.) represents the indicator function.
Finally, the mean of these three scores employed as the RL reward function:
R(m) R (m) + (m) + (s)).
This composite reward is also used the multi-objective criteria for selecting drug candidates from all generated molecules.
Fine-tuning For the reinforcement learning fine-tuning aimed pocket-aware 3D drug design, we execute 500 RL steps for each protein pocket, with batch size of 128 and constant learning Eq. (4)
rate of 10-4 The parameter takes less than 8 hours using 1 A 100 80G GPU and 128 CPU cores, with the computation of the Vina Dock reward running in parallel on the CPU cores.
100. The RL process for each protein pocket
Here the original SA has been lincarly transformed [0, 1], as illustrated in Appendix

---

Published as a conference paper at ICLR 2025
Table 2: Experimental results of 3DMolFormer and other baselines on pocket-aware 3D drug design, following the results reported in DecompDiff (Guan et al. 2023b). (1) (L) denotes that higher /
lower value is better. The best result in each column bolded.
Methods
Vina Score (4)
Vina Dock (4)
QED (t)
SA (f)
Success Rate (1)
Reference
-6.36
-7.45
0.48
0.73
25.0%
AR
-5.75
-6.75
0.51
0.63
7.1%
liGAN
-6.33
0.39
0,59
3.9%
GraphBP
-4.80
0.43
0.49
0.1%
Pocket2Mol
-5.14
-7.15
0.56
0.74
24.4%
TifgetDiff
-5.47
-7.80
0.48
0.58
10.5%
DecompDiff
-5.67
-8.39
0.45
0.61
24.5%
3DMoeFormer
-6.02
-9.48
0.49
0.78
85.3%
Results As shown in Table 2 the molecules designed by 3DMolFormer outperform those of all baselines across four metrics: Vina Score, Vina Dock, SA, and Success Rate. Notably, it exhibits significant advantage Success Rate, becoming the first method exceed the reference values pro- vided in the dataset for this key metric. Additionally, the result of QED also significantly surpasses the predefined threshold. This indicates that 3DMolFormer demonstrates superior performance in binding affinity optimization and multi-ohjective joint optimization compared to existing 3D drug design methods, highlighting its strong potential for real-world applications in drug discovery.
For further details, results, and case study of experiments pocket-aware 3D drug design, please refer to Appendix
CONCLUSION AND DISCUSSION
In this paper, we introduce 3DMolFormer for structure-based drug discovery, dual-channel wansformer-basee framework designed to process parallel sequences of tokens and numerical values representing pocket-ligane complexes. Through self-supervised large-scale pre- traininiing and super- vised fine-tuning. 3DMolFormer accurately and efficiently predict the binding poses of ligands to protein pockets. Furthermore, through reinforcement learning fine-tuning, 3DMolFormer can generate drug candidates that exhibit high binding affinities for given protein target, along with favorable drug-likeness and synthesizability. Above all, 3DMolFormer is the first machine learning framework that can simultaneously address both protein-ligand docking and pocket-aware 3D drug design, and it outperforms previous baselines in both tasks.
is noteworthy that many recent deep learning models for 3D molecules, such as Uni-Mol, Pocket2Mol, TargetDiff, and DecompDiff, which serve as baselines in our experiments, adhere the concept of "equivariance" introduced by geometric deep learning (Atz al. 2021 Isert et al. 2023a). However, the 3DMolFormer model does not explicitly enforce SE(3) Stmmmsyyymerr appears that through the normalization of 3D coordinates and random rotations during data aug- mentation, 3DMolFormer has acquired the SE(3)-equivariance by training on sufficiently large and diverse dataset. This approach aligns with recent successful methods in the field, including AlphaFold3 (Abramson et al. 2024). which also does not rely -Euivvaqui architectures.
Admittedly, our approach still has some limitations. First, 3DMolFormer does not account for the flexibility of proteins during ligand binding, which may affect the accuracy of subsequent bind- ing affinity prediction. Second, protein-ligand binding is dynamic process, but 3DMolFormer struggles capture this dynamism effectively. Finally. 3DMolFormer does not consider environ- mental factors such as temperature and pH, which can significantly influence the 3D conformation of the binding complex. These issues represent core challenges in current computational methods for structure-based drug discovery, and we look forward to future work addressing these limitations. Furthermore, the implementation details in 3DMolFormer have the potential be further optimized, for example, advanced methods of multi objeitvoeeective reinforcement learning (Liu et al. 2014) may be introduced into the drug design process.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew Ballard, Joshua Bambrick, al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pp. 1-3, 2024.
Sungsoo Ahn, Junsu Kim, Hankook Lee, and Jinwoo Shin. Guiding deep molecular optimization with genetic exploration. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12008-12021. Curran Associates, Inc., 2020.
Eric Alcaide, Zhifeng Gao, Guolin Ke, Yaqi Li, Linfeng Zhang, Hang Zheng, and Gengmo Zhou. Uni-mol docking v2: Towards realistic and accurate binding pose prediction. arXiv preprint arXiv:2405.11769, 2024.
Amr Alhossary, Stephanus Daniel Handoko, Yuguang Mu, and Chee- Kwoh. Fast, accurate, and reliable molecular docking with quickvina 2. Bioinformatics, 31(13):2214-23 2216, 2015.
Josep Arús-Pous, Simon Viet Johansson, Oleksii Prykhodko, Esben Jannik Bjerrum, Christian Tyr- chan, Jean-Louis Reymond, Hongming Chen, and Ola Engkvist. Randomized smiles strings improve the quality of molecular generative models. Journal of cheminformatics, 11(1):1-13, 2019.
Kenneth Atz, Francesca Grisoni, and Gisbert Schneider. Geometric deep learning molecular representations. Nature Machine Intelligence 3(12): 1023-1322,, 2021.
Minkyung Back, Frank DiMaio, Ivan Anishchenko. Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N Kinch, R Dustin Schaeffer, et al. Accurate prediction of protein structures and interactions using three-track neural network. Science, 373(6557):871- 876. 2021.
Viraj Bagal, Rishal Aggarwal, PK Vinod, and U Deva Priyakumar. Molgpt: molecular generation using a transformer-ecooder model. Journal of Chemical Information and Modeling, 62(9):2064- 2076, 2021.
Mostapha Benhenda. Can ai reproduce observed chemical diversity? bioRxiv, pp. 292177, 2018.
Richard Bickerton, Gaia Paolini, Jérémy Besnard, Sorel Muresan, and Andrew Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90-98, 2012.
Alexandre Blanco-Gonzalez, Alfonso Cabezon, Alejandro Goco-Gonzalez. Daniel Conde-Torres, Paula Antelo-Riveiro Angel Pinciro, and Rebeca Garcia-Fandino. The role of ai in drug discov- ery: challenges, opportunities, and strategies. Pharmaceuticals, 16(6):891, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901, 2020.
Martin Buttenschoen, Garrett M Morris, and Charlotte Deane. Posebusters: Ai-based docking methods fail to generate physically valid poses or generalise novel sequences. Chemical Sci- ence, 15(9):3130 3139, 2024.
Can Chen, Jingbo Zhou, Fan Wang, Xue Liu, and Dejing Dou. Structure-awart protein self- supervised learning. Bioinformatics, 2023.
Gabriele Corso, Hannes Stärk, Bowen Jing, Regina Barzilay, and Tommi Jaakkola. Diffdock: Dif- fusion steps, twists, and turns for molecular docking. In International Conference on Learning Representations (ICLR), 2023.
11

---

Published as a conference paper at ICLR 2025
Gabriele Corso, Arthur Deng, Nicholas Polizzi, Regina Barzilay, and Tommi Jaakkola. Deep confi- dent steps to new pockets: Strategies for docking generalization. In International Conference on Learning Representations (ICLR), 2024.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, pp. 4171-4186, 2019.
Chai Discovery, Jacques Boitreaud, Jack Dent, Matthew McPartlon, Joshua Meier, Vinicius Reis, Alex Rogozhnikov, and Kevin Wu. Chai-1: Decoding the molecular interactions of life. bioRxiv, pp. 2024-10, 2024.
Yuanqi Du, Tianfan Fu, Jimeng Sun, and Shengchao Liu. Molgensurvey; A systematic survey in machine learning models for molecule design. arXiv preprint arXiv:220. 14500, 2022.
Yuanqi Du, Arian R Jamasb, Jeff Guo, Tianfan Fu, Charles Harris, Yingheng Wang, Chenru Duan, Pietro Liò, Philippe Schwaller, and Tom L Blundell. Machine learning-aii middd generative molecular design. Nature Machine Intelligence, pp. 1-16, 2024.
Jerome Eberhardt, Diogo Santos-Martins, Andreas F Tillack, and Stefano Forli. Autodock vina 1.2. 0: New docking methods, expanded force field, and python bindings. Journal of chemical information and modeling, 61(8):3891 --899, 2021.
Peter Eckmann, Kunyang Sun, Bo Zhao, Mudong Feng, Michael K Gilson, and Rose Yu. Limo: Latent inceptionism for targeted molecule generation. In International Conference on Machine Learning. PMLR, 2022.
Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. Journal of cheminfor- matics, 1(1):1-11, 2009.
Xiaomin Fang, Lihang Liu, Jieqiong Lei, Donglong He, Shanzhuo Zhang, Jingho Zhou, Fan Wang, Hua Wu, and Haifeng Wang. Geemeety-enhanced molecular representation learning for property prediction. Nature Machine Intelligence, 4(2):127-134, 2022.
Shikun Feng, Minghao Li, Yinjun Jia, Weiying Ma, and Yanyan Lan. Protein-ligand binding repre- sentation learning from fine-grained interactions. In International Conference on Learning Rep- resentations, 2024a.
Wei Feng, Lvwei Wang, Zaiyun Lin, Yanhao Zhu, Han Wang, Jianqiang Dong, Rong Bai, Huting Wang, Jielong Zhou, Wei Peng, et al. Generation of 3d molecules in pockets via a language model. Nature Machine Intelligence, 6(1):62-73, 2024b.
Daniel Flam-Shepherd and Alán Aspuru- Guzik. Language models can generate molecules, materi- als, and protein binding sites directly in three dimensions xyz, cif, and pdb files. arXiv preprint arXiv:2305.05708, 2023.
Paul G Francoeur, Tomohide Masuda, Jocelyn Sunseri, Andrew Jia, Richard B lovanisci, Ian Snyder, and David R Koes. Three-dimensional convolutional neural networks and cross-docked data set for senecture-bsse drug design. Journal of chemical information and modeling, 60(9):4200- 4215, 2020.
Tianfan Fu, Wenhao Gao, Connor Coley, and Jimeng Sun. Reinforeed genetic algorithm for structure-based drug design. Advances in Neural Information Processing Systems, 35:12325- 12338, 2022.
Bowen Gao, Bo Qiang, Haichuan Tan, Yinjun Jia, Minsi Ren, Minsi Lu, Jingjing Liu, Wei- Ying Ma. and Yanyan Lan. Drugelip: Contrasive protein-molest molecule representation learning for virtual screening. Advances in Neural Information Processing Systems, 36, 2024a.
Bowen Gao, Minsi Ren, Yuyan Ni, Yanwen Huang, Bo Qiang, Zhi-Ming Ma. Wei- Wei-Ying Ma, and Yanyan Lan. Rethinking specificity in sbdd: Leveraging delta score and mergy-guided diffusion. arXiv preprint arXiv:2403. /2987, 2024b.
12

---

Published as a conference paper at ICLR 2025
Siavash Golkar, Mariel Pettee, Michael Eickenberg, Alberto Bietti, Miles Cranmer, Geraud Krawezik, Francois Lanusse, Michael McCabe, Ruben Ohana, Liam Parker, et al. xval: con- tinuous number encoding for large language models. arXiv preprint arXiv:2310.02989, 2023.
Jiaqi Guan, Wesley Wei Qian, Xingang Peng, Yufeng Su, Jian Peng, and Jianzhu Ma. 3d equivariant diffusion for target-aware molecule generation and affinity prediction. In International Confer- ence on Learning Representations, 2023a.
Jiaqi Guan, Xiangxin Zhou, Yuwei Yang. Yu Bao, Jian Peng, Jianzhu Ma, Qiang Liu, Liang Wang, and Quanquan Gu. Decompdiff: diffusion models with decomposed priors for structure-based drug design. In International Conference on Machine Learning, 2023b.
Charles Harris, Kieran Didi, Arian R Jamasb, Chaitanya Joshi, Simon V Mathis, Pictro Lio, and Tom Blundell. Benchmarking generated poses: How rational is structure-based drug design with generative models? arXiv preprint Xiv:2308.077433. 2023.
Xiuyuan Hu, Guoqing Liu, Yang Zhao, and Hao Zhang. De novo drug design using reinforcement learning with multiple gpt agents. In Thirty-sevnnth Conference Neural Information Processing Systems, 2023.
Xiuyuan Hu, Guoqing Liu, Quanming Yao, Yang Zhao, and Hao Zhang. Hamiltonian diversity: effectively measuring molecular diversity by shortest hamiltonian circuits. Journal of Cheminfor- matics, 16( 1):94, 2024.
Ross Irwin, Spyridon Dimitriadis, Jiazhen He, and Esben Jannik Bjerrum. Chemformer: a pre- trained transformer for computational chemistry. Machine Learning: Science and Technology, (1):015022, 2022.
Clemens Isert, Kenneth Atz, and Gisbert Schneider. Strueture-based drug design with geometric deep learning. Current Opinion in Structural Biology, 79:102548, 2023a.
Clemens Isert, Kenneth Atz, and Gisbert Schneider. Structure-based drug design with geometric deep learning. Current Opinion in Structural Biology, 79:102548, 2023b.
Dejun Jiang, Chang-Yu Yu Hsieh, Zhenxing Wu, Yu Kang, Jike Wang. Ercheng Wang, Ben Liao, Chao Shen, Lei Xu, Jian Wu, et al. Interactiongraphnet: A novel and efficient deep graph representation learning framework for accurate protein-ligany ligand interaction predictions. Journal of medicinal chemistry, 64(24):1820--182)
2021.
Wengong Jin, Regina Barzilay, and T. Jaakkola. Multi-objective molecule generation using terpretable substructures. In International Conference Machine Learning, pp. 4859-4859. PMLR, 2020.
Zygimantas Jocys, Joanna Grundy, and Katayoun Farrahi. Drugpose: benchmarking 3d generative methods for early stage drug discovery. Digital Discovery, 2024.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583-588) 2021.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017.
David Ryan Koes, Matthew Baumgartner, and Carlos Camacho. Lessons learned in empirical scoring with smina from the csar 2011 I benchmarking exercise. Journal of chemical information and modeling, 53(8): 1893-1944 2013.
Seul Lee, Scanie Lee, Kenji Kawaguchi, and Sung Ju Hwang. Drug discovery with dynamic goal- aware fragments. Proceedings of the 41th International Conference on Machine Learning, 2024.
Zeming Lin, Halil Akin, Roshan Rao. Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with language model. Science, 379(6637): 123-11130 2023.
13

---

Published as a conference paper at ICLR 2025
Chumming Liu, Xin Xu, and Dewen Hu. Multi-objective reinforcement learning: A comprehensive overview. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 45(3):385 -985, 2014.
Meng Liu, Youzhi Luo, Kanji Uchino, Koji Maruhashi, and Shuiwang Ji. Generating 3d molecules for target protein binding. International Conference on Muchine Leurning. 2022.
Zhihai Liu, Minyi Su, Li Han, Jie Liu, Qifan Yang, Yan Li, and Renxiao Wang. Forging the basis for developing -iigid interaction scoring functions. Accounts of chemical research, 50 (2):302-30% 2017.
Siyu Long, Yi Zhou, Xinyu Dai, and Hao Zhou. Zero-shot 3d drug design by sketching and gener- ating. Advances in Neural Information Processing Systems, 35:23894-23907, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Wei Lu, Qifeng Wu, Jixian Zhang, Jiahua Rao, Chengtao Li, and Shuangjia Zheng. Tankbind: Trigonometry-awan neural networks for drug-protein binding structure prediction. Advances in neural information processing systems, 35:7236-7244 2022.
Shitong Luo, Jiaqi Guan, Jianzhu Ma, and Jian Peng. A 3d generative model for structure-based drug design. Advances in Neural Information Processing Systems, 34:6229-624 2021.
Kit-Kay Mak, Yi-Hang Wong, and Mallikarjuna Rao Pichika. Artificial intelligence in drug discov- ery and development. Drug Discovery and Evaluation: Safety and Pharmacokinetic Assays, pp. 1-38, 2023.
Oscar Méndez-Lucio, Mazen Ahmad, Ehecatl Antonio del Rio-Chanona, and Jörg Kurt Wegner. A geometric deep learning approach predict binding conformations of bioactive molecules. Nature Machine Intelligence, 3(12):0033-103) 2021.
Alex Morehead, Nabin Giri, Jian Liu, and Jianlin Cheng. Deep learning for protein-ligand docking: Are we there yet? arXiv preprint arXiv:2405. 14108, 2024.
Garrett M Morris, Ruth Huey, William Lindstrom, Michel F Sanner, Richard K Belew, David S Goodsell, and Arthur J Olson. Autodock4 and autodocktools4: Automated docking with selective receptor flexibilityy. Journal of computational chemistry, (0(16))2785-2799 2009.
Mareus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. Journal of cheminformaties, 9(1):1-14, 2017.
Daniele Pala and David Clark. Caught between rock and hard place: current challenges in structure-based drug design. Drug Discovery Today, pp. 104106, 2024.
Qizhi Pei, Kaiyuan Gao, Lijun Wu, Jinhua Zhu, Yingce Xia, Shufang Xie, Tao Qin, Kun He, Tie- Yan Liu, and Rui Yan. Fabind: Fast and accurate protein-ligand binding. Advances in Neural Information Processing Systems, 36, 2024.
Xingang Peng, Shitong Luo, Jiaqi Guan, Qi Xic, Jian Peng, and Jianzhu Ma. Pocket2mol: Effi- cient molecular sampling based on 3d protein pockets. In International Conference Machine Learning, 2022.
Joanne Quinn, Joanne McEachen, Michael Fullan, Mag Gardner, and Max Drummy, Dive into deep learning: Tools for engagement. Corwin Press, 2019.
Rodrigo Quiroga and Marcos A Villarreal. Vinardo: A scoring function based on autodock vina improves scoring, docking, and virtual screening. PloS one, 11(5):e015518,, 2016.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Matthew Ragoza, Tomohide Masuda, and David Ryan Koes. Generating 3d molecules conditional receptor binding sites with deep generative models. Chemical science, 13(9):2701-2713, 2022.
14

---

Published as a conference paper at ICLR 2025
Arne Schneuing, Yuanqi Du, Charles Harris, Arian Jamasb, Ilia Igashov, Weitao Du, Tom Blun- dell. Pietro Lió, Carla Gomes, Max Welling, et al. Structure- drug design with equivariant 2022.
diffusion models. arXiv preprint arXiv:2210.1366
Philippe Schwaller, Teodoro Laino, Théophile Gaudin, Peter Bolgar, Christopher A Hunter, Costas Bekas, and Alpha A Lee. Molecular transformer: a model for uncertainty--aliiraaee chemical reaction prediction. ACS central science, 5(9):1572-1583, 2019.
Marwin HS Segler, Thierry Kogej, Christian Tyrchan, and Mark P Waller. Generating focused molecule libraries for drug discovery with recurrent neural networks. ACS central science, 4(1): 120-131, 2018.
Gregor Simm, Robert Pinsler, and José Miguel Hermández-LLoato. Reinforcement learning for molecular design guided by quantum mechanics. International Conference on Machine Learn- ing. pp. 9959.8969. PMLR, 2020.
Jacob O Spiegel and Jacob D Durrant. Autogrow4: an open-source genetic algorithm for de novo drug design and lead optimization. Journal of cheminformaties, 12:1-16, 2020.
Hannes Stärk, Octavian Ganea, Lagnajit Pattanaik, Regina Barzilay, and Tommi Jaakkola. Equibind: Geometric deep learning for drug binding structure prediction. In International conference on machine learning. pp. 20503-2055 200221. PMLR, 2022.
Minyi Su, Qifan Yang. Yu Du, Guoqin Feng, Zhihai Liu, Yan Li, and Renxiao Wang. Comparative assessment of scoring functions: the casf-2016 update. Journal of chemical information and modeling, 59(2):895 913, 2018.
Hampus Gummesson Svensson, Christian Tyrchan, Ola Engkvist, and Morteza Haghir Chehreghani. Utilizing reinforcement learning for de novo drug design. arXiv preprint arXiv:2303. 17615, 2023.
Oleg Trott and Arthur Olson. Autodock vina: improving the speed and accuracy of docking with new scoring function, efficient optimization, and multithreading. Journal of computational chemistry, 3122:455-461,,, 2010.
Rob LM Van Montfort and Paul Workman. Structure-based drug design: aiming for a perfect fit. Essays in biochemistry, 61(5):431 4-7,1143772400 2017.
Debby D Wang, Wenhui Wu, and Ran Wang. Structure-based, deep-learning models for protein- ligand binding affinity prediction. Journal of Cheminformatiis, 16(1):2, 2024a.
Jike Wang, Rui Qin, Mingyang Wang, Meijing Fang, Yangyang Zhang, Yuchen Zhu, Qun Su, Qiaolin Gou, Chao Shen, Odin Zhang, et al. Token-mol 1.0: Tokenized drug design with large language model. arXiv preprint arXiv:2407.07930. 2024b.
Yaqing Wang, Abulikemu Abuduweili, Quanming Yao, and Dejing Dou. Property- aware relation networks for few-shot molecular property prediction. Advances in Neural Information Processing Systems, 34: 1441144444, 2021.
Oliver Wieder, Stefan Kohlbacher, Mélaine Kuenemann, Arthur Garon, Pierre Ducrot, Thomas Sei- del, and Thierry Langer. A compact review of molecular property prediction with graph neural networks. Drug Discovery Today: Technologies, 37:1-12 2020.
Jun Xia, Yanqiao Zhu, Yuanqi Du, Yue Liu, and Stan Z Li. A systematic survey of chemical pre- trained models. International Joint Conference on Artificial Intelligence, 2023.
Chao Yang, Eric Anthony Chen, and Yingkai Zhang. Protein-ligand docking in the machine- learning era. Molecules, 27(14)44568, 2022.
Shuwen Yang, Ziyao Li, Guojie Song, and Lingsheng Cai. Deep molecular representation learn- ing via fusing physical and chemical information. Advances in Neural Information Processing Systems, 14:16346-16333 2021a.
Soojung Yang, Doyeong Hwang, Seul Lee, Seongok Ryu, and Sung Ju Hwang. Hit and lead discov- ery with explorative rl and fragment- Tragment-based molecule generation. Advances in Neural Information Processing Systems, 34:7924-7936, 2021b.
15

---

Published as a conference paper at ICLR 2025
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. Advances in neural information processing systems, 31, 2018.
Yangtian Zhang, Huiyu Cai, Chence Shi, Bozitao Zhong, and Jian Tang. E3bind: An end-to-end equivariant network for protein-ligand docking. In International Conference on Learning Repre- sentations (ICLR), 2023a.
Zaixi Zhang, Qi Liu, Hao Wang. Chengqiang Lu, and Chee-Kong Lee. Motif based graph self- supervised learning for molecular property prediction. Advances in Neural Information Process- ing Systems, 34:15870-15882, 2021.
Zaixi Zhang, Jiaxian Yan, Qi Liu, Enhong Chen, and Marinka Zitnik. A systematic survey in geo- metric deep learning for tructure-baseed drug design. arXiv preprint arXiv:2306.1 1 2023b.
Lingling Zhao, Yan Zhu, Junjie Wang, Naifeng Wen, Chunyu Wang, and Liang Cheng. A brief review of protein-ligand interaction prediction. Computational and Structural Biotechnology Journal, 20:2831-288 2022.
Kangyu Zheng, Yingzhou Lu, Zaixi Zhang, Zhongwei Wan, Yao Ma, Marinka Zitnik, and Tianfan Fu. Structure-based drug design benchmark: Do 3d methods really dominate? arXiv preprint arXiv:2406.03400. 2024.
Artem Zholus, Maksim Kuznetsov, Roman Schutski, Rim Shayakhmetov, Daniil Polykovskiy, Sarath Chandar, and Alex Zhavoronkov. Bindgpt: A scalable framework for 3d molecular de- sign via language modeling and reinforcement learning. arXiv preprint
2024.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. In International Conference on Learning Representations, 2023a.
Gengmo Zhou, Zhifeng Gao, Zhewei Wei, Hang Zheng, and Guolin Ke. Do deep learning methods really perform better in molecular conformation generation? arXiv preprint arXiv:2302.07061, 2023b.
Jinhua Zhu, Yingce Xia, Lijun Wu, Shufang Xie, Tao Qin, Wengang Zhou, Houqiang Li, and Tie- Yan Liu. Unified 2d and 3d pre-training of molecular representations. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pp. 2022.
16

---

Published as a conference paper at ICLR 2025
A PARALLEL SEQUENCES
Here is a real example of the parallel sequence of a pocket-ligand complex with a total length of 867, corresponding Figure [ pocket atom coordinates, ligand SMILES, ligand atom coordinates, and the start and end of each part marked by special tokens. The first amino acid of the pocket is particularly marked. Moreover, the [[x]', "ly]', "[z]" tokens are corresponding to values representing 3D coordinates in the numerical
and 1 The token sequence mainly consists of 4 parts: pocket atoms,
sequence.
CA |CCC:/CCTCCCCCCCCCCCCCCCCCCCCCCC
Tokens:
C CC, C CWT.CCCCCCCCCNNNNNCCCCCCCCCCCCCC C, "CT."0CCC..01CCCC....0"."""""""" CC, "CC"""C""CT""""""""""""""""""""" CCA', ",,""""""."""""""""""""""""""""" C., "_G_"11...000"NN"""11""0"""""""" C.. "CN"C"11"0000001TT1C""""""""""""
[x]". [yy [z]
[y]
[2] [ [y]
[z]'. [x] [yy"
[2]
"[]', [y]" [z [x]
[z].
"[x], [y]
12
[yy.. [2 [x]] 1yl [2
[x] 1y] [2]
[x]
"1]" [yl [2
[x]
[[']". 'ly]' "[]''
[x 144
12]4 [x] "yy"' [z]'. "[x]',
"x]', "[""',"" "1"''"""""""" [[]'
1y4
[x "yy
[z]'. "ly'.
[2
[x]
1yy'. 'l22 '[x]',
"['', "x]'''''' ".2
[x]
Tyl.
[2] [x
[yl'. [z]] "[']'
lyl' [z]
[x]'
[yl", "["],,
"[""""" "[z]",
[[]]' "[y]',
[yy., [2]
[x]
Tyy'.
[x]'. [yy] [z]',
[x] lyl
[z]' '[x] "x"". "z]' '[x]', "['"" " [[x]'
[y]
[z]'
[x]] [yy
[z]' [x]' "."",
[2]
[x]
[y]" "[z'', "[x]' "[y],, '[z]', "[']' "x]""""""""""""""""""" "[y], '[z]' [x] [y]
[2] [[x]"
[yl'. [z]'. [[x''
[yl [z]
[x] [y] [z] '[x]' '[y]' "[z], '[x]', "["" 'lyl', [z] [x]
[x] [yy', "[z'', "[x]'. "[y]', '[]]', """""
[yy'
[x]'. [yy'. [[]',
[x] [y]. [z] [x] [y
[z] "Iy",
[z]'.
[x] [y]".
"[x]', "[y]. '[z]', '[y',
[[]].
[y] [z]] [x
[yl'. '[x'''[x]' "y"'. [z] [x]', "[y]'. '[x]',
""z]". "["'"'" "[y", "Z]', [x]] "y"'. '
[x]". "[y]'. "lz'', [x]]' Ty]. [z]', "[''' "[z]', "x""""""''"" '[x]' [y]'. [Z]] [[x]' "Iy""
[z]'. '[yl'
[z]
[x]' [y] [[z]'
[y]'. '[z]'. '[x]', 'Iy]', [z]] [x]] [y] [zz] [[x]
[y]', [[z]', [x]] ly]
[z] [x] [y],,
[x]] "[y]", 'z]]' [[x]' [yy] [z] [x]
[yy] [
[x]". [y'' [z]] [x]
ly] [z] [x]],
[z]] [x] "y'", [zz]' [x]] [y] [z]
[X] Ty]
[z]'. [[']', [[y]' [2]
[x]
[y] [z]] [x]]
[y]] [z] [x]]' "ly]" [22 [X]
[y]
[2] [x]
[y]', "[]], [[]] lyr [z]
[x]
Ty] [z]
[y] [z]] "[x]' Tyl. [
[x]
[y]
[X]'. [yy" [2] [x]
1y]
[2] [x]
[y]
[x] [y]', [z]], [x] ly]
[z]
[x]
[2].. [x]] Ty.. [2]
[x]
[y] [2]
[z] '[]]', [2] [x]
ly]
[2
[yl'. [z]. [x
lyl [7]
[x] Ty] [z]].
[y]. "[]'' "y... [
[x]
Tyl.
[x]', 'lyl' [z]. [x] ly]
[z]
[x]
"yyl'
[Z] [x]] "yy'', "[]]' [[]]' Ty]
[2]
[x]
[z]". [x]', lyl. [2]
[x]
[y]
[z]
[x]]
[z] [[]]', "yy [z] [x]
lyl
[Z]
[y]". [z]] [x]]
ly] [
[
[z]]
[y] [Z]" [[x] [yl.
[x]
[y]
[x]". "y]" [2] [8]
[y]
[Z]
[x]
"yy]"
[x]", "y"", [Z] [x]
[y]
[2]
[X]
[z]', [x]','l'" [2]
[x]'
[y]"
'[x]'
[y] [z]]" '[x]', [[y]" "22.
[x]
[y]
12. [[x]
[[yl', "[z]','[']
lyl' [7]
[x]
[y] '[Z]'
'[x]' "ly'' "[]'' "lyl"
[[x]'
[y] [
[[x]', "Iy]", "[z]' [x]]
[y]
[z]
[x]
"yy]'
'Zz] '[x]', "ly]', "'''', [x]'
[y]"
[z]]
[x] ly
[[z]", '[x]',
"''''"""''''''''''"""""""""""""" '''''''''''''
"""''""""'"""""""" [Zz] "[''' .2..
[X]
[y]
12 [
[[x]', "[y"," 1 "'''''""'''"""...
" " "ly]'
[X]]
N "1
[
[y]
[
[x
[yl
". [X]] "
[X]
[y] [z]] [[x]
[[y]" [2].
[y] [2]
[PCE]]
ILS
",2"00100111111"1100001"0""1"""1 |ICCT
CLCS [x] "
17

---

Published as a conference paper at ICLR 2025
[x]' [y] [z] [x] 1y] [z] [x]
lyl.
[z]
[x]
1y
[2] [x
1y
[2] [x] lyl
[[''. [x]' lyl., [7]'. [x], 1yl', "[]" [x]'
1y 17
[x]'
1y
17
[x
1y1.
: 17 Xx]
Tyl.. '[]'' , [x]', Tyl. [2]' [x]' Tyl. [7]' Xx] Tyl.. 12]' 'x 1yl.
12
1x Tyl. 7
[x]' Tyyl, [z]', [x]". "yyl', 12] [x] lyl. [Z] [LCE]
Numbers: 1. 1.
1.1.
1.
1.
I.
I.
1.258
-0.361
0.197
1.411 -0.114 0.217 1.242 0.118 0.107 1.106 0.083
-0.099
1.655 -0.181 0.045
1.693 -0.462 0.086
1.418 -0.577 0.076 -0.424 0.62 1.849 -0.673 0.475 1.784 -0.824 0.383 2.028
-0.714
0.248 2.201
-0.619 0.224 1.609 -0.881 0.084 1.542 -0.463 0.299 1.344
-1.458 -0.708 -0.818 -1.275 -0.894 -0.94
-1.285 -1.149 -0.778 -1.411 -1.159 -0.568 -0.986 -0.789 -0.949 -0.871 -0.739 -0.676 -0.726 -0.929
-0.54 -0.866 -0.515 -0.525 -0.637 -0.826 -0.313 -0.723 -0.575 -0.298 -0.515 -2.414 0.742 -0.313
-2.275 0.905 -0.378 -2.308 1.201 -0.227 -2.425 1.35 -0.31 -1.984 0.824 -0.231 -1.956 0.529 -0.09
-1.698 0.47 -0.209 -1.479 0.513 0.143 -1.706 0.383 11755 0.086 0.046 -1558 0.29 0.172 -1.731
0.562 0.1 1.8114 -0.011
-. 0.526
0.019
11.15 0.045 -0. 0.011 -0.174
1.928 0. 171 -0.047 1.719 -0. 128 1.437 00.074-0.311 4.0 -0.129 1.757 0.514 -0.4 1.67
0.621 -0.628 0.417 0.846 -0.613 0.228 1.008 -0.363 0.288 0.909 -0. 0.409 0.755 -0.591 -0.066
0.623 -0.345 -0.106
0.814-0.14421..333.
168 2.133 -1.88 -0.408 1.991 -1.66 -0.492 2.095
-1.703 -0.508 2.335 -1.183 -0.637 2.028 -0.7 -0.61 1.832 -0.729 -0.497 1.871 -0.981 -0.692 1,556
-0.588 -0,495 1.631 -0.6 -0.6660.....3
-0821
1.399 -0,679 -0.674 1.169-1.117 -0.877 1.134
-0.873 -0.801 1.022 -0.812 -1,891 -0.518 -0.571 -2.055 -0.490 -0.657 -2.345 -0.518 -0.869 -2.419
0.416 -0.439 -2.009 -0.222 -0.382 -0.162 -0.133 -1.605 -0.2 -0.544 -1.519 -0.08 -0.143 -1 -1.349
-0.135 -0.392 -1.289 -0.072 0.109 -0.508 -2.274 0.335 -0.345 -2.184 0.588 -0.505 -2.131 0.786
0.399 -2.029 0.262 -0.179 -1.937 0.193 -0.35 -1.99 0.254 -0.585 -1,991 0.056 -0.232 -1,505 0.249
-0.588 0.808 0.382 -0.809 0.806 0.581 -0.839 0.578 0.694 -1.052 0.553 0.189 -1.044 0.78 0.07
1.033 0.52 -0.029 -1.041 0.999 1.244 0.351 0.238 1.426 0.432 0.457 1.353 0.35 0.742 1.477 0.45
0.926 1.424 0.737 0.433 1.153 0.801 0.335 1.082 0.581 0.142 -0.177 1.888 0.87 0.004 1.664 0.936
0.224 1.742 1.135 0.36 1.565 1.242 0.132 1.522 0.692 -0.081 1.397 0.519 0.304 1.71 0.529 -1.008
1.286 1.446 -1.146 1.092 -1.317 0.904 1.433 -1.261 0.856 1.667 -0.944 0.926 1.111 -0.747
1.057 0.919 -0.636 0.849 0.724 -0.832 1.303 0.773 11773 0.636 -0.746 1.678 0.408 -0.922 -1.728
0. -0.766 -1.731 0.16 -0.52 -1.389 0.408
0.434 -0.82 -.025 0.176 -0.756 -1.17
-0.044 -0.769 -0.777 0.206 -0.67 -1.603 -1.141 0.028 1.559 -0.893 0.123 1.694 -0.668 -0.013
1.614 -0.591 -0.235 -12259
-0.78 -1.259 0.481 -1.002 -1336 0.604 0..40.
-1.488
0.319 -1,491 -1.312 0.727 -1.026 -1.229 0.706 -0.729 -1.225 0.500 -0.597 0.872 -0.537
1.32 0.543 -0.337 -1.111 0.763 -0.295
-1251---880
-0,888
11100-111-11107003333----1--1111 -0.13-03331110
-1220333 -0,231
-1.836 -1.271 -0,142 -2.065 -0.913 -0.058 -15581 -0.693 -0. 11388 -0.672 -0.316 -1.267 -0.52
0.1 -1.351 -0.84
2.384 -0.05 -0.733 2.123 0.03 -0.443 2. 1.1 0.11 -0.361 2.014 0.306 -0.766 1.927
-0.202 -0.663 1.652 -0. 146 -0.809 1.472 0.01 -0.418 1.569 -0.244 -0.718 1.22 0.063 -0.324 1.316
-0.193 -0.478 1.138 -0.037 189 0.013 1.237 0.19 -0.096 0.964 0.344 -0.358 0.967 0.548 -0.396
1.1II 0.343 0.095 0.776 0.232 0.378 0.764 0.407 0.536 0.569 -0.061 0.389 0.682 0.912 0.482 1.165
0.846 0.7 1.349 0.635 0.619 1.554 0.435 0.487 1.489 0.752 0.946 1.188 0.974 1.064 1,009 0.85
1.269 0.822 1.203 1.191 1.169 -1.592 0.135 1.225 -1.603 -0.071 1.018 1.794 -0.288 1.113 -1.763
-0.394 1.334 -1.325 -0.19 0.955 11.346 -0.396 0.728 -1.137 0.029 0.868 0.794 0.122 -0.903 -0.524
0.748 -0.693 0.269 -1.24 -0.704 0.718 0.231 -0.421 1. 1.
1.
I. 1.
I. I.
0.637
1.213 -0.028 0.463 0.997
-0.136 0.502 0.924 -0.423 0.258 0.857 -0.581 0.091 0.665 -0.42 -0.147 0.711 -0.403 0.202 0.434
-0.286 0.039 0.245 -0.136 0.171 0.024 -0.035 0.039 -0.171 0.113 -0.234 -0.159 0.155 -0.382 -0.425
0.342 -0.778 -0.713 0.219 -0.674 -0.386 0.319 -0.287 -0.384
0.617 -0.374 0.062 0.048 -0.238 0.262
0.095 0.319 0.763 -0.862
0.596 0.657 -0.885 0.295 0.994
11051
0.123
0.552
-0.948 I.
18

---

Published as a conference paper at ICLR 2025
In addition, the factor g in Eq. 1 is set to 5.0. The performance of the model is not sensitive to the choice of q, because most of the 3D coordinates in our data (protein pockets and ligands) are in limited range.
B PRE-TRAINING
Data Source pockets for pre-training (3.2M), ligand conformations for pre-training (209M), and ground-truth protein-ligand comple for docking fine-tuning (17K): htts://htts:://itthhb com/
Docked protein-ligand complexes for pre-training and test for pocket-aware 3D drug design: httpes/s//ggggooooooo///gggggggg
In addition, samples with the maximal difference in coordinates in one dimension greater than 40 are removed in order to filter out those outliers that account for less than 0.1 % data.
Model Scaling The standard dual-channel model used in our paper follows the configuration of the GPT-2 small model (Radford al. 2019). An ablation study of pre-training is conducted de- termine the appropriate scale for 3DMolFormer, where the pre-training loss on the ligand validation is reported for each model size:
Layers
Heads
Embedding length
Pre-training Loss
8
256
0.325
12
8
256
0,254
12
12
256
0,229
12
12
768
0.178
16
12
768
0.178
16
16
768
0.180
The standard model size achieves the best performance compared with others, as result of which is utilized our design.
Hyper-parameters
The coefficient a in the loss function of Eq. 2 is set to 1.0. an ablation
study, we observe that the selection of a does not significantly affect the balance between CE loss and MSE loss:
CE Loss
MSE Loss
0.1
0.164
0.014
1.0
0.164
0.014
10.0
0.164
0.015
This may be because errors on the token sequences and those on the numerical sequences con- verge respectively during the large-scale pre-training.
The selection other parame-rrsseterss follows the common practice of the pre-training large language models (Radford al. 2019 Quinn et al. 2019).
C PROTEIN-LIGAND BINDING POSE PREDICTION SUPPLEMENT
Ducking Setup The exhaustiveness of all 4 search-based docking baselines Table is set to 8, following the settings in Uni-Mol (Zhou et al. 2023a).
Standard Errors
As shown in Table B the standard errors of the 3DMolFormer performance results are obtained by individual runs of supervised fine-tuning on protein-ligand docking. The minor standard errors further validate the robustness and soundness of 3DMolFormer.
19

---

Published as a conference paper at ICLR 2025
Table 3: Standard Errors of 3DMolFormer performance results in Table 1
Methods
%<1.0A (1)
%<2.0A (1)
%<3.0A (1)
%<5.0A (1)
Avg. (1)
3DMolFormer
43.8
84.910.5
96.41.022 98.8
1.29:10.22
Additional Experiments on PoseBusters PoseBusters (Buttenschoen et al. 2024 s widely- used benchmark for evaluating pootein-ligand docking methods, particularly focusing on the chal- lenges of blind docking, where the binding pocket information is not provided. However, in our study, we evaluate 3DMolFormer on PoseBusters using pocket information, providing different evaluation context compared to typical PoseBusters assessments conducted for state-of-the-an dock- ing approaches such as AlphaFold (Abramson et al. 2024), Chai-1 (Discovery et al. 2024 and Uni-Mol Docking V2 (Alcaide et al. 2024).
For experiments on PoseBusters, the blind docking baselines following the standard evaluation setup include AutoDock Vina (Trott & Olson 2010), DiffDock (Corso et al.) (203)) Uni-Mol Docking V2 (Alcaide et al. 2024). AlphaFold3 (Abramson et al. 2024). and Chai- (Discovery et al. 2024) For pocket-aware docking approaches including Uni-Mol (Zhou al.
2023a) and 3DMol- Former, we provide pocket information for docking. As shown in Table 4 our 3DMolFommer achieves higher pocket-aware docking accuracy than Uni-Mol, which is also higher than the blind docking accuracy of all state-of-the-art baselines.
Table 4: Experimental results of protein-ligand binding pose prediction on PoseBusters benchmark.
Methods
%<<..0A (1)
AutoDock Vina DiffDock
52.3
37.9
Uni-Mol Docking V2
77.6
AlphaFold3
76.3
Chai-1
77.1
Uni-Mol (pocket-aware)
74.8
3DMolFormer (pocket-aware)
81.5
D POCKET-AWARE 3D DRUG DESIGN SUPPLEMENT
Clarification the SA score It should be clarified that the SA ranges in [1, 10] defined in the original paper (Ertl & Schuffenhauer 2009), where a lower score is better. Following the pre- vious work on pocket-aware 3D drug design (Guan al. 2023b), report the linearly transformed SA score: SA = (10 - SAooggi) E [0, 1], where higher score is better.
Clarification on molecular diversity We do not include metrics for molecular diversity such as internal diversity [Benhenda] (2018) and Hamiltonian diversity Hu et al. (2024) in out evaluation, because existing metrics are based on 2D graph structures, while pocket-aware 3D drug design is a 3D molecular generation task.
In addition, in the 3D drug design experiments no more than 100 molecules are generated by each baseline method for each protein pocket. For 3ororFormer, exactly 100 unique molecules are generated and seleted for each protein pocket, which is a more stringent requirement.
Generation Setup
Case Study Visualization of the reference binding molecule and two molecules generated by 3DMolFormer on protein 4H3C:
Standard Errors and Ablation Study As shown in Table 5 the standard errors of the 3DMol- Former performance results obtained by 5 individual runs of RL fine-tuning pocket-aware 3D drug design. The minor standard errors further validate the robustness and soundness of 3DMol- Former.
20

---

Published as a conference paper at ICLR 2025
Reference
Designed Example 1
Designed Example 2
Molecule
Vina Dock
QED
SA
Reference
-8.0
0.55
0.91
Designed Example
-11.1
0.35
0.91
Designed Example 2
-10.6
0.48
0.75
In addition, conduct ablation study 3DMolFormer. The variant 3DMolFormer w/o RL refers to freezing the GPT weights for RI. fine-tuning, that is, generating molecules without fine- tuning for pocket-aware 3D drug design. The results indicate that the RL fine-tuning process is fundamental for this task.
Table Standard Errors of 3DMolFormer performance results Table 2 and results of the ablation study.
Methods 3DMolFormer
Vina Score (J)
Vina Dock (1)
QE)
SA (f)
Success Rate (f)
-6.02:0.27
9.48110.18
0.49100.1
0.78
85.3%
3DMolFormer w/o RL
-4.20
-5.03
0,46
0.50
2.1%
Distribution of Generated Molecules Figure demonstrates the distributions molecular weights, logP values, and the number of rotatable bonds of the 10,000 molecules designed by 3DMolFormer for all the 100 targets reported in Table 2 It is worth mentioning that all three metrics taken into account in drrug-likeneee measured by QED.
Distribution of Molecular Weights
Distribution of LagP
Distribution of Number Rotatable Bo
0.006
0.25
0.005
Frequency
0.004
Frequency
Frequency
0.15
0.003
0.10
0.10
0.05
0.05
200
2.5
150
Molecular Weight
LogP
Number Rotatable Bonds
Figure 4: The distributions of molecular weights, logP values, and the number of rotatable bonds of the molecules designed by 3DMolFormer.
Additional Evaluation by Delta Score Delta Score is a novel evaluation metric structure- based drug design that emphasizes the specificity of molecular binding (Gao al. 2024b). Unlike
21

---

Published as a conference paper at ICLR 2025
traditional docking scores that can inflate results due biases, Delta Score evaluates the selective affinity of a molecule for its target compared to other potential binding pockets, providing a more accurate measure of binding specificity and reducing the influence of promiscuous binding effects.
Results in Table 6 show that 3DMolFormer outperforms previous methods in terms of Delta Score, demonstrating its superior capability to generate molecules with higher specificity for their intended targets.
Table 6: Experimental results of Delta Score on pocket-aware 3D drug design.
Methods
Mean Delta Score (?)
Reference
1.158
AR
0.393
Pocket2Mol
0,437
TargetDiff
0.335
DecompDiff
0.354
3DMolformer
0.716
Additional Evaluation by PoseCheck Clash Score and Strain Energy are key metrics used in PoseCheck (Harris et al. (023) to evaluate the physical plausibility and stability of protein-ligand poses in structure-based drug design. Clash Score assesses steric clashes between atoms in the generated pose, while Strain Energy quantifies the energetic distortion from ideal molecular confor- mations. Both metrics ensure that generated poses align with physical and chemical principles.
Table 7 demonstrates that 3DMolFormer outperforms baselines on both metrics, highlighting its ability produce more physically realistic and energetically favorable docking poses.
Table 7: Experimental results of Delta Score on pocket-aware 3D drug design.
Methods
Mean Clash Score ())
Median Strain Energy (+)
Reference
4.59
102.5
LiGAN
3.40
18693.8
Pocket2Mol
5.62
194.9
TargetDiff
9.08
1241.7
3DMooFormmrr
3.25
183.3
Furthermore, DrugPose (Jocys al. 2024) offers broad range of metrics for 3D drug discovery, its overlap with PoseCheck the context of structure-based drug design makes PoseCheck a sufficient benchmark for our evaluation, ensuring comprehensive assessment without redundancy.
22


Reviews:
[{"review_id": "I8P82xBXoT", "reviewer": "Reviewer_U5kn", "summary": "This paper introduces 3DMolFormer, a unified transformer-based framework for SBDD, which is capable of docking prediction and pocket-aware 3D drug design. The motivations of 3DMolFormer are clearly stated: current computational docking methods lack accuracy, while current 3D pocket aware drug design methods are unable to take full advantage of 3D structural information due to factors such as difficulties in 3D information modelling and limited data regarding ground-truth protein-ligand complexes. To represent a 3D complex of a protein pocket and a small molecule ligand, 3DMolFormer uses a parallel sequence composed of the SMILES atom token sequences for the protein and small molecule, along with the numerical sequence for 3D coordinates. A GPT architecture is pretrained through autoregressive generation of the parallel sequence. Fine-tuning consists of: (1) a supervised protein-ligand binding pose prediction task, and (2) a multi-objective RL pocket-aware molecular generation task. The presented results suggest that 3DMolFormer is successful in both fine-tuning tasks; it accurately predicts binding poses of ligands to protein pockets, and is capable of generating molecules that display high binding affinity to protein targets, while being synthesizable and exhibiting drug-like qualities.", "questions": "- The authors state: “... the sampling of ligand SMILES utilizes the weights of the RL agent’s model, which are continuously updated during finetuning. In contrast, the generation of atomic 3D coordinates uses the weights from the model finetuned for docking, which remains unchanged during this process.” Why don’t the authors freeze the GPT weights during the sampling of ligands? If this hinders the model’s performance, why is this not shown in the ablation studies?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed GPT framework seems interesting and presents novelty in terms of representing 3D complexes.\n- 3DMolFormer presents strong results compared to other models on both fine-tuning tasks.\n- The presentation of this paper is clear and well-structured.", "weaknesses": "- The multi-objective optimization of the RL seems to be overly simplistic, including a reward function that assigns a constraint-based reward for QED and SA. Literature on multi-obj DRL shows using more sophisticated reward functions and multi-objective optimization techniques greatly improve agent performance and stability. \n- Minimal ablation studies are conducted, and all results are based on one run. More runs should be conducted to demonstrate the soundness of the model. \n\nMinor edits:\n- Line 482: the parameter σ in Eq. (4) was is to 100", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t73UhyzCHv", "reviewer": "Reviewer_69DH", "summary": "This paper introduce 3DMolFormer, a dual-channel Transformer-based model that can process atom sequence and coordinate information parallelly, and thus this model is claimed to be the first one that could simultaneously address both protein-ligand docking and pocket-aware 3D drug design, and it outperforms previous baselines in both tasks.", "questions": "1. For the pose evaluation, the model only check for RMSD. Could you also report other pose-relate metrics like steric clashes and strain-energy?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper introduce a novel transformer-based model that can handle docking and structure-based drug design simultaneously", "weaknesses": "1. This paper mention figure 1 multiple times when introducing model structure, however there is no figure 1 in the preprint.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nXp70IqW8y", "reviewer": "Reviewer_HfQS", "summary": "This paper proposes a novel dual-channel transformer-based framework that effectively handles both protein-ligand docking and structure-based drug design (SBDD) tasks.", "questions": "Refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Docking and structure-based drug design (SBDD) are indeed dual tasks. The method presented in this paper, which models both tasks simultaneously within a single framework, represents a promising and logical approach.\n2. By leveraging the similar architecture of GPT, the proposed method demonstrates significant scalability, including both model parameters and data volume, allowing for the effective utilization of large-scale datasets for pre-training.", "weaknesses": "As discussed in Section 5, the proposed method does not consider SE(3) symmetry explicitly but instead relies on data augmentation techniques. I think this aspect warrants further discussion and consideration. Although the experiments validate the method's effectiveness to some extent, I believe the persuasive power of these findings is limited when considering the following points:\n1. For docking task, as far as I know, the more advanced approach Uimol-docking v2 is not included in the baslines.\n2. For sbdd task, I noticed that the article directly employs evaluation metrics (Vina QED SA) as the reward function and utilizes reinforcement learning for fine-tuning. This approach may be considered unfair to other methods.  It may be necessary to incorporate abalation study as well as additional evaluation methods to comprehensively validate the effectiveness of the approach, for example, the delta score proposed in paper [1] may serve as a metric for assessing whether the method is overfitting to the Vina evaluation.\n\n\n[1]. Ren, Minsi, et al. \"Delta Score: Improving the Binding Assessment of Structure-Based Drug Design Methods.\" arXiv preprint arXiv:2311.12035 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "smBBjd2Dt4", "reviewer": "Reviewer_H1Mo", "summary": "The paper presents 3DMolFormer, a dual-channel transformer framework designed for protein-ligand docking and pocket-aware 3D drug design. It utilizes a parallel sequence format to represent 3D pocket-ligand complexes and employs a \"pre-training + fine-tuning\" approach on a large dataset to model 3D information effectively.", "questions": "-Could you clarify how well you perform on PoseBusters?\n-What are speed accuracy trade-offs in comparison to AlphaFold3 or Chai-1?\n-How the thresholds where chosen for the addition. By calculating using the formula. It seems that it was chosen to use QED 4.39?\n-What are the proteins in which you generated the molecules? How did you choose them? Are they diverse?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Unified Framework: 3DMolFormer integrates protein-ligand docking and pocket-aware 3D drug design into a single model.\nParallel Sequence Format: Represents 3D pocket-ligand complexes, facilitating effective modeling of both discrete and continuous information.\nLarge-Scale Pre-training: Utilizes a pre-training.\nEnhanced 3D Information Modeling: Effectively addresses challenges in modeling complex 3D interactions.\nNovel dual generative+docking : Applicable to multiple tasks within structure-based drug discovery with 3D information, improving efficiency in drug design processes.", "weaknesses": "Docking. The model should be benchmarked against PoseBusters. The state of the art here includes AF3 and Chai-1, which achieve an accuracy of 76–77%. These models are only trained on PDB data. I understand that cofolding is not part of this approach, but we still need to assess the trade-offs on a widely recognized benchmark.\n\nSynthetic Accessibility Score. It's stated that a score above 0.59 is given a value of 1. Why is this the case? Typically, a lower score indicates better synthetic accessibility. Is there a mistake, or are they using a different metric? I noticed that further clarification is provided in the Appendix.\n\nMulti-score Thresholds. The thresholds for multi-score criteria appear to be chosen arbitrarily. For example, a value of 1 is assigned to QED and SA scores that surpass a certain threshold. Could you clarify how these cutoffs were selected?\n\nDocking Formula. They provide a formula for redocking, but typically, a different target would require a distinct reference score, standardized by weight. There doesn’t seem to be a standard docking energy applied here.\n\nMolecule Distribution. The paper does not specify the distribution of generated molecules in terms of size. Is there variation in logP values? How about the number of rotatable bonds?\n\nBenchmarks for 3D Generation. I suggest using established benchmarks, such as CheckPose or DrugPose, for 3D generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)