3DMolFormer: A Dual-channel Framework for Structure-based Drug Discovery

Authors: Xiuyuan Hu, Guoqing Liu, Can Chen, Yang Zhao, Hao Zhang, Xue Liu
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: structure-based drug discovery, protein-ligand docking, 3d molecule generation, transformer

Abstract:
Structure-based drug discovery, encompassing the tasks of protein-ligand docking and pocket-aware 3D drug design, represents a core challenge in drug discovery. However, no existing work can deal with both tasks to effectively leverage the duality between them, and current methods for each task are hindered by challenges in modeling 3D information and the limitations of available data. To address these issues, we propose 3DMolFormer, a unified dual-channel transformer-based framework applicable to both docking and 3D drug design tasks, which exploits their duality by utilizing docking functionalities within the drug design process. Specifically, we represent 3D pocket-ligand complexes using parallel sequences of discrete tokens and continuous numbers, and we design a corresponding dual-channel transformer model to handle this format, thereby overcoming the challenges of 3D information modeling. Additionally, we alleviate data limitations through large-scale pre-training on a mixed dataset, followed by supervised and reinforcement learning fine-tuning techniques respectively tailored for the two tasks. Experimental results demonstrate that 3DMolFormer outperforms previous approaches in both protein-ligand docking and pocket-aware 3D drug design, highlighting its promising application in structure-based drug discovery. The code is available at: https://github.com/HXYfighter/3DMolFormer.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730531011960

thread_type:
official_review

<thread_object>
creator: reviewer_h1mo

<content>
summary:
The paper presents 3DMolFormer, a dual-channel transformer framework designed for protein-ligand docking and pocket-aware 3D drug design. It utilizes a parallel sequence format to represent 3D pocket-ligand complexes and employs a "pre-training + fine-tuning" approach on a large dataset to model 3D information effectively.

strengths:
Unified Framework: 3DMolFormer integrates protein-ligand docking and pocket-aware 3D drug design into a single model.
Parallel Sequence Format: Represents 3D pocket-ligand complexes, facilitating effective modeling of both discrete and continuous information.
Large-Scale Pre-training: Utilizes a pre-training.
Enhanced 3D Information Modeling: Effectively addresses challenges in modeling complex 3D interactions.
Novel dual generative+docking : Applicable to multiple tasks within structure-based drug discovery with 3D information, improving efficiency in drug design processes.

weaknesses:
Docking. The model should be benchmarked against PoseBusters. The state of the art here includes AF3 and Chai-1, which achieve an accuracy of 76–77%. These models are only trained on PDB data. I understand that cofolding is not part of this approach, but we still need to assess the trade-offs on a widely recognized benchmark.

Synthetic Accessibility Score. It's stated that a score above 0.59 is given a value of 1. Why is this the case? Typically, a lower score indicates better synthetic accessibility. Is there a mistake, or are they using a different metric? I noticed that further clarification is provided in the Appendix.

Multi-score Thresholds. The thresholds for multi-score criteria appear to be chosen arbitrarily. For example, a value of 1 is assigned to QED and SA scores that surpass a certain threshold. Could you clarify how these cutoffs were selected?

Docking Formula. They provide a formula for redocking, but typically, a different target would require a distinct reference score, standardized by weight. There doesn’t seem to be a standard docking energy applied here.

Molecule Distribution. The paper does not specify the distribution of generated molecules in terms of size. Is there variation in logP values? How about the number of rotatable bonds?

Benchmarks for 3D Generation. I suggest using established benchmarks, such as CheckPose or DrugPose, for 3D generation.

questions:
-Could you clarify how well you perform on PoseBusters?
-What are speed accuracy trade-offs in comparison to AlphaFold3 or Chai-1?
-How the thresholds where chosen for the addition. By calculating using the formula. It seems that it was chosen to use QED 4.39?
-What are the proteins in which you generated the molecules? How did you choose them? Are they diverse?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730533061845

thread_type:
official_review

<thread_object>
creator: reviewer_hfqs

<content>
summary:
This paper proposes a novel dual-channel transformer-based framework that effectively handles both protein-ligand docking and structure-based drug design (SBDD) tasks.

strengths:
1. Docking and structure-based drug design (SBDD) are indeed dual tasks. The method presented in this paper, which models both tasks simultaneously within a single framework, represents a promising and logical approach.
2. By leveraging the similar architecture of GPT, the proposed method demonstrates significant scalability, including both model parameters and data volume, allowing for the effective utilization of large-scale datasets for pre-training.

weaknesses:
As discussed in Section 5, the proposed method does not consider SE(3) symmetry explicitly but instead relies on data augmentation techniques. I think this aspect warrants further discussion and consideration. Although the experiments validate the method's effectiveness to some extent, I believe the persuasive power of these findings is limited when considering the following points:
1. For docking task, as far as I know, the more advanced approach Uimol-docking v2 is not included in the baslines.
2. For sbdd task, I noticed that the article directly employs evaluation metrics (Vina QED SA) as the reward function and utilizes reinforcement learning for fine-tuning. This approach may be considered unfair to other methods.  It may be necessary to incorporate abalation study as well as additional evaluation methods to comprehensively validate the effectiveness of the approach, for example, the delta score proposed in paper [1] may serve as a metric for assessing whether the method is overfitting to the Vina evaluation.


[1]. Ren, Minsi, et al. "Delta Score: Improving the Binding Assessment of Structure-Based Drug Design Methods." arXiv preprint arXiv:2311.12035 (2023).

questions:
Refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730560635538

thread_type:
official_review

<thread_object>
creator: reviewer_69dh

<content>
summary:
This paper introduce 3DMolFormer, a dual-channel Transformer-based model that can process atom sequence and coordinate information parallelly, and thus this model is claimed to be the first one that could simultaneously address both protein-ligand docking and pocket-aware 3D drug design, and it outperforms previous baselines in both tasks.

strengths:
1. This paper introduce a novel transformer-based model that can handle docking and structure-based drug design simultaneously

weaknesses:
1. This paper mention figure 1 multiple times when introducing model structure, however there is no figure 1 in the preprint.

questions:
1. For the pose evaluation, the model only check for RMSD. Could you also report other pose-relate metrics like steric clashes and strain-energy?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730586960945

thread_type:
official_review

<thread_object>
creator: reviewer_u5kn

<content>
summary:
This paper introduces 3DMolFormer, a unified transformer-based framework for SBDD, which is capable of docking prediction and pocket-aware 3D drug design. The motivations of 3DMolFormer are clearly stated: current computational docking methods lack accuracy, while current 3D pocket aware drug design methods are unable to take full advantage of 3D structural information due to factors such as difficulties in 3D information modelling and limited data regarding ground-truth protein-ligand complexes. To represent a 3D complex of a protein pocket and a small molecule ligand, 3DMolFormer uses a parallel sequence composed of the SMILES atom token sequences for the protein and small molecule, along with the numerical sequence for 3D coordinates. A GPT architecture is pretrained through autoregressive generation of the parallel sequence. Fine-tuning consists of: (1) a supervised protein-ligand binding pose prediction task, and (2) a multi-objective RL pocket-aware molecular generation task. The presented results suggest that 3DMolFormer is successful in both fine-tuning tasks; it accurately predicts binding poses of ligands to protein pockets, and is capable of generating molecules that display high binding affinity to protein targets, while being synthesizable and exhibiting drug-like qualities.

strengths:
- The proposed GPT framework seems interesting and presents novelty in terms of representing 3D complexes.
- 3DMolFormer presents strong results compared to other models on both fine-tuning tasks.
- The presentation of this paper is clear and well-structured.

weaknesses:
- The multi-objective optimization of the RL seems to be overly simplistic, including a reward function that assigns a constraint-based reward for QED and SA. Literature on multi-obj DRL shows using more sophisticated reward functions and multi-objective optimization techniques greatly improve agent performance and stability. 
- Minimal ablation studies are conducted, and all results are based on one run. More runs should be conducted to demonstrate the soundness of the model. 

Minor edits:
- Line 482: the parameter σ in Eq. (4) was is to 100

questions:
- The authors state: “... the sampling of ligand SMILES utilizes the weights of the RL agent’s model, which are continuously updated during finetuning. In contrast, the generation of atomic 3D coordinates uses the weights from the model finetuned for docking, which remains unchanged during this process.” Why don’t the authors freeze the GPT weights during the sampling of ligands? If this hinders the model’s performance, why is this not shown in the ablation studies?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732138606608

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We sincerely thank all reviewers for your valuable feedback and constructive suggestions. We have provided detailed point-by-point responses to each review to address your concerns.

In addition, we have uploaded a revised version of the paper. In this revised version, we use a **yellow background** to highlight the contents referenced in our responses and a **blue background** to indicate newly added content. These revisions are primarily located in **Sections 4 and 5**, as well as **Appendices C and D**. The updates include additional experiments, expanded discussions, and clarifications that align with the feedback provided.

We greatly appreciate your time and effort in reviewing our work and look forward to any further feedback you may have.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733458627564

thread_type:
public_comment

<thread_object>
creator: ~xiangyu_huang3

<content>
title:
Ask for more results

comment:
Dear Authors,

I am working on 3D molecular generation, and I came across this paper and was very interested in the idea of framing 3D structure generation as language modeling, as in ESM3 there are very complicated tokenization designs, while in this paper the authors only input every Euclidean coordinate. However, this paper claims to *effectively addressing the challenges of modeling 3D information in SBDD*, yet with limited evaluation in terms of 3D structures, and I found current discussion inadequate for me to truly appreciate this paper’s value. In hopes for a more comprehensive evaluation in order to better understand the value of this paper and possibly help the authors to demonstrate their results, I would like to raise several questions about the method.

## More 3D metrics
It's very impressive to see the generated molecules with such a low strain energy and high affinities, and a more thorough demonstration would help to provide more insight. 
- I am very curious about the Posebusters validity (PB-Valid) ratio for docking methods (since DL methods are known to violate the 3D physical constraints often) like in AlphaFold3, Chai-1, Boltz-1 and other recent docking methods. How does 3DMolFormer perform in terms of PB-Valid?
- Additionally, I checked the DecompDiff paper and found it important to assess bond length, bond angle and torsion angle distributions of the generated molecular structures. Can the authors provide the Jensen-Shannon Divergence between the molecules generated by their 3DMolFormer and reference molecules, too?

## Concerns on fair comparison
My biggest impression is that RL boosts the model’s performance so much, from barely working (2.1% Success Rate) to an astonishing level (85.3%). Can the authors explain why RL is so crucial here, and provide more implementation details? For example, on what dataset is the reward model being trained? What architecture is the reward model?

Moreover, as Reviewer HfQS pointed out, this paper might have made an unfair comparison between their approach (w/ RL) with non-optimization baselines (w/o RL). This brings out my third question.


## More literature review & SOTA baselines
I am not an expert in docking or drug design. But after checking the reference, it seemed to me that all the baselines are published no later than 2023, suggesting they are not SOTA now. Can the authors provide a more thorough list of related works and include the most recent models for comparison? Also, I am eager to see the missing literature review of molecule optimization, and how 3DMolFormer (w/ RL) compares with them.

Thank you very much! The authors’ feedback will be very helpful for my research.
</content>
</thread_object>

</thread>
<thread>
creation date:
1754848073232

thread_type:
public_comment

<thread_object>
creator: ~nahyun_kim3

<content>
title:
Questions regarding "3DMolFormer" and code reproducibility

comment:
Dear Authors,

I recently read your paper "3DMolFormer: A Dual-channel Framework for Structure-based Drug Discovery" with great interest and was particularly impressed by the results. I would be very grateful for your guidance on a few points.


## 1. Generation fine-tuning process
In Figure 3 of the paper, it appears that generation was performed including protein tokens, but in the released code, generation seems to start from the <LIG_START> token. Could you clarify which approach was actually used?


## 2. Generation fine-tuning results (Vina score and Vina dock)
The paper reports both Vina score and Vina dock results. Were these obtained by first generating SMILES strings with the generation fine-tuned model, then docking them using the docking fine-tuned model to get coordinates and scores? Or were they derived directly from the generation fine-tuned model?


## 3. About generation task datasets
According to the paper and the code you provided on GitHub, my understanding is that during the generation finetuning process, 128 samples are generated at each step for 500 steps. Then, after the RL finetuning for each protein pocket is complete, the top 100 samples are selected for evaluation.

I have a few questions regarding this process:

The dataset used for RL finetuning is also included in the pre-training data. Is it appropriate to use this data for the final evaluation? I am concerned about potential data leakage.

In other papers, performance is typically compared by randomly sampling 100 molecules. In contrast, this paper appears to generate a very large number of samples and then selects only the top 100 for evaluation. I am wondering if this might be an unfair comparison.

I am also curious whether a separate, specialized model was trained for each individual protein pocket.


## 4. Evaluation code
Would it be possible to share the evaluation code used to compute the scores reported in the paper for both docking and generation fine-tuning? This does not appear to be included in the GitHub repository.


## 5. Docking test results
I followed the code provided in the GitHub repository exactly and created my own docking evaluation script based on the methodology described in your paper. However, my results are significantly different from those reported. Since the evaluation was performed on data that had already been through the pre-processing pipeline, I am unsure why there is such a large discrepancy.

This leads me to ask were there any additional pre-training or finetuning steps performed that differ from what is described in the paper or the public GitHub code?

Here are my experimental results:

| Metric | My Result | Result in Paper |
| :--- | :--- | :--- |
| **RMSD < 1.0 Å** | 0.80% | 43.8%
| **RMSD < 1.5 Å** | 4.78% | 
| **RMSD < 2.0 Å** | 14.74% | 84.9% |
| **RMSD < 3.0 Å** | 34.26% | 96.4% |
| **RMSD < 5.0 Å** | 66.53% | 98.8% |
| **Average RMSD** | 4.52 | 1.29 |

As you can see, these values differ considerably from those in the paper.


## 6. Dataset repetition discrepancy
The paper mentions repeating the protein pocket dataset five times and the pocket-ligand dataset twenty times, whereas the github code sets the number of epochs to two and ten, respectively. Could you clarify which values were actually used?

Thank you very much for your time and for making the code publicly available. I would greatly appreciate any insights or materials you could share to help with faithfully reproducing your work.

Best regards.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission received feedback from four reviewers, almost all of whom provided positive reviews. Even though Reviewer H1Mo scored 5 in the first round and with no further feedback after rebuttal, the authors provided replies point-by-point and solved most of the issues from my perspective. Therefore, my recommendation is 'Accept (poster).