Reframing Structure-Based Drug Design Model Evaluation via Metrics Correlated to Practical Needs

Authors: Bowen Gao, Haichuan Tan, Yanwen Huang, Minsi Ren, Xiao Huang, Wei-Ying Ma, Ya-Qin Zhang, Yanyan Lan
Primary area: datasets and benchmarks
Keywords: Stucture-Based Drug Design, Model Evaluation, Benchmark

Abstract:
Recent advances in structure-based drug design (SBDD) have produced surprising results, with models often generating molecules that achieve better Vina docking scores than actual ligands. However, these results are frequently overly optimistic due to the limitations of docking score accuracy and the challenges of wet-lab validation. While generated molecules may demonstrate high QED (drug-likeness) and SA (synthetic accessibility) scores, they often lack true drug-like properties or synthesizability. To address these limitations, we propose a model-level evaluation framework that emphasizes practical metrics aligned with real-world applications. Inspired by recent findings on the utility of generated molecules in ligand-based virtual screening, our framework evaluates SBDD models by their ability to produce molecules that effectively retrieve active compounds from chemical libraries via similarity-based searches. This approach provides a direct indication of therapeutic potential, bridging the gap between theoretical performance and real-world utility. Our experiments reveal that while SBDD models may excel in theoretical metrics like Vina scores, they often fall short in these practical metrics. By introducing this new evaluation strategy, we aim to enhance the relevance and impact of SBDD models for pharmaceutical research and development.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729119111691

thread_type:
official_review

<thread_object>
creator: reviewer_8amh

<content>
summary:
The paper provides a comprehensive framework for evaluating structure-based drug design models, using metrics like virtual screening and similarity to known drugs.

The paper is well written. However, the paper has some limitations. Although it acknowledges the difficulty of synthesizing molecules derived from SBDD models, it lacks a systematic approach. A heavy reliance on virtual screening-based metrics can be misleading without experimental validation. For the paper to be more useful and applicable, more real-world case studies should be included.

strengths:
1. very detailed and well-explained in most of the sections.

2.  The propose a more practical framework for evaluating SBDD models, which can be useful.

weaknesses:
1. While the paper acknowledges the difficulty of synthesizing molecules generated by SBDD models, it does not present a systematic approach for addressing this problem.

2. In this paper, virtual screening-based metrics are heavily employed to determine the practical usefulness of generated molecules. While virtual screening is helpful, it could still be misleading if it is not validated with experimental data.

3. This paper needs to include more real-world case studies to be more valuable.

questions:
1. how well does this framework perform when applied to novel targets for which there may be no closely related compounds available?

2. Are there any risks associated with overfitting to the new evaluation metrics?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730162060319

thread_type:
official_review

<thread_object>
creator: reviewer_wyjc

<content>
summary:
This paper points out a curial problem in current SBDD models evaluation: The vina docking score metric is easy to inflate and other metrics are hard to test in wet lab. So the paper proposed three-level evaluation metrics: Similarity to Known Drugs and Actives, Virtual Screening Ability and Binding Affinity Estimation. They test on five models and claim that MolCRAFT outperforms other methods

strengths:
1. Provide a novel benchmark to tackle a curial problem in this area
2. Select a diverse set of models on SBDD

weaknesses:
1. Focus on testing deep learning models. Could test more on other based such as RL based or Genetics Algorithm based
2. Table 1's description is too simplified, hard to let people to understand why using real molecules as template is better
3. It would be better to briefly describe some terminology, i.e. FLAPP

questions:
1. The proposed benchmark relied on comparing similarity with known drug molecules, does this affect the search for de novo drug molecules?
2. Could you show the docking pose images of molecules generated by MolCRAFT and Pocket2Mol exhibit good similarity to one of the known active compounds
</content>
</thread_object>

</thread>
<thread>
creation date:
1730198545483

thread_type:
official_review

<thread_object>
creator: reviewer_pelf

<content>
summary:
This paper draws attention to pitfalls in evaluation metrics commonly used to assess generative models for structure-based drug discovery. While the authors focus their discussion on 3D methods, the pitfalls discussed extend to any generative model for drug discovery. The authors first highlight the problem with the commonly used docking score metric and how it can be exploited. Next, three sets of evaluation tasks are proposed: delta docking score (assess specific binding), similarity metrics (to known actives and FDA-approved drugs), and virtual screening (can a similarity search from generated molecules identify known actives?). The authors provide insights on the current performance of 3D SBDD generative models and highlight shortcomings of current model development to guide future research.

strengths:
* The authors explicitly highlight pitfalls of current docking score evaluations of generated molecules
* Performance across several 3D models is compared and contrasted
* The rationale is clear when proposing new metrics

weaknesses:
Overall, I agree there should be better metrics for SBDD assessment. However, many of the metrics and challenges proposed by the authors has been extensively reported in existing literature. There are also many existing works that are not 3D-based diffusion/flow models for SBDD that account for the limitations expressed by the authors. As a result, some of the limitations are limitations of the problem formulation and not intrinsically to SBDD metrics. I will organize the comments under each proposed metric.

### **Docking and Synthesizability**
* The authors focus on SBDD generative models but there is a lot of work in this field that does not involve geometric models that can perform SBDD and recent benchmarks also show that these 1D/2D methods outperform 3D methods [1]. The reason this is brought up is because in these works, the limitation of AutoDock Vina to exploit low QED molecules and large molecules is well documented. As a result of this, there are many papers that explicitly optimise docking jointly with QED (and often also with SA score). Here are a few works that do this, many of which have been published at conferences [2, 3, 4, 5]. Therefore, while I do think that many papers are overfitting AutoDock Vina by designing large molecules with low QED, many recent works have acknowledged this and are proposing objective functions that explicitly account for QED.
* A large reason the Delta score is poor for AutoDock Vina is that it is too crude. This is likely to diminish if one uses Glide docking, as the authors implicitly show with the enrichment. The goal could also be explicitly to dock well to 1 protein and not to another protein as was done here [6].
* It is true that metrics like SA score are imperfect to predict synthesizability but it can be well correlated [7, 8].
* Taking Tanimoto similarity to known molecules does not necessarily mean it is easier to synthesize. Synthesizability is complex and single atom changes can turn something synthesizable to something that is not.

### **Similarity Metrics (including Virtual Screening)**
* If a metric is introduced that rewards similarity to known molecules, then the optimisation objective could simply be similarity to these molecules. This has also been shown explicitly in [9] where the model generates molecules similar to a known active while getting good docking scores. In my opinion, we do not necessarily want to re-discover known molecules in the sense of generating similar molecules to a set of reference ligands. This is already done in distribution learning/transfer learning approaches that were detailed in this review [10]. The alternative and more general goal is to design molecules with arbitrary property profiles. In this case, the degree to which optimisation is successful is the most relevant metric. 
* Similarity to FDA-approved drugs may not be completely justified. Drugs are specific and drugs targeting 2 different proteins can be vastly different in structure. In Table 3, the authors also show that even the reference ligand have low similarity to the FDA-approved drugs.
* The problem with Vina scores that the authors elude to is that it is too crude. A very well known problem of docking algorithms is that they cannot reproduce experimental binding poses. Specifically, re-docking is a common task used to validate docking. In cheminformatics literature, this is known as sampling power [11]. Similarly, the ability to rank affinity with docking scores is known as scoring power.
* In ML literature, an equivalent is the assessment of how many binding poses are within some RMSD threshold, for example in the DiffDock paper [12]. If the docking cannot reproduce known experimental data, it is unclear the level of confidence to trust the output. The reason for bringing this up is because if there is a docking algorithm that can capture the binding interactions properly, designing molecules optimised for docking score can implicitly capture similarity. This was explicitly shown in cheminformatics literature [13] (although I still believe similarity to known molecules is not necessarily the best metric and can be hacked). The idea of having a better docking algorithm equating to better results is also shown by the authors in Table 1: comparing Glide with AutoDock Vina and Glide gets better enrichment. Then if Glide were optimised instead, the generated molecules would be better. I understand that this is not commonly done because Glide is proprietary software.

While I believe the sentiment of the authors is that better metrics can better reflect real impact, metrics do not replace wet-lab experiments. This line should not be bolded to give the false impression that predictive methods are close to accurately predicting wet-lab experiment. It would be better to re-phrase this line to convey that better metrics can better inform future model development that may improve wet-lab outcomes.

### **References**
[1] 3D SBDD Benchmark: https://chemrxiv.org/engage/chemrxiv/article-details/66bb0911a4e53c48763ac057

[2] Reinforced GA: https://arxiv.org/abs/2211.16508

[3] TacoGFN: https://arxiv.org/abs/2310.03223

[4] GEAM: https://arxiv.org/abs/2310.00841

[5] Saturn: https://arxiv.org/abs/2405.17066

[6] Negative Design: https://iclr.cc/virtual/2023/12911

[7] SA Correlation 1: https://jcheminf.biomedcentral.com/articles/10.1186/s13321-023-00678-z

[8] SA Correlation 2: https://arxiv.org/abs/2407.12186v1

[9] Similarity reward: https://www.nature.com/articles/s42256-022-00494-4

[10] Generative Design Review: https://www.nature.com/articles/s42256-024-00843-5

[11] Sampling and Scoring Power: https://pubs.rsc.org/en/content/articlelanding/2016/cp/c6cp01555g

[12] DiffDock: https://arxiv.org/abs/2210.01776

[13] DockStream: https://jcheminf.biomedcentral.com/articles/10.1186/s13321-021-00563-7

questions:
* What are the authors' thoughts on evaluating the 3D geometry of the generated molecules? PoseCheck [1] and PoseBusters [2] are now common in SBDD papers. The results show that unphysical poses is an ongoing challenge for 3D generative models. Generated molecules with poses that clash with the protein would not typically be considered further.

* The authors state that the metrics do not require the need to consider synthetic ability but synthesizability is a problem highlighted in the Methods section. I agree that wet-lab testing should never be a requirement for computational research but do the authors think there are metrics that can better assess synthesizability beyond SA score and beyond being implicitly considered via similarity to known actives?

* Minor comment: there is a typo in "actives" in figure 2.

[1] https://arxiv.org/abs/2308.07413

[2] https://pubs.rsc.org/en/content/articlehtml/2024/sc/d3sc04185a
</content>
</thread_object>

</thread>
<thread>
creation date:
1730243067047

thread_type:
official_review

<thread_object>
creator: reviewer_hhxu

<content>
summary:
This paper introduces a three part framework to assess SBDD-generated molecules. The first part is similarity between generated molecules and known active compounds, achieved using molecular fingerprints. The second part is a virtual screening metric, a combination of BEDROC and EF, to measure the ability of generated molecules to distinguish between active and inactive compounds. The final part is binding affinity to a target, in which Vina and delta scores, and DrugCLIP were used to achieve this. Results show that amongst the used models, MolCRAFT dominated across multiple metrics.

strengths:
- The problem that this paper attempts to address is one of high importance. Many generative/optimization models in this field rely on computational property predictors, which may not be accurate. The alternative would be to conduct real-world wet-lab experiments, which in many cases, is not feasible. 
- The three proposed assessment techniques capture different aspects of the generated molecules and make for a combination that avoids redundancy. 
- The datasets used (especially the testing set) seem to be satisfactory.

weaknesses:
- The authors state: “This provides a practical alternative to testing deep learning-generated molecules directly in wet-lab experiments.” I believe this is quite a bold statement and, unfortunately, am not fully convinced of this. While the authors provide a neat framework and results, I think more experiments on larger datasets are required to show that this statement holds true. Moreover, it would be much more convincing if the authors are able to demonstrate correlation between their metrics and actual wet-lab results on a subset of compounds. 


- I understand the rationale behind using similarity to known active compounds as an assessment. However, my concern is that this can be limiting – we can generate very desirable molecules that may be extremely different from the active compounds. 



Minor edits:
- Line 118: generated by docking software instead of real complex → complexes
- Line 307: Actually, ligands in PDBbind has a higher docking score → have
- This paragraph needs to be better written: "We refined the PDBbind dataset by excluding complexes with nuclear attachment and inaccurately recorded ligands. Then split into a 9:1 training and validation set. To assess the SBDD model’s generalization across diverse pockets, we removed samples with similar pockets using FLAPP for pocket alignment and similarity assessment. We remove all pockets from the training set with align rate more than 0.6 or 0.9 to the test set pocket. After the removal, the 0.6 version has 12344 pairs remaining while the 0.9 version has 17519 pairs remaining."
- Line 485: We do some visualizations → present 

- Figure 6: Be consistent with wording/capitalization: 
    - Molcraft Generated Mol 
    - Molecule generated by TargerDiff

- Figure 5 captions: include spaces after “(a) ...” and “(b) ...”

questions:
- How would your method evaluate a generated molecule that has a very different structure from known actives, but demonstrates strong binding affinity in docking simulations?

- Can you please clarify why you are so confident that this is a practical alternative to wet-lab experiments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733308881522

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Reviewer-Author Discussion Period

comment:
Dear AC and all reviewers,

We sincerely appreciate your time and efforts in reviewing our paper. During the rebuttal period, the reviewers provided invaluable feedback that allowed us to refine our work and address any areas of potential ambiguity. We sincerely thank the reviewers for their engagement in multiple rounds of discussions, which provided us with valuable insights and learning opportunities. **We are particularly pleased that, following the discussions, all reviewers acknowledged the value of our proposed benchmark framework to the structure-based drug design community.**

Reviewers have recognized the following **merits of our work**:

- **We are tackling a significant problem. [hHxU, pELf, WYJC]**: Proposing new metrics in SBDD is a crucial step toward enabling a more comprehensive evaluation, as it reduces reliance on existing computational property predictors, which are often susceptible to exploitation.

- **Proposed metrics will be valuable/useful to better assess SBDD models. [WYJC, pELf, hHxU, 8aMH]** : We believe that beyond using SBDD-generated molecules directly as drug candidates, they can be applied in more practical ways, such as in ligand-based virtual screening. Our proposed metrics offer a more comprehensive evaluation of the usefulness and effectiveness of current SBDD models across the entire drug discovery pipeline.

- **Our test set, training set, and structure-based splitting method have been positively recognized.[hHxU, pELf]** ： These ensure that our testing also reflects the generalization and transferability of different models. We retrained all models using these datasets to maintain fairness and reliability as a benchmark.


During the rebuttal process, we thoroughly addressed the reviewers’ comments and suggestions by providing detailed explanations, evidence, and incorporating feedback to enhance the manuscript. We demonstrated **strong alignment** between virtual screening metrics and wet-lab performance through examples that highlighted the importance of evaluating SBDD models under diverse deployment strategies. We clarified the rationale for using similarity-based metrics, explaining that **as a benchmarking paper**, our approach **does not constrain chemical space exploration**. We added the test results of an **RL and genetic algorithm based model**, RGA, to enrich our study. Additionally, we emphasized the robustness of our structure-based dataset splitting strategy, which incorporates a **highly diverse and carefully curated test set**, referencing **prior wet-lab studies** and **simulations on underexplored targets** to demonstrate the framework’s applicability, mitigate overfitting risks, and ensure reliability on untested targets, **effectively addressing concerns related to transferability**.

We are confident that all questions and suggestions raised by the reviewers have been thoroughly addressed during the rebuttal period, and our contributions have been explicitly recognized by the reviewers.


Based on the reviewer’s suggestions, we made the following revisions to the manuscript:

- Added more detailed explanations of the reliability of virtual screening and similarity-based metrics, along with examples from real-world studies **[8aMH]**.
- Included results of delta score calculations using Glide XP **[pELf]**.
- Added the Atom Efficiency results for different models as a reference **[pELf]**.
- Moved the FDA similarity metric and results to the appendix, as it might be too strict **[pELf]**.
- Included visualizations of docking poses in the appendix **[WYJC]**.
- Added the RGA method based on genetic and reinforcement learning to the appendix **[WYJC]**.
- Provided a more detailed explanation of Table 1, including the test set and partitioning method **[WYJC]**.
- Fixed various spelling errors and typos **[hHxU]**.
- Fixed some sentences that may cause ambiguity, and make our motivation more clear.

In summary, the **main contribution** of our work lies in proposing a set of metrics to evaluate SBDD models from a broader and more reliable perspective and providing a rigorously partitioned training and test set for fair comparisons. We also conducted comprehensive testing and analysis of current leading 3D-SBDD models. We hope our metrics can fairly assess the applicability of models in the drug discovery pipeline and provide insights from metric comparisons to guide the design of better models in the future. 

We sincerely thank the reviewers for their constructive feedback and thoughtful suggestions, which have significantly enhanced the quality of our paper. We will incorporate these insights into our future research to further contribute to advancements in the field of SBDD. Once again, we deeply appreciate your time and invaluable input.

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a framework for evaluating structure-based drug design (SBDD) based on practical metrics that may be better aligned with the efficacy of the designed molecules in real-world applications.
Reviewers generally agree that the proposed evaluation framework addresses an important challenge in SBDD and has the potential to improve the reliability of the evaluation results.
There have been moderate concerns regarding the novelty of the proposed approach, practical limitations on its expected impact in enhancing SBDD, and the need for clarifications.
The authors' rebuttals have addressed the concerns raised by the reviewers to some extent.