Reframing Structure-Based Drug Design Model Evaluation via Metrics Correlated to Practical Needs
Authors: Bowen Gao
Haichuan Tan
Yanwen Huang
Minsi Ren
Xiao Huang
Wei-Ying Ma
Ya-Qin Zhang
Yanyan Lan
Venue: ICLR 2025 Poster (2025)
Keywords: Stucture-Based Drug Design
Model Evaluation
Benchmark
Primary area: datasets and benchmarks

Abstract: Recent advances in structure-based drug design (SBDD) have produced surprising results, with models often generating molecules that achieve better Vina docking scores than actual ligands. However, these results are frequently overly optimistic due to the limitations of docking score accuracy and the challenges of wet-lab validation. While generated molecules may demonstrate high QED (drug-likeness) and SA (synthetic accessibility) scores, they often lack true drug-like properties or synthesizability. To address these limitations, we propose a model-level evaluation framework that emphasizes practical metrics aligned with real-world applications. Inspired by recent findings on the utility of generated molecules in ligand-based virtual screening, our framework evaluates SBDD models by their ability to produce molecules that effectively retrieve active compounds from chemical libraries via similarity-based searches. This approach provides a direct indication of therapeutic potential, bridging the gap between theoretical performance and real-world utility. Our experiments reveal that while SBDD models may excel in theoretical metrics like Vina scores, they often fall short in these practical metrics. By introducing this new evaluation strategy, we aim to enhance the relevance and impact of SBDD models for pharmaceutical research and development.

Published as a conference paper at ICLR 2025
REFRAMING STRUCTURE-BASED DRUG DESIGN MODEL EVALUATION VIA METRICS CORRELATED TO PRACTICAL NEEDS
Bowen Gaol22, Haichuan Tan Yanwen Huang', Minsi Ren', Xiao Huang,, Wei-Ying Ma'. Ya-Qin Zhang'. Yanyan Lan
IIstitute for AI Industry Research (AIR), Tsinghua University
"Department of Computer Science and Technology, Tsinghua University
Department of Pharmaceutical Science, Peking University
*Department of Artificial Intelligence, Westlake University $ Coolege of Intelligence and Computing, Tianjin University
Beijing Academy of Artificial Intelligence (BAAI)
ABSTRACT
Recent advances in structure-based drug design (SBDD) have produced surprising results, with models often generating molecules that achieve better Vina docking scores than actual ligands. However, these results are frequently overly optimistic due to the limitations of docking score accuracy and the challenges of wet-lab validation. While generated molecules may demonstrate high QED dtrug-likeness and SA (synthetic accessibilityy scores, they often lack true drug- like properties synthesizability. To address these limitations, we propose a model-level evaluation framework that emphasizes practical metrics aligned with real- weal-world applications. Inspired by recent findings on the utility of generated molecules in ligand- based vir- tual screening, framework evaluates SBDD models by their ability produce molecules that effectively retrieve active compounds from chemical libraries via similarity-based searches. This approach provides direct indication of therapeutic potential, bridging the gap between theoretical performance and real-world utility. Our experiments reveal that while SBDD models may excel in theoretical metrics like Vina scores, they often fall short in these practical metrics. By introducing this new evaluation strategy, we aim to enhance the relevance and impact of SBDD mod- els for pharmaceutical research and development. Code and data are available at https: wow hhw.://owen-eee httpa//oooooooooag//////aaaaaaaa
INTRODUCTION
The field of Structure-Based Drug Design (SBDD) has experienced remarkable advancements in recent years, with the development of models such Pocket2Mol (Peng al. 2022). TargetDifff (Gunn et al. [222). and MoICRAFT (Qu et al. 2024). These models are at the forefront of enhancing the efficiency and precision of drug discovery by generating molecules designed effectively bind to specific protein pockets. Despite these technological strides, the practical application of these models drug development remains formidable challenge. The crux of this challenge lies the verification of their efficacy, which is complicated by difficulties in synthesizing and testing these molecules laboratory settings.
The Vina docking score (Eberhardt et al. 2021 Trott & Olson 2010) is the standard metric used to estimate the binding abilities of molecules generated by SBDD models. provides an estimate based on empirical formula, serving
proxy for binding affinity. Studies have shown that SBDD models can generate molecules with Vina docking scores outperforming reference ligands (Guan)
al.
2023 2024
Qu al. 2024), suggesting significant potential in the field. However, its reliability
"Equal contribution
1 "Coresponnence lanyanyan@ai r. attiinguaaeet edu..c

---

Published as a conference paper at ICLR 2025
is increasingly questioned. As shown in Figure Vina scores can be inflated by simply increasing the number of atoms in a molecule or manipulated by various factors, as discussed in Appendix E| revealing susceptibility to overfitting. This suggests that reliance these easily manipulated metrics can lead to overly optimistic model evaluations. Additionally, as noted by Gao et al. (2024). recent advances in modeling have improved Vina scores, but estimates of specific binding ability, such as the delta score, remain unchanged or even worse, still falling short of reference ligand performance. Guo al. (02II) also done experiments showing that some docking algorithms are not even correlated with binding affinity.
Furthermore, the practical synthesis of molecules generated by current SBDD models often proves to be complex and unfeasible, which significantly impedes their validation in wet-lab experiments (Bradshaw et al. 2019 Gao & Coley 2020). This challenge, compounded by the long-term reliance on flawed metrics such as the Vina docking score, has led to notable shortcoming in SBDD disconnect from practical applications. This gap evident as the outputs of current SBDD models are theoretically promising, but prove challenging to utilize effectively in real-world settings. To
ID: jak2
D: abll
ID: pl1
-18
-16
Vina Dock Scove
-34-
-12
-10
-10
-4
1.0
0.6
08-
0.b
013
0.5
0.4
03
0.
20
20
Number
Figure 1: The relationship between Vina Dock Score/QED and number of atoms
address these critical shortcomings in the field of SBDD, we propose a new evaluation framework that extends beyond traditional theoretical estimation-based metrics. Recent attempts to incorporate SBDDDgenerated molecules into practical drug discovery processes have involved modifying these
molecules into existing or more easily synthesizable forms (Moret al. these molecules as reference templates for virtual screening (Shen et al. 2024 or conduct virtual
2023). Others have used
screening using characteristics like scafford of functional groups of the generated molecules (Bo let al. 2024) Inspired by these successful approaches, our proposed evaluation framework includes several new metrics designed to directly assess the practical usability and deployment capabilities of SBDD models from a way that more directly reflects the success rate of wet-lab experiment.
Our framework assesses three levels of evaluation. First, it evaluates the similarity of generated molecules to known active compounds, gauging their potential to be modified into viable drug candidates. Second, introduces a virtual screening-basse metric that directly measures the practical deployment capabilities of these molecules. Third, it continues to consider the estimated binding affinity, albeit in a more nuanced and critically evaluated manner.
By successfully meeting our proposed metrics, an SBDD model is more likely to produce molecules that are not only theoretically effective but also therapeutically viable in real-world drug discovery Pettings.Notably our metrics for using generated molecules in virtual screening directly correlate with and reflect the success rates of wet lab experiments. This approach aims bridge the significant gap between theoretical SBDD models and their practical application in the pharmaccutical industry, paving the way for more reliable and efficacious drug development processes.
We conducted extensive experiments using our dataset, which includes data derived from real crystal structures and is divided based on local structural similarities of the pockets, to train and evaluate major SBDD models. Our results show that, from practical deployment perspective, the molecules generated by current models fall significantly short of matching the quality of reference ligands. Despite achieving high Vina scores, their practical usability metrics reveal substantial gap. Our
2

---

Published as a conference paper at ICLR 2025
proposed evaluation pipeline is designed to help bridge this gap, offering direction that could enhance the practical applicability of future SBDD models.
2 RELATED WORK
Structure-Baad Based Drug Design involves generating small molecules with potential biological activity for a given protein pocket (Zhang et al. 2023). Several evaluation metrics have been designed to evaluate SBDD models. (Du al. (224))raadtiiiii Traditionally, biological activity is estimated using AutoDock Vina (Trott& & Olson 2010), a widely used docking software designed to predict the preferred binding orientation of small molecule (ligand) when bound to a larger protein (receptor) target.
Recently, with the development of deep generative models, several representative models have emerged. These include autoregressive models like AR (Luo et al. (022]) and Pocket2Mol (Peng et al. 2022), diffusion-based models like Targetdiff (Guan et al. 2023). and the newly developed Bayesian Flow Network model, MoICRAFT (Qu al. 2024).
Typically, these models are trained and tested using the CrossDocked dataset (Francoeur et al. 2020), which is constructed by cross-docking protein-small molecule pairs from the PDBbind dataset. However, this benchmark dataset presents several issues. First, the structures in CrossDocked are generated by docking software instead of real complexes, which may be inaccurate and cannot fully reflect the real interaction pattern. Secondly, the data selection in CrossDocked relies docking software, which leads to bias in the dataset, as the ligands selected for the training data tend to be favored by the docking software. Third, observed that many existing models use the test set validation set to select checkpoints during training, which risks data leakage. Given these concerns, it important to adopt a new dataset for more reliable evaluation of SBDD models.
METHODS
3.1
EVALUATION METRICS THAT REFLECT PRACTICAL NEEDS
Despite recent advancements in deep leamning- based generative models for drug design somig surpassing reference ligands in Vina docking scores -hhrer practical application in pharmaceutical settings remains limited. The ultimate goal is to generate drug-like molecules that specifically bind to intended targets, but current structure-based drug design (SBDD) models fall short in real-world applications.
some even
significant limitation lies in current evaluation metries, which fail to accurately assess a model's effectiveness in generating useful molecules. Most of the previous SBDD models published in major machine learning conferences primarily relied on Vina docking scores, biased toward molecules that achieve high scores yet are less effective practically. For example, shown in Table I Vina scores are less predictive than other methods for virtual screening. Additionally, Figure I demonstrates that merely increasing molecular size can inflate Vina scores while decreasing Quantitative Estimate of Drug-likeness (QED) values, revealing a vulnerability to overfitting. We also find vina score can be overfitted with more Hydroxyl groups(-OH), less percentage of Nitrogen and Oxygen atoms, and more Halogen atoms. Details can be found at Appedix E More importantly, despite improvements in Vina docking scores, the delta score a crucial metric for assessing specific binding capability still significantly lags behind that of reference ligands. This discrepancy could misdirect model devel- opment away from practical needs, underscoring the inadequacy of relying solely on Vina docking scores and highlighting the necessity for more relevant and accurate evaluation criteria.
Another challenge is the synthetic feasibility of molecules generated by deep learning approaches. Although these molecules often achieve high synthetic ability (SA) scores, studies (Bradshaw et al.) 2019 Gao & Coley] (2020) indicate they are frequently difficult to synthesize in practice, hindering wet-lab lab validation and application. This suggests that current theoretical metrics are impractical and do not accurately reflect the ability of models in terms of generating useful molecules.
To address these limitations, propose reevaluating how model outputs assessed. Instead of relying on theoretical coimatiionbbsse metrics, we advocate for metrics that directly reflect the usefulness of generated molecules without the need to consider synthetic ability.
3

---

Published as a conference paper at ICLR 2025
Recent research has involved successfully manually modifying generated molecules into synthesizable structures for wet-lab validation, drawing on the expertise of medicinal chemists who typically adjust molecules based actives targeting the same biological structures (Moret et al. 2023). Building this approach, we suggest shifting the evaluation paradigm. Rather than solely aiming for molecules ready for direct wet-lab experiments and binding affinity estimation, we should assess their potential to resemble active compounds. This perspective focuses on the feasibility of transforming generated molecules into practically useful compounds. A similarity or distance-based metric could then be employed to gauge this practical utility, reflecting more realistic and applicable measure of generated molecule's 's value drug development.
Furthermore, leveraging the capability of generative models in virtual screening to serve as templates for identifying similar compounds has shown promise, achieving significant hit rates (Shen et al. 2024). Thus, we suggest adding virtual screening metric to evaluate how well generated molecule can discriminate between active and inactive compounds, providing a direct measure its utility in drug discovery.
summary, as shown in Figure 2 deep learning-based generative models in drug design are structured across three levels:
our proposed evaluation metries for assessing the effectiveness of
1. Similarity To Know Actives:
These metrics evaluate the potential for modification and optimization of generated molecules. They assess how closely these molecules resemble known active compounds, facilitating easier synthesis and optimization.
2. Virtual Screening Ability: These metries determine the ability of generated molecules to distinguish between active and inactive compounds. This is crucial for identifying potential drug candidates that are more likely to succeed later stages of drug development.
3. Binding A Affinity Estimation: These metrics theoretically estimate the binding capabilities generated molecules to target structures. Beyond the conventional vina docking score, we also include other score functions.
Binding Affinity Estimation
Similarity-based Metrics
Soace
Delta score: 1.00
Active
Simbarty to Known Actvos
DrugClpp Encodors
b 0.42
N 0.09
e m-d
DrugGlip score: 0.02
c. Virtual Screening Ability Metrics
Screening
0g Screening
Input 100
Find Bimilar Mols
Good
EF 25.28
REDROCS2.RR
Database
decoys
Og Virtual Screening
Output
Figure 2: Our three-level evaluation metrics include: (a) Binding affinity estimation, which encom- passes the Vina docking score, delta score, and DrugCLIP score; (b) Similarity-based metrics that assess the resemblance between generated molecules and known actives; (c) Virtual screening ability metrics that evaluate the capability of the generated molecules to differentiate between actives and decoys when used as reference templates.
We introduce the details of those metrics.
Similarity to Known Drugs and Actives Our framework maps molecules into feature space using molecular fingerprints, specifically 2D Extended Connectivity Fingerprints (ECFP) and 3D Extended dimenssoil Fingerprints (E3FP), which are 1024-dimensional bit vectors. We also use deep learning- baased encoders like Uni-Mol (Zhou et al., 2022) and the DrugCLIP (Gao et al. 2023] molecular encoder, which aligns with binding pockets.
4

---

Published as a conference paper at ICLR 2025
Given target with N known actives d1, :2,...;; aN the Active Similarity score is:
Active Similarity(1 iE(1,2.....)
max (a(1) o(a)).
The underlying rationale for the similarity-based metrics is that similar molecules show similar binding behaviour(Bostróm et al., 2006), and for a generative model be considered effective, must be capable of generating molecules that are at least similar to one of the known active compounds. Furthermore, to ensure that the generated molecules exhibit "drug-like" properties, they should also demonstrate similarity to one or more known approved drugs.
Table 1: Results from virtual screening using docking software, and using real ligands as templates for similarity searches. Similarity is determined through various molecular fingerprints and deep learning encoders. BEDROC and EF@ are metrics of vitrual screening. The results demonstrate that using real ligands as templates for virtual screening yields better outcomes compared to virtual screening with the docking software Vina.
Category
Method
BEDROC 1
EF@
Glide
40.7
16.18
Docking
Vina
7.32
2D Fingerprints
39.32
24.95
3D Fingerprints
23.77
14.30
Ligand-basse baaad virtual screening
Uni-Mol Encoder
13.39
7.48
DrugCLIP Encoder
45.43
29.23
Virtual Screening Ability Relying solely on similarity- based metrics remains insufficient for comprehensive evaluation. In real-world drug discovery, known active compounds often have decoys that are structurally similar or share properties but are ineffective against the target. Therefore, if model predominantly generates decoys that exhibit high similarity active compounds, it cannot be considered effective. To address this, it is crucial to not only assess the similarity of generated molecules known actives but also evaluate the model's 's ability distinguish between actives and decoys. This ensures the model generates therapeutically relevant molecules rather than misleadingly similar but inactive compounds, providing more accurate measure of its true potential in drug discovery.
Thus, we introduce our virtual screening- based metries. Using the molecules generated by our models as references, we retrieve similar compounds from compound library and evaluate the accuracy identifying known actives. The library includes experimentally validated actives and decoys- structtrruul analogous actives but experimentally confirmed lack binding affinity. This makes the task of distinguishing actives from decoys particularly challenging. Additionally, we leverage different encoders for virtual screening, offering flexible approach better assess model performance across diverse molecular representations.
Specifically, we use the BEDROC and EF metrics to evaluate the effectiveness of virtual screening. BEDROC incorporates exponential weights that assign greater importance to early rankings. In the context of virtual screening, the commonly used variant is BEDROCss. where the top 2% of ranked candidates contribute to 80% the BEDROC score. The formal definition is shown in equation] [] where NTB, is the number of true binders in the top a%. Enrichment Factor (EF) also a widely NTB. used metric, calculated as EF NTB, xa where NTB, is the total number of binders in the entire screening pool. Table [ demonstrates that using real ligands as templates for virtual screening yields better results compared to virtual screening with docking software, validating the reliability of our proposed metric.
ENTB, R.
-0r/N
R. sinh(a/2) cosh(a/22 cosh (a/2 - ala) 1 co(1-Ra) (1)
BEDROC,
The virtual screening capability Notably, unlike the Vina docking score, which a theoretical estimation and shows poor correlation with actual binding affinity, the enrichment factor from virtual screening is directly correlated
evaluate SBDD models is central to proposed metrics.
5

---

Published as a conference paper at ICLR 2025
with, even equivalent to, the real hit rate in wet lab experiments. This means our metric aligns with practical needs and directly reflects the success rate of SBDD.
Binding Affinity Estimation Although the focus of this paper not the theoretical estimation of binding affinity, we still provide more comprehensive metrics to do the estimation. In this benchmark, we continue use Vina docking scores as one of the metrics to reflect the binding estimation. In addition to the conventional docking score, we also use the delta score proposed by (Gao et al., (2024) which provides a good estimation the specific binding ability of generated molecules. It provides an unbiased evaluation of whether the generated molecules possess structures that specifically contribute to binding the desired target, rather than structures that overfit the docking software to achieve high docking scores across all targets. To be specifically, for each target yi and the generated molecules Tij-J j E 1,2,.... .... m,- While the docking score is calculated as S(Zij. Vk). after random sample another target yku the delta score is calculated as:
Delta Score (1)) = m j=1
Σ
((25) U1) + S(zij 8k)),
for each i, we sample &s E {1,2..... n. with k +
To obtain a more accurate delta score with improved docking scores, we utilized both Glide SP and Glide XP (Friesner et 2006) for calculating the delta score. Glide XP is a more precise docking method compared to Glide SP, but it is also more time-consuming.
Beyond docking scores, we also employ machine learning- scoring function. DrugCLIP Gao et al. (202)) has demonstrated outstanding performance in virtual screening, making it a valuable evaluation metric for assessing the binding potential of generated molecules.
3.2 TEST DATASET
Previous work primarily tested models on the CrossDocked (Francoeur et al. 2020 test set, which has several limitations. It is randomly selected, lacks diversity checks, and is derived from synthesized data rather than real-world crystal structures. Additionally, each pocket in CrossDocked is paired with only one ground truth ligand, even though a single pocket can bind to multiple different ligands.
To address these limitations, propose new test set for SBDD models that created from Mysinger et al.] (2012). Our test set comprises 101 targets, with accurately recorded ligand and protein files, and includes diverse range of protein types as shown in the Appendix Each target is supplemented with significant number of actives and decoys, facilitating the evaluation of similarity and distance-based scores as well as virtual screening performance. A robustly generated molecule should effectively distinguish between actives and decoys. On average, each target in our test set contains 224.4 actives and 50 decoys for each active,The actives and decoys used have been validated through wet-lab experiments. Previous studies have demonstrated that virtual screening methods performing well on these targets can achieve matching results in real-world wet-lab experiments (Wang et al. 2024 Jia et al. 2024), indicating that virtual screening metrics can effectively align with real-world wet-lab performance.
3.3 TRAINING AND VALIDATION DATASET
Existing Models are commonly trained and tested on the CrossDocked (Francoeur et al. 2020) dataset, produced by docking software, which consists of synthesized protein-ligan complexes. In contrast, our benchmark utilizes real protein-ligand conformations from experimental crystal structures in PDBbind (Wang et al. 2004). Actually, ligands in PDBbind have higher docking score as well as delta score for specific binding ability. Details shown in Appendix G
We refined the PDBbind dataset by excluding complexes with nuclear attachment and inaccurately recorded ligands. Then split into a 9:1 training and validation set. To assess the SBDD model's generalization across diverse pockets, we removed samples with similar pockets using FLAPP for pocket alignment and similarity assessment. FLAPP(Sankar et al., (022) (Fast Local Alignment of Protein Pockets) is tool used to estimate the structural similarity (alignment rate) between two pockets. We calculated all the FLAPP scores between the training set and test set pockets and removed all pockets from the training set with FLAPP score greater than 0.6 0.9 relative to any

---

Published as a conference paper at ICLR 2025
test set pocket. After the removal, the 0.6 version has 12344 complex-ligandd pairs remaining while the 0.9 version has 17519 pairs remaining. Details can be found the Appendix G
EXPERIMENTS
4.1
TESTED MODELS
We select representative deep learning-based models for structure-based drug design evaluation. For voxel-grid based model, we use LiGAN (Ragoza et al. 2022). For autoregressive models, we choose AR (Luo et al. 2021) | and Pocket2Mol (Peng al. (022). For diffusion models, select Targetdiff (Guan et al. 2023). Additionally, MoICRAFT (Qu et al. 2024) is included as a generative model based the newly developed Bayesian flow network.
4.2 RESULTS OF BINDING ABILITY ESTIMATION
Table 2:
Evaluation Results for binding ability estimation. Results for Reference Ligands and best results are shown in bold text. Atom Efficiency defined by Vina score numbbrr of heavy atoms, for reference only.
Vina
Delta Score f
DrugCLIP score 1
Docking Score
Atom Efficiency
Glide SP
Glide XP
Reference Ligand
-9.633
-0.349
2.686
3.509
0.508
LiGAN
-5.175
-0.503
0.037
0.065
-0.016
AR
-7.255
-0,464
0.483
0.694
0.009
PDBhind 60
Pocket2Mol
-7.640
-0.479
0,531
0.553
-0.005
TargetDiff
-9.622
-0.283
0.325
0.099
MolCRAPT
-9.788
-0.364
0,973
1.301
0.145
LiGAN
-6.577
-0.457
0.107
-0.000
AR
-7.340
-0,446
0.523
0.007
PDBbind 90
Pocker2Mol
-8.195
-0.481
0.599
-0.006
TargetDiff -9.711
-0,287
0.238
0.095
MolCRAFT -9.778
-0,363
1.163
0.173
We first show the metrics that relevant theoretically binding affinity estimation. In Table 2 it is evident that both TargetDiff and MolCRAFT outperform the reference ligand in terms of average docking scores. However, when considering the Delta Score and DrugCLIP score, they lag significantly behind the reference ligand. Specifically, MoICRAFT, while the best performer among the evaluated methods, still shows a considerable disparity in Delta Score (0.973 vs. 2.686) and DrugCLIP score (0.173 vs. 0.508) when compared to the reference ligand. It is noteworthy that although TafgetDiff achieves a competitive docking score, its Delta Score inferior to those of aitoregressive-basee methods. This suggests that TargetDiff's high docking score might be the result of overfitting large atom numbers.
4.3 RESULTS OF SIMILARITY BASED METRICS
The evaluation results using similarity- based metries presented in Table While MoICRAFT excels in generating molecules that closely resemble known active compounds for specific targets, all models, along with other methods tested, significantly underperform compared the reference ligand. This suggests that current structure-based drug design (SBDD) models still lack effective conditional generation capabilities, highlighting a key area for further development. The distribution plots for amilarity-basee metrics across all targets in DUD-E are shown in Appendix Appendix[F.11F]
4.4 RESULTS OF VIRTUAL SCREENING-BASEE METIRCS
The results for virtual screening ability metrics are presented in Table 4 MoICRAFT outshines all other methods, regardless of the feature extractor utilized. Nonetheless, it underperforms when benchmarked against reference ligands. Optimistically, using the DrugCLIP molecular encoder for encoding generated molecules for similarity
virtual screening enables MolCRAFT to achieve a virtual screening efficacy comparable to that Vina, with an enrichment factor of 5,549 versus Vina's 7.32 (Table
[]. This result is particularly encouraging the speed of virtual screening

---

Published as a conference paper at ICLR 2025
Table 3: Evaluation Results for Similarity-Bann -Baeed Metires on real active molecules. Results for Reference Ligands and best results are shown in bold text.
2D Fingerprints
3D Fingerprints
Uni-Mol
DrugCIII
Reference Ligand
0.588
0.230
0.973
0.870
LiGAN
0.131
0.109
0,922
0,366
AR
0.157
0.122
0,949
0,449
PDBbind
Pocket2Mol
0.179
0.143
0,946
0.414
60
TargetDiff
0.169
0.143
0.958
0,478
MolCRAFT
0.208
0.161
0.965
0,522
LiGAN
0.136
0.112
0,931
0.397
AR
0.161
0.125
0.951
0.468
PDBbind 90
Pocket2Mol
0.185
0.146
0,945
0.419
TargetDiff 0.169
0.142
0.958
0,477
MolCRAFT 0.214
0.163
0.966
0.547
Table 4: Evaluation Results for Virtual Screening-Basse Metires. Results for Reference Ligands and best results are shown in bold text.
2D Fingerprints BEDROC
3D Fingerprints BEDROC
Uni-Mol
DrugCLIP
EF
EF
BEDROC
EF
BEDROC
EF
Reference Ligand
39.32
24.95
23.77
14.30
13.39
7.48
45.43
29.23
LiGAN
2,306
1.079
2.216
1.007
2.024
0.837
1.581
0.655
AR
4.938
2.567
4.407
2.196
3.266
1.501
3.698
1.796
PDBbind
Pocket2Mul
5.976
3.054
4.192
2.047
2.335
1.033
3,667
1.827
TargetDiff
4.062
1.957
3.747
1.763
2.995
1.340
4,400
2.260
MoICRAFT
7.584
3,953
5,521
2.792
4.868
2.357
7.265
3.968
LiGAN
2.240
1.004
2.610
1.183
2.151
0.932
1.450
0.636
AR
4.946
2.522
4.156
2,053
3.115
1.384
4,223
2.139
PDBbind 90
Pocket2Mol
6,277
3.240
4.359
2.174
2.970
1.312
3.215
1.541
TargetDDiff
4.431
2.174
3.888
1.861
3.967
1.828
4.122
2.117
MolCRAFT
9.032
4.825
6.423
3.284
5.299
2.456
9.782
5.549
with generated molecules is significantly faster than using docking software to dock all candidate compounds. The distribution plots for the enrichment factor of all targets in DUD-E across various models are displayed in Figure 3
Enrichment Factor
Enrichment Foctor
PacketZMal TargetDef
Packet2MMl TorgetDiff
(a) Morgan Fingerprints
ECFP3 Fingerprints
Figure 3: Distribution plots for Virtual Screening results on all targets in DUD-E with different models.
4.5 ANALYSIS
4.5.1
INSIGHTS REGARDING PERFORMANCE DIFFERENT MODELS
Figure Apresents radar plots comparing the performance of different models across various metrics in evaluation. The outer circle represents the metrics of reference ligands. It is evident that
8

---

Published as a conference paper at ICLR 2025
Split FLAPP 0.6
Split of FLAPP 0.9
FDA Sim.
80% of Pal.
FDA Sim.
40% Rel.
Spore
Acthess of Rat.
Deta Beore
ActiesySren
Pocket2Mol
BEGROC
VS
BEDROC 30%
Vi eF
30% of Ret
25%
25%
Figure 4: Radar plot shows the performance of different methods on part of our multifaceted metrics. Detailed definition and results refer to TDA--imilarity can be found at Appendix A
MolCRAFT outperforms other methods. Additionally, all methods fall significantly short of the reference ligands, even though some points the outer circle do not represent the full value of the metrics for reference ligands-such as the virtual screening enrichment factor, which is only 25% of its actual value. Despite this, the models still perform far below the reference ligands.
While achieving impressive Vina docking scores, Tifge overall performance is lacking. The generated molecules exhibit poor delta scores, low similarity to known active compounds, and struggle to distinguish between active and decoy molecules, even when compared to simpler autoregressive methods. This points potential flaw in the direction of diffusion- based SBDD models, which, despite their popularity (Guan et al. 2024,] Huang et al. 2024), may primarily overfit to Vina scores. The ease of generating large molecules inflates docking scores but fails to translate into truly useful relevant molecular structures for drug discovery.
Pocket2Mol ranked second-best for similarity to known actives and virtual screening using fingerprint- based extractors, its performance declines with deep encoder-based methods. The autoregressive nature limits its ability to generate larger molecules. Despite this, its good performance on vir- tual screening indicates that Pocket2Mol can generate valuable substructures or functional groups, though smaller molecules impact its docking scores negatively. We present some visualizations on Pocket2Mol's ability generate valuable substructures or functional groups in Appendix D
MoICRAFT ranks as a top performer across multiple metrics and excels in virtual screening. achieves results comparable to Vina docking but with greater speed, demonstrate that using SBDD models to generate molecules, then use those molecules as reference for virtual screening to find potential active molecules is practical alternative to directly the generated molecule to do wet-lab experiment.
Despite achieving higher docking scores, current models still fall short on other proposed metrics such as delta score, similarity to known actives, and virtual screening ability compared to reference ligands. Relying heavily on Vina docking scores can lead an overly optimistic view of the effectiveness of SBDD models. Interestingly, because SBDD models are often judged by their ability to generate molecules with high docking scores, many have shifted focus toward overfitting these scores. For instance, some models use trained Vina predictors to guide sampling, prioritizing the hacking of Vina scores rather than generating reliable and actual effective molecules. This approach can misguide development of SBDD models. We strongly advocate for evaluating SBDD models using metrics that are more practically relevant to drug discovery, rather than solely depending on inaccurate theoretical scoring functions that easy to be hacked overfitted.
4.5.2 ANALYSIS ON UNBIASED METRICS
In Figure we demonstrate that theoretical estimation-based metrics, such the Vina score and QED, are prone to overfitting due to their high correlation with molecular properties like atom count. To investigate whether proposed metrics exhibit similar behavior, conduct comparative analysis. As shown Figure metrics such as delta score, similarity to known actives, and virtual screening performance show correlation with atom count, indicating their robustness against.

---

Published as a conference paper at ICLR 2025
Moen Simillaiit
Delta Score
Number Atoms
Number Atoms
Number of Atoms
(a) delta score
(b) virtual screening
(c) similarity to known actives
Figure 5: Plots that show the relevance between the atom numbers and different metrics, including (a) delta score (b) virtual screening ability (c) similarity to known actives.
such biases. Note that as our metrics are based on similarity between molecules, rather than empirical formulas, they are less vulnerable to overfitting manipulation by specific factors.
4.6 VISUALIZATION OF PROPOSED METRICS
Goee Arlin
Moaa pawesoyyyy
Milrnsh
Scoe -656
CED
Acrie Simeerte 042
S 0.00
Case
Case 2
Figure 6: Cases that show the importance of using similarity-base bbasdd metrics evaluate the effective- ness of generated molecules
We do some visualizations to show the reason and importance of using metrics. Figure shows a molecule generated by MoICRAFT that, although diverging from the reference ligand, aligns with a known active of target AOFB, achieving high Morgan fingerprint similarity of 0.42 but receiving a bad Vina score of -6.56. This example underscores the necessity of using more practical metries, rather than solely relying Vina scores, and emphasizes the importance of considering all known actives in addition reference ligands for comprehensive model evaluation.
In Figure ()).. molecule from the TargetDiff model exhibits significant similarity to molecule on the FDA-approved list, despite its low QED score. This case demonstrates the significance of using similarity as metric. For details and definition the FDA-approved list and the definition of similarity, please refer to Appendix A This suggests its potential as a drug candidate through targeted modifications. This case highlights the importance of our practical related metries for more comprehensive assessment of drug- drug-likeness in generated molecules. More visualizations to show the necessity of our proposed metric be found in Appendix D
4.7 CONCLUSION
In this paper, we present a new benchmark of current Structure- -ased Drug Design (SBDD) models with metrics that are related practical needs. Unlike previous evaluations that primarily relied on Vina docking scores to estimate the efficacy of generated molecules, we advocate for assessing these models from a practical deployment perspective. Our findings reveal that while current methods often outperform reference ligands in terms of docking scores, they fall significantly short when evaluated using the more practical correlated metrics we propose. This discrepancy not only highlights the over-optimism in previous evaluations but also provides valuable insights reassessing the value of current SBDD models. We hope our evaluation framework will help bridge the gap between generative models and their practical applications.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by Beijing Academy of Artificial Intelligence (BAAI).
REFERENCES
Weichen Bo, Yangqin Duan, Yurong Zou, Ziyan Ma, Tao Yang, Peng Wang, Tao Guo, Zhiyuan Fu, Jianmin Wang, Linchuan Fan, et al. Local scaffold diversity-come contribuued generator for discovering potential nlrp3 inhibitors. Journal of Chemical Information and Modeling, 64(3):737-748, 2024.
Jonas Boström, Anders Hogner, and Stefan Schmitt. Do structurally similar ligands bind in a similar fashion? Journal of medicinal chemistry. 49(23):67 16-672,, 2006.
John Bradshaw, Brooks Paige, Matt J Kusner, Marwin Segler, and José Miguel Hernández-Lobatt model to search for synthesizable molecules. Advances in Neural Information Processing Systems, 32, 2019.
Yuanqi Du, Arian Jamasb, Jeff Guo, Tianfan Fu, Charles Harris, Yingheng Wang, Chenru Duan, Pietro Liò, Philippe Schwaller, and Tom L Blundell. Machine learning-aided generative molecular design. Nature Machine Intelligence. pp. 1-16, 2024.
Jerome Eberhardt, Diogo Martins, Andreas F Fiillack, and Stefano Forli. Autodock vina 1.2. New docking methods, expanded force field, and python bindings. Journal of chemical information and modeling, (188)3891-3899) 2021.
Paul G Francoeur, Tomohide Masuda, Jocelyn Sunseri, Andrew Jia, Richard lovanisci, Ian Snyder, and David R Koes. for structure-based drug design. Journal of chemical information and modeling, 60(9):2200444 4215, 2020.
convolutional neural networks and cross-docked data set
Richard A Friesner, Robert B Murphy, Matthew P Repaskyy Leah L Frye, Jeremy R Greenwood, Thomas A Halgren, Paul C Sanschagrin, and Daniel Mainz. Extra precision glide: Docking and scoring incorporating model of hydrophobic enclosure for protein- ligand complexes. Journal of medicinal chemistry, 49(21xx117-6119) 2006.
Tianfan Fu, Wenhao Gao, Connor Coley, and Jimeng Sun. Reinforced genetic algorithm for structure- based drug design. Advances in Neural Information Processing Systems, 12s12222223333 2022.
Bowen Gao, Bo Qiang. Haichuan Tan, Yinjun Jia, Minsi Ren, Minsi Lu, Jingjing Liu, Weig Ma, and Yanyan Lan. Drugelip: Contrasive protein-molecute molecule representation learning for virtual screen- ing. In NeurlPS 2023, 2023.
net www.mm www.oruuuunnddtaccccccccecec
Bowen Gao, Minsi Ren, Yuyan Ni, Yanwen Huang, Bo Qiang. Zhi-Ming Ma. Wei-Ying Ma, and Yanyan Lan. Rethinking specificity in sbdd: Leveraging delta score and energy diffusion. arXiv e-prints, pp. arXiv-2403, 2024.
Wenhao Gao and Connor W Coley. The synthesizability of molecules proposed by generative models. Journal of chemical information and modeling, 60(12):5714-522 2020.
Jiaqi Guan, Wesley Wei Qian, Xingang Peng, Yufeng Su, Jian Peng, and Jianzhu Ma. 3d equivariant diffusion for target-aware molecule generation and affinity prediction. In International Conference on Learning Representations, 2023.
Jiaqi Guan, Xiangxin Zhou, Yuwei Yang. Yu Bao, Jian Peng, Jianzhu Ma, Qiang Liu, Liang Wang, and Quanquan Gu. Decompdiff: diffusion models with decomposed priors for structure-basee drug design. arXiv preprint arXiv:2403.07900 2024.
Jeff Guo, Jon Paul Janet, Matthias R Bauer, Eva Nittinger, Kathryn A Giblin, Kostas Papadopoulos, Alexey Voronov, Atanas Patronov, Ola Engkvist, and Christian Margreitter. Dockstream: a docking wrapper to enhance de molecular design. Journal of cheminformatics, 13:1-21, 2021.
Zhilin Huang, Ling Yang, Xiangxin Zhou, Zhilong Zhang, Wentao Zhang, Xiawu Zheng, Jie Chen, Yu Wang, Bin CUI, and Wenming Yang. ligand interaction prior for binding-aware 3d molecule diffusion models. In International Conference on Learning Representations, 2024.
11

---

Published as a conference paper at ICLR 2025
Yinjum Jia, Bowen Gao, Jiaxin Tan, Xin Hong, Wenyu Zhu, Haichuan Tan, Yuan Xiao, Yanwen Huang, Yue Jin, Yafei Yuan, et al. Deep contrastive learning enables genome-wide virtual screening. bioRxiv, pp. 2024-09, 2024.
Shitong Luo, Jiaqi Guan, Jianzhu Ma, and Jian Peng. 3d generative model for structure- based drug design. Advances in Neural Information Processing Systems, 016622966399 2021.
Michael Moret, Irene Pachon Angona, Leandro Cotos, Shen Yan, Kenneth Atz, Cyrill Brunner, Martin Baumgartner, Francesca Grisoni, and Gisbert Schneider. Leveraging molecular structure and bioactivity with chemical language models for de novo drug design. Nature Communications, 14(1):114, 2023.
2023.
Michael M Mysinger, Michael Carchia, John J Irwin, and Brian K Shoichet. Directory of useful decoys, enhanced (dud-e): better ligands and decoys for better benchmarking. Journal of medicinal chemistry, 55(14):6582 6594, 2012.
Xingang Peng, Shitong Luo, Jiaqi Guan, Qi Xie, Jian Peng, and Jianzhu Ma. Pocket2mol: Efficient molecular sampling based on 3d protein pockets. In International Conference Machine Learning. pp. 17644-17655. PMLR, 2022.
Yanru Qu, Keyue Qiu, Yuxuan Jingjing Gong, Jiawei Han, Mingyue Zheng, Hao Zhou, and Wei-Ying Ma. Molcraft: Structure-based drug design in continuous parameter space. arXiv preprint arXiv:2404. 12141, 2024.
Matthew Ragoza, Tomohide Masuda, and David Ryan Koes, Generating 3D molecules conditional on receptor binding sites with deep generative models. Chem Sci, 13:2701-2713, Feb 2022. doi:
Santhosh Sankar, Naren Chandran Sakthivel, and Nagasuma Chandra. Fast local alignment protein pockets (flapp): a pystem-compiln program for large-scale binding site alignment. Journal of Chemical Information and Modeling, 62(19):4810-4811 2022.
Lingling Shen, Jian Fang, Lulu Liu, Fei Yang, Jeremy L. Jenkins, Peter Kutchukian, and He Wang. Pocket crafter: a 3d generative modeling based workflow for the rapid generation of hit molecules in drug discovery. Journal of Cheminformatics, 16(1):33, 2024.
Viet-Khoa Tran-Nguyen, Célien Jacquemard, and Didier Rognan. Lit-peba: unbiased data set for machine learning and virtual screening. Journal of chemical information and modeling, 60(9): 4263-4273, 2020.
2020.
Oleg Trott and Arthur Olson. Autodock vina: improving the speed and accuracy of docking with new scoring function, efficient optimization, and multithreading. Journal of computational chemistry, 31(2):455-46 2010.
Renxiao Wang, Xueliang Fang, Yipin Lu, and Shaomeng Wang. The pdbbind database: Collection of binding affinities for protein- ligand complexes with known three-dimensional structures. Journal of medicinal chemistry, 47 47(11)29777-2980, 2004.
Zhen Wang, Zhanfeng Wang, Maohua Yang, Long Pang, Fangyuan Nie, Siyuan Liu, Zhifeng Gao, Guojiang Zhao, Xiaohong Ji, Dandan Huang, et al. Enhancing challenging target screening multimodal protein-ligand contrastive learning. bioRxiv, pp. 2024-08 08, 2024.
Zaixi Zhang, Jiaxian Yan, Qi Liu, and Enhong Che. A systematic survey in geometric deep learning for structure-basee drug design. arXiv preprint arXiv:2306.11768, 2023.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: universal 3d molecular representation learning framework. In The Eleventh International Conference Learning Representations, 2022.
12

---

Published as a conference paper at ICLR 2025
A FDA-SIMILARITY AN AUXILIARY SIMILARITY-BASED METRIC
In addition to using similarity to known actives to measure the distance between generated molecules and real actives, we also proposed an FDA similarity metric, comparing the similarity of generated molecules to 2582 FDA-approved drugs to assess the
of generated molecules:
FDA Similarity(4)
(o(1) odd))).
JE(L2 2582]
where dj represents the j-th molecule in the FDA-approved drug database.
we believe that the 2582 molecules currently approved by the FDA as small-molecule drugs possess certain favorable drug-like chemical properties, such stability, activity in the human body, toxicity profile, absorption, and proper metabolism. These properties characterize a distribution within chemical space, and the distance to that chemical space can characterize the possibility new molecule to be modified to drug. If a model consistently generates small molecules across various targets that are far from the known drug-like space, the model's ability to generate drug-usable molecules may be questionable.
However, this metric may be overly strict; thus we place it in the appendix and provided the results of various models in Table on this metric for reference.
Table Results for Similarity- -Based Metrics including FDA-Approved drug similarity. Results for Reference Ligands and best results are shown in bold text.
Fingerprints Active
3D Fingerprints
Uni-Mol
DrugCLIP
FDA
FDA
Active
FDA
Active
FDA
Active
Reference Ligand
0.348
0.588
0.139
0.230
0.975
0.973
0.749
0.870
LiGAN
0.231
0.131
0.117
0.109
0.950
0.922
0.783
0.366
AR
0.237
0.157
0.116
0.122
0.966
0.949
0.747
0,449
PDBbind 60
Pocket2Mol
0.286
0.179
0.137
0.143
0.964
0.946
0.735
0,414
Diff
0.209
0.169
0.132
0.143
0.967
0.958
0.683
0.478
MoICRAFT
0.258
0.208
0.129
0.161
0.971
0.965
0.676
0,522
LiGAN
0.236
0.136
0.117
0.112
0.952
0.931
0.733
0.397
AR
0.229
0.161
0.113
0..125
0.965
0.951
0.730
0.468
PDBhind 90
Pocker2Mol
0.283
0.185
0.134
0.146
0.963
0.945
0.732
0.419
TargetDiff'
0.208
0.169
0.132
0.142
0.967
0.958
0.686
0.477
AOTCRAFT
0.258
0.214
0.128
0.163
0.972
0.966
0.681
0.547
Pocket2Mol demonstrates the highest similarity to known drugs on the FDA-approved list based on 2D Fingerprints.A notable observation is that the reference ligand shows greater similarity to known active compounds than to FDA-approved drugs in general. In contrast, the generative models display higher similarity to the broader category of FDA-approved drugs rather than specific known actives.
B ADDITIONAL BENCHMARK RESULTS FOR OPTIMIZATION- BASED METHOD.
We present the results of RGA (Fu et al. 2022), optimization-basee method, in Table After several rounds of optimization, RGA model can generate molecules with better Vina docking scores. However, the virtual screening metrics are not improved. And the Delta Score still quite limited.
C ADDITIONAL BENCHMARK RESULTS ON LIT-PCBA
In addition to the 101 targets used in the main paper, which utilizes decoys designed according to specific rules, we also benchmark the models with 15 additional targets that are derived from LIT-PCBA dataset Ngayen et al. 2020). an alternative virtual screening dataset. This dataset consists of decoys that have been actualized in wet-lab experiments and exhibit minimal bioactivity. The threshold used to differentiate between active compounds and decoys in LIT-PCBA more relaxed, making it significantly more stringent test.
13

---

Published as a conference paper at ICLR 2025
Table 6: Evaluation Results for optimization-basee method RGA.
Binding Affinty
Virtual Screening
Vina docking
Delta score
BEDROC
EF
Reference Ligand
-9.363
2.686
39.32
24.95
MoICRAFT
-9.788
0.973
7.584
3.953
RGA Initial
-9.000
0.177
4.042
1.951
RGA Final
-9.665
0.286
3.888
1.850
Table 7: Evaluation Results for Similarity- Based Metrics on LIT-PCBA dataset. Results for Reference
Ligands and best results are shown in bold text.
Similarity to Actives
Virtual Screening
Fingerprints
DrugCLIP
BEDROC
EF
Reference Ligand
0.269
0.613
4.332
3.641
LiGAN
0.141
0.529
1,527
0.889
AR
0.157
0.558
1.879
1.361
Pocket2Mol
0.187
0.538
2.363
1.711
TargetDiff'
0.167
0.508
2.095
1,237
MolCRAFT
0.189
0.556
2,498
1.577
Table 7 shows that LIT-PCBA is indeed a difficult dataset, with an enrichment factor of 3.64 when using actual ligands reference, indicating modest improvement over random selection. MoICRAFT remains the top-performing model overall.
D MORE VISUALIZATIONS
D.1 MOLECULE GRAPH
We provide additional visualizations in Figures 7 8 9 and to demonstrate the significance of our proposed metrics. Molecules generated by MoICRAFT and Pocket2Mol exhibit good similarity to one of the known active compounds, even though they only achieve moderate docking scores. is important to note that these molecules are not similar to the original ligands, highlighting the importance of using targets with multiple known actives for evaluating similarity metrics.
14

---

Published as a conference paper at ICLR 2025
Target: ada17
Ligand
Docking Score: -7.47
Known Active
0H
Similarity to Active: 0.37
OH
MolCRAFT generated Mol
HO
Figure 7: Cases that show the importance of using similarity-based metries to evaluate the effective- ness of generated molecules. Molecule generated by MolCRAFT.
Target: hivint
Ligand
Docking Score: -7.059
Known Active
Similarity to Active: 0.38
MolCRAFT generated Mol
Figure 8: Cases that show the importance of using similarity-based metrics to evaluate the effective- ness of generated molecules. Molecule is generated by MolCRAFT.
15

---

Published as a conference paper at ICLR 2025
Target parp1
Ligand
Docking Score: -7.627
Known Active
Similarity to Active: 0.36
HgN
Pocket2Mol generated Mol
Figure 9: Cases that show the importance of using similarity-based metries evaluate the effective- ness of generated molecules. Molecule generated by Pocket2Mol.
Target: fa10
Ligand
Docking Score: -6.455
Known Active
Similarity to Active: 0.44
Pocket2Mol generated Mol
NHgg
Figure 10: Cases that show the importance of using similarity-basee metrics to evaluate the effective- ness of generated molecules. Molecule is generated by Pocket2Mol.
D.2 DOCKING POSES
Here, present comparison of the binding poses between real actives and generated molecules with high similarity to actives. Figure II and Figure [2 illustrates molecules generated by MoICRAFT, B and are generated by Pocket2Mol.
16

---

Published as a conference paper at ICLR 2025
Figure 11: Binding poses between real active molecules (green) and generated molecules (pink) on
Target ROCK1, Molecule is generated by MolCRAFT.
Figure 12: Binding poses between real active molecules (green) and generated molecules (pink) on
Target HDAC2. Molecule is generated by MoICRAFT.
Figure 13: Binding poses between real active molecules (green) and generated molecules (pink) on Target ESR1. Molecule is generated by Pocket2Mol.
17

---

Published as a conference paper at ICLR 2025
Figure 14: Binding poses between real active molecules (green) and generated molecules (pink)
Target GLCM. Molecule is generated by Pocket2Mol.
E ADDITIONAL ANALYSIS ON VINA DOCKING BEING EASY TO BE OVERFITTED
In addition to atom number, we identified several factors that can lead to overfitting manipulation of Vina docking scores. These factors include count of hydroxyl groups (-OH), the percentage of nitrogen and oxygen atoms (N+O), and the count of halogens ((fluorine, chlorine, bromine, iodine, and astatine)). The correlations between these factors and Vina docking scores are illustrated in Figures 1 16
Mean Vina Score vs Hydroxyl (-OH) Count
-7.5
7.9258
-8.0
Mean Vina Score
8,1688
-8.2661
8,4653
-8.5
-9.0
0
1
2
3
Hydroxyl (-OH) Count
Figure 15: Vina docking scores have high correlation with the count of Hydroxyl groups
18

---

Published as a conference paper at ICLR 2025
Mean Vina Score vs N+0 Percentage
-4,8654
-5
-5. 5.9986
-6
Mean Vína Score
-7
7.1271
-8
-8.3187
8.9239
99.0117
-9
-10
0.0 to 0.1
0.1 to 0.2
0.2 to 0.3
0.3 to 0.4
0.4 0.5
0.5 to 0.6
N+O Percentage
Figure 16: Vina docking scores have high correlation with the ratio of Nitrogen(N) and Oxygen(O) atoms.
Mean Vina Score vs Halogen Count
7.8200
-9
Mean Vina Score
-10
1037
10.6908
-11
-12
12
Halogen Count
Figure 17: Vina docking scores have high correlation with the number of Halogen atoms.
19

---

Published as a conference paper at ICLR 2025
F ADDITIONAL DISTRIBUTION PLoTs
We provide the violin distribution plots of different targets in the test for different metrics.
F.1 DISTRIBUTION PLOT FOR SIMILARITY TO KNOWN DRUGS IN FDA APPROVED LIST
In Figure 18
Mal Sinitaiity
LIGAN
Pocket2Mol TargetDift MolCRAFT
RocketZMol TargetDiff MoICRAFT
(a) Morgan Fingerprints
(b) ECFP3 Fingerprints
lergetiiff
(c) Uni-Mol Molecular Encoder
(d) DrugCLIP Molecular Encoder
Figure 18: Distribution plots for max similarity to all drugs in FDA approves list on all targets in DUD-E with different models.
F.2 DISTIRBUTION PLOT FOR SIMILARITY TO KNOWN ACTIVES
In Figure 19
20

---

Published as a conference paper at ICLR 2025
0.70
Sinitarity
Max Simmillaat Mitl
0013
603
UGGAN
AR
Pocket2Moi largetDiff MolCRAFT
UIAN
AR
Pocket2/Mo targetbiff MolCRAFT
(a) Morgan Fingerprints
(b) ECFP3 Fingerprints
Sinilarit
Max Stmilaritt
0:00
08
Al
Pocker2Mol TargeiDiff MoICRAFT
UGAN
Pocket2Mol TargetDiff MeICRAFT
(c) Uni-Mol Molecular Encoder
(d) DrugCLIP Molecular Encoder
Figure 19: Distribution plots for max similarity to known actives on all targets in DUD-E with different models.
F.3 DISTIRBUTION PLOT FOR VIRTUAL SCREENING
In Figure 3
Enrichment Fector
Enrichment Factor
LIGAN
AR
Pocket2Mol TargetDiff MelCRAFT
UIAN
AR
PocketZMol TargetDiff MoICRAFT
(a) Morgan Fingerprints
(b) ECFP3 Fingerprints
Enrichment Tector
Enricchment Factor
LIGAN
AR
Pocket2Mol TargetDiff MoICRAFT
LIGAN
Pocket2Mol TargetDiff MolCRAFT
(c) Uni-Mol Molecular Encoder
(d) DrugCLIP Molecular Encoder
Figure 20: Distribution plots for Virtual Screening results on targets in DUD-E with different models.
21

---

Published as a conference paper at ICLR 2025
G COMPARISON OF PDBBIND AND CROSSDOCKED DATASET.
Table Glide Docking score and Delta Score of the pocket with reference ligand on PDBbind and CrossDocked dataset.
Glide Docking
Delta Score
CrossDocked
-6.37
1.05
PDBBind
-7.19
1.88
H TRAINING AND VALIDATION SET DETAILS
We constructed training and validation sets using data from PDBbind, renowned for its highly reliable, experimentally observed structures. The dataset underwent rigorous filtering to ensure quality and relevance. Initially, excluded all ligands that RDKit could not correctly interpret, including those with erroneous molecular structures and discontinuous molecules. Subsequently, we selected pockets from target PDB files using 10 Á distance threshold from the ligand. We further refined the dataset by excluding pockets containing nucleic acids (DNA/RNA) and repairing non-standard residues within the pockets. Rare non-standard residues that could not be repaired were removed. Additionally, protein pockets with fewer than 100 atoms were discarded. This comprehensive filtering process yielded final of 19,438 protein pockets, which we then used to construct our training and validation datasets.
To assess the ability of the SBDD generation model to generalize novel pocket types, we imple- mented a homology reduction based on pocket structural similarity between the training and test sets. Utilizing FLAAPP, we aligned pockets from the training set with those from the test set, quantifying structural similarity through the ratio of successfully aligned amino acids. Figure [ illlssrrrte the impact of varying FLAPP score thresholds on the number of remaining samples. To strike a balance between removing highly similar pockets and retaining selected thresholds of 0.6 and 0.9, resulting two distinct datasets, as detailed in Table / These datasets were subsequently divided into training and validation sets in 9:1 ratio through random sampling.
adequate volume of training data,
17500
1751
Training and Validation Set Size
15000
15346
13337
12500
12344
8620
7500
5000
2500
51
FLAPP scores
Figure 21: Relationship between FLAPP Score Threshold and Dataset Sizes
Figure 22 illustrates the FLAPP similarity scores between selected targets from the DUD-E dataset and protein pockets from the PDBbind database. The majority of cases have scores clustered the range of 0.2 to 0.4.
22

---

Published as a conference paper at ICLR 2025
FLAPP distribution: DUD-E PDBBind
Ikha4
Unrb
Salth
rent
es/2
0.8
0.4
0.0
xlep
lgf2r
tgfr1
pdesa
pperg
pyrd
0.5
0.2
cp3e4
1.0
0.4
0.0
def
hdac2
glcm
akt1
braf
egfr
0.8
0.2
Figure 22: FLAPP score distribution. Each figure is the distribution of FLAPP score between DUD-E target and all pdbbind pockets.
23

---

Published as a conference paper at ICLR 2025
Table 9: Data sizes for different thresholds
Threshold (FLAPP Score)
Data Size (PDBbind)
0.6
12,344
0.9
17,519
I TEST SET DETAILS
constructed new test sets utilizing data from the well-established virtual screening benchmarks DUD-E and LIT-PCBA. Following the removal of erroneous records, curated 101 test data points from DUD-E and 15 from LIT-PCBA. These test data cover various categories of protein targets, such G-Protein Coupled Receptors (GPCRs), kinases, and nuclear receptors. This diversity enables comprehensive assessment of the model's performance across different protein types. The classification and quantity of these data points are provided in Table I0
In DUD-E, each target in our test set contains an average of 224.4 active compounds and 50 decoys per active. For LIT-PCBA, each target includes average of 503.33 active compounds and 176,268.11 decoys. Consistent with our training and validation sets, we defined pockets for SBDD inputs by selecting regions within a 10 Á radius from the reference ligand. This systematic approach ensures a robust and comprehensive evaluation of the model's 's capabilities across diverse protein-ligand interactions.
Table 10: Distribution of protein target categories in our test set
Target Categories
DUD-E
LIT-PCBA
Kinase
26
Protease
15
0
Nuclear Receptor
GPCR
4
2
Miscellaneous
0
Ion Channel
0
Cytochrome P450
0
Other Enzymes
36
MODEL TRAINING DETAILS
We employed a new dataset to train and test across five distinet SBDD baselines. All models were trained on a single NVIDIA A100 80GB GPU. The training durations were follows: MoICRAFT required approximately 30 hours, TargetDiff took around 48 hours, Pocket2Mol also took about 48 hours, AR's main model and frontier model each required 48 hours, and LiGAN training took approximately 30 hours.
K MODEL SAMPLING DETAILS
For the 101 + 15 targets in our test set, we sampled 20 small molecules per target using each model. For the autoregressive-basee models, AR and TafgetDiff, which initially tend generate smaller molecules, we first sampled 100 molecules and then randomly selected 20 from this set to ensure uniformity in molecule size.
L EVALUATION PARAMETERS
For Morgan Fingerprint, we use radius = 2, length of bit vector = 1024.
For E3FP Fingerprint, length of bit vector 1024, radius multiplier 1.5.
24

---

Published as a conference paper at ICLR 2025
For Uni-Mol molecular encoder, we use the trained weights provided. For DrugCLIP related models, we use the trained weights provided.
25


Reviews:
[{"review_id": "7dqBvaLkLi", "reviewer": "Reviewer_hHxU", "summary": "This paper introduces a three part framework to assess SBDD-generated molecules. The first part is similarity between generated molecules and known active compounds, achieved using molecular fingerprints. The second part is a virtual screening metric, a combination of BEDROC and EF, to measure the ability of generated molecules to distinguish between active and inactive compounds. The final part is binding affinity to a target, in which Vina and delta scores, and DrugCLIP were used to achieve this. Results show that amongst the used models, MolCRAFT dominated across multiple metrics.", "questions": "- How would your method evaluate a generated molecule that has a very different structure from known actives, but demonstrates strong binding affinity in docking simulations?\n\n- Can you please clarify why you are so confident that this is a practical alternative to wet-lab experiments?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The problem that this paper attempts to address is one of high importance. Many generative/optimization models in this field rely on computational property predictors, which may not be accurate. The alternative would be to conduct real-world wet-lab experiments, which in many cases, is not feasible. \n- The three proposed assessment techniques capture different aspects of the generated molecules and make for a combination that avoids redundancy. \n- The datasets used (especially the testing set) seem to be satisfactory.", "weaknesses": "- The authors state: “This provides a practical alternative to testing deep learning-generated molecules directly in wet-lab experiments.” I believe this is quite a bold statement and, unfortunately, am not fully convinced of this. While the authors provide a neat framework and results, I think more experiments on larger datasets are required to show that this statement holds true. Moreover, it would be much more convincing if the authors are able to demonstrate correlation between their metrics and actual wet-lab results on a subset of compounds. \n\n\n- I understand the rationale behind using similarity to known active compounds as an assessment. However, my concern is that this can be limiting – we can generate very desirable molecules that may be extremely different from the active compounds. \n\n\n\nMinor edits:\n- Line 118: generated by docking software instead of real complex → complexes\n- Line 307: Actually, ligands in PDBbind has a higher docking score → have\n- This paragraph needs to be better written: \"We refined the PDBbind dataset by excluding complexes with nuclear attachment and inaccurately recorded ligands. Then split into a 9:1 training and validation set. To assess the SBDD model’s generalization across diverse pockets, we removed samples with similar pockets using FLAPP for pocket alignment and similarity assessment. We remove all pockets from the training set with align rate more than 0.6 or 0.9 to the test set pocket. After the removal, the 0.6 version has 12344 pairs remaining while the 0.9 version has 17519 pairs remaining.\"\n- Line 485: We do some visualizations → present \n\n- Figure 6: Be consistent with wording/capitalization: \n    - Molcraft Generated Mol \n    - Molecule generated by TargerDiff\n\n- Figure 5 captions: include spaces after “(a) ...” and “(b) ...”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "liasFW9wi7", "reviewer": "Reviewer_pELf", "summary": "This paper draws attention to pitfalls in evaluation metrics commonly used to assess generative models for structure-based drug discovery. While the authors focus their discussion on 3D methods, the pitfalls discussed extend to any generative model for drug discovery. The authors first highlight the problem with the commonly used docking score metric and how it can be exploited. Next, three sets of evaluation tasks are proposed: delta docking score (assess specific binding), similarity metrics (to known actives and FDA-approved drugs), and virtual screening (can a similarity search from generated molecules identify known actives?). The authors provide insights on the current performance of 3D SBDD generative models and highlight shortcomings of current model development to guide future research.", "questions": "* What are the authors' thoughts on evaluating the 3D geometry of the generated molecules? PoseCheck [1] and PoseBusters [2] are now common in SBDD papers. The results show that unphysical poses is an ongoing challenge for 3D generative models. Generated molecules with poses that clash with the protein would not typically be considered further.\n\n* The authors state that the metrics do not require the need to consider synthetic ability but synthesizability is a problem highlighted in the Methods section. I agree that wet-lab testing should never be a requirement for computational research but do the authors think there are metrics that can better assess synthesizability beyond SA score and beyond being implicitly considered via similarity to known actives?\n\n* Minor comment: there is a typo in \"actives\" in figure 2.\n\n[1] https://arxiv.org/abs/2308.07413\n\n[2] https://pubs.rsc.org/en/content/articlehtml/2024/sc/d3sc04185a", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The authors explicitly highlight pitfalls of current docking score evaluations of generated molecules\n* Performance across several 3D models is compared and contrasted\n* The rationale is clear when proposing new metrics", "weaknesses": "Overall, I agree there should be better metrics for SBDD assessment. However, many of the metrics and challenges proposed by the authors has been extensively reported in existing literature. There are also many existing works that are not 3D-based diffusion/flow models for SBDD that account for the limitations expressed by the authors. As a result, some of the limitations are limitations of the problem formulation and not intrinsically to SBDD metrics. I will organize the comments under each proposed metric.\n\n### **Docking and Synthesizability**\n* The authors focus on SBDD generative models but there is a lot of work in this field that does not involve geometric models that can perform SBDD and recent benchmarks also show that these 1D/2D methods outperform 3D methods [1]. The reason this is brought up is because in these works, the limitation of AutoDock Vina to exploit low QED molecules and large molecules is well documented. As a result of this, there are many papers that explicitly optimise docking jointly with QED (and often also with SA score). Here are a few works that do this, many of which have been published at conferences [2, 3, 4, 5]. Therefore, while I do think that many papers are overfitting AutoDock Vina by designing large molecules with low QED, many recent works have acknowledged this and are proposing objective functions that explicitly account for QED.\n* A large reason the Delta score is poor for AutoDock Vina is that it is too crude. This is likely to diminish if one uses Glide docking, as the authors implicitly show with the enrichment. The goal could also be explicitly to dock well to 1 protein and not to another protein as was done here [6].\n* It is true that metrics like SA score are imperfect to predict synthesizability but it can be well correlated [7, 8].\n* Taking Tanimoto similarity to known molecules does not necessarily mean it is easier to synthesize. Synthesizability is complex and single atom changes can turn something synthesizable to something that is not.\n\n### **Similarity Metrics (including Virtual Screening)**\n* If a metric is introduced that rewards similarity to known molecules, then the optimisation objective could simply be similarity to these molecules. This has also been shown explicitly in [9] where the model generates molecules similar to a known active while getting good docking scores. In my opinion, we do not necessarily want to re-discover known molecules in the sense of generating similar molecules to a set of reference ligands. This is already done in distribution learning/transfer learning approaches that were detailed in this review [10]. The alternative and more general goal is to design molecules with arbitrary property profiles. In this case, the degree to which optimisation is successful is the most relevant metric. \n* Similarity to FDA-approved drugs may not be completely justified. Drugs are specific and drugs targeting 2 different proteins can be vastly different in structure. In Table 3, the authors also show that even the reference ligand have low similarity to the FDA-approved drugs.\n* The problem with Vina scores that the authors elude to is that it is too crude. A very well known problem of docking algorithms is that they cannot reproduce experimental binding poses. Specifically, re-docking is a common task used to validate docking. In cheminformatics literature, this is known as sampling power [11]. Similarly, the ability to rank affinity with docking scores is known as scoring power.\n* In ML literature, an equivalent is the assessment of how many binding poses are within some RMSD threshold, for example in the DiffDock paper [12]. If the docking cannot reproduce known experimental data, it is unclear the level of confidence to trust the output. The reason for bringing this up is because if there is a docking algorithm that can capture the binding interactions properly, designing molecules optimised for docking score can implicitly capture similarity. This was explicitly shown in cheminformatics literature [13] (although I still believe similarity to known molecules is not necessarily the best metric and can be hacked). The idea of having a better docking algorithm equating to better results is also shown by the authors in Table 1: comparing Glide with AutoDock Vina and Glide gets better enrichment. Then if Glide were optimised instead, the generated molecules would be better. I understand that this is not commonly done because Glide is proprietary software.\n\nWhile I believe the sentiment of the authors is that better metrics can better reflect real impact, metrics do not replace wet-lab experiments. This line should not be bolded to give the false impression that predictive methods are close to accurately predicting wet-lab experiment. It would be better to re-phrase this line to convey that better metrics can better inform future model development that may improve wet-lab outcomes.\n\n### **References**\n[1] 3D SBDD Benchmark: https://chemrxiv.org/engage/chemrxiv/article-details/66bb0911a4e53c48763ac057\n\n[2] Reinforced GA: https://arxiv.org/abs/2211.16508\n\n[3] TacoGFN: https://arxiv.org/abs/2310.03223\n\n[4] GEAM: https://arxiv.org/abs/2310.00841\n\n[5] Saturn: https://arxiv.org/abs/2405.17066\n\n[6] Negative Design: https://iclr.cc/virtual/2023/12911\n\n[7] SA Correlation 1: https://jcheminf.biomedcentral.com/articles/10.1186/s13321-023-00678-z\n\n[8] SA Correlation 2: https://arxiv.org/abs/2407.12186v1\n\n[9] Similarity reward: https://www.nature.com/articles/s42256-022-00494-4\n\n[10] Generative Design Review: https://www.nature.com/articles/s42256-024-00843-5\n\n[11] Sampling and Scoring Power: https://pubs.rsc.org/en/content/articlelanding/2016/cp/c6cp01555g\n\n[12] DiffDock: https://arxiv.org/abs/2210.01776\n\n[13] DockStream: https://jcheminf.biomedcentral.com/articles/10.1186/s13321-021-00563-7", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vgSYtCv7fz", "reviewer": "Reviewer_WYJC", "summary": "This paper points out a curial problem in current SBDD models evaluation: The vina docking score metric is easy to inflate and other metrics are hard to test in wet lab. So the paper proposed three-level evaluation metrics: Similarity to Known Drugs and Actives, Virtual Screening Ability and Binding Affinity Estimation. They test on five models and claim that MolCRAFT outperforms other methods", "questions": "1. The proposed benchmark relied on comparing similarity with known drug molecules, does this affect the search for de novo drug molecules?\n2. Could you show the docking pose images of molecules generated by MolCRAFT and Pocket2Mol exhibit good similarity to one of the known active compounds", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Provide a novel benchmark to tackle a curial problem in this area\n2. Select a diverse set of models on SBDD", "weaknesses": "1. Focus on testing deep learning models. Could test more on other based such as RL based or Genetics Algorithm based\n2. Table 1's description is too simplified, hard to let people to understand why using real molecules as template is better\n3. It would be better to briefly describe some terminology, i.e. FLAPP", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W4BW673boQ", "reviewer": "Reviewer_8aMH", "summary": "The paper provides a comprehensive framework for evaluating structure-based drug design models, using metrics like virtual screening and similarity to known drugs.\n\nThe paper is well written. However, the paper has some limitations. Although it acknowledges the difficulty of synthesizing molecules derived from SBDD models, it lacks a systematic approach. A heavy reliance on virtual screening-based metrics can be misleading without experimental validation. For the paper to be more useful and applicable, more real-world case studies should be included.", "questions": "1. how well does this framework perform when applied to novel targets for which there may be no closely related compounds available?\n\n2. Are there any risks associated with overfitting to the new evaluation metrics?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. very detailed and well-explained in most of the sections.\n\n2.  The propose a more practical framework for evaluating SBDD models, which can be useful.", "weaknesses": "1. While the paper acknowledges the difficulty of synthesizing molecules generated by SBDD models, it does not present a systematic approach for addressing this problem.\n\n2. In this paper, virtual screening-based metrics are heavily employed to determine the practical usefulness of generated molecules. While virtual screening is helpful, it could still be misleading if it is not validated with experimental data.\n\n3. This paper needs to include more real-world case studies to be more valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)