CBGBench: Fill in the Blank of Protein-Molecule Complex Binding Graph
Authors: Haitao Lin
Guojiang Zhao
Odin Zhang
Yufei Huang
Lirong Wu
Cheng Tan
Zicheng Liu
Zhifeng Gao
Stan Z. Li
Venue: ICLR 2025 Spotlight (2025)
Keywords: Molecule Generation Benchmark
Target-Aware Drug Design
Generative Model
Primary area: datasets and benchmarks

Abstract: Structure-based drug design (SBDD) aims to generate potential drugs that can bind to a target protein and is greatly expedited by the aid of AI techniques in generative models. However, a lack of systematic understanding persists due to the diverse settings, complex implementation, difficult reproducibility, and task singularity. Firstly, the absence of standardization can lead to unfair comparisons and inconclusive insights. To address this dilemma, we propose CBGBench, a comprehensive benchmark for SBDD, that unifies the task as a generative graph completion, analogous to fill-in-the-blank of the 3D complex binding graph. By categorizing existing methods based on their attributes, CBGBench facilitates a modular and extensible framework that implements cutting-edge methods. Secondly, a single de novo molecule generation task can hardly reflect their capabilities. To broaden the scope, we adapt these models to a range of tasks essential in drug design, considered sub-tasks within the graph fill-in-the-blank tasks. These tasks include the generative designation of de novo molecules, linkers, fragments, scaffolds, and sidechains, all conditioned on the structures of protein pockets. Our evaluations are conducted with fairness, encompassing comprehensive perspectives on interaction, chemical properties, geometry authenticity, and substructure validity. We further provide insights with analysis from empirical studies. Our results indicate that there is potential for further improvements on many tasks, with optimization in network architectures, and effective incorporation of chemical prior knowledge. Finally, to lower the barrier to entry and facilitate further developments in the field, we also provide a single [codebase](https://github.com/EDAPINENUT/CBGBench) that unifies the discussed models, data pre-processing, training, sampling, and evaluation.

Published as a conference paper at ICLR 2025
CBGBENCH: FILL IN THE BLANK OF PROTEIN- MOLECULE COMPLEX BINDING GRAPH
Haitao Lin Li.t. Guojiang Zhao?:t, Odin Zhang Yufei Huang',
Lirong Wu', Zicheng Liu', Cheng Tan', Zhifeng Gao2, Stan Z. Li
1 AAT Lab, Research Center for Industries of the Future, Westlake University:
DP Technology: *Zhejiang University: Equall Contribution; *Corresponding Author; 1 Linhaitao@west wakee lakelakeeee. cn stan. zq. 11
cn
ABSTRACT
Structure-based drug design (SBDD) aims generate potential drugs that bind to a target protein and is greatly expedited by the aid of AI techniques in generative models. However, lack of systematic understanding persists due the diverse settings, complex implementation, difficult reproducibility, and task singularity. Firstly, the absence of standardization can lead to unfair comparisons and inconclu- sive insights. To address this dilemma, we propose CBGBench, a comprehensive benchmark for SBDD, that unifies the task as generative graph completion, analo- gous to fill-in-the-bland of the 3D complex binding graph. By categorizing existing methods based on their attributes, CBGBench facilitates modular and extensible framework that implements cutting-edge methods. Secondly, single de novo molecule generation task can hardly reflect their capabilities. To broaden the scope, adapt these models to range of tasks essential in drug design, considered sub- tasks within the graph fill-i--the-blant tasks. These tasks include the generative designation of de novo molecules, linkers, fragments, scaffolds, and sidechains, all conditioned on the structures of protein pockets. Our evaluations are conducted with faireess, encompassing comprehensive perspectives on interaction, chemical prop- erties, geometry authenticity, and substructure validity. We further provide insights with analysis from empirical studies. Our results indicate that there potential for further improvements
many tasks, with optimization in network architectures, and effective incorporation of chemical prior knowledge. Finally, to lower the barrier to entry and facilitate further developments in the field, we also provide single codebase (https: /github... com/ /EDAP INENUT CCBBBncc) that unifies the discussed models, data processssn, training, sampling, and evaluation.
INTRODUCTION
The rapid and remarkable progress in Graph Neural Networks (GNNs) and generative models have advanced the bio-molecule design in these years (Jumper al., 2021; Watson et al., 2023; Krishna et al., 2023). In structure- based drug design, Al-aided methods aim to learn the chemical space the molecules that can bind to certain proteins as targets, which decreases the general chemical space of molecule compounds (~ 1060) to more compact search space and enables the model to explore the potential binding drugs. Recent success in generative models such as diffusion models (Ho al., 2020; Lipman et al., 2022; Song et al., 2020) further enhanced these methods to fully exploit the targeted chemical space, and Al-aided SBDD has been propelled into another prominence.
Despite the significance of the SBDD and the development of various approaches, there remains lack of a comprehensive benchmark for this field covering various practical application scenarios. On the one hand, although different methods are proposed for the task, the experimental setup is not unified, and the evaluation protocol also differs. For example, GRAPHBP (Liu et al., 2022) use different training and test split of Crossdocked2020 (Francoeur et al., 2020) from concurrent works like POCKET2MOL (Peng et al., 2022) and 3DSBDD (Luo et al., 2022); And DIFFBP (Lin et al., 2022) and GRAPHBP employ Gnina (McNutt al., 2021) instead of AutoDock Vina (Trott & Olson, 2009) as the evaluator for obtaining docking score as binding affinity. Besides, the complex implementation of models makes the modules coupled, it is hard to figure out which proposed

---

Published as a conference paper at ICLR 2025
module contributes to the improvements in performance. For example, though TARGETDIFF (Guan et al., 2023b) and DIFFBP are both diffusion-basse methods, the first one uses EGNN (Satorras et al., 2022) to prediet the ground-truth position of each atom in the molecule while the later one uses GVP (Jing et al., 2020) to remove the added noise in a score-based way. To have a systematic understanding of the designation for diffusion models in molecule generation, the network architectures should be fixed to keep the expressivity equal. On the other hand, the task of de novo generation of molecules a branch of SBDD. There are also other important tasks, such as the design of molecular side chains in lead optimization, or linker design to combine functional fragments into single, connected molecule (Igashov et al., 2022; Guan et al., 2023a). It is highly meaningful to explore whether these methods can be successfully transferred to applications in drug optimization.
In this way, here we propose CBGBench as a benchmark for SBDD, with the latest state-of-the-an methods included and integrated into a single codebase (Appendix. C). Firstly, we unify the generation of the binding molecule as a graph completion task, i.e. fillin-the-blank of the 3D Complex Binding Graph. Therefore, the systematic categorization of existing methods is based on three dichotomies: (i) voxelized v.s. continuous position generation, (ii) one-shot v.s. auto-regressive generation, and based v.s. dull-enta-driven generation. As result, the methods can be easily modularized and extensible in a unified framework. Secondly, CBGBench introduces a unified protocol with comprehensive evaluation, including (i) chemical property, (ii) interaction, (iii) geometry and (iv) substructure analysis, with metrics extended and fair ranking considered. Moreover, thanks to our reformulation of the problem and extensible modular implementation, we can easily extend these methods to the other four tasks in lead optimization, including the designation of (i) linkers, (ii) fragments, (iii) side chains and (iv) scaffolds. Comprehensive evaluation is also conducted for these tasks, to explore the potential application value of the existing methods in lead optimization.
(iii)
As result, several brief conclusions can be reached as the following:
CNN-based methods by modeling voxelized density maps molecule representations remain highly competitive in target-aware molecule generation, and the diffusion- ones achieve the state-of-the-art.
For autoregressive methods, it is essential to enable the model to successfully capture the patterns of chemical bonds in training and generation of binding molecules.
Prior knowledge has been incorporated into the model in recent works, but the improvements remain limited. Effective design of integrating physical and chemical domain knowledge remains a challenge, leaving substantial room for future research.
Most evaluated methods can be well generalized to lead optimization. Empirical studies show that scaffold hopping is the most challenging task among them, while linker design is relatively the easiest. However, there is still large space for improvements in these tasks.
The conclusions drawn from the experimental results and with the evaluation protocol of CBGBench, are mainly consistent with those obtained on real-world discase targets, demonstrating generalizability and effectiveness of our benchmark.
BACKGROUND
2.1
PROBLEM STATEMENT
For a binding system composed of a protein-molecule pair as (P,M). in which P contains Nroc atoms of proteins and M contains Nig atoms of molecules, we represent the index set of the protein's 's atom as I and the molecule's atoms as Tig. with [reee Nirc and [Tgl Nig. The binding graph can be regarded as a heterogenous graph. One subgraph protein structures, as P = (Vree Erac), where V = (a;, X,, hez,, is the node set, and E = [(, (1,) (i.)e. is the edge set. Here, a; is the atom types with a; = 1,..., M,, x; E R is the corresponding 3D position, and s; 1, 20 is the amino acid types that i-th atom belongs to; elments in edge set means there exists a bond between i and j. with edge type et,- The other graph is the molecule structures, written as M = (Vig. Eng), where Vigg {(a;; x;) ie and Eig is the edge set with the same form
Binding Site
Docupted
Blank
Figure 1: Filling the blank in the complex binding site.
2

---

Published as a conference paper at ICLR 2025
Table 1: Categorization of included methods. LIGAN, 3DSBDD and VOXBIND model atom positions discrete variables; One-shot meth- ods maintain constant atom number in a generation, like diffusion-barte based ones; FLAG and D3FG use frag- ment motifs, and DECOMPDIFF uses arm&scaffodding priors.
Vorelized Position
Continuous Posttion
One-shol Generation
Auto-regressive Generation
Method
Continous Position
Generation
One-sbox
Domain
LIGAN
X
3DSBDD
x
x
PocKEr2MoL
/
x
x
Domsin-KnewweddeeeBe
Data-Driven
GRAPHBP
/
x
x
TARGErDRY
/
/
X
DID
/
/
X
FLAG
x
D3PG
/
V
Figure 2: The demonstration of the classification criteria for the existing methods. The gray thick curves outline the contact surface of the protein, and the circles represent different types of atoms.
DecomrDerr
/
/
MolCRAFt
V
VoxBIND
x
X
as Erec. Besides, there are virtual edges that exist between protein and molecule, which make up cross-edge set Ecr. Denote the probabilistic model for binding graph by p(M, PP). For the de novo molecule generation, the model aims to learn the probability of p(M(P), as to fill the blank of the protein pocket with atoms in a molecule, by using a generative model p(-).
comparable analogy can be made that if also divide the atoms in a molecule into known and to- be- he-generated parts, we can extend the aforementioned 'filing-in-the-blane In specific, for the missing blanks, we write the node set for generation as and the known atoms in the molecule as elements in the context set C (Veix: Eetx), in which Tetx Iig Igen- Therefore, the generative model for filling the blanks of the missing parts of molecules aims to learn the probability of p(G P). The newly-defined task of fill-in-the -prtiall- graph is of great significance in SBDD, especially in lead optimization, and we establish other four sub-tasks of it besides de novo design, with accessible datasets in Sec. 3.
task as shown in Figure. 1.
(Vgen= Egen) )
The
2.2 RELATED WORK AND TAXONOMY
The SBDD methods are initially combined with deep neural networks voxel grids, such as LIGAN (Masuda et al., 2020) generating atom voxelized density maps using VAE (Kingma Welling, 2022) and incorporating Convolutional Neural Networks (CNNs) as the architecture, and 3DSBDD (Luo et al., 2022) predicts whether the position on grids is occupied with which type of atoms in auto-regressive ways with Graph Neural Networks (GNNs). Then, the development of Equivariant Graph Neural Networks (EGNNs) boosts the SBDD methods to directly generate the continuous 3D positions, such as POCKET2MOL (Peng et al., 2022) generating the molecules' atom types, positions, and the connected bonds and GRAPIIB (Liu et al., 2022) employing nor- malizing flows generate these attributes, both in an auto-regressive way. The diffusion denoising probabilistic models (DDPM) (Ho et al., 2020) further propel the A Al-aided SBDD methods, such as TIFGETDIFF (Guan al., 2023b), DIFFBP (Lin et al., 2022) and DIFISBDD (Schneuing et al., 2022), inspired by EDM (Hoogeboom et al., 2022), generating full atoms' positions and element types, with different diffusion models. In recent years, domain knowledge has been used to constrain or guide the generation of binding drugs. For example, FLAG (Zhang et al., 2023b) and D3FG (Lin et al., 2023) use prior knowledge of fragment motifs to model the coarse structures, which generate fragments in molecules in one-shot and auto-regressive ways, respectively; And DECOMP (Guan et al., 2023c) harnesses virtual points searching for arm- and scaffold-clusterinn as prior knowledge, and using multivariate Gaussian process model atom positions in different clusters, with validity guidance for sampling. More recently, thanks to technological breakthroughs brought by generative AI, more related approaches have been proposed, which not only build upon previous modeling
3

---

Published as a conference paper at ICLR 2025
concepts but also incorporate new generative model techniques. MOLCRAFT (Qu et al., 2024), as the SBDD version of GEOBFN (Song et al., 2024), employs Bayesian Flow Network (Graves et al., 2024) as the variant of diffusion models. to address the continuous-discrete gap in modeling the elements' type and atoms' positions by applying continuous noise and smooth transformation; VOXBIND (Pinheiro al., 2024a), following their pioneering exploration of VoXMOL. (Pinheiro et al., 2024b), continues modeling the 3D voxelization of molecules in diffusion denosing way, with walk-jump sampling method used generate molecules. The latest FLEXSBDD (Zhang et al., 2024b) aims to generate the dynamic conformation of protein as well as design the binding molecules with Flow Mathcing techniques. Since the structure of protein targets may change significantly in FLEXSBDD, leading to different validation protocols, we will not conduct in-depth comparison here.
In this way, we can categorize these methods with the three standards:
Whether the positions of atoms are generated in continuous 3D space or voxelized grids.
Whether the generation process is auto-regressive or one-shot.
Whether the domain knowledge is introduced to integrate extra prior into the model.
To better articulate, we classify them according to Table. I, and Figure. 2 gives a simple demonstration of the criteria for taxonomy. A detailed review of these methods is given in Appendix. A.
3 TASK AND DATASET
For the de novo molecule generation, we follow the previous protocol to use Crossdocked2020 (Fran- coeur et al., 2020) and data preparation with splits proposed in LiGAN (Masuda et al., 2020) and 3DSBDD (Luo et al., 2022) as the training and test sets. Besides the de novo generation, the modular- ized methods can be extended to four subtasks which are branches of our defined tll-in-the-blantee of the complex binding graph, including the target-awars linker, fragment, side chain, and scaffold design (Zhang et al., 2024a), leading to the generative targets G for the probabilistic model p(g C, P) to be the four components, and the molecular context C as the rest, as shown in Figure. 3. We demonstrate the significance of the four tasks in lead optimization and how the corresponding datasets established below. The datasets are all established based on previous splits of Crossdocked2020 for a fair comparison.
Linker Design is critical strategy in fragment- drug discovery (Grenier et al., 2023). It focuses on creating linkers that connect two lead fragments and obtaining a complete molecule with enhanced affimity. Effective linker design can significantly influence the efficacy and pharmacokinetics of the drug, by ensuring that the molecule maintains the desired orientation and conformation when bound to its target (Erlanson et al., 2016). We define linkers following specific rules: (i) A linker should act as the intermediate structure connecting two fragments. (ii) A linker should contain at least two atoms on the shortest path between two fragments. (iii) Each connecting fragment must consist of more than five atoms.
Fragment Growing focus on expanding fragment on the lead compound to better fill the binding pocket (Bancet et al., 2020). It also relates to adjusting pharmacological properties, such as enhancing solubility or reducing toxicity (Hung al., 2009). We decompose the entire ligand into two fragments and select the smaller one for expansion based specific criteria: (i) Each fragment must contain more than five atoms.
(ii) The smaller fragment should be larger than half the size of the larger one.
Side Chain Decoration differs from fragment growing in that it permits modifications at multiple sites on the lead compound, whereas fragment growing typically modifies a single site. The taxonomy of arms- crns-sccafold in previous work (Guan et al., 2023c) is similar to side-chain-scafoldd while we focus on a chemist-intuitive approach, Demis-Murke decomposition (Bemis& & Murcko, 1996), which treats all terminal non-cyclic structures side chains.
Scaffold Hopping is introduced by Schneider et al. (1999), as a strategy in medicinal chemistry aiming replace the core structure of molecules to explore more diverse chemical space improve specific properties. While various empirical definitions exist (Sun et al., 2012), for consistency, the Bemis-Murko decomposition is utilized to define the scaffold structure.
Since chances are that there is no substructure in a molecule according to the discussed definition of the decomposition, here give the instance number of training and test set for each task in Table. 2.
4

---

Published as a conference paper at ICLR 2025
Table 2:
The instance number of training and test split in the datasets for the four tasks and De novo generation. The decompo- sition is conducted separately in the training and test set of Cross- Docked2020 avoid label leak- age. Side chain decoration and scaffold hopping are two dual tasks, so the training and test set numbers are the same.
Linker Design
Fragment Growing
Generated Molecule
Target atom
Condition
Dataset
Training
Test
Side Chain Decoration
Scaffold Hopping
t
De nove
99900
100
Linker
52685
43
Fragment
61379
61
Side Chain
70617
64
Scaffold
70617
64
Figure 3: Demonstration of the four tasks.
4 EVALUATION PROTOCOL
In previous works, the evaluation usually focuses on two aspects including interaction and chemical property. While lots of works start focus another two standards on molecule 3D structure's authenticity with geometry analysis and the 2D graphs reliability with subsutructure analysis, the evaluation protocol usually not unified. In this way, we extend them in a unified protocol including the four aspects as the following:
Substructure. To evaluate whether the model succeeds in learning the 2D graph structures for drug molecules. We expand previous metrics to functional groups, certain functional groups in the drugs act as pharmacophores, such as phenol and pyrimidine. We identify the 25 most frequently occurring functional groups by using EFG (Salmina et al., 2015) (Appendix. B.1 gives the details), and calculated mean frequency of each function group generated in a molecule and the overall multinomial distribution. In the same way, the two values can also be obtained according to rings and atom types. The generated frequency and distribution against the references lead to metrics on substructure analysis including the Jensen-Shannon divergence as JSD and mean absolute error as MAE, listed as (i) JSD (4)). (ii) MAE (4), (iii) rt (L), (iv) (4), (v) JSD (4), and (vi) &f (4), where 'at' means atom type, 'rt' means ring type and 'fg' means functional group.
Chemical Property. We continue the evaluation of chemical properties from most previous works, including (i) QED(t) as quantitative estimation of drug-likeness; (ii) SA as synthetic accessibility score; (iii) LogP represents the octanol-watte partition coefficient, and in general LogP values should be between -0.4 and 5.6 to good drug candidates (Ghose al., 1999); (iv) LPSK(() as the ratio of the generated drug molecules satisfying the Lipinski's ' rule of five.
Interaction. In evaluating the generated molecules' interaction with proteins, the binding affinity is usually the most important metric. Here we unify the affinity calculation with AutoDock Vina for fair comparison. However, previous studies show that larger molecule sizes will lead to a higher probability that the generated molecules can interact with the protein, resulting in higher predicted binding affinity (Abad-Zaae Zapatero & Metz, 2005; Hopkins et al., 2014). is also an observation in this paper the Pearson correlation of Vina Energy v.s. atom number reaches - -0.67 (See Appendix. B.2). Besides, in some structures, there are more atoms in the protein participating in the interaction, which also causes the average be unreasonable. Therefore, besides the commonly the mean Vina Energy. and (ii) IMP%(t) as the improvements indicating the ratio of the generated molecule with lower binding energy than the reference, we use mean percent binding gap (iii) (PBG(() (Appendix. B.3 gives computation) and ligand binding efficacy as (iv) LBE(T). written as LBF, Frima, indicating how much does a single atom in the molecule contribute to the binding affinity, to eliminate the effects of molecule sizes. We give detailed discussion on the reasonality of LBE in Appendix. B.4. Moreover, it is important for the generated molecules to keep the same or similar interaction patterns with proteins (Zhang al., 2023a). Therefore, we aim to figure out whether the SBDD models can learn the microscopic interaction patterns hidden in the data of the 3D conformations. We use PI (Adasme et al., 2021; li Zuo al., 2023; Salentin et al., 2015)
(i) Evina (4) as
5

---

Published as a conference paper at ICLR 2025
to characterize the 7 types of interactions and calculate per-pocket and overall JSD between the categorical distributions of generated types and reference types and MAE between the frequency of each generated types v.s. it of reference types, in which the frequency is defined as the average number of occurrences of each type of interaction produced by different single generated molecules binding the same protein, leading four metries including (v) JSDpp(4), (vi) JSDoA (+), (vii) MAEpp(U) (4), (viii) MAEoA (J), where 'PP" and 'OA' means per-pocket and overall respectively.
Geometry. The internal geometry an important characteristic for distinguishing between general point clouds and molecular structures. The torsion angles (Jing et al., 2022; Swanson et al., 2023) are flexible geometries. while bond lengths and bond angles are static to reveal whether the generated molecules have realistic structures, Hence, we evaluate the overall JSD of bond length and angle distribution between reference and generated ones, written as (i) JSDBL (L) and (ii) JSD (4). The other perspective for validating structural resonability is the clash, the occurrence of which is defined when the van der Waals radii overlap by 2 0.4A (Ramachandran et al., 2011). Hence, the ratio of number of atoms generating clashes with protein atoms to the total atom number is written as (iii) RatiOrea (4) (cross clashes at atom level). Besides, the ratio of molecules with clashes as (iv) Ratio (4) (molecule with clashes) are evaluated, which indicates if any atom in a molecule causes clash, we consider that molecule to have clash.
We here evaluate the 12 methods discussed in Table. 1, and use Friedman rank (Friedman, 1940; 1937; Wang et al., 2022) as the mean ranking method, to fairly compare the performance of different models in the four aspects, The ranking score calculated by (12 rank), and the final rank is according to the weighted mean ranking score. Appendix. B.3 gives detailed computation of the metrics, and their explanations as preliminary. The review of each method's ' evaluation aspect in the published papers is given in Apppendix. B.5.
5
BENCHMARK RESULT
5.1 De novo GENERATION
5.1.1 SETUP
Training. We use the default configuration in each model's released codebase as the hyper- parameter, and set the training iteration number as 5,000,000 for fair comparison. is noted that the loss of autoregressive methods exhibits faster convergence rate of loss, typically requiring only a few tens of thousands of epochs to reach the final best checkpoint. To eliminate the effect brought about by the architecture of GNNs, in implementation, we use GVP (Jing al., 2020) and EGNN (Satorras et al., 2022) with GAT (Veličković et al., 2018), as message- message-passing modules of auto-regressive and diffusion-basee -hased models, respectively. Especially, the GNN for encoding and decoding functional groups in D3FG is a combination of GAT and LoCS (Kofinas et al., 2022), CNN for processing atom density maps four convolution blocks in LIGAN, and larger-scaled UNet for VOXBIND. The details in hyperparameter tuning and training strategy are given in Appendix. D.
Evaluation. Following the previous protocol, we generate 100 molecules per pocket in the test set, while not all the generated molecules are chemically valid (See Appendix. E.4), which will lead to <10,000 molecules. In affinity evaluation, we employ three modes of Doock Vina, including 'Score", 'Minimize', and 'Dock'. MPBG and LBE are added to the "Vina Dock' mode. For Vina energy that larger than O, we think is invalid, so we will not report it, and give the lowest ranking in the column of metrics. Moreover, LogP only provides reference range for drug molecules, so we assigned a rank of to the generated molecules within the range and to those outside the range. In all the tables, values in bold are the best metric, and values with underline are the second and third.
5.1.2 RESULT ANALYSIS
Substructure. Table. 3 gives results on 2D substructures, showing that (i) Overall, the auto- regressive models perform worse than the one-shot ones, especially in ring-type evaluation, where the former ones are more likely to generate triangular and tetrahedral rings. (ii) Diffusion-based models also exhibit advantages in atom type, according to the results of TIRGETDIFF and MOLCRAFT, except for DIFFBP has difficulties in generating atoms with low occurrence frequencies. (iii Besides VOXBIND, MOLCRAFT shows the overall superiority in complex functional group generation, as states in addressing the mode collapse issues, while DECOMPDIFF and D3FG also exhibit advantages

---

Published as a conference paper at ICLR 2025
Table 3: Results of substructure analysis.
Table 4: Results of chemical property.
Metbods
Metrics
JSDa:
type
MAEs:
1SD
Ring type
MAE.:
Functional Group
ISD
MAA
Rank
LIGAN
QED
0.46
Logl
0.56
SA
LPSK
4.39
Rank
4.75
LIGAN
0.1167
0.8680
0.3163
0.2701
0.2468
0.0378
3DSEDD
0.45
0.47
0.63
4.72
1.50
3DSRDD
0,080
0.8444
0.3188
0.2457
0.2682
0.0994
6.50
GRAMIBP
0.44
3.29
0.64
4.73
3.75
GRAPHBP
0.1612
1.2266
0.5061
0.4322
0.6259
0.0705
11.33
POCKEY/2MUL
0.39
4.58
4.75
MockeT3MoL
0.0916
104977
0.3550
0.3545
0.2961
0.0622
8.50
TARGETDIEF
0.49
1.13
0.60
4.57
4.25
TARGETDEY
DEFSBDD
DUPBP
FLAG
0.0529
0.0533
0.2591
0.1032
0.2399
0.6166
1.7065
1.5491
0.2345
04531
022422
0,159
0,008
0.3370
0.2876
0.363
0.0411
0,0666
0.070
4.00
8.83
DIFFSBDD
DIVEP
FLAG
0.49
0.41
0.47
-0.15
0.29
0.34
0.58
4.47
4.89
4.93
4.25
5.25
3.50
D3FG
0.0644
0.8154
0.180
0.234
0
0.0516
5.17
D3FG
0.49
1.56
0.66
4.84
2,00
DECUMrDUp
0,0431
0.319
0.3431
0.200
0.1916
0.011
2.67
DECOMPDUF
0.49
3.75
MOLCRAFT
0,0490
0.3308
0.1
0.447
0,2469
0.0264
2.17
MOLCRAFT
0.48
0.88
0.67
4.39
4.00
VOXRIND
0.3564
0.0301
VOXBIND
0.54
2.22
0.65
4.70
2.75
Table 5: Results of interacton analysis.
Metrics
Vina Score
Vina Min
Vina Dock
PLIP Interaction
Rank
Methods
Paina
Evans
IMP%
Eiyica
IMP%
MPBG%
LBE
JSD
MAE
JSDpp
MAEpp
LIGAN
-6.47
62.13
-7.14
70.18
-7.70
72.71
4.22
0.3897
0.0346
0.0905
0.1451
0.3416
2.91
3DSBD
3.99
-3.75
17.98
-6.45
31.46
9.18
0.3539
0.0392
0.0934
0.1733
0.4231
7.17
1.67
4.57
-M0.83
0.3200
0.0462
0.1625
0.4855
11.33
PoCKET2MoL
5.23
31.06
-6.03
38.04
-7.05
48.07
-0.17
0.4115
0.0319
0.2455
0.1535
0.4522
5.67
TARGHTDUY
-5.71
38.21
-6.43
47.09
-741
$1.99
5,38
0.3557
0.0198
0.0600
0.1757
014687
4.67
DIRSBDD
12.67
-2.15
22.24
.5.53
29.76
-23.51
0.2920
0.0333
0.1461
0.1377
0.5265
9.25
DIFBP
8.60
19.68
-7.34
49.24
6.23
0.3481
0.0349
0.1430
0.1256
0.5639
7.41
FLAG
0.04
3.44
-3.65
11.78
-47.54
0.3319
0.0170
0.0277
0,2762
0.3976
9.00
D3FG
3.70
-2.59
11.13
46.78
28.90
-8.85
0.4009
0.0638
0.0135
0,1850
0.3641
8.17
DECOMPDOFF
19.66
-7.10
0.3460
0.0215
0.0769
0.4369
6.08
MOLCRAFT
-6.15
54.25
-6.9 9.9
56.13
-7.79
56.22
8.38
0.3638
0.0214
0.0780
0.1868
0.4574
3.75
VOXBIND
-6.16
41.80
-6.2
50.02
.7.68
$2.91
9.89
0.3588
0.0257
0.0533
0.1850
0.4606
4.00
due to the incorporation of prior knowledge in ring types and functional groups. DIFFBP and DIFFSBDD perform poorly due to inconsistencies in generating complex fragments. For detailed comparison, see Appendix. E.1.1
Chemical Property. The chemical property is calculated with 2D molecule graphs, so it can be greatly influenced by the molecule substructures. From Table, 4, we can conclude that (i) In terms of the four metrics, the differences among the compared methods not significant. (ii) D3FG shows the best overall properties, with the competitive QED, SA, and LPSK.
Interaction. From Table. 5 shows that (i) LIGAN and VOXBIND CNN-based methods gen- erate molecules that are initialized with high stability according to its competitive performance in Vina Score and Vina Min. It also performs well in Vina Dock mode, with positive MPBG and high LBE, providing good consistency in interaction patterns. (ii) Auto-regressive methods except POCKET2MOL can hardly capture the pocket conditions and generate stably-binding molecules well, with very low IMP% in all Vina modes, while POCKET2MOL is the state-of-the-art auto-regressive method in de novo generation. (iii) In diffusion- based methods, MOLCRAFT outperform the other competitors in overall interaction comparison, and TARGET DIFF and DECOMPDIFF perform compet- itively since the difference between these two methods is minimal. DIFFBP's performance in docking mode is comparable, but in other modes, it performs less satisfactorily. Performance of DIFFSBD is less than satisfactory. D3FG generates molecules with comparable Vina Energy but small atom numbers, leading to high LBE. For details, see Appendix. 1.2.
Geometry. From Table. 6, conclusions can be drawn that (i) MOLCRAFT generates overall the most realistie structures, according to the internal geometries, and DECOMPDIFF is comparable. (ii) The molecule-proted clashes can usually be avoided by diffusion diffusion-based models. However, in the auto-regres regressive models, only POCKET2MOL avoids clashes. The high frequency of clashes in auto-regressive methods can be explained as follows: These models first identify frontier amino acids or atoms within the molecule. During the placing of new atoms, incorrect binding site localization leads to an erroneous direction for the molecule's auto-regressively growing path, causing it to extend inward towards the protein. For detailed results, see Appendix. E.1.
Conclusion and Discussion. From Table. 7 and the previous discussion, we conclude that
(i) The CNN- methods like LIGAN and VOXBIND are highly competitive, especially in aspects of Interaction, which explains why such methods in the field of drug design and molecule generation still prevail in recent years. This is partly attributed to the fact
7

---

Published as a conference paper at ICLR 2025
Table 6: Results of geometry analysis.
Table 7: Ranking scores and overall ranking.
Methods
Metrics
J5Dr..
Static Geometry
JSD
Ratiouss
Clash
Ratloon
Rank
Methods
Weights
Subsrmac:
0.2
Chem.
0.2
Interact. 0,4
Geom.
0.2
Rank
LIGAN
0.4645
015675
0.00%
0,0718
5.75
LIGAN
1.45
5
0.8683
8.75
3D8BD
1.10
1.70
1.93
01659
7
GRAPHBP
0,5182
0.5645
0.8634
0.9974
11.50
GRAPHBI
0.13
1.65
0.27
0.10
11
PoCKET2MoL
0,5133
0.1922
0.0576
0.1199
8.50
POCKET2MOL
TARG
0.2689
0.369
0,0483
0.4920
4.50
TARGETDUFF
1.60
1.55
2.93
1.50
3
DDFSBDD
0,4588
0.6578
DIFFSBDD
0.73
1.70
1.10
0.95
5
DIFBP
0.3153
0.1621
0.0119
0.1077
5.25
DIVBP
1.38
1.84
8
FLAG
0.4215
0.4304
0.6777
0.9769
9.00
FLAG
0.63
1.55
1.20
0.60
10
D3FG
0.3727
014701
0,2115
0.8571
8.50
D3PG
1.37
2.00
1.53
0.70
6
DECOMPDIR
0.2576
(1:473
0,0462
0.5248
4.00
DECOMPDUFF
1.87
1.65
2.57
1.60
MOLCRAFT
0.2230
0.2683
0,0764
0.2691
2,00
1.60
2,00
VoXBIND
0,2701
0.3771
0.0103
0.1890
3.00
VOXBIND
1:37
1.85
3.20
1.80
2
that CNNs have advantage over GNNs in perceiving many-body patterns within a single filter (Townshend et al., 2021). As a result, it encourages further research into GNNs for 3D point clouds to develop architectures that can match the expressivity of CNNs.
(ii) In GNN-based methods, MOLCRAFT achieves the best overall performance, with TARGETD- IFF closely following. In contrast, DECOMPDIFT and D3FG as the variants of TARGETDIFF that incorporate domain knowledge show some degeneration in performance. It reveals that the current incorporation of physicochemical priors can hardly improve the quality of generated molecules. Effectively integrating domain knowledge to guide the model to generate structurally sound molecules remains challenge. For example, atom clashes are very common in generated molecules. While DECOMPDIFF employs the prior guidance for this, the problem has still not been fully solved. Integrating domain knowledge into the training process may mitigate this issue (Huang et al., 2024a; Adams & Coley, 2023).
(iii)
Only POCKET2MOL as an ruto-regressive method achieves competitive results, which we attribute to the following reasons: First, it utilizes chemical bonds to constrain atoms to grow orderly along chemical bonds rather than to grow based on distance to the pocket as in DIFFBP. Second, simultaneously predicts the bond types and employs contrastive learning by sampling positive and negative instances of atom positions real and fake bonds, which is not fully considered by FLAG, enhancing the model's 's ability to perceive chemical bond patterns. Therefore, believe that enabling the autoregressive methods model to successfully capture the patterns of chemical bonds is very essential.
5.2 EXTENSION ON SUBTASKS
Setup. Lead optimization is to strengthen the function property of the binding molecules by remodeling the existing partial 3D graph of molecules. We show the interaction analysis and chemical property in the main text and omit the interaction pattern analysis since maintaining the patterns is not necessary for lead optimization. The -mnaii-knowledg.. based methods can hardly be transferred for these tasks since different tasks require different priors, we have not compared them here. Besides, the voxelize-basse based methods are not easily extended to these tasks, and we regard the transferring of methods like LIGAN and 3DSBDD future work. Hence, we here compare 6 methods that model atoms' continuous positions with GNNs and have not employed domain knowledge. When training, since the number of training instances is smaller, we use the pretrained models on de novo generation and finetune the auto-regressive models with 1,000,000 iterations. For diffusion-basse models with one-shot generation, we train them from scratch because the zero-center-o ofrrr-o-mmssss (Satorras al., 2021) technique is shifted from employing protein geometric centers to using molecule context's ones. For detailed results, please refer to Appendix. E.2.
Conclusion and Discussion. From Table. 8 and the previous discussion, we conclude that
(i) Overall, the performance gap among these methods in the lead optimization is not as pronounced compared to de novo generation. MOLCRAFT, TARGET and POCKET2MOL maintain the good performance. Notably, in the evaluation of the stability of initial poses, the other three also complete the binding graph near the given partial molecular conformations well, according to the columns of Vina Score and Vina Min. The applicability of GRAPHBP also reflects the Argument. (ii) in our de novo geometry analysis: The failure of GraphBP mainly stems from the difficulty in locating the correct atoms for auto-regressive growth.
8

---

Published as a conference paper at ICLR 2025
Table 8: Results of subtasks for lead optimization.
Tisks
Methods
Metrics
Evisa
Vina Score
IMP%
Eccss
Vina Min
IMP%
Evinx
IMP%
Vina Dack
MPBQ%
LBE
QED
Log!"
Chem.
Prop
SA
LPSK
Rank
GRAPHBP
5.63
-0.97
12.43
-7.51
28. 18
-7.36
0.3288
0.41
0.86
0.70
3.60
4.75
Lioker
PocKET2Mo1
6.89
18.99
.7.19
25.04
-8.07
37.22
-3.85
0.3276
0.45
1.93
0.67
4.25
2.83
TARGFIDIFF
-7.23
36.11
.7.0
41.31
-8.49
50.73
2.61
0.2993
0.39
1,63
0.61
4,02
3.12
DIASBDD
-5.64
11.06
-6.38
19.15
-7.88
34.97
-4.42
0.3110
0.42
1.14
0.66
4.10
4.25
DIFFBP
-6.27
35.49
-7.19
36.80
-8.74
$1.33
6.60
0.3078
0.43
3.45
0.55
4.01
3.25
MOLCRAFT
-7.13
38,80
-7.81
42.56
-8.81
58.44
5.09
0.3334
0.43
1.06
0.67
4.14
1.50
GRAPHBP
-6.28
9,78
-7.
16.21
0.3749
0.54
4.66
3.58
Fragreen
PoCKET2MOL.
-687
22.78
-7.61
37.45
-8.33
54.05
-0.28
0.3310
0.46
1.02
0.63
4.07
2.00
TARGRIDDF
.6.06
24.56
-6.78
30.443
-7.96
42.000
-2.38
0,3003
0.45
1.43
D.58
4.28
3.33
DIRSBDD
19.14
0.3076
0.47
4.39
4.17
DIFFBP
4.51
22.31
-6.18
29.52
-7.90
45.20
.1.2
0.2952
0.46
2.24
0.49
4,30
4.08
Moi CRAFI
.6.75
21.12
-7.06
36.07
-7.92
43.02
-0.09
0.3236
0.46
1.27
0.51
4.64
2.58
Side chuin
GRAPHBP
5:01
10.15
-5:43
11:46
-6.14
9.71
-1.055
0.4459
0.61
1.93
0.76
4.93
3.75
PoCKET2MoL
5599
22.26
.6.56
33.29
0.49
2.91
TARGETDiW
25.00
23.90
-6.50
35.81
-7. 411
46.87
-2.55
0.5213
0.48
0.88
0.60
4.41
2.58
DID
4.43
15.12
-5,99
30.23
-7.58
44.09
-9.58
0.3178
0.43
1.20
0.65
4,03
3.83
DIFFBP
14.31
0.3143
0.49
4.50
4.25
Moi CRAPT
-6.10
24.10
-6.64
35.58
-7.49
41.67 11.67
-3.12
0.3227
0.44
1.22
0.61
4,36
2.42
GRAPURP
0.00
0.06
-3.900
0.99
-50.62
0.3797
0.43
0.14
0.76
4,98
4.16
Seaffold
PoCKE12MOL
-1.00
16.84
-5.71
23.08
-6.89
38.18
-8.07
0.3378
0.43
0.91
0.61
4.48
2.42
TARGEIDOY
-.52
31.47
-5.86
34.39
-7.06
44.32
-6.22
0,3038
0.43
0.59
4.26
2,00
DIFSBDD
-3.55
18.44
-1.90
22.12
-6.81
31.98
-10.23
0.2985
0.42
-0.13
0.53
4.29
4.17
DIFFBP
-2.09
13.89
.4.35
16.84
46.
32.43
-12.14
0. 0.3025
0.43
3.37
0.56
4.44
4.17
MoI CRAPT
.471
14.03
-5.35
30.16
-2.02
43.53
22.44
0.3146
0.42
1.01
0.56
4.55
2.83
(ii) In these tasks, scaffold hopping is the most challenging task, as the improvement of generated molecules relative to the reference is minimal; Linker design is relatively the casiest. Notably, in scaffolding, there are still some methods that fail.
(iii) There is a large space for improvements in these tasks, since the MPBG% metrics are usually negative, indicating that in most cases, the molecule is not optimized. Some edge- cutting techniques such as DPO (Rafailov et al., 2024; Cheng et al., 2024; Gu et al., 2024) and ITA (Yang et al., 2020; Kong et al., 2023; Lin et al., 2024) may be used to augment the optimized molecules as supervision signals for the models.
Additionally, several points warrant further detailed design. For instance, linker design, the fragments of the molecule that need to be connected may change orientation due to variations in the linker (Guan et al., 2023a); In side chain decoration, the protein's side chains, which are the main components in interaction, should be generated together with the molecule's 's side chains to achieve stable conformation as the entire complex (Luo et al., 2023; Huang et al., 2024b).
5.3 CASE STUDY ON REAL-WORLD DISEASE TARGETS
Introduction. In order to verify whether the included methods can generalize pharmaceutic targets related to disease and the applicability of CBGBench to real-world scenarios, we use the pretrained model in de novo generation, and apply them two proteins belonging the G-Protein- Coupled Receptor (GPCR) family: ARDB1 (beta-1 adrenergic receptor) and DRD3 (dopamine receptor D3). ARDB1 participates in the regulation of various physiological processes by responding to the neurotrans mitter epinephrine (adrenaline) and norepinephrine, and drugs that selectively activate or block this receptor are used in the treatment of various cardiovascular conditions. For DRD3, it is primarily expressed in the brain, particularly in areas such as the limbic system and the ventral striatum with functions of mediating the effects of the neurotrans mitter dopamine in the central nervous system.
Setup. On these targets, there are molecules reported active to them experimentally. we here randomly select 200 of them for each target and conduct two kinds of experiments. Firstly, we try to figure out the model can generate binding molecules that have similar chemical distributions with the actives. We use extended connectivity fingerprint (ECFP) (Rogers & Hahn, 2010) to get the molecule fingerprint and employ t-SNE (van der Maaten & Hinton, 2008) for 2-dimensional visualization. Secondly, we aim to find out the distribution of Vina Docking Energy and LBE of the generated and the actives as the metrics for binding affinities. Eight methods except for voxelized- based ones (inflexible to extend) and DECOMPDIFF (requiring complex preprooeppooessnngggg for domain knowledge) are tested on the two targets. Besides, we select 100 molecules randomly from TEOM-DRUG (Axelrod &
2022), randomized control sample set.

---

Published as a conference paper at ICLR 2025
Vina Docking Energy
Actives
Acthves GraphaP
Actives TargetDiff
-5
-10
412
Actves
Rocket2Mol
Graphep
TargeDiif
DiffSBD
Drffe
PLAG
D3PG
MatCraft
Actives DinsaDo
Actives
Actives FLAG
Dirite
(a) Distribution of Vina Dock Energy
Ligand Binding Efficacy
0.4
Actives MalCraft
DaFG
DOOM-DRUG
0.2-
0.0
Actives
PacketMMal
Graphap
BrgeDifff
DMSBDD
FLAG
D3FG
MolCraft
Figure 4: T-SNE visualization of chemical distributions of generated and active molecules on ADRBI.
Distribution of LBB
Figure 5: Distribution of binding affinities.
Conculusion and Discussion.
Figure. 4 and Figure. gives the results on target ADRB1.
(i) In Figure. 4, we can see that POCKET2MOL, TARGETDIFF, D3FG, and MOLCRAFT have better consistency in the chemical distribution of molecules, as evidenced by greater degree of overlap with the actives. addition, comparison to randomly selected molecules in GEOM-DRUG, these models show different preferences in generating binding molecules since the clustering center the chemical space differs.
(ii) Figure. 5 shows that in generating molecules based the real-world target ADRB1 related to hypertension and heart failure, D3FG exhibits superior performance. TARGETDIFF and MOLCRAFT perform comparably.
These conclusions are essentially consistent with the conclusion in Sec. 5.1, reflecting that the established evaluation protocols exhibit consistency and generalizability on real-world disease target data, especially in evaluating the binding affinity through Vina Dock. Besides, it is worth noting that DIFFBP, GRAPHIP, and POCKET2Mor can possibly generate molecules with small atom numbers and high LBE. This indicates that they have the potential to excel in lead discovery on ADRBI, as a good lead should possess good synthesizability and modifiability and smaller molecular weight. For DRD3, please refer to Appendix. E.3 for details.
CONCLUSION AND LIMITATION
CBGBench unifies the tasks of SBDD and lead optimization into a fll-in-the-bllak- 3D binding graph, comprehensively categorizes and modularizes existing methods, and integrates them into a unified codebase for fair comparison. Additionally, it extends existing evaluation protocols by incorporating more aspects with metries, addressing the issue of incomplete and diverse evaluation process. Extensive experiments including case studies on disease targets for molecule generation give insightful conclusions and identify future research directions.
However, there are certain limitations. Firstly, this codebase is based on GNNs, so the voxelized- grid-based methods with CNNs have not been included. Engineering the integration of these types of methods will be focus of our future work. Secondly, due to the inability to use wet lab experiments for validation, most metrics are obtained through computational methods, some which considered unable to accurately reflect the chemical properties of molecules, such as SA very important metrics but its deficiency is usually ignored by molecule design methods (Luo et al., 2024). Therefore, how to utilize AI to assist in accurate metric calculation will also be key focus of our future research. Besides, several works (Zheng et al., 2024) emphasized the potential ID/2D molecule designation in these year, which is also important aspect of research interest.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported by National Science and Technology Major Project (No. 20222D0115101), National Natural Science Foundation of China Project (No. 624B2115, No. U21A20427), Project (No. WU2022A009) from the Center of Synthetic Biology and Integrated Bioengineering of Westlake University, Project (No. WU2023C019) from Westlake University Industries of the Future Research Funding.
REFERENCES
Cele Abad-Zapater and James T Metz. Ligand efficiency indices as guideposts for drug discovery. Drug discovery today, 10(7):464-465 2005,
Keir Adams and Connor W. Coley. Equivariant slape-conditioned generation of 3d molecules for ligand-basee drug design. In The Eleventh International Conference on Learning Representations, 2023. URL https: : opwnreview.e...ee/ee.o... net/ fo Forum? 4MbGnp44 iQQ.
Melissa F. Adasme, Katja L Linnemann, Sarah Naomi Bolz, Florian Kaiser, Sebastian Salentin, V. Joachim Haupt, and Michael Schroeder. Plip 2021: expanding the scope of the protein-ligand interaction profiler to dna and ma. Nucleic Acids Research, 45:W530 W534, 2021.
Simon Axelrod and Rafael Gómez-Bombardee
Geom, energy-annotated molecular conformations for property prediction and molecular generation. Scientific Data, 18); 2022. doi: 10. 141597-022-01288- URL https:t///// htt////oiioooo huttps://doi.rrg/1100/ http///o.ooo......///////. 1038/
0220002222222 1002-01111 01288
Alexandre Bancet, Claire Raingeval, Thierry Lomberget, Mare Le Borgne, Jean-Françol Guichou, and Isabelle Krimm. Fragment linking strategies for structure-baset drug design. Journal of Medicinal Chemistry, 63(20):1 (1:00)::1120-1111 1435, 2020.
Guy W Bemis and Mark A Murcko. The properties of known drugs. 1. molecular frameworks. Journal of medicinal chemistry, 39(15):8887-289) 1996.
Xiwei Cheng, Xiangxin Zhou, Yuwei Yang, Yu Bao, and Quanquan Gu. Decomposed direct preference optimization for structure-based drug design, 2024. URL /tattps://axxi...oo abs/2407 13981.
Daniel A Erlanson, Stephen W Fesik, Roderick E Hubbard, Wolfgang Jahnke, and Harren Jhoti. Twenty years on: the impact fragments on drug discovery. Nature reviews Drug discovery, 15 (9):605-611) 619, 2016.
Paul G. Francoeur, Tomohide Masuda, Jocelyn Sunseri, Andrew Jia, Richard lovanisci, Ian Snyder, and David Ryan Koes. 3d convolutional neural networks and crossdocked dataset for structure- based drug design. Journal of chemical information and modeling. 2020.
Milton Friedman.
The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 22.675-701, 1937.
Milton Friedman. A comparison of alternative tests of significance for the problem of SmS rankings. Annals of Mathematical Statistics, 11:86-99 92, 1940.
Arup K. Ghose, Vellarkad N. Viswanadhan, and John J. Wendoloski. A knowledge-based approach in designing combinatorial or medicinal chemistry libraries for drug discovery. a qualitative and quantitative characterization of known drug databases. Journal of combinatorial chemistry, 55-68, 1999.
Alex Graves, Rupesh Kumar Srivastava, Timothy Atkinson, and Faustino Gomez. Bayesian flow networks, 2024. URL https::/ http//axvvvvo////// org/ aas//ab//// 0w..23200000000
Dylan Grenier, Solène Audebert, Jordane Preto, Jean- Françoianç Guichou, and Isabelle Krimm. Linkers in fragment-base drug design: an overview the literature. Expert Opinion on Drug Discovery, 1889)99771 2023.
11

---

Published as a conference paper at ICLR 2025
Siyi Gu, Minkai Xu, Alexander Powers, Weili Nie, Tomas Geffner, Karsten Kreis, Jure Leskovec, Arash Vahdat, and Stefano Ermon. Aligning target-aware molecule diffusion models with exact energy optimization, 2024. URL https: /ttp///axivvv//aa/ org/ htwpabs///////ooo/////////////a/ 2407 01 046.
Jiaqi Guan, Xingang Peng, Peiqi Jiang, Yunan Luo, Jian Peng, and Jianzhu Ma. Linkernet: Fragment poses and linker co-design with 3d equivariant diffusion. In Neural Information Processing Systems, 2023a.
Jiaqi Guan, Wesley Wei Qian, Xingang Peng, Yufeng Su, Jian Peng, and Jianzhu Ma. 3d equivariant diffusion for target-aware molecule generation and affinity prediction. In The Eleventh International Conference on Learning Representations, 2023b. URL https: /oporeeeeeeeeeeeeeeeeeeeeeee net/ forum? id=kJqXXPPMSED
Jiaqi Guan, Xiangxin Zhou, Yuwei Yang, Yu Bao, Jian Peng, Jianzhu Ma, Qiang Liu, Liang Wang. and Quanquan Gu. DecompDiff: Diffusion models with decomposed priors for structure-base based drug design. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 11827-11846. PMLR, html.
23-29 Jul 2023c. URL https:tppp.... httpp//ooceeddeeooo ngs ml r.rrr-prsssss.
Jonathan Ho, Ajay Jain, and P. Abbeel. Denoising diffusion probabilistic models. ArXiv, abs/2006. 1239, 2020.
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation 3d, 2022. URL https: / / /awxiv.vvvvv.///vv/o/.aaaa/vaa org/ abs/ 17003.
Andrew L Hopkins, Gynorgy M Keser"u, Paul D Leeson, David C Rees, and Charles H Reynolds. The role of ligand efficiency metries in drug discovery. Nature Reviews Drug Discovery, 13(2): 105-121, 2014. doi: 10. 10.1038/nr4413..
Lei Huang, Tingyang Xu, Yang Yu, Peilin Zhao, Xingjian Chen, Jing Han, Zhi Xie, Hailong Li, Wenge Zhong, Ka-Chun Wong, and Hengtong Zhang. A dual diffusion model enables 3d molecule generation and lead optimization based on target pockets. Nature Communications, 15, 2024a.
Yufei Huang, Odin Zhang, Lirong Wu, Cheng Tan, Haitao Lin, Zhangyang Gao, Siyuan Li, and Stan. Z. Li. Re-dock: Towards flexible and realistic molecular docking with diffusion bridge, 2024b.
Alvinemsp14W Hung, H Blundell, and Chris Abell. Application of fragment growing and fragment linking to the discov- ery of inhibitors of mycobacterium tuberculosis pantothenate synthetase. Angewandte Chemie International Edition, 48(45):8452-844) 2009.
Silvestre, Shijun Wen, Alessio Ciulli, Tomemsp144 14L
Ilia Igashov, Hannes Stärk, Clément Vignac, Victor Garcia Satorras, Pascal Frossard, Max Welling, Michael M. Bronstein, and Bruno E. Correia. Equivariant 3d-conditional diffusion models for molecular linker design. ArXiv, abs/2210.05274, 2022.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael J. L. Townshend, and Ron O. Dror. Learning from protein structure with geometric vector perceptrons. ArXiv, (ttp/2009.014111 2020.
Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and T. Jaakkola. Torsional diffusion for molecular conformer generation. ArXiv, 0hs/2206.01729) 2022.
John M. Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool. Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridg- land, Clemens Meyer, Simon A A Kohl, Andy Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Push- meet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596:583- 589, 2021.
12

---

Published as a conference paper at ICLR 2025
Peter W. Kenny. The nature of ligand efficiency. Journal of Medicinal Chemistry, 60(13):5583-5594 2017. doi: htw.ccce..mmmcmmm.....
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022.
Miltiadis Kofinas, Naveen Shankar Nagaraja, and Efstratios Gavves, Roto-translated local coordinate frames for interacting dynamical systems, 2022.
Xiangzhe Kong,
Wenbing Huang, and Yang Liu, Conditional antibody design as 3d equivariant graph translation. In The Eleventh International Conference on Learning Representations, 2023. URL https: /ttts::/opeee http///ppeeevvveeeeeeeeeeeeeeeee eewwe net/ / forum? 1ininiid=-HFSSS jIIPP.
Rohith Krishna, Jue Wang. Woody Ahern, Pascal Sturmfels, Preetham Venkatesh, Indrek Kalvet, Gyu Rie Lee, Felix S Morey-Burrows, Ivan V. Anishchenko, Ian R. Humphreys, Ryan McHugh, Dionne Vafeados, Xinting Li, George A. Sutherland, Andrew Hitchcock, C Neil Hunter, Alex Kang, Evans Brackenbrough, Asim K. Bera, Minkyung Baek, Frank DiMaio, and David Baker. Generalized biomolecular modeling and design with rosettafold all- all-atom. bioRxiv, 2023.
Jia li Zuo, Changqian Yu, Nong Sang, and Changxin Gao. Plip: Language-image pre-training for person representation learning. ArXiv, abs/2305.0853 2023.
Haitao Lin, Yufei Huang, Meng Liu, Xuan Cindy Li, Shuiwang Ji, and Stan Li, Diffbp: Generative 2022.
diffusion of 3d molecules for target protein binding. ArXiv,
Haitao Lin, Yufei Huang, Haotian Zhang, Lirong Wu, Siyuan Li, Zhiyuan Chen, and Stan Z. Li. conetiooal-group--e based diffusion for pocket-specific molecule generation and elaboration. ArXiv, abs/2306. 13769, 2023.
Haitao Lin, Lirong Wu, Yufei Huang, Yunfan Liu, Odin Zhang, Yuanqing Zhou, Rui Sun, and Stan Li. GeoAB: Towards realistic antibody design and reliable affinity maturation. In Forty-first International Conference on Machine Learning, 2024. URL https: /ttp://////p/eeeeee/ee net/ Forum?1d=6pHP5P
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. ArXiv, abs/2210.02747, 2022.
Meng Liu, Youzhi Luo, Kanji Uchino, Koji Maruhashi, and Shuiwang Ji. Generating 3d molecules for target protein binding. ArXiv, 0ttss/2040094100 2022.
Shitong Luo, Jiaqi Guan, Jianzhu Ma, and Jian Peng. A 3d generative model for structure-based drug design, 2022.
Shitong Luo, Yufeng Su, Zuofan Wu, Chenpeng Su, Jian Peng, and Jianzhu Ma. Rotamer density estimator is an unsupervised learner of the effect of mutations protein-protein interaction. In The Eleventh International Conference on Learning Representations, 2023. URL https:
h //oppneeview.rr net/ feret/forumiide lttp:iiddxxyx111111 X9Y11K2mD.
Shitong Luo, Wenhao Gao, Zuofan Wu, Jian Peng. Connor W. W. Coley, and Jianzhu Ma. Projecting molecules into synthesizable chemical spaces, 2024. URL https: /ttp:s///arre //tp://aaiivvii//a http:/awwivv...ga.baaa ab///466. 6 04628.
Tomohide Masuda, Matthew Ragoza, and David Ryan Koes. Generating 3d molecular structures conditional on receptor binding site with deep generative models. ArXiv, abs/2010. 14442, 2020.
Andrew T. McNutt, Paul G. Francoeur, Rishal Aggarwal, Tomohide Masuda, Rocco Meli, Matthew Ragoza, Jocelyn Sunseri, and David Ryan Koes. Gnina 1.0: molecular docking with deep learning. Journal of Cheminformatics, 13, 2021.
Christopher W Murray, Michael G Carr, Orla Callaghan, Gianluca Chessari, Miles Congreve, Steven Cowan, Paul Edwards, Mike Frederickson, Barry Graham, Rosemary McMenamin, Michelle A O'Brien, Susan Patel, Tim R Phillips, Gareth Williams, Alison J Woodhead, and Harren Jhoti. Fragment- based drug discovery applied to hsp90: Discovery of two lead series with high ligand efficiency. Journal of Medicinal Chemistry, 53(16):594-55555 2010. doi: 10. 1a2110021mmmmm 1004102.
13

---

Published as a conference paper at ICLR 2025
Open Babel development team. Open babel. URL http:://pppbbbbll..o//k/kkkkk Page.
Xingang Peng. Shitong Luo, Jiaqi Guan, Qi Xie, Jian Peng, and Jianzhu Ma. Pocket2mol: Efficient molecular sampling based on protein pockets, International Conference on Machine Learning, 2022.
Pedro O. Pinheiro, Arian Jamasb, Omar Mahmood, Vishnu Sresht, and Saeed Saremi. Structure-based drug design by denoising voxel grids, 2024a.
Pedro O. Pinheiro, Joshua Rackers, Joseph Kleinhenz, Michael Maser, Omar Mahmood, Andrew Mar- tin Watkins, Stephen Ra, Vishnu Sresht, and Saeed Saremi. 3d molecule generation by denoising voxel grids, 2024b.
Yanru Qu, Keyue Qiu, Yuxuan Song, Jingjing Gong, Jiawei Han, Mingyue Zheng, Hao Zhou, and Wei-Ying Ma. Molcraft: Structure-based drug design in continuous parameter space. ICML 2024, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model, 2024. URL https: / /arxivv.vvvv org wwtgg////////.. /tt.b////25..... 10000.182000
Srinivas Ramachandran, Pradeep Kota, Feng Ding, and Nikolay V. Dokholyan. Automated minimiza- tion of steric clashes in protein structures. Proteins: Structure, 79, 2011.
RDKit, online. RDKit: Open-source cheminformatics. accessed 11-April-2013].
[Online;
Charles Reynolds, Brett Tounge, and Scott Bembenek. Ligand binding efficiency: Trends, physical basis, and implications. Journal of medicinal chemistry, 51:2432-88 -8, 05 2008. doi: 10.1021/ jm701255b.
David Rogers and Mathew Hahn. Extendee--comnecivvty fingerprints. Journal of chemical information and modeling, 50 5:742-54, 2010.
Sebastian Salentin, Sven Schreiber, V. Joachim Haupt. Melissa F. Adasme, and Michael Schroeder. Plip: fully automated pigtein-ligand interaction profiler. Nucleic Acids Research, 43:W443 W447, 2015.
Elena S. Salmina, Norbert Haider, and Igor V. Tetko. Extended functional groups (efg): An efficient set for chemical characterization and structure-activity relationship studies of chemical compounds. Molecules, 21, 2015.
Victor Garcia Satorras, Emiel Hoogeboom, Fabian B. Fuchs, Ingmar Posner, and Max Welling. E(n) equivariant normalizing flows. Neural Information Processing Systems, 2021.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks, 2022.
Gisbert Schneider, Werner Neidhart, Thomas Giller, and Gerard Schmid. "scaffold-hopping" by topological pharmacophore search: contribution to virtual screening. Angewandte Chemie International Edition, 38(19):2894-289) 1999.
Ame Schneuing, Yuanqi Du, Charles Harris, Arian R. Jamasb, Ilia Igashov, Weitao Du Tom L. Blundell, Pietro Li'o, Carla Gomes, Max Welling, Michael M. Bronstein, and Bruno E. Correia. Structure-based drug design with equivariant diffusion models. ArXiv, abs/2210. 13695, 2022.
Yang Song, Jascha Narain Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ArXiv, abs/2011. 1.156,,,.13.... 2020.
Yuxuan Song, Jingjing Gong. Yanru Qu, Hao Zhou, Mingyue Zheng, Jingjing Liu, and Wei- Ying Ma. Unified generative modeling of 3d molecules via bayesian flow networks. ICLR 2024, 2024.
14

---

Published as a conference paper at ICLR 2025
Hongmao Sun, Gregory Tawa, and Anders Wallqvist. Classification of scaffold-hoppinn approaches. Drug discovery today, 17(7-8):310-2224 2012.
Kirk Swanson, Jake Williams, and Eric Jonas. Von mises mixture distributions for molecular conformation generation, 2023.
Raphael Townshend, Martin Vögele, Patricia Suriana, Alex Derry, Alexander Powers, Yianni Laloudakis, Sidhika Balachandar, Bowen Jing, Brandon Anderson, Stephan Eismann, Risi Kondor, Russ Altman, and Ron Dror. Atom3d: Tasks on molecules in three di- mensions. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Truck on Datasets and Benchmarks, volume 1, 2021. URL https: aalaatassss--eemmmma---reeeeeeee wwwp://cwee..........o..eee.e... cc/paper files/ paper/ pdf.
Oleg Trott and Arthur J. Olson. Autodock vina: Improving the speed and accuracy of docking with new scoring function, efficient optimization, and multithreading. Journal of Computational Chemistry, 31, 2009.
Laurens van der Maaten and Geoffrey E. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2605, 2008
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liō, and Yoshua Bengio. Graph attention networks, 2018.
Yidong Wang, Hao Chen, Yue Fan, Wangbin Sun, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang, Zhi Zhou, Lan-Zhe Guo, Heli Qi, Zhen Wu, Yu-Feng Li, Satoshi Nakamura, Weirong Ye, Marios Savvides, Bhiksha Raj. Takahiro Shinozaki, Bernt Schiele, Jindong Wang, Xingxu Xie, and Yue Zhang. Usb: A unified semi seni-super learning benchmark. ArXiv, 2bs/2200.07204 2022.
Joseph I. Watson, David Juergens, Nathaniel R. Bennett, Brian I.. Trippe, Jason Yim, Helen E. Eisenach, Woody Ahern, Andrew Borst, Robert J. Ragotte, Lukas Milles, Basile I. M. Wicky, Nikita Hanikel, Samuel J. Pellock, Alexis Courbet, William Sheffler, Jue Wang, Preetham Venkatesh, Isaac Sappington, Susana Vázquez Torres, Anna Lauko, Valentin De Bortoli, Emile Mathieu, Sergey Ovchinnikov, Regina Barzilay, T. Jaakkola, Frank DiMaio, Minkyung Back, and David Baker. De novo design protein structure and function with rfdiffusion. Nature, 620:1089 -1100, 2023.
Kevin Yang. Wengong Jin, Kyle Swanson, Regina Barzilay, and Tommi Jaakkola. Improving molecular design by stochastic iterative target augmentation. In International Conference on Machine Learning, pp. 10716- PMLR, 2020.
Odin Zhang, Jintu Zhang, Jieyu Jin, Xujun Zhang, Renling Hu, Chao Shen, Hanqun Cao, Hongyan Du, Yu Kang, Yafeng Deng. Furui Liu, Guangyong Chen, Chang- Yu Hsieh, and Tingjun Hou. Resgen is a pocket-aware aware 3d molecular generation model based on parallel multiscale modelling. Nature Machine Intelligence, 5:1020- 1030, 2023a.
Odin Zhang, Haitao Lin, Hui Zhang, Huifeng Zhao, Yufei Huang, Yuansheng Huang, Dejun Jiang, Chang-yu Hsich, Peichen Pan, and Tingjun Hou. Deep lead optimization: Leveraging generative ai for structural modification. arXiv preprint 2xXiv224441192300 2024a.
Zaixi Zhang, Yaosen Min, Shuxin Zheng, and Qi Liu. Molecule generation for target protein binding with structural motifs. In The Eleventh International Conference Learning Representations, net
2023b. URL https:://// http://pppeeeviieeeeeeeeeeeeeeee
Zaixi Zhang, Mengdi Wang, and Qi Liu. FlexSBDD: Structure-based drug design with flexible protein modeling. In The Thirtt-eighth Annual Conference Neural Information Processing Systems, 2024b. URL https: wwtpp/ppeeeeeeeeeee/eeeee net forum?
Kangyu Zheng, Yingzhou Lu, Zaixi Zhang, Zhongwei Wan, Yao Ma. Marinka Zitnik, and Tianfan Fu. Structure-based drug design benchmark: Do 3d methods really dominate?, 2024. URL httppp:////xvvvoo/////ass////// org/abs/ 2406 03403.
15

---

Published as a conference paper at ICLR 2025
A REVIEW ON THE METHODS' TIMELINE
Here we give a brief review of the 12 included methods in chronological order, with Table. ?? giving some standardized information on these methods.
LIGAN is firstly proposed to incorporate generative model for the SBDD task, in which the encoder and the decoder are 3-layer CNNs, and the Variational Auto-Encoder is employed to model the probability of the atom's density map in a molecule. split the training and test set in CrossDocked2 020..
Due to the rise of Graph Neural Networks (GNNs), 3DSBDD subsequently adopted GNNs for modeling this task. At that time, GNNs were more adept at handling pairwise information, autoregressive graph network modeling approach for molecular modeling. However, the development of graph networks for 3D tasks remained slow, leading 3DSBDD to continue modeling molecules using voxelized maps. 3DSBDD leverages graph networks to predict the voxel position that the next atom will occupy, as well the atom type, thereby generating molecules in an autoregressive manner. In addition, 3DSBDD further standardized the evaluation protocol of the experiment by using Vina Docking Energy to assess the stability of the generated molecules.
chosen effective method
The rise of EGNN has enabled autoregressive modeling of continuous atomic coordinates in the SBDD task while preserving equivariance and invariance properties. Following this development, GRAPHBP adopted the EGNN architecture, using a depth-first search approach to rank the molecules from near to far the protein surface. This allows for autoregressive molecular generation starting from the protein frontier and ultimately modeling a complete molecule. However, GRAPHBP did not consider critical chemical priors such bond lengths and bond angles between atoms, making it challenging to capture the internal patterns of molecules. The evaluation based on Gnina Docking, different from 3DSBDD
Mcanwhile, POCKET2MoL directly modeled atomic coordinates using GVP. Compared to GRAPHBP. POCKET2MOL took molecular bonds into account and employed contrastive learning, where false atoms were used to regularize the coordinates and bonding of real atoms. This significantly improved the stability of the generated molecules. The evaluation follows 3DSBDD.
The rise of generative models, particularly the widespread application diffusion models in the field of images, has drawn attention to these outting-edge generative techniques in the molecular domain as well. Diffusion models are inspired by particle systems in statistical physics, which aligns closely with the task of generating atomic coordinates in molecules. DDFFSBD, as pioneering works in applying diffusion models to the SBDD task, have also achieved significant results. DIFFSBD followed most of the concepts from EDM, such as the zero-center-o mass technique, and was tested and generalized on B Binding MOAD beyond CrossDocked2020
Meanwhile, DIFFBP also proposed use the Gaussian Diffusion model on atoms' positions and masked type diffusion for elements' types. DIFFBP analyzed the drawbacks of autore- gressive models from the perspective of the Boltzmann energy distribution in statistical physics and observed that molecular weight severely affects docking performance, being the first to adopt a grouping approach for model comparison.
After that, following previous explorations like GEODIFF and EDM in molecular (confor- mation) generation, TaRGETDIfF innovatively introduced diffusion model-based SBDD method, achieving new state-of-the -ffth-ar- (SOTA) performance. For validation, TARGETDIFF first proposed using Vina Score and Vina Min mode to assess the stability of the generated molecules' initial states. Additionally, the embeddings learned by the model can also be used for affinity prediction.
On the other hand, fragment-based generative methods are also in the early stages of exploration. FLAG follows the autoregressive generation approach of POCKET2MOL. breaking molecules into fragments and individual atoms, allowing for the generation of complex molecules with fewer steps. However, its modeling approach lacks sufficient consideration of chemical bond participation, which is a key reason for its suboptimal molecular generation performance.
16

---

Published as a conference paper at ICLR 2025
Afterward, D3FG combined fragment- generation with the diffusion method, leverag- ing protein generation techniques to establish a diffusion model that predicts the orientation matrix, central coordinates, and fragment types based on fragment information. The model then uses linkers to connect the molecule fragments. Additionally, D3FG enables molecular optimization by replacing fragments that are either prominent or insufficiently contributing.
In contrast to D3FG, which treats fragments rigid bodies, DECOMPDIFF considers fragments as flexible structures, thereby decomposing molecules into arms and scaffolds, and performing multivariate Gaussian diffusion within different decomposition clusters. Additionally, DECOMPDIFF employs non-intersection guidance to mitigate the issue of atomic overlap between the molecule and protein amino acids.
In addition, work on voxelized maps is steadily progressing. VOXBIND builds the pioneering work of VoXMoL, using ceffuusion-based denoising modeling on the voxelized density map of molecules. During the generation steps, it incorporates wj-sampling to reduce computational complexity, while also producing structurally stable molecules.
More recently, MOLCRAFT was proposed, utilizing the Bayes Flow Network, which is designed for modeling probabilistic densities' parameters. This allows molecules to avoid the difficulty of coupling the two modalities: the discrete variable atomic types and the continuous positions of atoms in the molecule. It effectively addresses issues such as mode collapse and has demonstrated outstanding performance as the most recent SOTA model.
Recently, several cocurrent benchmark works are also proposed for SBDD tasks, such as Zheng et al. (2024), which aims to demonstrate that 1D/2D molecular generation methods remain highly competitive compared to 3D approaches, which insightful for researchers to rethink that if 3D- molecule design is required for the tasks. In contrast, our work focuses solely on generative 3D SBDD tasks. Moreover, we provide a comprehensive evaluation of 3D molecular generation methods, incorporating over 10 3D-based methods compared to the five included in Zheng et al. (2024).
Table 9: A brief review of the included methods and task adaptation in our CBGBench.
Method
Time
Generative Model
Network Architecture
Price Knowledge
Evaluation Datasets
Task adaptation (CBGBench)
LIGAN
Oct, 2020
None
CrossIrckee20200
nove design
3DSBDD
Mar. 2022
Auto-regressive
GNN
None
ChosDDoked2120
Lead optimizarion
De nove design
GRAPHRP
Apr. 2022
Auto-regressive
EGNN
None
CrosDDecked2000
De novo Design Lead optimization
PoCKEr2MoL. May.
Auto ruto-regressive
None
CrosDDccee22200
De novo Design
optimizationn
CrossDelse2220 Binding MOAD
De novo Design
DIESBDD
Oct, 2022
DDPM
EGNN
None
Lead optimization
novo design
DIFEBP
Nov. 2022
DDPM
GVP
None
CrossDockee22000
Lead optimization
TARGETDRE
2023
None
GrosDoeke22000
De design
optimizarion
FLAG
Feb, 2023
Auto-regresive DDPM
GVP
Fragment
CrossDoked2220
De novo design
D3PG
May. 2023
EGNN IPA
Frigment
CrossDocked22000
De design
DECOMPDIFF
Feb, 2024
DDPM
EGNN
Arm&Scuffold
CrossDocked2020
De novo design
MOLCCRAP
2024
BFN
EGNN
None
CressDecked2000
De design optimization
VoXHIND
May. 2024
D0PM
CNN
None
Crosthekkd2000
De novo design
17

---

Published as a conference paper at ICLR 2025
B SUPPLEMENTARY EVALUATION DETAILS
B.1 INCLUDED FUNCTIONAL GROUPS
Here show the functional groups included in this paper, in which we follow D3FG and give demonstration of them in Table. 10.
Table 10: The included functional groups CBGBench. 'T' is the occurrence times of the functional group in the datasets (100,000 ligands). 'A,B,C' are the framing node index.
Smiles
2D graph
3D structures
A C
T
CM44 Cits
elecceel
1 0 131148
CA
NC=0
NH2
1
49023
O=CO
1
0 2
39863
CH
OH4 CRt
cleencel
NO
C0
3 2
15115
C CH11
cIncc2nc[nH]c22
7 3
11369
0.3
NS(=0)=0
1
10121
2N:0
0:2
O=)(O)(O)O
.0H2
1
7451
OHH
OCO
HB12
.110
1 0
6405
Crer1
N: Cws
elenenel
CHCa
o
3
5965
CH11
Ni CH11
clen[nH]el
2
5404
NH.3
18

---

Published as a conference paper at ICLR 2025
Smiles
2D graph
3D structures
A
B
C
T
DH-3
O=P(O)O
center(2.3) 5271
clooc2occc22c1
2
4
4742
cleesel
2
4
4334
N=CN
R43
Mn
2
4315
RIDS
NC(N=O
1
3
4167
O=cIcc(nHH|(E=O)nHH)
5
4145
clecc2nccec2e1
2
4
3519
clesen1
3466
clccc2[nH]]ene2l
4
6
3462
clc[nH]enl
4
2964
cl
0=[N+][O-]
0
2
2702
O=CNO
10
2
2477
NC(=OJO
2
2438
0=S=0
2
2375
2lccc2|nH]eeveel
4
2
2301
19

---

Published as a conference paper at ICLR 2025
B.2 LBE ANALYSIS
The LBE is motivated by the following phenomena in Fig. B.2, which shows that the Vina Energy has an strong correlation with the atom number. Besides, Appendix B4, give a more detailed analysis of LBE.
Aures BAG22 JAPGAL 2C, MOLEHEALA. S00r AWL. JHOMEEE
1UT WIMUTTAAL ASACCA SOC NC VESAA ANOWLE pROdIE
Pootes 101
CRUIE
SERUES
Rac
of
GSTPLINNNN ADVGLA WCJ.p CRLEIEEE FUEEEEW DUVIL pechees
TEPL JANEL AWN AAC ELPVE Gpeaess
Pigeas
OS
12
Figure 6: A demonstration of how the molecule size affects the binding energy. The 6 pockets that most linear-correlated with atom number.
B.3 METRIC CALCULATION
The metric of MPBG can be written as MPBG, Mean, Ex the indicator of the generated molecules in single protein and Mean,(-) calculates average along indicator i, with MPBG = Meany (MPBG,) The MPBG is a per-pocket metric that calculated within single pocket, which differs from other averaging metrics.
Erst
x 100%), in which is
For chemical property calculation, we use functions in RDKit (RDKit, online), and note that the SA the normalized one employed in the previous studies.
For interaction pattern analysis, we employ PLIP, which considers 7 kinds of interaction types includ- ing 'hydrophobic interactions', "hydrogen bonds", 'water bridges', 'æ-stacks', *и-cation interactions' "halogen bonds' and 'metal complexes'.
For MAE metrics that are related the molecule sizes, the generative model tends to achieve higher scores when the atom number is close to that of the reference. We suppose it is reasonable, considering that the size of the molecule itself is determined by factors such as the size and shape of the pocket, and these characteristics should be captured by the models.
B.4 DISCUSSION ON LBE.
it possible that smaller molecules exhibit higher efficacy, thus leading LBE to be meaningless? (Reynolds et al., 2008)
Meaning. The introduction of Ligand Binding Efficiency (LBE) is motivated by observations in Appendix B.2, where identified a strong positive correlation between the size of the generated molecules and the Vina docking energy. To address this size-related bias, LBE was incorporated additional metric. This approach is consistent with the basic principles of energy calculation,
20

---

Published as a conference paper at ICLR 2025
Table 11: Ratio of generated molecule's 's size and the corresponding LBE.
Method
Mol Size
[1.3]
(3,10)
(10,20)
(20,30)
>30
LiGAN
Ratio(%)
0.01
13.61
40.09
36.77
9.59
LBE
0.37
0.42
0.44
0.35
0.30
Ratio
0.01
15.48
54,44
22.63
7.42
3DSBDD
LBE
0.39
0.42
0,41
0.33
0.31
Ratio( ()
0.00
0.77
91.29
6.07
1.84
GRAPHBP
LBE
0.36
0.33
0.21
0.20
POCKET2MOL
Ratio (%)
0.08
24.22
48.83
20.19
6.64
LBE
0.43
0.45
0.44
0.33
0.30
TARGETDIFF
Ratio %))
0.00
10.24
28.17
39.51
22.05
0.44
0.44
0.31
0.29
Ratio %) LBE
0.13
21.54
41.43
28.01
8.86
DIFESRDD
0.37
0.38
0.30
0.24
0.22
Ratio (%)
0.00
7.29
32.19
43.08
17.42
DIFBBP
LBE
0.41
0,42
0.31
0.29
Ratio(%) (6)
0.04
39.01
54.29
6.52
0.13
FLAG
LBE
0.40
0.35
0.33
0.24
0.19
Ratio %)) LBE
0.00
8.31
61,70
27.72
2.55
D3FG
0.44
0.45
0.32
0.25
Ratio %))
0.00
9.99
35.50
33,98
20.51
DECOMPDIFF
0.38
0.38
0.33
0.30
Ratio (%)
0.00
9.18
30,97
39.62
20.21
MOLCRAFT
LBE
0.44
0.41
0.33
0.29
Ratio
0.00
6.10
36.64
43.39
15.85
VOXBIND
LBE
0.43
0.41
0.34
0.26
where the total binding energy is derived from the sum of interaction energies contributed by each atom, i.e. Evina ~ Ebind = _i= E.. Thus, using LBE as a measure of the average per-atom contribution to binding affinity is both logical and appropriate, as LBE numerous studies in medicinal chemistry have highlighted Ligand Efficiency as a critical evaluation criterion, advocating for its widespread use an effective metric for assessing molecular binding affinity (Hopkins et al., 2014; Kenny, 2017; Murray et al., 2010).
NN Natom Furthermore,
Eaind
Resonability. It can be usually concluded from Reynolds al. (2008) that 'smaller molecules tend to have higher LBE', while we believe this conclusion is incorrect. We claim that different protein pockets exhibit preference for molecule sizes. If this conclusion were true, would imply that molecules consisting of a single atom would have the optimal LBE? To explore this issue, in Table. 1, we give statistics of molecule sizes v.s. LBE of methods evaluated. shows that such molecules with extreme sizes (less than 4) that make the LBE meaningless are scarcely generated since they have small ratio. Ligands with sizes less than 20 usually have better LBE because the preference of the pockets, as consequence of protein binding sites being limited in their size, thus generating the preference of binding molecules' sizes. Larger ligand with sizes more than 20 gets decreased LBE because it becomes increasingly difficult to form optimal interactions with every site on the protein without introducing unfavorable ligand strain.
Little Effects of Extreme Samples. Besides, use weighted ranking with equal weights as previous benchmarks (Wang et al., 2022). to give an unbiased evaluation. Given that LBE accounts for one-twelfth of the total weight in interaction analysis, the impact of failed samples in LBE on calculating the LBE metrics and final ranking is minimal. Therefore, we claim that the inclusion of a minimal number of invalid LBE calculations will not render the overall ranking ineffective.
B.5 PREVIOUSLY-USEN METRICS FOR EVALUATION
The aspects of evaluation commonly used with different methods, while they hardly give very comprehensive evaluation. We give brief review of each method's evaluation on these aspects in Table. 12.
21

---

Published as a conference paper at ICLR 2025
Table 12: Different methods' evaluation aspects the published papers.
Method
Substructure
Geometry
Chem Property
Interaction
LIGAN
Figure. S6
Figure. S7, S8, S9
Figure. S3, 84, SS Table.
Figure. S13, S14 Table.
POCKET2MOL
Table.
Table. 3; Figure. 4
GRAPHBP
Table. 2; Figure. 5
Table. 1; Figure.
TARGETDIFF
Table.
Table. 1; Figure.
Tahle.
Table. 3, Figure.
DIFFBP
Table.
Table.
Table.
DIFFSBDD
Figure.
Table.
Table. 1, 2: Figure, 2
FLAG
Table.
Table. 2; Figure. 4 Table. Table. 1, Higure. 3 Table.
Table. Table. 4 Tahle. 3
Table.
D3FG
Table. 1, 3; Figure, 3
Table.
DECOMPDIFF
Table.
MOLCRAFT
Table.
Table. 2
Table.
VOXBIND
Figure.
Table.
Table. 1; Figure. 5
C CODEBASE STRUCTURE
In this section, we provide an overview of the codebase structure CBGBench, where four abstract layers are adopted. The layers include the core layer, algorithm layer, chemistry layer, and API layer in the bottom-up direction as shown in Fig. This codebase is licensed under the Apache License, Version 2.0. It provides robust and flexible framework for building and evaluating graph neural network models for structure-based drug design and lead optimization.
CBGBench: Complex Binding Graph Benchmark
Chem. Prop.
Geometry
Generation
Evaluation
Interaction
Substructure
Chemistry
Rdkit Utility
Vina Dacking
Scoring Func.
Protein Motif
Fragment Motif
Algorithm
Base
Embedding
Encoder GVP
Method
Config
Time
AutoRegressive
Dataset
TargetDiff DilISBD
Pocet2Mol
Data Loader
Angle
EGNN
GraphBP
Model
Atem
ITA
DiRBP
FLAG
Opliizze
Residue
Transform
Fragment
Core
Datase!
Utility
Module
Training
Sampling
LLoses
MLP
LR Scheduler
Sampler
Linker
Distribution
Attention
TensorBoard
Bond Constru.
Fragment
S033 group
Context
Side Chein
Sphere
Vanilla GNN
Checkpoints
Mol Saver
Scaffold
group
GIN
Logging
Figure 7: Structure of CBGBench Codebase, consisting of 4 layers. The core layer provides the common functions, datasets, and modules for CBG methods. The algorithm layer mainly implements the prevailing CBG algorithms. The Chemistry layer is used for data preprocessing and evaluation.
Core Layer. In the core layer, implement the commonly used functions for training and sampling CBG algorithms. Besides, the code regarding datasets, data loaders, and basic modules used in CBGBench is also provided in the core layer.
22

---

Published as a conference paper at ICLR 2025
Algorithm Layer. In the algorithm layer, first implement the base class for CBG algorithms, where we initialize the datasets, data loaders, and basic modules from the core layer. We modularize each method, first using embedding layers to project the imput into high-dimennional space, and then employing different 3D equivariant GNNs to construct the algorithm, including Diffusion-basee and auto-regresssiv--aseee ones. We further abstract the algorithms, enabling better code making it casier to implement new algorithms, The voxel-based methods are not included because our framework is mostly based on EGNNs, which will be added the codebase as future work, and DecompDiff requires different data preprocessing. Based on this, we support core CBG algorithms. More algorithms are expected to be added through continued extension.
and
Chemistry Layer. The chemistry layer is mostly built with post proceerocsssing utilities and evaluators such as atom type, bond angle, and docking modules. Besides, several prior decomposition functions are used as molecule parsers, and the molecule and protein fragment motifs serve as prior knowledge.
API Layer. We wrap the core functions and algorithms CBGBench the API layer as public Python package. It is friendly for users from different backgrounds who want employ CGB algorithms new applications. Training and sampling can done only a few lines of code. In addition, provide the configuration files of all algorithms supported in CBGBench with detailed parameter settings, allowing the reproduction of the results.
D EXPERIMENTAL IMPLEMENTATION DETAILS
Experiments are conducted based on Pytorch 2.0. I on hardware platform with Intel(R) Xcon(R) Gold 6240R @ 2.40GHz CPU and NVIDIA A100 GPU, We give the detailed parameters for training the included models Table. 13. Note that most the hyperparameters' combinations are directly ones. Specifically, The methods included in codebase either self- self-trained or trained by us when the official repository does not provide a pretrained checkpoint. For methods that are not included, validate them using the pretrained models that have been provided. In the Lead-Ope cond---timization tasks, the finetuning parameters are the same.
the
Table 13:
The arper-paraeetes in for training and sampling molecules for the SBDD methods. Different from GNN--bsee methods, CNN-based methods usually have different hidden dimension size, so we report the highest ones. 'Num steps' are the sampling steps which is usually used in diffusion-baa based methods, including DDPMs and BFNs. For other read the config files provided in the supplementary materials from /
detail, please
/traiir..
Method
Bat ea Size
Midden Dim
Layer num
Learning rate
Optimizer
Nun. Steps
LIGAN
8
128
4
1E-05
RMSprop
3DSBDD
4
256
6
IE-04
Adam
GRAPHBP
16
64
6
IE-04
Adam
/
POCKET2MOL
8
256
6
IE-05
Adam
TARGETDIEF
4
128
9
SE-04
Adam
1000
DITSBDD
16
256
6
1E-03
Adam
500
DIFFBP
16
256
6
1E-04
Adam
1000
FLAG
4
256
6
1E-04
Adam
/
D3FG
16
128
6
1E-04
Adam
1000
DECOMPDIFF
4
128
SE-04
Adam
1000
MOLCRAFT
8
128
9
SE-04
Adam
1000
VOXBIND
32
512
4
1E-05
AdamW W
100
E EXPERIMENTAL RESULT DETAILS
E. DE NOVO GENERATION
F. SUBSTRUCTURE
Table. 14 gives the generated molecules' atom distribution in detail. It shows DIFFBP and GRAPHBP tend to generate more atoms and have a low probability generating uncommon atoms. In contrast, D3FG and FLAGG, which directly motif library, have higher probability of generating these
23

---

Published as a conference paper at ICLR 2025
uncommon atoms. Table. 15 gives the generated molecules' ring distribution in detail. DIFFBP and DIFFSBDD perform poorly due to significant inconsistencies in generating large complex fragments. For example, they also generate large number of unreasonable triangular and tetrahedral rings. Similarly, these methods also fall short in generating complex functional groups, as shown in Table. 16. This leads to their subpar performance in substructure generation.
Table 14: Distribution of different atom types across different methods.
Method
C
N
0
F
CI
Ran
0,6715
0.1170
0. 11696
0.0131
0.0111
0.0112
0.0064
LIGAN
0.6477
0.0775
0.2492
0,0025
0.0234
0.0019
0.0003
3D5RDDD
0664
0.1311
01665
0.005
0.06
0.0000
0.000
GRAPHRP
08610
0.0997
00865
0.0016
0000
0.0099
0,031
2MoL
07623
0:8555
0.1413
TARGumbrr
DD
0.2000
0.1154
Due
019778
0.0030
01792
0,030
01000
0,000
00300
FLAG
015585
0.1541
0.207
00065
00313
0,847
01974
D3FG
0.7336
0,1158
0.1286
0.0356
0.033
0,0088
0.0000
DecosrDurr
0.6762
0.5978
0.1927
0,054
0.0189
0.0058
0.0033
CRLCK
B.6735
0.0917
0.2066
0.0108
09044
0.0058
0,035
VoxBIND
0.7380
0.1083
0.390
0.0000
0.006
0.0130
0.732
Table 15: Distribution of different ring sizes across various methods.
Method
Ram:
0.0130
0.0020
0.2855
0.0099
0.0005
LIGAN
0,2238
0.0698
0.2599
3.4089
0.0171
0.0096
IDSBDD
0.2970
0.0077
0.1538
0.5114
0.8181
0.0116
GRAPHBP
0.0000
0.2129
0.1922
0.1765
0.1533
0.1113
PoCKET2MoL
0.0000
0.1585
6.18
0.4375
0,1410
0.0478
TARGETDDY
0.0000
0.0188
0.2856
0.1209
0.0298
DIFFBDD
0.2854
0.0718
0.0193
0.0000
0.2155
0:2371
0.2115
0.1417
0,0707
FLAG
0.0000
0.0682
0.2716
0.5228
0.0996
0.0231
D3PG
0.0000
0.0201
0.2477
0.3966
0.0756
0.0283
DiCOMrDio
0.03022
0.0378
0.3407
0.4386
0.1137
0.01%
MOLCRAFT
0.0000
0.0022
0.2494
0.6822
0.0489
0,0072
VoXBIND
0.0232
0.0021
Table 16: Distribution of the top ten functional groups across different methods.
Method
clexeel
NC=O
O=CO
clocncel
cinec2nclaH]ccnl
NS(=0==0
O=P(OXO)O
OCO
elenenel
clenjnH]el
REF.
0:3920
0.1465
0.1192
0.0301
0.0191
0.9162
LIGAN
0.3464
0.0998
0.1549
0.0546
0.1028
0.0127
0.0490
0.0897
0.0197
0.0000
3DSHDD
D.3109
0.1488
0.1219
0.7769
0.0090
6.0010)
0.0814
0.0915
0.0148
0.0291
ORAPHBP
0,1888
0.0000
0,0000
PoCKET2MoL.
0.3794
0.1098
0.2906
0.0305
0.0000
0.0000
0.0150
0.0954
0.0030
0.0000
TARGETDUP
D.2729
0.1520
0.3085
0.0427
0.0061
0.0000
0.0241
0.0855
0.0069
0.00K5
DIFSBDD
0.0073
0.1985
0.5787
0.0194
0.0000
0.0073
0.0000
0.0145
0.0000
0,0000
DIFBP
0.1962
0.0108
0.4440
0.0010
0,0000
0.0000
DO0000
0.3289
0.0000
0.0000
FLAG
0.2223
0,1590
0.1736
0.0365
0.0009
0.0000
0.0014
0,3471
0.0135
0.0014
D3FG
0.3002
0.1597
0.2078
0.0323
0.0114
0.0000
0.0212
0.0751
0.0410
0,0000
DECOMPDUFE
0.2125
0.0527
0.0057
0.0128
MOLCRAFT
0.3960
0.1798
0.2361
0.0377
0.0001
0.0000
0.0362
0.0344
0.0053
0,0002
VOXBIND
0.4705
0.1138
0, 0,1663
0,0762
0.0001
0.0000
0.0191
0.0334
0.0110
0,0000
E.1 INTERACTION
PLIP Interaction Pattern. Table. 17 and 18 provide detailed interaction pattern analysis. Most models captured good interaction patterns, generating a substantial amount of hydrophobic and hydrogen interactions. DifISBDD excessively generated hydrophobic interactions. Additionally, for interactions such as r-stacking, 7-cation, and halogen interactions, which do not exist in the reference molecules, the models could probabilistically generate molecules capable of producing such interactions with the protein.
E.1 GEOMETRY
Here we give detailed bond length and bond angle distribution of the evaluated method's 'ss generated molecules and the reference, in Table. 19 and 20. It shows that the
24

---

Published as a conference paper at ICLR 2025
Table 17: Frequency of interaction type.
Method
hydrophobic
hydrogen
bridge
T-stacks
m-cation
balogen
metul
Roe.
3,000
0.0000
0.0000
0.0030
0,0000
0.0000
LIGAN
3.5078
0.0000)
0.1045
D:9441
3DSBDD
3,1237
3.0853
0.0000
0.1485
0.0434
0.0053
0.0000
GRAPHRP
5.2008
03809
0.0000
0.0007
0.0046
0.0050
0.0000
POCKEY2MLL
4.9266
1.88.10
0.0000
0.0176
0.0081
0.0045
0.0000
TARGETDIFF
4.244
3,5309
0.0000
0.1159
0,6377
0,0384
0.0000
DEYSBDD
3.3202
0.0000
0.0586
0.0338
0.0279
0.0000
DIFEBP
1.4830
0.0213
0.0039
0,0020
FLAG
193199
2.1490
0.0000
0.0134
0.8185
0.0528
D3PG
4.3149
2.4718
0.0000
0.0607
0.0260
0.0200
0.0000
DECOMPDEF
3.0244
3-5511
0.0000
0.1177
0.0436
0.0290
0.0000
Mor.CRAFT
4.0147
3,9456
0.0000
0.2038
0.0667
0.0377
0.0000
VOXBIND
4.4920
2.8385
0.0000
0.1659
0.0599
0.0001
0.0000
Table 18: Distibution of interaction type.
Method
bydrophobic
hydregen
water bridge
r-stacks
T-casion
balogen
metal
REa:
0,5000
0.5000
0.0000
0100000
0,0000
0.0000
0.0000
LIGAN
0.4614
0.5162
0.0000
0.0154
0.0065
0.0005
0.0000
3DSBD
0.4882
0.4822
0.000
0.0220
8.088
0.008
0.0000
GRAPHBP
0.8615
0.1336
0.0001
0.0010
0.0000
PoCKET2Mo1.
0.7206
0.2750
0.0026
0.0022
0.0077
0.0000
TARGETDIF
0.5327
0.4432
0,0000
0.0146
0.0047
0.0048
0.0000
DIFESDDD
0,5265
0,1198
0.0000
0.0134
0.0016
0.0008
0.0000
DReBP
0.6879
0,3398
0.0000
0,0020
3.004
0.0000
0.0000
FLAG
0.4638
0.5159
0.0300
0,0033
0:0045
0.0127
0.0000
D3PG
0.6259)
0.35%
0.0088
0.0029
0
DEcoMrDar
0.5188
0.2500
0.0155
3.0058
0.0038
0.0000
MOLCRAFT
0.4855
0:4771
0.0000
0.0246
0.0080
0.0045
0.0000
VoxBrrn
0,5943
0.3757
0.0000
0.0219
0.0007
0.2012
0.0000
Table 19: JSD Bond Length Comparisons across different methods.
METrOn
C-C
C-N
C-O
C-C
C-N
C-O
LIGAN
0.4986
0.4146
0.4560
0.4807
0.4776
0,4595
3DSBDD
0.2090
0,4258
0.5478
0.5170
0.6701
0.6448
GRAPURP
0.5038
0.4251
0.4973
0.6235
0.4629
0.5986
PoCKEr2MoL.
0.5667
0.5698
0.5433
0.4787
0.5989
0.5025
0.3101
0,2400
0.3072
0.1715
0.3629
DEYSBDD
0.3841
0.3708
0.3291
0.3043
0.3473
0.3647
DIFBP
0.5704
0.5256
0.5090
0.6161
0,6314
0,5296
FLAG
0.3460
0.3770
0.4433
D.4872
0.4464
014292
D3GF
0.4244
0.3227
0.3895
0.3860
0.3570
0.3566
DECOMPDIFF
0.2562
0.2007
0.2361
0.3091
MOLCRAFT
0.2473
0.1732
0.2341
0,3040
0.1459
0,2250
VOXBIND
0.3335
0.2577
0.3507
0.19
0.1459
0.3334
Table 20: JSD Bond Angle Comparisons across different methods.
J5DDA
LaGAN
3DSBDD
GRAPURP
PackuriMoL
TARCETDOE
DIESBDD
DIERP
FLAG
D3GF
DECOMPDim
MOLCLAFT
VOXBIND
0.4839
0.7307
0.4591
0,7027
0.8174
0,4922
CCNN
0.3161
0.7120
0.2252
0.2199
0,370
0.1925
0,3593
0.1987
0,1097
0.2201
0.2706
0.2315
0.2767
CC-N
0.4159
0.4376
3.3507
0.5171
0.3305
0.2064
CX-C
0.5209
0.3176
0,3081
0.4155
0,3597
0.192
0.1475
CN-N
0.5589
0.54013
04161
0.3505
0.2625
0.1813
CN-
0.7019
B.4926
06638
0.4653
0.7070
05500
0.4943
0.2002
0.3300
CN-C
0.3645
1.4011
0.113
0.4517
0.3667
04092
0,3636
CN-8
0.3387
0.214
0.7639
0.4212
0.5326
0.703
0.3917
0.954
0.7196
COC
0.1111
0.4219
0.4244
0.3756
0.5478
0.4066
0,3765
01188
0.1113
0,3305
CON
0.7893
0.3865
06637
0.6100
0.5655
0.3520
06257
0,6963
0,6065
0,3336
0,5436
C-C-N
0.450
04140
05415
0.3752
0.339
0.3225
0.4055
03361
0.3126
02574
0,1900
0,2505
C-C=C
077593
0.6866
07793
0.7373
0.3455
0.7549
0.775)
07419
0.760
07703
06188
0,8326
NAC-C
08151
0.3050
08396
0.5840
0.3417
0.7388
0.7928
03361
0.7133
07754
0:353
6.7130
N-C-
0.517
0.1705
0.179
0.5544
0.3058
0.4409
0.6764
04316
0.4464
0.2994
6360
0.2569
X-CO
0.7133
[12673
0.0244
0.5879
0.3926
0.4346
0.5023
0.4389
0.4967
0.302
0.3612
0,190
X--N
0.49g
[13670
0,3450
0.3986
0.2175
0.3558
0.5496
02264
(.3531
0,2913
0.1366
0.1360
N--O
0,5275
I13800
0.4255
0.2347
0.2664
0.3690
0.5719
0.363
0.3693
0.1197
0:075
0.2508
NN-O
0.8266
0.6488
0.7791
0.7639
0.5862
0.5912
0.7447
07117
036633
0.4331
0.6705
0,5859
N-CX
0.6595
1.3670
0,3450
0.3985
021155
0.3558
0.5498
0.664
0.3531
0,2993
0.1366
0.1360
N-C
0.5275
0,3800
0.4285
0,2347
0.2664
0.3690
0.5719
0.3166
0,3695
0.1197
03075
0.2508
E.2 LEAD OPTIMIZATION
We here give the details of interaction analysis. Since these tasks have provided partial of the molecules as the context, it is hard to tell the superiority of generated substructures or geometries. Table. 21 and 22 gives details. it shows that
25

---

Published as a conference paper at ICLR 2025
Table 21: Frequency of interaction type lead optimization tasks.
Method
hydropphobic
hydrogen
bridge
x-stacks
halogen
REP:
3,0000
3,0000
0.0000
0.0000
0.0000
0.0000
0.0000
GRAPHBP
5,5795
2,9812
0.0000
0.5362
0.3537
0.0000
0.0000
Linker
PoCKEY2MoL
5.1879
3.2739
0,9000
0.1708
0.1082
0.0326
0.0000
TARGETDIFF
5.3811
3,7133
0.0000
0,1675
0.0827
0.0404
0,0000
5.4132
3.1350
0.0000
DIFFRP
7.0697
2,6017
0.0000
0.1226
0.0678
0.0204
0.0000
GRAPHBP
3.0084
3.2071
0,0000
0.0947
0.0499
0.0196
0.0000
Fragmectt
PoCKeT2MoL
4.4064
4,0854
0.0000
0.1238
0.0956
0.0131
0,0000
TARGRDDIFF
4.3187
3,6436
0,9000
0.1040
0.0648
0,0405
0.0000
DIFSBDD
4.3086
3,6720
0.0000
0.0468
0.0000
DIFFBP
5.6128
2,7838
0,0000
0.0552
0.0339
0.0175
0.0000
Sidk Chain
GRAPHBP
3.0173
1.2207
0,9000
0,2029
0.0321
0.0001
0,0000
POCKET2MoL
3.0643
3.8911
0.0000
0,0872
0.0788
0.0194
0,0000
TARGETDIFF
3.3468
3.6206
0,9000
0.0975
0.0527
0.0851
0,0000
DIFSBDD
3.6822
3.3479
0,0000
0.0271
0.0242
0.0098
0.0000
DIFFBP
4.5444
2.5941
0,0000
0.0696
0.0349
0.0141
0,0000
GRAPHBP
0.9560
2.3953
0,9000
0,0000
0.0007
0,0007
0,0000
Scaffold
PaCKFY2Mal.
3,4761
TARGETDUPF
3.5427
4.1047
0.0000
0.0612
0.0383
0.0303
0.0000
DIFSBDD
4.6225
3.2649
0,9000
0.0021
0.0138
0,1786
0,0000
DIFBBP
5.0840
2,4424
0.0000
0.0061
0.0055
0.0110
0.0000
Table 22: Distibution interaction type on lead optimization tasks.
Method
hydrophobic
hydrogen
water bridge
x-stacks
s-cction
halogen
metal
Riar
0,5000
0.5000
0.0000
0,0000
0.0000
0,0000
0.0000
GRAPHBP
0.3154
0.0000
0.0567
Lanker
PocKEr2MoL.
0.5912
0.3733
0,0000
0.0195
0.0123
0.0037
0.0000
TARGErDDN
0.5734
0.3957
0.0000
0.0178
0.0888
0.0043
0.0000
DDF
0.6216
0.3600
0.0000
0.0132
0.0015
0.0035
0,0000
DIFFBP
4.1027
1,4830
0,0000
0.0213
0.0039
0,0021
0.0000
0.4716
0.5027
0.0000
Fraguncnt
POCKET2MoL
0.5051
0.4683
0.0000
0.0141
0.0110
0.0015
0.0000
TARGETDIFF
0.5285
0.4459
0.0000
0.0127
0.0079
0,0050
0.0000
DDESBDD
0,5289
0,4508
0.0000
0.0133
0.0011
0.0057
0.0000
DIFFBP
0.6601
0.3274
0.0000
0.0065
0.0040
0.0021
0.0000
Sidk Clain
GRAPHBP
0.6746
PucKEr2Mon.
0,4296
0,5455
0.0000
0.0122
0,99
0.0027
0.0000
TARGETDIFF
0.46417
0.5027
0,0000
0.0135
0.0073
0.0118
0,0000
DDVSBDD
0,5192
0,4721
0.0000
0.0038
0.034
0.0013
0.0000
DIFFBP
0.6262
0.3575
0.0000
0.0096
0.0048
0.0019
0,0000
GRAPHBP
0.2851
0.7144
0.0000
Scaffold
POCKET2MOL.
0,4752
0.5127
0,0000
0.0042
0.0043
0.0036
0.0000
TARGErDIE
0.4555
0,5278
0.0000
0.0079
0.0049
0.0039
0.0000
DIFFSBD
DIYBBP
0.6735
0.3235
0,0000
0.0008
0.0007
0.0015
0.0000
The optimization on the side chains has the least effect on the interaction pattern in an overall result and has the greatest influence on the linker.
GRAPHBP's failure in scaffold hopping also be reflected by the fact that the interaction pattern cannot be produced with the molecules.
E.3 CASE STUDY
DRD3. For DRD3, we give the distribution of different models' generated molecules in chemical space in Figure. and 9. Different methods capture different clustering centers of the actives, while their distributions show great gap from the GEOM-DRUG's randomly sampled molecules.
E.4 VALIDITY
Validity is an important metric to evaluate whether the molecules generated by the models are valid. There are two methods to reconstruct the 3D positions of the atoms into a molecule with bonds, one is used in TIFGERDIFF, POCKET2MOL and 3DSBDD, which name Refine; The second is to use Openbabel (Open Babel development team) the software, used DIFFSBDD. However, using these methods reconstruct molecules always carries the risk of broken bonds, which makes
26

---

Published as a conference paper at ICLR 2025
Vina Docking Energy
2ooi
Graph
Targ
Actves
RocketaMol
Graphep
TargeDiif
DiffSBDD
Drffer
PLAG
D3RG
MatCratt
Activess Dr1s8D0
Actives
Actives
D
(a)
Distribution of Vina Dock Energy
Ligand Ligand Binding Efficacy
0.8
0.6
Actives
Actives MoiCraft
Actives GEOM DRUG
D3FG
0.4
0.2
0.0
Actives
PacketMMal
Graphap
Brgeebiff
DMSBOD
Dmtaee
FLAG
DaFG
MolCraft
Figure 8: T-SNE visualization of chemical distributions of generated and active molecules on DRD3.
(b) Distribution of LBE
Figure
Binding affinities on DRD3.
Table 23: Denovo Validity and Rank
Method
Validity
Rank
LiGAN 0.42
12
3DSBDD
0.54
11
GraphBP
0.66
10
Pocket2Mol
0.75
6
TargetDiff
0.96
DiASBDD
0.71
7
DiffBP
FLAG
9
D3FG
0.77
DecompDiff
0.89
MolCraft
0.95
2
VoxBind
0.74
8
the strategy of selecting connected atoms form fragments and ultimately the final molecule crucial. Here, we define chemically valid molecule as one where the number of atoms in the largest fragment greater than 85% of the total number of atoms, and we use this as the criterion for reconstruction. Additionally, employed 'Refine + Openbabel' strategy, where if refinement is unsuccessful, Openbabel is added as a reconstruction method. Under this definition, the validity of various methods in the de novo task is shown in Table. 23. DecompDiff and TargetDiff' have the highest validity.
27


Reviews:
[{"review_id": "4vbjYIGfwh", "reviewer": "Reviewer_Fmfm", "summary": "This paper proposes a benchmark for SBDD, including a unified framework of generative graph completion for multiple tasks in the field and a comprehensive evaluation protocol.", "questions": "1. Why do you choose different GNN architectures for auto-regressive and diffusion-based models?\n2. MolCraft seems to be missing from the t-SNE visualization in Figure 8. Also, the distributions of Vina Dock Energy and LBE are not provided. Could you please provide these results?\n3. [credit to Associate PC] How might your benchmark and evaluation protocol need to be adapted to assess unified models that can generate both small molecules and macromolecules like proteins?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The unified code base is a nice contribution to the community and beneficial for future research.\n2. The evaluation protocol is comprehensive with a reasonable benchmark setting.\n3. The benchmarked methods are representative and state-of-the-art.", "weaknesses": "1. It would be better if the author could discuss the recent trend of training a unified model for small molecules and macromolecules such as proteins and nucleic acids, and its implications on the field of SBDD.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nkK47Qf6CP", "reviewer": "Reviewer_kydR", "summary": "The paper proposes CBGBench, a comprehensive benchmark for SBDD tasks, which aims to unify various generative models in a fill-in-the-blank framework for 3D complex binding graphs. It introduces a modular and extensible framework and evaluates multiple state-of-the-art methods across different metrics, including interaction, chemical properties, geometric authenticity, and substructure validity. The paper also introduces four sub-tasks: linker, fragment, side-chain, and scaffold design, to provide insights into lead optimization applications.", "questions": "Include more systems that showcase corner cases for a benchmark that are common in real scenarios. i.e KRAS12, BRD4\n\nThe comments were satisfactorily addressed.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper fills a notable gap in the SBDD domain by providing a well-structured and unified benchmarking framework. The comprehensive evaluation protocol addresses the diverse nature of generative tasks.\n\nThe study uses extensive metrics to evaluate models, including metrics like Ligand Binding Efficiency (LBE) to address the size bias in generated molecules.\n\nApplication to real pharmaceutical targets like ADRB1 and DRD3 demonstrates the practical potential of the benchmark and supports the generalizability of the findings.\n\nOverall it's a really well executed paper that focuses on an particular case of generative models, which is drug design, but deserved an acceptance to the main conference due to the relevance of the task.", "weaknesses": "Further testing on more diverse real world systems should be done. Exploring how this models behave in systems like KRAS12 for instance where the main goal is growing into subpockets, would enrich this study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5pd5M8wUD5", "reviewer": "Reviewer_g76L", "summary": "This paper introduces CBGBench, addressing the gap where the absence of standardization can lead to unfair comparisons and inconclusive insights. On one hand, this paper incorporates recent state-of-the-art studies into a unified framework for fair comparisons; on the other, it extends de novo SBDD tasks to include lead optimization and other related tasks. The paper conducts extensive experiments on these tasks and provides the corresponding code.", "questions": "See Weakeness. The W1 and W2 are my main concern, and if a satisfactory response can be provided to the above issues, I will consider increasing the score.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. This paper has evaluated almost all the prevailing SBDD methods in generative models in AI conference. \nS2. It adapt some of the models to a range of tasks essential in drug design, considered sub-tasks within the graph fill-in-the-blank tasks.\nS3. It establish a training, sampling and evaluation codebase, which is comprehensive and effective affter my testing.\nS4. The benchmark evaluates all the models for the two real-world target proteins, as a solid case study.", "weaknesses": "W1. Doubts of classification: In Line 165, it states that MolCraft using BFN as the variant of the diffusion models. However, I have two questions, Firstly, it appears that “one-shot model” in the paper refers to diffusion-based models. Given this, is it reasonable to classify BFN as a diffusion model? MolCraft generates in parameter space, while diffusion generates in data space, which I believe is a distinction. Is the authors’ classification reasonable in this regard?\n\nW2. Doubts of evaluation metrics:  The article compares the clash ratio between protein and molecule interactions, denoted as Ratio_cca . Additionally, it defines the internal clash ratio within molecules, denoted as  Ratio_cm . Regarding the internal clash within molecules, since there are bonds connecting atoms, the defined “van der Waals radii overlap by ≥0.4Å” does not hold. For example, if this definition were applied to a benzene ring, the clash ratio would be 1.0, which fails to reflect the actual structural integrity of the molecule. Therefore, I believe this metric is unreasonable. \n\nMinor: \nAdditionally, the article lacks citations for recent work related to SBDD and molecular generation, such as [1] [2]. I suggest that the authors include these references to enhance the completeness of the article.\n\n[1] Zaixi Zhang, Mengdi Wang, Qi Lium. FlexSBDD: Structure-Based Drug Design with Flexible Protein Modeling\n[2] Odin Zhang，et al. Deep Lead Optimization: Leveraging Generative AI for Structural Modification", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t2NRZohugC", "reviewer": "Reviewer_G2c6", "summary": "The paper proposes an extended benchmark protocol for SBDD methods, and does a large comparison of existing literature, and draws some insights from the results.\n\n-----\n\nPost-response update. My initial confusions have been adressed in the response, as well as issues in motivation. This is a useful contribution to SBDD's, with weaknesses wrt incrementality, presentation and experimental insights. I'm increasing my score to borderline, and I'm still leaning on rejection while I wouldn't mind acceptance.", "questions": "- It would be useful to extend the method table by also including which tasks they support, and their architectures and losses\n- Training losses should be discussed\n- The paper first describes the main de novo task very superficially, then it describes the subtasks more in-depth, then it goes back to describing the de novo task more in detail, and finally goes back to subtasks. It would have been much clearer to separate the main/sub tasks into different parts of the paper.\n- The task description is missing the objectives/metrics. Overall I didn’t really understand the role of the four subtasks. These are very specific, and surely each method needs to have custom support for them. Which ones do? Do we expect new methods to support these 4 tasks? Isn’t it enough that a method is good at the “basic” de novo generation? I’m a bit confused why we care about these at all, or what is their significance? It’s also unclear how they relate to the probabilistic model or ML modelling. Are these some kind of conditionals?\n- It’s unclear if the 4 tasks in table 2 have something to do with the main task. That is, do they share data in some sense? Are the 4 tasks more fine-grained versions of the de-novo data?\n- I’m a bit confused what’s the point of the “unified” probabilistic model notation. It seems that this is used nowhere in the paper after it’s introduction.\n- Why would you look at the MAE between two distributions? This doesn’t seem sensible. The distribution sizes and domains and types are mostly undefined [please use precise math]\n- What does it mean to measure “atom types” of functional groups? I don’t understand. I though that you compared just some distributions of 25 groups, which should give you a probability vector of length 25 to be compared. The at/rt/fg should have no role here. [It would help so much to define the stuff in precise math so that I wouldn’t have to guess.]\n- I fail to see where the accuracy of the de novo generation is described. I thought the task is to predict the correct ligand for a known protein (assuming there is only one correct ligand). Surely you want to measure how often you get this right. The paper also talks about probabilistic model of p(M|P). Where is this density evaluated: shouldn’t we see some logp values in the results?\n- Or is the task to just generate some ligands (irrespective if it was the correct one), as long as they have good binding and properties, or come from some distribution? If the task really is generative, then one expects to use learning target of maximizing the model likelihood of the observed data (both training and test folds). I don’t see this in the paper either. The learning problem needs to be precise.\n- Overall it seems that some metrics in the paper are about checking that the summary statistics of $p(ligand)$ or $p(ligand|protein)$ match the true distributions, and some are about making sure some $fitness(ligand,protein)$ is high. This is not formalised well, and it leads to the metrics and learning setting being vague and hand-wavy.  \n- I fail to see the motivation for the different Vina scores. If Vina is pathological, then why would we still use it? I don’t see how IMP or MPBG helps either if they are based on Vina. I don’t see convincing arguments why the LBE fixes the Vina pathology: the table 11 has values all of the place. I think here less would be more, and it would be much better to just have one good metric for binding than lots of binding metrics of varying quality. \n- I did not understand the PLIP stuff.\n- Overall the metrics are difficult to interpret. I don’t really know what the numbers mean. Is 0.4382 MAE good or bad? Is a 0.2345 JSD high or low? Is Vina -3.75 good or bad? Is IMP 22 good? I have no idea. It would be insightful to visualise the distributions, or use some human-understandable metrics. For instance, you could use distribution overlap percentage or something else.\n- It would have been useful to show some example generations, and their corresponding metrics.\n- Using 5M iterates in each method seems unfair. Different methods train in different ways. I think you need to analyse the discrepancy between your results and the published results to clarify this.\n- It seems that you change some methods architectures for no reason. This is not fair and will nullify the corresponding results. You can’t claim to benchmark method X if you change the method X from the publication.\n- It’s unclear if you reimplement the methods in your codebase, or just collect published codebases in one place. Can you clarify?\n- “We generate 100 molecules per pocket in test”. How do you do this? Why do you do this? Didn’t we have test data that tells us this? Now there are 10000 test molecules, while table 2 says that there are only 100. The learning setting is confusing, and using math to describe things would help. It seems that the entire setting in this paper is some kind of $D[p_gen || p_obs]$, which is not properly formalised.\n- “We show the interaction analysis and chemical property in the main context and omit the interaction pattern analysis since maintaining the patterns is not necessary for lead optimization.” I don’t understand this. How do you use “chemical property” as context? How do you use “analysis” as context? What is \"context\"?\n- I don’t understand what you do in the subtask training. Surely a method can’t be just applied in eg. linker generation if it wasn’t designed to do it? Are all 6 methods ones that support it, or do you somehow kludge this support to them? I’m really confused what do you train for extra 1M steps. What are even the loss functions? This is all super vague.\n- The results of subtasks are all over the place. I’m not sure what can you conclude from this.\n- Using ECFP to compare molecules in sec 5.3. is a very weak approach, and running it through tsne2d makes it even worse. This analysis has little to no value.\n- The setting in 5.3. is weak. Thera are only 100 random controls (why not way more, like a million?). It’s unclear how many molecules you sample, but this should be a large number.\n- Sec 5 concludes that the evaluation protocols are very consistent with real-world behavior. By far the best method in sec 5 is the DiffSBDD, which has by far best LBE, and ok Vina’s (Vina is pathological, so we should look at LBE anyways). But in Table 7 we see DiffSBDD being ranked 10/12, ie. one of the worst methods. It’s pretty clear that the real-world performance and de novo benchmark performance is not consistent.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper does a comprehensive benchmark, with lots of interesting results.\n- The authors contribute a unified codebase, which could be very useful for future research.", "weaknesses": "- The motivation of the work is weak. I’m not sure why we need all of this stuff, or what questions the paper answers. The paper seems to establish an extended benchmark protocol that adds few extra evaluations to standard benchmarks. This could be useful, but I found the new additions to be a bit hand-wavy. The paper also proposes “unified” tasks and notation, but why? These tasks and notation is already common.\n- The novel contributions of the work seem limited. The data and tasks seem to be from earlier works, and the benchmarks are added with just few extra evaluations, which also all seem to already exist. The main contribution is collecting all of this into one place. The performance comparison is a good contribution, but similar comparisons are done in every paper. The proposed new “unified” notation or task framing seems all well known. The main new contribution is probably the unified codebase, which can be very useful. The paper draws a bunch of conclusions from the results, which are interesting, but it’s unclear how novel they are.\n- As a benchmark paper the presentation of the results is limited: there are too many decimals, no standard deviations, and there are lots of massive tables of “raw” performance values, which are not digested into transferable insights. The analysis is superficial, and there are no ablations or attempts to explain how or why the different methods perform differently. \n- As a survey paper I didn't find this paper particularly good at summarising or organising the domain in terms of methods, data or metrics. Use of math is inconsistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lOOUxKwu1H", "reviewer": "Reviewer_ftZU", "summary": "The paper introduces a new benchmark for generative models in structure-based drug design. Recently, many machine learning models have been proposed to help solve the problem of designing compounds that can interact with their protein targets. However, the comparison of these methods is difficult due to the lack of standards and relevant benchmarks. CBGBench is a benchmark that aims at standardizing model comparison and proposing evaluation metrics that can be used to assess the performance of generative models. Additionally, the benchmark contains tasks that can be used to evaluate models' capabilities of fragment linking, scaffold hopping, fragment growing, and side chain generation. Several recent generative models were compared using the proposed benchmark, and conclusions were drawn from these experiments. Moreover, a case study was conducted to demonstrate the quality of compounds generated by these methods for two selected GPCR targets.", "questions": "1. What is the difference between fragment growing and side chain decoration on the implementation level? The text mentions ligand decomposition for the fragment growing task, but the detailed procedure of the decomposition is not explained. In Figure 3, how is the scaffold selected in the case of side chain decoration?\n2. Some of the employed evaluation metrics compare the generated distribution of molecules to a known distribution of molecules. Would this approach not penalize methods that generate molecules with novel scaffolds? Also in the t-SNE plot (Figure 4) that compares the set of generated molecules to the actives, the compounds generated by a model can be significantly different from the known actives but still active. A good example of this behavior would be genetic algorithms that optimize docking scores. These methods are not trained on any data, so they can easily learn to produce compounds that have a unique structure and dock well to the target protein.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\n- A new benchmark is proposed for comparing the performance of recently developed SBDD methods.\n- This benchmark also includes additional tasks like fragment linking or scaffold hopping, for which do not exist any standardized benchmarks.\n- A taxonomy of SBDD generative models is proposed.\n\nQuality:\n- The benchmark covers a wide range of generative models, including autoregressive and one-shot generative models.\n- Both evaluation methods and scoring network architectures are standardized.\n\nClarity:\n- The benchmark is clearly defined, and the diagrams in Figures 2 and 3 explain different tasks and model types.\n- The evaluation metrics are explained in an easy-to-follow way.\n\nSignificance:\n- The proposed benchmark should accelerate research on new generative models that generate molecules binding to a given protein.\n- The benchmarking code is attached to this submission, making it easier to test new models in the future.", "weaknesses": "Originality:\n- This work does not discuss other benchmarks for generative methods. In particular, Zheng et al. [1] have recently introduced an SBDD benchmark for generative models.\n\n[1] Zheng, Kangyu, et al. \"Structure-based Drug Design Benchmark: Do 3D Methods Really Dominate?.\" arXiv preprint arXiv:2406.03403 (2024).\n\nQuality:\n- The requirement that \"each connecting fragment must consist of more than five atoms\" seems arbitrary. Why are models that occasionally generate fewer atoms considered insufficient?\n- The benchmark could propose a method of computing confidence intervals, e.g. by sampling the generated molecules. It would facilitate the judgment of whether one of the tested models is significantly better than the others.\n\nClarity:\n- \"Real-world targets\" may be a confusing name for the structures used in the case study. The protein structures in the CrossDocked2020 dataset were also real-world targets, which might confuse some readers. Maybe it would be better to call the structures in the case study as selected targets.\n- Additionally, I recommend rephrasing statements like \"the established evaluation protocols exhibit strong consistency and generalizability on real-world target data.\" The real-world target data could be understood as binding data from laboratory experiments, but docking experiments usually do not correlate with such experimental data.\n- The text in Figures 4 and 5 may be too small.\n\nMinor comments:\n- A typo in line 68: \"Vina enery\"\n- A typo in line 475 \"the neurotrans-625 mitter\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)