CBGBench: Fill in the Blank of Protein-Molecule Complex Binding Graph

## Abstract

Abstract Structure-based drug design (SBDD) aims to generate potential drugs that can bind to a target protein and is greatly expedited by the aid of AI techniques in generative models. However, a lack of systematic understanding persists due to the diverse settings, complex implementation, difficult reproducibility, and task singularity. Firstly, the absence of standardization can lead to unfair comparisons and inconclusive insights. To address this dilemma, we propose CBGBench, a comprehensive benchmark for SBDD, that unifies the task as a generative graph completion, analogous to fill-in-the-blank of the 3D complex binding graph. By categorizing existing methods based on their attributes, CBGBench facilitates a modular and extensible framework that implements various cutting-edge methods. Secondly, a single de novo molecule generation task can hardly reflect their capabilities. To broaden the scope, we adapt these models to a range of tasks essential in drug design, considered sub-tasks within the graph fill-in-the-blank tasks. These tasks include the generative designation of de novo molecules, linkers, fragments, scaffolds, and sidechains, all conditioned on the structures of protein pockets. Our evaluations are conducted with fairness, encompassing comprehensive perspectives on interaction, chemical properties, geometry authenticity, and substructure validity. We further provide deep insights with analysis from empirical studies. Our results indicate that there is potential for further improvements on many tasks, with optimization in network architectures, and effective incorporation of chemical prior knowledge. Finally, to lower the barrier to entry and facilitate further developments in the field, we also provide a single codebase ( https://github.com/EDAPINENUT/CBGBench ) that unifies the discussed state-of-the-art models, data pre-processing, training, sampling, and evaluation.

## 1 Introduction

The rapid and remarkable progress in Graph Neural Networks (GNNs) and generative models have advanced the bio-molecule design in these years (Jumper et al., ; Watson et al., ; Krishna et al., ).
In structure-based drug design, AI-aided methods aim to learn the chemical space of the molecules that can bind to certain proteins as targets, which decreases the general chemical space of molecule compounds ($\sim 10^{60}$) to a more compact search space and enables the model to explore the potential binding drugs. Recent success in generative models such as diffusion models (Ho et al., ; Lipman et al., ; Song et al., ) further enhanced these methods to fully exploit the targeted chemical space, and AI-aided SBDD has been propelled into another prominence.

Despite the significance of the SBDD and the development of various approaches, there remains a lack of a comprehensive benchmark for this field covering various practical application scenarios. On the one hand, although different methods are proposed for the task, the experimental setup is not unified, and the evaluation protocol also differs. For example, GraphBP (Liu et al., ) use a different training and test split of Crossdocked2020 (Francoeur et al., ) from concurrent works like Pocket2Mol (Peng et al., ) and 3DSBDD (Luo et al., ); And DiffBP (Lin et al., ) and GraphBP employ Gnina (McNutt et al., ) instead of AutoDock Vina (Trott & Olson, ) as the evaluator for obtaining docking score as binding affinity. Besides, the complex implementation of models makes the modules coupled, so it is hard to figure out which proposed module contributes to the improvements in performance. For example, though TargetDiff (Guan et al., ) and DiffBP are both diffusion-based methods, the first one uses EGNN (Satorras et al., ) to predict the ground-truth position of each atom in the molecule while the later one uses GVP (Jing et al., ) to remove the added noise in a score-based way. To have a systematic understanding of the designation for diffusion models in molecule generation, the network architectures should be fixed to keep the expressivity equal. On the other hand, the task of de novo generation of molecules is a branch of SBDD. There are also other important tasks, such as the design of molecular side chains in lead optimization, or linker design to combine functional fragments into a single, connected molecule (Igashov et al., ; Guan et al., ). It is highly meaningful to explore whether these methods can be successfully transferred to applications in drug optimization.

In this way, here we propose CBGBench as a benchmark for SBDD, with the latest state-of-the-art methods included and integrated into a single codebase (See Appendix. C for details). Firstly, we unify the generation of the binding molecule as a graph completion task, i.e. fill-in-the-blank of the 3D Complex Binding Graph. In this way, the systematic categorization of existing methods is based on three dichotomies: (i) voxelized v.s. continuous position generation, (ii) one-shot v.s. auto-regressive generation, and (iii) domain-knowledge-based v.s. full-data-driven generation. As a result, the methods can be easily modularized and extensible in a unified framework. Secondly, CBGBench introduces a unified protocol with a comprehensive evaluation. In chemical property, the classical ones including QED, SA, LogP and LPSK are considered; For interaction, besides Vina enery, we extend the evaluation to interaction types between protein and molecules; For geometry, we considered static ones including bond length and bond angles, and clashes resulted from protein-molecule cross-clash between atoms in the generated molecules; In substructure analysis, we consider single atoms, rings, and functional groups as pharmacophores. Moreover, thanks to our reformulation of the problem and extensible modular implementation, we can easily extend these methods to the other four tasks in lead optimization, including the designation of (i) linkers, (ii) fragments, (iii) side chains and (iv) scaffolds. Comprehensive evaluation is also conducted for these tasks, to explore the potential application value of the existing methods in lead optimization.

As a result, several brief conclusions can be reached as the following:

- •
CNN-based methods by modeling voxelized density maps as molecule representations remain highly competitive in target-aware molecule generation, and the diffusion-based ones achieve the state-of-the-art.
- •
For autoregressive methods, it is essential to enable the model to successfully capture the patterns of chemical bonds.
- •
Prior knowledge has been incorporated into the model in recent works, but the improvements remain limited. Effective design of integrating physical and chemical domain knowledge remains a challenge, leaving substantial room for future research.
- •
Most evaluated methods can be well generalized to lead optimization. Empirical studies show that scaffold hopping is the most challenging task among them, while linker design is relatively the easiest. However, there is still a large space for improvements in these tasks.
- •
The conclusions drawn from the experimental results and with the evaluation protocol of CBGBench, are mainly consistent with those obtained on real-world targets, demonstrating generalizability and effectiveness of our benchmark.

## 2 Background

### 2.1 Problem Statement

For a binding system composed of a protein-molecule pair as $(\mathcal{P},\mathcal{M})$, in which $\mathcal{P}$ contains $N_{\mathrm{rec}}$ atoms of proteins and $\mathcal{M}$ contains $N_{\mathrm{lig}}$ atoms of molecules, we represent the index set of the protein’s atom as $\mathcal{I}_{\mathrm{rec}}$ and the molecule’s atoms as $\mathcal{I}_{\mathrm{lig}}$, with $|\mathcal{I}_{\mathrm{rec}}|=N_{\mathrm{rec}}$ and $|\mathcal{I}_{\mathrm{lig}}|=N_{\mathrm{lig}}$. The binding graph can be regarded as a heterogenous graph. One subgraph is protein structures, as $\mathcal{P}=(\bm{V}_{\mathrm{rec}},\bm{E}_{\mathrm{rec}})$, where $\bm{V}_{\mathrm{rec}}=\{(a_{i},\bm{\mathrm{x}}_{i},s_{i})\}_{i\in\mathcal{I}_{
\mathrm{rec}}}$ is the node set, and $\bm{E}_{\mathrm{rec}}=\{(i,j,e_{i,j})\}_{i,j\in\mathcal{I}_{\mathrm{rec}}}$ is the edge set. Here, $a_{i}$ is the atom types with $a_{i}=1,\ldots,M$, $\bm{\mathrm{x}}_{i}\in\mathbb{R}^{3}$ is the corresponding 3D position, and $s_{i}=1,\ldots,20$ is the amino acid types that $i$-th atom belongs to; elements in edge set means there exists

Figure: Figure 1: Filling the blank in the complex binding site.
Refer to caption: x1.png

a bond between $i$ and $j$, with edge type $e_{i,j}$. The other graph is the molecule structures, written as $\mathcal{M}=(\bm{V}_{\mathrm{lig}},\bm{E}_{\mathrm{lig}})$, where $\bm{V}_{\mathrm{lig}}=\{(a_{i},\bm{\mathrm{x}}_{i})\}_{i\in\mathcal{I}_{
\mathrm{lig}}}$ and $\bm{E}_{\mathrm{lig}}$ is the edge set with the same form as $\bm{E}_{\mathrm{rec}}$. Besides, there are virtual edges that exist between protein and molecule, which make up cross-edge set $\bm{E}_{\mathrm{crs}}$. Denote the probabilistic model for binding graph by $p(\mathcal{M},\mathcal{P})$. For the de novo molecule generation, the model aims to learn the probability of $p(\mathcal{M}|\mathcal{P})$, as to fill the blank of the protein pocket with atoms in a molecule, by using a generative model $p(\cdot)$.

A comparable analogy can be made that if we also divide the atoms in a molecule into known and to-be-generated parts, we can extend the aforementioned ‘filling-in-the-blank’ task as shown in Figure. 1. In specific, for the missing blanks, we write the node set for generation as $\mathcal{G}=(\bm{V}_{\mathrm{gen}},\bm{E}_{\mathrm{gen}})$, and the known atoms in the molecule as elements in the context set $\mathcal{C}=(\bm{V}_{\mathrm{ctx}},\bm{E}_{\mathrm{ctx}})$, in which $\mathcal{I}_{\mathrm{ctx}}=\mathcal{I}_{{\mathrm{lig}}}\setminus\mathcal{I}_{{
\mathrm{gen}}}$. Therefore, the generative model for filling the blanks of the missing parts of molecules aims to learn the probability of $p(\mathcal{G}|\mathcal{C},\mathcal{P})$. The newly-defined task of fill-in-the-partial-graph is of great significance in SBDD, especially in lead optimization, and we establish other four sub-tasks of it besides de novo design, with accessible datasets in Sec. 3.

### 2.2 Related Work and Taxonomy

The SBDD methods are initially combined with deep neural networks on voxel grids, such as LiGAN (Masuda et al., ) generating atom voxelized density maps using VAE (Kingma & Welling, ) and incorporating Convolutional Neural Networks (CNNs) as the architecture, and 3DSBDD (Luo et al., ) predicts whether the position on grids is occupied with which type of atoms in auto-regressive ways with Graph Neural Networks (GNNs). Then, the development of Equivariant Graph Neural Networks (EGNNs) boosts the SBDD methods to directly generate the continuous 3D positions, such as Pocket2Mol (Peng et al., ) generating the molecules’ atom types, positions, and the connected bonds and GraphBP (Liu et al., ) employing normalizing flows to generate these attributes, both in an auto-regressive way. The diffusion denoising probabilistic models (DDPM) (Ho et al., ) further propel the AI-aided SBDD methods, such as TargetDiff (Guan et al., ), DiffBP (Lin et al., ) and DiffSBDD (Schneuing et al., ), inspired by EDM (Hoogeboom et al., ), generating full atoms’ positions and element types, with different diffusion models. In recent years, domain knowledge has been used to constrain or guide the generation of binding drugs. For example, FLAG (Zhang et al., ) and D3FG (Lin et al., ) use prior knowledge of fragment motifs to model the coarse structures, which generate fragments in molecules in one-shot and auto-regressive ways, respectively; And DecompDiff (Guan et al., ) harnesses virtual points searching for arm- and scaffold-clustering as prior knowledge, and using multivariate Gaussian process to model atom positions in different clusters, with validity guidance for sampling. More recently, thanks to technological breakthroughs brought by generative AI, more related approaches have been proposed, which not only build upon previous modeling concepts but also incorporate new generative model techniques. MolCraft (Qu et al., ), as the SBDD version of GeoBFN (Song et al., ), employs Bayesian Flow Network (Graves et al., ) as the variant of diffusion models, to address the continuous-discrete gap in modeling the elements’ type and atoms’ positions by applying continuous noise and smooth transformation; VoxBind (Pinheiro et al., ), following their pioneering exploration of VoxMol (Pinheiro et al., ), continues modeling the 3D voxelization of molecules in a diffusion denosing way, with walk-jump sampling method used to generate molecules.

Figure: Table 1: Categorization of included methods. LiGAN, 3DSBDD and VoxBind model atom positions as discrete variables; One-shot methods maintain a constant atom number in a generation, like diffusion-based ones; FLAG and D3FG use fragment motifs, and DecompDiff uses arm&scaffolding priors.
Refer to caption: x2.png

In this way, we can categorize these methods with the three standards:

- •
Whether the positions of atoms are generated in continuous 3D space or voxelized grids.
- •
Whether the generation process is auto-regressive or one-shot.
- •
Whether the domain knowledge is introduced to integrate extra prior into the model.

To better articulate, we classify them according to Table. 1, and Figure. 2 gives a simple demonstration of the criteria for taxonomy. A detailed review of these methods is given in Appendix. A.

## 3 Task and Dataset

For the de novo molecule generation, we follow the previous protocol to use Crossdocked2020 (Francoeur et al., ) and data preparation with splits proposed in LiGAN (Masuda et al., ) and 3DSBDD (Luo et al., ) as the training and test sets.
Besides the de novo generation, the modularized methods can be extended to four subtasks which are branches of our defined fill-in-the-blank of the complex binding graph, including the target-aware linker, fragment, side chain, and scaffold design (Zhang et al., ), leading to the generative targets $\mathcal{G}$ for the probabilistic model $p(\mathcal{G}|\mathcal{C},\mathcal{P})$ to be the four components, and the molecular context $\mathcal{C}$ as the rest, as shown in Figure. 3. We demonstrate the significance of the four tasks in lead optimization and how the corresponding datasets are established below. The datasets are all established based on previous splits of Crossdocked2020 for a fair comparison.

Linker Design is a critical strategy in fragment-based drug discovery (Grenier et al., ). It focuses on creating linkers that connect two lead fragments and obtaining a complete molecule with enhanced affinity. Effective linker design can significantly influence the efficacy and pharmacokinetics of the drug, by ensuring that the molecule maintains the desired orientation and conformation when bound to its target (Erlanson et al., ). We define linkers following specific rules: (i) A linker should act as the intermediate structure connecting two fragments. (ii) A linker should contain at least two atoms on the shortest path between two fragments. (iii) Each connecting fragment must consist of more than five atoms.

Fragment Growing focus on expanding a fragment on the lead compound to better fill the binding pocket (Bancet et al., ). It also relates to adjusting pharmacological properties, such as enhancing solubility or reducing toxicity (Hung et al., ). We decompose the entire ligand into two fragments and select the smaller one for expansion based on specific criteria: (i) Each fragment must contain more than five atoms. (ii) The smaller fragment should be larger than half the size of the larger one.

Side Chain Decoration differs from fragment growing in that it permits modifications at multiple sites on the lead compound, whereas fragment growing typically modifies a single site. The taxonomy of arms-scaffold in previous work (Guan et al., ) is similar to side-chain-scaffold, while we focus on a chemist-intuitive approach, Bemis-Murko decomposition (Bemis & Murcko, ), which treats all terminal non-cyclic structures as side chains.

Scaffold Hopping is introduced by  Schneider et al. (), as a strategy in medicinal chemistry aiming to replace the core structure of molecules to explore more diverse chemical space or improve specific properties. While various empirical definitions exist (Sun et al., ), for consistency, the Bemis-Murko decomposition is utilized to define the scaffold structure.

Since chances are that there is no substructure in a molecule according to the discussed definition of the decomposition, we here give the instance number of training and test set for each task in Table. 2.

## 4 Evaluation Protocol

In previous works, the evaluation usually focuses on two aspects including interaction and chemical property. While lots of works start to focus another two standards on molecule 3D structure’s authenticity with geometry analysis and the 2D graphs reliability with subsutructure analysis, the evaluation protocol is usually not unified. In this way, we extend them in a unified protocol including the four aspects as the following:

Substructure.  To evaluate whether the model succeeds in learning the 2D graph structures for drug molecules. We expand previous metrics to functional groups, as certain functional groups in the drugs act as pharmacophores, such as phenol and pyrimidine. We identify the 25 most frequently occurring functional groups by using EFG (Salmina et al., ) (Appendix. B.1 gives the details), and calculated mean frequency of each function group generated in a molecule and the overall multinomial distribution. In the same way, the two values can also be obtained according to rings and atom types. The generated frequency and distribution against the references lead to 6 metrics on substructure analysis including the Jensen-Shannon divergence as JSD and mean absolute error as MAE, listed as (i) $\bm{\mathrm{JSD}_{\mathrm{at}}}(\downarrow)$, (ii) $\bm{\mathrm{MAE}_{\mathrm{at}}}(\downarrow)$, (iii) $\bm{\mathrm{JSD}_{\mathrm{rt}}}(\downarrow)$, (iv) $\bm{\mathrm{MAE}_{\mathrm{rt}}}(\downarrow)$, (v) $\bm{\mathrm{JSD}_{\mathrm{fg}}}(\downarrow)$, and (vi) $\bm{\mathrm{MAE}_{\mathrm{fg}}}(\downarrow)$, where ‘at’ means atom type, ‘rt’ means ring type and ‘fg’ means functional group.

Chemical Property.  We continue the evaluation of chemical properties from most previous works, including
(i) $\bm{\mathrm{QED}}$$(\uparrow)$ as quantitative estimation of drug-likeness; (ii) $\bm{\mathrm{SA}}$$(\uparrow)$ as synthetic accessibility score; (iii) $\bm{\mathrm{LogP}}$ represents the octanol-water partition coefficient, and in general LogP values should be between -0.4 and 5.6 to be good drug candidates (Ghose et al., ); (iv) $\bm{\mathrm{LPSK}}$$(\uparrow)$ as the ratio of the generated drug molecules satisfying the Lipinski’s rule of five.

Figure: Table 2: The instance number of training and test split in the datasets for the four tasks and De novo generation. The decomposition is conducted separately in the training and test set of CrossDocked2020 to avoid label leakage. Side chain decoration and scaffold hopping are two dual tasks, so the training and test set numbers are the same.
Refer to caption: x3.png

Interaction.  In evaluating the generated molecules’ interaction with proteins, the binding affinity is usually the most important metric. Here we unify the affinity calculation with AutoDock Vina for fair comparison. However, previous studies show that larger molecule sizes will lead to a higher probability that the generated molecules can interact with the protein, resulting in higher predicted binding affinity (Abad-Zapatero & Metz, ; Hopkins et al., ). It is also an observation in this paper as the Pearson correlation of Vina Energy v.s. atom number reaches $-0.67$ (See Appendix. B.2). Besides, in some structures, there are more atoms in the protein participating in the interaction, which also causes the average to be unreasonable. Therefore, besides the commonly-used (i) $\bm{E_{\mathrm{vina}}}(\downarrow)$ as the mean Vina Energy, and (ii) $\bm{\mathrm{IMP}\%}(\uparrow)$ as the improvements indicating the ratio of the generated molecule with lower binding energy than the reference, we use mean percent binding gap as (iii) $\bm{\mathrm{MPBG}}(\uparrow)$ (Appendix. B.3 gives computation) and ligand binding efficacy as (iv) $\bm{\mathrm{LBE}}(\uparrow)$, written as ${\mathrm{LBE}}_{i}=-\frac{{E_{\mathrm{vina}}}}{N_{\mathrm{lig}}}$, indicating how much does a single atom in the molecule contribute to the binding affinity, to eliminate the effects of molecule sizes. We give a detailed discussion on the reasonality of LBE in Appendix. B.4. Moreover, it is important for the generated molecules to keep the same or similar interaction patterns with proteins (Zhang et al., ). Therefore, we aim to figure out whether the SBDD models can learn the microscopic interaction patterns hidden in the data of the 3D conformations. We use PLIP (Adasme et al., ; li Zuo et al., ; Salentin et al., ) to characterize the 7 types of interactions and calculate per-pocket and overall JSD between the categorical distributions of generated types and reference types and MAE between the frequency of each generated types v.s. it of reference types, in which the frequency is defined as the average number of occurrences of each type of interaction produced by different single generated molecules binding to the same protein, leading to four metrics including (v) $\bm{\mathrm{JSD}_{\mathrm{PP}}}(\downarrow)$, (vi) $\bm{\mathrm{JSD}_{\mathrm{OA}}}(\downarrow)$, (vii) $\bm{\mathrm{MAE}_{\mathrm{PP}}}(\downarrow)$, (viii) $\bm{\mathrm{MAE}_{\mathrm{OA}}}(\downarrow)$, where ‘PP’ and ‘OA’ means per-pocket and overall respectively.

Geometry.  The internal geometry is an important characteristic for distinguishing between general point clouds and molecular structures. The torsion angles (Jing et al., ; Swanson et al., ) are flexible geometries, while bond lengths and bond angles are static to reveal whether the generated molecules have realistic structures. Hence, we evaluate the overall JSD of bond length and angle distribution between reference and generated ones, written as (i) $\bm{\mathrm{JSD}_{\mathrm{BL}}}(\downarrow)$ and (ii) $\bm{\mathrm{JSD}_{\mathrm{BA}}}(\downarrow)$. The other perspective for validating structural resonability is the clash, the occurrence of which is defined when the van der Waals radii overlap by $\geq 0.4$Å (Ramachandran et al., ). Hence, the ratio of number of atoms generating clashes with protein atoms to the total atom number is written as (iii) $\bm{\mathrm{Ratio}_{\mathrm{cca}}}(\downarrow)$ (cross clashes at atom level). Besides, the ratio of molecules with clashes as (iv) $\bm{\mathrm{Ratio}_{\mathrm{cm}}}(\downarrow)$ (molecule with clashes) are evaluated.

We here evaluate the 12 methods discussed in Table. 1, and use Friedman rank (Friedman, ; ; Wang et al., ) as the mean ranking method, to fairly compare the performance of different models in the four aspects. The ranking score is calculated by $(12-\mathrm{rank})$, and the final rank is according to the weighted mean ranking score. Appendix. B.3 gives detailed computation of the metrics, and their explanations as preliminary. The review of each method’s evaluation aspect in the published papers is given in Apppendix. B.5.

## 5 Benchmark Result

### 5.1 De novo Generation

#### 5.1.1 Setup

Training.  We use the default configuration in each model’s released codebase as the hyper-parameter, and set the training iteration number as 5,000,000 for fair comparison. It is noted that the loss of autoregressive methods exhibits a faster convergence rate of loss, typically requiring only a few tens of thousands of epochs to reach the final best checkpoint. To eliminate the effect brought about by the architecture of GNNs, in implementation, we use GVP (Jing et al., ) and EGNN (Satorras et al., ) with GAT (Veličković et al., ), as message-passing modules of auto-regressive and diffusion-based models, respectively. Especially, the GNN for encoding and decoding functional groups in D3FG is a combination of GAT and LoCS (Kofinas et al., ), CNN for processing atom density maps is four convolution blocks in LiGAN, and larger-scaled UNet for VoxBind. The details in hyperparameter tuning and training strategy are given in Appendix. D.

Evaluation.  Following the previous protocol, we generate 100 molecules per pocket in the test set, while not all the generated molecules are chemically valid (See Appendix. 23), which will lead to <10,000 molecules. In affinity evaluation, we employ three modes of AutoDock Vina, including ‘Score’, ‘Minimize’, and ‘Dock’. MPBG and LBE are added to the Vina Dock mode. For Vina energy that is larger than 0, we think it is invalid, so we will not report it, and give the lowest ranking in the column of metrics. Moreover, LogP only provides a reference range for drug molecules, so we assigned a rank of 1 to the generated molecules within the rank of 2 to those outside the range. In all the tables, values in bold are the best metric, and values with underline are the second and third.

#### 5.1.2 Result Analysis

Figure: Table 3: Results of substructure analysis.

Substructure.  Table. 3 gives results on 2D substructures, showing that (i) Overall, the auto-regressive models perform worse than the one-shot ones, especially in ring-type evaluation, where the former ones are more likely to generate triangular and tetrahedral rings. (ii) Diffusion-based models also exhibit advantages in atom type, according to the results of TargetDiff and MolCraft, except for DiffBP has difficulties in generating atoms with low occurrence frequencies. (iii) Besides VoxBind, MolCraft shows the overall superiority in complex functional group generation, as it states in addressing the mode collapse issues, while DecompDiff and D3FG also exhibit advantages due to the incorporation of prior knowledge in ring types and functional groups. DiffBP and DiffSBDD perform poorly due to inconsistencies in generating complex fragments. For a detailed comparison, see Appendix. E.1.1

Chemical Property.  The chemical property is calculated with 2D molecule graphs, so it can be greatly influenced by the molecule substructures. From Table. 4, we can conclude that (i) In terms of the four metrics, the differences among the compared methods are not significant. (ii) D3FG shows the best overall properties, with the competitive QED, SA, and LPSK.

Interaction. From Table. 5 shows that (i) LiGAN and VoxBind as CNN-based methods generate molecules that are initialized with high stability according to its competitive performance in Vina Score and Vina Min. It also performs well in Vina Dock mode, with a positive MPBG and high LBE, providing good consistency in interaction patterns. (ii) Auto-regressive methods except Pocket2Mol can hardly capture the pocket conditions and generate stably-binding molecules well, with very low IMP% in all Vina modes, while Pocket2Mol is the state-of-the-art auto-regressive method in de novo generation. (iii) In diffusion-based methods, MolCraft outperform the other competitors in overall interaction comparison, and TargetDiff and DecompDiff perform competitively since the difference between these two methods is minimal. DiffBP’s performance in docking mode is comparable, but in other modes, it performs less satisfactorily. Performance of DiffSBDD is less than satisfactory. D3FG generates molecules with comparable Vina Energy but small atom numbers, leading to high LBE. For details, see Appendix. E.1.2.

Geometry.  From Table. 6, conclusions can be drawn that (i) MolCraft generates overall the most realistic structures, according to the internal geometries, and DecompDiff is comparable. (ii) The molecule-protein clashes can usually be avoided by diffusion-based models. However, in the auto-regressive models, only Pocket2Mol avoids clashes. The high frequency of clashes in auto-regressive methods can be explained as follows: These models first identify frontier amino acids or atoms within the molecule. During the placing of new atoms, incorrect binding site localization leads to an erroneous direction for the molecule’s auto-regressively growing path, causing it to extend inward towards the protein. For detailed results, see Appendix. E.1.3.

##### Conclusion and Discussion.

From Table. 7 and the previous discussion, we conclude that

- (i)
The CNN-based methods like LiGAN and VoxBind are highly competitive, especially in aspects of Interaction, which explains why such methods in the field of drug design and molecule generation still prevail in recent years. This is partly attributed to the fact that CNNs have an advantage over GNNs in perceiving many-body patterns within a single filter (Townshend et al., ). As a result, it encourages further research into GNNs for 3D point clouds to develop architectures that can match the expressivity of CNNs.
- (ii)
MolCraft achieves the best overall performance, with TargetDiff closely following. In contrast, DecompDiff and D3FG as the variants of TargetDiff that incorporate domain knowledge show some degeneration in performance. This observation reveals that the current incorporation of physicochemical priors can hardly improve the quality of generated molecules. Effectively integrating domain knowledge to guide the model to generate structurally sound molecules remains a challenge. For example, atom clashes are very common in generated molecules. While DecompDiff employs the prior guidance for this, the problem has still not been fully solved. Integrating domain knowledge into the training process may mitigate this issue (Huang et al., ; Adams & Coley, ).
- (iii)
Only Pocket2Mol as an auto-regressive method achieves competitive results, which we attribute to the following reasons: First, it utilizes chemical bonds to constrain atoms to grow orderly along chemical bonds rather than to grow based on distance to the pocket as in DiffBP. Second, it simultaneously predicts the bond types and employs contrastive learning by sampling positive and negative instances of atom positions as real and fake bonds, which is not fully considered by FLAG, enhancing the model’s ability to perceive chemical bond patterns. Therefore, we believe that enabling the autoregressive methods model to successfully capture the patterns of chemical bonds is very essential.

### 5.2 Extension on Subtasks

Setup.  Lead optimization is to strengthen the function or property of the binding molecules by remodeling the existing partial 3D graph of molecules. We show the interaction analysis and chemical property in the main context and omit the interaction pattern analysis since maintaining the patterns is not necessary for lead optimization. The domain-knowledge-based methods can hardly be transferred for these tasks since different tasks require different priors, so we have not compared them here. Besides, the voxelize-based methods are not easily extended to these tasks, and we regard the transferring of methods like LiGAN and 3DSBDD as future work. Hence, we here compare 6 methods that model atoms’ continuous positions with GNNs and have not employed domain knowledge. When training, since the number of training instances is smaller, we use the pretrained models on de novo generation and finetune the auto-regressive models with 1,000,000 iterations. For diffusion-based models with one-shot generation, we train them from scratch because the zero-center-of-mass (Satorras et al., ) technique is shifted from employing protein geometric centers to using molecule context’s ones. For detailed results, please refer to Appendix. E.2.

Figure: Table 6: Results of geometry analysis.

Conclusion and Discussion.  From Table. 8 and the previous discussion, we conclude that

- (i)
Overall, the performance gap among these methods in the lead optimization is not as pronounced compared to de novo generation. MolCraft, Target and Pocket2Mol maintain the good performance. Notably, in the evaluation of the stability of initial poses, the other three also complete the binding graph near the given partial molecular conformations well, according to the columns of Vina Score and Vina Min. The applicability of GraphBP also reflects the Argument. (ii) in our de novo geometry analysis: The failure of GraphBP mainly stems from the difficulty in locating the correct atoms for auto-regressive growth.
- (ii)
In these tasks, scaffold hopping is the most challenging task, as the improvement of generated molecules relative to the reference is minimal; Linker design is relatively the easiest. Notably, in scaffolding, there are still some methods that fail.
- (iii)
There is a large space for improvements in these tasks, since the $\mathrm{MPBG}\%$ metrics are usually negative, indicating that in most cases, the molecule is not optimized. Some edge-cutting techniques such as DPO (Rafailov et al., ; Cheng et al., ; Gu et al., ) and ITA (Yang et al., ; Kong et al., ; Lin et al., ) may be used to augment the optimized molecules as supervision signals for the models.

Additionally, several points warrant further detailed design. For instance, in linker design, the fragments of the molecule that need to be connected may change orientation due to variations in the linker (Guan et al., ); In side chain decoration, the protein’s side chains, which are the main components in interaction, should be generated together with the molecule’s side chains to achieve a stable conformation as the entire complex (Luo et al., ; Huang et al., ).

### 5.3 Case study on Real-World Targets

Introduction.  In order to verify whether the included methods can generalize to pharmaceutic targets and the applicability of CBGBench to real-world scenarios, we use the pretrained model in de novo generation, and apply them to two proteins belonging to the G-Protein-Coupled Receptor (GPCR) family: ARDB1 (beta-1 adrenergic receptor) and DRD3 (dopamine receptor D3). ARDB1 participates in the regulation of various physiological processes by responding to the neurotransmitter epinephrine (adrenaline) and norepinephrine, and drugs that selectively activate or block this receptor are used in the treatment of various cardiovascular conditions. For DRD3, it is primarily expressed in the brain, particularly in areas such as the limbic system and the ventral striatum with functions of mediating the effects of the neurotrans-625
mitter dopamine in the central nervous system.

Setup.  On these targets, there are molecules reported active to them experimentally, we here randomly select 200 of them for each target and conduct two kinds of experiments. Firstly, we try to figure out if the model can generate binding molecules that have similar chemical distributions with the actives. We use extended connectivity fingerprint (ECFP) (Rogers & Hahn, ) to get the molecule fingerprint and employ t-SNE (van der Maaten & Hinton, ) for 2-dimensional visualization. Secondly, we aim to find out the distribution of Vina Docking Energy and LBE of the generated and the actives as the metrics for binding affinities. Eight methods except for voxelized-based ones (inflexible to extend) and DecompDiff (requiring complex data-preprocessing for domain knowledge) are tested on the two targets. Besides, we select 100 molecules randomly from GEOM-DRUG (Axelrod & Gómez-Bombarelli, ), as a randomized control sample set.

Figure: Table 8: Results of subtasks for lead optimization.

Figure: Figure 4: T-SNE visualization of chemical distributions of generated and active molecules on ADRB1.

Conculusion and Discussion.  Figure. 4 and Figure. 5 gives the results on target ADRB1.

- (i)
In Figure. 4, we can see that Pocket2Mol, TargetDiff, D3FG, and MolCraft have better consistency in the chemical distribution of molecules, as evidenced by a greater degree of overlap with the actives. In addition, in comparison to randomly selected molecules in GEOM-DRUG, these models show different preferences in generating binding molecules since the clustering center in the chemical space differs.
- (ii)
Figure. 5 shows that in generating molecules based on the real-world target ADRB1, D3FG exhibits superior performance. TargetDiff and MolCraft perform comparably.

These conclusions are essentially consistent with the conclusion in Sec. 5.1, reflecting that the established evaluation protocols exhibit strong consistency and generalizability on real-world target data.
Besides, it is worth noting that DiffBP, GraphBP, and Pocket2Mol can possibly generate molecules with small atom numbers and high LBE. This indicates that they have the potential to excel in lead discovery on ADRB1, as a good lead should possess good synthesizability and modifiability, smaller molecular weight, and stable binding conformations. For DRD3, please refer to Appendix. E.3 for details.

## 6 Conclusion and Limitation

In this paper, we propose CBGBench. It first unifies the tasks of SBDD and lead optimization into a 3D-graph completion problem as a fill-in-the-blank 3D binding graph and introduces four additional lead optimization tasks. Then, it comprehensively categorizes existing methods based on certain criteria, modularizes these methods based on the categorization, and integrates them into a unified codebase for fair comparison. Additionally, it extends existing evaluation protocols by incorporating aspects such as interaction pattern, ligand binding efficacy, and protein-atom clash ratio as important evaluation metrics, addressing the issue of incomplete and diverse evaluation process. Finally, through extensive experiments and by extending the pretrained models to real targets for molecule generation, we derive a series of insightful conclusions and identify future research directions.

However, there are certain limitations. Firstly, this codebase is primarily based on GNNs, so some voxelized-grid-based methods that require the use of CNNs have not been included. Engineering the integration of these types of methods will be a focus of our future work. Secondly, due to the inability to use wet lab experiments for validation, many of the metrics are obtained through computational chemistry methods. These computational functions are often considered unable to accurately reflect the chemical properties of molecules, such as the calculated SA (Synthetic Accessibility) and AutoDock Vina Energy. Therefore, how to utilize AI to assist in accurate metric calculation will also be a key focus of our future research.

## Appendix A Review on the Methods’ Timeline

Here we give a brief review of the 12 included methods in chronological order, with Table. LABEL:tab:appreview giving some standardized information on these methods.

- •
LiGan is firstly proposed to incorporate a generative model for the SBDD task, in which the encoder and the decoder are 3-layer CNNs, and the Variational Auto-Encoder is employed to model the probability of the atom’s density map in a molecule. It split the training and test set in CrossDocked2020.
- •
Due to the rise of Graph Neural Networks (GNNs), 3DSBDD subsequently adopted GNNs for modeling this task. At that time, GNNs were more adept at handling pairwise information, so an autoregressive graph network modeling approach was chosen as an effective method for molecular modeling. However, the development of graph networks for 3D tasks remained slow, leading 3DSBDD to continue modeling molecules using voxelized maps. 3DSBDD leverages graph networks to predict the voxel position that the next atom will occupy, as well as the atom type, thereby generating molecules in an autoregressive manner. In addition, 3DSBDD further standardized the evaluation protocol of the experiment by using Vina Docking Energy to assess the stability of the generated molecules.
- •
The rise of EGNN has enabled autoregressive modeling of continuous atomic coordinates in the SBDD task while preserving equivariance and invariance properties. Following this development, GraphBP adopted the EGNN architecture, using a depth-first search approach to rank the molecules from near to far on the protein surface. This allows for autoregressive molecular generation starting from the protein frontier and ultimately modeling a complete molecule. However, GraphBP did not consider critical chemical priors such as bond lengths and bond angles between atoms, making it challenging to capture the internal patterns of molecules. The evaluation is based on Gnina Docking, different from 3DSBDD.
- •
Meanwhile, Pocket2Mol directly modeled atomic coordinates using GVP. Compared to GraphBP, Pocket2Mol took molecular bonds into account and employed contrastive learning, where false atoms were used to regularize the coordinates and bonding of real atoms. This significantly improved the stability of the generated molecules. The evaluation follows 3DSBDD.
- •
The rise of generative models, particularly the widespread application of diffusion models in the field of images, has drawn attention to these cutting-edge generative techniques in the molecular domain as well. Diffusion models are inspired by particle systems in statistical physics, which aligns closely with the task of generating atomic coordinates in molecules. DiffSBDD, as pioneering works in applying diffusion models to the SBDD task, have also achieved significant results. DiffSBDD followed most of the concepts from EDM, such as the zero-center-of-mass technique, and was tested and generalized on Binding MOAD beyond CrossDocked2020.
- •
Meanwhile, DiffBP also proposed to use the Gaussian Diffusion model on atoms’ positions and masked type diffusion for elements’ types. DiffBP analyzed the drawbacks of autoregressive models from the perspective of the Boltzmann energy distribution in statistical physics and observed that molecular weight severely affects docking performance, being the first to adopt a grouping approach for model comparison.
- •
After that, following previous explorations like GeoDiff and EDM in molecular (conformation) generation, TargetDiff innovatively introduced a diffusion model-based SBDD method, achieving new state-of-the-art (SOTA) performance. For validation, TargetDiff first proposed using Vina Score and Vina Min mode to assess the stability of the generated molecules’ initial states. Additionally, the embeddings learned by the model can also be used for affinity prediction.
- •
On the other hand, fragment-based generative methods are also in the early stages of exploration. FLAG follows the autoregressive generation approach of Pocket2Mol, breaking molecules into fragments and individual atoms, allowing for the generation of complex molecules with fewer steps. However, its modeling approach lacks sufficient consideration of chemical bond participation, which is a key reason for its suboptimal molecular generation performance.
- •
Afterward, D3FG combined fragment-based generation with the diffusion method, leveraging protein generation techniques to establish a diffusion model that predicts the orientation matrix, central coordinates, and fragment types based on fragment information. The model then uses linkers to connect the molecule fragments. Additionally, D3FG enables molecular optimization by replacing fragments that are either prominent or insufficiently contributing.
- •
In contrast to D3FG, which treats fragments as rigid bodies, DecompDiff considers fragments as flexible structures, thereby decomposing molecules into arms and scaffolds, and performing multivariate Gaussian diffusion within different decomposition clusters. Additionally, DecompDiff employs non-intersection guidance to mitigate the issue of atomic overlap between the molecule and protein amino acids.
- •
In addition, work on voxelized maps is steadily progressing. VoxBind builds on the pioneering work of VoxMol, using diffusion-based denoising modeling on the voxelized density map of molecules. During the generation steps, it incorporates wj-sampling to reduce computational complexity, while also producing structurally stable molecules.
- •
More recently, MolCraft was proposed, utilizing the Bayes Flow Network, which is designed for modeling probabilistic densities’ parameters. This allows molecules to avoid the difficulty of coupling the two modalities: the discrete variable of atomic types and the continuous positions of atoms in the molecule. It effectively addresses issues such as mode collapse and has demonstrated outstanding performance as the most recent SOTA model.

**Table 9: A brief review of the included methods.**
| Method | Time | Generative Model | Network Structure | Prior Knowledge | Evaluation Datasets |
| --- | --- | --- | --- | --- | --- |
| LiGan | Oct. 2020 | VAE | CNN | None | CrossDocked2020 |
| 3DSBDD | Mar. 2022 | Auto-regressive | GNN | None | CrossDocked2020 |
| GraphBP | Apr. 2022 | Auto-regressive | EGNN | None | CrossDocked2020 |
| Pocket2Mol | May. 2022 | Auto-regressive | GVP | None | CrossDocked2020 |
| DiffSBDD | Oct. 2022 | DDPM | EGNN | None | CrossDocked2020<br>Binding MOAD |
| DiffBP | Nov. 2022 | DDPM | GVP | None | CrossDocked2020 |
| TargetDiff | Feb. 2023 | DDPM | EGNN | None | CrossDocked2020 |
| FLAG | Feb. 2023 | Auto-regressive | GVP | Fragment | CrossDocked2020 |
| D3FG | May. 2023 | DDPM | EGNN + IPA | Fragment | CrossDocked2020 |
| DecompDiff | Feb. 2024 | DDPM | EGNN | Arm&Scaffold | CrossDocked2020 |
| MolCraft | Apr. 2024 | BFN | EGNN | None | CrossDocked2020 |
| VoxBind | May. 2024 | DDPM | CNN | None | CrossDocked2020 |

## Appendix B Supplementary Evaluation Details

### B.1 Included Functional Groups

Here we show the functional groups included in this paper, in which we follow D3FG and give a demonstration of them in Table. 10.

**Table 10: The included functional groups in CBGBench. ‘T’ is the occurrence times of the functional group in the datasets (100,000 ligands). ‘A,B,C’ are the framing node index.**
| Smiles | 2D graph | 3D structures | A | B | C | T |
| --- | --- | --- | --- | --- | --- | --- |
| c1ccccc1 |  |  | 1 | 0 | 2 | 131148 |
| NC=O |  |  | 1 | 0 | 2 | 49023 |
| O=CO |  |  | 1 | 0 | 2 | 39863 |
| c1ccncc1 |  |  | 3 | 2 | 4 | 15115 |
| c1ncc2nc[nH]c2n1 |  |  | 7 | 3 | 6 | 11369 |
| NS(=O)=O |  |  | 1 | 0 | 2 | 10121 |
| O=P(O)(O)O |  |  | 1 | 0 | 2 | 7451 |
| OCO |  |  | 1 | 0 | 2 | 6405 |
| c1cncnc1 |  |  | 3 | 2 | 4 | 5965 |
| c1cn[nH]c1 |  |  | 2 | 3 | 1 | 5404 |

| Smiles | 2D graph | 3D structures | A | B | C | T |
| --- | --- | --- | --- | --- | --- | --- |
| O=P(O)O |  |  | 0 | 1 | center(2,3) | 5271 |
| c1ccc2ccccc2c1 |  |  | 3 | 2 | 4 | 4742 |
| c1ccsc1 |  |  | 3 | 2 | 4 | 4334 |
| N=CN |  |  | 1 | 0 | 2 | 4315 |
| NC(N)=O |  |  | 2 | 1 | 3 | 4167 |
| O=c1cc[nH]c(=O)[nH]1 |  |  | 7 | 1 | 5 | 4145 |
| c1ccc2ncccc2c1 |  |  | 3 | 2 | 4 | 3519 |
| c1cscn1 |  |  | 2 | 3 | 1 | 3466 |
| c1ccc2[nH]cnc2c1 |  |  | 5 | 4 | 6 | 3462 |
| c1c[nH]cn1 |  |  | 3 | 2 | 4 | 2964 |
| O=[N+][O-] |  |  | 1 | 0 | 2 | 2702 |
| O=CNO |  |  | 1 | 0 | 2 | 2477 |
| NC(=O)O |  |  | 1 | 0 | 2 | 2438 |
| O=S=O |  |  | 1 | 0 | 2 | 2375 |
| c1ccc2[nH]ccc2c1 |  |  | 3 | 4 | 2 | 2301 |

### B.2 LBE Analysis

Figure: Figure 6: A demonstration of how the molecule size affects the binding energy. The 6 pockets that are most linear-correlated with atom number.
Refer to caption: x57.png

### B.3 Metric Calculation

The metric of MPBG can be written as $\mathrm{MPBG}_{j}=\texttt{Mean}_{i}(\frac{E_{i,\mathrm{gen}}-E_{\mathrm{ref}}}
{E_{\mathrm{ref}}}\times 100\%)$, in which $i$ is the indicator of the generated molecules in a single protein and $\texttt{Mean}_{i}(\cdot)$ calculates average along indicator $i$, with $\bm{\mathrm{MPBG}}=\texttt{Mean}_{j}(\mathrm{MPBG}_{j})$. The MPBG is a per-pocket metric that is calculated within a single pocket, which differs from other averaging metrics.

For chemical property calculation, we use functions in RDKit (), and note that the SA is the normalized one employed in the previous studies.

For interaction pattern analysis, we employ PLIP, which considers 7 kinds of interaction types including ‘hydrophobic interactions’, ‘hydrogen bonds’, ‘water bridges’, ‘$\pi$-stacks’, ‘$\pi$-cation interactions’, ‘halogen bonds’ and ‘metal complexes’.

For MAE metrics that are related to the molecule sizes, the generative model tends to achieve higher scores when the atom number is close to that of the reference. We suppose it is reasonable, considering that the size of the molecule itself is determined by factors such as the size and shape of the pocket, and these characteristics should be captured by the models.

### B.4 Discussion on LBE.

- Is it possible that smaller molecules exhibit higher efficacy, thus leading LBE to be meaningless? (Reynolds et al., )

Meaning.  The introduction of Ligand Binding Efficiency (LBE) is motivated by observations in Appendix B.2, where we identified a strong positive correlation between the size of the generated molecules and the Vina docking energy. To address this size-related bias, LBE was incorporated as an additional metric. This approach is consistent with the basic principles of energy calculation, where the total binding energy is derived from the sum of interaction energies contributed by each atom, i.e. $E_{\mathrm{vina}}\sim E_{\mathrm{bind}}=\sum_{i=1}^{N_{\mathrm{atom}}}E_{i}$. Thus, using LBE as a measure of the average per-atom contribution to binding affinity is both logical and appropriate, as $\mathrm{LBE}=-\frac{E_{\mathrm{bind}}}{N_{\mathrm{atom}}}$. Furthermore, numerous studies in medicinal chemistry have highlighted Ligand Efficiency as a critical evaluation criterion, advocating for its widespread use as an effective metric for assessing molecular binding affinity (Hopkins et al., ; Kenny, ; Murray et al., ).

**Table 11: Ratio of generated molecule’s size and the corresponding LBE.**
| Method | Mol Size | [1,3] | (3,10] | (10,20] | (20,30] | >30 |
| --- | --- | --- | --- | --- | --- | --- |
| LiGAN | Ratio(%) | 0.01 | 13.61 | 40.09 | 36.77 | 9.59 |
| LBE | 0.37 | 0.42 | 0.44 | 0.35 | 0.30 |  |
| 3DSBDD | Ratio(%) | 0.01 | 15.48 | 54.44 | 22.63 | 7.42 |
| LBE | 0.39 | 0.42 | 0.41 | 0.33 | 0.31 |  |
| GraphBP | Ratio(%) | 0.00 | 0.77 | 91.29 | 6.07 | 1.84 |
| LBE | - | 0.36 | 0.33 | 0.21 | 0.20 |  |
| Pocket2Mol | Ratio(%) | 0.08 | 24.22 | 48.83 | 20.19 | 6.64 |
| LBE | 0.43 | 0.45 | 0.44 | 0.33 | 0.30 |  |
| TargetDiff | Ratio(%) | 0.00 | 10.24 | 28.17 | 39.51 | 22.05 |
| LBE | - | 0.44 | 0.44 | 0.31 | 0.29 |  |
| DiffSBDD | Ratio(%) | 0.13 | 21.54 | 41.43 | 28.01 | 8.86 |
| LBE | 0.37 | 0.38 | 0.30 | 0.24 | 0.22 |  |
| DiffBP | Ratio(%) | 0.00 | 7.29 | 32.19 | 43.08 | 17.42 |
| LBE | - | 0.41 | 0.42 | 0.31 | 0.29 |  |
| FLAG | Ratio(%) | 0.04 | 39.01 | 54.29 | 6.52 | 0.13 |
| LBE | 0.40 | 0.35 | 0.33 | 0.24 | 0.19 |  |
| D3FG | Ratio(%) | 0.00 | 8.31 | 61.70 | 27.72 | 2.55 |
| LBE | - | 0.44 | 0.45 | 0.32 | 0.25 |  |
| DecompDiff | Ratio(%) | 0.00 | 9.99 | 35.50 | 33.98 | 20.51 |
| LBE | - | 0.38 | 0.38 | 0.33 | 0.30 |  |
| MolCraft | Ratio(%) | 0.00 | 9.18 | 30.97 | 39.62 | 20.21 |
| LBE | - | 0.44 | 0.41 | 0.33 | 0.29 |  |
| VoxBind | Ratio(%) | 0.00 | 6.10 | 36.64 | 43.39 | 15.85 |
| LBE | - | 0.43 | 0.41 | 0.34 | 0.26 |  |

Resonability. It can be usually concluded from Reynolds et al. () that ‘smaller molecules tend to have higher LBE’, while we believe this conclusion is incorrect. We claim that different protein pockets exhibit a preference for molecule sizes. If this conclusion were true, it would imply that molecules consisting of a single atom would have the optimal LBE? To explore this issue, in Table. 11, we give statistics of molecule sizes v.s. LBE of methods evaluated. It shows that such molecules with extreme sizes (less than 4) that make the LBE meaningless are scarcely generated since they have a small ratio. Ligands with sizes less than 20 usually have better LBE because of the preference of the pockets, as a consequence of protein binding sites being limited in their size, thus generating the preference of binding molecules’ sizes. Larger ligand with sizes of more than 20 gets decreased LBE because it becomes increasingly difficult to form optimal interactions with every site on the protein without introducing unfavorable ligand strain.

Little Effects of Extreme Samples.  Besides, we use a weighted ranking with equal weights as previous benchmarks (Wang et al., ), to give an unbiased evaluation. Given that LBE accounts for one-twelfth of the total weight in interaction analysis, the impact of failed samples in LBE on calculating the LBE metrics and final ranking is minimal. Therefore, we claim that the inclusion of a minimal number of invalid LBE calculations will not render the overall ranking ineffective.

**Table 12: Different methods’ evaluation aspects in the published papers.**
| Method | Substructure | Geometry | Chem Property | Interaction |
| --- | --- | --- | --- | --- |
| LiGAN | Figure. S6 | Figure. S7, S8, S9 | Figure. 3, S3, S4, S5 | Figure. S13, S14 |
| Pocket2Mol | Table. 2 | Table. 3; Figure. 4 | Table. 1 | Table. 1 |
| GraphBP | - | Table. 2; Figure. 5 | - | Table. 1; Figure. 2 |
| TargetDiff | Table. 2 | Table. 1; Figure. 2 | Table. 3 | Table. 3, Figure. 4 |
| DiffBP | Table. 3 | - | Table. 2 | Table. 1 |
| DiffSBDD | - | Figure. 8, 9 | Table. 1 | Table. 1, 2; Figure. 2 |
| FLAG | Table. 3 | Table. 2; Figure. 4 | Table. 1 | Table. 1 |
| D3FG | Table. 1, 3; Figure. 3 | Table. 2 | Table. 4 | Table. 4 |
| DecompDiff | - | Table. 1, 2; Figure. 3 | Table. 3 | Table. 3 |
| MolCraft | Table. 1 | Table. 2 | Table. 2 | Table. 2 |
| VoxBind | - | Figure. 7 | Table. 1 | Table. 1; Figure. 5 |

### B.5 Previously-Used Metrics for Evaluation

The aspects of evaluation are commonly used with different methods, while they hardly give a very comprehensive evaluation. We give a brief review of each method’s evaluation on these aspects in Table. 12.

## Appendix C Codebase Structure

In this section, we provide an overview of the codebase structure of CBGBench, where four abstract layers
are adopted. The layers include the core layer, algorithm layer, chemistry layer, and API layer in the
bottom-up direction as shown in Fig. 7. This codebase is licensed under the Apache License, Version 2.0. It provides a robust and flexible framework for building and evaluating graph neural network models for structure-based drug design and lead optimization.

Figure: Figure 7: Structure of CBGBench Codebase, consisting of 4 layers. The core layer provides the common functions, datasets, and modules for CBG methods. The algorithm layer mainly implements the prevailing CBG algorithms. The Chemistry layer is used for data preprocessing and evaluation.
Refer to caption: x58.png

Core Layer. In the core layer, we implement the commonly used core functions for training and sampling CBG
algorithms. Besides, the code regarding datasets, data loaders, and basic modules used in CBGBench is also
provided in the core layer.

Algorithm Layer.  In the algorithm layer, we first implement the base class for CBG algorithms, where
we initialize the datasets, data loaders, and basic modules from the core layer. We modularize each method, first using embedding layers to project the input into high-dimensional space, and then employing different 3D equivariant GNNs to construct the algorithm, including Diffusion-based and Auto-regressive-based ones. We further abstract the algorithms, enabling
better code reuse and making it easier to implement new algorithms. The voxel-based methods are not included because our framework is mostly based on EGNNs, which will be added to the codebase as future work, and DecompDiff requires different data preprocessing. Based on this, we support 7 core CBG algorithms. More algorithms are expected to be added through continued extension.

Chemistry Layer. The chemistry layer is mostly built with post-processing utilities and evaluators such as atom type, bond angle, and docking modules. Besides, several prior decomposition functions are used as molecule parsers, and the molecule and protein fragment motifs serve as prior knowledge.

API Layer.  We wrap the core functions and algorithms in CBGBench in the API layer as a public Python
package. It is friendly for users from different backgrounds who want to
employ CGB algorithms in new applications. Training and sampling can be done in only a few lines
of code. In addition, we provide the configuration files of all algorithms supported
in CBGBench with detailed parameter settings, allowing the reproduction of the results.

## Appendix D Experimental Implementation Details

Experiments are conducted based on Pytorch 2.0.1 on a hardware platform with Intel(R) Xeon(R) Gold 6240R @ 2.40GHz CPU and NVIDIA A100 GPU.
We give the detailed parameters for training the included models in Table. 13. Note that most of the hyperparameters’ combinations are directly the officially-provided ones. Specifically, The methods included in our codebase are either self-trained or trained by us when the official repository does not provide a pretrained checkpoint. For methods that are not included, we validate them using the pretrained models that have been provided. In the Lead-Optimization tasks, the finetuning parameters are the same.

**Table 13: The hyper-parameters in for training and sampling molecules for the SBDD methods. Different from GNN-based methods, CNN-based methods usually have a different hidden dimension size, so we report the highest ones. ‘Num steps’ are the sampling steps which is usually used in diffusion-based methods, including DDPMs and BFNs. For other hyper-parameters in detail, please read the config files provided in the supplementary materials from ./configs/denovo/train.**
| Method | Batch Size | Hidden Dim | Layer num | Learning rate | Optimizer | Num Steps |
| --- | --- | --- | --- | --- | --- | --- |
| LiGan | 8 | 128 | 4 | 1E-05 | RMSprop | / |
| 3DSBDD | 4 | 256 | 6 | 1E-04 | Adam | / |
| GraphBP | 16 | 64 | 6 | 1E-04 | Adam | / |
| Pocket2Mol | 8 | 256 | 6 | 1E-05 | Adam | / |
| TargetDiff | 4 | 128 | 9 | 5E-04 | Adam | 1000 |
| DiffSBDD | 16 | 256 | 6 | 1E-03 | Adam | 500 |
| DiffBP | 16 | 256 | 6 | 1E-04 | Adam | 1000 |
| FLAG | 4 | 256 | 6 | 1E-04 | Adam | / |
| D3FG | 16 | 128 | 6 | 1E-04 | Adam | 1000 |
| DecompDiff | 4 | 128 | 6 | 5E-04 | Adam | 1000 |
| MolCraft | 8 | 128 | 9 | 5E-04 | Adam | 1000 |
| VoxBind | 32 | 512 | 4 | 1E-05 | AdamW | 100 |

## Appendix E Experimental Result Details

### E.1 De novo Generation

#### E.1.1 Substructure

Table. 16 gives the generated molecules’ atom distribution in detail. It shows DiffBP and GraphBP tend to generate more C atoms and have a low probability of generating uncommon atoms. In contrast, D3FG and FLAG, which directly use a motif library, have a higher probability of generating these uncommon atoms. Table. 16 gives the generated molecules’ ring distribution in detail. DiffBP and DiffSBDD perform poorly due to significant inconsistencies in generating large complex fragments. For example, they also generate a large number of unreasonable triangular and tetrahedral rings. Similarly, these methods also fall short in generating complex functional groups, as shown in Table. 16. This leads to their subpar performance in substructure generation.

**Table 14: Distribution of different atom types across different methods.**
| Method | C | N | O | F | P | S | Cl |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Ref. | 0.6715 | 0.1170 | 0.1696 | 0.0131 | 0.0111 | 0.0112 | 0.0064 |
| LiGan | 0.6477 | 0.0775 | 0.2492 | 0.0005 | 0.0224 | 0.0019 | 0.0008 |
| 3DSBDD | 0.6941 | 0.1311 | 0.1651 | 0.0025 | 0.0063 | 0.0010 | 0.0000 |
| GraphBP | 0.8610 | 0.0397 | 0.0868 | 0.0036 | 0.0040 | 0.0039 | 0.0010 |
| Pocket2Mol | 0.7623 | 0.0855 | 0.1413 | 0.0025 | 0.0044 | 0.0027 | 0.0013 |
| TargetDiff | 0.6935 | 0.0896 | 0.1924 | 0.0110 | 0.0059 | 0.0052 | 0.0025 |
| DiffSBDD | 0.7000 | 0.1154 | 0.1611 | 0.0081 | 0.0017 | 0.0093 | 0.0031 |
| DiffBP | 0.9178 | 0.0030 | 0.0792 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| FLAG | 0.5585 | 0.1341 | 0.2077 | 0.0265 | 0.0312 | 0.0347 | 0.0074 |
| D3FG | 0.7336 | 0.1158 | 0.1286 | 0.0056 | 0.0035 | 0.0088 | 0.0040 |
| DecompDiff | 0.6762 | 0.0978 | 0.1927 | 0.0064 | 0.0149 | 0.0088 | 0.0033 |
| MolCraft | 0.6735 | 0.0917 | 0.2056 | 0.0103 | 0.0094 | 0.0058 | 0.0035 |
| VoxBind | 0.7359 | 0.1083 | 0.1390 | 0.0000 | 0.0046 | 0.0120 | 0.7382 |

#### E.1.2 Interaction

##### PLIP Interaction Pattern.

Table. 18 and  18 provide detailed interaction pattern analysis. Most models captured good interaction patterns, generating a substantial amount of hydrophobic and hydrogen interactions. DiffSBDD excessively generated hydrophobic interactions. Additionally, for interactions such as $\pi$-stacking, $\pi$-cation, and halogen interactions, which do not exist in the reference molecules, the models could probabilistically generate molecules capable of producing such interactions with the protein.

**Table 17: Frequency of interaction type.**
| Method | hydrophobic | hydrogen | water bridge | $\pi$-stacks | $\pi$-cation | halogen | metal |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Ref. | 3.0000 | 3.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| LiGan | 3.1354 | 3.5078 | 0.0000 | 0.1045 | 0.0441 | 0.0034 | 0.0000 |
| 3DSBDD | 3.1237 | 3.0853 | 0.0000 | 0.1405 | 0.0434 | 0.0053 | 0.0000 |
| GraphBP | 5.2008 | 0.8039 | 0.0000 | 0.0007 | 0.0046 | 0.0060 | 0.0000 |
| Pocket2Mol | 4.9366 | 1.8840 | 0.0000 | 0.0176 | 0.0081 | 0.0045 | 0.0000 |
| TargetDiff | 4.2444 | 3.5309 | 0.0000 | 0.1159 | 0.0377 | 0.0384 | 0.0000 |
| DiffSBDD | 3.9009 | 3.3202 | 0.0000 | 0.0986 | 0.0338 | 0.0279 | 0.0000 |
| DiffBP | 4.1027 | 1.4830 | 0.0000 | 0.0213 | 0.0039 | 0.0000 | 0.0000 |
| FLAG | 1.9319 | 2.1490 | 0.0000 | 0.0134 | 0.0185 | 0.0528 | 0.0000 |
| D3FG | 4.3149 | 2.4718 | 0.0000 | 0.0607 | 0.0260 | 0.0200 | 0.0000 |
| DecompDiff | 3.9284 | 3.4531 | 0.0000 | 0.1177 | 0.0436 | 0.0290 | 0.0000 |
| MolCraft | 4.0147 | 3.9456 | 0.0000 | 0.2038 | 0.0667 | 0.0377 | 0.0000 |
| VoxBind | 4.4920 | 2.8395 | 0.0000 | 0.1659 | 0.0599 | 0.0001 | 0.0000 |

#### E.1.3 Geometry

Here we give detailed bond length and bond angle distribution of the evaluated method’s generated molecules and the reference, in Table. 20 and  20. It shows that the

**Table 19: JSD Bond Length Comparisons across different methods.**
| Method | C-C | C-N | C-O | C=C | C=N | C=O |
| --- | --- | --- | --- | --- | --- | --- |
| LiGan | 0.4986 | 0.4146 | 0.4560 | 0.4807 | 0.4776 | 0.4595 |
| 3DSBDD | 0.2090 | 0.4258 | 0.5478 | 0.5170 | 0.6701 | 0.6448 |
| GraphBP | 0.5038 | 0.4231 | 0.4973 | 0.6235 | 0.4629 | 0.5986 |
| Pocket2Mol | 0.5667 | 0.5698 | 0.5433 | 0.4787 | 0.5989 | 0.5025 |
| TargetDiff | 0.3101 | 0.2490 | 0.3072 | 0.1715 | 0.1944 | 0.3629 |
| DiffSBDD | 0.3841 | 0.3708 | 0.3291 | 0.3043 | 0.3473 | 0.3647 |
| DiffBP | 0.5704 | 0.5256 | 0.5090 | 0.6161 | 0.6314 | 0.5296 |
| FLAG | 0.3460 | 0.3770 | 0.4433 | 0.4872 | 0.4464 | 0.4292 |
| D3GF | 0.4244 | 0.3227 | 0.3895 | 0.3860 | 0.3570 | 0.3566 |
| DecompDiff | 0.2562 | 0.2007 | 0.2361 | 0.2590 | 0.2844 | 0.3091 |
| MolCraft | 0.2473 | 0.1732 | 0.2341 | 0.3040 | 0.1459 | 0.2250 |
| VoxBind | 0.3335 | 0.2577 | 0.3507 | 0.1991 | 0.1459 | 0.3334 |

### E.2 Lead Optimization

We here give the details of interaction analysis. Since these tasks have provided a partial of the molecules as the context, it is hard to tell the superiority of generated substructures or geometries. Table. 22 and 22 gives details. it shows that

- •
The optimization on the side chains has the least effect on the interaction pattern in an overall result and has the greatest influence on the linker.
- •
GraphBP’s failure in scaffold hopping also can be reflected by the fact that the interaction pattern cannot be produced with the molecules.

**Table 21: Frequency of interaction type on lead optimization tasks.**
|  | Method | hydrophobic | hydrogen | water bridge | $\pi$-stacks | $\pi$-cation | halogen | metal |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Ref. | 3.0000 | 3.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| Linker | GraphBP | 5.5795 | 2.9812 | 0.0000 | 0.5362 | 0.3537 | 0.0000 | 0.0000 |
| Pocket2Mol | 5.1879 | 3.2759 | 0.0000 | 0.1708 | 0.1082 | 0.0326 | 0.0000 |  |
| TargetDiff | 5.3811 | 3.7133 | 0.0000 | 0.1675 | 0.0827 | 0.0404 | 0.0000 |  |
| DiffSBDD | 5.4132 | 3.1350 | 0.0000 | 0.1153 | 0.0132 | 0.0308 | 0.0000 |  |
| DiffBP | 7.0697 | 2.6017 | 0.0000 | 0.1226 | 0.0678 | 0.0204 | 0.0000 |  |
| Fragment | GraphBP | 3.0084 | 3.2071 | 0.0000 | 0.0947 | 0.0499 | 0.0196 | 0.0000 |
| Pocket2Mol | 4.4064 | 4.0854 | 0.0000 | 0.1238 | 0.0956 | 0.0131 | 0.0000 |  |
| TargetDiff | 4.3187 | 3.6436 | 0.0000 | 0.1040 | 0.0648 | 0.0405 | 0.0000 |  |
| DiffSBDD | 4.3086 | 3.6720 | 0.0000 | 0.1087 | 0.0089 | 0.0468 | 0.0000 |  |
| DiffBP | 5.6128 | 2.7838 | 0.0000 | 0.0552 | 0.0339 | 0.0175 | 0.0000 |  |
| Side Chain | GraphBP | 3.0173 | 1.2207 | 0.0000 | 0.2029 | 0.0321 | 0.0001 | 0.0000 |
| Pocket2Mol | 3.0643 | 3.8911 | 0.0000 | 0.0872 | 0.0708 | 0.0194 | 0.0000 |  |
| TargetDiff | 3.3468 | 3.6206 | 0.0000 | 0.0975 | 0.0527 | 0.0851 | 0.0000 |  |
| DiffSBDD | 3.6822 | 3.3479 | 0.0000 | 0.0271 | 0.0242 | 0.0098 | 0.0000 |  |
| DiffBP | 4.5444 | 2.5941 | 0.0000 | 0.0696 | 0.0349 | 0.0141 | 0.0000 |  |
| Scaffold | GraphBP | 0.9560 | 2.3953 | 0.0000 | 0.0000 | 0.0007 | 0.0007 | 0.0000 |
| Pocket2Mol | 3.2220 | 3.4761 | 0.0000 | 0.0282 | 0.0295 | 0.0246 | 0.0000 |  |
| TargetDiff | 3.5427 | 4.1047 | 0.0000 | 0.0612 | 0.0383 | 0.0303 | 0.0000 |  |
| DiffSBDD | 4.6225 | 3.2649 | 0.0000 | 0.0021 | 0.0138 | 0.1786 | 0.0000 |  |
| DiffBP | 5.0840 | 2.4424 | 0.0000 | 0.0061 | 0.0055 | 0.0110 | 0.0000 |  |

### E.3 Case Study

##### DRD3.

For DRD3, we give the distribution of different models’ generated molecules in chemical space in Figure. 8. Different methods capture different clustering centers of the actives, while their distributions show a great gap from the GEOM-DRUG’s randomly sampled molecules.

Figure: Figure 8: T-SNE visualization of chemical distributions of generated and active molecules on t-SNE.
Refer to caption: x59.png

### E.4 Validity

**Table 23: Denovo Validity and Rank**
| Method | Validity | Rank |
| --- | --- | --- |
| LiGAN | 0.42 | 12 |
| 3DSBDD | 0.54 | 11 |
| GraphBP | 0.66 | 10 |
| Pocket2Mol | 0.75 | 6 |
| TargetDiff | 0.96 | 1 |
| DiffSBDD | 0.71 | 7 |
| DiffBP | 0.78 | 4 |
| FLAG | 0.68 | 9 |
| D3FG | 0.77 | 5 |
| DecompDiff | 0.89 | 3 |
| MolCraft | 0.95 | 2 |
| VoxBind | 0.74 | 8 |

Validity is an important metric to evaluate whether the molecules generated by the models are valid. There are two methods to reconstruct the 3D positions of the atoms into a molecule with bonds, one is used in TargerDiff, Pocket2Mol and 3DSBDD, which we name as Refine; The second is to use Openbabel () the software, used in DiffSBDD. However, using these methods to reconstruct molecules always carries the risk of broken bonds, which makes the strategy of selecting connected atoms to form fragments and ultimately the final molecule crucial. Here, we define a chemically valid molecule as one where the number of atoms in the largest fragment is greater than 85% of the total number of atoms, and we use this as the criterion for the reconstruction. Additionally, we employed a ‘Refine + Openbabel’ strategy, where if refinement is unsuccessful, Openbabel is added as a reconstruction method. Under this definition, the validity of various methods in the de novo task is shown in Table. 23. DecompDiff and TargetDiff have the highest validity.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Why do you choose different GNN architectures for auto-regressive and diffusion-based models?
2. MolCraft seems to be missing from the t-SNE visualization in Figure 8. Also, the distributions of Vina Dock Energy and LBE are not provided. Could you please provide these results?
3. [credit to Associate PC] How might your benchmark and evaluation protocol need to be adapted to assess unified models that can generate both small molecules and macromolecules like proteins?
soundness: 3
strengths: 1. The unified code base is a nice contribution to the community and beneficial for future research.
2. The evaluation protocol is comprehensive with a reasonable benchmark setting.
3. The benchmarked methods are representative and state-of-the-art.
summary: This paper proposes a benchmark for SBDD, including a unified framework of generative graph completion for multiple tasks in the field and a comprehensive evaluation protocol.
weaknesses: 1. It would be better if the author could discuss the recent trend of training a unified model for small molecules and macromolecules such as proteins and nucleic acids, and its implications on the field of SBDD.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: Include more systems that showcase corner cases for a benchmark that are common in real scenarios. i.e KRAS12, BRD4

The comments were satisfactorily addressed.
soundness: 3
strengths: The paper fills a notable gap in the SBDD domain by providing a well-structured and unified benchmarking framework. The comprehensive evaluation protocol addresses the diverse nature of generative tasks.

The study uses extensive metrics to evaluate models, including metrics like Ligand Binding Efficiency (LBE) to address the size bias in generated molecules.

Application to real pharmaceutical targets like ADRB1 and DRD3 demonstrates the practical potential of the benchmark and supports the generalizability of the findings.

Overall it's a really well executed paper that focuses on an particular case of generative models, which is drug design, but deserved an acceptance to the main conference due to the relevance of the task.
summary: The paper proposes CBGBench, a comprehensive benchmark for SBDD tasks, which aims to unify various generative models in a fill-in-the-blank framework for 3D complex binding graphs. It introduces a modular and extensible framework and evaluates multiple state-of-the-art methods across different metrics, including interaction, chemical properties, geometric authenticity, and substructure validity. The paper also introduces four sub-tasks: linker, fragment, side-chain, and scaffold design, to provide insights into lead optimization applications.
weaknesses: Further testing on more diverse real world systems should be done. Exploring how this models behave in systems like KRAS12 for instance where the main goal is growing into subpockets, would enrich this study.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: See Weakeness. The W1 and W2 are my main concern, and if a satisfactory response can be provided to the above issues, I will consider increasing the score.
soundness: 3
strengths: S1. This paper has evaluated almost all the prevailing SBDD methods in generative models in AI conference. 
S2. It adapt some of the models to a range of tasks essential in drug design, considered sub-tasks within the graph fill-in-the-blank tasks.
S3. It establish a training, sampling and evaluation codebase, which is comprehensive and effective affter my testing.
S4. The benchmark evaluates all the models for the two real-world target proteins, as a solid case study.
summary: This paper introduces CBGBench, addressing the gap where the absence of standardization can lead to unfair comparisons and inconclusive insights. On one hand, this paper incorporates recent state-of-the-art studies into a unified framework for fair comparisons; on the other, it extends de novo SBDD tasks to include lead optimization and other related tasks. The paper conducts extensive experiments on these tasks and provides the corresponding code.
weaknesses: W1. Doubts of classification: In Line 165, it states that MolCraft using BFN as the variant of the diffusion models. However, I have two questions, Firstly, it appears that “one-shot model” in the paper refers to diffusion-based models. Given this, is it reasonable to classify BFN as a diffusion model? MolCraft generates in parameter space, while diffusion generates in data space, which I believe is a distinction. Is the authors’ classification reasonable in this regard?

W2. Doubts of evaluation metrics:  The article compares the clash ratio between protein and molecule interactions, denoted as Ratio_cca . Additionally, it defines the internal clash ratio within molecules, denoted as  Ratio_cm . Regarding the internal clash within molecules, since there are bonds connecting atoms, the defined “van der Waals radii overlap by ≥0.4Å” does not hold. For example, if this definition were applied to a benzene ring, the clash ratio would be 1.0, which fails to reflect the actual structural integrity of the molecule. Therefore, I believe this metric is unreasonable. 

Minor: 
Additionally, the article lacks citations for recent work related to SBDD and molecular generation, such as [1] [2]. I suggest that the authors include these references to enhance the completeness of the article.

[1] Zaixi Zhang, Mengdi Wang, Qi Lium. FlexSBDD: Structure-Based Drug Design with Flexible Protein Modeling
[2] Odin Zhang，et al. Deep Lead Optimization: Leveraging Generative AI for Structural Modification

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 2
questions: - It would be useful to extend the method table by also including which tasks they support, and their architectures and losses
- Training losses should be discussed
- The paper first describes the main de novo task very superficially, then it describes the subtasks more in-depth, then it goes back to describing the de novo task more in detail, and finally goes back to subtasks. It would have been much clearer to separate the main/sub tasks into different parts of the paper.
- The task description is missing the objectives/metrics. Overall I didn’t really understand the role of the four subtasks. These are very specific, and surely each method needs to have custom support for them. Which ones do? Do we expect new methods to support these 4 tasks? Isn’t it enough that a method is good at the “basic” de novo generation? I’m a bit confused why we care about these at all, or what is their significance? It’s also unclear how they relate to the probabilistic model or ML modelling. Are these some kind of conditionals?
- It’s unclear if the 4 tasks in table 2 have something to do with the main task. That is, do they share data in some sense? Are the 4 tasks more fine-grained versions of the de-novo data?
- I’m a bit confused what’s the point of the “unified” probabilistic model notation. It seems that this is used nowhere in the paper after it’s introduction.
- Why would you look at the MAE between two distributions? This doesn’t seem sensible. The distribution sizes and domains and types are mostly undefined [please use precise math]
- What does it mean to measure “atom types” of functional groups? I don’t understand. I though that you compared just some distributions of 25 groups, which should give you a probability vector of length 25 to be compared. The at/rt/fg should have no role here. [It would help so much to define the stuff in precise math so that I wouldn’t have to guess.]
- I fail to see where the accuracy of the de novo generation is described. I thought the task is to predict the correct ligand for a known protein (assuming there is only one correct ligand). Surely you want to measure how often you get this right. The paper also talks about probabilistic model of p(M|P). Where is this density evaluated: shouldn’t we see some logp values in the results?
- Or is the task to just generate some ligands (irrespective if it was the correct one), as long as they have good binding and properties, or come from some distribution? If the task really is generative, then one expects to use learning target of maximizing the model likelihood of the observed data (both training and test folds). I don’t see this in the paper either. The learning problem needs to be precise.
- Overall it seems that some metrics in the paper are about checking that the summary statistics of $p(ligand)$ or $p(ligand|protein)$ match the true distributions, and some are about making sure some $fitness(ligand,protein)$ is high. This is not formalised well, and it leads to the metrics and learning setting being vague and hand-wavy.  
- I fail to see the motivation for the different Vina scores. If Vina is pathological, then why would we still use it? I don’t see how IMP or MPBG helps either if they are based on Vina. I don’t see convincing arguments why the LBE fixes the Vina pathology: the table 11 has values all of the place. I think here less would be more, and it would be much better to just have one good metric for binding than lots of binding metrics of varying quality. 
- I did not understand the PLIP stuff.
- Overall the metrics are difficult to interpret. I don’t really know what the numbers mean. Is 0.4382 MAE good or bad? Is a 0.2345 JSD high or low? Is Vina -3.75 good or bad? Is IMP 22 good? I have no idea. It would be insightful to visualise the distributions, or use some human-understandable metrics. For instance, you could use distribution overlap percentage or something else.
- It would have been useful to show some example generations, and their corresponding metrics.
- Using 5M iterates in each method seems unfair. Different methods train in different ways. I think you need to analyse the discrepancy between your results and the published results to clarify this.
- It seems that you change some methods architectures for no reason. This is not fair and will nullify the corresponding results. You can’t claim to benchmark method X if you change the method X from the publication.
- It’s unclear if you reimplement the methods in your codebase, or just collect published codebases in one place. Can you clarify?
- “We generate 100 molecules per pocket in test”. How do you do this? Why do you do this? Didn’t we have test data that tells us this? Now there are 10000 test molecules, while table 2 says that there are only 100. The learning setting is confusing, and using math to describe things would help. It seems that the entire setting in this paper is some kind of $D[p_gen || p_obs]$, which is not properly formalised.
- “We show the interaction analysis and chemical property in the main context and omit the interaction pattern analysis since maintaining the patterns is not necessary for lead optimization.” I don’t understand this. How do you use “chemical property” as context? How do you use “analysis” as context? What is "context"?
- I don’t understand what you do in the subtask training. Surely a method can’t be just applied in eg. linker generation if it wasn’t designed to do it? Are all 6 methods ones that support it, or do you somehow kludge this support to them? I’m really confused what do you train for extra 1M steps. What are even the loss functions? This is all super vague.
- The results of subtasks are all over the place. I’m not sure what can you conclude from this.
- Using ECFP to compare molecules in sec 5.3. is a very weak approach, and running it through tsne2d makes it even worse. This analysis has little to no value.
- The setting in 5.3. is weak. Thera are only 100 random controls (why not way more, like a million?). It’s unclear how many molecules you sample, but this should be a large number.
- Sec 5 concludes that the evaluation protocols are very consistent with real-world behavior. By far the best method in sec 5 is the DiffSBDD, which has by far best LBE, and ok Vina’s (Vina is pathological, so we should look at LBE anyways). But in Table 7 we see DiffSBDD being ranked 10/12, ie. one of the worst methods. It’s pretty clear that the real-world performance and de novo benchmark performance is not consistent.
soundness: 3
strengths: - The paper does a comprehensive benchmark, with lots of interesting results.
- The authors contribute a unified codebase, which could be very useful for future research.
summary: The paper proposes an extended benchmark protocol for SBDD methods, and does a large comparison of existing literature, and draws some insights from the results.

-----

Post-response update. My initial confusions have been adressed in the response, as well as issues in motivation. This is a useful contribution to SBDD's, with weaknesses wrt incrementality, presentation and experimental insights. I'm increasing my score to borderline, and I'm still leaning on rejection while I wouldn't mind acceptance.
weaknesses: - The motivation of the work is weak. I’m not sure why we need all of this stuff, or what questions the paper answers. The paper seems to establish an extended benchmark protocol that adds few extra evaluations to standard benchmarks. This could be useful, but I found the new additions to be a bit hand-wavy. The paper also proposes “unified” tasks and notation, but why? These tasks and notation is already common.
- The novel contributions of the work seem limited. The data and tasks seem to be from earlier works, and the benchmarks are added with just few extra evaluations, which also all seem to already exist. The main contribution is collecting all of this into one place. The performance comparison is a good contribution, but similar comparisons are done in every paper. The proposed new “unified” notation or task framing seems all well known. The main new contribution is probably the unified codebase, which can be very useful. The paper draws a bunch of conclusions from the results, which are interesting, but it’s unclear how novel they are.
- As a benchmark paper the presentation of the results is limited: there are too many decimals, no standard deviations, and there are lots of massive tables of “raw” performance values, which are not digested into transferable insights. The analysis is superficial, and there are no ablations or attempts to explain how or why the different methods perform differently. 
- As a survey paper I didn't find this paper particularly good at summarising or organising the domain in terms of methods, data or metrics. Use of math is inconsistent.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. What is the difference between fragment growing and side chain decoration on the implementation level? The text mentions ligand decomposition for the fragment growing task, but the detailed procedure of the decomposition is not explained. In Figure 3, how is the scaffold selected in the case of side chain decoration?
2. Some of the employed evaluation metrics compare the generated distribution of molecules to a known distribution of molecules. Would this approach not penalize methods that generate molecules with novel scaffolds? Also in the t-SNE plot (Figure 4) that compares the set of generated molecules to the actives, the compounds generated by a model can be significantly different from the known actives but still active. A good example of this behavior would be genetic algorithms that optimize docking scores. These methods are not trained on any data, so they can easily learn to produce compounds that have a unique structure and dock well to the target protein.
soundness: 3
strengths: Originality:
- A new benchmark is proposed for comparing the performance of recently developed SBDD methods.
- This benchmark also includes additional tasks like fragment linking or scaffold hopping, for which do not exist any standardized benchmarks.
- A taxonomy of SBDD generative models is proposed.

Quality:
- The benchmark covers a wide range of generative models, including autoregressive and one-shot generative models.
- Both evaluation methods and scoring network architectures are standardized.

Clarity:
- The benchmark is clearly defined, and the diagrams in Figures 2 and 3 explain different tasks and model types.
- The evaluation metrics are explained in an easy-to-follow way.

Significance:
- The proposed benchmark should accelerate research on new generative models that generate molecules binding to a given protein.
- The benchmarking code is attached to this submission, making it easier to test new models in the future.
summary: The paper introduces a new benchmark for generative models in structure-based drug design. Recently, many machine learning models have been proposed to help solve the problem of designing compounds that can interact with their protein targets. However, the comparison of these methods is difficult due to the lack of standards and relevant benchmarks. CBGBench is a benchmark that aims at standardizing model comparison and proposing evaluation metrics that can be used to assess the performance of generative models. Additionally, the benchmark contains tasks that can be used to evaluate models' capabilities of fragment linking, scaffold hopping, fragment growing, and side chain generation. Several recent generative models were compared using the proposed benchmark, and conclusions were drawn from these experiments. Moreover, a case study was conducted to demonstrate the quality of compounds generated by these methods for two selected GPCR targets.
weaknesses: Originality:
- This work does not discuss other benchmarks for generative methods. In particular, Zheng et al. [1] have recently introduced an SBDD benchmark for generative models.

[1] Zheng, Kangyu, et al. "Structure-based Drug Design Benchmark: Do 3D Methods Really Dominate?." arXiv preprint arXiv:2406.03403 (2024).

Quality:
- The requirement that "each connecting fragment must consist of more than five atoms" seems arbitrary. Why are models that occasionally generate fewer atoms considered insufficient?
- The benchmark could propose a method of computing confidence intervals, e.g. by sampling the generated molecules. It would facilitate the judgment of whether one of the tested models is significantly better than the others.

Clarity:
- "Real-world targets" may be a confusing name for the structures used in the case study. The protein structures in the CrossDocked2020 dataset were also real-world targets, which might confuse some readers. Maybe it would be better to call the structures in the case study as selected targets.
- Additionally, I recommend rephrasing statements like "the established evaluation protocols exhibit strong consistency and generalizability on real-world target data." The real-world target data could be understood as binding data from laboratory experiments, but docking experiments usually do not correlate with such experimental data.
- The text in Figures 4 and 5 may be too small.

Minor comments:
- A typo in line 68: "Vina enery"
- A typo in line 475 "the neurotrans-625 mitter"

Meta-review: The paper makes a solid contribution by introducing a novel benchmark for models applied to structure-based drug discovery. 

Reviewers appreciated the scope of the evaluated methods, the design of the subtasks, and the shared codebase. All reviewers voted to accept the paper. 

I agree with the reviewers and I am happy to recommend the work for the ICLR program.

Decision: Accept (Spotlight)