Redefining the task of Bioactivity Prediction
Authors: Yanwen Huang
Bowen Gao
Yinjun Jia
Hongbo Ma
Wei-Ying Ma
Ya-Qin Zhang
Yanyan Lan
Venue: ICLR 2025 Poster (2025)
Keywords: Bioactivity Prediction
New Dataset
Primary area: datasets and benchmarks

Abstract: Small molecules are vital to modern medicine, and accurately predicting their bioactivity against protein targets is crucial for therapeutic discovery and development. However, current machine learning models often rely on spurious features, leading to biased outcomes. Notably, a simple pocket-only baseline can achieve results comparable to, and sometimes better than, more complex models that incorporate both the protein pockets and the small molecules. Our analysis reveals that this phenomenon arises from insufficient training data and an improper evaluation process, which is typically conducted at the pocket level rather than the small molecule level. To address these issues, we redefine the bioactivity prediction task by introducing the SIU dataset-a million-scale Structural small molecule-protein Interaction dataset for Unbiased bioactivity prediction task, which is 50 times larger than the widely used PDBbind. The bioactivity labels in SIU are derived from wet experiments and organized by label types, ensuring greater accuracy and comparability. The complexes in SIU are constructed using a majority vote from three commonly used docking software programs, enhancing their reliability. Additionally, the structure of SIU allows for multiple small molecules to be associated with each protein pocket, enabling the redefinition of evaluation metrics like Pearson and Spearman correlations across different small molecules targeting the same protein pocket. Experimental results demonstrate that this new task provides a more challenging and meaningful benchmark for training and evaluating bioactivity prediction models, ultimately offering a more robust assessment of model performance.

Published as a conference paper at ICLR 2025
REDEFINING THE TASK OF BIOACTIVITY PREDICTION
Yanwen Huang2 Bowen Ga Yinjun Jia', Hongbo Ma², Wei-Ying Ma', Ya-Qin Zhang', Yanyan Lan 1,43
"Istitute for AI Industry Research (AIR), Tsinghua University
DDpaarmeent of Pharmaceutical Science, Peking University
Department of Computer Science and Technology, Tsinghua University *Beijing Academy of Artificial Intelligence (BAAI)
ABSTRACT
Small molecules are vital modern medicine, and accurately predicting their bioac- tivity against protein targets is crucial for therapeutic discovery and development. However, current machine learning models often rely on spurious features, leading to biased outcomes. Notably, simple pocket-only baseline can achieve results comparable to, and sometimes better than, more complex models that incorporate both the protein pockets and the small molecules. This phenomenon arises from insufficient training data and an improper evaluation process, which is typically conducted at the pocket level rather than the small molecule level. To address these issues, we redefine the bioactivity prediction task by introducing the SIU dataset-a million-scale Structural small molecule-protein Interaction dataset for Unbiased bioactivity prediction task, which is 50 times larger than the widely used PDBbind. The bioactivity labels in SIU are derived from wet experiments and organized by label types, ensuring greater accuracy and comparability. The complexes in SIU are constructed using majority vote from three commonly used docking software programs, enhancing their reliability. Additionally, the structure of SIU allows for multiple small molecules to associated with each protein pocket, enabling the redefinition of evaluation metrics like Pearson and Spearman correlations across different small molecules targeting the same protein pocket. Experimental results demonstrate that this new task provides more challenging and meaningful bench- mark for training and evaluating bioactivity prediction models, ultimately offering more robust assessment of model performance. Dataset and Code available at: itttps://gile / com bowen -aao-aaoooo-
INTRODUCTION
Small molecules are essential active components in life-saving therapeutic drugs, with their safety and efficacy intricately linked to interactions with various protein targets within the human body. Consequently, bioactivity prediction is critical task in the drug discovery process (Tropsha et al.) 2024; Gaulton & Overington 2010), driven by the rapid advancement of machine learning methods. In this context, "bioactivity" encompasses the diverse biological effects resulting from small molecule- protein interactions, including binding responses-commont
quantified by the dissociation constant (K2) and the inhibition constant (K;)-as well as functional responses, typically assessed through the half-maximal inhibitory concentration (IC50) and the half-maximal effective concentration (ECso).
Recently, various 3D machine learning models have been proposed in this direction (Townshend et al., 2020: Zhou et al., 2022 Gao et al., 2023a; Luo et al.) 2023), achieving significant advancements. These methods utilize the structural information of small molecules and protein targets as inputs to learn mapping function between these inputs and bioactivity labels. This methodology is inherently sound and explainable, as biological insights suggest that the biological effect of a small
'Equal contirbution
I Work was done while Yanwen Huang was an intern at AIR. "oorrsponndence 1anyanyan@al tai r.tiiighua wwwhhaa.eduu...

---

Published as a conference paper at ICLR 2025
molecule largely depends on its 3D shape complementarity with its protein targets (Verma et al. 2010), a principle known as the key-lock modulation theory (Koshland Jr 1995 Eschenmoser 1995). Nevertheless, the applications of these methods have not yielded satisfactory results regarding drug discovery capabilities. For instance, when using predicted biological labels to differentiate between active and inactive molecules-an essential task in virtual screening- taeiin predictive models often fail to compete with widely used docking methods, as noted in Shen et al. (2021) and a et al. (2023b).
Our analysis reveals that these models can be easily biased to some spurious features, leading to inaccurate predictions based on shortcuts. We propose a pocket-only baseline to diagnose the current bioactivity prediction task. While previous works assume that the bioactivity labels are determined by the interaction between small molecules and protein targets, they tend to assess only the protein target while ignoring the provided small molecules, representing degenerate solution, As shown in Figure TA and B. experiments on the widely used Atom3D ligand binding affinity (LBA) prediction dataset (Townshend et al. 2020) demonstrate that this pocket-only approach achieves, or can even outperforms, models utilizing the complex information across both 30% and 60% sequence identity splits, These results support our claim by suggesting that statistical irregularities in the data enable a model to achieve bioactivity predictions beyond what should be possible without access to the small molecule information.
A
(1)
correiation (1)
Predictions
Label Values
Predicted Value
Value of Labe's
134
Mol pockett only
pocket only
Paarznn come
Identity
60 identity
09
identity
60 identity
Label Value
Figure 1: Analysis of Atom3D bioactivity prediction task. The evaluation metrics include (A)
Root Mean Squared Error (RMSE) and (B) Pearson correlation. The models tested include: a GNN model using the full progedn-ligand complexes as inputs, Uni-Mol model with both a small molecule encoder and a protein encoder, and Uni-Mol model with only a protein encoder which only takes pocket side information. Performance is evaluated across different sequence identity splits (30% and 60%). It shows that Pocket-only model can overfit Atom3D bioactivity prediction task. (C) Predicted versus actual label values for various small molecules within a single protein target.
Upon further analysis, we find that the key issue stems from the improper definition of the current bioactivity prediction task, particularly in terms of both data construction and evaluation metrics.
From data perspective, the constructed training data is not sufficient for developing robust bioactivity predictor. Although previous works have utilized different training data, they are all derived form PDBbind (Wang et al. 2004 2005), which contains only about 20,000 small molecule- protein target pairs. More importantly. yl, for each protein target, these datasets typically feature only a single small-molecule ligand. This introduces bias into the training data, causing models to primarily learn the bioactivity range for each protein target rather than differentiating between various small molecules interacting with the same target. As demonstrated in Figure [CC when testing model with different small molecules for the same target, even with both protein and small molecule information provided, the model generates predictions that cluster around the mean bioactivity value of the target, while the actual label values vary across much wider range. This behavior suggests that the model trained on the current dataset fails to differentiate between different small molecules. This also helps to explain why pocket-only baselines can achieve unexpectedly good metric values.
From an evaluation perspective, the current metrics fail to accurately reflect how well models capture the interactions between protein target and diverse small molecules. Specifically, established metrics like Pearson and Spearman correlations are computed across different protein targets rather than across multiple small molecules for the same target. This approach primarily measures differences between various protein targets. Consequently, models can overfit by relying predominantly on pocket information without truly learning the nuances of small molecule binding.
2

---

Published as a conference paper at ICLR 2025
To address these issues, we propose redefining the bioactivity prediction task in this paper. Our strategy involves constructing a novel, large-scale structural dataset of small pollecule-proeeine interactions, featuring multiple small molecules for each protein target, and evaluating metrics across these different small molecules. A significant challenge lies in constructing a large-scale dataset of reliable small molecule-protein complexes, as high-quality structural data depends on aabor-intensive and time-conuuming wet-lab experiments. To tackle this, we first sourced, cleaned, and deduplicated small molecules and protein targets from relevant databases containing high-quality bioactivity labels. For each protein and its various pockets, we utilized multiple docking software programs, such as Vina (Trott& & Olson 2010), to dock associated molecules, generating primary interaction complex structures. Subsequently, majority vote mechanism was employed to obtain high-quality interaction poses. Furthermore, we differentiated between various label types, such as K,, K,, IC50. and EC50. to mitigate potential biases associated with label types during training and evaluation. This resulted in a large-scale Structural dataset of small molecule-protenn Interactions for Unbiased bioactivity prediction, namely SIU.
The SIU dataset comprises over 5.34 million conformations and features 1.38 million rigorously curated bioactivity annotations, each clearly designated by label types. This extensive dataset provides comprehensive coverage of diverse small molecules, surpassing the limitations of datasets restricted molecules structurally similar co-crystal ligands. It also includes wide array of protein targets across all major protein classes, with each protein linked to multiple PDB IDs that reflect distinct pocket conformations (not necessarily different binding sites). Notably, SIU differs from existing datasets that often overlook critical distinctions between label types, making it more suitable for fair bioactivity prediction and comparison.
With the availability of multiple small molecules with bioactivity labels for each protein target in SIU, we redefine the evaluation metrics by calculating values among different small-molecule ligands with the same target, rather than across different targets. The results are then averaged across targets using mean pooling. This approach ensures that the evaluation metrics accurately reflect the biactivity difference between small molecules within the same targets, thereby mitigating the aforementioned evaluation bias.
We compare the experimental results of training several classical baseline models PDBbind and SIU. Two key findings highlight the outperformance of SIU over PDBbind. First, when evaluated using traditional metries like RMSE, Pearson, and Spearman correlations across different targets, models trained on SIU demonstrate sigmificant improvements compared to those trained on PDBbind, reflecting the value of the inclusion of more structural data. Notably, this performance enhancement persists even after removing data with high sequence identity from the test set, while models trained on PDBbind do not undergo the same removal. Second, our redefined metrics reveal substantial drop in performance when evaluating small molecules within the same target. For instance, the Pearson correlation for K; can decrease from 0.485 0.036. This indicates that the new task is more challenging and that the bioactivity prediction abilities of the previous models may be overestimated due to improper task definitions. These results underscores the importance of the unbiased bioactivity prediction task we introduced, which we believe will advance the development of machine learning models that are truly beneficial for drug discovery.
RELATED WORK
Commonly used bioactivity prediction tasks include the Comparative Assessment of Scoring Func-
tions (CASF) task (Cheng et al. 2009. Li et al.) 2014bba Su et al.] 2018) and the Atom3D LBA task (Townshend et al. 2020). Both tasks are derived from the PDBbind dataset (Wang et al., 2004 2005). which is widely used and contains complex structures of small molecule-prose -rootcn interactions along with their corresponding bioactivity labels. However, the data cleaning and splitting methods differ between these tasks. The CASF-2016 task (Su et al.) 2018) consists of 285 -itein-ligand complexes, each labeled with an experimentally measured binding affinity. Since it does not provide a dedicated training set, prior research typically relies on self-defined training datasets derived from PDBbind. In contrast, the LBA task in Atom3D (Townshend et al. 2020) provides predefined training and testing splits, using sequence identity-based splits on 30% and 60% to ensure that test results reflect the model's 's generalization ability. This task combines different label types, including ICan. K,, and K.,. into a unified prediction variable, with total of 4,463 complexes in the dataset.
3

---

Published as a conference paper at ICLR 2025
In this work, we introduce the SIU dataset address specific challenges in bioactivity prediction tasks. Similarly, large-scale, high-quality datasets like Papyrus (Béquignon et al. 2023). curated from diverse sources, address other critical aspects and contribute valuable resources the field.
Atom3D also introduced two widely adopted baseline models: a poxel-grid-basee 3D convolutional neural network (3D-CNN) and graph neural network (GNN) (Townshend et al., 2020). Recent advances bioactivity prediction have been driven by the application of pretrained models, such Uni-Mol (Zhou al., 2022) and ProFSA (Gao et al. 2023a). These models utilize large-scale pretraining on molecular and structural data to achieve state-of-the-art performance across various bioactivity prediction tasks. In Atom3D, binding affinity prediction models are evaluated using RMSE, Mean Absolute Error (MAE), Pearson correlation, and Spearman correlation metrics.
3
METHODS
3.1
STU DATASET CONSTRUCTION
A
Data cleaning and deduplication
Multi-software docking
Unbiased tasks
Ligands
ChEMBL
40
Ligand Poses
PDB1 1 (g
PDB2 Lggeon
BindingDB
Docking
PDB1 L
PROTEIN BANK
B
iC50
K
PDB2
Glide GOLD Vina
1,323,386
994,409
Char No Food
The
201,458
103,435
56,485 19,448
4,463
285
(UniPrat IDe):
targets
(POB IDs): 9,544
Protein
pockets
Small moleoule-
protein pairs: 249,631
poses:
molecule
Small
values:
and their
types
siu datanel
BU
StU K.
SIU
SIU
PoBbind
LBA
CABE 2018
1,720
1,313,386
1,385,201
Figure 2: Construction and features of the SIU dataset. (A) The construction pipeline began with the collection of small molecules and protein targets from established databases, followed by data cleaning and deduplication. The small molecules underwent a comprehensive multi-software docking process, where they were prepared and docked to their experimentally validated targets. For quality control, the resulting poses were filtered through a voting mechanism, resulting in a dataset organized by both PDB and assay, designed to enable unbiased bioactivity prediction.
(B) The SIU dataset offers large-scale structural data, making it more than fifty times the size of PDBbind and significantly larger than datasets currently used for bioactivity prediction tasks. (C) The SIU dataset meticulously structured enhance unbiased bioactivity prediction. features multiple pockets (identified by PDB IDs) associated with the same protein target, multiple small molecules mapped to individual pockets (green), multiple high-quality docking poses per small molecule, and detailed label type annotations corresponding to all bioactivity values (orange).
Bioactivity label data cleaning and deduplication. Non-structural bioactivity data were retrieved from ChEMBL (Mendez et al., 2019 Gaulton et al. 2012) and BindingDB (Chen et al.) 2001 Liu et al. 2007] Gilson et al. 2016). Non-drug-like small molecules were excluded based on criteria such molecular weight (150-650 Da), the presence of at least carbon atom, and
4

---

Published as a conference paper at ICLR 2025
minimum of nine heavy atoms (details in Appendix C.1 Each small molecule retained its original IUPAC International Chemical Identifier (InChl) keys (Heller et al. 2015) and Simplified Molecular Input Line Entry System (SMILES) notations (Weininger) 1988; Weininger al. 1989) to prevent mismatches arising from different software calculations. Small molecules were deduplicated using Extended-Coonecctiity Connectivity Fingerprints (ECFP) (Rogers & Hahn 2010). Molecules with Tanimoto similarity greater than 0,8 were clustered, and representatives were selected based on bioactivity, ensuring both quality and structural diversity while reducing computational expense in molecular docking. Deduplication was applied only to protein targets with a small molecule count exceeding 2, 1464, the 90th percentile across all targets.
Protein target information for each assay was standardized using UniProt IDs (Consortium 2015 umi 2017), ensuring consistency across datasets and alignment with structural data. Protein structures were retrieved, and pockets were extracted. An area within a 15 A radius of the co-crystal ligand in the same complex structure defined as a distinct pocket (identified by a single PDB ID), even if it belongs to the same binding site as pockets from other PDB files of the same protein. A filtering mechanism excluded PDB files with non-specific or irrelevant ligands, and pockets were further deduplicated using Fast Local Aligmment of Protein Pockets (FLAPP). Bioactivity labels were standardized to molar units (mol/ L) and converted to their negative logarithms, following conventions for drug-target binding affinity datasets (Ozüürk et al. 2018). The resulting dataset, featuring structural pocket information, non-structural small molecule SMILES, and bioactivity labels.
Feasibility study
RMSD: 1.544
0.10
Glide docking score: -9,388
GOLD Score: 92.36
Success Ratio
www.ingiinnggnnnno..nn Raslo
0.70
0.00
34
Outoff
Pocket PDB ID: 3PB7
RMSD: 1.985
D RMSD: 8.095
Glide docking score:
docking score:
GOLD Score: 97.49
GOLD Score: 79.71
Pocket PDB ID: 3PB7
Pocket PDB ID: 3PB7
Figure 3: Quality control of SIU structural data. (A) A feasibility study of our methods showing the impact of root mean square deviation (RMSD) success (when the pose simultaneously passes the consensus filter and has an RMSD < 2 A compared to the co-crystal pose) and remaining ratios (the ratio of poses passing the filter) was analyzed using co-crystal poses, treated the ground truth, and redocked into their original PDB pockets according to our docking procedure. (B-D) Visualization of our pose consensus mechanism, where RMSD is calculated between different docking poses from different software (within same pocket PDB ID 3PB7). A single Glide docking pose is compared with the top three docking poses generated by GOLD. (B) RMSD 1. 1.544 A: well-superimposed poses; (C) RMSD 1.985 A: similar predicted binding modes; (D) RMSD 8.095 A: fundamentally different predicted binding modes.
Structural data construction via multi-software docking SIU employs multiple docking software programs (Friesner et al. 2004 Verdonk et al., 22003] Trott & Olson 2010), reducing reliance on any individual docking software. Initial 3D conformations for the small molecules
generated prior to docking using the Glide LigPrep module with default settings. The preprocessed data were organized into formats compatible with the chosen docking software. Protein targets were prepared, and grid
5

---

Published as a conference paper at ICLR 2025
files generated according to each software's e specific requirements to ensure compatibility. Small molecules were then docked into the pockets of the protein structures (detailed in Appendix C.2)p
For quality control, the SIU structural data underwent a majority voting mechanism: only docking poses consistent across at least two of the three docking software were retained. This consensus-bard based approach mitigated the inclusion of erroneous misleading docking poses, thereby improving the overall quality and reliability of the dataset.
We investigated the selection of the consensus filtering RMSD cutoff by evaluating the trade-off between pose accuracy and the quantity of retained data. Experiments were conducted to assess the impact of varying RMSD cutoff's on these factors (Figure3A), In this experiment, we re-docked small- molecule ligands with known co-crystal structures using different docking software. A successful docking pose was defined as one with an RMSD of less than Á compared to the experimental structure. The results demonstrate that with an RMSD cutoff below 2 A, significant number of molecules are retained, and the success rate of the poses satisfactory. However, the RMSD cutoff increases, the number of retained poses rises slightly, but their accuracy decreases substantially. This suggests that our consensus method effective for quality control of docked structures. Furthermore, Figure 3-D show that when the RMSD is around 2 Á, key interactions are preserved, indicating potentially valid docking result. Based on these observations, an RMSD cutoff of À was selected as the optimal threshold.
3.2 DATASET OVERVIEW
Large-scale. The SIU dataset comprises 5,342,250 conformations detailing small molecule- protein interactions, each entry providing comprehensive structural and bioactivity information, as shown in figure [ It includes 1,385,201 bioactivity labels derived from wet experiments, each with standardized values and clearly annotated label types. The top four label types by small molecule- protein pair count are half-maximal inhibitory concentration ICso (994,409), K, (201,458), half- maximal effective concentration EC30 (103,435), and K2 (56,485), which form the primary subset used in our subsequent experiments.
Diversity. SIU offers an extensive range data, encompassing 214,686 diverse small molecules and 1,720 distinct protein targets. It includes experimentally validated low-bioactivity or inactive molecules, which are often absent in structural datasets from wet experiments, thus providing valuable negative data for Al-driven drug discovery (AIDD). The dataset features broad protein type coverage, including proteins from different species and major protein classes. As illustrated in Figure 4D, the assay values of different protein targets vary significantly. This broad coverage ensures comprehensive representation of small molecule-proteii interaction modes, enhancing the relevance of our bioactivity prediction tasks to real biological environments.
High-quality. The structural information on small molecule-protein interactions in SIU is of high quality, due to multi-software voting mechanism that maximizes docking accuracy within computational limits. As detailed in the structural data construction section, we achieved a satisfactory balance between data accuracy and scale, presenting high-quality data unobtainable with a single docking software or solely by ranking based on software- settware-preditedd docking scores. Docking software often provides successful simulated docking poses within the top-ranking positions, but these not always ranked first by docking scores. Our method, however, is based on the consistency of docking pose sampling across different algorithms. By examining consensus among different docking algorithms, we effectively ensure more accurate docking pose data.
Well-organized. SIU's bioactivity labels are meticulously curated and systematically organized by PDB IDs and label types, ensuring data integrity and enabling effective PDB-wise and assay-wise comparisons. This organization offers robust resource for unbiased bioactivity prediction, addressing the limitations of existing datasets that often fail to distinguish clearly between different bioactivity label types. Traditional machine learning measurements of correlations in bioactivity prediction tasks are often ineffective due to the lack of clarity in existing datasets. SIU can also address this problem, ensuring more precise and meaningful analyses. Our structured approach facilitates nuanced assessments, such as evaluating the impact of specific small molecule transformation on protein interactions comparing the efficacy of different compounds within the same protein pocket context.

---

Published as a conference paper at ICLR 2025
A
B
Pairwise b-test p value differences between assay values of different label types
The distribution of assay values for different iabol types
8Cgp
6Gx
4.
C
Pairwise 1-test value differences between assay values of different protein targets
distribution
for different targets
Figure 4: Differences in assay values across label types and protein targets. (A) The mean assay values vary among representative label types, as shown by a heatmap of pairwise t-test p-values. Smaller p-values (lighter colors) indicate significant differences.
(B) Violin plots illustrate the distributions of label values for different label types. (C) Differences in mean assay values among ten protein targets. (D) The distributions of label values for different protein targets.
Versatile-usage. By providing well-structured SIU dataset, we not only support unbiased bioac- tivity prediction but also enable a deeper understanding of the molecular mechanisms underlying these interactions. This level of detail is also crucial for training robust models for other small silecule-proteinn meueture-bbsed downstream tasks, such as protein-ligant docking, virtual screening. and molecular generation, as discussed Appendix [F] Additionally, we provide parallel data of small pairs that have not been subjected to docking, meaning they lack structural ligand poses but retain valuable data points that either could not generate pose or did not meet quality control standards. This alternative dataset may be useful for evaluating bioactivity prediction models that do not rely on structural ligand poses, particularly when comparing methods that require 3D input data.
3.3 REFRAMING THE BIOACTIVITY PREDICTION TASK
Organization different label types. We organized the data by label types to address the common issue of mixing K2 and K, data, while also ensuring that other bioactivities are not neglected. As illustrated in Figure 4A and 43, the physical meanings the label types differ, leading to variations in their mean values and distributions. This highlights the importance of not mixing different label types and suggests that they should be treated as distinct tasks.
approach. As shown in Figure 4C, we selected 10 different protein targets along with their corresponding ICso label values and calculated the pairwise t-test p-values, A higher p-value (darker color) indicates a higher similarity in the mean values of different targets. This observation is further corroborated by Figure where violin plots depict the distribution of label values for each of the representative targets, highlighting variations both mean values and overall distributions. The figure clearly demonstrates that most target pairs exhibit significantly different distributions. These differences introduce bias into the dataset and explain why utilizing only pocket information can still achieve strong Pearson and Spearman correlation performance, shown in Figure]) [
Unbiased correlation metrics with group-by-pont
Thanks to the fact that our dataset provides multiple reframe the task and introduce new bioactivity prediction metrics, enabling more unbiased benchmarking of the models.
ligands for each protein, we
In the traditional machine learning approach, Pearson correlation is calculated across all ligand- pocket pairs without considering the individual protein pocket. Given N ligand-pocket pairs, where ú; represents the predicted bioactivity and y; represents the true bioactivity for each ligand-pocket
7

---

Published as a conference paper at ICLR 2025
pairs, the Pearson correlation is computed as:
Ei=1 (9: (1: 5)
- (i: 5) >i= (yi -0)²
where: 1 is the mean of the predicted bioactivities across all ligand-pocke pairs. T is the mean of the true bioactivities across all hgand-poccket pairs.
We calculate Pearson correlation after grouping by protein pockets (PDB IDs). For each protein pocket t, with n, ligands and their corresponding predicted bioactivities Gú,, and true bioactivities 3i, first compute the Pearson correlation re for each pocket:
E (Di, (vs. U.)
Ti ii= (Di, - D1)²
(Vii Dt)²
where: Vt is the mean predicted bioactivities for ligands within the same pocket t. Dr is the mean of true bioactivities for ligands within the same pocket t.
Once we have computed the Pearson correlation for each pocket, the overall correlation considering all protein pockets (Pearson") )) is obtained by mean pooling:
T
T
where T is the total number of pockets.
The similar method is also applied to the Spearman correlation to get Spearman".
This grouped-by waaeeed-by-oore by-pocket approach offers an unbiased and more useful evaluation, as it ensures that the correlation reflects the model's ability to predict bioactivities for different small- molecule ligands within the same protein pocket, reducing bias introduced by variations between different pockets.
Dataset Splits. To ensure robust evaluation and flexibility, the dataset includes multiple predefined splitting strategies. These include sequence identity filters at thresholds of 90%, 60%, and 30%, as well as combined sequence identity and structural similarity filter. A manually curated test set focuses on biologically meaningful tasks by incorporating representative protein targets across diverse classes, offering insights into the generalizability of predictions for key biochemically relevant targets. Additionally, bioactivity prediction models can be assessed using a 10-fold cross- tross-validation framework, providing reliable and unbiased approach for diverse training and testing scenarios (details in Appendix A.3
EXPERIMENTS AND ANALYSIS
4.1 EXPERIMENTS
We conducted experiments using several classical models to provide baseline results and analyze our SIU dataset. The models tested include voxel-grid based 3D-CNN model, Graph Neural Network (GNN) model, and pretrained models such as Uni-Mol and ProFSA (Gao et al., 2023a) as it achieves SOTA result for pootein-ligame binding affinity prediction task. Our experiments were performed in both Multi-Task Learning (MTL) and single-target settings. In the MTL setting, all data were combined to train a single MTL model. In the single-target setting, the Uni-Mol model was trained separately on individual labels.
The metries used in our analysis include RMSE, MAE, general Pearson and Spearman correlation, and the correlation after grouping by PDB IDs. The general Pearson and Spearman correlations are calculated by mixing pairs of protein pockets and molecules. The grouped correlation metrics calculated for different molecules within single protein pocket. We use Pearson" represent Pearson correlation after grouping by PDB ID (pocket), and Spearman to represent Spearman correlation after grouping by PDB IDs.
Results for multi-task learning is shown in Table I and the results for single task learning is shown in Table 2
8

---

Published as a conference paper at ICLR 2025
Table 1: Results for multi task learning with different label types. We show results for 3D-CNN,
GNN, Uni-Mol, and ProFSA trained on SIU 0.9 version.
RMSE 4
MAE
Pearson f
Pearson
Spearman
Spearman
3D-CNN
1.560
1.275
0.158
0.044
0.154
0.040
IC50
GNN
1.412
1.141
0.336
0.241
0.316
0.235
Uni-Mol
1.353
1.092
0.462
0.343
0.466
0.351
ProFSA
1.361
1.108
0.382
0.331
0.356
0.317
3D-CNN
1.518
1.234
0.128
0.010
0.128
0.004
EC50
GNN
1.334
1.025
0.444
0.108
0,481
0.120
Uni-Mol
1.273
1.017
0.428
0.178
0.461
0.144
ProFSA
1.255
0,971
0.438
0.204
0.495
0.154
3D-CNN
1.534
1,260
0.201
0.025
0.200
0.021
GNN
1.814
1,504
0.247
0.099
0.107
0.058
Uni-Mol
1.390
1.133
0.375
0.092
0.324
0.056
ProFSA
1.374
1.142
0.405
0.149
0.365
0.127
3D-CNN
1.503
1,233
0.173
0.024
0.167
0.038
Kg
GNN
1.711
1.431
-0.068
0.065
-0.147
0.033
Uni-Mol
1.429
1,223
-0.084
0.155
-0.175
0.144
ProFSA
1.546
1.334
-0.172
0.057
-0.205
0.029
Table 2: Results for single task training with different label types. We show the results with Uni-Mol model on PDBbind dataset, our SIU 0.6 version and 0.9 version dataset.
Train Set
RMSE
MAE
Pearson
Pearson" 1
Spearman f
Spearman"
PDBbind
1.575
1.279
0,430
0.245
0.425
0.229
IC50
SIU 0.6
1.407
1.138
0.461
0.317
0.463
0.31
SIU 0.9
1.357
1.099
0.470
0.345
0.474
0.347
ECs0
SIU 0.6
1.400
1.163
0.280
0.171
0.284
0.150
SIU 0.9
1.340
1.096
0.384
0.196
0.379
0.142
PDBbind
1.315
1.085
0.368
0.040
0.323
0.026
K;
SIU 0.6
1.255
1.034
0.472
0.106
0.452
0.112
SIU 0.9
1.235
1.017
0.485
0.036
0.452
0.041
PDBbind
1.565
1.308
0.041
0.010
0.004
0.006
K,1
SIU 0.6
1.389
1.192
-0.149
0.052
-0.206
0.022
SIU 0.9
1.364
1.141
-0.033
0.103
-0.082
0.065
4.2 ANALYSIS
Different label types. The bioactivity prediction difficulty varies among different label types. The K₂ task is the most challenging, primarily due to the varying correlations between different label types, as shown in Figure 4 and 43. Although the means of K; and K, labels do not differ statistically, the distribution of these data groups is different. The intrinsic differences in label types of bioactivity arise from the principles of the wet-lab experiments used to measure them. Binding assays focus on the direct interaction between the small molecule and the protein target, providing insights into the strength and specificity of this binding through metrics like K, and Ka. using techniques such as surface plasmon resonance (SPR) (Schasfoort] 2017 [Egglbiennee al. 2003) and isothermal titration calorimetry (ITC) (Leavitt & Freire 2001). p. In contrast, functional assays measure the biological response elicited by the small molecule on the target, capturing its effect on biological system and often quantified by IC50 and EC5n by enzyme activity assays (Bisswanger) 2014 Hall 1996) or other wet experiment techniques. The inherent differences in what these assays measure mean that within
their values cannot be directly compared (Yung-Chi & Prusoff 1973). Furthermore, the categories of binding and functional assays, metries should not be used interehangeably. as K; and K, describe different aspects of binding affinity, just IC50 and EC50 describe different aspects of biological response.
Influence of our unbiased metrics grouping by PDB ID (pocket) across all label types results in a significant decline in both Pearson and Spearman correlations. This observation suggests that is more challenging correlation when assessing binding affinities for different molecules within the same pocket after grouping. This challenge primarily arises from the different distribution of binding affinities across various protein pockets, as shown Figure 4C and 4D. Furthermore, these findings highlight that conventional machine learning approaches to measuring correlation without grouping by target
As demonstrated in Figure calculating the correlation at after
achieve high

---

Published as a conference paper at ICLR 2025
Pearson
PDBbind
Pearson
SIU 60
Spearman
0.1
Spearman
1.0
0.1
-02
1.00
Figure 5: (A) Pearson and Spearman correlations for various label types, calculated both before and after grouping by PDB IDs. (B) Pearson correlations after grouping PDB IDs for different assay types trained different datasets.
may not effectively capture model's ability to differentiate between molecules targeting the same protein. Such discriminatory capacity is crucial in drug discovery, emphasizing the importance of focusing on molecular interactions specific to each target rather than general correlations across diverse targets, This underscores the necessity of our dataset, which measures correlation within the same PDB IDs, providing more relevant assessment of deep learning model's 's utility in drug discovery. Also, as the correlation calculated within same pocket, cannot overfitted with pocket only information, as it will result in similar prediction results for different molecules, which would lead to NaN when calculating the pearson or spearman correlation.
Effectivness of training on our dataset. We compare models trained on the PDBbind 2020 dataset with those trained on SIU versions 0.6 and 0.9. Notably, the PDBbind 2020 dataset was used in its en- tirety, without implementing any filtering techniques exclude pockets similar those in the test set. As illustrated in Table I0 and Figure models trained on the SIU datasets outperform those trained on PDBbind, despite the latter's lack of homology removal. This underscores the effectiveness of our large-scale dataset in enhancing model learning for binding affinity prediction. Also the 0.9 version gives better performance compared to the 0.6 version, indicating the influence of removing homology and scaling law of the dataset. We also provide the results of using docked structures of PDBbind to train the model, which is shown in Appendix E. The results yield the same conclusion.
5 CONCLUSION
We identified and further analyzed the inherent biases present mainstream bioactivity prediction tasks. These tasks tend to introduce bias in both the training and testing processes. During training, the bias arises due to the limited scale of small molecule-protee there is only one small molecule associated with each protein pocket. This limits the model's ability to learn the underlying interactions between small molecules and protein targets, leading to overfitting to the value range of these protein pockets. Furthermore, during testing, existing metries primarily assess the models' ability to discriminate between different protein pockets, neglecting their ability to rank various small molecules that interact with the same protein pocket. To address these critical challenges, we redefined the bioactivity prediction task by introducing novel, large-scale, high-un quality structural dataset with well-organized labels. We also developed new metrics that specifically evaluate a model's s ability to rank different small molecules for each protein target. Our analysis, which included testing several classical models as baselines, demonstrates that our dataset improve model performance. Moreover, our proposed metrics provide more challenging and meaningful evaluation of bioactivity prediction models. Therefore, the task we introduced and the SIU dataset created represent valuable contributions to the field.
pairs; most current datasets,
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by Beijing Academy of Artificial Intelligence (BAAI).
REFERENCES
Uniprot:
the universal protein knowledgebase. Nucleic acids research, 45(D1):D158 -D69,, 2017.
Lucia Altucci, Mark D Leibowitz, Kathleen M Ogilvie, Angel R de Lera, and Hinrich Gronemeyer. Rar and ixr modulation in cancer and metabolic disease. Nature reviews Drag discovery, 6(10): 793-810. 10, 2007.
Misty M Attwood, Doriano Fabbro, Aleksandr V Sokolov, Stefan Knapp, and Helgi B Schiöth. Trends kinase drug discovery: targets, indications and inhibitor design. Nature Reviews Drug Discovery, 20( 20,1)),33,,.,, 2021.
Olivier JM Béquignon, Brandon J Bongers, Willem Jespers, Adriaan P IJzerman, B van der Water, and Gerard JP van Westen. Papyrus: a large-scale curated dataset aimed at bioactivity predictions. Journal of cheminformatics, 15(1):3, 2023.
Hans Bisswanger. Enzyme assays. Perspectives in Science, 1(1-6)41-55, 2014.
Matthias Bureik, Michael Lisurek, and Rita Bernhardt. The human steroid hydroxylases eypl lb1 and cyp1 1b2. 2002.
Xi Chen, Ming Liu, and Michael Gilson. Bindingdb: a web-accessible molecular recognition database. Combinatorial chemistry & high throughput screeming, 4(8):719-725, 2001.
Tiejun Cheng, Xun Li, Yan Li, Zhihai Liu, and Renxiao Wang. functions on diverse test set. Journal of chemical information and modeling, 1974):107-- 2009.
Comparative assessment of scoring
Philip Cohen, Darren Cross, and Pasi A Jänne. Kinase drug discovery 20 years after imatinib: progress and future directions. Nature reviews drug discovery, 20(7):551-569, 2021.
UniProt Consortium. Uniprot: a hub for protein information. Nucleic acids research, 43(D1):
D204-D212, 2015.
áB Danielson. The cytochrome p450 superfamily: biochemistry, evolution and drug metabolism in humans. Current drug metabolism, 3(6):561-597, 2002.
Ilia G Denisov, Thomas M Makris, Stephen G Sligar, and Iime Schlichting. Structure and chemistry of cytochrome p450. Chemical reviews, 105(6): 2005.
Patrick Englebienne, Anne Van Hoonacker, and Michel Verhas. Surface plasmon resonance: princi-
ples, methods and applications in biomedical sciences. Spectroscopy, 17(2-3):255-255 273. 2003.
Albert Eschenmoser. One hundred years lock-and-key principle. Angewandte Chemie International Edition in English, 33(23-24) 1995.
Paul G Francoeur, Tomohide Masuda, Jocelyn Sunseri, Andrew Jía, Richard B Iovanisci, Ian Snyder, and David R Koes. Three- Three-dimensioonll convolutional neural networks and cross-docked data set structure-based drug design. Journal of chemical information and modeling, 4009):4200-4215, 2020.
Richard A Friesner, Jay Banks, Robert B Murphy, Thomas A Halgren, Jasna Klicic, Daniel Mainz, Matthew P Repasky, Eric H Knoll, Mee Shelley, Jason K Perry, et al. Glide: new approach for rapid, accurate docking and scoring. 1. method and assessment of docking accuracy. Journal of medicinal chemistry, 47(7):1739- 1749, 2004.
Bowen Gao, Yinjun Jia, YuanLe Mo, Yuyan Ni, Wei- Ying Ma, Zhi-Ming Ma. and Yanyan Lan. Self-supervised pocket pretraining via protein International Conference Learning Representations, 2023a.
alignment. In The Twelfth

---

Published as a conference paper at ICLR 2025
Bowen Gao, Bo Qiang, Haichuan Tan, Yinjun Jia, Minsi Ren, Minsi Lu, Jingjing Liu, Wei- Ying Ma, and Yanyan Lan. Drugelip: Contrasive protein molecule representation learning for vir- tual screening. In NeurlPS 2023, 2023b. URL https::/ppeerrereeee net ftr.eerrrrewwe..eeeoooour...e.. fort/forum?id- 1AbCgNcxm7
Anna Gaulton and John P Overington. Role of open chemical data in aiding drug discovery and design. Future Medicinal Chemistry, 2(6):903-990)) 2010.
Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, et al. Chembl: a large-scale bioactivity database for drug discovery. Nucleic acids research, 100DI::DI0O0D11U 100-D110,, 107, 2012.
Michael K Gilson, Tiging Liu, Michael Baitaluk, George Nicola, Linda Hwang, and Jenny Chong. Bindingdb in 2015: a public database for medicinal chemistry, computational chemistry and systems pharmacology. Nucleic acids research, 440D1)11045-D10053 2016.
George M Hall. Methods of testing protein functionality. Springer Science & Business Media, 1996.
Alexander S Hauser, Misty M Attwood, Mathias Rask-Andersen, Helgi B Schiöth, and David E Gloriam. Trends in gper drug discovery: new agents, targets and indications. Nature reviews Drug discovery, 16(12):829-842) 2017.
Stephen R Heller, Alan McNaught, Igor Pletnev, Stephen Stein, and Dmitrii Tchekhovskoi. Inchi, the iupac international chemical identifier. Journal of cheminformatics, 7:1-34, 2015.
Daniel Koshland Jr. The key-lock theory and the induced fit theory. Angewandte Chemie International Edition in English, 33(23-24):2375-2533 1995.
Stephanie Leavitt and Emesto Freire. Direct measurement of protein binding energetics by isothermal titration calorimetry. Current opinion in structural biology. 11(5):560- -606 2001.
Yan Li, Li Han, Zhihai Liu, and Renxiao Wang. Comparative assessment of scoring functions on an updated benchmark: 2. evaluation methods and general results. Journal of chemical information and modeling, 54(6): 1001:177711777 2014a.
Yan Li, Zhihai Liu, Jie Li, Li Han, Jie Liu, Zhixiong Zhao, and Renxiao Wang. Comparative assessment of scoring functions on updated benchmark: 1. compilation of the test set. Journal of chemical information and modeling,
2014b.
Tiqing Liu, Yuhmei Lin, Xin Wen, Robert N Jorissen, and Michael K Gilson. Bindingdb: a web- accessible database of experimentally determined protein-ligand binding affinities. Nucleic acids research, 15(suppl_1):D1 1191))):-DDD200 DD01,, 2007.
Yunan Luo, Yang Liu, and Jian Peng. Calibrated geometric deep learning improves dinase-drug binding predictions. Nature Machine Intelligence, 5(12):1390-1400 2023.
David Mendez, Anna Gaulton, A Patrícia Bento, Jon Chambers, Marleen De Veij. Eloy Félix, María Paula Magariños, Juan F Mosquera, Prudence Mutowo, Michal Nowotka, et al. Chembl: towards direct deposition of bioassay data. Nucleic acids research, 47(D1):D930-D940, 2019.
H Mori and M Mishina. Structure and function of the nmda receptor channel. Neuropharmacology, 34(10):1219-1237, 1995.
Hakime Öztürk, Arzucan Özgür, and Elif Ozkirimli. Deepdta: deep drug-targe binding affinity prediction. Bioinformatics, 34(17):881--i22 2018.
Liyan Qu and Xiuwen Tang. Bexarotene: promising anticancer agent. Cancer chemotherapy and pharmacology, 65:201-205, 2010.
Barry Reisberg, Rachelle Doody, Albrecht Stöffler, Frederick Schmitt, Steven Ferris, and Hans Jörg Möbius. Memantine in moderate-to-sseere alzheimer's disease. New England Journal of Medicine, 348(14):1333- 2003.
David Rogers and Mathew Hahn. extended-connectivin fingerprints. Journal of chemical information and modeling, 50(5):742-754, 2010.
12

---

Published as a conference paper at ICLR 2025
Santhosh Sankar, Naren Chandran Sakthivel, and Nagasuma Chandra. Fast local alignment of protein pockets (flap)): a system- cetee--ompiled program for lcrge-scale binding site alignment. Journal of Chemical Information and Modeling, 62(19))4810-433 4819, 2022.
Richard Schasfoort. Introduction surface plasmon resonance. 2017.
Chao Shen, Ye Hu, Zhe Wang. Xujun Zhang, Jinping Pang, Gaoang Wang, Haiyang Zhong. Lei Xu, Dongsheng Cao, and Tingjun Hou. Beware of the generic machine learning-base scoring functions in structure-based virtual screening. Briefings Bioinformatics, 22(3):bbaa070, 2021.
Minyi Su, Qifan Yang, Yu Du, Guoqin Feng, Zhihai Liu, Yan Li, and Renxiao Wang. Comparative assessment of scoring functions: the casf-2016 update. Journal of chemical information and modeling, 59(2):895 913, 2018.
Raphael JL Townshend, Martin Vögele, Patricia Suriana, Alexander Derry, Alexander Powers, Yianni Laloudakis, Sidhika Balachandar, Bowen Jing. Brandon Anderson, Stephan Eismann, et al. Atom3d: Tasks on molecules in three dimensions. arXiv preprint WxXiv2202.040033 2020.
Raphael John Lamarre Townshend, Martin Vögele, Patricia Adriana Suriana, Alexander Derry, Alexander Powers, Yianni Laloudakis, Sidhika Balachandar, Bowen Jing, Brandon M Anderson, Stephan Eismann, et al. Atom3d: Tasks molecules in three dimensions. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Alexander Tropsha, Olexandr Isayev, Alexandre Vameek, Gishert Schneider, and Artem Cherkasov. Integrating qsar modelling and deep learning in drug discovery: the emergence of deep qsar. Nature Reviews Drug Discovery, 23(2):141-1555 2024.
Oleg Trott and Arthur Olson. Autodock vina: improving the speed and accuracy of docking with new scoring function, efficient optimization, and multithreading. Journal of computational chemistry, 31(2):455-411 2010.
Marcel L Verdonk, Jason C Cole, Michael J Hartshorn, Christopher W Murray, and Richard D Taylor. Improved prgtein-ligand docking using gold. Proteins: Structure, Function, and Bioinformatics, 52(4):609-623, 2003.
Jitender Verma, Vijay M Khedkar, and Evans C Coutinho. 3d-qsar in drug design-a a review. Current topics in medicinal chemistry, 10(1):95-115, 2010.
Renxiao Wang, Xueliang Fang, Yipin Lu, and Shaomeng Wang. The pdbbind database: Collection of binding affinities for protein- ligand complexes with known three-dimensional structures. Journal of medicinal chemistry, 47(12))2977-2999
2004.
Renxiao Wang, Xueliang Fang, Yipin Lu, Chao- Yie Yang, and Shaomeng Wang. The pdbbind database: methodologies and updates. Journal of medicinal chemistry, 48(12):4111441 2005,
David Weininger. Smiles, a chemical language and information system. 1. introduction to methodol- ogy and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36 1988.
David Weininger, Arthur Weininger, and Joseph Weininger. Smiles. 2. algorithm for generation of unique smiles notation. Journal of chemical information and computer sciences, 29(2)9771011 1989.
Cheng Yung-Chi and William Prusoff. Relationship between the inhibition constant (ki) and the concentration of inhibitor which causes 50 per cent inhibition (i50) of an enzymatic reaction. Biochemical pharmacology, 22(23):3099-3108 1973.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang and Guolin Ke. Uni-mol: universal 3d molecular representation learning framework. In The Eleventh International Conference Learning Representations, 2022 2022.
13

---

pose2. sdf Published as a conference paper at ICLR 2025
APPENDIX
LIST OF CONTENTS FOR APPENDIX
A Detailed data usage description
B General statistics and quality evaluation
C Data construction methodology
D Model training details
E Additional experimental results
F Other potential applications of the SIU dataset
A DETAILED DATA USAGE DESCRIPTION
A.1 A PICKLE FILE THAT CONTAINS PROCESSED DATA AND LABELS FOR THE DATASET
The dataset is provided dictionary containing processed information. Each key represents UniProt ID, and the corresponding value is a list of dictionaries. Each dictionary represents data point and contains the following keys:
source data PDB ID and UniProt ID information, dictionary for different labels, including Ki, Kd, IC50, BC50, InChIKey the ligand, SMILES of the ligand
label
ik smi
A.2 STRUCTURE FILES
The structure files are organized as follows:
DIR
uniprotid
pdb_id id
pocket_pdb__1 file
inchikeyl
posel.sdf
pose2.sdf
pose3.
inchikey2
posel. sdf
pose3. sdf
Users can use this structure to process their own data for their models.
A.3 DATA SPLITS
To ensure both flexibility and robust evaluation, we provide a range of predefined dataset splitting strategies. Splits for the training and test sets of our task provided. However, do not impose fixed split for the dataset, allowing users the flexibility to perform their own splits. These options provide users with the flexibility evaluate models under various levels of sequence and structural
14

---

Published as a conference paper at ICLR 2025
similarity constraints. A 10-fold cross-validation setting is also provided, as it provides a reliable and unbiased framework for machine learning model evaluation by ensuring diverse training and testing scenarios.
The updated version of the dataset includes five different predefined splitting strategies:
1. 90% Sequence Identity Filter: A fixed test set is provided, and proteins with a sequence identity greater than 90% to the test set are removed from the training set.
2. 60% Sequence Identity Filter: A fixed test set is provided, and proteins with a sequence identity greater than 60% to the test set are removed from the training set.
3. 60% Sequence Identity + Structural Similarity Filter: A fixed test set is provided, and proteins with sequence identity greater than 60% to the test set are removed from the training set. Additionally, protein pockets with a structural similarity greater than 20% to the test set are also excluded from the training set.
4. 30 % Sequence Identity Filter: A fixed test set is provided, and proteins with a sequence identity greater than 30% to the test set are removed from the training set.
5. 10-Fold Cross-Valle Validation Split: The dataset is divided into 10 clusters. Any pair of proteins with a sequence identity greater than 60% are placed within the same cluster. This split can be used for 10-fold cross-validation.
Splitting methods. To ensure the generalizability of the experimental findings with SIU, we employed a manual curation approach for dataset splitting. We selected a set of 10 representative protein targets serve as the test set. These targets were intentionally chosen to cover diverse range of protein classes, including well-known drug targets such as G-Protein Coupled Receptors (GPCRs) (Hauser et al. 2017), kinases (Attwood et al. 202T Cohen al. (021), and cytochromes (Danielson 2002). This selection strategy was designed to encompass the bioactivity landscape across various protein functionalities, thereby enhancing the applicability of our results to wider range of potential drug discovery applications. We conducted non-homology analyses at two levels, 0.6 and 0.9, to ensure the independence and diversity of the training and test sets. For both versions 0.9 and 0.6, we have 21,528 data pairs allocated for testing. Specifically, version 0.9 includes 1, 207 data pairs for training and validation, while version 0.6 includes 386,330 data pairs for these purposes.
Table 3: The curated test set of 10 protein targets, covering a diverse range of protein classes and displaying an even distribution of small molecule-pockes pair counts.
UniProt
Gene name
Class
Small molecule- pocket pair count
P61073
CXCR4 HUMAN
GPCR
1376
P42866
OPRM MOUSE
GPCR
2379
Q00535
CDKS.HUMAN HUMAN
Kinase
2189
Q04759
HPCA_HUMAN
Kinase
2320
P15538
CHIB1_HUME _HUMAN
Cytochrome
2427
P13631
RARG HUMAN
Nuclear Receptor
1888
Q12879
NMDE1
OMDEI_HUMAN HUMAN
Ion Channel
2144
Q9UGNS
HARP2.HUMAN
Epigenetic
2251
Q86WV6
STING HUMAN
Others
2495
Q96SW2
CRBN_HUMAN
Others
2059
Test set construction. ensure the robustness and generalizability of the experimental findings with SIU, we meticulously curated test set composed of 10 protein targets, as listed in Table 3 These targets were selected to represent a wide range of protein classes, including G-Protein Coupled Receptors (GPCRs), kinases, cytochrome, nuclear receptor, ion channel, epigenetic, and others, ensuring broad coverage of the bioactivity landscape. For example, CCIIB_HUMANY belongs to the cytochrome P450 family, which is involved in the metabolism of various drugs (Bureik et al.) 2002;; Denisov et al. 2005). "RARG HUMAN" belongs to the Nuclear Receptor family, with drugs like bexarotene used for certain cancers (Altucci et al. 2007 Qu & Tang 2010). "NMDE1 HUMAN"
15

---

Published as a conference paper at ICLR 2025
represents the NMDA receptor, critical glutamate receptor in neurons implicated in various neuro- logical disorders, with memantine being an approved NMDA receptor antagonist for moderate to severe Alzheimer's s discase (Mori & Mishina 1995 Reisberg al. (003). Including these targets across various functionalities enhances the applicability of our results in drug discovery.
B GENERAL STATISTICS AND QUALITY EVALUATION
B.1 GENERAL STATISTICS OF THE DATASET
SIU represents large-scale, high-quality dataset of small molecule-protein interactions, meticulously organized to facilitate unbiased bioactivity prediction, both PDB-wise and assay-type-wist The dataset comprises total of 5,342,250 conformations. Each instance in the dataset provides detailed information about small molecule- -erctinn interactions, including the coordinates and element types of each atom in the small molecule and the corresponding pockets of each interaction. Additionally, the assay value and type of each conformation, along with other critical information, are carefully obtained and retained from the original bioactivity databases. This includes the UniProt and PDB ID of the protein pockets, well as the InChl keys (Heller et al., 2015) and SMILES Weininger (1988): Weininger et al. (1989) notations of the small molecules.
Table 4: The label count for 4 representative label
types in SIU total, SIU 0.9, and 0.6 versions.
SIU 0.9 version
SIU 0.6 version
Total
Train
Valid
Test
Total
Train
Valid
Test
MTL
1272335
1125727
125080
21528
407858
347697
38633
21528
ICo0
962063
854230
94859
12974
320594
276969
30651
12974
EC50
97952
84067
9508
4377
32842
25675
2790
4377
K; 198091
175442
19447
3202
47946
40188
4556
3202
Ka 54570
47347
5347
1876
17509
14003
1630
1876
B.2 FURTHER EVALUATION OF LABEL VALUE DIFFERENCES ACROSS LABEL TYPES
To further support our rationale for dividing predictions by label type, we consider the potential coupling relationships between label types and proteins, which could lead to distributional differences across label types. To address this, we selected only pairs that contain all four label types, ensuring that the corresponding list of proteins is identical for each label type.
A Pairwise 1-test p-valuo differences between values of different label types
The distribution assay values diffforont labal types
K2
ECis
Figure 6: Differences in assay values across different label types. (A) Pairwise differences mean assay values among these label types, as shown by heatmap of pairwise t-test p-values. Smaller p-values (lighter colors) indicate significant differences. (B) Violin plots illustrate the distributions of label values for different label types.
Despite this refinement, the figures still reveal distributional differences across label types. Further- more, by ensuring that each pair includes all four label types, calculated the Pearson correlation between label types for a given protein. The results are presented in Table
Thus, continue to believe that separating these types into distinct tasks is worthwhile approach.
16

---

Published as a conference paper at ICLR 2025
Table Correlation matrix for IC5o. EC Ka, and K2 values. The table shows pairwise correlations between the different bioactivity measures.
ICso
ECso
K
K,
ICso
0.0773
-0.0080
0.5849
BC50
0,0773
0.3752
0.2673
K.d K;
-0.0080
0.3752 0.2673
0.0463
0.5849
0.0463
B.3 FURTHER EVALUATION LABEL VALUE DIFFERENCES ACROSS PROTEIN POCKETS.
In Figure 4 we have vitualized the label value ranges difference in ten targets. To further provide more comprehensive and convincing demonstration, in 7 we provide here a heatmap and violin plot across more randomly sampled protein targets following the same logic. These representations further confirm that our findings remain consistent and valid in this broader context.
t-test differences assay values different protein targets
The distribution assay valuee different protein targets
Figure 7: Differences in assay values across more random-sampled protein targets. (A) Pairwise differences in mean assay values among these protein targets, shown by heatmap of pairwise t-test p-values. Smaller p-values (lighter colors) indicate significant differences. (B) Violin plots illustrate the distributions of label values for different protein targets.
B.4 FURTHER EVALUATION OF STRUCTURAL DATA QUALITY
We implemented rigorous quality control measures, such the consensus method described earlier, to enhance the overall reliability of the generated structrual data. The intention of our work is not to propose new docking algorithm but rather to create dataset that facilitates redefining the bioactivity prediction task. In this effort, our focus was on selecting robust and widely accepted mainstream docking methods.
Further evaluation in cross-docking scenario. We conducted experiment focusing target with 20 PDB structures with co-crystal ligands within the same binding site. After aligning these complex structures. we performed cross-docking on them and calculated the RMSD between the docked ligand poses in all holo pockets and the corresponding co-crystal ligand poses.
As shown in Table there is a significant improvement across various metrics for the docking poses that pass the filter compared to those that fail, highlighting the effectiveness of our consensus algorithm in ensuring the quality of docking poses.
Table 6: RMSD statistics structures that passed or failed the filter. The table shows mean and median RMSD values, well as the percentage of structures with RMSD below A, 2 Á, and 3 Á.
Metric
RMSD
RMSD
Percentage
Percentage Á RMSD
Percentage 3 Á RMSD
Mean
Median
Á RMSD
Passed the Filter
3.24
2.02
17.12%
49.42%
61 11. 16.44%
Failed the Filter
5.76
5.24
1.80%
6.26%
17

---

Published as a conference paper at ICLR 2025
B.5 FURTHER COMPARISON WITH PDBBIND
Additionally, as illustrated in Table 7 the dataset encompasses over 1,385,201 assay labels, each derived from corresponding wet-lab bioactivity experiments, ensuring the reliability and accuracy of the bioactivity information. SIU includes 1,720 diverse protein targets. with each protein poten- tially possessing multiple distinct binding pockets, verified through rigorous deduplication methods, resulting in total of 9,6622 unique pockets. The dataset also features a substantial and diverse collec- tion of small molecules, totaling 214,686, across all pockets. Importantly, we have only included protein pocket-small molecule pairs confirmed to be active or inactive through wet-lab experiments, amounting to over 1,291,33 1,32 million pairs.
Table 7: Comparison of PDBbind and SIU datasets, showing the number of pocket-molecule pairs, average molecules per pocket, and unique pockets and molecules.
Dataset
Pocket-molecule pairs
Avg. molecules per pocket
Unique pockets
Unique molecules
PDBbind
19,443
1
19,443 9,544
19,443
SIU
1,312,827
137.6
214,686
B.6 COMPARISON WITH FEP DATASET
By predicting binding free energy, FEP can be considered a bioactivity prediction method to some extent. However, handling one pocket at time represents inherent limitation of these methods, as they do not actively group data to address biases when evaluating the model. When evaluating models with FEP dataset, it is still necessary to calculate metrics as average across different targets to assess general performance. While reporting metrics for individual targets can highlight performance of one model specific targets, it does not provide a clear comparison of overall performance. Another limitation of the FEP dataset is the small number of targets and the limited number of small molecules per target. While this setup might serve as a held-out test for machine learning models, addresses a different aspect from the motivation of this paper, which aims to provide a more comprehensive and unbiased evaluation.
C DATA CONSTRUCTION METHODOLOGY
C.I DATA PREPROCESSING METHOD DETAILS
Label data extraction and cleaning. We retrieved non-structural bioactivity data from ChEMBL (Release 33) and BindingDB (version 202404), applying rigorous filtering to refine the dataset. For ChEMBL, data were extracted via SQLite, focusing on records with an wwwayss.onfidence.see sccore of 9, targeting a single protein, and classified as binding (B) or functional (F). We included entries with non-null values for activities.standarddrellll activities. .units were 'pM', 'nM', or "HM''. Further refinement was applied to activi- ties. actsvactiviyy comment to capture specific biological activity descriptions, and molecular weight (compound properties.mw freebase) was restricted to between 150 and 650 Da. For small molecules, stringent filtering criteria were used to exclude non-drug-lit molecular weight of 150-650 Da, containing at least one carbon atom, and having minimum of nine heavy atoms. Each small molecule retained its original IUPAC International Chemical Identifier (InChl) keys and Simplified Molecular Input Line Entry System (SMILES) notations. From BBn we extracted relevant data from the .tsv file and standardized protein target information using UniProt IDs. The ChEMBL and Binding datasets were merged by matching InChl keys for small molecules with UniProt IDs for protein targets, ensuring accurate alignment of bioactivity labels with their corresponding small molecule-protein interactions.
and activitieesstandard.ve value, where
entities. We selected molecules with
Label data deduplication. To deduplicate small molecules, the Extended Connectivity Fingerprint (ECFP) (Rogers & Hahn, 2010) was utilized. ECFP is a type of molecular fingerprint commonly used in cheminformatics to represent the structural features of small molecules. It extends the circular fingerprint method by encoding a molecule into a fixed-length bit string that reflects its substructural characteristies. ECFPs are widely applied in various computational chemistry tasks,
18

---

Published as a conference paper at ICLR 2025
PSBECAAC Delltt CMthuit
Defon
390003
X0000
Defore
250090
156890
400300
30000
100300
50090
20000
1003
1000
000
100 723
273
PS0 IMGEE
100P LATAanti AitrBhoorr
G
No XXX00ce DLRT AMPTee
Hg.
200
0.3
800 1009 1200
-35
15
25 50 755 100 125 1500 175
of
Figure 8: Property Distributions of Small Molecules Before and After Deduplication. Key characteristics such as molecular weight, water-oil partition coefficient (logP), hydrogen bond acceptor (HBA) count, and hydrogen bond donor (HBD) count remain consistent, despite a reduction in the total number of small molecules. (A-D) Distributions of small molecules across different ranges of molecular properties. The y-axes represent the number of small molecules within specific property ranges. (E-H) Cumulative distributions of the same molecular properties, with the y-axes indicating cumulative probabilities.
such as virtual screening, similarity searching, and machine learning. In this work, small molecules with high structural similarity (Tanimoto similarity > 0.8) were grouped into clusters. One or more representatives from each cluster, prioritized based on bioactivity, were selected to deduplicate the dataset while maintaining structural diversity. Notably, only small molecules within pockets where the molecule count exceeds the 90th percentile threshold (2,146) across all pockets were subjected to deduplication. As illustrated in Figure [] the diversity of small molecules was preserved following the application of this deduplication method. Fast Local Alignment of Protein Pockets (FLAPP) (Sankar jet al. 2022) is a program that used to calculate the structure similarity between two protein pockets. We remove structurally redundant pockets using this program. The threshold is set 0.9.
Pocket definition. Biologically speaking, a protein typically exhibits limited number of distinct binding sites, defined as regions accommodating specific ligands. It's important to differentiate between these binding sites and the "pockets" identify, which are characterized by the ligands used for their definition rather than their spatial distribution on the protein surface. In essence, multiple pockets can potentially reside within a single binding site, since a lot of binding sites have complex structures with multiple ligands therefore have a lot of PDB IDs. Different PDB IDs represent distinct resolved structures for single protein, all associated with the same UniProt ID. For each PDB ID, a single pocket was extracted, defined as the region centered on the co-crystal ligand within a 15 Á radius.
Mapping small molecule pocket. While our method be considered form of cross-docking, it fundamentally differs from the well-known CrossDocked2020 dataset (Francoeur et al. 2020). Specifically, we use only small molecules with experimentally determined bioactivity values to dock to their corresponding protein targets. This ensures that each small molecule is docked to at least the correct protein target. Additionally, our consensus docking not only generates structural data but also serves as a reverse strategy for identifying the appropriate binding pocket, thereby enhancing the reliability of the pocket assignments.
C.2 DOCKING METHOD DETAILS
Constructing a structural dataset of this scale required careful optimization of docking parameters to balance performance and computational resource efficiency. Ligands from bioactivity databases, provided as 1D representations (SMILES), were first processed with Glide LigPrep to generate up to 32 stereoisomers per molecule, accounting for potential variations in stereochemistry chiral centers. Ionization states were predicted at physiological pH to ensure accurate representation of charged forms. Multiple conformations were prepared for each small molecule to account for molecular
19

---

Published as a conference paper at ICLR 2025
flexibility.. Protein structures preprocessed by predicting their ionization states, removing water molecules, and preparing docking grid files centered on the co-crystal ligand. Docking was performed with default settings. The resulting conformations were filtered through a consensus pipeline to select the final set conformations included in the SIU dataset.
D MODEL TRAINING DETAILS
For GNN Model, we use the same model in Atom3D (Townshend et al. 2021). We train the model using NVIDIA A100 GPU. The batch size is 256, the max number of epochs is 20, the optimizer Adam, the learning rate is 1e-3.
For 3D-CNN Model, we use the same model in Atom3D. We train the model using NVIDIA A100 GPU. The batch size 256, the max number of epochs is 20, the optimizer is Adam, the learning rate le-4.
For Uni-Mol model, we use the pretrained model weights provided. The pretrained molecular encoder and pocket encoder outputs concatenated and passed through four-layer Multi-Layer Perceptron (MLP) with hidden dimension 1024, 521, 256, 128. We use four NVIDIA A100 GPU to train the model. The batch size is 384, the max number of epochs is 50, the optimizer Adam, the learning rate is le-4.
For ProFSA model, we use the pretrained model weights provided. The pretrained molecular encoder and pocket encoder outputs are concatenated and passed through a four-layer MLP with hidden dimension 1024, 521, 256, 128. We use four NVIDIA A100 GPU train the model. The batch size is 384, the max number of epochs is 50, the optimizer is Adam, the learning rate is 1e-4.
E ADDITIONAL EXPERIMENTAL RESULTS
E. ADDITIONAL RESULTS FOR USING THE DOCKED STRUCTURE OF PDBBIND
We conduct additional results using the docked structures instead of the original complex of PDBbind. The result is shown in Table
Table 8: Results for single task training with different label types. We show the results with Uni-Mol model on PDBbind dataset, our SIU 0.6 version and 0.9 version dataset. Pdocked is the model trained on PDBbind but with the docked structure instead of original complex structure.
Train Set
RMSE
MAE
Pearson
Pearson"
Spearman
Spearman"
PDBbind
1.575
1.279
0.430
0.245
0.425
0.229
IC50
Pdocked
1.600
1.304
0.439
0.241
0.436
0.229
SIU 0.6
1.407
1.138
0.461
0.317
0.463
0,311
SIU 0.9
1.357
1.099
0.470
0.345
0.474
0.347
SIU 0.6
1,400
1.163
0.280
0.171
0.284
0.150
ECs0
SIU 0.9
1.340
1.096
0.384
0.196
0.379
0.142
PDBbind
1.315
1.085
0.368
0.040
0.323
0.026
K,
Pdocked
1.301
1.059
0.365
0.035
0.311
-0.001
SIU 0.6
1.255
1.034
0.407
0.106
0,452
0.112
SIU 0.9
1.235
1.017
0.385
0.036
0.452
0.041
PDBbind
1.565
1.308
0.041
0.010
0.004
0.006
K
Pdocked
1.447
1.231
-0.015
0.034
-0.062
0.011
SIU 0,6
1.389
1.192
-0.049
0.052
-0.206
0.022
SIU 0.9
1.364
1.141
0.033
0.103
-0.082
0.065
E.2 ADDITIONAL RESULTS FOR DIFFERENT SPLITTING
The results for combining 60% sequence identity and 0.2 FLAPP structure similarity are shown in T
20

---

Published as a conference paper at ICLR 2025
The results for 10 folds split shown in Table I0 Those results pertain to the Uni-Mol model.
Table 9: Results for single task training with different label types. We show the results with Uni-Mol model SIU that is based our original 0.6 split, with FLAPP to remove additional similar pockets from the train set.
RMSE
MAE
Pearson
Pearson"
Spearman
Spearman"
ICso
1.389
1.117
0.411
0.353
0.412
0.355
EC
1.399
1.164
0.212
0.165
0.232
0.154
K;
1.318
1.096
0.456
0.122
0.443
0.1 18
Kd
1.349
1.154
-0.087
-0.042
-0.123
-0.14
Table 10: Results for single task training with different label types. We show the results with Uni-Mol model on SIU that is splitter in to 10 folds based on sequence identity. The result is trained on 8 folds, validate 1 fold, and tested another fold.
RMSE
MAE
Pearson
Pearson"
Spearman
Spearman"
IC50
1.332
1.046
0.421
0.216
0.408
0.203
EC5o
1.322
1.123
0.494
0.227
0.520
0.228
K
1.545
1.281
0.356
0.143
0.355
0.133
K
1.678
1.369
0.474
0.149
0.347
0.153
F OTHER POTENTIAL APPLICATIONS OF THE SIU DATASET
The SIU dataset offers a wealth of opportunities for advancing drug discovery by addressing a wide range of applications. Beyond its core use in unbiased bioactivity prediction, the dataset is being extended to support pairwise ranking tasks, ensuring that comparisons are restricted to docking poses from the same PDB structure and bioactivity data with consistent label types. Additionally, SIU can be leveraged as virtual screening resource. Another promising application lies in training models for molecular optimization based on pairwise data, distinguishing molecules with high and low bioactivity. Furthermore, SIU serves as a supplementary dataset for docking pose prediction tasks, providing reliable resource. Finally, the automated pipeline demonstrated showcases the feasibility and scalability of generating large-scale, high-quality datasets. Future efforts could refine these methods to construct even larger datasets, enhancing the understanding of small molecule-proten interactions and accelerating progress in AIDD.
this work
21


Reviews:
[{"review_id": "EeMWXhcZrN", "reviewer": "Reviewer_nZBp", "summary": "The paper addresses significant issues in current evaluation strategies for bioactivity prediction, identifying problems with data preparation and evaluation metrics that result in inadequate benchmarking. For instance, approaches that only consider protein features in protein-small molecule interaction predictions may outperform methods that consider both due to overfitting. To tackle these challenges, the authors propose a new large dataset SIU based on mining wet lab bioactivity data and software-based docking to obtain structures, along with new evaluation metrics. The evaluations demonstrate the superiority of SIU over existing datasets and support the adoption of the new metrics.", "questions": "- Would a complementary analysis focused solely on small molecules, rather than only proteins, lead to the same conclusions? For example, could training a small-molecule-only baseline on PDBbind yield similar insights?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**. The paper presents original research with novel isights into the issues in bioactivity predictions and novel methodology for constructing bioactivity data for machine learning.\n\n**Quality**. The experiments are sound and convincing, clearly demonstrating the inadequacies of prior evaluation methods and the value of the new dataset and metrics.\n\n**Clarity**. The paper is easy to follow.\n\n**Significance**. The paper addresses a highly important issue of inadequate benchmarking in bioactivity prediction. The new SIU dataset is substantially larger than existing alternatives.", "weaknesses": "**Major Comments**\n\n- The availability of the dataset is not discussed. Will it be publicly accessible, and if so, in what form?\n- Data deduplication and diversity measurement methods are only briefly mentioned and not described anywhere in the text. Specifically, please elaborate on the ECFP and FLAPP methods used for deduplication of small molecules and pockets, respectively, and analyze the diversity and statistics after deduplication.\n- Although the paper proposes a large dataset, further analysis of the data splitting approach would be beneficial. The meanings of 0.6 and 0.9 non-homology levels are unclear (Does 0.6 signify 60% sequence identity upon alignment? How are sequences aligned?). Furthermore, is sequence similarity-based splitting sufficient with respect to data leakage? Recent work suggests that structure-based splitting may be a better choice [[1](https://arxiv.org/abs/2402.18396), [2](https://www.biorxiv.org/content/10.1101/2024.07.17.603955v1)].\n- The related work section is brief, only covering a few related datasets. Including related work on bioactivity prediction baselines to justify the selected methods for experiments and discussing prior evaluation metrics would strengthen the paper.\n- A table summarizing the statistics (for example, number of proteins, ligands, pairs, unique pockets, unique ligands, etc.) for the training and test sets of SIU and other datasets like PDBbind would be beneficial.\n\n**Minor Comments**\n\n- Abstract: please clarify what SIU stands for. From the abstract it is not clear that SIU is a dataset.\n- Figure 1: What is the GNN architecture?\n- Figure 1C: the meaning of the red line and the target are not clear.\n- Line 153: please specify the cutoffs being referenced.\n- Figure 2: Poses are visualized in the same way as ligands, which is confusing.\n- Line 209: “identified by PDB IDs” contradicts earlier text, which states that multiple pockets may arise from the same PDB ID.\n- Figure 3: clarify the meaning of RMSD—is it the RMSD between a docking pose and the ground truth from PDB, or between docking poses obtained with different methods to quantify consensus?\n- Section: “Structural data construction via multi-software docking”: Same as above, RMSD is used in two different senses, which is not always clear. \t\n- Section: “Structural data construction via multi-software docking”: Are the three chosen docking methods independent? Why were these specific methods selected?\n- Line 280: the claim “diverse small molecules” is unsupported by experiments.\n- Line 346: “for each target” is slightly confusing—are there only 10 targets in the dataset?\n- Line 392: clarify the statement, “We conducted non-homology analyses at two levels, 0.6 and 0.9”—does this imply a maximum of 60% and 90% sequence identity between training and test proteins?\n- Line 424: UniMol reference appears to be missing.\n- Tables 1 and 2: highlighting the highest values in bold would improve clarity.\n- Line 496: “pdb” should be in uppercase (PDB).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "umNtFX8Znl", "reviewer": "Reviewer_KtDx", "summary": "The paper has two main contributions. First, it presents a new large-scale dataset of small molecule-protein interactions for unbiased bioactivity prediction, named SIU. Second, the authors redefine evaluation metrics for bioactivity prediction by proposing to average correlation coefficients per protein before calculating the overal coefficients, and by stratifying performance evaluation across four key types of bioactivity labels. The benefits of the proposed dataset and metrics are demonstrated empirically through the evaluation of several standard models.", "questions": "- Could the authors clarify how the RMSE and MAE metrics are averaged? It would also be helpful to understand why RMSE* and MAE* are not defined using the same approach as Pearson* and Spearman* from Pearson and Spearman, respectively.\n- Could the authors explain why grouping by pockets is considered equivalent to grouping by PDB IDs? Is not it possible for a single protein to have multiple pockets?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Originality\n\nThe work’s originality lies primarily in its focus: rather than routinely proposing new models using existing datasets without analyzing their limitations, it critically examines the dataset and evaluation metrics themselves.\n\nQuality\n\nThe technical quality of the work is high. The methods employed are appropriate and well-justified.\n\nClarity\n\nThe paper includes a thorough introduction that provides clear context and a strong foundation for the study.\n\nSignificance\n\nThis work is highly significant for the machine learning community, as it introduces both a new dataset and an updated evaluation protocol for the important task of bioactivity prediction, while effectively demonstrating limitations in previous datasets and evaluation methods.", "weaknesses": "Major Concerns:\n\n- Data and Code Availability: A primary contribution of this work is the benchmark for bioactivity prediction, which includes a new dataset, evaluation metrics, and a specific data split. However, without readily accessible resources, the value of the work may be limited, as reproducing the benchmark independently could be challenging (e.g., requiring extensive runs across multiple docking software tools). It would be helpful if the authors addressed data availability, perhaps by providing a link to an anonymous GitHub repository.\n\n- Simulated Poses vs. Crystal Structures: The dataset consists exclusively of simulated poses rather than crystal structures. Since the dataset is described as high-quality, a discussion of the tradeoffs between experimental crystal structures and generated poses would add valuable context. For example, the dataset is constructed based on a consensus from two out of three docking tools. Additional information on whether this approach assures high quality—especially if one tool generates a different pose—could be beneficial. The authors might also consider presenting results from training on SIU and evaluating on PDBbind to explore any potential distribution shifts.\n\n- Sequence Identity-Based Data Splitting: The paper uses sequence identity-based data splitting; however, since the task is defined in terms of 3D structures, a structure-based split may offer a more appropriate approach, especially considering that similar structures may arise from different sequences.\n\n- Validation Fold Construction: Details on the construction of the validation fold would clarify this important aspect, as it affects both model training outcomes and the usability of the dataset in future work.\n\n- Test Fold Curation: While the manual curation of the test fold is appreciated, it would be helpful to understand the choice of focusing on only 10 protein targets, as this might affect the generalizability of the evaluation. Could the authors expand it to more proteins?\n\n- Redefining the Bioactivity Prediction Task: Since the paper redefines the bioactivity prediction task, a clearer description of the specific inputs and outputs considered would be helpful. For instance, it is unclear which ligand pose is used as input when three docking tools are involved. Additionally, Section 3.3, \"Reframing the Bioactivity Prediction Task,\" does not discuss the RMSE and MAE metrics used for evaluation.\n\n- Figure 3 Interpretation: Interpreting Figure 3 is challenging without a more precise definition of the evaluation using co-crystal poses from PDB complexes. The text does not clarify which PDB complexes were included and does not clearly define Success Ratio and Remaining Ratio.\n\nMinor Concerns:\n\n- The abstract does not define what SIU stands for.\n- The process for filtering based on extended-connectivity fingerprints could be elaborated. Is it based on pairwise Tanimoto similarities between fingerprints?\n- Line 330: The statement, \"Our structured approach facilitates nuanced assessments, such as evaluating the impact of specific small molecule modifications on protein interactions or comparing the efficacy of different compounds within the same protein pocket context,\" is not clear, especially as some filtering of ligands was involved based on extended-connectivity fingerprints.\n- Line 282: The acronym \"AIDD\" could be defined for clarity.\n- Line 346: It appears that Figure 4(D) should be replaced with Figure 4(B).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eMJ0lnPeSJ", "reviewer": "Reviewer_MuEF", "summary": "The paper proposes a new benchmark for training and testing of bioactivity prediction models, a critical task in computational biology. The curation of the training data involves filtering a set of more than one million bioactivities, cluster them into predictions around specific pockets and predict approximate structures for each of them. The curation of the test set the selection of a small subset of test complexes and the definition of the task as correlations within each target. Finally, the authors train a number of models on this new benchmark and highlight how the poor results on this assessment suggests more work needs to be done.", "questions": "1. Figure 3: how is success ratio defined?\n\n2. What type of assay does the test set contain? I assume it does not contain measurements across all different assay types. In this case how are spearman and person correlations computed for al assay types in the test set?\n\n3. How will the dataset be released? What format and what license?\n\n4. Is Pearson computed also on inactives? What proportion of the test and training set are inactive?\n\n5. Do I understand correctly that the test set only has 10 targets? \n\n6. In line 365 I assume the assay type is also used for the grouping, could the authors confirm?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I believe that the curation of a dataset for protein small molecule activity prediction is very important. Current datasets are too small and used in the incorrect way as the authors also represent. In this work, the authors do a very extensive and carefully thought out job towards building such dataset.", "weaknesses": "Building and proposing a new benchmark constitutes a significant responsibility: once the benchmark is released and, as I hope, the community adopts it, it will become very hard to correct any wrong choice made at this stage and these may lead to a lot of work being largely wasted (as I believe it is happening with current benchmarks for this task). Therefore, my bar is not just having an improvement over what is public but making sure that researchers in the field would not have concerns about adopting such a dataset.\n\nBased on this premise below I’m presenting a list of concerns that I would appreciate the authors responding to or addressing:\n\n1. The authors only retained datapoints for which the docking methods were consistent in their majority. While this appears sensible, it also creates a bias in the kind of molecules that the model considers (intuitively only “easier” complexes). It could be interesting to ablate the addition of non-consensus poses in the training set or in the test set for some of the baselines.\n\n2. The success proportion presented appears somewhat misleading. It is well known that redocking or holo-structure docking (that the authors use to validate the structure generation) is significantly easier than cross-docking (docking to the protein structure resolved when binding to a different protein, which the authors have to use when generating new structures). Therefore the experiments on the success should be performed on the cross docking task. This can be performed by taking from PDB structures of ligands bound to the same pocket.\n\n3. Saying that structures generated with the docking programs are of high quality seems a bit exaggerated or at least lacking experimental backing (see comment above on the difference between redocking and cross-docking). Multiple works in the molecular docking community have shown significant issues with the apo or cross docked performance of the methods applied for docking in this work.\n\n4. Assay types might be a bit misleading as the name of the “label type” as there are many different types of assays producing for example Kd values.\n\n5. The definition of different pockets from PDB IDs of co-crystal ligands and the subsequent deduplication are very unclear. While this seems a relatively non-important detail it becomes critical once the authors decide to use this assignment to compute correlation over.\n\n6. For protein ligand complexes from the bioactivity assays the way that these are assigned to a specific PDB ID / pocket is not clear to me. The available information is the Uniprot but then how does one go from having a specific Uniprot to deciding which of the PDB IDs corresponding to the different pockets to assign a complex to.\n\n7. Basing the benchmark of the output of the docking models e.g. by filtering out complexes with non-consensus poses and grouping by predicted pocket, will not allow fair comparisons for non-structure based methods or methods based on a different docking algorithm. I would recommend the authors remove such filtering steps for the test set to avoid the applicability of the benchmark being limited.\n\n8. Line 326 the authors say that the dataset is unbiased. Could they specify clearly what they mean by this term? Once again the claim does not seem fully justified.\n\n9. In paragraph 3.3, the authors discuss the value of dividing the prediction for different assay types. To motivate this they apply statistical significance test to the mean of the distribution of different assay types. However, it is unclear to me why this would be a good motivation for that. Differences in averages might just mean that these assays were applied to different types of proteins, while what the authors would need to show to motivate this point is that if they were applied to the same complex the results would be different depending on the assay. This work was actually performed by a recent paper [1] which however showed no very clear outcomes when it comes to the differentiation between different assay types. An alternative way to try to demonstrate this point would be to put all datapoints in the same “task” and then train and evaluate a single task model.\n\n10. I believe that saying that “In the traditional approach” or “conventional approach” people would look for pearson across different targets is misleading. This has been indeed unfortunately done recently in a number of ML papers, however, more established benchmarks in the field such as FEP+ do look at correlations computed per target.\n\nI would raise my score if a convincing response or revision were made on these points.\n\n[1] Combining IC50 or Ki Values from Different Sources Is a Source of Significant Noise. Landrum, Riniker.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yFlksy68p2", "reviewer": "Reviewer_6s1k", "summary": "The authors present a new dataset for bioactivity prediction together with a more fair metric for evaluation of ML models for bioactivity prediction. The authors present valuable insights regarding the weaknesses of current state of the art which convincingly motivates their work. The new carefully curated dataset together with the new metric tackles the identified weaknesses of current SOTA.", "questions": "1. The presented majority vote system over the 3 docking system is interesting even from the point of view of docking. What percentage of poses passed this filter? Did you notice some 2 of the 3 tools to tend to agree more with each other? It would be interesting to benchmark this method against the individual tools on some docking dataset (e.g. PDBBind) - for example take the performance of the 3 individual tools evaluated on PDBBind and compare it to the performance of the best of the 3 tools evaluated just over the datapoints which passed the majority vote filter. This benchmarking is not a strict requirement for the rebuttal period from me, but I would be really interested to see it.\n\n2. Could you please describe in more detail how were the models trained for the bioactivity prediction? E.g. the docking model UniMol is described in SuppMat to have an MLP on top of it. What is the input and what is the output of such model. Is the MLP trained to predict a single scalar value for a protein-ligand pair? Please explain.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Identification of the issues with current SOTA: overfitting to protein pockets due to small diversity of current datasets in terms of containing different small molecules docked to a single protein pocket; issues in reporting the overall correlation instead of computing the partial correlations per target and pooling the partial correlations which can help hide the problem of overfitting to protein pockets\n\n2. Introduction of a new large dataset for bioactivity prediction\n\n3. Careful curation of the dataset using a custom method of majority vore over different docking tools used for generation of the docked pose. Separation by different essay types is also useful.", "weaknesses": "1. There are few smaller unclarities in the text:\n- On line 174, I did not understand what it is meant by \"anti-logged\"\n- I did not find what method was used for generation of the small molecule conformers, it would be good to clarify this near the statement \"Initial 3D conformations for the small molecules were generated prior to docking.\"\n- Figure 4 is discussed in the text, but the caption could use more detailed description. Just by looking at the figure it is not clear what we are looking at.\n- The statement “We conducted non-homology analyses at two levels, 0.6 and 0.9,” is not clear. Did the authors check for sequence identity less than 60/90 pct. between test and train set? Or the complement of 40/10 pct.? Please clarify.\n\n2. No mention of the Papyrus dataset ( https://jcheminf.biomedcentral.com/articles/10.1186/s13321-022-00672-x ). It should be discussed in related work and authors should compare SIU to Papyrus in all relevant aspects, such as size, data origin and level of curation.\n\n3. Detailed, practical description of the dataset, which would facilitate its usage by developers of ML models is missing. I suggest putting to supplementary material more detailed and practical description of what is provided in the dataset. PDB files containing the structure of the protein-ligand complexes as docked by some of the docking software? And the labels are just a single scalar value per datapoint? Please explain in detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)