Rethinking the generalization of drug target affinity prediction algorithms via similarity aware evaluation
Authors: Chenbin Zhang
Zhiqiang Hu
Jiang Chuchu
Wen Chen
JIE XU
Shaoting Zhang
Venue: ICLR 2025 Oral (2025)
Keywords: Drug-Target Affinity Prediction
Similarity-Aware Evaluation
Primary area: datasets and benchmarks

Abstract: Drug-target binding affinity prediction is a fundamental task for drug discovery. It has been extensively explored in literature and promising results are reported. However, in this paper, we demonstrate that the results may be misleading and cannot be well generalized to real practice. The core observation is that the canonical randomized split of a test set in conventional evaluation leaves the test set dominated by samples with high similarity to the training set. The performance of models is severely degraded on samples with lower similarity to the training set but the drawback is highly overlooked in current evaluation. As a result, the performance can hardly be trusted when the model meets low-similarity samples in real practice. To address this problem, we propose a framework of similarity aware evaluation in which a novel split methodology is proposed to adapt to any desired distribution. This is achieved by a formulation of optimization problems which are approximately and efficiently solved by gradient descent. We perform extensive experiments across five representative methods in four datasets for two typical target evaluations and compare them with various counterpart methods. Results demonstrate that the proposed split methodology can significantly better fit desired distributions and guide the development of models.

Published as a conference paper at ICLR 2025
RETHINKING THE GENERALIZATION OF DRUG TARGET AFFINITY PREDICTION ALGORITHMS VIA SIMILARITY AWARE EVALUATION
Chenbin Z Zhiqiang Hu Chuchu Jiang" Wen Chen', Jie Xu², Shaoting Zhang
MoleculeMindd PPekigg University, $Shanghai Al L Laboratory, "Sense Research
cheenblncchanggmmoeeeullemei com, huzqlpku. edu. cn, (jiangchuchu, xujie, zhangshaotingg org. en, chenven@sensetingg imeeenttttiteet......... com
ABSTRACT
Drug-target binding affinity prediction is a fundamental task for drug discovery. It has been extensively explored in literature and promising results are reported. However, in this paper, we demonstrate that the results may be misleading and cannot be well generalized to real practice. The core observation is that the canon- ical randomized split of dominated by samples with high similarity to the training set. The performance of models is severely degraded on samples with lower similarity the training set but the drawback is highly overlooked in current evaluation. As a result, the performance can hardly be trusted when the model meets low-similarity samples in real practice. To address this problem, we propose a framework of similarity aware evaluation in which a novel split methodology is proposed to adapt to any desired distribution. This is achieved by formulation of optimization problems which are approximately and efficiently solved by gradient descent. We perform extensive experiments across five representative methods in four datasets for two typical target evaluations and compare them with various counterpart methods. significantly better
set in conventional evaluation leaves the test set
Results demonstrate that the proposed split methodology desired distributions and guide the development of models. Code is released at attps://githun htw.s/hwww....ooooooooeeeeeeeeee
INTRODUCTION
Drug-target binding affinity (DTA) prediction a fundamental and crucial task for drug discovery. It evaluates the effectiveness of drug candidates, samples, and sees its application in a large- scale virtual screening where most ineffective candidates are filtered to save experimental cost and time (Chatterjee et al.] 2023). DTA is quantitatively measured by inhibition constant Ki, half maximal inhibitory concentration IC50, etc., which are all real-valued (Monteiro et al. 2022). The prediction performance is commonly evaluated by mean absolute error (MAE) and coefficient of determination (R²).
The task of DTA prediction has been extensively studied for decades (Chen al. 2018 Askr al. 2023). Related works can be categorized as structure-based sequence-based and similarity-basee (Wu et al. 2018 Chuang al. 2020). Structure-basse methods rely 3D structures of samples, target proteins, or their complexes. Although theoretically accessible to most comprehensive infor- mation following the dogma "structure determines function", structure-based methods are limited by available 3D structures, especially experimentally verified structures, and also hindered in prac- tice by poor time efficiency. In contrast, sequence-based and similarity-based methods are fast and do not set 3D structures as prerequisite (Xu et al. 2017 Zhang et al. (022). Instead, they take
"Equal contribution. Corresponding Author.

---

Published as a conference paper at ICLR 2025
as imput residual sequences, Simplified Molecular-Inpul Line-Entry System (SMILES) sequences, fingerprint sequences, atom-bond graphs, or the derived pairwise similarities, which are easier to acquire with lower cost. Moreover, these sequences and similarities are readily processed by di- versified sophisticated backbones including convolutional neural networks (CNNs) (Özüürk et al.,
2018
Li et a 2019 [uu et al. 2023), recurrent neural networks (RNNs) (Karimi et al. 2019] Yuan
et al.
2022), graph neural networks (GNNs) (Nguyen et al. [2021, Yang et al.) 2022
Tang et al. Song al.
2022
Wang et al. 2022) and transformers (Chithrananda et al. 2020 Zhao et 2022
2023
Jiang et al. (023), and enjoy the beneñits of the development of deep learning techniques. As
a result, sequence-basee based and similarity- based methods are shown to reach new high performance
and are drawing increasing attention.
Although promising results are reported, we find, surprisingly, that these results may be misleading- Take the task of IC50 prediction for target EGFR as an example, as shown in Figure T we evaluate five state-of-the-ant and representative methods and the best-performing one, SAM--DTA (Hu al.] 2023), achieves a MAE of 0.6012 and R² of 0.6505 for the whole test set. However, if we dive into the performance and divide the test set according to the similarity of the sample to the training set, we find clear performance degradation for low-similarity samples: the MAE deteriorates to 1.2970 for samples with similarity less than 1/3 and R² to -0.6385. The gap is huge. Nevertheless, poor performance on low-similarity samples does affect the whole performance since they only occupy a negligible proportion: only 16 samples with similarity less than 1/3 out of a total of 873 samples in the test set (Figure []]. In other words, the test set is dominated by high-similarity samples and performance for low- low-similarity samples is overwhelmed in current evaluations. We will show that the phenomenon exists across different similarity measures, performance metrics, datasets, and methods, and therefore it is general. Consequently, the evaluation will be misleading to practitioners, especially when the trained model meets low-similarity samples when used in real practice.
We argue that the core of the problem lies in the canonical randomized split of the test set. Ran- domized split follows the assumption of independent identical distribution (I.ID.),, which is the foundation of most statistical learning theories. However, in drug discovery samples are not neces- sarily independent of cach other: in practice. mutually similar variants are more likely to be tested together in high-throur thehuuhhuu experiments, while at the same time, they have to avoid high similarity to approved drugs for intellectual property issues (Harren et al. 2024). Empirically, drugs devel- oped at different times show significant distinction in their properties (Sheridan et al. 2022). As result, practitioners would not always expect the samples they are going to test to follow the same distribution as historical samples. This in turn raises request model development that the test set should satisfy a desired distribution. For example, one may need a test set that is uniform at differ- ent similarity bins; others may ask the test set samples be all limited within predefined similarity bounds, and so on (Li & Yang 2017 Simm et al. 2021 Luo et al. 2024 Tricarico et al. 2024).
We formulate the test set split with desired distribution as a combination optimization problem. The problem is infeasible to solve for optimum due to efficiency issues. In this work, we address this challenge by relaxing it to continuous optimization problem where samples are allowed to coexist in the training set and test set with "probability" or weight. Further, the objective function contains non-differentiable operations including taking the maximum and counting in similarity bins, and are approximated in this work by differentiable counterparts. We will show that the degree of approxi- mation is adjustable by introduced hyper-parameters. Next, the resulting optimization problem has no closed-form solution, and hence we have resorted to Lagrangian multipliers with numerical method implemented by PyTorch and Cooper (Gallego- GGallego-Posada & Ramirez 2022). Finally, we an- solution and find the non-negligible approximation error induced by the relaxation. To this end, we introduce a regularization term that penalizes samples whose weight is far from bipartition. We refer to our strategy as Similarity A Aware Evaluation, abbreviated SAE. By doing all this, we have managed to achieve test set splits with various desired distributions.
alyze the continuously-vvaiee
Extensive experiments are performed substantiate the effectiveness of our split strategy. To begin with, our split strategy can achieve a uniformly distributed test set across various similarity bins (Figure M.. Subsequently, we evaluate the performance of five DTA prediction methods on this test set. The results underscore distinct relationship between the performance and the corresponding similarity levels, suggesting a more comprehensive assessment of balanced split across varied meth- ods in comparison to the strategy of randomized split. Moreover, in scenarios where the samples practitioners intend to test deviate from the distribution of existing samples, our split strategy can effectively split the training set and internal test set based the similarity distribution of the test
2

---

Published as a conference paper at ICLR 2025
Evaiuation IC:50 EGFR by randomized split
Evaliation IC:50 by (balanced) split.
Pereentage of sample in each bins randomized split
1,5
1.51
12
1.2
13
111
(33, 2/3
0.7
0.7
(2/31]
06
0.6
85
0.5
Ovenil 22)
0.1
0.0
MoKLR
PhenstiO7
SAM-U74
0.0
CheedEar
M
SAMADA
(25,
(18, 293
0,8
0.5
Parcentage sample by SAE (balanced) split
0.7
0.6
0.4
0.0
33.0%
03
0.3
02
02
33.0%5
-0
(1/3, 2/1)
02
02
03
0.3
044
(18) 213)
FoainDDA
MokLE
Phand
SAM-DTTA
ChouBRR3
MoCLE
PhamoGT
SAM-DTA
(2%,
Figure 1: Comparison of randomized split and SAE (balanced) split at ICSO for EGFR. The random- ized split led 88% of test samples yielding high similarity (> 2/3) the training set. In contrast, our SAE (balanced) split strategy ensures more balanced distribution similarities. The evalu- ation of five DDA prediction methods demonstrates that the performance aligns with the similarity levels. In the randomized split, the overall performance closely resembles that of high-similarity samples, thus failing to evaluate the performance when encountering low-similarity samples.
set (Figure 4). Here we conduct hyper-parameter searches on the internal test set across five DTA prediction methods to assess the efficacy of our split strategy. Compared to previous split strategies, our split strategy facilitates the selection of optimal syper-parameters, enhancing performance on the external test set (Figure 5). In other scenarios where practitioners specify predefined similarity constraints for the test set samples, such as a maximum similarity limit 0.4 or 0.6, or even a range bounded by minimum and maximum similarity of 0.4 and 0.6, our split strategy ensures the majority samples in the test set adhering to these requirements (Figure(6) 6).
PROBLEM OF RANDOMIZED SPLIT
In this section, we will show that the imbalanced distribution of samples and the consequent over- shadowing of low-similarity samples is general for randomized split of test set across differentt similarity measures, performance metries, datasets, and methods. We will first give the details the example in Figure and then explore possible variants.
In the example demonstrated by Figure we take the IC50 dataset of target EGFR with a total of 4,361 samples, which is one of the largest datasets from the BindingDB database, and IC50 has been converted to its negative logarithm form, pTC50 = - log TC50 (Molar) following the convention. Then, randomly split the dataset into a training set and test set with a ratio 8:2. Subsequently, we train and validate the DTAA prediction models on the training set and evaluate their performance on the test set. should be noted that follow the original hyper-parameter tuning procedure in the source code of each DTA prediction method.
able to find. The dataset has originated
To perform the fine-grained evaluation with respect the similarity, first define the pairwise similarity for sample and Ø2,
PuirmiseSSimlarill(ee... = Similarily (Fealure(xx). Fealure( (1)
and then derive the similarity the union of the training set by aggregation, for sample x: E TestSet.
3

---

Published as a conference paper at ICLR 2025
Table 1: Variations of Similority TrainingSet related feature extraction, similarity measure, aggregation functions, and performance metrics. We choose PharmHGT and SAM-DTA as the example methods for the detailed showcase.
Randomized Split (MAE)
Feature: RDKit fingerprint
Feature: Avalon fingerprint
Bin
Count (Ratio)
PharmHGT SAM-DTA
Count (Ratio)
PharmHGT SAM-DTA
[0 1/3]
8 (0.0092)
1.7551
1.6787
0 (0,0000)
(1/3, 2/3]
34 (0.0389)
1.3214
1.0040
28 (0.0321)
1.4646
1.3319
(2/3, I]
831 (0.9519)
0.6015
0.5743
845 (0.9679)
0.6128
0.5770
overall
873 (1.0000)
0.6401
0.6012
873 (1.0000)
0.6401
0.6012
SimalarityMeasure: Sokal similarity
SimilarityMasuure: Dice coefficient
[0 1/3]
33 (0.0378)
1.5051
1.2444
0 (0.0000)
(1/3, 2/3]
398 (0.4559)
0.7066
0.6619
33 (0.0378)
1.5051
1.2444
(2/3, 1]
442 (0.5063)
0.5157
0.4985
840 (0.9622)
0.6061
0.5759
overall
873 (1.0000)
0.6401
0.6012
873 (1.0000)
0.6401
0.6012
Aggregation: Top-3
Aggregation: Top-5
[0 1/3]
17 (0.0195)
1.5188
1.3149
24 (0.0275)
1.5627
1.3469
(1/3, 2/3]
171 (0.1959)
0.8890
0.7748
240 (0.2749)
0.8014
0.7228
(2/3, 1]
685 (0.7847)
0.5562
0.5401
609 (0.6976)
0.5402
0.5239
overall
873 (1.0000)
0.6401
0.6012
873 (1.0000)
0.6401
0.6012
Randomized Split (R²)
Feature: RDKit fingerprint
Feature: Avalon fingerprint
Bin
Count (Ratio)
PharmHGT SAM-DTA
Count (Ratio) PharmHGT SAM-DTA
[0 1/3]
8 (0.0092)
0.1555
0.2579
0 (0.0000)
(1/3, 2/3]
(0.0389)
-0.0371
0.3529
28 (0.0321)
0.1028
0.2921
(2/3. I]
831 (0.9519)
0.6327
0,6706
845 (0.9679)
0.6169
0.6672
overall
873 (1.0000)
0.5928
0.6505
873 (1.0000)
0.5928
0.6505
SimalarityMeasure: Sokal similarity
SimalarityMeauure: Dice coefficient
[0 1/3]
33 (0.0378)
-0.1562
0.1866
0 (0.0000)
(1/3, 2/3]
398 (0.4559)
0.5412
0.6057
33 (0.0378)
-0.1562
0.1866
(2/3. 1]
442 (0.5063)
0.6942
0.7156
840 (0.9622)
0.6290
0.6711
overall
873 (1.0000)
0.5928
0.6505
873 (1.0000)
0.5928
0.6505
Aggregation: Top-3
Aggregation: Top-5
[0 1/3]
17 (0.0195)
-0.6280
-0.3148
24 (0.0275)
-0.2379
0,0891
(1/3, 2/3]
171 (0.1959)
0.4283
0.5743
240 (0.2749)
0.4756
0.5794
(2/3, 1]
685 (0.7847)
0.6483
0.6712
609 (0.6976)
0.6584
0.6804
overall
873 (1.0000)
0.5928
0.6505
873 (1.0000)
0.5928
0.6505
MollaarityToTTiiinnngSeete = Aggregation PairviseSSittrrittt.. TroiningSet
(2)
where in the example of Figure we set Feature the Morgan fingerprint and SimslarityMMasurr as the Tanimoto coefficient, which are both commonly used to measure the similarity of samples, and we set Aggregation as the maximum function (Bajusz et al. 2015 Ying et al., 2021).
Next, we compare other variants for these functions. For the feature extractor Feature, compare other widely used molecular descriptors including Avalon fingerprint and RDKit fingerprint (a.k.a. topological fingerprint): for the function SimalarttyMMesure coefficient, which are both symmetric for its parameters; and finally for the Aggregation function, special
compare Sokal similarity and Dice
compare the general top-k averaging where the maximum function can be

---

Published as a conference paper at ICLR 2025
Table 2: Comparison randomized split and SAE (balanced) split at IC50 for BACEI, Ki for Carbonic anhydrase and Carbonic anhydrase II. We choose PharmHGT and SAM-DTA as the example methods for the detailed showcase.
IC50 for Target BACEI (MAE)
Randomized Split
SAE (balanced) Split
Bin
Count (Ratio)
PharmHGT SAM-DTA
Count (Ratio)
PharmHGT SAM-DTA
[0 1/3]
10 (0.0108)
1,3743
1.1204
309 (0.3330)
1.1397
1.0309
(1/3, 2/3]
67 (0.0722)
0.6334
0.6928
311 (0.3351)
0.6410
0.6693
(2/3, I]
851 (0.9170)
0.4611
0,4594
308 (0.3319)
0.4747
0,4808
overall
928 (1.0000)
0.4834
0.4834
928 (1.0000)
0.7518
0.7272
IC50 for Target BACEI (R²)
[0 1/3]
10 (0.0108)
-0.0553
0.3702
309 (0.3330)
-0.2983
-0.1261
(1/3, 2/3]
67 (0.0722)
0.6789
0.6439
311 (0.3351)
0.5848
0.5637
(2/3, 1]
851 (0.9170)
0.7113
0.7150
308 (0.3319)
0.7797
0.7803
overall
928 (1.0000)
0.7190
0.7256
928 (1.0000)
0.5329
0.5665
for Target Carbonic anhydrase (MAE)
Randomized Split
SAE (balanced) Split
Bin
Count (Ratio)
PharmHGT SAM-DTA
Count (Ratio)
PharmHGT SAM-DTA
[0 1/3]
7 (0.0079)
1.1467
0.8798
264 (0.2983)
0.8410
0.8729
(1/3, 2/3]
205 (0.2316)
0.5843
0.6605
311 (0.3514)
0.6706
0.6877
(2/3. 1]
673 (0.7605)
0.4986
0.4896
310 (0.3503)
0.6039
0.5740
overall
885 (1.0000)
0.5236
0,5323
885 (1.0000)
0.6981
0.7031
Ki for Target Carbonic anhydrase I (R2)
[0 1/3]
7 (0.0079)
-0.3232
0.1308
264 (0.2983)
-0.0389
-0,0282
(1/3, 2/3]
205 (0.2316)
0.5733
0.4820
311 (0.3514)
0.3642
0.3478
(2/3. 1]
673 (0.7605)
0.5037
0.5270
310 (0.3503)
0.3917
0.4262
overall
885 (1.0000)
0.5257
0,5174
885 (1.0000)
0.2994
0.3071
Ki for Target Carbonic anhydrase II (MAE)
Randomized Split
SAE (balanced) Split
Bin
Count (Ratio)
PharmHGT SAM-DTA
Count (Ratio) PharmHGT SAM-DTA
[0 1/3]
8 (0.0087)
0.5879
0.6645
244 (0.2667)
1.0450
1.0564
(1/3, 2/3]
201 (0.2197)
0.6807
0,7009
342 (0.3738)
0.7389
0.7572
(2/3.
706 (0.7716)
0.5615
0.5426
329 (0.3596)
0.6172
0.5813
overall
915 (1.0000)
0.5879
0.5785
915 (1.0000)
0.7768
0.7738
Ki for Target Carbonic anhydrase II (R?)
[0 1/3]
8 (0.0087)
0.6739
0.4277
244 (0.2667)
-0.0885
0.0087
(1/3, 2/3]
201 (0.2197)
0.5803
0.5742
342 (0.3738)
0.4657
0,4690
(2/3.
706 (0.7716)
0.5509
0.5932
329 (0.3596)
0.4760
0.5346
overall
915 (1.0000)
0.5684
0.5938
915 (1.0000)
0.3776
0.4192
of = Note that averaging or taking the median over the whole training set is not suitable. This because the majority of samples in the training set have low similarity to specific sample, and averaging or taking the median over the whole training set is not able to tell whether there exists any high-similarity ones. The results are shown in Table Here choose example methods
for detailed showcase (PharmHTT (Jiang et al. (023) and SAM-DTA (Hu et al. 2023)) )) while the results of other methods be found in the appendix.
For the prediction method, as shown in Figure I we select five thate-rf-the-art and representative DTA prediction methods. Molecular Contrastive Learning of Representations (MoICLR) sam-
5

---

Published as a conference paper at ICLR 2025
ples as atom-bond graphs and employs GCN and GIN to learn the molecular representations by contrastive pairs (Wang et al. 2022). Sequence-agnostic model for drug-target binding affinity pre- diction (SAM-DTA), on contrast, takes as input the Simplified Molecular- Input Line-Entry System (SMILES of samples and processes it using 1D-CNN with dilated parallel residual blocks (Hu et al.) 2023). SMILES is also utilized in FusionDTTA but is processed by unified LSTM model with lin- car attention mechanism (Yuan et al., 2022). Finally, we include two transformer-basee methods. One is ChemBERTa which takes as input SMILES of samples and builds a model with 12 attention heads and 6 layers (Chithrananda et al. 2020). The other is PharmHGT which leverages a unique caaammaccphhoicccoossttiii
heterogeneous molecule graph and two various transformers to extract chemical properties and predict molecular attributes (Jiang et al. 2023).
We also investigate the problem with other tasks and datasets. Specifically, for the task of IC50 pre- diction, we also perform experiments in the dataset of target BACHI with total of 4,636 samples, and we further extend the experiments to the task of Ki prediction for targets Carbonic anhydrase I and Carbonic anhydrase II, with 5,307 and 5,487 samples respectively. For all of these datasets, we apply the same preprocessing as that of target EGFR, except that taking the negative logarithm form not applicable to Ki datasets. The results are collectively presented in Table 2 Here choose PharmHTT and SAM-DTA as the example methods for a detailed showcase, while the comprehen- sive collection of results can be found in the appendix.
In summary, extensive experiments demonstrate the generality of the imbalanced distribution of samples caused by randomized split and the consequent overshadowing of low-similarity samples. This problem will be analyzed and addressed in the following section.
SIMILARITY AWARE EVALUATION
In this section, we will elaborate on the proposed Similarity Aware Evaluation (SAE) which aims at obtaining a test set with the desired distribution. We will exemplify the method for a test set that uniform at similarity-based bins (see Figure [| for 3 similarity-basee bins), and then extend it to other desired distributions.
The split for test set that is uniform across similarity-based bins can be formulated natorial optimization problem as follows. Given a dataset X = i = 1, .... .N. a pairwise similarity matrix {sky [0, 1]. su 0, i 1,2, .... N:j 1, 2, ...,... N]. a ratio a, and K bins with boundaries (br, k = 0, 1,2 .... K}. find subset (test set) X C X, Xxs = a.,, such that
combi-
K (ok aN/K)²
f(Xx)) = Σ
(3)
aN/K
is minimized, where
0k Hrr E X : -k- < <r; S bb))
(4)
72 max 8ij 23EX
(5)
X X Xts
(6)
X1r denotes the training set, the similarity of x; to the training set, and o the count for each of the K bins. Note that the objective function f is essentially the x² statistics in the Chi-Square (x2) Test, where oş is the observed count in each bin and aN/K K is expected. Note also that we specially set 84 = 0 in the pairwise similarity matrix. This has no effect on the problem itself but can avoid that r; falls trivially to 1 due to the maximum operation for the relaxed problem below.
The combination optimization problem infeasible to solve for optimum. As a result, we relax to continuous optimization problem where samples are allowed to coexist in the training set and test set by the introduction of the weights {w, E [0, 1]. i = 1,2, ..., 2,..... N} and by |X,sl = aN we have constraints E; w; a... Next, we have to deal with non-differentiable operations in the objective function if including taking the maximum and counting in similarity-base bins. For the maximum function in the calculation of ri, we approximate it by the LogSumExp operation with hyper-parameter B,
= max Sij = max(1 - w5)sis ~ 3 Ex
log Σ exp (8(1 - wg w))881)
j
6

---

Published as a conference paper at ICLR 2025
12
151
Z
2
40
15
[a] Mar
(6) 8
[e)
(d) 100
Figure 2: Impact of the hyper-parameter 3 on the approximation the maximum function in Eq. To illustrate this impact, consider simplified scenario involving only two random variables: X and Y. (a) Z Max(X, X, Y); (b-d) Z = 1/3 (og((op((22)) (x)) + (x))) A larger value of 3 results accurate approximation, with 3 100 yielding excellent result.
1.0
1.0
1.0
1.
0.8
988
y0.6
y0.6
0.4
0.4
0.6
0.4
0.
0.0
00
00
(4) lfj < S
(8) a
[e] 0.1
(d) a 0.01
Figure 3: Influence of the hyper-parameter a in Eq.
we analyze a specific case where k/3,Ck (2k: 1)/6. (a) y T(bi < r S b2); (b-d) y
K 3, bk
exp ((- (203)) /E exp (-(r- - c*) /220))))
decrease in the value leads to more precise estimation, with o 0.01 producing outstanding result.
In terms counting for similarity- event of a sample falling into specific bin by continuous score which depends on how far r; of the sample deviates from the center of the bin. The score function is defined following the bell-shaped normal distribution with the center of the bin as the expectation and tunable standard deviation. For sample, scores across all bins are normalized. Specifically, denote Gj = (bx-1 + bx)/2 as the center of each bin, d% the tunable standard deviation, we have:
bins the calculation of oje, we approximate the discrete
Ok = 1ar E Xts 0k-1 < S bb
(8)
Σ
Wi w;T[ba-1 <ri bk)
(9)
exp (-(r; (20))
Σ Wi
V2noi
(10)
/2k0, exp (-(ri ck' )2 (2/()))))
where I is the indicator function. In this paper, we set dg = o, = 1,2,... K. Thus, we obtain the following expression:
Ok Σ wi E exp (-(r; Cx
exp
Σ wi softmax
(r: (ri-c)) cx)²
(1I)
)/ /20=))
20
Figure and Figure 3 illustrate the error induced by these two differentiable approximations, with respect to hyper-paramete (3 and a, respectively. In Figure 2 we compare between values of 1, 10, and 100 and plot the surface for a special case of maximum over two variables. It can be seen that a larger achieves a better approximation, but is also prone to overflow in practice. We use 100 throughout the paper. For Figure 3 on the other hand, show the comparison of values between 1, 0.1, and 0.01 for an example case of the indicator of the second bin for 3-bin setting by k/3. The degree of approximation gets better when the value of decreases, and is pretty well when ø = 0.01. For the sake of flexibility, we set 0 = 0.1(b. bx-1) in rest of the paper.
At the moment in practice, we find that the approximation enror induced by relaxing w; from {0, 1} to [o, 1) is not negligible. In fact, considerable proportion of w; solved is neither near 0 nor 1. To address this
to be ready arrive at the approximated optimization function. However,

---

Published as a conference paper at ICLR 2025
issue, are inspired by the concept of entropy, and propose to add regularization term, Ieg Σ (wy log(w;) + (1 ui)) log(1 w;))
(12)
where À is hyper-paraee parameter that balances between the objective function and the regularization term. Finally, we have the optimization problem,
(o aN/KY²
mènimize we
+ Lrgg
(13)
aN/K
subject to
Σ Wi a.
(14)
0 S wy
1, = 1,2,
(15)
where
(- (Ti [-(ii-a 22
Og Σ wi softmax
(16)
20²
k
- log Σ exp (B(1-wy)8) wy)s8a)
(17)
Tog Σ (w log(w:) + (1 ws) log(1 wi))
(18)
Note that the optimization problem has no closed- form solution, and hence we have resorted to Lagrangian multipliers with numerical method implemented by PyTorch and Cooper.
For other desired distributions, one can modify the objective function f in straightforward way while the approximation tricks and regularization term can be retained, and the resulting optimiza- tion function can also be solved by Lagrangian multipliers with numerical method. Generally, if the expected count in each bin Eg, k = 1,2, ..., K, the objective function can be readily modified as
(ok
)2
+ beg
(19)
Ek
4 EXPERIMENTS
4.1 BALANCED SPLIT
In Section demonstrated that within the context of the randomized split, suboptimal perfor- mance on low-similarity samples does not significantly impact the overall performance, as they only occupy negligible proportion. To avoid disregarding samples with low similarity, we implemented a "balanced split" using similarity aware split strategy to achieve a uniformly distributed test set across various similarity bins ([1/3, 2/3), (1/3, (2/3, 1).. Figure [ shows comparison of ran- domized split and SAE (balanced) split at IC50 for EGFR. The randomized split strategy yielded a case in which 88% of test samples have high similarity (> 2/3) to the training set, while our split strategy guarantees a more evenly distributed range of similarities. The evaluation at the SAE (bal- anced) split reveals that the performance of each model aligns with the respective similarity levels. Hence, SAE (balanced) split provides more accurate representation of the performance.
Additional results at other tasks and datasets are delineated Table 2 Given the space constraint, we provide experimental results of two representative DTA prediction methods. Notably, analo- gous phenomena are observed across the remaining three datasets. The comprehensive collection of results can be found in the appendix.
4.2
MIMIC SPLIT
When prior knowledge about the external dataset- such as the distribution similarity to existing samples-2n available, proach enables optimal hyperparameter selection for the DTA prediction method, thereby enhancing performance on the external dataset.
construct an internal test set that mirrors this distribution. This ap-
8

---

Published as a conference paper at ICLR 2025
0.4
0.4
Ponabage
0.3-
0.3
03-
0.3
92-
0.2
0.2
0.-
9.0
0.0
0.
0.
(0) Extemal Test Set
(b) Random
(c) Scaffold
(d) SIMPD
(c) SAE (mimic)
Figure 4: The similarity distribution of the internal test set across different split strategies. (b) Randomized split leads to a scenario where most internal test samples are highly similar to the training set. (c) Scaffold split produces a more balanced distribution. (d) SIMPD split yields distribution similar to the random split. (e) Our SAE (mimic) split brings the internal test set's distribution closest to that of the external test set.
PisiaaDTA
ChenBERTa
MACLR
PhanEGGT
SAM-DIA
0.8
0.6
04
Rembog
Stamfad
SIMPO
SAF
(mbmi))
Radino
Reaffed
SRMP
sapp (minlo))
Randaa
Shantad
STARP
Randew
Shaffed
BAF (manic)
Rendroo
SIMPP
(ONIS)))
2G Ferral Trat
Park
0.7
0.6-
PicterITTA
ChrmBRRTh
MrCLR
PhamElGT
SAM-DTA
0.4
0.2
0.-
Ranbon
SBMPO
Ste
(mina))
Radoop
Rostood
spuro
Sa.p (mipi))
Randon
Bradtid
SMP
348
(mikni)
Randon
Smaffedd
SDMRD
SAF
(mini))
Randem
Bratfed
SMPP
(mmni)
Figure 5: Comparison of the generalization ability of different split strategies at ICSO for EGFR across five DTA prediction methods. The external test set performance of the mimic split surpasses that other split strategies.
We conducted experiments the task of predicting IC50 values using the EGFR target dataset. Specifically, 70.2% samples were procured from ChEMBL (Zdrazil et al. 2024). while the remain- ing samples were obtained from PubChem (Kim et al. 2023), the US Patent and the scientific lit- erature available in BindingDB (Gilson et al.) 2016).. For our analysis, we classified the ChEMBL- derived samples as internal data, while those obtained from the other sources as the external test set. We first computed the similarity distribution between the external test set and the internal data, as shown in Figure (a). Subsequently, we employed the Randomized split, Scaffold split, and SIMPD split (Landrum et al. 2023) to split the internal data into training set and an internal test set with ratio of 70% and 30%. The similarity distributions between the internal test set and the training set for these splits are depicted in Figure
respectively. Finally, we utilized the proposed strategy to split the internal data, thereby mimicking the similarity distribution observed in the external test set. The results are illustrated in Figure 4 (e). We refer to this split strategy as "mimic split".
We searched for hyper-parameters. Details of the hyper-parameter for each method are provided in the appendix. Experimental results are shown in Figure 5 our SAE (mimic) split strategy consistently yields op- timal
such as optimizer, learning rate, batch size, and other relevant
sets for all five DTA prediction methods. Among the various split strategies, the scores of our SAE (mimic) split on the internal test set are the most closely aligned with those on the external test set.

---

Published as a conference paper at ICLR 2025
Presaing
0.3
837
10.3
(82)
103 (64
18
68
10
(63
(a) Test data distribution with 0-0.4 split
(b) Test data distribution with 0-0.6 split
(c) Test data distribution with 0.4-0.6 split
Figure 6: Other applications of our split strategy on the IC50 dataset of target EGFR. (a) 86% of the test samples satisfied the desired distribution with maximum similarity of 0.4, (b) 88% of the test samples met the criteria for a maximum similarity of 0.6, and (c) 77.7% of the test samples fulfilled the requirements for a similarity range between 0.4 and 0.6.
4.3 OTHER APPLICATIONS
Beyond achieving balanced splits, our strategy supports distributions with maximum similarities of 0..4 or 0.6 or a range between 0.4 and 0.6. In a 7:3 train-test split on the EGFR target's IC50 dataset (Figure 6). SAE ensured 86%, 88%, and 77.7% of test samples met the criteria, respectively. This underscores the flexibility of the strategy in accommodating diverse split needs. Moreover, since SAE can flexibly achieve desired distributions by capturing the similarity between pairs of data samples, it can also be applied to QSAR scenarios, including ADMET prediction, drug de- sign (De et al. [022 Tropsha et al.) 2024), as well as the prediction of protein-protein interactions (PPI) (Sharma & Bhatia 2021) } and drug-drug interactions (DDI) (Dmitriev et al. 2019).
RELATED WORKS
When evaluating machine learning methods, it is essential to set aside a test set for benchmark- ing (Wu et al.] 20118). The similarity between the training set and the test set significantly influences the performance of these methods (Sheridan et al. [2004] Cherkasov et al.] (2014) Pahikkala et al.) 2015 Sieg et al., 2019 Nguyen et al.] 2022 Atas Guvenilir & Dogan| 2023 Harren et al. 2024). However, in the field of chemical data, imbalanced data distributions are inherent and unavoid- able challenge (Harren et al. 2024 Yang et al.) 2020; Tossou et al. 2024). Therefore, it is crucial to design dataset split strategies that account for these imbalances and ensure meaningful evaluation of model performance (Li & Yang. 2017; Sheridan et al.) 2022). The commonly used random splitting method may fail meet the requirements due to inherent data bias. A typical solution is exclude
all samples in the training set that are similar to those in the test set (Li et al. 2021 Scantlebury
al. 2023, Luo al., 2024 2017 Wan al., 2019; Atas Guvenilir & Dogan (023) Recently,
several advanced split strategies have been proposed, including scaffold split (Bemis & Murcko, 1996 Fang et al., 20222 Zhou et al., (2023 Liu et al.) (2024), time split (Guan et al.) [023] Stärk al,
2022 stratified split (Wu et al. [2088] Chen et al.) 2022). physicochemical properties- based split (Kalemati al. 2024), cold-drug split (Huang et al. 2021), and SIMPD (Landrum et al.) 2023). among others.
6 CONCLUSION
In this paper, we show the generality of the imbalanced distribution of samples by randomized split and the consequent overshadowing of low-similarity samples. To address the issue, we proposed a novel and flexible similarity aware split strategy for the test set to achieve desired distribution like uniform discrete distribution, which can deliver comprehensive evaluation for various drug-target binding affinity prediction algorithms. Furthermore, we utilized the similarity aware split to create "mimic split", splitting the training set and internal test set by replicating the distribution found an across
external test set. Our mimic split consistently aids in selecting the optimal various deep-learning methods. In the end, our split strategy enables the generation of distributions with minimum maximum similarity constraints as required.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENT
This work partially supported by the Shanghai Artificial Intelligence Laboratory.
REFERENCES
Heba Askr, Enas Elgeldawi, Heba Aboul Ella, Yaseen AMM Elshaier, Mamdouh M Gomaa, and Aboul Ella Hassanien. Deep learning drug discovery: an integrative review and future chal- lenges. Artificial Intelligence Review, 56(7):597566037, 2023.
Heval Atas Guvenilir and Tunca Dogan. How to approach machine learning-based prediction of brng/compound-targe target interactions. Journal of Cheminformatics, 15(1):16, 2023.
Dávid Bajusz, Anita Rácz, and Károly Héberger. Why is tanimoto index an appropriate choice for fingerprint-basee similarity calculations? Journal of cheminformatics, 7:1-13, 2015.
Guy W Bemis and Mark A Murcko. The properties of known drugs. 1. molecular frameworks. Journal of medicinal chemistry, 39(15):2887-2893, 1996.
Ayan Chatterjee, Robin Walters, Zohair Shafi, Omair Shafi Ahmed, Michael Sebek, Deisy Gysi, Rose Yu, Tina Eliassi-Rad, Albert-Lászle Barabási, and Giulia Menichetti. Improving the gen- eralizability of liggad binding predictions with ai-bind. Nature communications, 14(1):
1989, 2023.
Hongming Chen, Ola Engkvist, Yinhai Wang, Marcus Olivecrona, and Thomas Blaschke. The rise of deep learning in drug discovery. Drug discovery today, 23(6): 1241-1250, 2018.
Wenlin Chen, Austin Tripp, José Miguel Hermández-Looato. Meta-learning adaptive deep kernel gaussian processes for molecular property prediction. arXiv preprint urXiv:2205.02708, 2022.
Artem Cherkasov, Eugene N Muratov, Denis Fourches, Alexandre Varnek, Igor Baskin, Mark Cronin, John Dearden, Paola Gramatica, Yvonne C Martin, Roberto Todeschini, et al. Qsar modeling: where have you been? where are you going to? Journal of medicinal chemistry, 57 (12):4977-5010, 2014.
Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta: large-scale self- supervised pretraining for molecular property prediction. arXiv preprint arXhv:2010.09885, 2020.
Kangway V Chuang, Laura M Gunsalus, and Michael J Keiser. Learning molecular representations for medicinal chemistry: miniperspective. Journal of Medicinal Chemistry, 63(16))8705-8722,
2020.
Priyanka De, Supratik Kar, Pravin Ambure, and Kunal Roy. Prediction reliability of qsar models: an overview of various validation tools. Archives of Toxicology, 96(5):127--12955 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized mlm. Advances in Neural Information Processing Systems, 36, 2024.
Alexander Dmitriev, Alexey Lagunin, Dmitry A Karasev, Anastasia V Rudik, Pavel V Pogodin, Dmitry A Filimonov, and Vladimir V Poroikov. Prediction of drug-drug interactions related to inhibition or induction of drrug-metabolinn enzymes. Current Topics in Medicinal Chemistry, 19(5):319-335 936,,,333300000 2019.
Xiaomin Fang, Lihang Liu, Jieqiong Lei, Donglong He, Shanzhuo Zhang, Jingbo Zhou, Fan Wang, Hua Wu, and Haifeng Wang. Geonetry--ehanned molecular representation learning for property prediction. Nature Machine Intelligence, 4(2):177-334, 2022.
Jose Gallego-Posada and Juan Ramirez. Cooper: toolkit for Lagrangian-based constrained opti- mization. pectps://qithae
com/ www./ooopeoooooo/o/coooooo
2022.
Michael K Gilson. Tiqing Liu, Michael Baitaluk, George Nicola, Linda Hwang, and Jenny Chong. Bindingdb in 2015: public database for medicinal chemistry, computational chemistry and sys- tems pharmacology. Nucleic acids research, 44(D1))D004-D11053 2016.
11

---

Published as a conference paper at ICLR 2025
Jiaqi Guan, Wesley Wei Qian, Xingang Peng, Yufeng Su, Jian Peng, and Jianzhu Ma. 3d equiv- ariant diffusion for target-aware molecule generation and affinity prediction. arXiv preprint arXiv:2303.03543, 2023.
Tobias Harren, Torben Gutermuth, Christoph Grebner, Gerhard Hessler, and Matthias Rarey. Mod- em machine-leaming for binding affinity estimation of protein ligand complexes: Progress, op- portunities, and challenges. Wiley Interdisciplinary Reviews: Computational Molecular Science, 14(3)==1716, 2024.
Zhiqiang Hu, Wenfeng Liu, Chenbin Zhang, Jiawen Huang, Shaoting Zhang, Huiqun Yu, Yi Xiong, Hao Liu, Song Ke, and Liang Hong. Sam-dta: a sequence-agnostic model for drug-target binding affinity prediction. Briefings in Bioinformatics, 24(1):bbac533, 2023.
Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. arXiv preprint arXiv:2102.0954 2021.
James P Hughes, Stephen Rees, S Barrett Kalindjian, and Karen I. Philpott. Principles of early drug discovery. British journal of pharmacology, 162(6):1239-1244 2011.
Yinghui Jiang, Shuting Jin, Xurui Jin, Xianglu Xiao, Wenfan Wu, Xiangrong Liu, Qiang Zhang, Xiangxiang Zeng, Guang Yang, and Zhangming Niu. Pharmacophoric<oosstaaneee heterogeneous graph transformer model for molecular property prediction. Communications Chemistry, 6(1):60, 2023.
Mahmood Kalemati, Mojtaba Zamani Emani, and Somayyeh Koohi. Degan-dta: Predicting drug- target binding affinity with deep convolutional generative adversarial networks. BMC genomics, 25(1):411, 2024.
Mostafa Karimi, Di Wu, Zhangyang Wang, and Yang Shen. Deepaftinity: interpretable deep learn- ing of compound penteen affinity through unified recurrent and convolutional neural networks. Bioinformatics, 35(18) (13333)33333333333 2019.
Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Ben- jamin A Shoemaker, Paul A Thiessen, Bo Yu, et al. Pubchem 2023 update. Nucleic acids research, (((D)):D1333-D1388) 2023.
Sowmya Ramaswamy Krishnan, Navneet Bung, Gopalakrishnan Bulusu, and Arijit Roy. Accel- erating de novo drug design against novel proteins using deep learning. Journal of Chemical Information and Modeling, 31(2):621-600) 2021.
Damjan Krstajic, Ljubomir J Buturovic, David E Leahy, and Simon Thomas. Cross- pitfalls when selecting and assessing regression and classification models. Journal of cheminfor- matics, 6:1-15, 2014.
Gregory A Landrum, Maximilian Beckers, Jessica Lanini, Nadine Schneider, Nikolaus Stiefl, and Sereina Riniker. Simpd: an algorithm for generating simulated time splits for validating machine learning approaches. Journal of cheminformatics, 15(1):119, 2023.
J Thomas Leonard and Kunal Roy. On selection of training and test sets for the development of predictive qsar models. QSAR& & Combinatorial Science, 25(3)2235-251, 2006.
Hongjian Li, Gang Lu, Kam-Heung Sze, Xianwei Su, Wai-Yee Chan, and Kwong-Sak Leung- Machine-learning scoring functions trained on complexes dissimilar to the test set already out- perform classical counterparts on a blind benchmark. Briefings in bioinformatics, (2b6)bbb,,225, 2021.
Shiwei Li, Sanan Wu, Lin Wang, Fenglei Li, Hualiang Jiang, and Fang Bai. Recent advances in predicting protein- protein interactions with the aid of artificial intelligence algorithms. Current Opinion in Structural Biology, 73:102344, 2022.
Yang Li and Jianyi Yang. Structural and sequence similarity makes significant impact on machine- learning-based scoring functions for protein- ligand interactions. Journal of chemical information and modeling, 57(4): 1007 2017.
12

---

Published as a conference paper at ICLR 2025
Yanjun Li, Mohammad A Rezaei, Chenglong Li, and Xiaolin Li. Deepatom: A framework for protein-ligam binding affinity prediction. In 2019 IEEE International Conference on Bioinfor- matics and Biomedicine (BIBM), pp. 303-310. IEEE, 2019.
Zhiyuan Liu, Yaorui Shi, An Zhang, Enzhi Zhang, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. Rethinking tokenizer and decoder masked graph modeling for molecules. Advances in Neural Information Processing Systems, 36, 2024.
Ding Luo, Dandan Liu, Xiaoyang Qu, Lina Dong, and Binju Wang. Enhancing generalizability in protein-ligand binding affinity prediction with multimodal contrastive learning. Journal of Chemical Information and Modeling, 64(6):1822-1906 2024.
Yunan Luo, Xinbin Zhao, Jingtian Zhou, Jinglin Yang, Yanqing Zhang, Wenhua Kuang, Jian Peng, Ligong Chen, and Jianyang Zeng. A network integration approach for drug-target interaction prediction and computational drug repositioning from heterogeneous information. Nature com-
munications, 8(1):573,201
Neann Mathai, Ya Chen, and Johannes Kirchmair. Validation strategies for target prediction meth- ods. Briesings in bioinformatics, 21(3):911-802, 2020.
Nelson RC Monteiro, Carlos JV Simões, Henrique V Ávila, Maryam Abbasi, José L. Oliveira, and Joel Arrais. Explainable deep drug-target representations for binding affinity prediction. BMC bioinformatics, 23(1):237, 2022.
Thin Nguyen, Hang Le, Thomas P Quinn, Tri Nguyen, Thuc Duy Le, and Svetha Venkatesh. Graphdta: predicting drug-target binding affinity with graph neural networks. Bioinformatics, 14(8):1140-1144 147, 2021.
Tri Minh Nguyen, Thin Nguyen, and Truyen Tran. Mitigating cold-start problems in drug-target affinity prediction with interaction knowledge transferring. Briefings in Bioinformatics, 23(4): bbac269, 2022.
Hakime Özüürk, Arzucan Özgür, and Elif Ozkirimli. Deepdta: deep drug target binding affinity prediction. Bioinformaticx, 34(17):1821-i82) 2018.
Tapio Pahikkala, Antti Airola, Sami Pietilä, Sushil Shakyawar, Agnieszka Szwajda, Jing Tang. and Tero Aittokallio. Toward more realistic drug-targed target interaction predictions. Briefings in bioinfor- matics, 16(2):325-337, 2015.
Yungki Park and Edward M Marcotte. Flaws in evaluation schemes for pair- mnir-input computational predictions. Nature methods, 9(12):1134-111 1136, 2012.
Tomasz Puzyn, Aleksandra Mostrag-S Saatrag-Szlicttnng. Agnieszka Gajewicz, Michat Skrzyáski, and An- drew P Worth. Investigating the influence of data splitting on the predictive ability of qsar/qspr models. Structural Chemistry, 22:795-804, 2011.
Jack Scantlebury, Lucy Vost, Anna Carbery, Thomas E. Hadfield, Oliver M Turnbull, Nathan Brown, Vijil Chenthamarakshan, Payel Das, Harold Grosjean, Frank Von Delft, et al. small step to- ward generalizabiltty: training machine learning scoring function for structure-based virtual screening. Journal of Chemical Information and Modeling, 63110)2290-2974, 2023.
Smriti Sharma and Vinayak Bhatia. Recent trends in qsar in modelling of drug-protein and protein- protein interactions. Combinatorial Chemistry & High Throughput Screening, 103))1101 2021.
Robert P Sheridan. Time-split cross- cross-validttion as method for estimating the goodness of prospec- tive prediction. Journal of chemical information and modeling, 53(4):783-790, 2013.
Robert P Sheridan, Bradley P Feuston, Vladimir N Maiorov, and Simon K Kearsley. Similarity to molecules in the training set is good discriminator for prediction accuracy in qsar. Journal of chemical information and computer sciences, 1912 2004.
13

---

Published as a conference paper at ICLR 2025
Robert P Sheridan, Chris Culberson, Elizabeth Joshi, Matthew Tudor, and Prabha Karnachi. Pre- diction accuracy of production admet models as function of version: activity cliffs rule. Journal of Chemical Information and Modeling, (2(14):3275-3288 2022.
Jochen Sieg, Florian Flachsenberg, and Mauthias Rarey. In need of bias control: evaluating chemical data for machine learning in structure-based virtual screening. Journal of chemical information and modeling, 59(3):47--91, 2019.
Jaak Simm, Lina Humbeck, Adam Zalewski, Noe Sturm, Wouter Heyndrickx, Yves Moreau, Bernd Beck, and Ansgar Schuffenhauer. Splitting chemical structure data sets for federated privacy- preserving machine learning. Journal of cheminformatics, 13:1-14, 2021.
Yuanbing Song. Jinghua Chen, Wenju Wang, Gang Chen, and Zhichong Ma. Double-head trans- former neural network for molecular property prediction. Journal of Cheminformaties, 15(1):27, 2023.
Megan Stanley, John F Bronskill, Krzysztof Maziarz, Hubert Misztela, Jessica Lanini, Marwin Segler, Nadine Schneider, and Marc Brockschmidt. Fs-mol: few-shot learning dataset of molecules. In Thirty-fifth Conference Neural Information Processing Systems Dulasets and Benchmarks Track (Round 2), 2021.
Hannes Stärk, Octavian Ganea, Lagnajit Pattanaik, Regina Barzilay, and Tommi Jaakkola. Equibind: Geometric deep learning for drug binding structure prediction. In International conference on machine learning, pp. 10503-20521.. PMLR, 2022.
Chunyan Tang, Cheng Zhong, Mian Wang, and Fengfeng Zhou Fmgnn: A method to predict compound-protein interaction with pharmacophore features and physicochemical properties of amino acids. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 20(2): 1030-1040, 2022.
Prudencio Tossou, Cas Wognum, Michael Craig, Hadrien Mary, and Emmanuel Noutahi. Real-world molecular out-of-distributton: Specification and investigation. Journal of Chemical Information and Modeling, 64(3):697-711, 2024.
Giovanni A Tricarico, Johan Hofmans, Eelke B Lenselink, Miriam -ópez-Ramos, Marie-Pierre Dréanic, and Pieter FW Stouten. Construction of balanced, chemically dissimilar training, vali- dation and test sets for machine learning on molecular datasets. 2024.
Alexander Tropsha, Olexandr Isayev, Alexandre Varnek, Gisbert Schneider, and Artem Cherkasov. Integrating qsar modelling and deep learning in drug discovery: the emergence of deep qsar. Nature Reviews Drug Discovery, 13(2):141-155, 2024.
Fangping Wan, Lixiang Hong, An Xiao, Tao Jiang, and Jianyang Zeng. Neodti: neural integration of neighbor information from a heterogeneous network for discovering new drug-target interactions. Bioinformatics, 35(1):104-111, 2019.
Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4(3):279- 287, 2022.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learn- ing. Chemical science, 9(2):513-530 2018.
Zheng Xu, Sheng Wang, Feiyun Zhu, and Junzhou Huang. Seq2seq fingerprint: An unsupervised deep molecular embedding for drug discovery. In Proceedings of the 8th ACM international conference on bioinformatics, computational biology, and health informatics, pp. 285-294, 2017.
Jincai Yang, Cheng Shen, and Niu Huang. Predicting pretending: artificial intelligence for protein-ligand interactions lack of sufficiently large and unbiased datasets. Frontiers in phar- macology, 11 11:69, 2020.
14

---

Published as a conference paper at ICLR 2025
Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman- Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et al. Analyzing learned molecular repre- sentations for property prediction. Journal of chemical information and modeling, 59(8):3370- 3388, 2019.
Ziduo Yang, Weihe Zhong, Lu Zhao, and Calvin Yu-Chian Chen. Mgraphdta: deep multiscale graph binding affinity prediction. Chemical science, 13(3):
neural network for explainable 816 833, 2022.
Cheng Zhi Ying, Chieu Hai Leong, Liew Wen Xing Alvin, and Chua Jing Yang. Improving the workflow of chemical structure elucidation with morgan fingerprints and the tanimoto coefficient. In IRC-SET 2020: Proceedings of the 6th IRC Conference on Science, Engineering and Technol- ogy, July 2020, Singapore, pp. 14.24.35 Springer, 2021.
Weining Yuan, Guanxing Chen, and Calvin Yu-Chian Chen. Fusiondta: attention-base bbaeed feature polymerizer and knowledge distillation for drug-target binding affinity prediction. Briefings in Bioinformatics, 23(1 23(1):bbarn (b1):bbab506, 2022.
Barbara Zdrazil, Eloy Felix, Fiona Hunter, Emma J Manners, James Blackshaw, Sybilla Corbett, Marleen de Veij, Harris loannidis, David Mendez Lopez, Juan Mosquera, al. The chembl database in 2023: a drug discovery platform spanning multiple bioactivity data types and time periods. Nucleic acids research, 52DD1))D1100-D1122 2024.
Xiao-Chen Zhang, Cheng-Kun Wu, Jia-Cai Yi,
Zeng, Can-Qun Yang, Ai-Ping Lu, Ting-Jun Hou, and Dong-Sheng Cao. Pushing the boundaries of molecular property prediction for drug discovery with multitask learning bert enhanced by smiles enumeration. Research, 2022: 0004, 2022.
Qichang Zhao, Guihua Duan, Mengyun Yang. Zhongjian Cheng, Yaohang Li, and Jianxin Wang- Attentiondta: Drug-target binding affinity prediction by sequence-based deep learning with at- tention mechanism. IEEF/ACM transactions on computational biology and bioinformaties, 20 (2):852-863, 2022.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. 2023.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 RELATED WORKS ON DATA SPLITTING
In molecular machine learning, including general QSAR tasks, the challenge of fair predictive eval- uation has been a longstanding issue (Cherkasov et al. 2014). While randomized split remains the most commonly used strategy for data splitting, it is not always the optimal choice for evaluating machine learning methods. Consequently, various alternative split strategies have been developed to better evaluate the machine learning methods:
Time split (Sheridan 2013 Stärk et al. 2022 Guan et al. 2023) is employed for datasets containing temporal information, where the model is trained on historical data and tested on more recent data. It may effectively replicate real-world scenarios, however, significant number of datasets are devoid of time-specific information. In some situations, when the time span is too large or the data distribution changes significantly over time, the model may struggle to perform well on the test set.
Scaffold split (Bemis & Murcko, 1996) is technique that splits the dataset based on the structural framework of each sample. It is often leveraged in situations involving out-of- distribution data provide measure of generalization capabilities (Stanley et al.] 2021 Fang et al.) 2022 Zhou et al.] 2023 Liu et al. 2024). Because scaffold split does not enforce stratification during the partitioning process, may result in class imbalance (Yang
et al. 2019).
Stratified split also called stratified random sampling is a sampling method designed to ensure that each fold of a dataset maintains the same distribution of classes as the entire dataset. It achieves this by first dividing the data into different output strata based on class labels and then executing random partition with the guaranteeing that the entire label range is encompassed within each set (Krstajic et al. 2014 Wu et al. 2018 Mathai et al., 2020 Chen et al. 2022)..
Cold-drug split (Huang et al. 202T) is a method for dividing a dataset into multi-protein prediction tasks, where the dataset is split based on entity types, such as proteins, drugs, or DNAs. The process begins by randomly splitting the dataset into training, validation, and test sets based on one chosen entity type. Subsequently, all data samples associated with specific entity are assigned to the same set to ensure no overlap across splits, ensuring that there is no overlap of the chosen entity type across the splits.
SIMPD split (Landrum et al. 2023) mimics temporal splits in scenarios where temporal information not accessible. This approach was developed by observing and analyzing disparities observed between earlier and subsequent samples within the scope of medicinal chemistry projects.
Dissimilar split (Atas Guvenilir & Dogan 2023) divides dataset into training and test sets by ensuring that samples in each set are dissimilar. This strategy prevents similar samples from appearing in both sets, thereby increasing the difficulty of predictions.
This challenge is closely intertwined with the broader problem of nut-of-distributtin (OOD) gen- eralization (Tossou et al.) 2024). demonstrating its relevance far beyond the confines of individual lasks such as DTA prediction. In fact, machine learning models tend to perform well when the training set shares similar distribution with the test set (Leonard & Roy 2006 Puzyn et al. 2011 Cherkasov et al. 2014). However, previous split strategies often yield test sets with distributions that closely mimic the training set (as shown in Figure A.2). Such alignment between the training and test set distributions can lead to overly optimistic assessments of model's generalization ability, as fails to account for scenarios where the model is applied to data with significantly different char- acteristics. SAE provides effective solution to this issue by enabling more precise control over data distributions through its ability to capture the similarities between data samples. This approach ensures greater adaptability to wider range of scenarios.
A.2 DISCUSSION ABOUT THE APPLICATION OF QSAR SCENARIOS
Quantitative Peructre-Activing Relationship (QSAR) modeling is a widely used in silico approach for predicting the biological or chemical properties of molecules (De et al. 2022). Previous studies
16

---

Published as a conference paper at ICLR 2025
on QSAR (Sheridan et al. 2004 have shown that prediction accuracy is highly correlated with the similarity between the molecule being predicted and its closest neighbor in the training set. This observation is similar to patterns found in the DTA prediction task. Therefore, our SAE method can also be extended to QSAR tasks.
For instance, Krishnan al. 2021) introduced de novo drug design method that incorporates pre-trained model alongside transfer learning to generate novel inhibitors targeting the human JAK2 protein. In this approach, transfer learning was utilized capture the features of the target-related chemical space. If the characteristics of the target-related chemical particppry the distribu- tion of the external dataset- are already well understood, our SAE can be applied replicate this distribution during the splitting of training and test sets, thereby enhancing the overall performance.
Similarly, in the task of protein-protein interaction prediction, improper construction of the data split among training, validation, and test sets can lead to severe data leakage and overly optimistic results (Li et al., 2022). To address this issue, one proposed solution to divide the test set into three distinct classes (Park& & Marcotte| [2012: CI, where test pairs consist of proteins that are both present in the training set; C2, where test pairs involve one protein present in the training set; and C3, where neither protein in the test pair is found in the training set. Notably, the three classes can be viewed as specific cases of our SAE split strategy. Furthermore, the SAE approach can be flexibly applied to constructing test sets with varying levels of difficulty to more effectively evaluate the model's generalization.
A.3 TIME COMPLEXITY AND SPACE COMPLEXITY ANALYSIS
Given the number of iterations M. the number of samples N, and the number of bins K, we analyze the time complexity of a single iteration in Eq. 1 which involves both forward and backward propagation. During forward propagation, computing Og involves O(N-K) operations, iterating over N samples and K bins, with softmax and exponential computations. The computation of r, more expensive, requiring O(N2) operations due to the nested summation over N samples. The regularization term lyeg involves simple summation over N, contributing ((N) operations. Combining these, the time complexity of one forward propagation is dominated by the O(N2) and O(N K) terms, resulting O(N2 + N K), which simplifies to O(N2) because K < N. For backward propagation, the computation of gradients with respect to w; involves similar operations, which results in the same complexity of O(N)). Additionally, the process of checking constraints involves N + Lagrangian multipliers. The forward and backward propagation for this constraint-checking step each have a complexity of O(N). Combining all of these components, the time complexity of one iteration is O(N2), and the total time complexity of SAE across all iterations is O(M 22). The overall space complexity of SAE is primarily determined by the storage requirements for Sig and the intermediate values needed for computing gradients from r4 to wy. As a result, the space complexity is O
requires
Empirically, for the IC50 dataset of target EGFR which contains N = 4, 361 samples, the desired distribution is a uniform over bins [1/3, 2/3]. (1/3, 2/3], (2/3, 1]. We set the number of iterations to M = 20, 000. On a single 3090 GPU, SAE completes the process in approximately 270 seconds, utilizing 2,410 MiB of GPU memory.
We would also like to emphasize that SAE is used in the model development stage and as a result, when the model is developed, it will longer affect the efficiency for high-throughput inference. Note also that SAE needs only to be performed once for a fixed dataset, meaning that it can be reused by different models as long as they are developed it takes may be overwhelmed by the time used by the heavy model development. When scaling to large datasets, should be noted that almost all operations within one iteration are parallelable, and thus it will benefit significantly from more powerful GPU devices and distributed computation.
the same dataset. As result, the time
A.4 DISCUSSION ON THE APPLICATION OF SAE TO LARGE- DATASETS
In pharmacompany (private) drug libraries for early drug discovery, there might be 200,000 to 10s samples (Hughes et al. 2011). When scaling to these large datasets, solution based on the sparse matrix is applicable. Specifically, Figure A. shows that the majority of pairwise similarities are low. Suppose the desired distribution is uniform over bins (1/3, 2/3), (1/3, 2/3), (2/3, 1) just in
17

---

Published as a conference paper at ICLR 2025
2.5c6
2.0c6
Count
1.566-
1/3] 95.89%
1.0c6
5.005
(1/3,
4.11%
0.0c0
0.0
0.2
1/3
0.4
0.6
0.8 1.0
Similarity
Figure A.1: The similarity distribution of all sample pairs in the IC50 prediction task for the target EGFR. The dataset consists of 4,361 samples, resulting in a total of 9,506,980 pairwise similarity calculations. Among these, 95.89% of the pairs exhibit similarities of no greater than 1/3, while only 11% have similarities exceeding 1/3.
Pansatnge
0.2
02-
0.2
0.0
80
0.
00
(8.50)
(5.00)
(8] Extemaa Test Set
(b) Random
(c) Scaffold
(d) SIMPD
022
0.2-
02-
0.0
0.0
0.0
0.
(22
(21
186
( SAE (mimiee
( Stratified (max)
(g) Stratified (wyg)
(c) Dissimilar
Figure A.2: The similarity distribution of the internal test set across different split strategies. (b) Randomized split leads to a scenario where most internal test samples are highly similar to the training set. (c) Scaffold split produces more balanced distribution. (d) SIMPD split yields distribution similar to the random split. (e) Our SAE (mimic) split brings the internal test set's distribution closest to that of the external test set. (f) Stratified split based on the maximum similarity of each ligand to all others in the dataset. (g) Stratified split based on the average similarity each ligand to all others in the dataset. (h) Dissimilar split guarantees that the similarity will remain below 0.5.
the previous section, over 95% entries in the similarity matrix are less than 1/3 and can be safely set to zero without the interference of the results. The time and space complexity can be significantly reduced in this way. Given N 106, the number of pairs with similarity greater than 1/3 would be approximately 0.05 [(N - 1)/2, which about 2.5 x 10 We can store these similarities in sparse format, represented tuples (Index of sample A. Index of sample B, Similarity value). Each index can be encoded using 20 bits (sufficient to represent 20 1,048,576 positions), and the
similarity value be quantized into 4 bits (Dettmers al. 2024 Consequently, the total storage
18

---

Published as a conference paper at ICLR 2025
Table A.I: Comparison of the generalization ability of different split strategies at IC50 for EGFR across five DTA prediction methods.
Internal Test MAE
Internal Test R
External Test
External Test R²
Split
Method
MAE
TTs
0.6445
0.5399
1.1198
0.0957
ChemBERTa
0.6389
0.5574
1.0600
0.1517
Random
MolCLR
0,5830
0,6318
1.0976
0.1546
PharmHGT
0.6166
0.5739
1.1107
0.1323
SAM-DTA
0.5805
0.6301
0.9863
0.3206
FusionDTA
0.9626
0.1544
1.0863
0.1243
ChemBERTa
0.9314
0.1997
1.1972
-0.0491
Scaffold
MolCLR
0,8627
0.3145
1.1585
0.0405
PharmHGT
0.8537
0.3427
1.0930
0,1594
SAM-DTA
0.8725
0.3311
1.0187
0.3034
FusionDTA
0.7215
0.3292
1.1417
0.0528
ChemBERTa
0.6954
0.3642
1.1010
0.0878
SIMPD
MoICLR
0.6334
0.4775
1,2131
-0.0016
PharmHGT
0,6742
0.3958
1.1588
0.0133
SAM-DTA
0.6271
0.4867
1.0058
0.3083
FusionDTA
0.6753
0.5346
1.0886
0.1517
ChemBERTa
0,6752
0.5384
1.1504
0.0223
Stratified (max)
MolCLR
0.5968
0.6504
1.0917
0.1207
PharmHGT
0.6092
0.6302
1.0694
0.1811
SAM-DTA
0.6019
0.6404
1.0345
0,2722
FusionDTA
0.6490
0.5713
1.0957
0.1206
ChemBERTa
0.6724
0.5191
1.1258
0.0896
Stratified (avg)
MolCLR
0.5939
0.6368
1.1556
0.1019
PharmHGT
0.6103
0.5895
1.0938
0.1667
SAM-DTA
0.6099
0.6159
0.9946
0.3345
FusionDTA
0.9425
-0.1256
1.2788
-0.1063
ChemBERTa
0,8927
-0,0139
1,6402
-0.5971
Dissimilar
MolCLR
0.8462
0.0592
1.3355
-0.1366
PharmHGT
0.9011
-0.0029
1.6006
-0.5237
SAM-DTA
0.9239
-0.0845
1.2140
-0.0039
FusionDTA
0.9130
0.2919
1.0605
0.2122
ChemBERTa
0.8976
0.2736
1.0452
0.2477
SAE (mimic)
MolCLR
0.8536
0.3653
1.0002
0,2981
PharmHGT
0.8826
0.3200
1.0609
0.1861
SAM-DTA
0.8545
0.3770
0.9773
0.3367
requirement can calculated as:
(20bits 20bits + 4bits) x 2.5 x 10tH 5.5Byles 2.5 10a0 128GB
This size is manageable and can even be stored in memory.
A.5 SUPPLEMENTARY EXPERIMENTAL RESULTS OF MIMIC SPLIT
For thorough comparison with other split strategies, we implemented stratified split (Wu et al. 2018 Chen et al. 2022 and dissimilar split Atas Guvenilir & Dogan 2023) at IC50 for EGFR. For the stratified split, we first compute the pairwise similarities for the full dataset, resulting in similarity matrix of size N x N (where N is the number of samples in the dataset, with the diagonal values set to zero). Next, we calculate the maximum/average similarity for each row, yielding similarity vector of size N, which represents the maximun/average similarity of each ligand all others in
19

---

Published as a conference paper at ICLR 2025
Table A.2: Detailed comparison of the generalization ability of different split strategies IC50 for EGFR across five DTA prediction methods.
Extrenal Test MAE
Bin
Count
Split
FusionDTA ChemBERTa MolCLR PharmHGT SAM-DTA
Random
1.2442
1.1167
1.2385
1.3261
1.1343
Scaffold
1.2293
1.2242
1.1630
1.1840
1.1363
SIMPD
1,4297
1.3223
1.7376
1.3382
1.1179
[0, 1/3]
120
Stratified (max)
1.2311
1.1663
1.2414
1.2179
1.1747
Stratified (avg)
1.5371
1.4478
1,6005
1.3962
1.3666
Dissimilar
1,4134
1.2363
1.1912
1.2424
1.4161
SAE (mimic)
1.0626
1.0315
1,1848
1.1082
1.0435
Random
1.1416
1.0874
1.0983
1. 1.0989
0.9879
Scaffold
1.0729
1,2273
1.2021
1.0891
1.0209
SIMPD
1.1545
1.1272
1.2134
1.1614
1.0158
(1/3, 2/3] 1026
Stratified (max)
1.0756
1.1611
1.0927
1.0677
1.0317
Stratified (avg)
1.0815
1.1450
1.1384
1.0835
0.9808
Dissimilar
1.3106
1.6954
1.3676
1.6504
1.2513
SAE (mimic)
1.0531
1.0594
0.9979
1. 1.0666
0.9743
Random
0.9559
0.9060
1.0208
1.0516
0.9015
Scaffold
1.0604
1.0346
0.9449
1.0543
0.9335
SIMPD
0.9741
0.9022
1.0066
1.0775
0.9191
(2/3, 1]
186
Stratified (max)
1.0718
1.0883
1.0017
0,9933
0.9683
Stratified (avg)
0.9694
0.9047
1.0415
1.0102
0.8967
Dissimilar
1.0201
1.6720
1,2858
1.6248
0.8773
SAE (mimic)
1.1003
0.9762
0.8938
1.0322
0.9511
External Test R
Bin
Count
Split
FusionDTA ChemBERTa MoICLR PharmHGT SAM-DTA
Random
-0.5661
-0.5827
-0.5009
-0.6605
-0.2855
Scaffold
-0.1335
-0.3903
-0.1887
-0.1096
-0,0121
SIMPD
-0.8717
-0.7917
-1.3400
-0,6245
-0.1181
[0, 1/3]
120
Stratified (max)
-0.4278
+0.4195
-0.7345
-0.3873
-0.3106
Stratified (avg)
-0.8887
-1.0024
-1.1185
-0.5790
-0.3962
Dissimilar
-0.4209
-0.0677
0.0122
-0.0173
-0.4122
SAE (mimic)
-0.0736
-0.1140
-0.3105
-0.2592
-0.0783
Random
0.0351
0.1066
0.1433
0.1005
0.2950
Scaffold
0.0987
-0.1211
-0.5502
0.1355
0.2824
SIMPD
0.0078
0.0332
-0.0169
-0.0337
0.2780
(1/3, 2/3] 1026
Stratified (max)
0.1305
-0.0321
0.1016
0.1530
0.2633
Stratified (avg)
0.1017
0.0476
0.1156
0.1430
0.3188
Dissimilar
-0,2307
-0,8279
-0,2834
-0,7374
10.1244
SAE (mimic)
0.1791
0.1848
0.2632
0.1237
0.3020
Random
0.0869
0.1390
-0.0844
0.0577
0.2594
Scaffold
-0.0667
+0.0351
0.1475
-0.0412
0.2110
SIMPD
0.1642
0.2476
0.0842
0.0139
0.2880
(2/3, 1]
186
Stratified (max)
0.0272
-0.0595
0.1063
0.1111
0.1550
Stratified (avg)
0.1114
0.2234
0.0081
0.0860
0.3187
Dissimilar
0.0288
-1.0679
-0.3223
-0.9576
0.2743
SAE (mimic)
-0,2135
0.1123
0.1807
0.0350
0.1359
the dataset. Finally, divide the dataset into K bins based the maximal/average similarity and perform random sampling within each bin to create the test set. We refer to the two variations of this stratified split strategy "Stratified (max)," which the maximum similarity for binning.
20

---

Published as a conference paper at ICLR 2025
and "Stratified (avg)", which uses the average similarity. The similarity distributions of the stratified
split are shown Figure A.2 (f) and Figure A.2 (g). The distribution result dissimilar split is
(h). A comparison of the generalization ability of different split strategies is shown in Table A. SAE performs better than stratified sampling and dissimilar split with clear
shown in Figure A.2
margin.
We also analyze the performance of the different brackets in the external dataset. As shown in Table A.2 SAE improves performance in the low- and mid-similarity brackets, but in the high- similarity one. We believe this is because the internal test set by SAE has more samples in the low- and mid-similarity brackets and thus performance in these brackets receives more attention compared with other split strategies.
A.6 COMPARISON ACROSS DIFFERENT SIMILARITY MEASURES AND FINGERPRINTS
As for comparison across different similarity measures and fingerprints, we conducted experiments on similarity measure choices including Tanimoto, Cosine, Sokal, and Dice, and fingerprint choices including Morgan (ECFP), RDKFP (RDKit), and Avalon. As shown by Table A.3 split results are less affected by similarity measure choices but more influenced by fingerprint choices. In all settings of similarity measures and fingerprints, SAE outperforms other approaches by achieving a split that closer to the desired distribution (uniform distribution in this case).
21

---

Published as a conference paper at ICLR 2025
Table A.3: Comparison across different similarity measures and fingerprints, the desired distribution is uniform distribution across the bins [0, 1/3]. (1/3, 2/3J. and (2/3, 1]. The numbers in this table are presented in the form [Sample counts in the first bin. Sample counts in the second bin, Sample counts in the third bin].
Stratified (avg)
0. 27. 846
1. 14, 858
0. 6, 867
29. 416, 428
15. 74, 784
6. 73, 794
2. 27. 844
1. 14. 858
0. 6. 867
13. 99. 761
4. 33, 836
1, 28. 844
Stratified (max)
32, 841
0. 13. 860
0. 6. 867
34, 423, 416
82, 777
8. 76. 789
0. 34, 839
14. 859
0. 8. 8.865
12. 99. 762
39, 833
0. 30, 843
14,
0.
16, 657, 200
1. 124, 748
16. 857
689, 126. 58
135, 624, 114
23, 629, 221
17. 672, 184
134, 738
0, 23. 850
228, 547, 98
635. 236
1, 324, 548
SIMPD
6, 159, 708
1. 78. 794
0. 21, 21, 852
172, 510, 191
85. 236. 552
26, 275, 572
9. 163, 701
2. 83, 788
0, 26. 847
80, 273. 520
15, 184, 674
2, 154, 717
Scaffold
0. 33, 840
0. 17. 856
0. 7, 866
33. 398. 442
80. 774
7. 63, 803
33, 840
19. 854
0. 7. 866
98, 759
8. 34, 831
0. 28, 845
Random
19.
0.
0.
16.
SAE (balanced)
145, 436, 292
18, 426. 429
429.. 435
292, 289. 292
291, 291, 291
291, 291, 291
182, 378. 313
60. 463, 350
32. 416, 425
290. 299, 284
289. 292, 292
220. 325, 328
Fingerprint
Morgan
RDKFP
Avalon
Morgan
RDKHP
Avalon
Morgan
RDKFP
Avalon
Morgan
RDKFP
Avalon
Similarity Measure
Tanimoto
Cosine
Sokal
Dice
22

---

Published as a conference paper at ICLR 2025
Table A.4: sepee-parameters used in the mimic split experiment for each method, the following search options are derived from the default parameter settings of each method.
Method
Hyper-paramert parameter
Options
Optimizer
[Adam, SGD]
SAM-DTA
Learning rate
[le-3, le-4, 1e-5] [10, 32, 64]
Batch size
Optimizer
[Adam, SGDJ
Learning rate
MoICLR
([le-3,5e-3] (Ic-4, 5e-4), (le-5, 5c-5)]
(prediction head, GNN encoder)
Dropout ratio
[0.3, 0.5] [Mean, Max, Add]
Readout pooling
Optimizer
[Adam, SGDJ
Learning rate
[le-2, 1e-3, le-4]
FusionDTA
Batch size
[128, 256]
Loss function
[L1, MSE]
Optimizer
[Adam, SGD]
Learning rate
[le-2, 1e-3, 1e-4]
PharmHGT
Activation function
[Sigmoid, ReL.U]
Loss function
[RMSE, MAE]
Optimizer
[Adam W, Adafactor] [4e-3, 4e-4, 4e-5, 4e-6]
ChemBERTa
Learning rate
Batch size
[4. 8, 16]
23

---

Published as a conference paper at ICLR 2025
Table A.5: Variations of Similarily To TrainingSel related to feature extraction, similarity measure, aggregation functions, and performance metrics. The methods for the detailed showcase are Fu- sionDTA and ChemBERTa.
Randomized Split (MAE)
Feature: RDKit fingerprint
Feature: Avalon fingerprint
Bin
Count (Ratio)
FusionDTA ChemBERTa Count (Ratio) FusionDTA ChemBERTa
1/3]
(0.0092)
1.4442
1.4679
0 (0.0000)
(1/3, 2/3]
34 (0.0389)
1.1294
1.1306
28 (0.0321)
1.3299
1,2340
(2/3,
1]
831 (0.9519)
0.6407
0.6546
845 (0.9679)
0.6451
0.6623
overall
873 (1.0000)
0.6671
0.6806
873 (1.0000)
0.6671
0.6806
Similarity Measure: Sokal similarity
SimilarityMeasurre Dice coefficient
[0 1/3]
33 (0.0378)
1.2751
1.2711
0 (0.0000)
(1/3, 2/3]
398 (0.4559)
0.7366
0.7234
33 (0.0378)
1.2751
1,2711
(2/3, I]
442 (0.5063)
0.5591
0.5980
840 (0.9622)
0.6432
0.6574
overall
873 (1.0000)
0.6671
0.6806
873 (1.0000)
0.6671
0.6806
Aggregation: Top-3
Aggregation: Top-5
[0 1/3]
17 (0.0195)
1.1567
1.3484
24 (0.0275)
1.3682
1.3330
(1/3, 2/3]
171 (0.1959)
0.8839
0.8722
240 (0.2749)
0.7861
0,8094
(2/3, 1]
685 (0.7847)
0.6008
0.6162
609 (0.6976)
0.5926
0.6041
overall
873 (1.0000)
0.6671
0.6806
873 (1.0000)
0.6671
0.6806
Randomized Split (R3)
Feature: RDKit fingerprint
Feature: Avalon fingerprint
Bin
Count (Ratio)
FusionDTA ChemBERTa
Count (Ratio)
FusionDTA ChemBERTa
1/3]
(0.0092)
0.2319
0.2536
0 (0.0000
(1/3, 2/3]
34 (0.0389)
0.2253
0.1829
28 (0.0321)
0.2166
0.2573
(2/3,
1]
831 (0.9519)
0.5899
0.5796
845 (0.9679)
0.5845
0.5698
overall
873 (1.0000)
0.5697
0.5585
873 (1.0000)
0.5697
0.5585
SimilarityMeasure: Sokal similarity
SimilarityMeasure: Dice coefficient
1/3]
33 (0.0378)
0.0871
0.0615
0 (0.0000)
(1/3, 2/3]
398 (0.4559)
0.5068
0.5186
33 (0.0378)
0.0871
0.0615
(2/3, 1]
442 (0.5063)
0.6469
0.6117
840 (0.9622)
0.5898
0.5793
overall
873 (1.0000)
0.5697
0.5585
873 (1.0000)
0.5697
0.5585
Aggregation: Top-3
Aggregation: Top-5
1/3]
17 (0.0195)
-0.0685
-0.3530
24 (0.0275)
-0.0237
-0.0345
(1/3, 2/3]
171 1959)
0,4390
0,4408
240 (0.2749)
0.4881
0.4647
(2/3, 1]
685 (0.7847)
0.5930
0.5837
609 (0.6976)
0.5965
0.5903
overall
873 (1.0000)
0.5697
0.5585
873 (1.0000)
0.5697
0.5585
24

---

Published as a conference paper at ICLR 2025
Table A.6: Variations of Similarity S related to feature extraction, similarity measure, aggregation functions, and performance metrics. The method for the detailed showcase is MolCLR.
Randomized Split (MAE)
Feature: RDKit fingerprint
Feature: Avalon fingerprint
Bin
Count (Ratio)
MolCLR
Count (Ratio)
MolCLR
[0 1/3]
(0.0092)
1.4442
0 (0.0000)
(1/3, 2/3]
34 (0.0389)
1.1294
28 (0.0321)
1.3299
(2/3,
1]
831 (0.9519)
0.6407
845 (0.9679)
0.6451
overall
873 (1.0000)
0.6671
873 (1.0000)
0.6671
Mieasurt Sokal similarity
SimilarityMeasun Dice coefficient
1/3]
33 (0.0378)
1.2751
0 (0.0000)
(1/3, 2/3]
398 (0.4559)
0.7366
33 (0.0378)
1.2751
(2/3, I]
442 (0.5063)
0.5591
840 (0.9622)
0.6432
overall
873 (1.0000)
0.6671
873 (1.0000)
0.6671
Aggregation: Top-3
Aggregation: Top-5
[0 1/3]
17 (0.0195)
1,1567
24 (0.0275)
1.3682
(1/3, 2/3]
171 (0.1959)
0,8839
240 (0.2749)
0.7861
(2/3, I]
685 (0.7847)
0.6008
609 (0.6976)
0.5926
overall
873 (1.0000)
0.6671
873 (1.0000)
0.6671 1
Randomized Split (R?)
Feature: RDKit fingerprint
Feature: Avalon fingerprint
Bin
Count (Ratio)
MolCLR
Count (Ratio)
MoICLR
[0 1/3]
(0.0092)
0,2319
0 (0.0000)
(1/3, 2/3]
34 (0.0389)
0.2253
28 (0.0321)
0.2166
(2/3, 1]
831 (0.9519)
0.5899
845 (0.9679)
0.5845
overall
873 (1.0000)
0.5697
873 (1.0000)
0.5697
SimilarityMeasure Sokal similarity
Similarity Measure: Dice coefficient
[0 1/3]
33 (0.0378)
0.0871
0 (0.0000)
(1/3, 2/3]
398 (0.4559)
0.5068
33 (0.0378)
0.0871 1
(2/3. 1]
442 (0.5063)
0.6469
840 (0.9622)
0.5898
overall
873 (1.0000)
0.5697
873 (1.0000)
0.5697
Aggregation: Top-3
Aggregation: Top-5
[0 1/3]
17 (0.0195)
-0.0685
24 (0.0275)
-0.0237
(1/3, 2/3]
171 19.1959)
0.4390
240 (0.2749)
0.4881
(2/3, 1]
685 (0.7847)
0.5930
609 (0.6976)
0.5965
overall
873 (1.0000)
0.5697
873 (1.0000)
0.5697
25

---

Published as a conference paper at ICLR 2025
Table A.7: Comparison Randomized Split and SAE (balanced) Split at ICS0 for BACEI, Ki for Carbonic anhydrase I and Carbonic anhydrase II. The methods for the detailed showcase FUA and ChemBERTa.
IC50 for Target BACEI (MAE)
Randomized Split
SAE (balanced) Split
Bin
Count (Ratio)
FusionDTA ChemBERTa Count (Ratio)
FusionDTA ChemBERTa
[0 1/3]
10 (0.0108)
1.3020
1.1440
309 (0.3330)
1.2117
1.2503
(1/3, 2/3]
67 (0.0722)
0.7270
0.6719
311 (0.3351)
0.7444
0.6599
(2/3,
1]
851 (0.9170)
0.5105
0.5267
308 (0.3319)
0.5310
0.5352
overall
928 (1.0000)
0.5347
0.5439
928 (1.0000)
0.8292
0.8151
IC50 for Target BACEI (R²)
[0 1/3]
10 (0.0108)
0.0422
0.3204
309 (0.3330)
-0.5113
-0.5641
(1/3, 2/3]
67 (0.0722)
0.5980
0.6325
311 (0.3351)
0.4238
0,5787
(2/3, 1]
851 (0.9170)
0.6651
0.6446
308 (0.3319)
0.7076
0.7235
overall
928 (1.0000)
0.6755
0.6673
928 (1.0000)
0.4213
0,4548
Ki for Target Carbonic anhydrase (MAE)
Randomized Split
SAE (balanced) Split
Bin
Count (Ratio)
FusionDTA ChemBERTa Count (Ratio)
FusionDTA ChemBERTa
1/3]
(0.0079)
0.9363
0.9564
264 (0.2983)
1.0252
0.9245
(1/3, 2/3]
205 (0.2316)
0.7086
0.7085
311 (0.3514)
0.7362
0.7228
(2/3, I]
673 (0.7605)
0.5203
0.5440
310 (0.3503)
0.6181
0.6060
overall
885 (1.0000)
0.5673
0.5854
885 (1.0000)
0.7810
0.7421
Ki for Target Carbonic anhydrase I (R3)
1/3]
(0.0079)
0.0634
0.1421
264 (0.2983)
-0.4131
-0.1076
(1/3, 2/3]
205 (0.2316)
0.3536
0.3761
311 (0.3514)
0.2106
0,2829
(2/3, 1]
673 (0.7605)
0.4500
0.4231
310 (0.3503)
0.3532
0.3299
overall
885 (1.0000)
0,4259
0.4161
885 (1.0000)
0.1253
0.2334
Ki for Target Carbonic anhydrase II (MAE)
Randomized Split
SAE (balanced) Split
Bin
Count (Ratio)
FusionDTA ChemBERTa
Count (Ratio)
FusionDTA ChemBERTa
1/3]
(0.0087)
0.8465
0.5778
244 (0.2667)
0.9849
0.9314
(1/3, 2/3]
201 (0.2197)
0.6817
0.7419
342 (0.3738)
0.8265
0,7390
(2/3,
1]
706 (0.7716)
0.5605
0.5997
329 (0.3596)
0.6040
0.6072
overall
915 (1.0000)
0.5896
0.6307
915 (1.0000)
0.7888
0.7429
Ki for Target Carbonic anhydrase II (R?)
1/3]
(0.0087)
0.0349
0.3581
244 (0.2667)
0.0488
0.2523
(1/3, 2/3]
201 (0.2197)
0.5603
0.4667
342 (0.3738)
0.3416
0.4686
(2/3, 1]
706 (0.7716)
0,5513
0.5146
329 (0.3596)
0.4499
0.4744
overall
915 (1.0000)
0.5570
0.5087
915 (1.0000)
0.3583
0.4659
26

---

Published as a conference paper at ICLR 2025
Table A.8:
Comparison of Randomized Split and SAE (balanced) Split ICS0 for BACEI, Ki for Carbonic anhydrase and Carbonic anhydrase II. The method for the detailed showcase is MolCLR.
IC50 for Target BACEI (MAE)
Randomized Split
SAE (balanced) Split
Bin
Count (Ratio)
MoICLR
Count (Ratio)
MolCLR
[0 1/3]
10 (0.0108)
1.2452
309 (0.3330)
1.4141
(1/3, 2/3]
67 (0.0722)
0.6952
311 (0.3351)
0.6940
(2/3, I]
851 (0.9170)
0.4878
308 (0.3319)
0.4784
overall
928 (1.0000)
0,5109
928 (1.0000)
0.8622
IC50 for Target BACEI (R²)
[0 1/3]
10 (0.0108)
-0.0492
309 (0.3330)
-0.9211
(1/3, 2/3]
67 (0.0722)
0.6184
311 (0.3351)
0.5107
(2/3, I]
851 (0.9170)
0.6919
308 (0.3319)
0.7746
overall
928 (1.0000)
0.6974
928 (1.0000)
0.3713
Ki for Target Carbonic anhydrase (MAE)
Randomized Split
SAE (balanced) Split
Bin
Count (Ratio)
MolCLR
Count (Ratio)
MolCLR
[0 1/3]
(0.0079)
0.8510
264 (0.2983)
1.0059
(1/3, 2/3]
205 (0.2316)
0.6141
311 (0.3514)
0.7549
(2/3, I]
673 (0.7605)
0.4762
310 (0.3503)
0.5755
overall
885 (1,0000)
0.5111
885 (1.0000)
0.7669
Ki for Target Carbonic anhydrase (R²)
[0 1/3]
(0.0079)
0.3127
264 (0.2983)
-0.3331
(1/3, 2/3]
205 (0.2316)
0.5338
311 (0.3514)
0.1585
(2/3, 1]
673 (0.7605)
0.5598
310 (0.3503)
0,4342
overall
885 (1.0000)
0.5572
885 (1.0000)
0.1552
Ki for Target Carbonic anhydrase II (MAE)
Randomized Split
SAE (balanced) Split
Bin
Count (Ratio)
MoICLR
Count (Ratio)
MolCLR
[0 1/3]
8 (0.0087)
1.0278
244 (0.2667)
0.8873
(1/3, 2/3]
201 (0.2197)
0.6882
342 (0.3738)
0.6907
(2/3, I]
706 (0.7716)
0.5497
329 (0.3596)
0.6232
overall
915 (1.0000)
0.5843
915 (1.0000)
0.7189
Ki for Target Carbonic anhydrase II (R?)
[0 1/3]
(0.0087)
-0.3461
244 (0.2667)
0.2192
(1/3, 2/3]
201 (0.2197)
0.5635
342 (0.3738)
0.5347
(2/3, I]
706 (0.7716)
0,5964
329 (0.3596)
0,4409
overall
915 (1.0000)
0.5856
915 (1.0000)
0.4740
27


Reviews:
[{"review_id": "SQS3F0OGLl", "reviewer": "Reviewer_C9Wr", "summary": "The paper focus on analysing the evaluation methods on Drug-Target binding Affinity (DTA) prediction. The paper first shows that state-of-the-art methods for drug target affinity prediction tend to perform better on drugs that are similar to the drugs used to train the models, than on those with a lower similarity. Authors also shows that current sampling strategies used for the evaluation of DTA tend to report a performance value that is much closer to the similar drugs in training, while the performance for dissimilar drugs is much worst. \n\nAuthors then propose a split methodology called Similarity Aware Evaluation (SAE). The proposed split strategy is capable of reflect any desired distribution, by formulating the problem as an optimisation problem, that can be approximated by gradient decent.", "questions": "What exactly do authors mean when writing \"samples are allowed to coexist in trainset and testset\"? (lines 84 and 311) One possible interpretation of this sentence could be that a sample can be used for training the model and later also used to validate the performance. This seems not to be the right interpretation given several parts of the mathematical section of the paper, yet this sentence made me wonder if this is indeed the case. \n\nIn terms of reporting performance, what are the advantages of using SAE over reporting the performance on the different similarity brackets (as done in Figure 1, left column)? This would allow to better pick the model that better suits the similarity bracket instead of having one model for all the drugs.\n\nRelated to the previous question, it would be interesting to see the performance on the different brackets in the external dataset of figure 5. Does using SAE improves the performance across all the similarity brackets?\n\nThe research topic could also be extended to other QSAR tasks, and while not needed, a small commentary about this could enrich section \"4.3 Other Applications\".\n\nDo authors consider providing the code implementing SAE, and the code to replicate the experiments?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper presents thorough evaluations on current SOTA methods for DTA, showing that performance is quite skewed by similar drugs used for training. It clearly illustrates the difference in performance at different levels of drug similarity, under different representations and with several similarity measures.\n\nThe proposed similarity aware evaluation and the math behind it is well explained and overall easy to understand. \n\nThe experiments to validate the proposed evaluation are reasonable. The influence and impact of the parameters controlling it are reasonably justified.", "weaknesses": "The paper needs to be proof read for English. While the main ideas in the mathematical part can be understood, the paper writing needs to be improved. (too many missing articles, for example)\n\nWhile the results presented by the authors do indeed show that current DTA prediction methods tend to perform better on drugs with high similarity comparing to the ones used for training and poorly on the ones with low similarity, this fact is hardly novel or surprising. The influence and challenges of similarity in drug related tasks has been previously documented, but not referenced in the paper. In QSAR related literature this is a known fact, and other areas predicting quantitative properties of drugs do control for this. For example:\n\nSheridan, Robert P., et al. \"Similarity to molecules in the training set is a good discriminator for prediction accuracy in QSAR.\" Journal of chemical information and computer sciences 44.6 (2004): 1912-1928.\n\nIn other areas, such as Drug-Target Interaction prediction this is also often reported by measuring performance on dissimilar drugs in the testing set, for example:\n\nLuo, Yunan, et al. \"A network integration approach for drug-target interaction prediction and computational drug repositioning from heterogeneous information.\" Nature communications 8.1 (2017): 573.\n\nWan, Fangping, et al. \"NeoDTI: neural integration of neighbor information from a heterogeneous network for discovering new drug–target interactions.\" Bioinformatics 35.1 (2019): 104-111.\n\nThis is similar to what authors report on Figure 1 on the left column. All being said, authors do a good job quantifying and reporting the difference in performance with respect to the similarity. \n\nThe related works section itself is reduced and quite compressed, but most of the related work is covered in the introduction.\n\nIn the abstract and in the text, the use of the word “target” in the sentence “target distribution” is confusing in the context of this paper, since it could be easily confused with “target protein distribution”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gyATbpVKt8", "reviewer": "Reviewer_dEEq", "summary": "This paper addresses a critical limitation of evaluation schema in current drug-target affinity (DTA) prediction models. Traditional methods often employ a randomized dataset split, leading to test sets dominated by samples similar to the training set. This approach causes inflated model performance on high-similarity samples while overlooking the degradation in performance on low-similarity samples, which are common in real-world scenarios. To address this, the authors propose a novel \"Similarity Aware Evaluation\" (SAE) framework, which introduces an optimized splitting methodology to better match target distributions across similarity bins. This paper demonstrates that SAE enables a more accurate assessment of model generalizability, thereby enhancing performance on external test sets and improving robustness in real-world applications.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces the Similarity Aware Evaluation (SAE) framework, an innovative methodology for creating test sets that better represent real-world data distributions. \n- The authors conduct extensive experiments across multiple datasets and state-of-the-art models, demonstrating the widespread impact of similarity bias.", "weaknesses": "- The continuous relaxation and hyperparameter tuning required in the SAE framework might introduce additional computational complexity and practical challenges for large datasets or high-throughput applications. Could you provide further clarification on the time complexity of the SAE framework, especially in terms of how these processes impact scalability and efficiency in such contexts?\n- The performance of the proposed SAE framework likely varies depending on the similarity measurement used, such as Tanimoto or cosine similarity, and may also be influenced by the choice of fingerprint (e.g., ECFP, MACCS, rdikit). A discussion comparing the performance of the SAE framework across these different similarity measures and fingerprints would provide a more comprehensive understanding of its effectiveness.\n- While the paper compares SAE with randomized and scaffold splits, it could benefit from a more detailed analysis against other existing methods tailored for similarity-based evaluations, such as time-based, stratified, and cold-drug splits. To provide a clearer benchmark, it would be helpful to include a comparison table showing the performance of SAE against these methods on a common dataset. Additionally, comparisons with specific methods [1,2,3] could be included, or an explanation could be provided for why certain comparisons may not be applicable. This would allow for a more comprehensive evaluation of SAE's effectiveness relative to existing approaches.\n\n[1] Pahikkala, Tapio, et al. \"Toward more realistic drug–target interaction predictions.\" Briefings in bioinformatics 16.2 (2015): 325-337.\n\n[2] Nguyen, Tri Minh, Thin Nguyen, and Truyen Tran. \"Mitigating cold-start problems in drug-target affinity prediction with interaction knowledge transferring.\" Briefings in Bioinformatics 23.4 (2022): bbac269.\n\n[3] Atas Guvenilir, Heval, and Tunca Doğan. \"How to approach machine learning-based prediction of drug/compound–target interactions.\" Journal of Cheminformatics 15.1 (2023): 16.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fQghHpZwNj", "reviewer": "Reviewer_PwZH", "summary": "This paper targets the performance evaluation of drug-target binding affinity prediction, focusing on analyzing effective data partitioning into training and validation sets. The paper proposes a similarity-aware evaluation (SAE) as an alternative to the conventional random split approach. Recognizing that the similarity of test set molecules to those in the training set significantly impacts machine learning prediction performance, the paper first categorizes validation set molecules into three groups based on their similarity scores with training set molecules. Through extensive experimentation, the paper demonstrates (1) that with random splitting, a large portion of the validation set is occupied by molecules similar to those in the training set, regardless of the feature representation used, and (2) that the overall prediction accuracy aligns closely with the prediction accuracy for training-set-similar molecules dominating the validation set.\n\nThe paper argues that, in real-world external test scenarios, there is no inherent focus on molecules similar to those in the training set, and hence, an evaluation approach that is independent of similarity distribution is preferable. However, this cannot be achieved through traditional random splitting. The paper then introduces SAE as a sampling strategy that considers similarity bins and optimizes the scaled distance between the similarity distribution within each bin and a uniform distribution, achieving combinatorial optimization. Given the practical challenges of this discrete combinatorial optimization, the authors propose an approximate solution using continuous relaxation.\n\nThe experimental results indicate that SAE’s continuous relaxation-based sampling yields a uniform similarity distribution in the test set, providing a more accurate estimate of model performance in testing. Additionally, the paper demonstrates that when the distribution of an external test set is known, the optimization can be adapted to align with this specific distribution instead of a uniform one.", "questions": "Have you tried a direct approach like this (stratified sampling), where the dataset is first divided into K bins based on similarity to other molecules (stratification), and then random sampling (random splitting) is applied within each bin to create a test set that is uniform at  similarity-based bins?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper is highly readable, with well-organized arguments, making it a thought-provoking study not only for the specific task of DTA but also for general evaluation considerations in molecular machine learning.\n- Instead of focusing on prediction methods, this paper addresses the critical issue of how machine learning predictions should be evaluated appropriately. For tasks like drug-target binding affinity (DTA) prediction, practical utility is the key metric, and the gap between benchmarking scores and real-world applicability has been a long-debated issue. It is valuable that such research is presented at venues like ICLR.\n- In standard molecular task benchmarking, prediction performance is often evaluated using random split (or scaffold split, as also compared in the paper) for dividing training/validation/test sets. The authors support with data that if an evaluation independent of similarity to the training set is desired, current evaluation practices of using random split or scaffold split may yield an overly optimistic assessment.\n- Specifically for the DTA task, the authors empirically demonstrate that regardless of the similarity metric, dataset, prediction target (e.g., IC50 or Ki), or molecular feature representation, as long as random split is used, (1) molecules similar to those in the training set will dominate the validation set, and (2) consequently, only the prediction accuracy for training-set-similar molecules affects the overall performance score. The prediction accuracy for low-similarity molecules, which are fewer in number, is often overlooked. However, considering the actual needs in DTA, this point cannot be ignored.\n- As an alternative to the random split, the authors propose a similarity-aware evaluation (SAE), defining it as a combinatorial optimization problem that considers similarity bins and aims to create a uniform distribution within each bin. They also propose a practical solution for SAE using continuous relaxation to approximate the discrete combinatorial optimization.", "weaknesses": "- The technical novelty of the work is relatively limited, as the proposed SAE is essentially a specific case of \"stratified sampling\" from statistics. The \"similarity bins\" that underpin SAE effectively serve as strata defined by similarity, meaning that if the dataset is pre-divided into strata based on molecule similarity, applying random sampling within each stratum would naturally achieve uniformity across similarity bins. Therefore, it seems feasible to implement this approach more directly without relying on the combinatorial optimization and continuous relaxation proposed by the authors. From this perspective, the combinatorial optimization formulation may appear to overly complicate the problem.\n- In molecular machine learning, including general QSAR tasks, the challenge of fair predictive evaluation has been a longstanding issue. For instance, it was highlighted as a concern for inadequate QSAR model validation over a decade ago [1]. Additionally, this challenge is closely related to out-of-distribution (OOD) generalization [2], and thus it is not exclusive to the DTA task. In [2], a similar analysis to this paper is presented, introducing a \"deployment-to-train\" distribution-based splitting protocol that resembles the approach proposed in this paper. Note that [2] does not propose any new splitting method, which does not diminish the contribution of this paper.\n  - [1] Cherkasov A, Muratov EN, Fourches D, Varnek A, Baskin II, Cronin M, Dearden J, Gramatica P, Martin YC, Todeschini R, Consonni V, Kuz'min VE, Cramer R, Benigni R, Yang C, Rathman J, Terfloth L, Gasteiger J, Richard A, Tropsha A. QSAR modeling: where have you been? Where are you going to? J Med Chem. 2014 Jun 26;57(12):4977-5010.\n  - [2] Tossou P, Wognum C, Craig M, Mary H, Noutahi E. Real-World Molecular Out-Of-Distribution: Specification and Investigation. J Chem Inf Model. 2024 Feb 12;64(3):697-711. doi: 10.1021/acs.jcim.3c01774.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WW2P0Kkd7D", "reviewer": "Reviewer_MUd6", "summary": "This paper studies the problem of generalization in the problem of predicting of drug target binding for drug repurposing, where the affinity score to a specific target is computed on a library of drugs. This paper first draws attention through empirical results to the fact that good performance on a testing set which low-similarity points are underrepresented might be misleading, as the trained model might fail in a real-life setting. Second, the paper introduces an approach to splitting the training and testing subsets for the training of models, that aims at achieving a target distribution on points in the testing set, that might be less biased for the evaluation of a prediction model.", "questions": "- Could you provide the code for your method or explain why it is not included? Having access to the implementation would greatly aid reproducibility.\n- People often split the data into three subsets: training and validation sets (to apply the method on the training set and use the metrics computed on the validation set to update parameters) and testing set (to evaluate the performance of the method). In that case, should the SAE method be applied iteratively? As it is, it can only split data into two groups. \n- For Figure 6, could you specify the exact target distribution used to generate these plots, including any constraints or parameters?\n\nSummary of suggested modifications:\n- Please provide experimental code or justify why it is not included. \n- Mention how the approach could be adapted for three-way splits, or what modifications would be needed to support this common scenario.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Clarity: The submission is experimentally rigorous, and well-written. \n- Quality: Several interesting claims are made and illustrated by relevant experiments (for instance, that performance on the testing set is correlated with the proportion of low-similarity samples and that those are often a small part of the dataset). The tested baselines are relevant, and the impact of all relevant hyperparameters is appropriately assessed.\n- Significance: The question is important, in particular for healthcare applications, and the algorithmic contribution is valuable, as it is a tractable approach (optimization-wise). \n- Originality: The algorithm proposed to splitting data can handle any target distribution.", "weaknesses": "- Clarity: The section on related works should be present earlier in the paper. \n- Quality: The results shown are not easily reproducible, as the code is not shared. Some aspects to make the method truly appealing to the end users (training for prediction models) are missing: the computational cost of the approach (the computational resources needed to run the experiments), the empirical runtime on the data set of approximately 4,000 samples for instance, and how it compares for these aspects to the selected baselines (random, scaffold, SIMPD). The impact of the number of bins K on the closeness between solution distribution and target distribution is not discussed. \n- Significance: There seems to be a significant gap between the target distribution and the one returned at the end (Figure 6 : 20% of points are present in bins that should remain empty), which might be due to the several approximations made to obtain a tractable optimization problem. I assume that the approach to obtain this figure is to set to zero the number of elements expected in bins associated with similarity ranges higher than the prescribed threshold, but this is not specified in the paper. Finally, the approach relies on the precomputation of a similarity matrix that might be too large to be stored in-place. All the data sets in the paper are of size less than 5,500, but in pharmacompany (private) drug libraries for early drug discovery, there might be 200,000 to $10^6$ compounds [1].\n\n[1] Hughes, James P., et al. \"Principles of early drug discovery.\" British journal of pharmacology 162.6 (2011): 1239-1249.\n\nEven though I rated the paper 8, I think the reproducibility is the largest issue and the absence of code is not justified in the paper. This is a major concern to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)