A new framework for evaluating model out-of-distribution generalisation for the biochemical domain
Authors: Raul Fernandez-Diaz
Hoang Thanh Lam
Vanessa López
Denis C. Shields
Venue: ICLR 2025 Poster (2025)
Keywords: Machine learning evaluation
AI4Science
Biochemistry
Proteins
Small molecules
Protein Language Models
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We present a new metric for estimating the the expected performance of a model against any specific target deployment distribution(s)

Abstract: Quantifying model generalization to out-of-distribution data has been a longstanding challenge in machine learning. Addressing this issue is crucial for leveraging machine learning in scientific discovery, where models must generalize to new molecules or materials. Current methods typically split data into train and test sets using various criteria — temporal, sequence identity, scaffold, or random cross-validation — before evaluating model performance. However, with so many splitting criteria available, existing approaches offer limited guidance on selecting the most appropriate one, and they do not provide mechanisms for incorporating prior knowledge about the target deployment distribution(s).

To tackle this problem, we have developed a novel metric, AU-GOOD, which quantifies expected model performance under conditions of increasing dissimilarity between train and test sets, while also accounting for prior knowledge about the target deployment distribution(s), when available. This metric is broadly applicable to biochemical entities, including proteins, small molecules, nucleic acids, or cells; as long as a relevant similarity function is defined for them. Recognizing the wide range of similarity functions used in biochemistry, we propose criteria to guide the selection of the most appropriate metric for partitioning. We also introduce a new partitioning algorithm that generates more challenging test sets, and we propose statistical methods for comparing models based on AU-GOOD.

Finally, we demonstrate the insights that can be gained from this framework by applying it to two different use cases: developing predictors for pharmaceutical properties of small molecules, and using protein language models as embeddings to build biophysical property predictors.

Published as a conference paper at ICLR 2025
A NEW FRAMEWORK FOR EVALUATING MODEL OUT- OF-DISTRIBUTION GENERALISATION FOR THE BIO- CHEMICAL DOMAIN
Raúl Fernández-Diar
Thanh Lam Hoang & Vanessa Lopez IBM Research Dublin, D02 PN40, Ireland. {t. hoang, vanlopez]e ie. 1.m. com
School of Medicine
Conway Institute Biomolccular and Biomedical Research University College Dublin
Dublin, DO4 VI V8W8, Ireland.
IBM Research Dublin, D02 PN40, Ireland.
wew.rrnneeeiaaaaaccceeeeeee .ee
Denis C. Shields
School Medicine
Conway Institute of Biomolecular and Biomedical Research
University College Dublin
Dublin, D04 VIW8, Ireland.
denis shie l ds@ucd.
ABSTRACT
Quantifying model generalization to out-of-distriutiion data has been a longstand- ing challenge machine learning. Addressing this issue is crucial for leveraging machine learning in scientific discovery, where models must generalize to new molecules or materials. Current methods typically split data into train and test sets using various criteria temporal, sequence identity, scaffold, random cross-validation - before evaluating model performance. However, with so many splitting criteria available, existing approaches offer limited guidance selecting the most appropriate one, and they do not provide mechanisms for incorporating prior knowledge about the target deployment distribution(s).
To tackle this problem, we have developed novel metric, AU-GOOD, which quantifies expected model performance under conditions of increasing dissimilar- ity between train and test sets, while also accounting for prior knowledge about the target deployment distribution(s), when available. This metric is broadly applica- ble to biochemical entities, including proteins, small molecules, nucleic acids, or cells; long relevant similarity function is defined for them. Recognizing the wide range of similarity functions used in biochemistry, we propose criteria to guide the selection of the most appropriate metrie for partitioning. We also intro- duce a new partitioning algorithm that generates more challenging test sets, and we propose statistical methods for comparing models based AU-GOOD.
Finally, we demonstrate the insights that can be gained from this framework by applying it to two different use cases: developing predictors for pharmaceutical properties of small molecules, and using protein language models as embeddings to build biophysical property predictors.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
The last decade has been characterised by the impact that the introduction of machine learning mod- els has had in the acceleration of scientific discovery. These models are frequently used to predict the properties of entities (drug candidates, materials, cells, etc.) that are inherently different from those present their training distribution. This deployment scenario, known as out-of-distribution (OOD), is particularly frequent within the biochemical domain which encompasses both biological and chemical modelling. Proper OOD evaluation of models is necessary, within this domain specif- ically, due the enormous economic and societal impact that wrong predictions might have, for example, when a drug candidate that was predicted as non-toxic (Huang et al., (021a) goes through the latter phases of the drug development pipeline, including pre-clinical clinical trials. Further tasks where OOD evaluation is critical are: modelling of the interaction between a known molecular target and new compounds (Corso et al. 2024p, the prediction of the structure of molecular targets mediating disease (Mirdita et al.j 2022), or cell type annotation (Fischer et al. 2024). Overall, ro- bust OOD evaluation necessary for the development of trustworthy predictive models, driving advancements in biochemistry.
Prior literature in the biochemistry domain, has already highlighted the importance of evaluating model generalisation to OOD data, though there is no prior work attempting to build framework for both measuring the generalisation capabilities of any given model and to provide a statistical method to compare the performance of different models. Instead, prior works have focused on developing algorithms for: i) measuring pairwise similarity values between biochemical entities (Steinegger & Söding, 2017 Rogers & Hahn| (2100) Orrsi & Reymond, (202)) ii) splitting dataset into independent train-test subsets (Butina) 1999| Ramsundar et al., 2019 Tom et al. 2023: Steshin
2023 Teufel et al., 2023), iii) combining different partitioning algorithms simulate the compo- sition of a single target deployment distribution (Tossou et al., 2024), and iv) exploring the effect of overlap between training and testing on model performance (Ektefaie al.) (024).. Due to the absence of a comprehensive framework for evaluating model generalisation to out-of-distribution (OOD) data, validating claims about machine learning generalisation remains challenging.
In response to this gap. we first present a framework to study and quantify model generalisation to OOD data for biochemistry. We propose novel partitioning algorithm that, unlike existing methods (Butina 1999; Ramsundar et al.) 2019] Tom et al. 2023, Steshin 2023 Teufel et al.) 2023), is broadly applicable across various biochemistry contexts, including proteins and small molecules. Our approach is agnostic to the underlying data types; instead, it relies on any similarity functions that can be defined between two data instances. In contrast to previous methods that choose a certain similarity function assuming that it can capture generalisation within the context of a certain modelling task (Butina 1999 Ramsundar et al.. 2019] Tom et al., 2023 Steshin 2023] Teufel et al.) 2023 Tossou al.) 2024| Ektefaie et al., 2024). our framework provides quantitative measurements
indicating how appropriate a similarity metric is for guiding the partitioning of a certain dataset, with minimal relience domain knowledge. We also propose new similarity functions based on physico-chemice descriptors, rather than structure- strueture-based fingerprints and a statistical metric to compare the generalisation capabilities of different models.
2 HESTIA-GOOD FRAMEWORK
This section presents the Hestia- GOOD framework for quantifying out-of-distributin generalisation in biochemistry. It is divided into three subsections.
1. We present the AU-GOOD metric for quantifying model generalisation to target deploy- ment distribution(s). This metric corresponds with the expected empirical risk of model trained on a given dataset with varying train/test similarity conditioned on any arbitrary target deployment distribution(s).
2. We propose an algorithm for dataset partitioning that i) does not remove any data points, ii) creates strict boundaries between training and testing, and iii) supports multiple similarity thresholds.
3. We propose a statistical test to compare the generalisation capabilities of two different models.
2

---

Published as a conference paper at ICLR 2025
2.1 AU-GOOD METRIC FOR QUANTIFYING MODEL GENERALISATION THE TARGET DEPLOYMENT DISTRIBUTION(S)
We consider the problem of estimating model performance against any arbitrary target deployment distribution(s) that are out-of-distribution of the data available during model training. Our approach consists on three steps:
1. Partition the available data (training data) into training/testing subsets with the condition that there are no elements in the testing set that are similar (up to a threshold value) to those in training. Multiple partitions are created by applying different thresholds to this similarity.
2. The model is trained in each training subset and its performance evaluated against the corresponding testing subset. Then, we describe model performance as function of the similarity threshold. We refer to this function as the Generalisation Of-iistribution (GOOD) curve.
3. Given any number of arbitrary target deployment distribution(s), we calculate the similarity between each of its elements and the training data. For each element in the deployment distribution, we keep the maximum similarity and calculate a histogram of their distribution such that each bin corresponds to one of the thresholds in the GOOD curve. The AU- GOOD, or Area Under the GOOD curve, can be calculated as the weighted average of all points in the GOOD curve where the weight of each point corresponds to the value of the histogram for the corresponding threshold.
Formally, we demonstrate that the AU-GOOD metric corresponds to the expected empirical risk of the model conditioned on the target deployment distribution (see Appendix A).
Illustration AU-GO00 calculation
A. Enamplex GOOD
B.. Dnumpte statierity distribotton Torpet
Doumple All-GOOD with Target
0.99
Nodeta
Soald A
0.94
Models
Naki
32
83
2.4
6.0 tratning testing
8.0
65.
64
6.5
60
1.0
09
0.2
3.2
8.5
Soniaattl
training and target ditetbbttion
Eooupple sonclaatts Darget
Exammplee Target
Hodel
100
Mastmal strallrrity bebrows
Stntlartyy threshold
Figure 1: Illustration of the calculation of the AU-GOOD metric. A-C correspond to synthetic data.
To illustrate how the AU-GOOD metric is calculated and what insights can be drawn from its anal- ysis, let us consider a simple simulation, which will serve to illustrate the relationship between GOOD curve, target distribution(s) and AU-GOOD. Let us consider two models, "Model A" and "Model B", with GOOD curves as shown Figure A In Figure A Model A performs better when predicting molecules similar to the training data, whereas for Model B there is comparatively better performance at low similarity values.
Let us now consider two different target deployment distributions, "Target 1" and "Target 2". The histograms reflecting their similarity to the training data are shown in Figures B-C. Target (Figure B) comprises entities with low similarity to the training data, while Target (Figure C represents the opposite case, where most entities are similar to those the model has been trained on.
The AU-GOOD metric allows for the evaluation of model generalisation relative to a target deploy- ment distribution. Therefore, it allows us to determine that, though apparently Model has a more consistent performance through different similarity thresholds, Model A is better choice for Target
3

---

Published as a conference paper at ICLR 2025
2, which is skewed towards molecules similar to the training data. Conversely, Model B is better choice for Target 1.
This simple example also highlights one of the main strengths of the AU-GOOD metric, which is that it allows to obtain estimators of model generalisation for an arbitrary number of target de- ployment distributions without the need for additional experiments. This is significant advantage when compared with other approaches for out-of of-dissribution performance estimation, like MOOD (Tossou al. 2024) which focuses on generating test sets that closely resemble the target deploy- ment distribution and requires repeating the training/evallation cycle for each new target deployment distribution; whereas with ours, only the similarity histogram needs be updated. Our approach is also similar to Ektefaie et al.] (2024) with three main differences: 1) their method defines ar- bitrarily certain similarity threshold and considers model performance against the proportion of similar entities between training and testing; whereas, our approach defines a certain proportion of similar entities (0%) and explores the effect on model performance of the different similarity thresholds; 2) their method assumes that whatever similarity function they have chosen is good at capturing the relevant properties that mediate any given task; however, our method makes no such assumption and we provide series of analyses to determine that GOOD curve reasonably de- scribes
generalisatio; and 3) their approach does not provide any estimation of model performance conditioned the target deployment distribution(s), which limits its practical application in real-world scenarios.
Finally, our study is limited by the lack of experimental follow up studies to confirm empirically the advantages of our method. The closest study to ours in this regard are the aforementioned MOOD from (Tossou al. (2024) and Spectra (Ektefaie al. 2024), neither of which provided metric to evaluate the approach on "ground truth" data. Development of scenarios where such a ground truth exists will help inform evaluation of the performance of our method.
2.2 DATASET PARTITIONING TO SIMULATE SIMILARITY VALUE
DATA GIVEN A
The AU-GOOD values can also be affected by the sampling performed during the partitioning step, as it may bias the composition of the test set. There are three conditions that the partitioning al- gorithms have to fullfil are: 1) not remove any data point, otherwise the comparison between thresholds could also correspond to the loss of information and would make them not directly com- parable: 2) to enforce strict boundaries between training and testing subsets for any given similarity threshold (Walsh et al. 2020). otherwise would not be measuring strictly the dependence on the similarity threshold, but also (implicitly) the effect of overlap within any given threshold; and 3) to allow for multiple thresholds, otherwise cannot describe model performance as a function of the threshold.
All prior partitioning algorithms described for biochemical data violate at least one of the three conditions: perimeter and maximum dissimilarity do not fulfill condition 2 and scaffold split does not fulfill condition 3 (Ramsundar et al. (2019), cluster splits relying on hierarchical clustering (Tom) let al., (023) do not fulfill conditions 2 and 3: GraphPart (eefell et al. 2023) or Lo-Hi splitting (Steshin 2022) do not fulfill condition 1. The Butina split (Butina) 1999) generates clusters where every member of a given cluster is guaranteed to be closer to the centroid than the threshold, however the frontiers between clusters are not strict, and therefore does not fulfill condition 2.
Here, we propose the CCPart (Connected Components Partitioning) new strategy for partitioning the datasets that fulfills all three previous conditions and that is extensible to any biochemical data. The algorithm first constructs graph, where the nodes are all elements in the datasets and the edges are drawn between nodes with similarity greater than the threshold. It then identifies all unconnected subgraphs. The testing set is iteratively built by assigning the smallest subgraphs until the desired size is reached. Smaller subgraphs are considered more unique. CCPart thus focuses the evaluation on the most dissimilar regions of the dataset distribution. Additionally, stratified sampling can be applied to maintain label balance in classification tasks. Figure 2 illustrates this process (see Algorithm STI.
In any case, our framework is agnostic to the specific partitioning strategy used and future devel- opment of alternative algorithms that fulfill the three conditions could improve the accuracy of the
4

---

Published as a conference paper at ICLR 2025
Similarity Threshold
Traning
Tewingg
Figure 2:
Schema showing how CCPart builds the training and testing subsets based on the similarity graph at different similarity thresholds.
AU-GOOD calculation. also an open research question what the impact is violating any of the three assumptions the reliability of the AU-GOOD metric.
2.3 STATISTICAL TEST FOR COMPARING AU-GOOD VALUES
Generally, comparison between the statistical significance of the difference in performance between two given models made by comparing their confidence intervals. This standard practice, implicitly assumes that performance is normally distributed. However, the performance values at different thresholds cannot be assumed to be normally distributed. Therefore, the analysis of the statistical significance between individual AU-GOOD values requires different kind of statistical test. We propose to use the non-paaaect equivalent to the one-tailed T-test, the one-tailed paired Wilcoxon ranked-sum test Wilcoxon (945);; Ash et (2024). Finally, we want to compare the performance of n. models, n X pairwise tests can be performed. In cases where we are performing multiple tests (n 5), the Bonferroni correction for multiple testing (Bonferroni] 1936) can be applied so that the significance threshold depends on the number of models: 0.05/n n (Ash et al. 2024
2.4 HESTIA: COMPUTATIONAL EMBODIMENT
The partitioning algorithms rely on pairwise similarity calculations and scale in complexity and memory footprint by (N In response to this, our implementation improves computational and memory efficiency by relying sparse matrices and parallelism. We have run toy examples with datasets of up to 400,000 protein sequences within day in 32 CPU cores and 200GB of RAM, thus believe that scaling does not limit the usefulness of the method within the domain. In other domains, algorithmic improvements might be required for scaling to very large datasets.
We have open soureed the code including: 1) wrappers for most common similarity funetions be- tween a) small molecules, b) biosequences including protein and nucleic acids, c) protein structures, and d) embeddings from pre-trained representation learning models; 2) implementation of differ- ent partitioning algorithms such as: a) GraphPart and b) CCPart; and 3) calculation of AU-GOOD values. More details in Appendix
EXPERIMENTS
We have chosen two different real-world use cases to demonstrate the insights that can be drawn from using our framework for the comparison of ML models within the biochemical domain. The first one is the selection of predictive models for pharmaceutical properties given a target deployment distribution and the second is the selection of what protein language model embedding to use for different target deployment distributions.
3.1
USE CASE 1: PREDICTION OF PHARMACEUTICAL PROPERTIES OF SMALL MOLECULES
ADMET (Absorption, Distribution, Metabolism, Excretion, and Toxicity) properties encompass dif- ferent biophysical attributes of small molecules that may affect both their pharmacokinetic and phar- macodynamic behaviour within the organism (Wang et al. 2024). The estimation of which model will generalise better to which target deployment distribution, will allow researchers decide what model to use for each particular project. For the purposes of this study, we have selected the
Github Repository: https:://ithub.coo com/ HM/ WM/Heatin GOOD
5

---

Published as a conference paper at ICLR 2025
Drug Repurposing Hub (Corsello et al. 2017) as the library of choice as it represents realistic scenario with known approved drugs.
We examine our framework with six datasets containing different ADMET properties from the Ther-
apeutics Data Commons collection (Huang et al. 2021b): Ames' mutagenicity (Xu et al.) 2012)) cell
effective permeability in Caco-2 cell line (Wang et al. 2016), drug-induced liver injury (DILI) (Xu
et al. 2013), acute toxicity LD50 (Zhu et al. 2209), drug half-life (Obach et al.) 2008). and parallel
artificial membrane permeability assay (PAMPA) (Siramshetty et al., 2021). We report model per- formance with Matthew's 's correlation coefficient for classification tasks (Ames, DILI, and PAMPA)
and Spearman's 's correlation coefficient for regression tasks (Caco2, Half-life, and LD50).
Each experiment described uses as featurization method Extended-CConnectvvte FingerPrints (ECFP) with radius 2 and 2,048 bits, as they are a well-known baseline for all the datasets under consideration (Huang et al. 202 Each experiment consists of 5 independent runs with different seeds. Each run for a dataset and similarity metric has the exact same partitions, because the par- titioning algorithm is heuristic and not stochastic, therefore any variance between runs depends on the learning algorithm and the Bayesian hyperparameter-optiiisttoo which conducted for each run independently with Optuna (Akiba et al. 2019) (see Table $3
3.1.1 WHAT IS THE BEST SIMILARITY FUNCTION?
Experimental setup
The first step for using the Hestia-GOOD framework is to determine what is the optimal similar- ity function to use. We evaluated three types of fingerprints: ECFP with radius 2 and 2,048 bits, MACCS, and chiral Min-hashed atom pair (MAPc) with radius and 2,048 bits. We considered binary set similarity functions for the binary fingerprints (ECFP and MACCS): Tanimoto (Rogers & Tanimoto 1960). Dice (Dice 1945), Sokal (Sokal & Michener 1958). and Rogot-Goldberg (Rogot & Goldberg 1966). We also considered cosine similarity as an alternative geometry-based similarity function. For MAPe, which is not a binary fingerprint, we only considered the Jaccard similarity. Additionally, we have also considered alternative similarity metrics that are not fingerprint-based including: the euclidean distance between embeddings from two pre-trained chemical language models: Molformer (Ross et al.) 2022) and ChemBERTa-2 (77M MLM) (Ahmad et al. 2022); the canberra distance between two types of vectors of physicochemical descriptors 1) BUTCD (Stanton 1999) and Lipinski (Lipinski al. 1997). More details in Appendix C
Criteria for selecting similarity function
We observed that different similarity functions led to GOOD curves with notably different shapes (see Appendix
- (D). The two properties of the curves that experienced the biggest variance across similarity functions were 1) their dynamic range, i.e., the difference between the minimum and max- imum similarity thresholds that could be used to generate viable partitions with test sets with at least 18.5% of the total dateP) and 2) their monotonicity, measured as the Spearman's 's p correlation coef- ficient between similarity threshold and model performance. A necessary assumption of the GOOD and AU-GOOD calculations is that model performance can be described ilarity threshold. Therefore, Spearman's 's correlation coefficient serves as a robust, non-parametric diagnostic test to ensure that any GOOD curve meets this assumption, without assuming linearity.
function of the sim-
We propose to use both metrics as selection criteria:
1. Rank all similarity functions by their largest dynamic range and choose the top functions.
2. Compare the monotonicity of the GOOD curves generated with those similarity functions and select the similarity function with the highest monotonicity.
The first criterion aims to produce GOOD curves that span greater similarity range to minimise the impact of noise. The second focuses on improving discrimination between low and high similarity thresholds for better evaluation of model generalization. It also serves as a diagnostic test to ensure that the GOOD curves meet the assumption of model performance dependence on the similarity thresholds. If none of the similarity functions achieves reasonable monotonicity, it is likely that
We keep the conventional 20% value and allowed for 1.5% margin of error.

---

Published as a conference paper at ICLR 2025
custom similarity function is required; or that there underlying issues with the data generation process that need to be corrected a dataset specific way.
Choice of similarity function is dataset-specific
We applied these criteria to the datasets and similarity metrics considered the study. Table I sum- marises the decisions and the properties of the GOOD curves obtained in each case and Appendix- contains a more in-depth analysis.
Overall, MAPe Jaccard seems be the most versatile similarity metric, being the optimal choice for three out of six datasets. It has the highest dynamic range and good monotonicity in the majority of datasets. This is consistent with the literature that describes it fine-grained and sensitive tool for performing similarity searches in chemical databases (Orsi & Reymond 2024).
It is also interesting that Molformer and ChemBERTa-2, both similarity metrics based the homonymn molecular language models, show the best relative performance in the same two datasets PAMPA and Caco2. This suggests that the models are able capture chemical properties relevant for the respective tasks that are then exploited when partitioning. It also worth noting that, in both cases, the Molformer similarity has larger dynamic range, demonstrating greater resolution low thresholds.
Finally, one of the new similarity metries proposed in this study, Lipinski, the best for the half-life dataset which proved quite challenging for the other alternatives.
Table 1: Similarity function chosen per dataset.
Dataset
Similarity metric
Dynamic range
Monotonicity
Ames
MAPc Jaccard
0.75
0.82 + 0.02
DILI
MAPe Jaccard
0.85
0.81 0.02
PAMPA-NCATS
Molformer
0.80
0.13 + 0.02
Caco2
Molformer
0.85
0.76 + 0.03
Half-life
Lipinski
0.60
0.46 0.06
LD50
MAPc Jaccard
0.70
0.94 + 0.02
The main conclusion that can drawn from these results is that there no one-size-fits-al simi- larity function. Different datasets require different similarity metrics. Therefore, before starting any analysis regarding model generalisation, a careful selection of the similarity metric used to measure generalisation is crucial. This initial step of analysis is missing from previous works (Butina 1999
Ramsundar et al. 2019 Steshin
2023
Tom et al., 2023; Tossou et al. 2024 Hiteffaie et al.) 2024).
Remarkably, the PAMPA dataset has low monotonicity with all similarity metrics considered in this study, which makes the interpretation of the GOOD curve unreliable. We believe that this behaviour is due to the idiosyncrasies of this dataset which is relatively small (+s 2, 000 molecules) and highly unbalanced (1,739 positives to 295 negatives). The imbalance is further exacerbated by the partitioning algorithm with the test set containing, in most partitions, around two thirds of all negatives s 200. This dataset provides a concrete showcase of the limitations of our framework, which are: 1) it depends on the similarity metrics and certain tasks may require custom similarity functions and 2) it may not be reliable for highly unbalanced datasets as the partitioning can increase the imbalance. For this reason, the PAMPA dataset will be excluded from the next step, as the GOOD curve and AU-GOOD values are not reliable.
In any case, we highly recommend performing an initial analysis with any similarity function to determine whether the GOOD has an appropriate dynamic range and meets the assumption of cor- relation between similarity threshold and model performance.
Qualitative analysis of CCPart
OOD partitioning algorithm
We visualised the overlap between training and testing partitions (generated with the corresponding chosen similarity metric). The visualisation of the chemical space is achieved by using uniform man-
ifold approximation and projection (UMAP) (MeInnes et al., 20188 Ross et al. 2022 Ahmad et al. 2022) to project the ECFP fingerprints of each dataset into two dimensions. Figure 3 shows how the testing partition of the Cell-effective permeability dataset gets clustered together at low thresholds and gets more evenly distributed over the training data distribution the threshold increases. It is
7

---

Published as a conference paper at ICLR 2025
particularly interesting how the partitions at threshold 0.9 still less evenly distributed than those obtained with random partitioning. Figure $2 contains similar representations for the rest of the datasets considered in the study.
UMAT rpprosntatioos partitions Gelletilctionn permenbilly
Therslato 0L10
Thorsbako 1.
Thredadds Bi
Korsken partilionin
Figure 3: UMAP representation of the chemical space covered by the training (blue) and testing (red) partitions of the Cell-effective permeability dataset at different similarity thresholds.
Overall, these visualizations provide qualitative and empirical demonstration that the CCPart al- gorithm able to generate -iiitii-iitttttiionnn partitions.
3.1.2 WHAT MODELS GENERALISE BETTER TO THE DRUG REPURPOSING HUB?
Experimental setup
We evaluated the performance of four traditional ML algorithms to build models for the ADMET datasets: K-nearest neighbours (KNN), support vector machines (SVM), random forest (RF), and light gradient boosting (LightGBM) and evaluated their AU-GOOD values using as target deploy- ment distribution the Drug Reporpusing Hub Corsello et al. (2017).
Demonstration of an in-depth Hestia analysis
To demonstrate the depth of the analysis that can be performed with our framework, let us consider one of the datasets in detail. Figure $3 contains equivalent analyses for the rest of the datasets. Figure A shows the GOOD curves for all models considered. The behaviour of KNN is particularly interesting it tends to perform worse lower similarity thresholds and better higher similarity. This observation is not surprising in itself (KNN explictly depends on the distance to neighhouring data points within the data), but exemplifies the type of situation our framework is built to address.
Figure 4 B shows the distribution of maximal similarities between training and target deployment distribution. In this particular case, seems that the distribution is skewed towards higher similari- ties. F shows the distribution of AU-GOOD values of the Matthew's 's correlation coefficient calculated for each model against the target deployment distribution. It shows that RF has the best average AU-GOOD value, even though it shows greater variance across runs. Interestingly. KNN the second best model, despite performing worse at the lower similarity thresholds. This is because the maximal similarity distribution is skewed towards higher similarities and, therefore, the performance at those thresholds has a greater contribution towards the final AU-GOOD score.
Figure DD shows the p-values of comparing the 4 models against each other with the Wilcoxon signed-rank test. In this case, the alternative hypothesis is that Model (y-axis) has better AU- GOOD values than Model B (x-axis). This clearly shows that RF is significantly better (p < 0.001) than the other three models.
Demonstration of summary of the Hestia analysis
The results for the rest of the datasets are summarised in Table 2 The significant rank reflects the Wilcoxon signed-rank test results. For each model, it is calculated as the difference between the total number of models and the number of models to which it is significantly superior. For example, the significant rank for RF in Figure 4 would be 4 (number of models) 3 (number of models that are significantly inferior to RF) = 1; KNN would be = 2, SVM and LightGBM would be
4 -
8

---

Published as a conference paper at ICLR 2025
mulagenwiitty
B. Mosinal similuity distellatio
Wilcome Unalect
1,000
0.0334
0.862
1040
Model
10M0
SON
5X0020
0.2
1000
12
M.006 029332
1.0000
200000
0.5
60
9.2
SMN Lakiiae
990
LagAGaNi Io
1004
Strulurity thoeceldd
Masriaal coutuuitt
Model
Model
Figure 4: HESTIA analysis for Ames' mutageneicity dataset. A. GOOD curve with MAPc (radius 2 and 2,048 bits) and jaceard index; B. Maximal similarity between entities in Ames' mutageneicity dataset and target distribution (Drug Repurposing Hub); AU-GOOD values across 5 different runs; Wilcoxon test p-values with alternative hypothesis Model Model B.
Table Comparison between the different downstream models for each dataset. AU-GOOD refers to average AU-GOOD value across 5 runs, the error corresponds to the standard error of the mean. The AU-GOOD value is calculated for the classification tasks for the Matthew's correlation coeffi- cient (MCC) for Ames, DILI, and PAMPA. For the regression tasks ( CCaco2, Half life. and LD50)
Spearman's 's p is used. S. Rank: Significant rank.
KNN
LighGGMM
Dataset
AU-GOOD
S. Kank
ATRGOOD
K.nk
AL-GROD
S. Rank
AU
S. Kank
+ 0.01
2
0.58
0.00
0,90 0.01
+ 0.01
DML
0.95 + 0.00
0.96
0.00
0.95
0.00
0.77 + 0.04
Cloo2
0.87 # 0.00
0.92
0.00
0.92
0.00
0.88 4 0.00
L3350
0.86 = 0.00
0.83 + 0.00 0.00
0.89 = 0.00
D.83 4 0.00
Average
0.90 = 0.01
2.6
0.91
0.01
2.2
0.87 0.02
2.0
0.72 4 0.05
48
3.2
USE CASE 2: EVALUATION OF PROTEIN LANGUAGE MODELS
Protein Language Models (PLMs) are pre-trained representation learning models that rely on the transformer architecture and are trained with masked language modelling objective on protein se- quences. Here, we explore which PLM generates an embedding space that is better suited for gener- alising to different target deployment distributions. We considered a biophysical property prediction task, Optimal temperature for catalysis (Li et al. 2022), from (Chen et al. 2024). To this end, we downloaded all protein sequences for four organisms in the SwissProt database (Consortium, 2019) target deployment distributions: human (Homo sapiens), Baker's 's yeast (Saccharomyces cerevisiae strain YJM789), and bacteria (Escherichia coli strain CFT073 / ATCC 700928 / UPEC) and the human hepatitis B virus (HPV-B) Orthohepadnavirus hepatitis B virus.
3.2.1 EXPERIMENTAL SETUP
We considered the following models: ESM2-8M (Lin et al.) 2022). ESM2-150M (Lin et al. 2022).
ESM2-650M (Lin al., 2022). ProtBERT (Elnaggar et al. 2021). Prot-T5-XL (Elnaggar et al. 202T]. Prost-T5 (Elnaggar et al. 202T), Aukh-base (Elnagaar et al. 2023), and Ankh-large (Elnag- gar et al., 2023). We extracted the embeddings for all sequences
downstream datasets and for each sequence s we computed its embeddings using average pooling (Unsal et al. (022). We used SVM the downstream model to reduce the amount of noise in our analyses due to model selection.
3.2.2 WHAT IS THE BEST SIMILARITY FUNCTION?
The task considering concerns protein sequence property prediction. We decided to com- pare two computationally efficient similarity functions MMSeqs2 and MMSeqs2 with prior k-mer prefiltering (Steinegger & Söding 2017). However, for other protein tasks where data points evolutionary related we recommend using alternative metrics like Hamming distance like in the Fluorescence task in the TAPE benchmarks |ao et al. (2019) or distance in phylogenetic trees.
The results clearly indicate that MMSeqs2 with prefilter is the best option in all respects i) dynamic range, (Figure A) and ii) monotonicity (Figure $4 B).

---

Published as a conference paper at ICLR 2025
3.2.3 WHAT PLM GENERATES THE MOST USEFUL EMBEDDINGS?
Figure 5 shows the analyses of the different PLM embeddings (More details in Figure $ Clearly, the bigger models (Prot-TS-XL, ESM2-650M, and Prost-T5) tend to generalise better than their smaller counterparts. This is particularly clear with the ESM2 models which are ordered in as- cending performance according to their size ESM2-8M, ESM2-150M, and ESM2-650M. The worst performing models are the Ankh models, which surprising as the original paper (Elnaggar et al.) 2023) reported better performance in several tasks. This result is not conclusive as we are only evaluating a single task. However, it is an interesting finding that opens the question as to whether scaling down model parameter size might be detrimental for the ability of the model embeddings to generalise to new data.
It is also worth noting that, although the ranking of the models is the same regardless of the target distribution, in some cases, like human or the hepatitis virus, the use of a much smaller model like ESM2-150M instead of Prost-T5 could be statistically justified.
cabilyds agrinet distrbzttons
Haraao
All AU--O000 rere
conerticion
call
Hosan Erqutitta viras AU-.U00
product
product
rith
calk taon
E.8
10
1-3
3.8
16
8.8
10
Wilhomen
Wilermen prodecm
MiDresse prodhams
Willowe proateee
Nodel
Madel
Model
Modal
Figure 5: Hestia analyses for Optimal Temperature of Catalysis, comparing different PLM repre- sentations. AU-GOOD values correspond to Spearman's 's p. The variance corresponds to 5 different runs.
4 CONCLUSION
The AU-GOOD is metric that estimates the expected model performance against any target deployment distribution(s), and thus provides a quantifiable value to guide the selection of the most appropriate model to use in different deployment scenarios, without the requirement for additional experiments. It is applicable to any biochemical entities for which a relevant similarity function can be defined. The calculation of this metric requires partitioning the training data into training/testing subsets that are increasingly dissimilar. We propose CCPart a new partitioning algorithm to generate strict splits at multiple thresholds, without the removal of any data points.
We provide a robust framework for analysing whether a similarity function is appropriate for par- titioning a given dataset, with series of quantitative metrics to provide a formal analysis of the reliability of the resulting GOOD curves. Finally, we discuss how to obtain statistical support for comparing different AU-GOOD values. Our experiments show that different datasets may require different similarity functions that better capture the chemical relationships in the context of the spe- cific modelling problem and that the selection of the appropriate similarity metric for each dataset has significant impact on the GOOD curves.
We demonstrate the use of this framework for two different use cases: the development of models for predicting properties of small molecules, and the selection of a protein language model to generate embeddings upon which to build biophysical property predictors.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Walid Ahmad, Elana Simon, Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta-2: Towards chemical foundation models. arXiv preprint arXiv:2209. 220.022200..11,, 2022.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama, Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery data mining, pp. 2623-26666622 2633,,63,, 2019.
Jeremy Ash, Cas Wognum, Raquel Rodríguez-Pérez, Matteo Aldeghi, Alan Cheng, Djork-Arné Clevert, Ola Engkvist, Cheng Fang. Daniel J Price, Jacqueline M Hughes-Oliven et al. Practically significant method comparison protocols for machine learning in small molecule drug discovery. 2024.
Carlo Bonferroni.
Teoria statistica delle classi calcolo delle probabilita. Pubblicazioni del R istituto superiore di scienze economiche e commericiali di firenze, :23-62, 1936.
Darko Butina. Unsupervised data base clustering based on daylight's fingerprint and tanimoto sim- ilarity: A fast and automated way to cluster small and large data sets. J. Chem. Inf. Comput. Sci., 3944)7777-700 1999. doi: 10. 10.100//C1900331.. URL https::////ooo /ttp:/://2oi..r11100002222
c19803381
John-Mare Chandonia, Naomi K Fox, and Steven Brenner. Scope: classification of large macro- molecular structures in the structural classification of proteins-exteneee extended database. Nucleic acids research, 47(D1))D455-4481 2019.
Bo Chen, Xingyi Cheng. Pan Li, Yangli-ao Geng, Jing Gong, Shen Li, Zhilei Bei, Xu Tan, Boyan Wang, Xin Zeng, et al. xtrimopglm: unified 100b-scale pre-trained transformer for deciphering the language of protein. arXiv preprint wxXiv2201.006/99 2024.
UniProt Consortium. Uniprot: a worldwide hub of protein knowledge. Nucleic acids research, 47
(D1):D506-DD5)) D515, 2019.
Steven M Corsello, Joshua A Bittker, Zihan Liu, Joshua Gould, Patrick McCarren, Jodi E Hirschman, Stephen Johnston, Anita Vreic, Bang Wong, Mariya Khan, et al. The drug re- purposing hub; a next-generation drug library and information resource. Nature medicine, 23(4): 405-408, 2017.
Gabriele Corso, Arthur Deng, Benjamin Fry, Nicholas Polizzi, Regina Barzilay, and Tommi Jaakkola. Deep confident steps to new pockets: Strategies for docking generalization. ArXiv, 2024.
Lee R Dice. Measures of the amount of ecologic association between species. Ecology, 26(3): 297-302, 1945.
Joseph L Durant, Burton A Leland, Douglas R Henry, and James G Nourse. Reoptimization of mdl keys for use in drug discovery. Journal of chemical information and computer sciences, 42(6): 1280, 2002.
Yasha Ektefaie, Andrew Shen, Daria Bykova, Maximillian Marin, Marinka Zitnik, and Maha Farhat. Evaluating generalizability artificial intelligence models for molecular datasets. bioRxiv, 2024.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang. Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. Prottrans: Toward un- derstanding the language of life through self-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 44(10):7122-712) 2021.
Ahmed Elnaggar, Hazem Essam, Wafaa Salah-Eldin, Walid Moustafa, Mohamed Elkerdawy, Char- lotte Rochereau, and Burkhard Rost. Ankh: Optimized protein language model unlocks general- purpose modelling, arXiv preprint 200:2300.00668.. 2023.
Felix Fischer, David S Fischer, Roman Mukhin, Andrey Isaev, Evan Biederstedt, Alexandra-Chlos Villani, and Fabian Theis. sctab: Scaling cross-tissue single-cell annotation models. Nature Communications, 15666,611, 2024.
11

---

Published as a conference paper at ICLR 2025
Limin Fu, Beifang Niu, Zhengwei Zhu, Sitao Wu, and Weizhong Li. Cd-hit: accelerated for cluster- ing the gexeerannnt sequencing data. Bioinformatics, 28(23):3150-311 3152, 2012.
Aric Hagberg and Drew Conway, Networkx: Network analysis with python. URL: https://networkk.
github. io, 2020.
Christof Hannesschlaeger, Andreas Horner, and Peter Pohl. Intrinsic membrane permeability to small molecules. Chemical reviews, 1199))5922-5553, 2019.
David Z Huang. Christian Baber, and Sogole Sami Bahmanyar. The challenges of generalizability in artificial intelligence for adme/tox endpoint and activity prediction. Expert opinion drug discovery, 16(9); 1045-1066,,, 2021a.
Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. arXiv preprint arXiv:2102.0985 002409948 2021b.
Oliver Kramer. Scikit-learn. Machine learning for evolution strategies, pp. 45-53, 2016.
Greg Landrum. Rdkit documentation. Release, (11-79):4, 2013.
Gang Li, Filip Buric, Jan Zrimec, Sandra Viknander, Jens Nielsen, Aleksej Zelezniak, and Mar- tin KM Enggvist. Learning deep representations of enzyme thermal adaptation. Protein Science, 41812):e4480, 2022.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos San- tos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, al. Language models of protein sequences at the scale evolution enable accurate structure prediction. BioRxiv, 2022:500902, 2022.
Christopher A Lipinski, Franco Lombardo, Beryl W Dominy, and Paul J Feeney. Experimental and computational approaches to estimate solubility and permeability in drug discovery and develop- ment settings. Advanced drug delivery reviews, 23(1-3):3 1997.
Alex CW May. Percent sequence identity: the need to be explicit. Structure, 12(5):737-738, 2004.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 1rXiv:1802.03422 2018.
Milot Mirdita, Konstantin Schütze, Yoshitaka Moriwaki, Lim Heo, Sergey Ovchinnikov, and Martin Steinegger. Colabfold: making protein folding accessible to all. Nature methods, 196))679-682, 2022.
Saul B Needleman and Christian Wunsch. A general method applicable to the search for similar- ities in the amino acid sequence of two proteins. Journal of molecular biology, 48(3):443-45 453, 1970.
Scott Obach, Franco Lombardo, and Nigel J Waters. Trend analysis database of intravenous pharmacokinetic parameters in humans for 670 drug compounds. Drug Metabolism and Disposi- tion, 36(7):1385-144) 1405, 2008
Markus Orsi and Jean-Louis Reymond. One chiral fingerprint to find them all. Journal of chemin- formatics, 16(1):53, 2024.
Valery O Polyanovsky, Mikhail A Roytberg, and Vladimir G Tumanyan. Comparative analysis of the quality of a global algorithm and a local algorithm for alignment of two sequences. Algorithms for molecular biology, 6:1-12, 2011.
Bharath Ramsundar, Peter Eastman, Patrick Walters, Vijay Pande, Karl Leswing, and Zhenqin Wu. Deep Learning for the Life Sciences. O' O'Reilly Media, 2019. https:: www. amazon. com/ Deep- Lea iatarninng-eee Li fe- Swweeesciiiee----eeeee wwwscopyycpp14920....
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with tape. Advances in neural infor- mation processing systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2025
Peter Rice, lan Longden, and Alan Bleasby. Emboss: the european molecular biology open software suite. Trends in genetics, 16(6):276-21) 2000.
David Rogers and Mathew Hahn. dekeenedd-conectiing fingerprints. Journal of chemical informa- lion and modeling, 50(5)7442-75, 2010.
David J Rogers and Taffee T Tanimoto. A computer program for classifying plants: The computer is programmed to simulate the taxonomic process of comparing each with every other case. 1960.
Science, 1114444):11111111
Eugene Rogot and Irving D Goldberg. A proposed index for measuring agreement in test-retest studies. Journal of chronic diseases, 19(9)9991-1066 1966.
Jerret Ross, Brian Belgodere, Vijil Chenthamarakshann Inkit Padhi, Youssef Mroueh, and Payel Das. Large-scale chemical language representations capture molecular structure and properties. Nature Machine Intelligence, 4(12): 1256-124,, 2022.
Vishal Siramshetty, Jordan Williams, ac-Trung Nguyen, Jorge Neyra, Noel Southall, Ewy Mathé, Xin Xu, and Pranav Shah. Validating adme qsar models using marketed drugs. SLAS DISCOV. ERY: Advancing the Science of Drug Discovery, 26(10):1326 13361136,, 2021.
Temple F Smith, Michael S Waterman, et al. Identification of common molecular subsequences. Journal of molecular biology, 147(1):195-197, 1981.
Robert R Sokal and Charles Michener. A statistical method for evaluating systematic relation- ships. University of Kansas Scientific Bulletin, 3806):140--38, 1958.
David T Stanton. Evaluation and use of beut descriptors in qsar and qspr studies. Journal of chemical information and computer sciences, 39(1): 1--20, 1999.
Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology. 35(11) (5(11):1026-108 2017.
Simon Steshin. Lo-hi: Practical ml drug discovery benchmark. Advances in Neural Information
Processing Systems, 36:64526-64554, 2023.
Felix Teufel, Magmús Halldór Gíslason, José Juan Almagro Armenteros, Alexander Rosenberg Jo- hansen, Ole Winther, and Henrik Nielsen. Graphpart: homology partitioning for biological se- quence analysis. NAR genomics and bioinformatics, 5(4):lqad088, 2023.
Gary Tom, Riley J Hickman, Aniket Zinzuwadia, Afshan Mohajeri, Benjamin Sanchez-Lengeling. and Alán Aspuru-Guzik. Calibration and generalizability of probabilistic models on low-data chemical datasets with dionysus. Digital Discovery, 2(3):759-774, 2023.
Prudencio Tossou, Cas Wognum, Michael Craig, Hadrien Mary, and Emmanuel Noutahi. weal-world molecular outtof-distributtin:: Specification and investigation. Journal of Chemical Information and Modeling, 2024.
Serbulent Unsal, Heval Atas, Muammer Albayrak, Kemal Turhan, Aybar C Acar, and Tunca Dogan. Learning functional properties of proteins with language models. Nature Machine Intelligence, (3):227-245, 2022.
Michel van Kempen, Stephanie S Kim, Charlotte Tumescheit, Milot Mirdita, Cameron LM Gilchrist, Johannes Söding, and Martin Steinegger. Foldseek: fast and accurate protein structure search. Biorxiv, pp. 2022-02, 2022.
Pauli Virtanen, Ralf Gommers, Travis Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, al. Scipy 1.0: funda- mental algorithms for scientific computing in python. Nature methods, 1733):261-27,, 2020.
Ian Walsh, Dmytro Fishman, Dario Marcia-Gasulla Tina Titma, Jen Harrow. Fotis E Psomopoulos, and Silvio CE Tosatto. Recommendations for machine learning validation in biology. arXiv, 2020.
13

---

Published as a conference paper at ICLR 2025
Ning-Ning Ning Wang, Jie Dong, Yin-Hua Deng, Min-Feng Zhu, Ming Wen, Zhi-Jiang Yao, Ai- Ai-Ping Lu, Jian-Bing Wang, and Dong-Sheng Cao. Adme properties evaluation drug discovery: prediction of caco-2 cell permeability using combination of nsga-ii and boosting. Journal chemical information and modeling, 56(4))763-773, 2016.
Wenyi Wang, Fjodor Melnikov, Joe Napoli, and Prashant Desai. Advances in the application of in silico admet models-an an industry perspective. Computational Drug Discovery: Methods and Applications, 2:495-535, 2024.
Frank Wilcoxon. Individual comparisons by ranking methods. biom bull (6): 80-83, 1945.
Congying Xu, Feixiong Cheng, Lei Chen, Zheng Du, Weihua Li, Guixia Liu, Philip W Lee, and Yun Tang. In silico prediction of chemical ames mutagenicity. Journal of chemical information and modeling, 52(1 1:2840-22444 2012.
Youjun Xu, Ziwei Dai, Fangjin Chen, Shuaishi Gao, Jianfeng Pei, and Luhua Lai. Deep learning for drug-ine drug-induced liver injury. Journal of chemical information and modeling, 55(10):2085-2093, 2015.
Hao Zhu, Todd Martin, Lin Ye, Alexander Sedykh, Douglas Young, and Alexander Trop- sha. Quantitative structure- activity relationship modeling of rat acute toxicity by oral exposure. Chemical research in toxicology, 1212211113111 2009.
14

---

Published as a conference paper at ICLR 2025
A MATHEMATICAL FORMALISATION OF AU-GOOD METRIC
This Appendix section discusses the formal definition of the AU-GOOD metric and its connection with statistical learning theory. It is divided into four subsections, follows: i) empirical risk in terms of the dataset partitioning strategy, ii) definition of the AU-GOOD metric as an estimation of model generalisation a target distribution, iii) theoretical considerations regarding the assumption of unbiased sampling during dataset partitioning, and iv) empirical considerations regarding model performance metrics.
A.1 EMPIRICAL RISK TERMS THE DATASET PARTITIONING STRATEGY
This subsection demonstrates that the empirical risk can be expressed in terms of the partitioning operation used divide the data into training and testing subsets.
Let us consider model Sa defined by set of parameters 0 and trained data distribution P(x, y) where each datum is defined by set of features æ E and a set labels y E y. The model will attempt to approximate an unknown function g that provides mapping from the feature space to the label space fa ~ g: X y. The quality of the approximation can be described in terms of loss function C that measures the error in each individual mapping. In the end, the parameters of the model will be optimised to minimise the expectation of the loss across the whole population which is also known the population risk:
R(fo) =Reewji--ia
In practice, we do not have access to the data distribution, only to subset, D ~ P(x,8) y). Without access to the data distribution, the population risk has The empirical risk of model given a dataset 'D is calculated by partitioning 'D into two mutually exclusive subsets T, D such that T n E = 0, where T is the train subset and subset defined by the partitioning operation denoted as 4: D + T,& E. The model trained on the train subset is denoted as fo.r (x) Therefore, the empirical risk can be estimated in terms of the partitioning strategy:
approximated by the empirical risk.
the test
1 Σ ((for())) (Z,9)EE
Ro(fo)==
(2)
where n is the number elements in
A.2 DEFINITION OF METRIC ESTIMATING MODEL GENERALISATION A TARGET DISTRIBUTION
This subsection builds upon the previous definition of empirical risk in terms of the partitioning operator and derives metric for model generalisation to any target data distribution based on the similarity between the elements the training and target distributions.
Let us consider hypothetical function that measures the similarity between the features of any two elements in the population, 8: X' x + R. Let us also consider partitioning operator that enforces maximal value of similarity À between the elements in training and testing. Such operator would necessarily be a function of both the similarity function and the threshold similarity X, Then, the empirical risk would depend the similarity function and the similarity threshold:
(foo)
1 1 72
((forr
(3)
Model generalisation to target deployment distribution Ex that has been drawn from population distribution Ex ~ P" (z, y) different from P(x, 3)) could then be described as the expectation of the empirical risk across the distribution of the similarity Àg denoted P(A.(P") induced from the unknown target distribution Pa P= y). within the bounds of a minimal similarity Ao and maximal similarity An:
15

---

Published as a conference paper at ICLR 2025
(Jo = [ALpP- - An Rs. P()s)P"dda
(4)
where P(A,|P5 PP) is the probability density distribution of the similarity between D and EL, i.e. the expected distribution of similarities between D and =": P(A,|P*) (omax V wiy Iy D2E*} >,). This integral can be approximated numerically by:
An
Go(folP*) = E PA
(5)
λ=
A.3 SAMPLING CONSIDERATION
In this subsection, discuss the limitations of the assumption of unbiased sampling made in the
previous subsections.
In many biological domains, data points noisy, and it's quite possible that the type of discon- nected subgraphs generated by similarity-based partitioning algorithms would tend to bias the testing towards unrepresentative regions of the (bio)chemical space. This behaviour would lead unsta- ble GOOD curves where there is low monotonic correlation between similarity threshold and model performance and would therefore be identified when following the guidelines outlined in Subsection 3.1.1.
However, the GOOD curve monotonic, there could some specific partitions given thresh- olds that show unstable behaviour. Given that the AU-GOOD is an expected value, with a sufficient number of thresholds (n > 10) and wide enough dynamic range, the estimated and the true popu- lation AU-GOOD values will tend converge.
A.4 MODEL PERFORMANCE METRIC CONSIDERATIONS
Finally, in this subsection we discuss practical limitations, regarding the effect that choice of model performance metric might have on the calculated AU-GOOD values and our general recommenda- tion.
The GOOD curve generates test partitions with different label distributions at different thresholds. Thus, both the monotonicity and AU-GOOD calculation could be affected by the label distribution shifts across different thresholds.
For this reason, we recommend the use of model performance metrics that are not sensitive to label imbalance, like Matthew's correlation coefficient or weighted Fl for classification tasks and Spear- man's Pearson's correlation coefficient for regression tasks.
B HESTIA COMPUTATIONAL SUITE
B. SIMILARITY CALCULATION
Hestia implements a diverse array of pairwise similarity functions s(x, y) which need to fulfill two conditions:
1. The similarity between an entity and itself is maximal.
2. They are symmetric, szz y) syy,
Biological sequences. The similarity function calculated for protein sequences and nucleic acids is the sequence identity in pairwise alignments. Local alignments calculated using the MMSeqs2
implementation (Steinegger & Söding 2017) of the Waterman-Sie -mith algorithm (Smith et al. 1981) with without prior k-mer prefiltering. Global alignments calculated using the EMBOSS im- plementation (Rice et al., (2000) of the Needleman Wunch algorithm (Needleman & Wunsch] 1970). More information regarding the empirical differences between local and global alignments can be
16

---

Published as a conference paper at ICLR 2025
The denominator used to calculate the identity can be the length of the longest or the shortest sequence, as well as the length of the full alignment. Choice of the most appropriate denominator depends on the dataset MMyy 2004).
found in (Polyanovsky et al.) 2011 Teufel al. 2023
Protein structures. The similarity function calculated for a pair of protein structures is the probabil- ity that they belong the same SCOPe family (Chandonia et al. 2019). This probability is approx- imated using the Foldseek alignment algorithm (van Kempen et al. 2022) with both sequence and structural interaction representation (3Di+AA) in either global (Foldseek-TM) or local (Foldseek) modes.
For both biological sequences and protein structures, pairwise alignments are not necessarily sym- metric so we enforce condition 3 by taking the maximal similarity for each pair of entities: s* == max[s((x y), 2)).
Small molecules. Similarity between small molecules can be calculated using a variety of similarity
functions including: Tanimoto (Rogers & Tanimoto 1960), Dice (Dice 1945). Rogot-Goldbert
Rogot & Goldberg (1966). Sokal (Sokal & Michener 1958), euclidean, manhattan, canberra, or
cosine similarity between various types fingerprints including extended connectivity fingerprints
(ECFP) (Rogers & Hahn 2010), MACCS (Durant et al. 2002). or MAPc (Orsi & Reymond 2024);
an physicochemical descriptors.
Representation learning pre-trained model embeddings. Similarity between embeddings in- cludes traditional geometrical distance functions like cosine, euclidean, manhattan, canberra, as well offering the option to use any custom similarity function. The distance functions d are transformed into similarities with the following expression: s(x, 1+d(z.y)
B.2 SIMILARITY CORRECTION ALGORITHMS
Basic notation. Let D be an arbitrary dataset comprised of n entities. D can be expressed as graph G(N, F.) where the nodes (N) are the set of all entities in D and the edges (F), the subset of all pairwise similarity measurements between those entities s N,, N2) with values above a threshold X, thus, E [m, n22) V n1, n2 E N such that s(n,, > }}. Let 7 and V be two partitions of D such that T' n V 0. Then, Ey, the forbidden edges, can be defined as the subset of all similarities between any two entities that ftt, v) A}.
and that above the threshold, Es = {tt, v) Vt E T, E E such
In this paper, similarity correction techniques refer to algorithms targeting the reduction of Ey. We discuss them extensively outlined below.
Similarity reduction. Similarity reduction aims at reducing redundant entities from D. This is achieved by two-step process comprising clustering step and redundancy reduction step in which the representative entities of each cluster are selected and the rest of cluster members are re- moved. Hestia relies on custom implementations of greedy incremental clustering and greedy linear cover-set algorithms for the clustering step. These algorithms are commonly used in the context of sequence clustering, by specialised software like CD-HIT (greedy incremental clustering) (Fu al., 2012) and MMSeqs (both) (Steinegger & Söding 2017). Our custom implementations generalise their utility to any arbitrary data type for which a pairwise distance matrix can be calculated.
Random partitioning. Random partitioning algorithms aim to divide the dataset into subsets through unbiased sampling. The idea is to generate partitions with similar distributions. Strictly speaking, it could only be considered similarity correction algorithm under the assumption of in- dependent and identically distributed data, e.g., after performing similarity reduction on the dataset. Hestia leverages the corresponding scikit- teiikitleeare implementation (Kramer) 2016) of the algo- rithm.
GraphPart generalisation. Similarity partitioning algorithms aim at dividing the dataset into subsets or partitions such that Es 0 between any two partitions. This is achieved through the removal of entities whenever necessary. Hestia relies custom implementation of the Graph-Part algorithm (Teufel et al. 2023) that generalises it to any similarity function and biochemical data
type.
The algorithm starts with a clustering step using limited agglomerative clustering with single linkage with the restriction that the clustering stops when either a) cluster reaches the expected partition
17

---

Published as a conference paper at ICLR 2025
size N/n where N is the number of entities in the dataset and the number of desired partitions or b) there are no more edges above the threshold x. Then, clusters are iteratively merged into the partitions that the generated partitions have balanced distribution of labels (if the dataset has categorical labels) and similar number of entities. The number of interpartition neighbours (entities with E X) is checked and entities are moved the partition with which they have the most neighbours, If the number of entities with at least one interpartition neighbour is greater than O, then the e log (i/100) the number of interpartition neighbours and i, the current iteration. This iterative process continues until c Our custom implementation leverages Scipy (Virtanen et al. 2020) and Networkx Hagberg & Conway 2020
entities with most interpartition neighbours are removed, where is
Connected components partitioning.
The algorithm first identifies the set of all unconnected subgraphs of G. {U1, U2.... Uz}. These un- connected subgraphs, by definition, will not have any intercluster neighbours, otherwise they would not be unconnected. We have observed that generally this strategy leads to one cluster being popu- lated with most entities and a variable number of much smaller subgraphs (10 10*). The smaller cluster is, the more unique it is with respect to the dataset distribution. Based on this assump- tion, the algorithm builds the evaluation set by assigning subgraphs in ascending order of number of members, i... smaller subgraphs first. As above, there is also the optional additional objective of maximising evaluation label balance. This sampling strategy biases the evaluation subset towards the regions of the dataset distribution that are the most unconnected and thus the most dissimilar to other members of the dataset. See Algorithm
Algorithm CCPart (Connected Components Partitioning) algorithm
1: Define G.D
2: U U Find all unconnected subgraphs of G
3: Sort U in order ascending number of elements
4: T,E 0,0
5: while [&] & 0.185 x [D] do
6: Add & first element of U
7: Remove first element from U
8:
end while
9:
T - U
10: return T,E
C MOLECULAR FINGERPRINTS AND PHYSICOCHEMICAL DESCRIPTORS
ECFP coomeeitd fingerprints) (Rogers & Hahn 2010 are binary fingerprints where an "on" bit represents the presence of substructure and "off" bit, its absence. The radius de- termines the number of hops consider in the molecular graph to define each substructure, whereas the number of bits determines the level of information compression applied with smaller numbers leading to higher compression and increasing the likelihood of "collisions" ". i.c., two substructures being assigned the same bit. We have chosen the values of radius 2 and 2,048 bits as it their most common configuration (Rogers & Hahn 2010 Orsi & Reymond, (024)) and an early exploratory analysis showed that they were the most cost effective configuration.
MACCS (Molecular ACCess System) (Durant et al. 2002) are binary fingerprints, as well, with 166 bits. They differ from ECFPs in that there direct mapping between the bits and specific, pre-defined, chemical substructures.
MAPe (MinHashed Atom-Pair chiral fingeprint) (Orsi Reymond 2024) fingerprints are hash- based fingerprints, as ECFP, but they are not binary, each position can be occupied by different hash-values. They have been shown to outperform any other fingerprint similarity searches with unique properties such being able to discriminate between different stereoisomers of the same molecule. Number of bits and radius was selected based on the best configuration reported by its authors.
18

---

Published as a conference paper at ICLR 2025
The Lipinski physicochemical descriptors vector that introduce in this study for the pur- poses of measuring molecular similarity that has been inspired by Lipinski's rule of (Lipinski et al., 1997). It describes the following properties of a molecule: 1) number of H acceptors, 2) number of H donors, 3) number of heteroatoms, 4) number of rotable bonds, 5) number of saturated carbocycles, 6) number of rings. The calculation of each the properties was performed through the implementation in RDKit (Landrum) (201)).
The BUTC is vector with terms describing the lower and upper bounds of the confidence interval of the following molecular characteristics related to the maximum value of the Burden Cluster-based Van der Waals atomic surface area (Stanton [999]: and 2) upper and lower bound of the mass eigenvalue; 3 and 4) upper and lower bound of the eigenvalue of the Gasteiger chargea; 5 and 6) upper and lower bound of the eigenvalue for Crippen logP; and 7 and 8) upper and lower bound of the eigenvalue for molar refractivity. It is calculated through the implementation in Landrrua 2013).
D CHOICE OF SIMILARITY FUNCTION FOR THE ADMET DATASETS
Figure $I displays detailed results per dataset and Tables 2 contain the raw data from each of the experiments. is clear that similarity functions with small GOOD curve dynamic ranges, like all involving MACCS fingerprints or the BUTC descriptors, also demonstrate the biggest variance in their GOOD curve monotonicity. This is completely reasonable, as the smaller the dynamic range, the smaller the number of points comprising the GOOD curve and the more sensitive that the monotonicity calculation will be to random noise.
For the Ames' mutagenicity test dataset the similarity function with the largest dynamic range is Molformer (0.85), however this similarity function is ranked 9th in monotonicity (0.24 + 0.02). The next largest dynamic range belongs MAPc Jaccard (0.75), which is also the highest monotonicity (0.82 + 0.01). The GOOD curve corresponding to MAPc Jaccard shows three phases. First, an approximately stationary phase at thresholds 0.2-0.455 then a phase or steep growth 0.60 and, finally, a phase with a more controlled growth 0.60 - 0.95. It is also worth noting that for this dataset all the similarity functions with dynamic range below 0.4 (with the exception of RG ECFP2), have either a high variance on their monotonicity or very low monotonicity. This observation suggests that even for such a large dataset (= 7, 000 molecules), the increased number of thresholds is necessary to avoid random noise.
For the Drug-Induced Liver Injury, DIILI, dataset the two largest dynamic ranges are Molformer and MAPc Jaccard (0.85), with MAPc Jaccard ranked 3rd in monotonicity (0.81 + 0.02) and Molformer 16th (0.33 + 0.02). The GOOD curve of MAPe-Jaccard shows only one phase of steady growth. Interestingly, for this small dataset (476 molecules), the GOOD curve is less smooth than in the rest of the datasets, with small jumps between one threshold and the next. However, this jumps well within the error ranges and the overall trend of the curve is clearly monotonie. Another interesting aspect of this dataset is that all similarity functions show decent monotonicity (lowest is 0.33 + 0.02), which may respond to the molecular diversity within the dataset.
The PAMPA NCATS dataset is the most challenging of all the datasets considered in this study. Only the Molformer similarity function achieves positive monotonicity, albeit a small one (0.13 + 0.02). We hypothesize that this is a consequence of the highly unbalanced nature of the dataset (1,739 positives 295 negatives), which further exacerbated by CCPart. Another con- tributing factor to this behaviour is the nature of the chemical predictive task, cell permeability, which is not mediated by the scaffold of the molecule or its overall structure, but for smaller local properties like the charges at the ends of the molecule or its lipophilicity (Hannesschlaeger et al.) 2019). This is also shown in the Caco2 dataset (see below), which also shows uncharacteristic low monotonicity for fingerprint-basee similarity metrics. Overall, this dataset shows the importance of working with good quality datasets and the need for further research into similarity functions within the biochemical domain.
Similarly, the Caco2 cell effective permeability dataset shows that MAPe-Jaccard and Molformer are tied as the similarity functions with largest dynamic range (0.85), but Molformer has the 2nd best monotonicity (0. 00.03 and MAPc Jaccard the 5th 0.00). Interestingly the 1st similarity function in monotonicity is ChemBERTa-2 (0.80 + 0.05), which is also based Language Model.
19

---

Published as a conference paper at ICLR 2025
These results are consistent with those seen for the PAMPA dataset, and indicate some dependence on the chemical task (in both cases cell penetration). If we look at the GOOD curve for Molformer, though monotonic, most of the variance due to the threshold is concentrated at very low thresh- olds (0.1 0.25) and then the growth of the curve is very modest. is clear from this observation and the previous
with PAMPA that there isi need for further research into similarity functions that do not focus as much in global molecular structure (like common fingerprints), but also on the physicochemical characteristics of the molecules.
The Half-life dataset is quite idiosyncratic, as well. The three similarity functioons with the largest dynamic range (Molformer, 0.85; MAPc Jaccard. 0.80; and Sokal ECFP2, 0.70) all show monono- tonicity below or around 0.0. The fourth largest dynamic range corresponds to the Lipinski simi- larity function, which new physicochemical similarity function that we first introduced this study to address the shortcomings of other alternatives. It has decent, albeit small, monotonicity (0.46 0.06). Interestingly, the best monotonicity achieved by the other physicochemical simi- larity function we have introduced in this study (BUTC) with 0.80 = 0.06, however, it has two short a dunamime range (0.25). The Lipinski GOOD curve shows some noise at the beginnig, proba- bly caused by pathological sampling by CCPart at the lowest threshold: and then a steep growth (0.40 0.50). From 0.50 until the end model performance remains static. The dataset is quite small (=s 667 molecules) and this behaviour could be explained by lack of internal chemical diversity. Alternatively, the Lipinski similarity function might not be fine-grained enough to properly exploit the existing diversity in the dataset. Overall, the results from this dataset paired with the previous two datasets highlight the importance of considering more diverse similarity functions that do not rely on the traditional hash-based fingerprints; and the need within the field for more research into other types of similarity functions.
Finally, the LD50 dataset shows similar behaviour to that of Ames, which may be explained by both of them being the biggest. Interestingly, Molformer shows larger dynamic range (0.85) than MAPc Jaceard (0.70); however Molformer ranks 12th in monotonicity (0.48 + 0.02) and MAPs- Jaccard, 6th (0.94 + 0.00), but has an almost perfect monotonicity.
20

---

Published as a conference paper at ICLR 2025
Comparino stsiiacct Sanstera ADMET
wdilitlll Bal Ennggaag
andapnokily puliil ledk Sciespingg 1f
Cottrr
Colto
Staiaaoo 2014
NACCS
Codta MG0O
Color
Onde 103/5
Kile Jurnet
NDC
Sex DXF/
01000
Seabbeity Sembbed
0071
Stadlvity
Coltee
Saatbrty RaatoA
MAOCE
000D masmvlitt
Seedlerit
14M0 daiity Srannialle
Draeeb norge
5 6008 man eseavedity
JAMPP UNDATS
come
000 Me000
Nasiaill
Colbs -4000
Fantiees
Sala HULLS
-18
.85
03
0.4
04
Effective (CRE-)) Meng vl.)
Permesllly (Cte03) Weng
Eltectiee Derreabile (C6te2 Weng
3.1
546- 0172
305
G00D curve Emenii raage
cane ressoeldty
Stellarity devebbdt
& G00D meestelete
DC
Codo -400E
Nadiurity
M600K
Eile: (079)
WFC
-85
Acuse teeicity LEG0 (Dve et all
Acite tedcity LOGE con et
Acxte tndecity LID0 (2be ot
Stadlarity 2017
Dey (007)
Cotcaa
Eab: MLCCE
With Joned
Dar 0310
Bun 4V23
AD 0112
212
GO0D
-18
-83
Stedluttty Dembbe
Figure 1: A: Analysis of dynamic range. B: Analysis of monotonicity, violin plot show the disper- sion across 5 runs. C: GOOD curves, error bars correspond to the standard error of the mean across 5 runs with independent hyperparameter optimisation. The similarity metric chosen for each dataset is highlighted.
21

---

Published as a conference paper at ICLR 2025
Table 1: Complete analysis of all similarity functions (Part 1). Measurement error corresponds to the standard error of the mean for 5 independent runs.
Datusel
Similarity metric
Monotunicity
Dynamic runge
ames
Lamimoto ECTP2
0.74 0.00
0.35
Tanimulo MACCS
-0.82 0.01
0.20
attecs
BCFP2
0.74 0.01
0.35
Dixe - MACCS
-0,60 1 0.01
0.10
Counse BCFF2
0.67 4 0.00
0.35
Coline MACCS
-0,00 4 0.01
0.10
ames
Sokal BCPP2
0.77 # 0.00
0.70
Socal MACUS
0.03 0.02
0.35
amis
KGG BCHP2
0.60 #b 0.02
0.15
RG MACUS
0.20 E 0.10
0:05
MAPe. Jaccard
0.82 # 0.01
0:75
ames
MolForrer
0.24 # 0.01
0.85
BUTC
0,90 0.04
0.20
amis
Lipiskki
0.51 # 0.02
0.70
ChemBERTa-3
0.66 + 0.07
0.55
ese02
Tanimoto- RCFP2
0.33 + 0.00
0.65
CA002
Tanimoto MACCS
00.02 0.00
0.25
41002
Dice - FCFP2
0.16 0.001
0.50
MACCS
0.40 0.02
0.10
Cowine ECFP2
00.28 4 0.01
0.50
C1002
Cosine - MACCS
00600 0.01
0.10
ECTP2
0.55
0.75
Ca0o2
Sosal MACCS
0.32 + 0.01
0.40
RG ECTP2
-0,14 1:14:100 0.04
0,20
C3602
RG MACUS
-1.00 0,00
0.05
C3092
MAP: Jaccand
0.45 1 0.01
0.85
MolFormer
0.76 0.02
0.85
Caco2
BUTC
-0,50 0,00
0.10
66602
Lipinski
0.41 + 0.02
0.60
C3002
CSemBEKia
0.80 0.05
0.35
Tanimoto BCFF2
0.76 0.00
0.70
diNi
Tanimoto - MACOS
0.69 2 0.00
0.35
BCFF2
0.76 0.00
0.55
MACCS
0.62 : 0.01
0.20
dill
Cosate - BCFP2
0.82 = 0.00
0.55
Cosine MACCS
0.76
0,20
dili
Sokal BCPP2
0.54 22 0.01
0.80
deN
Sokal MACCS
0.54 25 0.02
0,50
RG- RCFP2
0.91 = 0.00
0.25
d5h
RG MACCS
0.40 2 0.005
0,10
MAPe Jaccard
0.81 2
0.85
Molfoormer
0.33 0.001
0.85
diRi
BUTC
0.36 0.04
0.15
Lipinski
0.41 0.04
0.60
dihi
ChemBERTa-3
0.80 2 0.05
0,55
pumpu ncals
Tamimuto ECFP2
-0.33 1: 0.01
0,50
pampa
Tunimoto MACUS
00400 0.01
0.15
pumip Icalls
Dive ECTP2
0,22 1 0.01
0.35
nculs
Dice - MACUS
11.00 4 0.00
0.15
pompa
RCFP2
0.49
0.01
0.35
pampa neats
Counte MACCS
-1.00 2: 0.00
0.05
pampa nests
Sokal - RCFPP
-0.28 2 0.02
0.65
pampa ncats
Sokal MACUS
-0.66 0.02
0.25
gampu nests
ECFP2
-0.36 1:66:000 0.02
0.15
pampa
RG- MACCS
-1.00 0.00
0.00
pampu neals
MAPc- Juccarl
-0.27 0.02
0.80
pampa nestis
MedFormer
0.13 0.02
0.80
pumpu
BUTC
-1.00 0.00
pampa nc98s
I lipinki
-0.30 +: 0.05
0.50
pumpa
C
-0,18 1: 0.07
0.40
balt life
Tamimoto ECTP2
-0.34 0.00
0.60
half life
Timimoto MACCS
-0.70 1:000000 0.01
0.20
ball life
Dive BCHP2
-0.47 4 0.01
0.40
MACCS
0.50 0.00
0,10
balf life
Counne ECFP2
-0.87 0.00
0.40
ball life
Cosine MACCS
-1.00 0.00
0.05
life
Sokal BCFP2
-0.88
0.00
0.70
balf life
Sokal MACCS
-0.11 4 0.01
0.35
RGG BCHP2
0.48
0.01
0,20
balf hife
KG MACUS
-1.00 1: 0.00
0.05
Balf life
MAPo Jascard
-0.12 2: 0.01
0.80
MolFormer
-0.08
0.01
0.85
half Life
BUTTC
0.80 22 0.04
0.15
half life
Lipinsii
0.46 0.005
0.60
life
ChemRERTa-3
0.36 0.04
0.55
22

---

Published as a conference paper at ICLR 2025
Table 2: Complete analysis of all similarity functions (Part II). Measurement corresponds to the standard error of the mean for independent runs.
Dataset
Similarity metrrcc
Monotonicity
Dynamic range
1450
Tanimoto ECFP2
0.96 =: 0.D0
0.45
1850
MACCS
0.54
0.01
Dioe
0.97
0.00
1850
Dice MACCS
1.00 + 0.00
0.05
Cosine HCFP?
1050
Cosine MACCS
1.00 U.IK
0.05
M40
Sokal ECFP2
0.93 0.000
0.65
0.67
M50
RG RCFP2
1.00 0.00
0.10
MACC'S
1.00
M150
MAPc Jaccard
0.0000.00
0.70
1050
Moll ormer
0.45 1 0.02
0.83
BUTC
1050
Lipinski
0.51 0.06
0.70
E HYPERPARAMETER SEARCH FOR HPO
Table $ describes the hyperparameter space defined for experiments.
Table 3: Hyperparameter search space for each learning algorithm.
Model
Trials
Hyperparaneter search space
Type
Beat
x 10
Yes
kemel
categorical
lincar, rbf, sigmoid
NA
degree (only kernel poly)
inseger
coctll (only with poly sigmood)
Boat
10 8.1
Yes
opeilon (only regression)
10-5.1
inseger
1:30
No
Weights
caleporical
uniform, distance
NA
algorithm
categonical
leaf size (oaly with ball or lid tree)
integer
100
No
power of Minkowski metric
integer
1-3
No
Inteper
10:5,000
categerical
gini", ennoppy S, luz lose".
RF
200 minimum samples split maximum
2- 100
MSE** MAB** friedmmm MSE*
NA
No
catepyrical
lag2. sgri
complesity panmeter (cop.nlpha)
foot
10 10-3
Yes
number eatimators minimam stmples
10-5,000
integer
LightGBM
200
minimum split gain regularization leaming
foaa
10
10
Yos
10,
Yes
float
7
Yes
F VISUALISATION OF THE CHEMICAL SPACE
This appendix contains the visualisation of the level overlap between the training and testing partitions of the dataset in the chemical space defined by the ECFP fingerprints.
23

---

Published as a conference paper at ICLR 2025
Threshedd: 0.20
Threshold: 0.35
Thresbeld: 0.3
Random partilioning
partitioning
30
Threshold:
Thresood
Threshohh
Thrnshrldi 0.10
Threcholdi 0.55
Thrnchedd 0.8
Random partitioning
Thorebuldd.c 0.35
Threcholds
Thrreheld.
Random partitinning
Halttn
partitioning
Figure 2: UMAP representation of the chemical space covered by the training (blue) and testing
(red) partitions of all molecular datasets considered during this study.
24

---

Published as a conference paper at ICLR 2025
G DETAILED HESTIA ANALYSES FOR ALL ADMET DATASETS
This appendix contains the detailed Hestia analyses for all ADMET datasets considered in this study.
Compacaso of Saarsin slgecahiu fur ADMETT GREAIECI
Postee ineuuity Aorbecie
Aorrictin
Model
VM
SMM
Light
5P0
Figure 3: Hestia analyses for the comparison of learning algorithms for the 6 ADMET datasets considered in this study.
H DETAILED HESTIA ANALYSES FOR ALL PROTEIN DATASETS
Comparison similenty protein datasets
Analyyis dynamii
Analydi uumotosicity Optimal Iemperaturs Catalyats
000D ouree drnonndo sangee
Optttnal Temperattrr Catalyete
0.40
GOODD
0.20
Statlartty tunction
Simtlactly function
Figure 4: A: Analysis of dynamic range. Analysis of monotonicity. Violin plot show the disper- sion in the GOOD curve slope across 8 alternative representation methods and 5 runs per represen- tation (total of 40 experiments)
25

---

Published as a conference paper at ICLR 2025
impeertcc macipa lorgnn Corti
Figure 5: Analyses for the comparison of PLM embeddings algorithms for the optimal temperature for catalysis.
26


Reviews:
[{"review_id": "GOCjeBln6j", "reviewer": "Reviewer_b5Gp", "summary": "This paper proposed an interesting framework for evaluating the model generalization to the out-of-distribution data (OOD) specifically with-in the biochemical domain. The author first proposed a metric that measures the model generalization based on existing similarity measures. A dataset partitioning algorithm and non-parametric statistical testing methods are proposed to complete the framework for model generalization comparisons.", "questions": "My questions are listed in the weakness part.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents a comprehensive framework that not only measures generalization capabilities but also provides a statistical method for comparing different models based on the new AU-GOOD metric​\n\n2. In the experimental section and the Appendix, the authors present a detailed and systematic analysis of selecting the similarity function, including its crucial relationship with the robustness of the GOOD metric in predicting the OOD data. \n\n3. The paper includes extensive experiments on multiple use cases to demonstrate the framework's effectiveness. Including selecting the model with more generalization abilities in case 1: predicting pharmaceutical properties of small molecules and case 2: predicting protein sequence properties. \n\n4. The open source code repo is clean and well-organized.", "weaknesses": "1. I am a bit confused by the formulation process in section 2.1 and found the formulation process is somewhat misleading. Here is a simple example, given four data points with features x1, x2, x3, x4. Assume the similarity score is: s(x1, x2) = 0.2. s(x1, x3) = 0.3, s(x1, x4) = 0.5, s(x2, x3) = 0.7, s(x2, x4) = 0.8, s(x3, x4) = 0.9. Assume the maximal value of similarity \\lambda between the training and testing elements is 0.85. In this case, \\lambda_{s}=0.85 only restricts x3 and x4 to be both in either the training set or the test set. Therefore, \\lambda_{s}=0.85 results in different partitions of these four data points. E.g (x1, x3, x4) for training (x2) for testing or (x3, x4) for training and (x1, x2) for testing, etc. I am wondering if it is proper to formulate eq.3 from eq.2 by eliminating the \\phi, since \\lambda_{s} can only  be considered as a partition constraint rather than partition method. \n\n2. In addition, assuming \\phi is not omitted in equation 5, and empirical risk R is rewrite as (say) a function of \\phi. This indicates that, under the same threshold, the AU_GOOD value is not only dependent on the target distribution, but also the partition algorithms. It’s better to discuss different partition algorithms, and how it impacts the AU_GOOD, or, justify that the proposed CCPart can successfully partition the Out-of-distribution sets. \n\n---\n\nMinor concerns\n\n3. For equation 2, \\mathcal{T} denotes the training subset, and the empirical risk is calculated among the test set \\mathcal{E}, the subscript of x should not be i. \n\n4. It’s better to enlarge the captions, x-labs and y-labs in Figure 1.\n\n5. Another minor shortcoming of this paper is that it lacks a ‘ground truth label’ for model generalization comparison. In use case 1, Figure 5 shows that the Random Forest is significantly better than the rest models in OOD generalization. However, this conclusion cannot be further verified. It is better to know exactly which model does better in handling the actual OOD data. (Though it is well recognized that RF and lightgbm is good at handling the outliers)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KVOgprOY6W", "reviewer": "Reviewer_53DR", "summary": "This paper proposes AU-GOOD, a metric for assessing machine learning model generalization to out-of-distribution biochemical data. AU-GOOD quantifies expected performance under increasing train-test set dissimilarity, integrating prior target distribution knowledge. A novel partitioning algorithm, CCPart, generates challenging test sets by prioritizing dissimilar data points. Statistical methods for model comparison based on AU-GOOD are also provided. The accompanying open-source code includes implementations of various similarity functions, partitioning algorithms, and AU-GOOD calculation, applied to two use cases: small molecule property prediction and protein language model embedding evaluation.", "questions": "Are there any plans to extend the work to larger datasets and deep learning models? Often, datasets for virtual screening are much larger, on the order of millions, and GNN and GCN models start to exhibit state-of-the-art performance. \n\nWhat motivates the use of various performance metrics? Why, for example, Spearman for Caco2, Halflife and LD509, but Matthew for Ames, DILI, and PAMPA? It seems that the behaviour of the GOOD curve is heavily dependent on this choice. Would the conclusions change for different metrics? And how might that affect the interpretation of AU-GOOD, if the metric can so quickly fluctuate and change?\n\nHow does CCPart compare with other splitting methods that are already available, such as Butina splits, scaffold splits, cluster splits etc.? \n\nTable 1, I think the ranking should be 2,4,1,4 for the models read left to right?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Authors presented a detailed analysis of AU-GOOD, and presented a CCPart for splitting sets based on pairwise similarities. CCPart can be applied to any dataset that has useful pairwise similarity metrics in the data, which means it can be applied to other datasets outside of biochemical applications. The authors perform a wide variety of tests on various datasets, and also study various protein language model embeddings, and their performances on OOD. The authors take care to use statistical tests to confirm statistical significance.", "weaknesses": "The font in the figures are extremely small. Please enlarge to improve readability. Especially the plots in the SI, ie. SI fig 1. \n\nThe use of pairwise similarity measures heavily restricts the size of the dataset that can be split. As such, most datasets the authors look at are on the smaller end. This also means that the models studied are usually traditional ML methods (KNN, SVM, RF, etc.).\n\nVarious works have already tried structure-only fingerprints-based splitting methods. This includes work done by the MOOD paper from Tossou et al. (2024), DeepChem from Ramsundar et al. (2019), and Dionysus from Tom et al. (2023). In particular, CCPart is quite similar to Butina or scaffold splitting (from DeepChem), which gathers clusters of fingerprints or scaffolds and sets a cutoff threshold for Tanimoto similarity. \n\nOverall, the AUGOOD metric, and the GOOD curve, is not convincing to me as a reliable measure of OOD performance. Looking at the curves in SI Fig 2, the variations in the GOOD curve are not as well behaved as those in the main text, such as Fig 1, or Fig 5. Fitting a linear line to the GOOD curve to attain a \"slope\" also seems strange considering the huge variations in the performance metric. The curves also dramatically change as the performance metric is changed (ie. between Matthew vs Spearman vs Pearson coefficients). The error bars are very large on the performance metrics, but the authors seem to use the metrics of model performance interchangeably.\n\nAdditionally, the choice of AU the GOOD curve is confusing to me. Take for example, Fig 1A, where the authors conclude that Model A is less generalizable than B, because B performes similarily across the different similarity thresholds. However, the area under the curve would actually be quite similar for model A and B. While this is an example to illustrate the GOOD curve, it is also an example that shows how AUGOOD may not be a useful metric. The authors may defer to the slope of a linear fit, but as seen in Fig 2 of SI, the GOOD curve is rarely linear, or even monotonic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hSRODI35SU", "reviewer": "Reviewer_mZMB", "summary": "This paper introduces a formalized approach for creating training and test splits that assesses the generalization performance of a machine learning algorithm on biological data as a function of a (chosen) similarity metric between data points.", "questions": "* I would ask the authors to think of ways to demonstrate that their approach outperforms other existing methods.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "* The paper is generally well written.\n* It is broadly applicable to a wide range of biological domains given the generality of the similarity metric based approach.", "weaknesses": "* The paper is written in an overly formalized manner. Sections 2.1 and 2.2 say very little that’s non-obvious. It can be described in 1-2 paragraphs of plain English. The over-formalization of very simple concepts seem largely intended to make the matter seem more complex than it actually is. Similarly for sec 2.3, which introduces more novel methods and is not strictly a pedagogical section, but the description is far too long and unnecessarily mathematical.\n* While the basic concepts are overly formalized, concepts that could actually use more formal precision, e.g., the behavior of different GOOD curves, are instead described using phrases like “visually incoherent GOOD curves” without actually specifying what “visually incoherent” means.\n* In most biological domains, data points are evolutionarily related (e.g. proteins, species, etc), and so the assumption that there will be disconnected subgraphs is not really correct, at least not for stringent levels of generalization. I.e., it might be possible to find a threshold by which the graph does become disconnected, but that threshold would only segregate highly similar data points and would result in a very easy test.\n* In many biological domains, data points are noisy, and it’s quite possible that the type of disconnected subgraphs found in the proposed algorithm would basically correspond to pathological outliers. Thus I would caution against naive application of this approach as it may result in test sets comprising largely unrepresentative data points.\n* Ultimately there’s little utility / novelty here. The approach does not introduce anything substantially different from the existing literature, and in the places where it does differ, it is hard to judge whether this approach will actually work in practice. The experiments performed are not compared to any other than baseline and the discussion about the resulting GOOD curves is largely informal and speculative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)