Lo-Hi: Practical ML Drug Discovery Benchmark
Authors: Simon Steshin
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Molecular Property Prediction
Benchmark
Hit Identification
Lead Optimization
ML Drug Discovery
TL;DR: In Hit Identification, practitioners search for new molecules, but test ML models on molecules similar to the training set. In Molecular Optimization, researchers assume models can distinguish minor modifications, but do not test this assumption.

Abstract: Finding new drugs is getting harder and harder. One of the hopes of drug discovery is to use machine learning models to predict molecular properties. That is why models for molecular property prediction are being developed and tested on benchmarks such as MoleculeNet. However, existing benchmarks are unrealistic and are too different from applying the models in practice. We have created a new practical \emph{Lo-Hi} benchmark consisting of two tasks: Lead Optimization (Lo) and Hit Identification (Hi), corresponding to the real drug discovery process. For the Hi task, we designed a novel molecular splitting algorithm that solves the Balanced Vertex Minimum $k$-Cut problem.  We tested state-of-the-art and classic ML models, revealing which works better under practical settings. We analyzed modern benchmarks and showed that they are unrealistic and overoptimistic.

Review: https://openreview.net/forum?id=H2Yb28qGLV

Lo-Hi benchmark: https://github.com/SteshinSS/lohi_neurips2023

Lo-Hi splitter library: https://github.com/SteshinSS/lohi_splitter

Lo-Hi: Practical ML Drug Discovery Benchmark
Simon Steshin Independent Researcher s simon. www.sitggaaiiill...... com
Abstract
Finding new drugs is getting harder and harder. One of the hopes of drug discov- ery is machine learning models predict molecular properties. That is why models for molecular property prediction are being developed and tested on benchmarks such MoleculeNet. However, existing benchmarks unrealistic and are too different from applying the models in practice. We have created a new practical Lo-Hl benchmark consisting of two tasks: Lead Optimization (Lo) and Hit Identification (Hi), corresponding to the real drug discovery process. For the Hi task, we designed novel molecular splitting algorithm that solves the Balanced Vertex Minimum k-Cut problem. We tested state state-of-the-as of-the-art and classic ML models, revealing which works better under practical settings. We analyzed modern benchmarks and showed that they are unrealistic and overoptimistic.
Review: jattps:/ http:://wwt..oooooeeeccaaaaaaaaa Lo-Hi benchmark: Lo-Hi splitter httppppr//gtttaocccm///eeaassi/p
Introduction
Drug discovery the process of identifying molecules with therapeutic properties 0 serve as drug, molecule must possess multiple properties simultaneously [2]. It must be stable [3], yet easily eliminated from the body (4) able to reach its target [5 non-toxic 8 cause minimal
side effects [9], and therapeutically active the target [10
To identify such molecules, researchers develop Molecular Property Prediction (MPP) models. These models are used in virtual screening, during which the models are used to make predictions for large number of molecules. after which the molecules with the best estimated property are selected
for experimental validation [2 13 15 16].
To enable comparisons between various architectures, the research community relies standardized benchmarks to evaluate model performance [17 [8] 19 20 21 22 23]] The prevailing assumption is that models with superior metrics these benchmarks more suitable for real-world applications.
believe this assumption is false. Modern benchmarks test in non-realistic conditions on impractical tasks. In this paper, introduce two practical drug discovery ML tasks Hit Identification and Lead Optimization - that are often encountered in most drug discovery campaigns. We demonstrate that none of the benchmarks assess models for these tasks, which is why propose seven new practical datasets that better imitate real-life drug discovery scenarios. To prepare Hi datasets, we designed novel molecular splitter algorithm that solves Balanced Vertex Minimum C-Cut problem using Integer Linear Programming with heuristics.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Hit identification (Hi)
Lead Optimization (Lo)
Train Test
Train
*
Figure 1: Hit Identification (Hi) task
Figure 2: Lead Optimization (Lo) task
Practical drug discovery
2.1 Hit Identification (Hi)
Drug discovery involves multiple stages. When the potential mechanism of action is known, an early step Hit Identification. During this phase, chemists search for a "hit" molecule with the potential to become drug [24]. A viable hit must exhibit some level of activity towards the target (c.g., Ki less than 10 μM) and possess novelty, meaning it eligible for patent protection.
Clinical trials can incur costs in the hundreds of millions of dollars [25] 26), making it risky to pursue non- non-patentnn non-patentable molecules. Consequently, companies prioritize patentable molecules from the outset. Novelty is an essential aspect of this process. In practice, medicinal chemists often pre-filter their chemical libraries before the virtual screening, eliminating molecules that exhibit Tanimoto similarity above a specific threshold to molecules with known activity 10
16 27
2.2 Lead Optimization (Lo)
Following Hit Identification, the next stage is Lead Optimization. Once a novel, patentable molecule with activity towards the desired target is identified, its closest analogs also likely to exhibit activity [30 In this phase, medicinal chemists often make minor modifications to the hit molecule enhance its target activity, selectivity, and other properties [31 [32 Unlike in Hit Identification, novelty usually is not a priority during Lead Optimization. Instead, the objective to discover molecules that are similar the original hit but possess improved characteristics.
This is related to the field of goal-directed molccular generation [33], which occasionally formulated as the problem of searching for molecules with maximal activity within the e-neighborhood of known hit 34 35), within some scaffold [36), or as an optimization process in the latent space [37 These works use predictive ML models to distinguish between minor chemical modifications [38 The effectiveness of these models in their ability to distinguish between molecules with small modifications remains unclear.
2.3 Model Selection
In this context, Lead Optimization (Lo) and Hit Identification (Hi) represent contrasting tasks. Hi ML models are expected to demonstrate their generalization capabilities by predicting properties of molecules significantly different from the training set. Conversely, Lo ML models should predict properties minor modifications of molecules with known activity.
For selecting the appropriate ML model or adjusting its hyperparameters. it is crucial to test the model under conditions similar to its intended application. Hi, models must identify novel molecules markedly different from known active ones, implying the models should be tested on molecules distinctly different from the training set. Conversely, Lo applies MI. models to molecules resembling known active ones. The null Strivture-Activity Relationship hypothesis assumes that

---

small modifications will not alter the molecule's properties 30 39]. Consequently, Lo ML models should be evaluated based on their ability to make predictions that surpass the null hypothesis.
We show that modern benchmarks mix these two scenarios, which is why it remains unclear how effectively models can generalize truly novel molecules and how proficiently they can distinguish minor modifications, thereby guiding molecular optimization. Furthermore, we untangle these scenarios using& novel benchmark, demonstrating that different architectures are better suited for different tasks.
3 Novelty
But how to measure novelty? From commercialization and regulatory perspective, a novel molecule is one that be patented. Contemporary patents consist of human- human-written written text and incorporate non-trivial substitutions, rendering the assessment of a molecule's s patentability a complex task for legal professionals. To date, this process has not been automated. Additionally. during the Hit Identification, not only must the hit be patentable - its neighborhood should be patentable as well to make Lead Optimization possible. When dealing with an extensive chemical library, these challenges make patentability an impractical criterion for determining novelty.
While chemists agree that minor substitutions likely do not alter molecule's function, consensus on the distinction between a new molecule and modified version an existing one remains elusive. One might hope to find a general similarity threshold that would separate similar molecules with presumably the same activity from distinct molecules. This hope led the "0.85 myth" [40], which proved be false due the significant variability of such thresholds across different targets [41]. About molecular similarity, was said [42 41), "Similarity is in the eye of the beholder." Nevertheless, practical criterion exists.
In study 4 143 experts from regulatory authorities(FDA, FDA of Taiwan, EMA, PMDA) were shown 100 molecular pairs and asked if the molecules should be regarded as structurally similar. This study is especially notable because it simulates real-life scemario in which a regulatory committee (EMA's Committee for Medicinal Products for Human Use) decides if a new drug is novel enough to grant it a beneficial status: orphan drug designation. The study shows that when two molecules have Tanimoto similarity x 0.4 with ECFP4 fingerprints 44 half of the experts regard them dissimilar, which is sufficient to conclude the novelty of the drug. See Appendix for our reproduction.
While Tanimoto similarity with ECFP4 fingerprints has its own disadvantages such as size bias 45 its simplicity makes possible quickly measure the similarity of molecules, so there are practical [6] and theoretical 46 47 works that use "Taanmmoto similarity < 0.4" novelty criterion. Because of the practical evaluation in real-life scenarios and efficiency, we assess novelty using Tanimoto similarity with ECFP4 fingerprints in our benchmark.
4 Our contribution
We suggest two new practical drug discovery ML tasks - Hit Identification and Lead Optimization that better imitate real-life scenarios;
We designed novel molecular splitting algorithm for the Hi task;
We propose seven new practical datasets;
We demonstrate that modern ML drug discovery benchmarks simulate impractical scenarios;
We evaluate modern and classic ML algorithms on our benchmark.
Modern benchmarks test neither Lo nor Hi
To select a model or its hyperparameters, we must evaluate them under the conditions in which they will be used. We demonstrate that none of the modern benchmarks correspond to realistic conditions, thus raising questions about their suitability for evaluating practical machine learning models. Although it is impossible to examine every existing benchmark due to their sheer number, we have opted to assess variety of benchmarks using different data, different preprocessing techniques, and originating from different authors. We will initially focus standard benchmarks, while in
3

---

ESOL train
HIV train
TDC.CYP2D6__eet th train
Moldata train
ESOL test
HIV test
CYP2D_ Veith test
Moldata test
Figure Common benchmarks contain highly similar molecules across different splits.
we will explore more exotic and specialized ones. We provide additional analysis in Appendix
the section 6
5.1 MoleculeNet: ESOL
MolcculeNet [17] is a widely-used benchmark for ML drug discovery, consisting of 17 datasets, including the ESOL dataset with water solubility data. This dataset is frequently used in studies for model comparisons [48 49 50
51 52 53 While the authors recommend a random train-test split, we discovered that it leads to 76% of the test molecules having neighbor in the train with Tanimoto similarity > 0.4, making ESOL unsuitable for Hi scenario (see Fig. B It is also unclear how well the dataset represents the Lo scenario. the distinct 24% of the train contributes to the evaluation, and the RMSE metric can significantly improved over the constant baseline, even without identifying minor chemical modifications.
5.2 MoleculeNet: HIV
In benchmarks, train and test sets are typically split using random, scaffold, or occasionally time splits when time stamps available. It has been frequently observed that random splits result in highly similar molecules in both train and test sets, leading to overly optimistic and impractical estimations. For example: "Random splitting, common in machine learning, is often not correct for chemical data" [17]. Thus, scaffold splitting [54] is sometimes suggested as an alternative.
Scaffold splitting (54 is method where each molecule is represented by graph consisting of ring systems, linkers and side chains. group of molecules may correspond a single graph, which case they are assigned to the same partition.
The MoleculeNet HIV dataset (ubiquitous in evaluation
recommends using scaffold splitting, as it is believed to better reflect the process of discovering new molecules: "As we are more interested in discovering new categories of HIV inhibitors. scaffold splitting [...] is recommended" [17]. Although scaffold splitting makes the train set more distinct from the test set, still insufficient for the scenario of finding novel molecules. We discovered that 56% of HIV test molecules have train neighbor with a Tanimoto similarity > 0.4, indicating the dataset is unsuitable for the scenario (see Fig. 3. Additionally, the dataset unsuitable for the scenario due its binary label, rather than continuous value.
55
5.3 Therapeutic Data Commons: TDC. CYw.06_V2VV..1ee
The Therapeutic Data Commons [8 novel platform designed for evaluating machine learning models in drug discovery. We examined the largest dataset, TDC. CYP.D6. CYP2D6_Veith, within the Single- Instance Learning Tasks category. Although a scaffold split is recommended, the dataset deemed unsuitable for the Hi scenario, as 78% of test molecules possess training neighbor exhibiting Tanimoto similarity greater than 0.4 (see Fig. 3}. Additionally, the dataset is unsuitable for the scenario due to its binary label, rather than continuous value.

---

5.4 MolData
MolData [9 is recent benchmark for MPP based on PubChem data. The authors themselves analyzed the novelty of the molecules within the subset and found their scaffold split to be unrealistic: "[...] more than 44% of the molecules within the MolData dataset have at least one other similar molecule to them with Tanimoto Coefficient of 0.7 or higher. This high percentage of the similarity can denote lack of diversity within this portion of the dataset."
The authors use scaffold split to increase the difference between the train and the test, but we found that least 88% of the molecules in the test have similar molecule in the train with a Tanimoto similarity > 0.4, making the dataset unsuitable for the Hi scenario (see Fig. 3 Additionally, the dataset is unsuitable for the Lo scenario due to its binary label, rather than continuous value.
Related works
6.1
Out-of-distribution MPP
In the Hi scenario, the training and test sets are significantly different to simulate the search for new molecules, making the Hi scenario interpretable an Dut-Of-Distribution (OOD) task. Although OOD benchmarks already exist for machine learning-bai based drug discovery, they do not align with practical drug discovery applications.
DrugO0D 20 is drug discovery benchmark dedicated Out-Of-Distribttion prediction. The authors accurately observe that "In the field of Al-aided drug discovery, the problem of distribution shift [...] is ubiquitous", and therefore suggest assay-based, scaffold-based, or molecular size-based partitions to create a distribution shift. However, their benchmark does not correspond to practical involves predicting average activity for all ChEMBL targets simultaneously in ligand-based drug discovery datasets. This approach is impractical because, in reality, researchers are not interested in average activity across ChEMBL targets, but rather in activity on a specific target.
drug discovery,
GOOD (1 is a benchmark explicitly designed to separate covariate and concept shifts. We examined the GOOD-HIV HIV dataset, as it is MPP benchmark containing a significant amount of diverse data (see the #Circles in Appendix[A A). In all Out-Of-Distrbuttion test partitions, more than 40% of molecules
have neighbor in the training unsuitable for the Hi scenario.
Tanimoto similarity more than 0.4, making the dataset
During the NeurIPS review period, Valence Labs and Laval University jointly published work independent from ours - investigating research is akin to our Hi-scenario, but the authors studied different data splits, employed different models, and investigated uncertainty calibration.
generalization [60]. Their
6.2 Activity cliff
Activity cliff is a pair of structurally similar compounds that are active against the same bio-target but significantly different in binding potency [6] 22).
Recently, several remarkable studies have emerged, seeking to establish a standard benchmark for Activity Cliff Prediction 22 2 These works have inspired us to create our own benchmark. While Activity Cliff Prediction may be advantageous in the Lo scenario, it is inadequate for guiding molecule optimization. In the Lo scenario, is essential not only to identify extreme activity cliffs with 10x difference in activity but also to predict minor activity fluctuations. We propose alternative splits and metries in the Lo scenario.
7 Results
7.1 Hi-splitter
To simulate the Hi scenario, necessary to divide the dataset into training and testing subsets such that any pair of molecules from different partitions has ECFP4 Tanimoto similarity of less than 0.4. Despite the fact that the scaffold splitter produces more diverse division than random splitter, we observe that it is inadequate for an effective Hi scenario.
5

---

A greedy approach is to implement the conventional scaffold split and discard from the test set any molecules excessively similar to those in the training set. The issue with this method, however, is that data points are costly, and it is desirable to minimize the number of discarded molecules. Consequently, we have developed a novel algorithm for strict dataset splitting, which discards fewer molecules than the greedy algorithm.
Let us consider molecules X [I1 We construct a neighborhood graph G (V, B), where each molecule I; corresponds to a vertex e E V. Two vertices are connected by an edge if and only if the associated molecules have a similarity to each other greater than threshold t: evu E E + T(Ie, > t, where T is Tanimoto Similarity with ECFP4 fingerprints. In such graph, connected components can be assigned to the training or testing sets independently. However, in practice. 95% of the molecules belong to a single connectivity component. Our goal is to remove the minimum number of vertices such that the giant component breaks up into multiple components with size constraints, thereby enabling us to distribute them between the training and testing sets.
This problem is known as the Balanced Vertex Minimum k-Cut and has been extensively researched in literature [62 63 64]. A review of similar problems can be found elsewhere [65 66]. Similar to we formulate the Integer Linear Programming formulation.
22
Let K denote the set of integers (1,2,...ke k}. Let G = (V, E) represent a simple connected graph that we going to split:
V T-U Vi V /
For all vertices E V and for all integers i K, let associate binary indicator yt such that:
1, if E V/ yb 0, otherwise
Note that Vo denotes the set of removed vertices, so if Eiek vi = O then the vertex v is in Vo-
Let by E N be lower bound on the cardinality [V/ of partition V,. We need it to get partitions with size constraints, e.g. 80% in train and 20% in test. Let Wa be the weights of the nodes. In simple formulation we take we We formulate the Balanced Vertex Minimum k: Cut as follows:
max ΣΣ
w.y'
JEKUEL
Σ U S1
AOEV
(2)
KK
2 + vol 1
Vi t K, Veau E
(3)
Σい
VieK
OEV
1 E 10
WEEEVVV
(5)
Equation (1) minimizes the weight of removed molecules Vo. Equation (2) states that each vertex should be in one partition maximum. Equation 3 ensures there is no connectivity between different partitions, so for each edge evv E E if is in V. partition, then the other vertex is either in V as well, or in Vo, meaning it was removed. Equation (4) puts constraints on the size of the partitions V.
While this formulation was effective on small-scale graphs (around 100 vertices), we found it too slow for a real DRD2 activity dataset with 6k molecules. To our knowledge, existing literature [62 65 67 68) typically involves small graphs from the standard benchmarks with maximum node count of several hundred. To expedite computation, we implemented graph coarsening approach. The basic idea is outlined here, while the formal algorithm is detailed in Appendix E
We initially performed Butina clustering (69] on molecules and created coarse graph wherein the vertices correspond not to individual molecules but to clusters of molecules. In the coarsened graph,

---

each vertex is assigned weight we equal to the number of molecules in the respective cluster. This approach accelerated computations and enabled us partition the HIV dataset consisting of 40k molecules, removing fewer vertices than would be the case with the greedy approach (See Table
Table 1: Number of removed molecules for 0.9:0.1 split
Method
DRD2
HIV
Greedy
1066 (17.0%)
5851 (14.2%)
Hi-Splitter
97 (1.5%)
1598 (3.8%)
7.2 Lo-Hi benchmark
We conducted an analysis of numerous drug discovery benchmarks, yet seemed to align with the actual drug discovery process. Consequently, we prepared seven datasets and are now making them available to the community.
We selected datasets that represent realistic drug discovery problems, contain substantial amounts of qualitative data, and cover diverse chemical space. Diversity was assessed using the recently introduced #Circles metric [70]. As the source code was unavailable at the time of writing, we employed own implementation of the greedy algorithm outlined in [70]: Appendix H. We provide additional statistics for the original datasets in the Appendix(A]
We propose three distinct train-test splits for each dataset. We advise adjusting hyperparameters solely on the first split, applying the same hyperparameters to train and assess models on splits #2 and #3, and comparing models by averaging metrics across the splits. Datasets are released under the MIT license.
7.2.1 Hi
In the Hit Identification scenario, we aim to predict binary labels for new molecules that significantly differ from those the training set. We prepared four datasets which we divided into training and testing sets, ensuring that the Tanimoto similarity between any molecule in the test set and those in the training set less than 0.4. See Fig. []] Additionally, we show that such a split predicts experimental outcomes better than the scaffold split (Appendix F).
DRD2-Hi involves predicting dopamine receptor inhibition, GPCR target of therapeutic importance in schizophrenia 72] and Parkinson's 's disease [73] To create this dataset, we obtained Ki data for DRD2 from ChEMBL30 (75). cleaned it (see Appendix A),, and binarized it so that molecules with Ki < 10uM are considered active.
HIV-Hi is HIV dataset from the Drug Therapeutics Program AIDS Antiviral Screen that measures the inhibition of HIV replication. We obtained the prepared dataset from MoleculeNet [17].
DRD2-Hi and HIV-Hi are large. However, real-life data often comes in limited quantities. To simulate this crucial scenario, we prepared smaller challenging dataset, KDR-Hi. This dataset is based on the ChEMBL IC50 data associated with vascular endothelial growth factor receptor 2, kinase target for cancer treatment (76). Its creation process was similar to that of DRD2-Hi, but we restricted the training folds to just 500 molecules.
The Sol-Hi dataset draws from a public solubility dataset at Biogen 777. such that molecules with a solubility of less than 10 pg/mL were assigned a positive label.
We binarized this data
Each dataset was divided into distinct training and testing sets. We used the Hi-splitter with A 3 to obtain three highly dissimilar subsets: {Fi, F2, F3)- These subsets were then combined to create three distinct folds, each with unique test set. For instance, for the first fold, the training was train, = Fi, F2} and the test set was lest, }. This methodology enabled us to assess the variability in quality resulting from using different data with the same models
Hi Metric
For our benchmark, we have selected the PR AUC. As a simple binary classification metric without parameters, it is implemented in most libraries and normalized to range of [0. 1].
'The DRD2-Hi preparation code be found at notebooks/tata//03. swwss_dprdddppspppppppppppppppla ipynb.
7

---

The PR AUC favors early recognition models [78 and does not appeal to wrong intuition among readers in an unbalanced setting.
7.2.2 Lo
In the Lead Optimization scenario, aim predict the activity of molecules that are highly similar to those in the training set. As similar molecules tend to exhibit similar activity, our focus is not on predicting binary labels, but rather on ranking, which indicates whether modification increases the activity or not.
To simulate the Lead Optimization scenario, we isolated clusters of molecules with intracluster similarity 2 0.4 and consisting of 2 5 molecules. We included them in the test dataset. In practical Lead Optimization, the activity of given hit is already known; thus, for each cluster, we retained exactly one molecule with a similarity 2 0.4 to that cluster in the training set. See Fig. 2 We provide additional analysis in Appendix [
In order to confirm the validity of the Lo benchmark, we ensured that the intracluster variation in activity exceeds the experimental noise (See Appendix [B]. This step is critical, as if the variance within a cluster of similar molecules not significantly greater than the experimental noise, it would not make sense to test models such molecules under these conditions, even ideal model would struggle to make accurate predictions. The formal pseudocode can be found in Appendix C
We assembled three datasets: DRD2-Lo, KCNH2-Lo and KDR-Lo. Both DRD2-Lo and KDR-Lo are based on the same data as their respective Hi counterparts but feature different split between training and testing sets. KCNH2 is an ion channel that regulates heartbeat, and its inhibition can cause dangerous side effects (79.. Consequently, bioassays for KCNH2 are used as a screening method for cardiotoxicity. We extracted IC50 data from ChEMBL30, cleaned it, and divided it into training and testing sets.
Each dataset was split three times using different random seeds, resulting in three folds. This method enables us to assess the variability in quality when using different data with the same models
Lo Metric Our goal is to determine whether the models can make better predictions than assuming "the modified molecule active in the same manner as the original hit." We chose Spearman's 's correla- metric, calculated within each cluster and averaged across clusters. This metric
tion coefficient does not rely on intracluster variation, depends solely on the ranking of molecules within the cluster, and is normalized, between minus one (ideally wrong), zero (random) and one (ideal), rendering easily interpretable.
7.3 Evaluation
We evaluated both traditional and state-of-the-art ML models on our benchmark. We meticulously executed the hyperparameter search, following the procedure outlined in Appendix H Results are presented in Table [] The best scores for fingerprint and graph models are highlighted in bold. should be noted that the mean and standard deviation were calculated not for random seeds, but within different folds.
We found that the best models varied for the Hi and Lo tasks. The most effective model for the Hi task was the Chemprop graph neural network [80), aligning with its real-world success [[6 28 29 The second-best were gradient boosting and KNN for the small KDR-Hi Hi dataset.
Conversely, for the Lo task, the SVM model was found to be the most proficient, which corroborates previous works on activity cliffs (33 Only for the small KDR-Lo did Chemprop outperform SVM, but even then, the performance was not satisfactory. We further provide per-cluster Spearman distri- bution for SVM in Appendix[[ [[] The limited inability of Chemprop to distinguish minor modifications may be due the limited expressivity of graph neural networks. However, this was surprising to us, considering the limited expressivity of binary fingerprints as well.
"The DRD2-Lo preparation code can be found at nwtebookss/tata//4..coo spdpsppppppp__d____pppp 1o. ipynb.

---

Table 2: Lo-Hi results. PR AUC for Hi and mean Spearman for Lo.
Model
DRD2-HE
HIV-Hi
XD8-Hi
Bol-Bi
DED2-1o
KCMB2-Lo
XDB-1e
Dommy baseline
0.677=0.061
0.04+00014
0.609H0.081
0.21540.008
0.000:0.0
0.000s0.0
0.000+0.0
KNN (Tamimato distance, ECFP4)
0.70640.044
0.067=0.029
0.646200048
0.426a0.022
0.19520.053
0.16420.014
0.130+0.034
KNN (Tanimeto distance, MACCS)
0.702:0.042
0.072=0.036
0.610=0.072
0.422:0.000 009
0.211:0.041
0.036=0.022
0.07
Gradient Boostring (ECFP4)
0,08
0.607a0.067
0.429:0.000
0.145+0.052
0.3720.003
0.07620.036
Gradient Boostring (MACCS)
0.751=0.063
0.058=0.03
0.603+0.004
0.502#0.045
0. 1977+00043
0.216=0.052
0.100+0.026
SVM (ECFP5)
0.677=0.061
0.04+0.014
160.08
0.298+0.047
0.311=0.015
0.472=0.014
0.158+00051
SVM (MACCS)
0.01240:015
0.60510.088
0.30840.021
0.219
0.133g0.024
0.07440.024
MLP (ECTP4)
0.717=0.063
0.049=0.019
0.626:0.047
0.402+0.017
0.094+0.059
0.146:0.040
0.08540.0300
MLP (MACCS)
0.69640.048
0.052a0.018
0.61320.077
0.46240.048
0.02640.083
0.174g0.031
0,06540.027
Chemprop
0.782+0.062
0.148=0.114
0.676#0.026
0.618#0.03
0.298=0.003
0.375=0.067
0.161=00024
Graphormer SJ 82
0.72910.039
0.09620.070
8 Conclusion
In virtual screening, practitioners aim to discover novel molecules and filter their chemical libraries using Tanimoto similarity. Despite this common practice, there are no benchmarks that simulate this particular scenario. In molecular optimization, researchers employ predictive models to guide the optimization process in step-by-step manner. However, it remains unclear whether these models possess the capacity to distinguish minor modifications.
We identified several limitations in current drug discovery benchmarks and proposed more realistic and practical alternative in the form of the Lo-Hi benchmark. By introducing two tasks, Lead Optimization (Lo) and Hit Identification (Hi), which closely resemble real drug discovery scenarios, we created an environment to evaluate machine learning models under more representative conditions. We emphasize the importance of testing models under conditions similar to their intended application.
Furthermore, the paper critically assesses existing benchmarks and related works, highlighting their inadequacies and the need for better evaluation methods. To address these issues, we suggest alternative datasets. To build them, we designed novel molecular splitter algorithm for the Hi task.
Different models proved to be better suited to different tasks. Our evaluation of both classical and modern ML models revealed Chemprop the state-of-ther oftth-of-hhe-rt for the Hi task, and SVM with ECFP4 fingerprints as the state-of-thh-an for the Lo task.
The paper's key contributions include the introduction of the Lo-Hi benchmark, a comprehensive analysis the limitations of modern ML drug discovery benchmarks, a novel molecular splitter algorithm and the evaluation of modern and classic ML algorithms on the proposed benchmark. This work sets the stage for a more accurate and reliable evaluation of machine learning models in the field of drug discovery, ultimately leading to better decision-making and improved outcomes in the search for new therapeutic compounds.
Limitations
We have conducted hyperparameter tuning, although performing it thoroughly poses challenge [+ Convincing evidence supporting particular architecture could be garnered from an open online contest with prizes, accompanied by undisclosed test dataset. We faced numerous technical difficulties in executing and modifying Graphormer (see Appendix H.7). As such, we cannot definitively determine if Graphormer's failure is consequence of its architecture or the result of improper dependency pinning by the authors.
84
Our findings indicate that the Integer Linear Programming solution proves to be significantly more effective than the greedy approach. However, we have not explored different formulations in our study. Therefore, it is possible that more efficient methods for splitting molecular datasets could exist.
We endeavored to encompass a diverse range of ligand-based drug discovery problems (GPCR inhibition, kinase inhibition, cardiotoxicity, solubility) in our benchmark. However, it is infeasible to capture every potential molecular property prediction task. We advise practitioners to use benchmark results to shortlist models, but also to test them against specific objectives.

---

10
Future work
Our focus was on medium and large datasets, yet many small datasets contain fewer than 100 data- points. It would be beneficial to have smaller datasets similar to [86] but tailored for Hi generalization.
While our emphasis was on ligand-based drug discovery, where the goal is to predict molecule's property, there is also structure-based drug discovery. This approach not only involves predicting molecular properties but also incorporates protein information. Hence, it would be advantageous to have structure-beed basee drug discovery datasets that are divided not just by protein (or pockets [87] similarity but also with Hi generalization across ligands.
major ongoing challenge in molecular generative models is ensuring synthesizabilitty meaning that generated molecules be made in the real world. Hi splits can help test the generalizability of synthesizability models. But, it's important to remember that Lo/Hi splits assume similar molecules have similar properties. While this holds for physico-chemical attributes, this premise remains to be validated in the context of feasibility measures.
11 Potential Harmful Consequences
One of the primary concerns arises from the ability to design molecules that are unfamiliar to medical chemists and experts in the field. While the novelty of these compounds can be advantageous for pushing the boundaries current scientific knowledge, it also raises the potential risk of misuse, especially if malicious actors were use the system to generate harmful or toxic compounds for hostile purposes. Given their unfamiliar nature, these molecules might not immediately raise flags upon review by experts during synthesis orders chemical laboratories. This could open the door for the creation and dissemination harmful compounds.
12 Acknowledgements
This work was funded by Gleb Pobegailo.
References
[1] Shu-Feng Zhou and Wei-Zhu Zhong. 2017.
Drug design and discovery: principles and applications,
[2] Jing Lin, Diana C Sahakian, SM De Morais, Jinghai J Xu, Robert J Polzer, and Steven Winter. The role of absorption, distribution, metabolism, excretion and toxicity in drug discovery. Current topics in medicinal chemistry, 3(10): 1154, 2003.
[3] Sabina Podlewska and Rafal Kafel. citiaaboonooliigg platform for metabolic stability predic- tions. International journal of molecular sciences, 19(4):1040, 2018
[4] Mohsen Sharifi and Taravat Ghafourian. Estimation of biliary excretion of foreign compounds using properties of molecular structure. The AAPS journal, 16:65-78, 2014.
[5] Patrizia Crivori, Gabriele Cruciani, Pierre-Alain Carrupt, and Bernard Testa. Predicting blood- brain barrier permeation from three-dimensional molecular structure. Journal of medicinal chemistry, 43( (3(11):2204-22 2216, 2000.
[6] Qiang Tang, Fulei Nie. Qi Zhao, and Wei Chen. A merged molecular representation deep learning method for blood-braie barrier permeability prediction. Briefings Bioinformatics. 23(5), 2022.
[7] Raimund Mannhold and Han Van de Waterbeemd. Substructure and whole molecule approaches for calculating log p. Journal of Computer-Aided Molecular Design, 15:337-354, 2001.
[8] Ashok K Sharma, Gopal N Srivastava, Ankita Roy, and Vineet K Sharma. Toxim: a toxicity prediction tool for small molecules developed using machine learning and chemoinformatics approaches. Frontiers in pharmacology, 8:880, 2017.
10

---

[9] Rakesh Kanji, Abhinav Sharma, and Ganesh Bagler. Phenotypic side effects prediction by optimizing correlation with chemical and target profiles of drugs. Molecular BioSystems, 11 (1 (11):2900-2006, 2015.
[10] Brian J Bender, Stefan Gahbauer, Andreas Luttens, Jiankun Lyu, Chase M Webb, Reed M Stein, Elissa A Fink, Trent Balius, Jens Carlsson, John J Irwin, et al. A practical guide to large-scale docking. Nature protocols, 16(10):4799 4832, 2021.
[11] Reed M Stein, Hye Jin Kang, John D McCorvy, Grant C Glatfelter, Anthony J Jones, Tao Che, Samuel Slocum, Xi-Ping Huang, Olena Savych, Yurii S Moroz, et al. Virtual discovery of melatonin receptor ligands to modulate circadian rhythms. Nature, 579(7800):609
2020.
[12] Ralf Mueller, Alice Rodriguez, Eric Dawson, Mariusz Butkiewicz, Thuy T Nguyen, Stephen Oleszkiewicz, Annalen Bleckmann, C David Weaver, Craig W Lindsley, P Jeffrey Conn, et al. Identification of metabotropic glutamate receptor subtype 5 potentiators using virtual high- throughput screening. ACS chemical neuroscience, ((4):288-303) 2010.
[13] Liying Zhang, Denis Fourches, Alexander Sedykh, Hao Zhu, Alexander Golbraikh, Sean Ekins, Julie Clark, Michele C Connelly, Martina Sigal, Dena Hodges, et al. Discovery of novel antimalarial compounds enabled by gsar-based virtual screening. Journal of chemical information and modeling, 53(2):475 492, 2013.
[14] Bruno J Neves, Rafael F Dantas, Mario R Senger, Cleber C Melo-Filho, Walter CG Valente, Ana CM de Almeida, João M Rezende-Neto, Elid FC Lima, Ross Paveley, Nicholas Furnham, al. Discovery of new anti-schistosomal hits by integration of qsar-based virtual screening and high content screening. Journal of medicinal chemistry, 59(15)70775 -7880 2016.
[15] Georges E Janssens, Xin-Xuan Lin, Lluís Millan-Ariñoo, Alan Kavšek, llke Sen, Renée Seinstra, Nicholas Stroustrup, Ellen AA Nollen, and Christian G Riedel. Transrriptomies-aseed screening identifies pharmacological inhibition of hsp90 as a means to defer aging. Cell Reports, 27(2): 467-480, 2019.
[16] Jonathan M Stokes, Kevin Yang. Kyle Swanson, Wengong Jin, Andres Cubillos-Ruiz, Nina M Donghia, Craig R MacNair, Shawn French, Lindsey A Carfrae, Zohar deep learning approach to antibiotic discovery. Cell, 180(4):688-702 2020.
et al.
[17] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9():5113-330, 2018.
[18] Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learn- ing datasets and tasks for drug discovery and development. arXiv preprint 202.v:210. 2021.
[19] Arash Keshavarzi Arshadi, Milad Salem, Arash Firouzbakht, and Jiann Shiun Yuan. Moldata, a molecular benchmark for disease and target based machine learning. Journal of Cheminformat- ics, 14(1):1-18, 2022.
[20] Yuanfeng Ji, Lu Zhang, Jiaxiang Wu, Bingzhe Wu, Long-Kai Huang, Tingyang Xu, Yu Rong, Lanqing Li, Jie Ren, Ding Xue, al. Drugood: benchmark for ai-aided drug discovery- a focus on affinity prediction problems with noise annotations. arXiv preprint (tXiv:2201.09633 2022.
(ood) dataset curator and
[21] Shurui Gui, Xiner Li, Limei Wang, and Shuiwang Ji. Good: A graph out-of-distribution benchmark. arXiv preprint arXiv:2266.08522 2022.
[22] Ziqiao Zhang, Bangyi Zhao, Ailin Xie, Yatao Bian, and Shuigeng Zhou. Activity cliff prediction: Dataset and benchmark. arXiv preprint arXiv:2302. 07541, 2023.
[23] Derek van Tilborg, Alisa Alenicheva, and Francesca Grisoni. Exposing the limitations of molecular machine learning with activity cliffs. Journal of Chemical Information and Modeling, 62(23):5938-5551 2022.

---

[24] György M Keserú and Gergely M Makara. Hit discovery and hit-to-lead approaches. Drug discovery today, 11(15-1))741-788, 2006.
[25] Asher Mullard. How much do phase iii trials cost? Nature Reviews. Drug Discovery, 17(11): 777-777, 2018
[26] Linda Martin, Melissa Hutchens, Conrad Hawkins, and Alaina Radnov. How much do clinical trials cost. Nat Rev Drug Discov, 16(6):381-382, 2017.
[27] Jiankun Lyu, Sheng Wang. Trent E Balius, Isha Singh. Anat Levit, Yurii S Moroz, Matthew J O' O'Meara, Tao Che, Enkhjargal Algaa, Kateryna Tolmachova, et al. Ultra-large library docking for discovering new chemotypes. Nature, 566(7743).224-23) 2019.
[28] Felix Wong, Satotaka Omori, Nina M Donghia, Erica Zheng, and James J Collins. Discovering mooel-molecule senolytics with deep neural networks, Nature Aging, pages 1-17, 2023.
[29] Gary Liu, Denise B Catacutan, Khushi Rathod, Kyle Swanson, Wengong Jin, Jody C Mo- hammed. Anush Chiappino-Pery Saad A Syed, Meghan Fragis, Kenneth Rachwalski, et al. Deep learning- sarning-guided discovery of an antibiotic targeting acinetobacter baumannii. Nature Chemical Biology, pages 1-9, 2023.
[30] RD Brown and YC Martin. An evaluation of structural descriptors and clustering methods for use in diversity selection. SAR and QSAR in Environmental Research, 8(1-2):23-39 1998.
[31] Bie Verbist, Günter Klambauer, Liesbet Vervoort, Willem Talloen, Ziv Shkedy, Olivier Thas, Andreas Bender, Hinrich WH Göhlmann, Sepp Hochreiter, QSTAR Consortium, et al. Using transcriptomies to guide lead optimization in drug discovery projects: Lessons learned from the qstar project. Drug discovery today, 20(5):505-513, 2015.
[32] George Papadatos, Anthony WJ Cooper, Visakan Kadirkamanathan, Simon JF Macdonald, lain M McLay, Stephen D Pickett, John M Pritchard, Peter Willett, and Valerie J Gillet. Analysis of neighborhood behavior in lead optimization and array design. Journal of chemical information and modeling. 49(2):195-28) 208, 2009.
[33] Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher. Guacamol: bench- marking models for de novo molecular design. Journal of chemical information and modeling. 59(3):1096-1 108, 2019.
[34] Tianfan Fu, Cao Xiao, Xinhao Li, Lucas M Glass, and Jimeng Sun. Mimosa: Multi-constraint molecule sampling for molecule optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 125-133, 2021.
[35] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junetion tree variational autoencoder for molecular graph generation. In International conference on machine learning, pages 3323-2332.. PMLR, 2018
[36] Maxime Langevin, Hervé Minoux, Maximilien Levesque, and Mare Bianciotto. Scaffold- constrained molecular generation. Journal of Chemical Information and Modeling, 60(12): 5637-5646, 2020.
[37] William Godinez, Eric J Ma, Alexander T Chao, Luying Pei, Peter Skewes-Cox. Stephen M Canham, Jeremy L Jenkins, Joseph M Young, Eric J Martin, and W Armand Guiguemde. Design of potent antimalarials with generative chemistry. Nature Machine Intelligence, 1(2):180-186,
2022.
[38] Wenhao Gao, Tianfan Fu, Jimeng Sun, and Connor Coley. Sample efficiency matters: a benchmark for practical molecular optimization. Advances in Neural Information Processing Systems, 35:21342-21355 2022.
[39] David E Patterson, Richard D Cramer, Allan M Ferguson, Robert D Clark, and Laurence E Weinberger. Neighborhood behavior: a useful concept for validation of "molecular diversity" descriptors. Journal of medicinal chemistry, 39(16):3049-3055 1996.
12

---

[40] Yvonne C Martin, James L Kofron, and Linda M Traphagen. Do structurally similar molecules have similar biological activity? Journal of medicinal chemistry. (119)):350-3356 2002.
[41] Gerald Maggiora, Martin Vogt, Dagmar Summfe, and Jurgen Bajorath. Molecular similarity in
medicinal chemistry: miniperspective. Journal of medicinal chemistry, 57(8):3186-3200) 2014.
[42] Alex Zhavoronkov and Alán Aspuru-Guzik. Reply to "assessing the impact of generative ai on medicinal chemistry'. Nature Biotechnology, 38(2): (8(2):146-146, 2020.
[43] Pedro Franco, Nuria Porta, John D Holliday, and Peter Willett. The use of 2d fingerprint methods to support the ussessment of structural similarity in orphan drug legislation. Journal of cheminformatics, 6:1-10, 2014.
[44] David Rogers and Mathew Hahn. Extended-connectivv fingerprints. Journal of chemical information and modeling, 50(5):742-75,, 2010.
[45] John D Holliday, Naomie Salim, Martin Whittle, and Peter Willett. Analysis and display of the size dependence of chemical similarity coefficients. Journal of chemical information and computer sciences, 43(3): 119-828, 2003.
[46] Wei Lu, Qifeng Wu, Jixian Zhang, Jiahua Rao, Chengtao Li, and Shuangjia Zheng. Tankbind: Trigonometryawware neural networks for drug-protein binding structure prediction. bioRxiv, pages 2022-06, 2022.
[47] Seul Lee, Jachyeong Jo, and Sung Ju Hwang. Exploring chemical space with score-based out-of- distribution generation. In International Conference on Machine Learning, pages 18872-18892. PMLR, 2023.
[48] Hyeoncheol Cho and Insung S Choi. Enhanced deep-learning prediction of molecular properties via augmentation of bond topology. ChemMedChem, 14(17):1604-160) 1609, 2019.
[49] Yoonho Jeong, Jihoo Kim, Yeji Kim, and Insung Choi. Development of a chemically intuitive filter for chemical graph convolutional network. Bulletin of the Korean Chemical Society, 43(7); 934-936, 2022.
[50] Benson Chen, Regina Barzilay, and Tommi Jaakkola. Path-augmented graph transformer network. arXiv preprint arXiv: 12712, 2019.
[51] Prateeth Nayak, Andrew Silberfarb, Ran Chen, Tulay Muezzinoglu, and John Byrnes. Trans- former based molecule encoding for property prediction. arXiv preprint arXiv:2011.0355 2020.
[52] Ross Irwin, Spyridon Dimitriadis, Jiazhen He, and Esben Jannik Bjerrum. Chemformer: a pre- trained transformer for computational chemistry. Machine Learning: Science and Technology, 3(1):015022, 2022.
[53] Yao Zhang et al. Bayesian semi-supervised learning for uncertainty-calibrated prediction of molecular properties and active learning. Chemical science, 10(35))8154-8163, 2019.
[54] Guy W Bemis and Mark A Murcko. The properties of known drugs. molecular frameworks. Journal of medicinal chemistry, 39(15):2887-289) 1996.
[55] Bihter Das, Mucahit Kutsal, and Resul Das. Effective prediction of drug target interaction on hiv using deep graph neural networks. Chemometrics and Intelligent Laboratory Systems, 230: 104676, 2022.
[56] Garrett B Goh, Charles M Siegel, Abhinav Vishnu, and Nathan O Hodas. Chemnet: A transferable and generalizable deep neural network for small-molecule property prediction. Technical report, Pacific Northwest National LabN(L,,..,, Richland, WA (United States), 2017.
[57] Shion Honda, Shoi Shi, and Hiroki R Ueda. Smiles transformer: Pre-trained molecular fingerprint for low data drug discovery. arXiv preprint arXiv: 2019.
13

---

[58] Chao Shang, Qinqing Liu, Ko-Shin Chen, Jiangwen Sun, Jin Lu, Jinfeng Yi, and Jinbo Bi Edge aatention-based multi-relational graph convolutional networks. arXiv preprint arXiv: 1802.04944, 2018.
[59] Karim Abbasi, Antti Poso, Jahanbakhsh Ghasemi, Massoud Amanlou, and Ali Masoudi- Nejad. Deep transferable compound representation across domains and tasks for low data drug discovery. Journal of chemical information and modeling. 59(1 1):4528-45555 2019.
[60] Prudencio Tossou, Cas Wognum, Michael Craig, Hadrien Mary, and Emmanuel Noutahi. Real- world molecular aut-of-distribution. Specification and investigation. 2023.
[61] Dilyana Dimova and Jürgen Bajorath. Advances in activity cliff research. Molecular informatics. 35(5)1181-11, 2016.
[62] Denis Cornaz, Fabio Furini, Mathieu Lacroix, Enrico Malaguti, A Ridha Mahjoub, and Sébastien Martin. Mathematical formulations for the balanced vertex k-separator problem. In 2014 International Conference Control, Decision and Information Technologies (CoDIT), pages 176-181. IEEE, 2014.
[63] Egon Balas and Cid C de Souza. The vertex separator problem: polyhedral investigation. Mathematical Programming, 103(3):583-608 2005.
[64] Stephan Schwartz. An overview of graph covering and partitioning. Discrete Mathematics, 345 (8):112888 2022.
[65] Denis Cornaz, Fabio Furini, Mathicu Lacroix, Enrico Malaguti, A Ridha Mahjoub, and Sébastien Martin. The vertex k-cut problem. Discrete Optimization, 31:8-28, 2019.
[66] Paolo Paronuzzi. Models and algorithms for decomposition problems. 2020.
[67] Fabio Furini, Ivana Ljubić, Enrico Malaguti, and Paolo Paronuzzi. On integer and bilevel formulations for the k-vertex problem. Mathematical Programming Computation, 12: 133-164, 2020.
[68] Yangming Zhou, Gezi Wang, and MengChu Zhou. Detecting k-vertex cuts in sparse networks via a fast local search approach. IEEE Transactions on Computational Social Systems, 2023.
[69] Darko Butina. Unsupervised data base clustering based on daylight's fingerprint and tanimoto similarity: A fast and automated way cluster small and large data sets. Journal of Chemical Information and Computer Sciences, 39(4):747-750, 1999.
[70] Yutong Xie, Ziqiao Xu, Jiaqi Ma, and Qiaozhu Mei. How much space has been explored? measuring the chemical space covered by databases and muchine-generann molecules. The Eleventh International Conference Learning Representations.
[71] Thelma Beatriz González-Castre Yazmin Hernandez- -Daz. Isela Esther Juárez-Rojop María Lilia López-Narváez, Carlos Alfonso Tovilla-Zárate, Alma Genis-Mendoza, and Mariela Alpuin-Reyes. The role of c957t, taqi and ser31 leys polymorphisms of the drd2 gene in schizophrenia: systematic review and meta-analysis. Behuvioral and Brain Functions, 12(1):
1-14, 2016.
[72] Ritushree Kukreti, Sudipta Tripathi, Pallav Bhatnagar, Simone Gupta, Chitra Chauhan, Shob- hana Kubendran, YC Janardhan Reddy, Sanjeev Jain, and Samir Brahmachari. Association of drd2 gene variant with schizophrenia. Neuroscience letters, 392(1 392(1-2):66 (2((12):68-7111 2006.
[73] V McGuire, SK Van Den Eeden, CM Tanner, F Kamel. DM Umbach, K Marder, R Mayeux, Ritz, GW Ross, H Petrovitch, et al. Association of drd2 and drd3 polymorphisms with parkinson's is disease in multiethnic consortium. Journal of the neurological sciences, 307(1-2): 22-29, 2011.
[74] Dongjun Dai, Yunliang Wang, Lingyan Wang, Jinfeng Li, Qingqing Ma, Jianmin Tao, Xingyu Zhou, Hanlin Zhou, Yi Jiang, Guanghui Pan, et al. Polymorphisms of drd2 and drd3 genes and parkinson's discase: A meta-analysis. Biomedical reports, 2(2):275-21,, 2014.
14

---

[75] David Mendez, Anna Gaulton, A Patricia Bento, Jon Chambers, Marleen De Veij, Eloy Félix, María Paula Magariños, Juan F Mosquera, Prudence Mutowo, Michal Nowotka, al. Chembl: towards direct deposition of bioassay data. Nucleic acids research, 47(D1))19300-9900 2019.
[76] Siddharth J Modi and Vithal M Kulkarni. Vascular endothelial growth factor receptor (vegfr- 2)kkdr inhibitors: medicinal chemistry perspective. Medicine in Drug Discovery, 2: 2:100009, 2019.
[77] Cheng Fang, Ye Wang. Richard Grater, Sudarshan Kapadnis, Cheryl Black, Patrick Trapa, and Simone Sciabola. Prospective validation of machine learning algorithms for absorption, distribution, metabolism, and excretion prediction: An industrial perspective. Journal of Chemical Information and Modeling, 2023.
[78] Takaya Saito and Mare Rehmsmeier. The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets. PloS one, 10(3):00118422 2015.
[79] Kelly Rac Chi. Revolution dawning in cardiotoxicity testing. Nature reviews. Drug discovery, 12(8):565, 2013.
[80] Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et al. Analyzing learned molec- ular representations for property prediction. Journal of chemical information and modeling, 59
(8):3370-3388, 2019.
[81] Yu Shi, Shuxin Zheng, Guolin Ke, Yifei Shen, Jiacheng You. Jiyan He, Shengjie Luo, Chang Liu, Di He, and Tie- Yan Liu. Benchmarking graphormer on large-scale molecular modeling datasets. arXiv preprint arXiv:2203.00 04810, 2022. URL (ttpsttps::. http:://waxxai.aaaeee abaaaaa//2222/ 04810
[82] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Vie-Yan Liu. Do transformers really perform badly for graph representation? In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. http:lttts:///ppee let//forum?2d=DelloOorrr
[83] Joelle Pineau, Philippe Vincent- Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d' Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program). The Journal of Machine Learning Research, ((1):7449-7478, 2021.
[84] Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? large-scale study. Advances neural information processing systems, 31, 2018.
[85] Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. arXiv preprint 1707v0/707.05588 2017.
[86] Megan Stanley, John F Bronskill, Krzysztof Maziarz, Hubert Misztela, Jessica Lanini, Marwin Segler, Nadine Schneider, and Marc Brockschmidt. Fs-mol, A few-shot learning dataset of molecules. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[87] Zaixi Zhang and Qi Liu. Learning subpocket prototypes for generalizable structure-based drug design. arXiv preprint arXiv:2305.139%0 2023.
[88] Tuomo Kalliokoski, Christian Kramer, Anna Vulpetti, and Peter Gedeck. Comparability of mixed ic50 data-a statistical analysis. PloS one, 8(4)xe61007, 2013.
[89] Christian Kramer, Tuomo Kalliokoski, Peter Gedeck, and Anna Vulpetti. The experimental uncertainty of heterogenoous public k i data. Journal of medicinal chemistry, 55(1 1):5165-51733 2012.
15

---

A Lo-Hi benchmark
Lo-Hi is a practical ML drug discovery benchmark comprising two tasks: Hit Identification (Hi) and Lead Optimization (Lo). Hi corresponds to binary classification problem, wherein the goal is to identify novel hits that differ significantly from the training dataset [1O] 27 28 This is why
there are no molecules in the test set with ECFP4 Tanimoto similarity exceeding 0.4 to the training set. Models are compared using the PR AUC metric.
Lo is a ranking problem that pertains to optimizing molecules or guiding molecular generative models. The test set consists of clusters of similar molecules that are largely dissimilar from the training set, except for one molecule representing a known hit. The task involves ranking the activity of the molecules within clusters, hence we use mean intercluster Spearman correlation evaluate models. To ensure that the variation in intracluster activity stems from actual differences in activity rather than random noise, we selected clusters demonstrating high variation, as detailed in Appendix B and
The datasets each consist of three folds. We advise using the first fold for hyperparameter selection, and then applying these hyperparameters across all folds.
Datasets are released under the MIT license. Authors bear all responsibility in case of violation of rights. Datasets are small .csv files, that is why we are going to keep them in the public GitHub repository. Reviewers can find datasets in data folder.
In this section, we provide further information regarding the datasets and preprocessing steps. The size and diversity of the original datasets are displayed in Table
Table 3: Original datasets
Dataset
Size
#Circles 77 (0.5)
Active fraction
DRD2 (Ki)
8482
837
0.731
HIV
41127
19222
0.035
KDR (IC50)
8826
791
0.643
Sol
2173
1763
0,216
KCNH2 (IC50)
11159
2128
NA
A.1 Data preprocessing
We began by canonicalizing all SMILES using RDKit 2022.9.5.
For DRD2-Hi. Hi, DRD2-Lo, KDR-Hi, KDR-Lo and KCNH2-Lo utilized data from the ChEMBL30 [75] database. We collected data points that measured Ki (for DRD2) and IC50 (for KCNH2 and KDR) with confidence_acore which standard_ units were "nM". We converted aaaadarddvvaue to logarithmic scale, also known as frequently-aae asked questions/
6. We selected those for
/hchmbll/////bl..... chembl. http://chhmmb....oooo.....
hwwriiions//hhmeieeeeeeeeeeeeeee chembl data- wwwstiisststtttetooeeoe is -ww-iisssphhhhlll-
For binary DRD2-Hi and KDR-Hi we binarized the data such that log activity values greater than 6 (which 10 muM) were designated as and all others as O. We removed any ambiguous data points (e.g. with standard_ "randard_relation of "<" and activity value more than 10 muM, because those could not be binarized reliably). Following this, we selected data points with identical SMILES, discarding any with differing binarized activities.
For the continuous DRD2-Lo, KDR-Lo and KCNH2-Lo datasets, we selected data points that had standard_ batardddeeatiionnn of '=' and log activity value greater than 5 but less than 9, We selected data points with identical SMILES, discarding any with activity differences greater than 1.0. For the remaining data, we took the median of each group.
Lo--Hi benchmark: http://///ggttt....o////ee/aaaaa
16

---

DRD2-Hi: The nearest distance between train and test
140
120
100
40
20
0.20
0.25
0.30
0.35
0.40
ECFP4 Tanimoto Similarity
Figure 4: DRD2-HH Hi: Fold I
HIV-Hi: The nearest distance between train and test
800
600
400
0.10
0.15
0.20
0.25
0.30
0.35
0.40
ECFP4 Tanimoto similarity
Figure 5: HIV-Hi: Fold
17

---

Table folds
Dataset
Train
Test
Train 2
Test
Train
Test 3
DRD2-18 Hi
2385
1190
2381
1194
2384
1191
HIV-Hi
15696
7847
15695
7848
15695
7848
KDR-Hi
500
3116
500
3125
500
2285
Sol-Hi
1442
721
1442
721
1442
721
Table 5: Lo folds
Dataset
Train
Test
Train
Test
Train
Test 3
DRD2-LO
2206
267
2128
267
2257
262
KCNH2-Lo
3313
406
3313
406
3313
406
KDR-Lo
500
437
500
520
500
417
DRD2-Hi: Fold train
HIV-Hi: Fold train
DRD2 Hi: Fold test
HIV-Hi: Fold test
Figure 6: The most similar pairs of molecules between train and test.
Figure 7: Example of Lo cluster in DRD2-Lo
Figure 8: Example of Lo cluster in KCNH2-Lo
18

---

B Lo dataset is not just noise
Experimental data inherently contain noise. Consequently, selecting similar molecules may result in clusters that possess such small variation that it could be solely attributable to experimental noise, thereby invalidating the Lo task. This potential issue underlines the importance of ascertaining that the clusters exhibit significant signal to ensure the validity of the task.
As reported 88 the standard deviation for the same ligand-protein pair's pIC50 is aprC5 0.20 when measured in the same laboratory, and PpC500 R 0.68 in the ChEMBL database. In similar work [89] standard deviation for ChEMBL. pki was found to be OpKi 0.56. Therefore, based these findings, we opted to select only those clusters that displayed a standard deviation exceeding 0.70 for pIC50 and more than 0.60 for pKi. These selection criteria enhance the confidence the validity of the Lo task by prioritizing clusters with significant intracluster variation.
EPED oh
COMD thates
66
Figure 9: Within cluster variability is higher than noise standard deviation.
C Lo algorithm
The Python implementation be found in code/ py.
Algorithm 1 Get Lo Split
Input:
List of molecular SMILES S, similarity threshold i, minimum cluster size m, maximum
number of clusters M, activity values V, standard deviation threshold std;
Output: List of SMILES clusters C, list of remaining training SMILES train_S
I: procedure GETLOSPLIT(S, t, m, M, V, std,)
2:
C, train S SELECIDISTINCTCLUSEER
(S, m, M, V, std,)
3: for each cluster in C' do
4:
Move central molecule from cluster to train_ 5: end for
6: return C, ttrin_
7:
end procedure
19

---

Algorithm 2 Select Distinct Clusters
Input:
List of molecular SMILES S, similarity threshold t, minimum cluster size maximum number of clusters M, activity values V, standard deviation threshold std,
Output: List of SMILES clusters C, list of the rest training SMILES train_S
1: function SELECTDISTINCCCCUSSERRRR t, m, M, V, std,)
2: train_$ S
3: Initialize list C' as empty
4: while length of < M do
5:
Compute fingerprints F from SMILES in train_$
6:
Compute total number of neighbors N for each fingerprint in Compute ST D standard deviation of of neighbors for each fingerprint in F Set central idæ to None
7:
8:
9:
Set least_ meiis__nighhors to max(N) for each idæ in 0.. train do
Find the smallest cluster that meets criteria if [N(ixx) m and STD(idx) > std, and N(irr] < least__tte ceiest_eeihhorss then central idx idx trast_neighbors <- M(ida]
11:
13:
end if end for
15:
16:
central_ien idd is None then break
b Exit if there are no more clusters that meet criteria
17:
18:
end
19:
Add central idæ molecule and its neighbors to list of clusters C Remove the cluster and its neighbors from train_s end while
20:
21:
22: return C, train_5
23:
end function
D Additional benchmarks analysis
Distribution of Tanimoto Similarity between the nearest molecules between train and test.
ESOL: The nearest distance between train and test
8
7-
4-
3-
2-
1
ECFP4 Tanimoto Similarity
Figure 10: ESOL
20

---

HIV: The nearest distance between train and test
300
250
200
150
100
0.2
0.4
0.6
0.8
1.0
ECFP4 Tanimoto Similarity
Figure 11: HIV
CYP2D6 Veith: The nearest distance between train and test
60
40
30
20
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
ECFP4 Tanimoto similarity
Figure 12: TDC
21

---

We additionally analyzed other ligand-based MoleculeNet datasets.
Table 6: Fraction test molecules in various MoleculeNet datasets with a Tanimoto similarity >0.4 to the train set using ECFP4 fingerprints.
Dataset QM7
Fraction of Test Molecules Similar to Train Set
0.93
QM8
0.98
QM9
0.99
FreeSolv Lipophilicity PCBA
0.8
0.67
>0.93
MUV
0.96
BACE
0.77
Tox21
0.52
SIDER
0.48
E
Graph coarsening algorithm
The Python implementation can be found in httee/min_verexx__cttt We are planning to release it a pip package.
Algorithm 3 Calculate Neighbors
Input: Graph G (V, F), similarity threshold
Output: List of tuples n__ neiighobbrrrs
function AAL.CU.ATTNNFIGHHRRRSC.CC 0)
2: Initialize list neighbors empty
3: for each node e in do
4: Initialize total_ notal_neighsorrs as O
5: for each edge e incident on node do if el ee'similarity'] > 9 then
6:
total neighbors +- total_ nettl_neighhors + end if 9: end for
7:
8:
10: Append (total_neight index of v) to o-eeiihbors
11:
end for
return n_neighhorrs
13:
end function
22

---

Algorithm Cluster Nodes
Input: Sorted list n neighbors, Graph G = (V, E), similarity threshold
Output: Cluster assignment node to cluster, number of clusters total clusters
1: function CLUSTERNODES(R..
G. 0)
2: Initialize array mode_to_clune
of size [V
3: Initialize total_ _eeds__uterssss as
4:
for each tuple (count, node) in n neighors do if sende_tooclustte [node] 1 then mode_to gonde_to_clusse tlo_olusterreeeeeeeeeeeeeeeeeee_ [node] t- for each edge incident node node do if e( similaaiity adjacent node t c(1]
5:
6:
Assign new cluster
7:
8:
0 then
9:
10:
if node to wwwsstrrraajaceeet node] = -1 then
11:
node_to www.ttto_ssseeeeeeeeeeeeeee hww.:uuutterrayarnnn.oooooo mode total _ustees
12:
end if end if end for total rofal_clusters
13:
14:
16:
end
end for
18:
return node to cluster, _ctsserss
end function
Algorithm 5 Build Coarse Graph
Input:
Cluster assignment node cluster, number of clusters colal_clusters Graph G (V, E) Output: Coarsened Graph Geoarse
function BULDDCAASSEGRAHHOOd to cluster, total clusters, G) 2: Compute clusters_st size, count of each unique element in node to to_ccuster
3: Initialize Geuare an empty graph
4: for cluster in () to total coil__csutrrssse -1 do
b Add nodes
5: Add node cluster with weight clusters ciarrsssiicccusssrr Geourse
6: end for
7:
lotta_ccuutteee 8: Initialize oonnecled_cluster as empty set
for cluster to total clusters do
D Add edges
9: Get nodes of cluster as [hiisclusser_jiddiess iindices where mode_lo_cluster equals cluster
10:
for each node in this cluster clusterinddices indices do
11:
for each edge incident on node node do
12:
Add node, to cluster[e[[]] to connected_clustee
13:
end for end for for each connected cluster in connected clusters do _cccsteedclusterrrreeeeeeee
14:
15:
_lauster Add edge from cluster to connected cluster in Gooarse
16:
17:
end for end for
18:
19:
return Genare
end function
Algorithm 6 Main Procedure
Input: Graph G (V, E), similarity threshold
Output: Coarsened graph Geoars
1: procedure COARSEGRAPHIC, 0)
2: n_neighbors &- CAL CAL.CULATEENEHHHBBORSCC.
3: Sort t__neighbors descending order of first element of each tuple
4: node to cluster, mrde__t__cuuter, total clusters + GLUSTERNODDES("_neieeeee
G, 5: Geoarse BULDCORRSTGR.PHHOOd to tessee, total dafal_clusters G)
6: return Geoarse
7: end procedure
23

---

Hi-split predicts virtual screening hit rate better than scaffold split
For effective virtual screening, predicting experimental outcomes prior experimentation is paramount. In this study, we compare the predictive performance of the novel Hi-split approach with the traditional scaffold split method under a Hit Identification scenario. Following existing literature [10] 11 16
27 28 we simulate testing on novel molecules with an ECFP4 Tanimoto similarity of I 0.4 to the training set. The dataset is partitioned using both splitting methods to form separate training and validation sets for hyperparameter selection. Hyperparameter search is performed for gradient boosting on ECFP4 fingerprints, identified as the most efficient model that facilitates quick training.
After selecting the optimal hyperparameters, performance metries are computed on the validation set. Subsequently, the model is retrained on the combined training and validation sets, and performance metrics for the hold-out test set are calculated to simulate the application trained model in virtual screening. The results are summarized in Table
Table 7: Hi-split scaffold split
Dataset
Validation
Test
DRD2-Hi (Hi split)
0.603
0.677
DRD2-Hi (Scaffold split)
0.872
0.663
HIV-Hi (Hi split)
0.069
0.084
HIV-Hi (Scaffold split)
0.189
0.078
The Hi-split method demonstrates superior predictive performance for virtual screening hit rate
compared the scaffold split method, which over-optimistic in the Hit Identification scenario. also improved the test evaluation metric, although the difference is not substantial. The improved performance of the Hi-split may be attributed to the selection of more regularized models.
G Novelty consensus analysis
We have reproduced the work presented in 43 using binary ECFP4 fingerprints, as calculated by RDKit version 2022.9.5. The results can be found in Figure [3 For this particular work, we selected 0.40 as the novelty threshold.
data
Experts think molecules are too similar
0.8
0.6
0.0
ECFP4 Tanimoto similarity
Figure 13: Sigmoid fit to data

---

H Hyperparameter optimization
We used random or grid search to optimize hyperparameters for all models except for the Graphormer, which was too slow for meticulous hyperparameter search. Here we provide optimization parameters and additional commentary on the training.
We utilized a single NVIDIA RTX 2070 SUPER with CUDA 11.7 and calculated binary 1024 ECFP4 and MACCS fingerprints using RDKit 2022.9.5.
H.1 Dummy baseline
Always predicts the same constant value.
H.2 KNN
We used scipy. spatial .ww.tandssteeee.eee. jaccard as the distance metric, as it outperformed the stan- dard Euclidian distance our use case. We used grid search with all combinations of parameters. For ECFP4 it was:
params = {
'n neighigbo''': [3, 5, 7, 10], weights': ['uniform', 'distance'] ']
and for MACCS:
params = {
"n_neigbbor''; [3, 5, 7, 10, 12, 15], weights' ['umiform', 'distance']]
}
H.3 Gradient Boosting
We used 30 iterations of random search with these parameters:
params
"p_esstmmate :n [10, 50, 100, 150, 200, 250, 500], learning_rate'; [0.01, 0.1, 0.3, 0.5, 0.7, 1.0), subsample' [0.4, 0.7, 0.9, 1.0),
imin_samplee__plit> [2, 3, 5, 7],
"min_samples_leaf leaf'; [1, 3, 5], 'max_depth': [2, 3, 4],
imax_feturess: [None, 'sqrt']
}
H.4 SVM
We used grid search with these parameters:
params
C': [0.1, 0.5, 1.0, 2.0, 5.0],
}
H.S MLP
We implemented beed-forward neural network using Pytorch 2.0.0+cul 17 and Pytorch Lightning 2.0.2. consisted of several feed-forward layers with optional dropout layers. We used early stopping to prevent overfitting with patience 20 for the Hi tasks, and 10 for the Lo tasks. We used learning rate 0.01. We used batch size 32. We conducted 30 iterations of random search. For ECFP4 we used these parameters:
25

---

param dict {
'layers': [
[1024, 32, 32],
[1024, 16, 16],
[1024, 32],
[1024, 8, 4],
[1024, 4]
},
' 'dropout': [0.0, 0.0, 0.2, 0.4, 0.6], '12': [0.0, 0.0, 0.001, 0.005, 0.01],
}
For MACCS we used these parameters:
dict {
'layers': [
[167, 32, 32]
[167, 16, 16],
[167, 32].
[167, 8, 4],
[167, 4]
],
'dropout'; : 10.0, 0.0, 0.2, 0.4, 0.6]. '12': [0.0, 0.0, 0.001, 0.005, 0.01],
}
After the selection of the best hyperparameters, we selected a fixed number of the training epochs using early stopping. We used the same number of epochs for all the folds.
H.6 Chemprop
We used Chemprop 1 1.5.2 with rdkit features. We found the evaluation metrics to be little better with them, but was SOTA for Hi even without them:
- ---features .eeeettt rdkit_2d Bokkit_dd_nrraaiee normalized', ---no_ aetoo_ffaturee_ssalaaee
We used 20 iterations of random search with these parameters:
firam_dicttti param dict {
depth' ['3', '4', '5', '6'],
> dropout'; ['0.0', 10.2', 20.3', '0.5', '0.7"),
-- wwwhhhddd_siizee ['600', '1200', '2400', *3600'],
---ffn --fff_num_laaaaeeaeeaa lan.ummm_lyss: [1', 2 '3',
--hidddnnsizee" ['600', 1220'', 22400', 33600']
}
We selected the number of epochs using only the first fold. After the hyperparameters were selected, we trained the model and did not expose to the test data. The full command for training Chemprop for HIV-Hi dataset:
chemprop_train --data_path dataa/hihhivtrrain..co - --dataset_type classification \
--save_dis checkpoints/hii/hi/
--config-path configs/hiv
-separatee_vl__ptth data/hii/hiv/brain...cow
-ssparate_.ess__athh data/hi/hiv/trainn.too
--metric 'prc-auc'
-eepochs 40
--features geaiuureeggeerttorrrroooooooooor rdkit_22 2d nnnnoormlizee
Sakit_2d_normalize
-no loo_ffature_scccling
26

---

For the DRD-Hi the best hyperparameters were:
{
"depth": 6,
"dropout": 0.0,
2400, "ffn_num Sacn_num__aae aum__ayerrs 1, "hidden_size": 2400 }
For the HIV-Hi the best hyperparameters
{
"depth": 6,
"'ropoutt: 0.2,
3600, "ffn_num wat_nnum_ayerr"" 2, "hidden_sizer 3600
}
H.7 Graphormer
We used Graphormer with the last commit plied two fixes, that we found in issues 77717776666606600002222230000000 ap- [58##issuecomment- -
and https://github. com///icrreeoot///rrploomee http:///gihuuu..ccaaa/tooooooo/s issues #essuess/100tisssuocommeettttt 1207122200000 that solved our problems. However, we set up an in-house Graphormer some time ago and currently, it cannot be reinstalled from scratch due to multiple broken dependencies.
We modified code calculate and track PR AUC metrics, add datasets, and evaluate trained models. We manually optimized the hyperparameters over approximately 10 iterations. We found Graphormer to be inferior Chemprop, which is consistent with our previous experience with different datasets.
We faced numerous technical difficulties in executing and modifying Graphormer [8 82 due to improper dependency pinning by the authors. We found the training to be slow, which limited our ability optimize hyperparameters. Because of technical difficulties, we decided not test it for the Lo task.
H.8 HIV-Hi balance
HIV-Hi is a highly unbalanced binary classification problem with only 3% of positive examples. Due to this imbalance, we experimented with weighted options of classical ML algorithms manually resampled positive examples for neural networks.
27

---

I Spearman distribution
The test set of the Lo datasets composed of molecular clusters. To evaluate the models, the Spearman correlation coefficient is calculated within each cluster, comparing the actual activity values the predicted ones. The final Lo metric is the average of the Spearman coefficients across all clusters.
In the following, present histogram of Spearman coefficients for the best models across various datasets. Note that the KDR-Lo dataset is more challenging than both DRD2-Lo and KCNH2-Lo.
KCNH2-Lo: Test clusters
4.0
3.5
3.0
2.0
1.5
0.5
0.0
0.0
0.2
0.4
0.6
0.8
Spearman coefficient
Figure 14: KCNH2-Lo Spearman coefficient distribution for SVM-ECFP4
DRD2-Lo: Test clusters
0.75
-0.50
-025
0.00
0.25
0.50
0.75
1.00
Spearman coefficient
Figure DRD2-Lo Spearman coefficient distribution for SVM-BCFP4
28

---

KDR-Lo (SVM): Test clusters
4.0
3.0
2.0
1.5
0.5
0.0
-0.75
-0.50
-025
0.00
0.25
0.50
0.75
1.00
Spearman coefficient
Figure 16: KDR-Lo Spearman coefficient distribution for SVM-ECFP4
KDR-Lo (Chemprop): Test clusters
-075
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Spearman coefficient
Figure 17: KDR-LA Spearman coefficient distribution for Chemprop
29


Reviews:
[{"review_id": "whOivM9RGH", "reviewer": "Reviewer_jiMQ", "summary": "", "questions": "", "limitations": "See above", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "## Emphasis on testing models under representative conditions\nThe paper underscores the significance of assessing machine learning models within contexts analogous to their projected deployment. I find this approach compelling and believe it could lead to beneficial subsequent studies.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "nLvUYHyRnC", "reviewer": "Reviewer_H78M", "summary": "", "questions": "", "limitations": "1. The authors' focus on two target proteins, DRD2 and HIV, while important, may not fully reflect the complexity and diversity of practical drug discovery scenarios. Despite the authors' commendable effort in designing the benchmark tasks, the limited range of target proteins may not capture the real-world challenges and intricacies of applying machine learning in drug discovery. Expanding the scope to include a broader array of targets would likely provide a more comprehensive and realistic benchmarking environment.\n2. The impact of outliers, especially in cases where the molecule seems entirely implausible but the model confidently predicts a positive outcome, is not explored in the manuscript. This oversight may limit the benchmark's utility in assessing model robustness and reliability, as handling such outliers is a common challenge in practical applications. The authors could consider addressing this as a separate task within their benchmark to enhance its real-world relevance and applicability.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper is relevant and timely, considering the increasing importance of machine learning in drug discovery.\n2. It introduces a novel benchmark that is grounded in real-world drug discovery processes.\n3. The development of the molecular splitting algorithm is a noteworthy contribution.\n4. The comprehensive testing of ML models under practical settings provides valuable insights.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "VTBT5dxNGi", "reviewer": "Reviewer_ZG59", "summary": "", "questions": "", "limitations": "The authors very nicely describe the limitations of their work. I think the discussion in section 9 and elsewhere is more than adequate.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Very good motivation for the benchmark: the authors clearly understand how molecular property prediction models will be deployed and design the benchmark accordingly. The design makes up for deficiencies in other benchmarks (chiefly data splitting). It is refreshing to see a paper propose a benchmark around real-world drug discovery practices, instead of taking a problem setting from computer vision / NLP and trying to make a version with molecules!\n- Very clear writing throughout. The paper was a pleasure to read.\n- Case studies on other datasets in section 5 were very informative. I think this analysis like this should be more common in D&B submissions. A lot of D&B submissions are very quick to dismiss other datasets based on the _intention_ of the creators rather than the contents of the dataset (e.g. \"MoleculeNet was not designed with this kind of splitting in mind, so our dataset must be important.\"). I appreciate that you actually showed this.\n- Experiments in section 7 were well-done and informative. I especially appreciate the extensive hyperparameter tuning done by the authors (many submission just use the default values, making the results meaningless in my opinion).", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "KTmmPLtj7B", "reviewer": "Reviewer_igzy", "summary": "", "questions": "", "limitations": "The mentioned limitations almost exclusively focus on ML issues such as hyperparmater tuning and various architectural difficulties, which is certainly helpful. This is a dataset submission though, so comments on data limitations would also be very helpful. Currently the authors have written:\n\n\"We selected datasets that represent realistic drug discovery problems, contain substantial amounts of qualitative data, and cover a diverse chemical space...\" \n\nwhich is great, but no dataset is absolutely perfect. Could the authors discuss any important limitations of the dataset for future work to be aware of or consider improving?", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "As above the paper tackles a very important topic for multiple research communities. The introductory logic is compelling, clear and well-written for the difference between lead optimization and hit identification, how the drug-discovery process works and interacts (or fails to interact) with frontier machine learning and so on. The supplementary material is extensive and shows that the author have made considerable efforts to provide clear evidence for claims in the paper, e.g. with substantial analysis into hyperparameters and different architectures, etc..", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The authors present two benchmarks for drug discovery modeled after two separate tasks in real-world drug discovery. To create the datasets, they propose a new variant of a graph cut algorithm. The authors give a critique of existing benchmarks and give practical results for ML models on their benchmark.

The reviewers at first glance had a high variance (3, 4, 7, 9). Two reviewers were very positive on the work. Both igzy and ZG59 mentioned the strong motivation for the benchmark, the clear writing, and informative case studies and experiments. igzy said “potentially very high impact and useful for the ML-biomedical boundary community, as it points out both major flaws and clear roadmaps for solutions in the important area of drug discovery datasets in a clearly written way, demonstrating substantial ML and domain knowledge.”

The two negative reviewers were less active in the discussion. Reviewer jiMQ made factual errors in their review and did not reply to the authors’ response. One of reviewer H78M’s main concerns was the lack of a feasibility/synthesizability task, however two other reviewers noted that this request is infeasible and beyond the scope of the paper.

After reading through the discussions, I agree with reviewers igzy and ZG59 about the high quality of this work, and the high potential for impact. I recommend acceptance.
Decision: Accept (Poster)