SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Simon Steshin
simon.steshin@gmail.com
Finding new drugs is getting harder and harder. One of the hopes of drug discovery is to use machine learning models to predict molecular properties. That is why models for molecular property prediction are being developed and tested on benchmarks such as MoleculeNet. However, existing benchmarks are unrealistic and are too different from applying the models in practice. We have created a new practical  $Lo-Hi$  benchmark consisting of two tasks: Lead Optimization (Lo) and Hit Identification (Hi), corresponding to the real drug discovery process. For the Hi task, we designed a novel molecular splitting algorithm that solves the Balanced Vertex Minimum  $k$ -Cut problem. We tested state-of-the-art and classic ML models, revealing which works better under practical settings. We analyzed modern benchmarks and showed that they are unrealistic and overoptimistic.
Review: https://openreview.net/forum?id=H2Yb28qGLV
Lo-Hi benchmark: https://github.com/SteshinSS/lohi_neurips2023
Lo-Hi splitter library: https://github.com/SteshinSS/lohi_splitter
Drug discovery is the process of identifying molecules with therapeutic properties [1]. To serve as a drug, a molecule must possess multiple properties simultaneously [2]. It must be stable [3], yet easily eliminated from the body [4], able to reach its target [5, 6, 7], non-toxic [8], cause minimal side effects [9], and therapeutically active on the target [10, 11].
To identify such molecules, researchers develop Molecular Property Prediction (MPP) models. These models are used in a virtual screening, during which the models are used to make predictions for a large number of molecules, after which the molecules with the best estimated property are selected for experimental validation [12, 13, 14, 15, 16].
To enable comparisons between various architectures, the research community relies on standardized benchmarks to evaluate model performance [17, 18, 19, 20, 21, 22, 23]. The prevailing assumption is that models with superior metrics on these benchmarks are more suitable for real-world applications.
We believe this assumption is false. Modern benchmarks test in non-realistic conditions on impractical tasks. In this paper, we introduce two new practical drug discovery ML tasks — Hit Identification and Lead Optimization — that are often encountered in most drug discovery campaigns. We demonstrate that none of the benchmarks assess models for these tasks, which is why we propose seven new practical datasets that better imitate real-life drug discovery scenarios. To prepare Hi datasets, we designed a novel molecular splitter algorithm that solves Balanced Vertex Minimum  $k$ -Cut problem using Integer Linear Programming with heuristics.
Figure 1: Hit Identification (Hi) task
Figure 2: Lead Optimization (Lo) task
2 Practical drug discovery
2.1 Hit Identification (Hi)
Drug discovery involves multiple stages. When the potential mechanism of action is known, an early step is Hit Identification. During this phase, chemists search for a "hit" - a molecule with the potential to become a drug [24]. A viable hit must exhibit some level of activity towards the target (e.g., Ki less than  $10\mu \mathrm{M}$ ) and possess novelty, meaning it is eligible for patent protection.
Clinical trials can incur costs in the hundreds of millions of dollars [25, 26], making it risky to pursue non-patentable molecules. Consequently, companies prioritize patentable molecules from the outset. Novelty is an essential aspect of this process. In practice, medicinal chemists often pre-filter their chemical libraries before the virtual screening, eliminating molecules that exhibit a Tanimoto similarity above a specific threshold to molecules with known activity [10, 11, 16, 27, 28, 29].
2.2 Lead Optimization (Lo)
Following Hit Identification, the next stage is Lead Optimization. Once a novel, patentable molecule with activity towards the desired target is identified, its closest analogs are also likely to exhibit activity [30]. In this phase, medicinal chemists often make minor modifications to the hit molecule to enhance its target activity, selectivity, and other properties [31, 32]. Unlike in Hit Identification, novelty usually is not a priority during Lead Optimization. Instead, the objective is to discover molecules that are similar to the original hit but possess improved characteristics.
This is related to the field of goal-directed molecular generation [33], which is occasionally formulated as the problem of searching for molecules with maximal activity within the  $\varepsilon$ -neighborhood of a known hit [34, 35], within some scaffold [36], or as an optimization process in the latent space [37]. These works use predictive ML models to distinguish between minor chemical modifications [38]. The effectiveness of these models in their ability to distinguish between molecules with small modifications remains unclear.
2.3 Model Selection
In this context, Lead Optimization (Lo) and Hit Identification (Hi) represent contrasting tasks. Hi ML models are expected to demonstrate their generalization capabilities by predicting properties of molecules significantly different from the training set. Conversely, Lo ML models should predict properties of minor modifications of molecules with known activity.
For selecting the appropriate ML model or adjusting its hyperparameters, it is crucial to test the model under conditions similar to its intended application. In Hi, models must identify novel molecules markedly different from known active ones, implying the models should be tested on molecules distinctly different from the training set. Conversely, Lo applies ML models to molecules resembling known active ones. The null Structure-Activity Relationship hypothesis assumes that
small modifications will not alter the molecule's properties [30, 39]. Consequently, Lo ML models should be evaluated based on their ability to make predictions that surpass the null hypothesis.
We show that modern benchmarks mix these two scenarios, which is why it remains unclear how effectively models can generalize to truly novel molecules and how proficiently they can distinguish minor modifications, thereby guiding molecular optimization. Furthermore, we untangle these scenarios using a novel benchmark, demonstrating that different architectures are better suited for different tasks.
3 Novelty
But how to measure novelty? From a commercialization and regulatory perspective, a novel molecule is one that can be patented. Contemporary patents consist of human-written text and incorporate non-trivial substitutions, rendering the assessment of a molecule's patentability a complex task for legal professionals. To date, this process has not been automated. Additionally, during the Hit Identification, not only must the hit be patentable — its neighborhood should be patentable as well to make Lead Optimization possible. When dealing with an extensive chemical library, these challenges make patentability an impractical criterion for determining novelty.
While chemists agree that minor substitutions likely do not alter a molecule's function, a consensus on the distinction between a new molecule and a modified version of an existing one remains elusive. One might hope to find a general similarity threshold that would separate similar molecules with presumably the same activity from distinct molecules. This hope led to the "0.85 myth" [40], which proved to be false due to the significant variability of such thresholds across different targets [41]. About molecular similarity, it was said [42, 41], "Similarity is in the eye of the beholder." Nevertheless, a practical criterion exists.
In study [43] 143 experts from regulatory authorities(FDA, FDA of Taiwan, EMA, PMDA) were shown 100 molecular pairs and asked if the molecules should be regarded as structurally similar. This study is especially notable because it simulates a real-life scenario in which a regulatory committee (EMA's Committee for Medicinal Products for Human Use) decides if a new drug is novel enough to grant it a beneficial status: orphan drug designation. The study shows that when two molecules have Tanimoto similarity  $\approx 0.4$  with ECFP4 fingerprints [44], half of the experts regard them as dissimilar, which is sufficient to conclude the novelty of the drug. See Appendix G for our reproduction.
While Tanimoto similarity with ECFP4 fingerprints has its own disadvantages such as size bias [45], its simplicity makes it possible to quickly measure the similarity of molecules, so there are practical [16] and theoretical [46, 47] works that use "Tanimoto similarity  $< 0.4$  " as a novelty criterion. Because of the practical evaluation in real-life scenarios and efficiency, we assess novelty using Tanimoto similarity with ECFP4 fingerprints in our benchmark.
4 Our contribution
- We suggest two new practical drug discovery ML tasks — Hit Identification and Lead Optimization — that better imitate real-life scenarios;
- We designed a novel molecular splitting algorithm for the Hi task;
- We propose seven new practical datasets;
- We demonstrate that modern ML drug discovery benchmarks simulate impractical scenarios;
- We evaluate modern and classic ML algorithms on our benchmark.
5 Modern benchmarks test neither Lo nor Hi
To select a model or its hyperparameters, we must evaluate them under the conditions in which they will be used. We demonstrate that none of the modern benchmarks correspond to realistic conditions, thus raising questions about their suitability for evaluating practical machine learning models. Although it is impossible to examine every existing benchmark due to their sheer number, we have opted to assess a variety of benchmarks using different data, different preprocessing techniques, and originating from different authors. We will initially focus on standard benchmarks, while in
ESOL train
HIV train
ESOL test
TDC.CYP2D6_Veith train
HIV test
TDC.CYP2D6_Veith test
Figure 3: Common benchmarks contain highly similar molecules across different splits.
Moldata train
Moldata test
the section 6, we will explore more exotic and specialized ones. We provide additional analysis in Appendix D.
5.1 MoleculeNet: ESOL
MoleculeNet [17] is a widely-used benchmark for ML drug discovery, consisting of 17 datasets, including the ESOL dataset with water solubility data. This dataset is frequently used in studies for model comparisons [48, 49, 50, 51, 52, 53]. While the authors recommend a random train-test split, we discovered that it leads to  $76\%$  of the test molecules having a neighbor in the train with Tanimoto similarity  $>0.4$ , making ESOL unsuitable for Hi scenario (see Fig. 3). It is also unclear how well the dataset represents the Lo scenario, as the distinct  $24\%$  of the train contributes to the evaluation, and the RMSE metric can be significantly improved over the constant baseline, even without identifying minor chemical modifications.
5.2 MoleculeNet: HIV
In benchmarks, train and test sets are typically split using random, scaffold, or occasionally time splits when time stamps are available. It has been frequently observed that random splits can result in highly similar molecules in both train and test sets, leading to overly optimistic and impractical estimations. For example: "Random splitting, common in machine learning, is often not correct for chemical data" [17]. Thus, scaffold splitting [54] is sometimes suggested as an alternative.
Scaffold splitting [54] is a method where each molecule is represented by a graph consisting of ring systems, linkers and side chains. A group of molecules may correspond to a single graph, in which case they are assigned to the same partition.
The MoleculeNet HIV dataset (ubiquitous in evaluation [55, 56, 57, 58, 59]) recommends using scaffold splitting, as it is believed to better reflect the process of discovering new molecules: "As we are more interested in discovering new categories of HIV inhibitors, scaffold splitting [...] is recommended" [17]. Although scaffold splitting makes the train set more distinct from the test set, it is still insufficient for the scenario of finding novel molecules. We discovered that  $56\%$  of HIV test molecules have a train neighbor with a Tanimoto similarity  $>0.4$ , indicating the dataset is unsuitable for the Hi scenario (see Fig. 3). Additionally, the dataset is unsuitable for the Lo scenario due to its binary label, rather than a continuous value.
5.3 Therapeutic Data Commons: TDC.CYP2D6_Veith
The Therapeutic Data Commons [18] is a novel platform designed for evaluating machine learning models in drug discovery. We examined the largest dataset, TDC.CYP2D6_Veith, within the Single-Instance Learning Tasks category. Although a scaffold split is recommended, the dataset is deemed unsuitable for the Hi scenario, as  $78\%$  of test molecules possess a training neighbor exhibiting a Tanimoto similarity greater than 0.4 (see Fig. 3). Additionally, the dataset is unsuitable for the Lo scenario due to its binary label, rather than a continuous value.
5.4 MolData
MolData [19] is a recent benchmark for MPP based on PubChem data. The authors themselves analyzed the novelty of the molecules within the subset and found their scaffold split to be unrealistic: “[...] more than  $44\%$  of the molecules within the MolData dataset have at least one other similar molecule to them with a Tanimoto Coefficient of 0.7 or higher. This high percentage of the similarity can denote lack of diversity within this portion of the dataset.”
The authors use scaffold split to increase the difference between the train and the test, but we found that at least  $88\%$  of the molecules in the test have a similar molecule in the train with a Tanimoto similarity  $>0.4$ , making the dataset unsuitable for the Hi scenario (see Fig. 3). Additionally, the dataset is unsuitable for the Lo scenario due to its binary label, rather than a continuous value.
6.1 Out-of-distribution MPP
In the Hi scenario, the training and test sets are significantly different to simulate the search for new molecules, making the Hi scenario interpretable as an Out-Of-Distribution (OOD) task. Although OOD benchmarks already exist for machine learning-based drug discovery, they do not align with practical drug discovery applications.
Drug00D [20] is a drug discovery benchmark dedicated to Out-Of-Distribution prediction. The authors accurately observe that "In the field of AI-aided drug discovery, the problem of distribution shift [...] is ubiquitous", and therefore suggest assay-based, scaffold-based, or molecular size-based partitions to create a distribution shift. However, their benchmark does not correspond to practical drug discovery, as it involves predicting average activity for all ChEMBL targets simultaneously in ligand-based drug discovery datasets. This approach is impractical because, in reality, researchers are not interested in average activity across ChEMBL targets, but rather in activity on a specific target.
GOOD [21] is a benchmark explicitly designed to separate covariate and concept shifts. We examined the GOOD-HIV dataset, as it is a MPP benchmark containing a significant amount of diverse data (see the #Circles in Appendix A). In all Out-Of-Distribution test partitions, more than  $40\%$  of molecules have a neighbor in the training set at a Tanimoto similarity more than 0.4, making the dataset unsuitable for the Hi scenario.
During the NeurIPS review period, Valence Labs and Laval University jointly published work — independent from ours — investigating Molecular-Out-Of-Distribution generalization [60]. Their research is akin to our Hi-scenario, but the authors studied different data splits, employed different models, and investigated uncertainty calibration.
6.2 Activity cliff
Activity cliff is a pair of structurally similar compounds that are active against the same bio-target but significantly different in binding potency [61, 22].
Recently, several remarkable studies have emerged, seeking to establish a standard benchmark for Activity Cliff Prediction [22, 23]. These works have inspired us to create our own benchmark. While Activity Cliff Prediction may be advantageous in the Lo scenario, it is inadequate for guiding molecule optimization. In the Lo scenario, it is essential not only to identify extreme activity cliffs with a  $10\mathrm{x}$  difference in activity but also to predict minor activity fluctuations. We propose alternative splits and metrics in the Lo scenario.
7.1 Hi-splitter
To simulate the Hi scenario, it is necessary to divide the dataset into training and testing subsets such that any pair of molecules from different partitions has a ECFP4 Tanimoto similarity of less than 0.4. Despite the fact that the scaffold splitter produces a more diverse division than random splitter, we observe that it is inadequate for an effective Hi scenario.
A greedy approach is to implement the conventional scaffold split and discard from the test set any molecules excessively similar to those in the training set. The issue with this method, however, is that data points are costly, and it is desirable to minimize the number of discarded molecules. Consequently, we have developed a novel algorithm for strict dataset splitting, which discards fewer molecules than the greedy algorithm.
Let us consider molecules  $X = \{x_{1},\dots,x_{n}\}$ . We construct a neighborhood graph  $G = (V,E)$ , where each molecule  $x_{i}$  corresponds to a vertex  $v_{i} \in V$ . Two vertices are connected by an edge if and only if the associated molecules have a similarity to each other greater than threshold  $t$ :  $e_{vu} \in E \Leftrightarrow T(x_v,x_u) > t$ , where  $T$  is Tanimoto Similarity with ECFP4 fingerprints. In such a graph, connected components can be assigned to the training or testing sets independently. However, in practice, 95% of the molecules belong to a single connectivity component. Our goal is to remove the minimum number of vertices such that the giant component breaks up into multiple components with size constraints, thereby enabling us to distribute them between the training and testing sets.
This problem is known as the Balanced Vertex Minimum  $k$ -Cut and has been extensively researched in literature [62, 63, 64]. A review of similar problems can be found elsewhere [65, 66]. Similar to [62] we formulate the Integer Linear Programming formulation.
Let  $K$  denote the set of integers  $\{1, 2, \ldots, k\}$ . Let  $G = (V, E)$  represent a simple connected graph that we are going to split:
$$
V = \cup_ {i = 1} ^ {k} V _ {i} \cup V _ {0} \qquad \forall i \neq j \quad V _ {i} \cap V _ {j} = \emptyset
$$
For all vertices  $v \in V$  and for all integers  $i \in K$ , let us associate a binary indicator  $y_v^i$  such that:
$$
y _ {v} ^ {i} = \left\{ \begin{array}{l l} 1, & \text {i f} v \in V _ {i} \\ 0, & \text {o t h e r w i s e} \end{array} \right.
$$
Note that  $V_{0}$  denotes the set of removed vertices, so if  $\sum_{i\in K}y_v^i = 0$  then the vertex  $v$  is in  $V_{0}$ .
Let  $b_{i} \in \mathbb{N}$  be a lower bound on the cardinality  $|V_{i}|$  of partition  $V_{i}$ . We need it to get partitions with size constraints, e.g. 80% in train and 20% in test. Let  $w_{v}$  be the weights of the nodes. In simple formulation we take  $w_{v} = 1$ . We formulate the Balanced Vertex Minimum  $k$ -Cut as follows:
$$
\max  \sum_ {i \in K} \sum_ {v \in V} w _ {v} y _ {v} ^ {i} \tag {1}
$$
$$
\sum_ {i \in K} y _ {v} ^ {i} \leq 1 \quad \forall v \in V \tag {2}
$$
$$
y _ {u} ^ {i} + y _ {v} ^ {j} \leq 1 \quad \forall i \neq j \in K, \forall e _ {u v} \in E \tag {3}
$$
$$
\sum_ {v \in V} y _ {v} ^ {i} \geq b _ {i} \quad \forall i \in K \tag {4}
$$
$$
y _ {v} ^ {i} \in \{0, 1 \} \quad \forall i \in K, \forall v \in V \tag {5}
$$
Equation (1) minimizes the weight of removed molecules  $V_{0}$ . Equation (2) states that each vertex  $v$  should be in one partition maximum. Equation (3) ensures there is no connectivity between different partitions, so for each edge  $e_{uv} \in E$  if  $u$  is in  $V_{i}$  partition, then the other vertex  $v$  is either in  $V_{i}$  as well, or in  $V_{0}$ , meaning it was removed. Equation (4) puts constraints on the size of the partitions  $V_{i}$ .
While this formulation was effective on small-scale graphs (around 100 vertices), we found it too slow for a real DRD2 activity dataset with 6k molecules. To our knowledge, existing literature [62, 65, 67, 68] typically involves small graphs from the standard benchmarks with a maximum node count of several hundred. To expedite computation, we implemented a graph coarsening approach. The basic idea is outlined here, while the formal algorithm is detailed in Appendix E.
We initially performed Butina clustering [69] on molecules and created a coarse graph wherein the vertices correspond not to individual molecules but to clusters of molecules. In the coarsened graph,
each vertex is assigned a weight  $w_{v}$  equal to the number of molecules in the respective cluster. This approach accelerated computations and enabled us to partition the HIV dataset consisting of 40k molecules, removing fewer vertices than would be the case with the greedy approach (See Table 1).
Table 1: Number of removed molecules for 0.9:0.1 split
Method	DRD2	HIV
Greedy	1066 (17.0%)	5851 (14.2%)
Hi-Splitter	97 (1.5%)	1598 (3.8%)
7.2 Lo-Hi benchmark
We conducted an analysis of numerous drug discovery benchmarks, yet none seemed to align with the actual drug discovery process. Consequently, we prepared seven datasets and are now making them available to the community.
We selected datasets that represent realistic drug discovery problems, contain substantial amounts of qualitative data, and cover a diverse chemical space. Diversity was assessed using the recently introduced #Circles metric [70]. As the source code was unavailable at the time of writing, we employed our own implementation of the greedy algorithm outlined in [70]: Appendix H. We provide additional statistics for the original datasets in the Appendix A.
We propose three distinct train-test splits for each dataset. We advise adjusting hyperparameters solely on the first split, applying the same hyperparameters to train and assess models on splits #2 and #3, and comparing models by averaging metrics across the splits. Datasets are released under the MIT license.
7.2.1 Hi
In the Hit Identification scenario, we aim to predict binary labels for new molecules that significantly differ from those in the training set. We prepared four datasets which we divided into training and testing sets, ensuring that the Tanimoto similarity between any molecule in the test set and those in the training set is less than 0.4. See Fig. 1. Additionally, we show that such a split predicts experimental outcomes better than the scaffold split (Appendix F).
DRD2-Hi involves predicting dopamine receptor inhibition, a GPCR target of therapeutic importance in schizophrenia [71, 72] and Parkinson's disease [73, 74]. To create this dataset, we obtained Ki data for DRD2 from ChEMBL30 [75], cleaned it (see Appendix A), and binarized it so that molecules with  $\mathrm{Ki} < 10\mu \mathrm{M}$  are considered active.
HIV-Hi is an HIV dataset from the Drug Therapeutics Program AIDS Antiviral Screen that measures the inhibition of HIV replication. We obtained the prepared dataset from MoleculeNet [17].
DRD2-Hi and HIV-Hi are large. However, real-life data often comes in limited quantities. To simulate this crucial scenario, we prepared a smaller challenging dataset, KDR-Hi. This dataset is based on the ChEMBL30 IC50 data associated with vascular endothelial growth factor receptor 2, a kinase target for cancer treatment [76]. Its creation process was similar to that of DRD2-Hi, but we restricted the training folds to just 500 molecules.
The Sol-Hi dataset draws from a public solubility dataset at Biogen [77]. We binarized this data such that molecules with a solubility of less than  $10~\mu \mathrm{g / mL}$  were assigned a positive label.