SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Each dataset was divided into distinct training and testing sets. We used the Hi-splitter with  $k = 3$  to obtain three highly dissimilar subsets:  $\{F_1, F_2, F_3\}$ . These subsets were then combined to create three distinct folds, each with a unique test set. For instance, for the first fold, the training set was  $train_1 = \{F_1, F_2\}$  and the test set was  $test_1 = \{F_3\}$ . This methodology enabled us to assess the variability in quality resulting from using different data with the same models.
Hi Metric For our benchmark, we have selected the PR AUC. As a simple binary classification metric without parameters, it is implemented in most libraries and normalized to a range of [0, 1].
<sup>1</sup>The DRD2-Hi preparation code can be found at notebooks/data/03_split_drd2_hi.ipynb.
The PR AUC favors early recognition models [78] and does not appeal to wrong intuition among readers in an unbalanced setting.
7.2.2 Lo
In the Lead Optimization scenario, we aim to predict the activity of molecules that are highly similar to those in the training set. As similar molecules tend to exhibit similar activity, our focus is not on predicting binary labels, but rather on ranking, which indicates whether a modification increases the activity or not.
To simulate the Lead Optimization scenario, we isolated clusters of molecules with intracluster similarity  $\geq 0.4$  and consisting of  $\geq 5$  molecules. We included them in the test dataset. In practical Lead Optimization, the activity of a given hit is already known; thus, for each cluster, we retained exactly one molecule with a similarity  $\geq 0.4$  to that cluster in the training set. See Fig. 2. We provide additional analysis in Appendix A.
In order to confirm the validity of the Lo benchmark, we ensured that the intracluster variation in activity exceeds the experimental noise (See Appendix B). This step is critical, as if the variance within a cluster of similar molecules is not significantly greater than the experimental noise, it would not make sense to test models on such molecules — under these conditions, even an ideal model would struggle to make accurate predictions. The formal pseudocode can be found in Appendix C.
We assembled three datasets: DRD2-Lo, KCNH2-Lo and KDR-Lo. Both DRD2-Lo and KDR-Lo are based on the same data as their respective Hi counterparts but feature a different split between training and testing sets. KCNH2 is an ion channel that regulates heartbeat, and its inhibition can cause dangerous side effects [79]. Consequently, bioassays for KCNH2 are used as a screening method for cardiotoxicity. We extracted IC50 data from ChEMBL30, cleaned it, and divided it into training and testing sets.
Each dataset was split three times using different random seeds, resulting in three folds. This method enables us to assess the variability in quality when using different data with the same models.2
**Lo Metric** Our goal is to determine whether the models can make better predictions than assuming "the modified molecule active in the same manner as the original hit." We chose Spearman's correlation coefficient as our metric, calculated within each cluster and averaged across clusters. This metric does not rely on intracluster variation, depends solely on the ranking of molecules within the cluster, and is normalized, between minus one (ideally wrong), zero (random) and one (ideal), rendering it easily interpretable.
7.3 Evaluation
We evaluated both traditional and state-of-the-art ML models on our benchmark. We meticulously executed the hyperparameter search, following the procedure outlined in Appendix H. Results are presented in Table 2. The best scores for fingerprint and graph models are highlighted in bold. It should be noted that the mean and standard deviation were calculated not for random seeds, but within different folds.
We found that the best models varied for the Hi and Lo tasks. The most effective model for the Hi task was the Chemprop graph neural network [80], aligning with its real-world success [16, 28, 29]. The second-best were gradient boosting and KNN for the small KDR-Hi dataset.
Conversely, for the Lo task, the SVM model was found to be the most proficient, which corroborates previous works on activity cliffs [23]. Only for the small KDR-Lo did Chemprop outperform SVM, but even then, the performance was not satisfactory. We further provide a per-cluster Spearman distribution for SVM in Appendix I. The limited inability of Chemprop to distinguish minor modifications may be due to the limited expressivity of graph neural networks. However, this was surprising to us, considering the limited expressivity of binary fingerprints as well.
2The DRD2-Lo preparation code can be found at notebooks/data/04_split_drd2_1o.ipynb.
Table 2: Lo-Hi results. PR AUC for Hi and mean Spearman for Lo.
Model	DRD2-Hi	HIV-Hi	KDR-Hi	Sol1-Hi	DRD2-Lo	KCNH2-Lo	KDR-Lo
Dummy baseline	0.677±0.061	0.04±0.014	0.609±0.081	0.215±0.008	0.000±0.0	0.000±0.0	0.000±0.0
KNN (Tanimoto distance, ECFP4)	0.706±0.047	0.067±0.029	0.646±0.048	0.426±0.022	0.195±0.053	0.164±0.014	0.130±0.034
KNN (Tanimoto distance, MACCS)	0.702±0.042	0.072±0.036	0.610±0.072	0.422±0.009	0.211±0.041	0.036±0.022	0.071±0.02
Gradient Boosting (ECFP4)	0.736±0.05	0.08±0.038	0.607±0.067	0.429±0.006	0.145±0.052	0.37±0.003	0.076±0.036
Gradient Boosting (MACCS)	0.751±0.063	0.058±0.03	0.603±0.074	0.502±0.045	0.197±0.043	0.216±0.032	0.100±0.026
SVM (ECFP4)	0.677±0.061	0.04±0.014	0.611±0.081	0.298±0.047	0.311±0.015	0.472±0.014	0.158±0.051
SVM (MACCS)	0.713±0.05	0.042±0.015	0.605±0.082	0.308±0.021	0.219±0.02	0.133±0.024	0.074±0.034
MLP (ECFP4)	0.717±0.063	0.049±0.019	0.626±0.047	0.403±0.017	0.094±0.059	0.146±0.040	0.085±0.030
MLP (MACCS)	0.696±0.048	0.052±0.018	0.613±0.077	0.462±0.048	0.026±0.083	0.174±0.031	0.065±0.027
Chemprop [80]	0.782±0.062	0.148±0.114	0.676±0.026	0.618±0.03	0.298±0.035	0.375±0.067	0.161±0.024
Graphormer [81, 82]	0.729±0.039	0.096±0.070	-	-	-	-	-
In virtual screening, practitioners aim to discover novel molecules and filter their chemical libraries using Tanimoto similarity. Despite this common practice, there are no benchmarks that simulate this particular scenario. In molecular optimization, researchers employ predictive models to guide the optimization process in a step-by-step manner. However, it remains unclear whether these models possess the capacity to distinguish minor modifications.
We identified several limitations in current drug discovery benchmarks and proposed a more realistic and practical alternative in the form of the  $Lo-Hi$  benchmark. By introducing two tasks, Lead Optimization (Lo) and Hit Identification (Hi), which closely resemble real drug discovery scenarios, we created an environment to evaluate machine learning models under more representative conditions. We emphasize the importance of testing models under conditions similar to their intended application.
Furthermore, the paper critically assesses existing benchmarks and related works, highlighting their inadequacies and the need for better evaluation methods. To address these issues, we suggest alternative datasets. To build them, we designed a novel molecular splitter algorithm for the Hi task.
Different models proved to be better suited to different tasks. Our evaluation of both classical and modern ML models revealed Chemprop as the state-of-the-art for the Hi task, and SVM with ECFP4 fingerprints as the state-of-the-art for the Lo task.
The paper's key contributions include the introduction of the  $Lo-Hi$  benchmark, a comprehensive analysis of the limitations of modern ML drug discovery benchmarks, a novel molecular splitter algorithm and the evaluation of modern and classic ML algorithms on the proposed benchmark. This work sets the stage for a more accurate and reliable evaluation of machine learning models in the field of drug discovery, ultimately leading to better decision-making and improved outcomes in the search for new therapeutic compounds.
9 Limitations
We have conducted hyperparameter tuning, although performing it thoroughly poses a challenge [83, 84, 85]. Convincing evidence supporting a particular architecture could be garnered from an open online contest with prizes, accompanied by an undisclosed test dataset. We faced numerous technical difficulties in executing and modifying Graphormer (see Appendix H.7). As such, we cannot definitively determine if Graphormer's failure is a consequence of its architecture or the result of improper dependency pinning by the authors.
Our findings indicate that the Integer Linear Programming solution proves to be significantly more effective than the greedy approach. However, we have not explored different formulations in our study. Therefore, it is possible that more efficient methods for splitting molecular datasets could exist.
We endeavored to encompass a diverse range of ligand-based drug discovery problems (GPCR inhibition, kinase inhibition, cardiotoxicity, solubility) in our benchmark. However, it is infeasible to capture every potential molecular property prediction task. We advise practitioners to use benchmark results to shortlist models, but also to test them against specific objectives.
10 Future work
Our focus was on medium and large datasets, yet many small datasets contain fewer than 100 datapoints. It would be beneficial to have smaller datasets similar to [86] but tailored for Hi generalization.
While our emphasis was on ligand-based drug discovery, where the goal is to predict a molecule's property, there is also structure-based drug discovery. This approach not only involves predicting molecular properties but also incorporates protein information. Hence, it would be advantageous to have structure-based drug discovery datasets that are divided not just by protein (or pockets [87]) similarity but also with Hi generalization across ligands.
A major ongoing challenge in molecular generative models is ensuring synthesizability, meaning that generated molecules can be made in the real world. Hi splits can help test the generalizability of synthesizability models. But, it's important to remember that Lo/Hi splits assume similar molecules have similar properties. While this holds for physico-chemical attributes, this premise remains to be validated in the context of feasibility measures.
11 Potential Harmful Consequences
One of the primary concerns arises from the ability to design molecules that are unfamiliar to medical chemists and experts in the field. While the novelty of these compounds can be advantageous for pushing the boundaries of current scientific knowledge, it also raises the potential risk of misuse, especially if malicious actors were to use the system to generate harmful or toxic compounds for hostile purposes. Given their unfamiliar nature, these molecules might not immediately raise flags upon review by experts or during synthesis orders at chemical laboratories. This could open the door for the creation and dissemination of harmful compounds.
12 Acknowledgements
This work was funded by Gleb Pobegailo.
[1] Shu-Feng Zhou and Wei-Zhu Zhong. Drug design and discovery: principles and applications, 2017.
[2] Jing Lin, Diana C Sahakian, SM De Morais, Jinghai J Xu, Robert J Polzer, and Steven M Winter. The role of absorption, distribution, metabolism, excretion and toxicity in drug discovery. Current topics in medicinal chemistry, 3(10):1125-1154, 2003.
[3] Sabina Podlewska and Rafal Kafel. Metstabon—online platform for metabolic stability predictions. International journal of molecular sciences, 19(4):1040, 2018.
[4] Mohsen Sharifi and Taravat Ghafourian. Estimation of biliary excretion of foreign compounds using properties of molecular structure. The AAPS journal, 16:65-78, 2014.
[5] Patrizia Crivori, Gabriele Cruciani, Pierre-Alain Crrupt, and Bernard Testa. Predicting blood-brain barrier permeation from three-dimensional molecular structure. Journal of medicinal chemistry, 43(11):2204-2216, 2000.
[6] Qiang Tang, Fulei Nie, Qi Zhao, and Wei Chen. A merged molecular representation deep learning method for blood-brain barrier permeability prediction. Briefings in Bioinformatics, 23(5), 2022.
[7] Raimund Mannhold and Han Van de Waterbeemd. Substructure and whole molecule approaches for calculating log p. Journal of Computer-Aided Molecular Design, 15:337-354, 2001.
[8] Ashok K Sharma, Gopal N Srivastava, Ankita Roy, and Vineet K Sharma. Toxim: a toxicity prediction tool for small molecules developed using machine learning and chemoinformatics approaches. Frontiers in pharmacology, 8:880, 2017.
A Lo-Hi benchmark
$Lo-Hi$  is a practical ML drug discovery benchmark $^3$ , comprising two tasks: Hit Identification (Hi) and Lead Optimization (Lo). Hi corresponds to a binary classification problem, wherein the goal is to identify novel hits that differ significantly from the training dataset [10, 11, 16, 27, 28]. This is why there are no molecules in the test set with ECFP4 Tanimoto similarity exceeding 0.4 to the training set. Models are compared using the PR AUC metric.
Lo is a ranking problem that pertains to optimizing molecules or guiding molecular generative models. The test set consists of clusters of similar molecules that are largely dissimilar from the training set, except for one molecule representing a known hit. The task involves ranking the activity of the molecules within clusters, hence we use mean intercluster Spearman correlation to evaluate models. To ensure that the variation in intracluster activity stems from actual differences in activity rather than random noise, we selected clusters demonstrating high variation, as detailed in Appendix B and C.
The datasets each consist of three folds. We advise using the first fold for hyperparameter selection, and then applying these hyperparameters across all folds.
Datasets are released under the MIT license. Authors bear all responsibility in case of violation of rights. Datasets are small .csv files, that is why we are going to keep them in the public GitHub repository. Reviewers can find datasets in data folder.
In this section, we provide further information regarding the datasets and preprocessing steps. The size and diversity of the original datasets are displayed in Table 3.
Table 3: Original datasets
Dataset	Size	#Circles [70] (0.5)	Active fraction
DRD2 (Ki)	8482	837	0.731
HIV	41127	19222	0.035
KDR (IC50)	8826	791	0.643
Sol	2173	1763	0.216
KCNH2 (IC50)	11159	2128	NA
A.1 Data preprocessing
We began by canonicalizing all SMILES using RDKit 2022.9.5.
For DRD2-Hi, DRD2-Lo, KDR-Hi, KDR-Lo and KCNH2-Lo we utilized data from the ChEMBL30 [75] database. We collected data points that measured Ki (for DRD2) and IC50 (for KCNH2 and KDR) with confidence_score  $\geq 6$ . We selected those for which standard.units were in "nM". We converted standard_value to logarithmic scale, also known as pChembl(https://chembl.gitbook.io/chembl-interface-documentation/frequently-asked-questions/chembl-data-questions#what-is-pchembl).
For binary DRD2-Hi and KDR-Hi we binarized the data such that log activity values greater than 6 (which is  $< 10\mathrm{muM}$ ) were designated as 1, and all others as 0. We removed any ambiguous data points (e.g. with standard Relation of "<" and an activity value more than  $10\mathrm{muM}$ , because those could not be binarized reliably). Following this, we selected data points with identical SMILES, discarding any with differing binarized activities.
For the continuous DRD2-Lo, KDR-Lo and KCNH2-Lo datasets, we selected data points that had standard Relation of  $\mathbf{\Psi}^{\prime} = \mathbf{\Psi}^{\prime}$  and a log activity value greater than 5 but less than 9. We selected data points with identical SMILES, discarding any with activity differences greater than 1.0. For the remaining data, we took the median of each group.
<sup>3</sup>Lo-Hi benchmark: https://github.com/SteshinSS/lohi_neurips2023
DRD2-Hi: The nearest distance between train and test
Figure 4: DRD2-Hi: Fold 1
HIV-Hi: The nearest distance between train and test
Figure 5: HIV-Hi: Fold 1
Table 4: Hi folds
Dataset	Train 1	Test 1	Train 2	Test 2	Train 3	Test 3
DRD2-Hi	2385	1190	2381	1194	2384	1191
HIV-Hi	15696	7847	15695	7848	15695	7848
KDR-Hi	500	3116	500	3125	500	2285
Sol-Hi	1442	721	1442	721	1442	721
Table 5: Lo folds
Dataset	Train 1	Test 1	Train 2	Test 2	Train 3	Test 3
DRD2-Lo	2206	267	2128	267	2257	262
KCNH2-Lo	3313	406	3313	406	3313	406
KDR-Lo	500	437	500	520	500	417
Figure 6: The most similar pairs of molecules between train and test.
Figure 7: Example of Lo cluster in DRD2-Lo
Figure 8: Example of Lo cluster in KCNH2-Lo
B Lo dataset is not just noise
Experimental data inherently contain noise. Consequently, selecting similar molecules may result in clusters that possess such a small variation that it could be solely attributable to experimental noise, thereby invalidating the Lo task. This potential issue underlines the importance of ascertaining that the clusters exhibit a significant signal to ensure the validity of the task.
As reported [88], the standard deviation for the same ligand-protein pair's pIC50 is  $\sigma_{pIC50} \approx 0.20$  when measured in the same laboratory, and  $\sigma_{pIC50} \approx 0.68$  in the ChEMBL database. In similar work [89] standard deviation for ChEMBL pKi was found to be  $\sigma_{pKi} \approx 0.56$ . Therefore, based on these findings, we opted to select only those clusters that displayed a standard deviation exceeding 0.70 for pIC50 and more than 0.60 for pKi. These selection criteria enhance the confidence in the validity of the Lo task by prioritizing clusters with significant intracluster variation.
Figure 9: Within cluster variability is higher than noise standard deviation.
C Lo algorithm
The Python implementation can be found in code/splits.py.
Algorithm 1 Get Lo Split
Input: List of molecular SMILES  $S$ , similarity threshold  $t$ , minimum cluster size  $m$ , maximum number of clusters  $M$ , activity values  $V$ , standard deviation threshold  $std_{t}$
Output: List of SMILES clusters  $C$ , list of remaining training SMILES train_S
1: procedure GETLOPLIT(S, t, m, M, V, stdt)
2:  $C, \text{train}_S \gets \text{SELECT DISTINCT CLUSTERS}(S, t, m, M, V, \text{std}_t)$
3: for each cluster in  $C$  do
4: Move central molecule from cluster to train_S
5: end for
6: return  $C$ , train\_S
7: end procedure
Algorithm 2 Select Distinct Clusters
Input: List of molecular SMILES  $S$  , similarity threshold  $t$  , minimum cluster size  $m$  , maximum number of clusters  $M$  , activity values  $V$  , standard deviation threshold  $\text{std}_t$    
Output: List of SMILES clusters  $C$  , list of the rest training SMILES train_S   
1: function SELECTDISTINCTCLUSTERS(S, t, m, M, V,  $\text{std}_t$ )   
2: train_  $S\gets S$    
3: Initialize list  $C$  as empty   
4: while length of  $C <   M$  do   
5: Compute fingerprints  $F$  from SMILES in train_S   
6: Compute total number of neighbors  $N$  for each fingerprint in  $F$    
7: Compute STD standard deviation of  $V$  of neighbors for each fingerprint in  $F$    
8: Set central_idx to None   
9: Set least_neighbors to max(N)   
10: for each idx in 0..|train_S| do Find the smallest cluster that meets criteria   
11: if  $N[idx] > m$  and STD  $[idx] > std_{t}$  and  $N[idx] <$  least_neighbors then   
12: central_idx  $\leftarrow$  idx   
13: least_neighbors  $\leftarrow N[idx]$    
14: end if   
15: end for   
16: if central_idx is None then Exit if there are no more clusters that meet criteria   
17: break   
18: end if   
19: Add central_idx molecule and its neighbors to list of clusters  $C$    
20: Remove the cluster and its neighbors from train_S   
21: end while   
22: return C, train_S   
23: end function
D Additional benchmarks analysis
Distribution of Tanimoto Similarity between the nearest molecules between train and test.
Figure 10: ESOL
Figure 11: HIV
Figure 12: TDC
We additionally analyzed other ligand-based MoleculeNet datasets.
Table 6: Fraction of test molecules in various MoleculeNet datasets with a Tanimoto similarity  $> {0.4}$  to the train set using ECFP4 fingerprints.
Dataset	Fraction of Test Molecules Similar to Train Set
QM7	0.93
QM8	0.98
QM9	0.99
FreeSolv	0.8
Lipophilicity	0.67
PCBA	>0.93
MUV	0.96
BACE	0.77
Tox21	0.52
SIDER	0.48
E Graph coarsening algorithm
The Python implementation can be found in code/minvertex_k_cut.py. We are planning to release it as a pip package.
Algorithm 3 Calculate Neighbors  
Input: Graph  $G = (V, E)$ , similarity threshold  $\theta$   
Output: List of tuples  $n\_ neighbors$   
1: function CALCULATEIGHBORS( $G, \theta$ )  
2: Initialize list  $n\_ neighbors$  as empty  
3: for each node  $v$  in  $V$  do  
4: Initialize total_neighbors as 0  
5: for each edge  $e$  incident on node  $v$  do  
6: if  $e[('similarity') > \theta$  then  
7: total_neighbors ← total_neighbors + 1  
8: end if  
9: end for  
10: Append (total_neighbors, index of  $v$ ) to  $n\_ neighbors$   
11: end for  
12: return  $n\_ neighbors$   
13: end function
Algorithm 4 Cluster Nodes
Input: Sorted list n_neighbors, Graph  $G = (V,E)$ , similarity threshold  $\theta$   
Output: Cluster assignment node_to_cluster, number of clusters total_clusters  
1: function CLUSTERNODES(n_neighbors, G,  $\theta$ )  
2: Initialize array node_to_cluster of size  $|V|$  as  $-1$   
3: Initialize totalClusters as 1  
4: for each tuple (count, node) in n_neighbors do  
5: if node_to_cluster(node] = -1 then  
6: node_to_cluster(node] ← total_clusters  
7: for each edge e incident on node node do  
8: if e['similarity'] >  $\theta$  then  
9: adjacent_node ← e[1]  
10: if node_to_cluster[adjacent_node] = -1 then  
11: node_to_cluster[adjacent_node] ← total_clusters  
12: end if  
13: end if  
14: end for  
15: totalClusters ← totalclusters + 1  
16: end if  
17: end for  
18: return node_to_cluster, total_clusters  
19: end function
Algorithm 5 Build Coarse Graph
Input: Cluster assignment node_to_cluster, number of clusters totalClusters, Graph  $G = (V,E)$    
Output: Coarsened Graph  $G_{\mathrm{coarse}}$    
1: function BUILDCOARSEGRAPH(node_to_cluster, total_Clusters, G)   
2: Compute clusters_size, count of each unique element in node_to_cluster   
3: Initialize  $G_{\mathrm{coarse}}$  as an empty graph   
4: for cluster in 0 to total_Clusters - 1 do Add nodes   
5: Add node cluster with weight clusters_size[cluster] to  $G_{\mathrm{coarse}}$    
6: end for   
7: for cluster in 0 to total_Clusters - 1 do Add edges   
8: Initialize connected_Clusters as an empty set   
9: Get nodes of cluster as this_cluster_values where node_to_cluster equals cluster   
10: for each node in this_cluster_values do   
11: for each edge e incident on node node do   
12: Add node_to_cluster[e[1]] to connected_Clusters   
13: end for   
14: end for   
15: for each connected_cluster in connected_Clusters do   
16: Add edge from cluster to connected_cluster in  $G_{\mathrm{coarse}}$    
17: end for   
18: end for   
19: return  $G_{\mathrm{coarse}}$    
20: end function
Algorithm 6 Main Procedure
Input: Graph  $G = (V,E)$  , similarity threshold  $\theta$    
Output: Coarsened graph  $G_{\mathrm{coarse}}$    
1: procedure COARSEGRAPH(G,  $\theta$  1   
2: n_neighbors  $\leftarrow$  CALCULATENEIGHBORS(G,  $\theta$  3 Sort n_neighbors in descending order of first element of each tuple   
4: node_to_cluster, total_clusters  $\leftarrow$  CLUSTERNODES(n_neighbors,G,  $\theta$  5:  $G_{\mathrm{coarse}}\gets$  BUILDCOARSEGRAPH(node_to_cluster,total_clusters,G)   
6: return  $G_{\mathrm{coarse}}$    
7: end procedure
F Hi-split predicts virtual screening hit rate better than scaffold split
For effective virtual screening, predicting experimental outcomes prior to experimentation is paramount. In this study, we compare the predictive performance of the novel Hi-split approach with the traditional scaffold split method under a Hit Identification scenario. Following existing literature [10, 11, 16, 27, 28], we simulate testing on novel molecules with an ECFP4 Tanimoto similarity of  $\leq 0.4$  to the training set. The dataset is partitioned using both splitting methods to form separate training and validation sets for hyperparameter selection. Hyperparameter search is performed for gradient boosting on ECFP4 fingerprints, identified as the most efficient Hi model that facilitates quick training.