SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jun Xia\*, Lecheng Zhang\*, Xiao Zhu\*, Yue Liu, Zhangyang Gao, Bozhen Hu, Cheng Tan, Jiangbin Zheng, Siyuan Li, Stan Z. Li†  
School of Engineering, Westlake University  
{xiajun, zhanglecheng, stan.zq.li}@westlake.edu.cn
Molecular Property Prediction (MPP) is a crucial task in the AI-driven Drug Discovery (AIDD) pipeline, which has recently gained considerable attention thanks to advancements in deep learning. However, recent research has revealed that deep models struggle to beat traditional non-deep ones on MPP. In this study, we benchmark 12 representative models (3 non-deep models and 9 deep models) on 15 molecule datasets. Through the most comprehensive study to date, we make the following key observations: (i) Deep models are generally unable to outperform non-deep ones; (ii) The failure of deep models on MPP cannot be solely attributed to the small size of molecular datasets; (iii) In particular, some traditional models including XGB and RF that use molecular fingerprints as inputs tend to perform better than other competitors. Furthermore, we conduct extensive empirical investigations into the unique patterns of molecule data and inductive biases of various models underlying these phenomena. These findings stimulate us to develop a simple-yet-effective feature mapping method for molecule data prior to feeding them into deep models. Empirically, deep models equipped with this mapping method can beat non-deep ones in most MoleculeNet datasets. Notably, the effectiveness is further corroborated by extensive experiments on cutting-edge dataset related to COVID-19 and activity cliff datasets.
Molecular Property Prediction (MPP) is a critical task in computational drug discovery, aimed at identifying molecules with desirable pharmacological and ADMET (absorption, distribution, metabolism, excretion, and toxicity) properties. Machine learning models have been widely used in this fast-growing field, with two types of models being commonly employed: traditional nondeep models and deep models. In non-deep models, molecules are fed into traditional machine learning models such as random forest and support vector machines in the format of computed or handcrafted molecular fingerprints [64]. The other group utilizes deep models to extract expressive representations for molecules in a data-driven manner. Specifically, the Multi-Layer Perceptron (MLP) could be applied to computed or handcrafted molecular fingerprints; Sequence-based neural architectures including Recurrent Neural Networks (RNNs) [43], 1D Convolutional Neural Networks (1D CNNs) [22], and Transformers [25, 54] are exploited to encode molecules represented in Simplified Molecular-Input Line-Entry System (SMILES) strings [71]. Additionally, molecules can be naturally represented as graphs with atoms as nodes and bonds as edges, inspiring a line of works to leverage such structured inductive bias for better molecular representations [20, 76, 79, 58]. The key advancements underneath these approaches are Graph Neural Networks (GNNs), which
Figure 1: Exemplary molecular descriptors and their corresponding models in our benchmark. SVM: Support Vector Machine [84]; RF: Random Forest [61]; XGB: eXtreme Gradient Boosting [8]; MLP: Multi-Layer Perceptron; CNN: 1D Convolution Neural Network [32]; RNN: Recurrent Neural Network (GRU) [46]; TRSF: TRanSFormer [67]; GCN: Graph Convolution Network [33]; MPNN: Message-Passing Neural Network [20]; GAT: Graph Attention neTwork [68]; AFP: Attentive FP [77]; SPN: SphereNet [38]. The above-mentioned abbreviations are applicable throughout the entire paper.
consider graph structures and attributive features simultaneously in the learning process [33, 68, 24]. More recently, researchers incorporate 3D conformations of molecules into their representations for better performance, whereas pragmatic considerations such as calculation cost, alignment invariance, uncertainty in conformation generation, and unavailable conformations of target molecules limited the practical applicability of these models [5, 17, 57, 16, 38]. We summarize the widely-used molecular descriptors and their corresponding models in our benchmark, as shown in Figure 1. Despite the fruitful progress, previous studies [41, 29, 79, 65, 30, 13, 66] have observed that deep models struggled to outperform non-deep ones on molecular datasets. However, these studies neither consider the emerging powerful deep models (e.g., Transformer [25], SphereNet [37]) nor explore various molecular descriptors (e.g., 3D molecular graph). Also, they did not investigate the reasons why deep models often fail on molecules.
To narrow this gap, we present the most comprehensive benchmark study on molecular property prediction to date, with a precise methodology for dataset inclusion and hyperparameter tuning. Our empirical results confirm the observations of previous studies, namely that deep models generally struggle to outperform traditional non-deep counterparts, even without accounting for the slower training of deep learning algorithms. Moreover, we observe several interesting phenomena that challenge the prevailing beliefs of the community, which can guide optimal methodology design for future studies.
Furthermore, we aim to understand why deep models often underperform non-deep ones in MPP. Specifically, we transform the original molecular data to observe the performance changes of various models, uncovering the unique patterns of molecular data and the differing inductive biases of various models. These in-depth empirical studies shed light on the benchmarking results: Deep models struggle to learn non-smooth target functions that map molecular data to labels, while the target functions are often non-smooth in MPP. This means that small changes in the chemical structure of a molecule may result in large changes in molecular properties. Additionally, deep models tend to attend to molecule features as a whole, especially handling the molecular fingerprints, while partial substructures known as functional groups are the most informative for molecules. On the other hand, XGB and random forest are well-suited for molecules because they make decisions based on each dimension of molecular features separately. Based on these phenomena and analyses, we develop a novel feature mapping method for molecule data before feeding them into models. Theoretically, we show that our method can help deep models learn non-smooth target functions that map molecules to properties. Moreover, our method is readily pluggable into various deep methods for performance improvement.
We highlight the following contributions: (I) We provide the most comprehensive benchmark on MPP tasks to date and expose the limitations of deep models on molecule datasets. Our findings offer new and valuable insights for the fast-growing AIDD community. (II) We empirically investigate the unique patterns of molecular data and inductive biases of various models, providing explanations for why deep models often cannot beat non-deep ones on MPP tasks. (III) We develop a simple-yet-effective feature mapping method to help deep models learn the non-smooth target functions with theoretical guarantees. (IV) We verify the effectiveness of our method through extensive experiments on MoleculeNet datasets, a cutting-edge dataset related to COVID-19 and activity cliff datasets.
In this section, we elaborate on various molecular descriptors and their respective learning models.
2.1 Fingerprints-based Molecular Descriptors
Molecular fingerprints (FPs) serve as one of the most important descriptors for molecules. Typical examples include Extended-Connectivity Fingerprints (ECFP) [53] and PubChemFP [70]. These fingerprints encode the neighboring environments of heavy atoms in a molecule into a fixed bit string with a hash function, where each bit indicates whether a certain substructure is present in the molecule. Traditional models and MLPs can take these fingerprints as 'raw' input. However, the high-dimensional and sparse nature of FPs introduces additional efforts for feature selection when they are fed into certain models. Additionally, it is difficult to interpret the relationship between properties and structures because the hash functions are non-invertible.
2.2 Linear Notation-based Molecular Descriptors
Another option for molecules is linear notations, among which SMILES [71] is the most frequently-used one owing to its versatility and interpretability. In SMILES, each atom is represented as a respective ASCII symbol; Chemical bonds, branching, and stereochemistry are denoted by specific symbols. However, a significant fraction of SMILES strings does not correspond to chemically valid molecules. As a remedy, a new language named SELF-referencing Embedded Strings (SELFIES) for molecules was introduced in 2020 [34]. Every SELFIES string corresponds to a valid molecule, and SELFIES can represent every molecule. Naturally, RNNs, 1D CNN, and Transformers are powerful deep models for processing such sequences [69, 86, 25, 55, 82]. However, the poor scalability of the sequential notations and the loss of spatial information limit the performances of these approaches.
2.3 2D and 3D Graph-based Molecular Descriptors
Molecules can be represented with graphs naturally, with nodes as atoms and edges as chemical bonds. Initially, [15] first adopted convolutional layers to encode molecular graphs to neural fingerprints. Following this work, [11] employs the atom-based message-passing scheme to learn expressive molecular graph representations. To complement the atom's information, [31] utilized both the atom's and bonds' attributes, and MPNN [20] generalized it to a unified framework. Also, multiple variants of the MPNN framework are developed to avoid unnecessary loops (DMPNN [79]), to strengthen the message interactions between nodes and edges (CMPNN [58]), to capture the complex inherent quantum interactions of molecules (MGCN [39]), or take the longer-range dependencies (Attentive FP [76]). More recently, some hybrid architectures [54, 81, 42, 45] of GNNs and transformers are emerging to capture the topological structures of molecular graphs.
The 3D molecular graph is composed of nodes (atoms), and their positions in 3D space and edges (bonds). The advantage of using 3D geometry is that the conformer information is critical to many molecular properties, especially quantum properties. In addition, it is also possible to directly leverage stereochemistry information such as chirality given the 3D geometries. Recently, multiple works [57, 56, 14, 38, 3] have developed message-passing mechanisms tailored for 3D geometries, which enable the learned molecular representations to follow certain physical symmetries, such as equivariance to translations and rotations. However, the calculation cost, alignment invariance, uncertainty in conformation generation, and unavailable conformations of target molecules limited the applicability of these models in practice.
3 Benchmark Representative Models on Multiple Molecular Datasets.
In this section, we present a benchmark on 15 molecular datasets with 12 representative models.
Fingerprints  $\longmapsto$  SVM, XGB, RF, and MLP. Following the common practice [30, 63, 50], we feed the concatenation of various molecular fingerprints including 881 PubChem fingerprints (PubchemFP), 307 substructure fingerprints (SubFP), and 206 MOE 1-D and 2-D descriptors [80] to
SVM, XGB, RF, and MLP models to comprehensively represent molecular structures, with some pre-processing procedures to remove features (1) with missing values; (2) with extremely low variance (variance  $< 0.05$ ); (3) have a high correlation (pearson correlation coefficient  $>0.95$ ) with another feature. The retained features are normalized to the mean value of 0 and variance of 1. Additionally, considering that traditional machine models (SVM, RF, XGB) cannot be directly applied in the multi-task molecular datasets, we split the multi-task dataset into multiple single-task datasets and use each of them to train the models. Finally, we report the average performance of these single tasks.
SMILES strings  $\longmapsto$  CNN, RNN, and TRSF. We adopt the 1D CNNs from a recent study [32], which include a single 1D convolutional layer with a step size equal to 1, followed by a fully connected layer. As for the RNN, we use a 3-layer bidirectional gated recurrent units (GRUs) [10] with 256 hidden vector dimensions. Additionally, we use the pre-trained SMILES transformer [25] with 4 basic blocks and each block has 4-head attentions with 256 embedding dimensions and 2 linear layers. The SMILES are split into symbols (e.g., 'Br', 'C', '=', '('', '2')') and then fed into the transformer together with the positional encoding [67].
2D Graphs  $\longmapsto$  GCN, MPNN, GAT, and AFP. As in previous studies [76], we exhaustively utilized all readily available atom/bond features in our 2D graph-based descriptors. Specifically, we have incorporated 9 atom features, including atom symbol, degree, and formal charge, using a one-hot encoding scheme. In addition, we included 4 bond features, such as type, conjugation, ring, and stereo. The resulting encoded graphs were then fed into GCN, MPNN, GAT, and AFP models. Further details on the graph descriptors used in our experiments can be found in [76].
3D Graphs  $\longmapsto$  SPN. We employ the recently proposed SphereNet [38] for molecules with 3D geometry. Specifically, for quantum mechanics datasets (QM7, QM8, and QM9) that contain 3D atomic coordinates calculated with ab initio Density Functional Theory (DFT), we feed them into SphereNet directly. For other datasets without labeled conformations, we used RDKit [35]-generated conformations to satisfy the request of SphereNet.
Datasets splits, evaluation protocols and metrics, hyper-parameters tuning. Firstly, we randomly split the training, validation, and test sets at a ratio of 8:1:1. And then, we tune the hyper-parameters based on the performance of the validation set. Due to the heavy computational overhead, GNNs-based models on the HIV and MUV datasets are in 30 evaluations; all the models on the QM7 and QM8 are in 10 evaluations; all the models on the QM9 dataset are in one evaluation. And then, we conduct 50 independent runs with different random seeds for dataset splitting to obtain more reliable results, using the optimal hyper-parameters determined before. Following MoleculeNet benchmark [72], we evaluate the classification tasks using the area under the receiver operating characteristic curve (AUC-ROC), except the area under the precision curve (AUC-PRC) on MUV dataset due to its extreme biased data distribution. The performance on the regression task are reported using root mean square error (RMSE) or mean absolute error (MAE). Kindly note that we report the average performance across multi-tasks on some datasets because they contain more than one task. Additionally, to avoid the overfitting issue, all the deep models are trained with an early stopping scheme if no validation performance improvement is observed in successive 50 epochs. We set the maximal epoch as 300 and the batch-size as 128. We provide more details including hyper-parameters tuning space for each model in the appendix.
3.2 Observations
Table 1 documents the benchmark results for various models and datasets, from which we can make the following Observations:
Observation 1. Deep models underperform non-deep counterparts in most cases.
As can be observed in Table 1, non-deep models rank as the top one on 11/15 datasets. Kindly note that we report the results of each task in the QM9 dataset in the appendix. On some datasets such as MUV, QM7, and BACE, three non-deep models can even beat any deep models.
Observation 2. The failure of deep models should not be solely attributed to the limited size of molecular datasets.
Intuitively, many previous works [21, 79, 59] pointed out that the small size of molecular datasets is a bottleneck for deep learning models and propose various strategies accordingly [49, 48]. Here, we complement this pre-dominant belief with a new opinion with empirical evidence. As shown in Table 1, all the non-deep models can outperform any deep ones on some larger-scale datasets (e.g.,
Table 1: The comparison of representative models on multiple molecular datasets. The standard deviations can be seen in the appendix for the limited space. No.: Number of the molecules in the datasets. The top-3 performances on each dataset are highlighted with the grey background. The best performance is highlighted with bold. Kindly note that 'TRSF' denotes the transformer that has been pre-trained on 861,000 molecular SMILES strings. The results on QM 9 can be seen in the appendix.
Dataset (No.)	Metric	SVM	XGB	RF	CNN	RNN	TRSF	MLP	GCN	MPNN	GAT	AFP	SPN
BACE (1,513)	AUC_ROC	0.886	0.896	0.890	0.815	0.559	0.835	0.887	0.880	0.846	0.886	0.879	0.882
HIV (40,748)	AUC_ROC	0.817	0.823	0.826	0.733	0.639	0.748	0.791	0.834	0.814	0.812	0.819	0.818
BBBP (2,035)	AUC_ROC	0.913	0.926	0.923	0.760	0.693	0.897	0.918	0.915	0.872	0.902	0.893	0.905
ClinTox (1,475)	AUC_ROC	0.879	0.919	0.933	0.685	0.813	0.963	0.890	0.889	0.868	0.891	0.907	0.912
SIDER (1,366)	AUC_ROC	0.626	0.638	0.644	0.591	0.515	0.641	0.617	0.633	0.603	0.614	0.620	0.613
Tox21 (7,811)	AUC_ROC	0.820	0.837	0.838	0.766	0.734	0.817	0.834	0.830	0.816	0.829	0.845	0.827
ToxCast (8,539)	AUC_ROC	0.725	0.785	0.778	0.735	0.74	0.780	0.781	0.767	0.736	0.768	0.788	0.772
MUV (93,087)	AUC_PRC	0.093	0.072	0.069	0.045	0.094	0.059	0.018	0.056	0.019	0.055	0.044	0.058
SARS-CoV-2 (14,332)	AUC_ROC	0.599	0.700	0.686	0.688	0.649	0.643	0.638	0.646	0.640	0.683	0.651	0.663
ESOL (1,127)	RMSE	0.676	0.583	0.647	2.569	1.511	0.718	0.653	0.773	0.695	0.661	0.594	0.671
Lipop (4,200)	RMSE	0.683	0.585	0.626	1.016	1.207	0.947	0.633	0.665	0.669	0.680	0.664	0.630
FreeSolv (639)	RMSE	1.063	0.715	1.014	2.275	2.205	1.504	1.046	1.316	1.327	1.304	1.139	1.159
QM7 (6,830)	MAE	42.814	52.726	51.403	81.165	158.160	64.363	86.060	64.530	107.013	78.217	59.973	55.727
QM8 (21,786)	MAE	0.0364	0.0126	0.0098	0.0205	0.0295	0.0232	0.0104	0.0154	0.0109	0.0187	0.0098	0.0103
Figure 2: The performance of various models on the smoothed datasets. Left: ESOL (Regression); Middle: Lipop (Regression); Right: QM7 (Regression). Kindly note that we only smooth the regression datasets because the labels of classification datasets are not suitable for smoothing.
MUV and QM 7). However, in some small datasets (e.g., ClinTox and ESOL), some deep models can beat partial non-deep ones. Therefore, we argue that there could be other reasons for the failure of deep models, not solely the dataset size. We will provide further analysis in Sec. 4.
Observation 3. XGB and RF exhibit a particular advantage over other models.
In the experiments shown in Table 1, we can see that the XGB and RF models consistently rank among the top three on each dataset. Additionally, tree models rank as the top one on 8/15 datasets. Next, we will explore why tree models are well-suited for molecular fingerprints in Sec. 4.
4 Empirical Study: Why above phenomena would occur?
In this section, we attempt to understand which characteristics of molecular data lead to the failure of powerful deep models. Also, we aim to understand the inductive biases of XGB and RF that make them well-suited for molecules, and how they differ from the inductive biases of deep models. The details of the experiments in this section can be found in the appendix.
Explanation 1. Unlike image data, molecular data patterns are non-smooth. Deep models struggle to learn non-smooth target functions that map molecules to properties.
We design two experiments to verify the above explanation, i.e., increasing or decreasing the level
Table 2:  $\mathrm{RMSE}_c$  are the prediction RMSE on cliff molecules, respectively.  $\Delta \mathcal{P}$  is the predicted bioactivity change rate when transitioning from non-cliff to cliff molecules with subtle structural changes. The top-3 performances on each dataset are highlighted with the grey background. The best performance is highlighted with bold.
Target name (Response type)	Metric	SVM	XGB	RF	CNN	RNN	TRSF	MLP	GCN	MPNN	GAT	AFP
CB1 (Agonism \( EC_{50} \))	\( RMSE_c \)	0.773	0.767	0.770	0.944	0.823	0.888	0.807	0.992	0.989	0.975	0.967
\( \Delta \mathcal{P} \)	15.04%	21.13%	20.76%	2.07%	10.15%	9.42%	13.32%	4.13%	3.85%	1.17%	4.35%
DAT (Inhibition \( K_i \))	\( RMSE_c \)	0.744	0.696	0.730	0.894	0.783	0.934	0.792	1.003	0.921	1.042	0.995
\( \Delta \mathcal{P} \)	20.64%	23.03%	23.95%	2.73%	14.18%	9.34%	15.02%	5.83%	5.15%	2.27%	5.08%
\( PPAR\alpha \)(Agonism \( EC_{50} \))	\( RMSE_c \)	0.671	0.678	0.685	0.962	0.825	0.968	0.713	0.870	0.872	0.929	0.823
\( \Delta \mathcal{P} \)	21.07%	22.93%	23.14%	11.29%	13.48%	18.39%	15.29%	1.83%	5.18%	4.93%	11.73%
DOR (Inhibition \( K_i \))	\( RMSE_c \)	0.861	0.854	0.836	1.098	1.036	1.032	0.874	1.259	1.152	1.281	1.179
\( \Delta \mathcal{P} \)	25.26%	28.41%	23.95%	10.02%	9.83%	10.25%	15.18%	9.77%	12.52%	11.36%	13.11%
of data smoothing in the molecular datasets. Firstly, we 'increase' the molecular data smoothing level by smoothing the labels based on similarities between molecules. Specifically, let  $\mathcal{D}$  denote the molecular dataset and  $(x_{i},y_{i})\in \mathcal{D}$  be  $i$ -th molecule and its label, we smooth the target function as follows,
$$
\widehat {y _ {i}} = \frac {\sum_ {x _ {j} \in \mathcal {N} _ {x _ {i}}} s \left(x _ {i} , x _ {j}\right) y _ {j}}{\sum_ {x _ {j} \in \mathcal {N} _ {x _ {i}}} s \left(x _ {i} , x _ {j}\right)}, \tag {1}
$$
where  $s(\cdot, \cdot)$  denotes the Tanimoto coefficient of the extended connectivity fingerprints (ECFP) between two molecules that can be considered as their structural similarity.  $\mathcal{N}_{x_i}$  is the  $k$ -nearest neighbor set of  $x_i$  (including  $x_i$ ) picked from the whole dataset based on the structural similarities.  $\widehat{y}_i$  denotes the label after smoothing. We smooth all the molecules in the dataset in this way and use the smoothed label  $\widehat{y}_i$  to train the models. The results are shown in Figure 2, where '0-smooth' denotes the original datasets. '10-smooth' and '20-smooth' mean  $k = 10$  and  $k = 20$ , respectively. As can be observed, the performance of deep models improves dramatically as the level of dataset smoothing increases, and many deep models including MLP, GCN, and AFP can even beat non-deep ones after smoothing. These phenomena indicate that deep models are more suitable for the smoothed datasets.
Secondly, we 'decrease' the level of data smoothing using the concept of activity cliff [40, 60] from chemistry, which means a situation where small changes in the chemical structure of a drug lead to significant changes in its bioactivity. We provide an example activity cliff pairs in Figure 3. Apparently, the target function of activity cliffs that map molecules to the activity values is less smoothing than normal molecular datasets. We then evaluate the models on the activity cliff datasets [66]. The test set contains molecules that are chemically similar to those in the training set but exhibit either a large difference in bioactivity (cliff molecules) or similar bioactivity (non-cliff molecules). As shown in Table 2, the non-deep models consistently outper
form deep ones on these activity cliff datasets. Furthermore, the deep models exhibit less significant prediction change rates compared to the non-deep ones. This observation suggests that deep models are indeed less sensitive to subtle structural changes compared to non-deep ones. Our explanation is consistent with the conclusions in deep learning theory [51], i.e., deep models struggle to learn high-frequency components of the target functions. However, traditional models such as XGB and RF can learn piece-wise target functions, and do not exhibit such bias. Our explorations uncover several promising avenues to enhance deep models' performance on molecules: smoothing the target functions or improving deep models' ability to learn the non-smooth target functions.
$K_{1} = 3 \mathrm{nM}$
$K_{\mathrm{r}} = 126 \mathrm{nM}$
Figure 3: Exemplary of Activity Cliffs (ACs) on the target named dopamine D3 receptor (D3R).  $\mathrm{K}_i$  means the bioactivity values. The figure is adapted from a previous work [66] with permission.