Understanding the Limitations of Deep Models for Molecular property prediction: Insights and Solutions
Authors: Jun Xia
Lecheng Zhang
Xiao Zhu
Yue Liu
Zhangyang Gao
Bozhen Hu
Cheng Tan
Jiangbin Zheng
Siyuan Li
Stan Z. Li
Venue: NeurIPS 2023 poster (2023)
Keywords: Graph Neural Networks
TL;DR: We comprehensively evaluate various models on molecular property prediction, explain the failure of deep models and propose a feature mapping method for molecule data to improve the performance of deep models.

Abstract: Molecular Property Prediction (MPP) is a crucial task in the AI-driven Drug Discovery (AIDD) pipeline, which has recently gained considerable attention thanks to advancements in deep learning. However, recent research has revealed that deep models struggle to beat traditional non-deep ones on MPP. In this study, we benchmark 12 representative models (3 non-deep models and 9 deep models) on 15 molecule datasets. Through the most comprehensive study to date, we make the following key observations: \textbf{(\romannumeral 1)} Deep models are generally unable to outperform non-deep ones; \textbf{(\romannumeral 2)} The failure of deep models on MPP cannot be solely attributed to the small size of molecular datasets; \textbf{(\romannumeral 3)} In particular, some traditional models including XGB and RF that use molecular fingerprints as inputs tend to perform better than other competitors. Furthermore, we conduct extensive empirical investigations into the unique patterns of molecule data and inductive biases of various models underlying these phenomena. These findings stimulate us to develop a simple-yet-effective feature mapping method for molecule data prior to feeding them into deep models. Empirically, deep models equipped with this mapping method can beat non-deep ones in most MoleculeNet datasets. Notably, the effectiveness is further corroborated by extensive experiments on cutting-edge dataset related to COVID-19 and activity cliff datasets.

Understanding the Limitations of Deep Models for Molecular property prediction: Insights and Solutions
Jun Xia, Lecheng Zhang, Xiao Zhu, Yue Liu, Zhangyang Gao, Bozhen Hu, Cheng Tan, Jiangbin Zheng, Siyuan Li, Stan Z. Li School of Engineering, Westlake University {xiajun, zhanglecheng, stan. 1isttelllaaeeeeee edu. cn
Abstract
Molecular Property Prediction (MPP) is crucial task in the Al-driven Drug Discovery (AIDD) pipeline, which has recently gained considerable attention thanks advancements in deep learning. However, recent research has revealed that deep models struggle to beat traditional non-deep ones on MPP. In this study, benchmark 12 representative models (3 non-deep models and 9 deep models) on 15 molecule datasets. Through the most comprehensive study to date, we make the following key observations: (i) Deep models are generally unable outperform non-deep ones; (ii) The failure of deep models on MPP cannot be solely attributed the small size of molecular datasets; (iii) particular, some traditional models including XGB and RF that use molecular fingerprints as inputs tend perform better than other competitors. Furthermore, we conduct extensive empirical investigations into the unique patterns of molecule data and inductive biases of various models underlying these phenomena. These findings stimulate us to develop a simple-yet-effectivy feature mapping method for molecule data prior to feeding them into deep models. Empirically, deep models equipped with this mapping method can beat non-deep ones in most MoleculeNet datasets. Notably, the effectiveness is further corroborated by extensive experiments on cutting-edge dataset related to COVID-19 and activity cliff datasets.
Introduction
Molecular Property Prediction (MPP) is critical task in computational drug discovery, aimed at identifying molecules with desirable pharmacological and ADMET (absorption, distribution, metabolism, excretion, and toxicity) properties. Machine learning models have been widely used in this fast-growing field, with two types of models being commonly employed: traditional non- deep models and deep models. non-deep models, molecules are fed into traditional machine learning models such random forest and support vector machines in the format of computed or handcrafted molecular fingerprints [64]. The other group utilizes deep models to extract expressive representations for molecules in data-driven manner. Specifically, the Multi-Layer Perceptron (MLP) could be applied to computed or handcrafted molccular fingerprints; Sequence- neural architectures including Recurrent Neural Networks (RNNs) 43]. ID Convolutional Neural Networks (1D CNNs) (22). and Transformers 255 54 are exploited to encode molecules represented in Simplified Molecular- Line Entry System (SMILES) strings (71). Additionally, molecules can be naturally represented as graphs with atoms as nodes and bonds as edges, inspiring line of works to leverage such structured inductive bias for better molecular representations (0 76 79 58).
The key advancements underneath these approaches are Graph Neural Networks (GNNs), which
"quual Contribution.
Conresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Aloms fe40.r96
Z
0
Descriplors
Nctcacjo:1)CI-O00
1
C
Bonds
Models
SVM
RF
XGB
MLP
CNN
RNN
TRSF
GCN
MPNN
GAT
AFP
SPN
(8) Fingerprints (FPs)
(b) SMILES string
2D Graph
(D)3D Graph
Figure 1: Exemplary molecular descriptors and their corresponding models in our benchmark. SVM:
Support Vector Machine [84]; RF: Random Forest [6]]; XGB: eXtreme Gradient Boosting [8:
MIP: Multi-Layer Perceptron: CNN: ID Convolution Neural Network (32] RNN: Recurrent Neural
Network (GRU) [46]; TRSF: Former [67]: GCN: Graph Convolution Network (33; MPNN:
Message-Passing Neural Network 20); GAT Graph Attention nwokk [68]; AFP: Attentive FP [77]:
SPN: SPhereNet (38). The above-mentioned abbreviations are applicable throughout the entire paper.
consider graph structures and attributive features simultaneously in the learning process 33 68 More recently, researchers incorporate 3D conformations of molecules into their representations for better performance, whereas pragmatic considerations such as calculation cost, alignment invariance, uncertainty conformation generation, and unavailable conformations of target molecules limited the practical applicability of these models [7]
24
(38). We summarize the widely-used molecular [7] 16,
descriptors and their corresponding models in our benchmark, as shown in Figure [ Despite the fruitful progress, previous studies [41
13 (66] have observed that deep models struggled to outperform non-deep ones on molecular datasets. However, these studies neither consider the emerging powerful deep models (e.g., Transformer [25]] SphereNet (37]) nor explore various molecular descriptors (e.g., 3D molecular graph). Also, they did not investigate the reasons why deep models often fail molecules.
29 79
65
30
To narrow this gap, we present the most comprehensive benchmark study molecular property prediction to date, with precise methodology for dataset inclusion and hyperparameter tuning. Our empirical results confirm the observations of previous studies, namely that deep models generally struggle to outperform traditional non-deep counterparts, even without accounting for the slower training of deep learning algorithms. Moreover, we observe several interesting phenomena that challenge the prevailing beliefs of the community, which can guide optimal methodology design for future studies.
Furthermore, we aim to understand why deep models often underperform non-deep ones in MPP. Specifically, transform the original molecular data observe the performance changes of various models, uncovering the unique patterns of molecular data and the differing inductive biases of various models. These in-depth empirical studies shed light on the benchmarking results: Deep models struggle to learn non-smooth target functions that map molecular data to labels, while the target functions are often non-smooth in MPP. This means that small changes in the chemical structure of a molecule may result in large changes in molecular properties. Additionally, deep models tend to attend molecule features as whole, especially handling the molecular fingerprints, while partial substructures known as functional groups are the most informative for molecules. On the other hand, XGB and random forest are well-suited for molecules because they make decisions based on each dimension of molecular features separately. Based on these phenomena and analyses, we develop novel feature mapping method for molecule data before feeding them into models. Theoretically, we show that our method can help deep models learn non-smooth target functions that map molecules to properties. Moreover, our method is readily pluggable into various deep methods for performance improvement.
We highlight the following contributions: (1) We provide the most comprehensive benchmark on MPP tasks to date and expose the limitations of deep models on molecule datasets. Our findings offer new and valuable insights for the fast-growing AIDD community. (II) We empirically investigate the unique patterns of molecular data and inductive biases of various models, providing explanations for why deep models often cannot beat non-deep ones on MPP tasks. (III) We develop a simple-yet- effective feature mapping method to help deep models learn the non-smooth target functions with theoretical guarantees. (IV) We verify the effectiveness of our method through extensive experiments on MoleculeNet datasets, cutting-edge dataset related to COVID-19 19 and activity cliff datasets.

---

2
Related Work
In this section, we claborate on various molecular descriptors and their respective learning models.
2.1 Fingerprints-based Molecular Deseriptors
Molecular fingerprints (FPs) serve as one of the most important descriptors for molecules. Typical examples include fingerprints encode the neighboring environments of heavy atoms in a molecule into a fixed bit string with hash function, where cach bit indicates whether certain substructure is present in the molecule. Traditional models and MLPs can take these fingerprints as 'raw' input. However, the high-dimensional and sparse nature of FPs introduces additional efforts for feature selection when they are fed into certain models. Additionally, it is difficult to interpret the relationship between properties and structures because the hash functions are non-invertible.
Fingerprints (ECFP) [53] and PubChemFP [70]. These
2.2 Linear Notation-based Molecular Descriptors
Another option for molecules is linear notations, among which SMILES [7T] is the most frequently- used one owing to its versatility and interpretability. In SMILES, each atom is represented respective ASCII symbol; Chemical bonds, branching, and stereochemistry are denoted by specific symbols. However, significant fraction of SMILES strings does not correspond to chemically valid molecules. As a remedy, a new language named SELF- reffeeeenennnnnggg Embedded Strings (SELFIES) for molecules was introduced in 2020 [34]. Every SELFIES string corresponds to valid molecule, and SELFIES can represent every molecule. Naturally, RNNs, ID CNN, and Transformers are powerful deep models for processing such sequences [69 86 25 35 82). However, the poor scalability of the sequential notations and the loss of spatial information limit the performances of these approaches.
2.3 2D and 3D Graph-based Molecular Descriptors
Molecules can be represented with graphs naturally, with nodes as atoms and edges chemical bonds. Initially, [15] first adopted convolutional layers to encode molecular graphs to neural fingerprints. Following this work, [ employs the atom-based gessaae-passing scheme to learn expressive molecular graph representations. To complement the atom's 's information, [31 utilized both the atom's 's and bonds' attributes, and MPNN (20) generalized it to unified framework. Also, multiple variants of the MPNN framework are developed to avoid unnecessary loops (DMPNN (791), to strengthen the message interactions between nodes and edges (CMPNN [58)), to capture the complex inherent quantum interactions of molecules (MGCN (39 or take the longer-range dependencies (Attentive FP (6)). More recently, some hybrid architectures 54
45 of GNNs and transformers
emerging to capture the topological structures of molecular graphs.
The 3D molecular graph is composed of nodes (atoms), and their positions in 3D space and edges (bonds). The advantage of using 3D geometry is that the conformer information is critical to many molecular properties, especially quantum properties. In addition, it is also possible to directly leverage stereochemistry information such as chirality given the 3D geometries. Recently, multiple works (57 56 14, 38 have developed pesssage-passing mechanisms tailored for 3D geometries, which enable the learned molecular representations to follow certain physical symmetries, such as equivariance to translations and rotations. However, the calculation cost, alignment invariance, uncertainty in conformation generation, and unavailable conformations of target molecules limited the applicability of these models in practice.
3 Benchmark Representative Models on Multiple Molecular Datasets.
In this section, we present a benchmark on 15 molecular datasets with 12 representative models.
3.1 Experimental Setups
Fingerprints - SVM, XGB, RF, and MLP. Following the common practice (30
63
50
we feed the concatenation of various molecular fingerprints including 881 PubChem fingerprints (PubchemFP). 307 substructure fingerprints (SubFP), and 206 MOE 1-D and 2-D descriptors (80) to

---

SVM, XGB, RF, and MLP models to comprehensively represent molecular structures, with some pre prepprceesiing procedures remove features (1) with missing values; (2) with extremely low variance (variance < 0.05); (3) have a high correlation (pearson correlation coefficient > 0.95) with another feature. The retained features are normalized to the mean value of and variance of 1. Additionally, considering that traditional machine models (SVM, RF, XGB) cannot be directly applied in the multi-task molecular datasets, we split the multi-task dataset into multiple single-task datasets and use each of them to train the models. Finally, we report the average performance of these single tasks.
SMILES strings + CNN, RNN, and TRSF. We adopt the 1D CNNs from a recent study [32], which include single ID convolutional layer with a step size equal to 1, followed by a fully connected layer. As for the RNN, we use a 3-layer bidirectional gated recurrent units (GRUs) [TO] with 256 hidden vector dimensions. Additionally, we use the pre- pre-trained SMILES transformer (25] with 4 basic blocks and each block has 4-head attentions with 256 embedding dimensions and 2 linear layers. The SMILES are split into symbols (e.g., 'Br', 'C", '=', 'C., (2) and then fed into the transformer together with the positional encoding 67
2D Graphs - GCN, MPNN, GAT, and AFP. As in previous studies 76 we exhaustively utilized all readily available atom/bond features in our 2D graph-based descriptors. Specifically, we have incorporated 9 atom features, including atom symbol, degree, and formal charge, using a one-hot encoding scheme. In addition. we included bond features, such as type. conjugation, ring, and stereo. The resulting encoded graphs were then fed into GCN, MPNN. GAT, and AFP models. Further details on the graph descriptors used in our experiments can be found in (76).
3D Graphs + SPN. We employ the recently proposed SphereNet 38 for molecules with 31) geometry. Specifically, for quantum mechanics datasets (QM7, QM8, and QM9) that contain 31) atomic coordinates calculated with ab initio Density Functional Theory (DFT), we feed them into SphereNet directly. For other datasets without labeled conformations, we used RDKit [35 -generatd conformations to satisfy the request of SphereNet.
Datasets splits, evaluation protocols and metrics, hyper- prper-parameters tuning. Firstly, we randomly split the training, validation, and test sets at a ratio of 8: 1:1. And then, we tune the hyper-parametee based the performance of the validation set. Due to the heavy computational overhead, GNNs- based models on the HIV and MUV datasets are in 30 evaluations; all the models on the QM7 and QM8 in 10 evaluations; all the models the QM9 dataset in one evaluation. And then, we conduct 50 independent runs with different random seeds for dataset splitting to obtain more reliable results, using the optimal saper-parameters determined before. Following MoleculeNet benchmark (72), we evaluate the classification tasks using the area under the receiver operating characteristic curve (AUC-ROC), except the area under the precision curve (AUC-PRC) on MUV dataset due to its extreme biased data distribution. The performance on the regression task are reported using root mean square error (RMSE) or mean absolute error (MAE). Kindly note that we report the average performance across multi-tasks on some datasets because they contain more than one task. Additionally, avoid the overfitting issue, all the deep models are trained with an early stopping scheme if no validation performance improvement is observed in successive 50 epochs. We set the maximal epoch as 300 and the batch-size as 128. We provide more details including eopeerparammters tuning space for each model in the appendix.
3.2 Observations
Table I documents the benchmark results for various models and datasets, from which we can make the following Observations:
Observation I. Deep models underperform non-deep counterparts in most cases.
As can be observed in Table [ non-deep models rank the top one 11/15 datasets. Kindly note that we report the results of each task in the QM9 dataset in the appendix. On some datasets such as MUV, QM7, and BACE, three non-deep models can even beat any deep models.
Observation 2. The failure of deep models should not be solely attributed to the limited size of molecular datasets.
pointed out that the small size of molecular datasets is a bottleneck for deep learning models and propose various strategies accordingly 44 48 Here, we complement this pre-dominant belief with a new opinion with empirical evidence. As shown in Table I all the non-deep models can outperform any deep ones on some larger-scale datasets (e.g..
Intuitively, many previous works 21
79 59

---

Table 1: The comparison of representative models on multiple molecular datasets. The standard deviations can be seen in the appendix for the limited space. No.: Number of the molecules in the datasets. The top-3 performances on each dataset are highlighted with the grey background. The best performance is highlighted with bold. Kindly note that "TRSF" denotes the transformer that has been
pre-trained on 861, 000 molecular SMILES strings. The results on QM can be seen in the appendix.
Dataset (No.)
Metric
SVM
XGB
RF
CNN
RNN
TRSW
MLP
GCN
MPNN
GAT
AFP
SPN
BACE (1,513)
AUC_ROC 0.886
0.896
0.890
0.815
0.559
0.835
0.887
0.880
0.846
0.886
0.879
0.882
HIV (40,748)
AUC_ROC ROC 0.817
0.823
0.826
0.733
0.639
0.748
0.791
0.834
0.814
0.812
0.819
0.818
BBBP (2.035)
AUC_ROC 0,913
0.926
0.923
0,760
0.693
0.897
0.918
0.915
0.872
0.902
0.893
0.905
CliaTox (1.475)
AUC_ AUC_ROC 0.879
0.919
0,933
0.685
0.813
0.963
0.890
0.889
0.868
0.891
0.907
0,912
SIDER (1.366)
AUC ROC 0.626
0.638
0.644
0.591
0,515
0.641
0.617
0.633
0.603
0.614
0.620
0.613
Tox21 (7,811)
AUC_ROC 0.820
0.837
0.838
0,766
0.734
0.817
0.834
0.830
0,816
0.829
0.845
0.827
ToxCast (8,539)
AUC_ 0.725
0.785
0.778
0.735
0.74
0.780
0.781
0.767
0.736
0.768
0.788
0.772
MUV (93,087)
AUC_PRC 0.093
0.072
0.069
0.045
0.094
0.059
0.018
0.056
0.019
0.055
0.044
0.058
SARS-CoV-2 (14.332)
AUC_ROC
0.599
0.700
0.686
0.688
0.649
0.643
0.638
0.646
0.640
0.683
0.651
0.663
ESOL (1,127)
RMSE
0.676
0,583
0.647
2.569
1,511
0.718
0.653
0,773
0.695
0.661
0,594
0,671
Lipop (4,200)
RMSE
0.683
0.585
0.626
1.016
1.207
0,947
0.633
0.665
0.669
0.680
0.664
0.630
FreeSolv (639)
RMSE
1.063
0.715
1.014
2.275
2,205
1.504
1.046
1,316
1.327
1.304
1.139
1.159
QM7 (6,830)
MAE
42.814
52.726
51.403
81.165
158.
64.363
86.060
64.530
107.013
78.217
59.973
55.727
QM8 (21,786)
0.0364
0.0126
0.0099
0.0205
0.0295
0.0232
0.0104
0.0154
0.0109
0.0187
0.0098
0.0103
0.9
120
0.8
100
0.7
0.7
0.6
0.5
MIE
0.6
0.4
0.5
0.3
0.2
20
0.4
0.1
0-smooth
10-smooth
20-smooth
0-smooth
10-smooth
20-smooth
0-smooth
10-smooth
20-smooth
Figure 2: The performance of various models on the smoothed datasets. Left: ESOL (Regression); Middle: Lipop (Regression); Right: QM7 (Regression). Kindly note that we only smooth the regression datasets because the labels of classification datasets not suitable for smoothing.
MUV and QM 7). However, in some small datasets (e.g., ClinTox and ESOL), some deep models can beat partial non-deep ones. Therefore, we argue that there could be other reasons for the failure of deep models, not solely the dataset size. We will provide further analysis in Sec. 4
Observation 3. XGB and RF exhibit particular advantage over other models. In the experiments shown in Table [] we can see that the XGB and RF models consistently rank among the top three on each dataset. Additionally, tree models rank as the top one on 8/15 datasets. Next, we will explore why tree models are well-suited for molecular fingerprints in Sec. 4
4 Empirical Study: Why above phenomena would occur?
In this section, we attempt to understand which characteristics of molecular data lead to the failure of powerful deep models. Also, we aim to understand the inductive biases of XGB and RF that make them well-suited for molecules, and how they differ from the inductive biases of deep models. The details of the experiments in this section can be found in the appendix.
Explanation 1. Unlike image data, molecular data patterns are non-smooth. Deep models struggle to learn non-smooth target functions that map molecules to properties. We design two experiments to verify the above explanation, i.c., increasing decreasing the level
5

---

Table 2: RMSE, are the prediction RMSE on cliff molecules. respectively. AP is the predicted bioactivity change rate when transitioning from non-cliff to cliff molecules with subtle structural changes. The top-3 performances on each dataset are highlighted with the grey background. The best performance is highlighted with bold.
Target name
Metric
SVM
XGB
RF
CNN
RNN
TRSF
MLP
GCN MPNN
GAT
AFP
(Response type)
CBI
RMSE.
0.773
0.767
0.770
0.944
0.823
0.888
0.807
0.992
0.989
0.975
0.967
(Agonism FCpo)
AP
15.04%
21.13%
20.76%
2.07%
10.15%
9.42%
13.32%
4.13%
3.85%
1.17%
4.35%
DAT
RMSE.
0.744
0.696
0.730
0.894
0.783
0.934
0.792
1.003
0.921
1.042
0.995
(Inhibition K,)
AP
20.64%
23.03%
23.95%
2.73%
14.18%
9.34%
15.02%
5.83%
5.15%
2.27%
5.08%
PPARa
RMSE.
0.671
0.678
0.685
0.962
0.825
0.968
0.713
0.870
0.872
0,929
0.823
(Agonism ECso)
AP
21.07%
22.93%
23.14%
11.29%
13.48%
18.39%
15.29%
1.83%
5.18%
4.93%
11.73%
DOR
RMSE,
0.861
0.854
0.836
1.098
1.036
1.032
0.874
1,259
1.152
1.281
1.179
(Inhibition K.)
AP
25.26%
28.41%
23.95%
10.02%
9.83%
10.25%
15.18%
9.77%
12.52%
11.36%
13.11%
of data smoothing in the molecular datasets. Firstly, we 'increase" the molecular data smoothing level by smoothing the labels based on similarities between molecules. Specifically, let D denote the molecular dataset and (Iz, E DD be i-th molecule and its label, we smooth the target function as follows,
EnSeN,, s(1; f1 = ExSEN., s()),
where -(:-)) denotes the Tanimoto coefficient of the extended connectivity fingerprints (ECFP) between two molecules that can considered as their structural similarity. N₂,, is the k-nearest neighbor set of z; (including x;)) picked from the whole dataset based on the structural similarities. Di denotes the label after smoothing. We smooth all the molecules in the dataset in this way and use the smoothed label in to train the models. The results are shown in Figure 2 where 'O-smooth' denotes the original datasets. '10-smooth' and '20-smooth' mean h = 10 and observed, the performance of deep models improves dramatically the level of dataset smoothing increases, and many deep models including MLP, GCN, and AFP can even beat non-deep ones after smoothing. These phenomena indicate that deep models are more suitable for the smoothed datasets.
20, respectively, As can be
Secondly, we 'decrease' the level of data smooth- ing using the concept of activity cliff 40 60) from chemistry, which means a situation where small changes in the chemical structure of drug lead to significant changes in its bioactivity. We provide an example activity cliff pairs in Figure Appar- ently, the target function of activity cliffs that map molecules to the activity values is less smoothing than normal molecular datasets. We then evaluate the models the activity cliff datasets [[66]. The test set contains molecules that are chemically sim- ilar to those in the training set but exhibit either large difference in bioactivity (cliff molecules) or similar bioactivity (non-cliff molecules). As shown in Table the non-deep models consistently outper- form deep ones on these activity cliff datasets. Furthermore, the deep models exhibit less significant prediction change rates compared to the non-deep ones. This observation suggests that deep models are indeed less sensitive to subtle structural changes compared to non-deep ones. Our explanation is consistent with the conclusions in deep learning theory [51], i.e., deep models struggle to learn high-frequency components of the target functions. However, traditional models such as XGB and RF can learn piece-wise target functions, and do not exhibit such bias. Our explorations uncover several promising avenues to enhance deep models' performance on molecules: smoothing the target functions improving deep models' ability to learn the non-smooth target functions.
K= 126
Figure 3: Examplary of Activity Cliffs (ACs) on the target named dopamine D3 receptor (D3R). K, means the bioactivity values. The figure is adapted from a previous work [66] with permission.

---

Explanation 2. Deep models undesirably mix different dimensions of molecular features, whereas tree models make decisions based on each dimension of the features separately.
SVN
XGB
MLP
GCN
MPNN
GAT
0.67
0.96
1.6
0.94
0.86
0.85
0.92
1.4
0.84
0.83
1.2
0.88
0.82
1.0
0.86
0.81
0.84
0.8
0.79
Original
Transformed
Original
Transformed
Original
Transformed
Figure 4: The performance of various models on the orthogonally transformed datasets. Left: FreeSolv (Regression); Middle: ClinTox (Classification): Right: Tox21 (Classification). Kindly note that we did not evaluate CNN, RNN, and TRSF on the transformed datasets because we cannot apply the orthogonal transformations to the input SMILES strings.
Typically, features in molecular data carry meanings individually. As we elaborated in Sec. 2.1 each dimension of molecular fingerprints often indicates whether certain substructure is present in the molecule; each dimension of nodes/edges features in molecular graph data indicates a specific characteristic of the atoms/bonds (e.g., atom/bond type, atom degree). To verify the above explanation, we mix the different dimensions of molecular features I2 E Rd using an orthogonal transformation before feeding them into various models,
- Q.
(2)
where Q Rdldd is the orthogonal matrix and f; is the molecular feature after transformation. Kindly note that the meaning of x; depends on the input molecular descriptors in the experiments. Specifically, for SVM, XGB, RF, and MLP, xi denotes the molecular fingerprints; for GNN models, t; can denote the atom features and bond features in the molecular graphs, i.e., we apply orthogonal transformations both the atom features and bond features. As can be observed in Figure 4 the performance of tree models deteriorates dramatically and falls behind most deep models after the orthogonal transformation. dimensions of r; according to the matrix-vector product rule. In other words, the molecular features after orthogonal transformation no longer carry meanings individually, accounting for the failure of tree models that make decisions based each dimension of the features separately. The learning style of models is more suitable for molecular data because only a handful of features (e.g., certain substructures) are most indicative of molecular properties [47]. On the other hand, the performance decreases of deep models are less significant, and most deep models can beat tree models after transformations. We explain this observation as follows. Without the loss of generality, we assume that a linear layer of deep models can map the original molecular feature o; to the label y/s
because each dimension of f is linear combination of all the
Y1 W + b,
where W and denote the parameter matrix and the bias term of the linear layer, respectively. And then, we aim to learn a new linear layer mapping the transformed model feature f; to label yo:
W S + W Or, + G,
Vi
where W and denote the parameter matrix and the bias term of the new linear layer, respectively. Apparently, to achieve the same results the original feature, we only have to learn W so that W QW because @- Q as an orthogonal matrix, and also i -= b. Therefore, applying the orthogonal transformation molecular features barely impacts the performance of deep models. The empirical results in Figure confirm this point although some performance changes are observable due to uncontrollable random factors.
7

---

5 Methodology
Although we have shown and explained the superiority of non-deep models on molecular data, deep models have numerous advantages over the traditional approaches: (i) GNNs can exploit the structura//geometric
inductive biases of 2D/3D molecular graphs, alleviating the manual efforts to capture the topology of the networks; (ii) The pre-trained representations with deep models are beneficial to various downstream tasks, showing promising values in drug discovery [75] 74, 9 73). Therefore, we aim empower the deep models beat the non-deep ones on MPP tasks including activity cliff cases.
As explained before, deep models struggle learn the non-smooth target functions of molecular data, a phenomenon referred to "spectral bias" in literature (51). To overcome such bias, prior works 44 [8 have experimentally found that heuristic sinusoidal mapping of the input features allows MLPs to learn the non-smooth target functions. However, these mapping methods would undesirably mix the original features. Please refer the appendix for detailed discussions due to the limited space. As remedy, we introduce a new method named Independent Feature Mapping (IFM) that embeds each dimension of molecular features separately before feeding them into models. Denoting a molecular feature as æ E R., we formulate IFM as,
fe [sin [v) cos(v)),
[2rq ................................ 2Tc22]
(5)
where || denotes the concatenation of two vectors, c = =[c1 C2, C.. ck] are the learnable parameters initialized from (0, o) and fz E R2kxd We study the influence of the hyperparameters I: and in the appendix. Since cos(a (a- b) = cos a cos + sin asn b we have,
Sa Sz cos cos(2 - x')) = go(l - x'),
where is the dot product and x' is another molecular feature. Therefore, IFM can map data points to a vector space so that their dot product achieves a certain distance metric, which is an expected characteristic for feature mapping methods [52] 4 23) In what follows, will provide theoretical
justifications on the effectiveness of our IFM following a previous study [62]. As revealed in previous works, deep models Specifically, let / be a fully-connected deep network with weights initialized from a Gaussian distribution N, the NTK theory shows that the width of the layers in / becomes infinite and the learning rate for stochastic gradient descent (SGD) approaches zero, the function (x; 0) converges during training to the kernel regression solution using the neural tangent kernel (NTK), which is:
be approximated with Neural Tangent Kernel (NTK) 28
6 36 62)
bNTK(T x') = EE-~~ <
[1((;; 0)
(1('
00
00
When the inputs are limited to a hypersphere, the NTK for MLP can be expressed dot product kernel (a kernel in the form hYTK( : x²) for a scalar function hyTk IR - R). In our cases, the input to the deep models would be fa. the composed kernel of IFM and NTK can be formulated as,
bNTK (fz f2² = hNTK (ge (x x (hNTk o ge) -²),
(8)
thus, training deep models on these mapped molecular features corresponds to kernel regression with the stationary composed NTK function WNTK o Je- Considering that the parameters IFM creates a composed NTK that is not only stationary but also tunable. It enables us to dramatically control the range of frequencies that can be learned via manipulating the parameters
tunable,
6 Experiments
6.1 Experimental Settings
In our experiments, we equip various deep models with IFM. Specifically, for MLPs with fingerprints as inputs, we employ the proposed feature mapping method to the fingerprints (after feature selection and standardization) directly; For molecular graphs, we map both the features of atoms and bonds before feeding them into the GNNs. The other settings are the same as the benchmarking experiments. If a deep model named 'z' (e.g.. MLP) is equipped with IFM, we re-name it as 'IFM-z' (e.g.. IFM-MLP) our results. Also, we evaluate the non-deep models equipped with IFM in the appendix.
8

---

Table 3: The performance comparison on multiple molecular datasets. The best performance on each dataset is highlighted with bold. The 'P-Best (Model)' denotes the best result in Table I and its corresponding model name. The results of 12 tasks on QM 9 can be seen in the appendix.
Datuset (No.)
Metrie
MLP
GCN
MPNN
GAT
AFP
P-Best (Medel) IFM-MLP IFM-GCN IFM-MPNN IEM-GAT IFM-AFP
BACH(11,51)
AUC
AUC_ROC ROC 0.887
0.880
0.846
01886
0.579
(XGB)
0.915
0.903
0.866
0.834
0.907
HIV (4,0748)
AUC_ROS RCC_ROC 0.791
0.834
0.814
0.812
0.819
0.334 (GCN)
0.816
0.862
0.8-66
0.838
0.849
BBAP (2,035)
AUC. RCC 0.918
0.915
0,872
0.902
0.893
0.926 (XGB)
0.937
0.945
0.908
0.933
0.940
ClinTox (1,475)
AUC: ROC 0,890
0.889
0.868
0.891
0.907
0.963 (TRSF)
0.941
0.938
0,929
0.953
0.959
SIDER (1.366)
AUC_ROS IIC_ROC 0.617
0.633
0.603
0.614
0.620
0.644 (RF)
0.646
0.647
0.652
Tox21 (7.811)
AUC_ROS
0.830
0.816
0.829
0.845
(AFP)
0.819
0.853
ToxCast (8,539)
AUC_ROC 0.781
0.767
0.736
0.768
0.758
0.788 (AFP)
0.795
0,790
0.772
0.797
0,806
MUV (93,087)
AUC_ PRC 0,018
0.056
0.019
0,055
0.044
0.093 (SVM)
0.052
0.113
0.068
0.124
0.097
SARS-CoV-2 (14,352)
AUC_ROC RCC 0.638
0.541
0.6-10
0,683
0.651
0,700 (XGB)
0.675
0.682
0.686
0.716
0,704
ESOL (1.127)
RMSE
0.653
0.773
0.695
0.661
0.594
0.583 (XGB)
0.587
0.728
0.673
0.566
0,561
Lipop (4,200)
RMSE
0.633
0.665
0.669
0.680
0.664
0,585 (XGB)
0.556
0.577
0.568
0.584
0.578
FredSalv 6639)
RMSE
1,016
1,316
1.327
1.304
1.139
0.715 (XGB)
0.862
0.916
0,911
0.908
0.883
QM7 (6.830)
MAE
86.060
61.530
107.013
78.217
59.973
(SVM)
66.570
38.793
84,918
59.595
33.775
QM8 (21,786)
MAE
0.8104
0,0154
0,01099
0,0187
0.0198
0.0098 (AFP)
0.0091
0.0114
0.00885
0.0139
0.0079
Table 4: The results on activity cliff datasets. The best result for each dataset is highlighted in 'bold'.
Turget (Response type) Metric MLP GCN MPNN GAT AFP P-Best (Model) IFM-MLP IFM-GCN IFM-MPNN IFM-GAT IFM-AFP
CBI (Agonism EC:o)
RMSE, 0.807 0.992 0.989 0.975 0.967
0.767 (XGB)
0.748
0.741
0.746
DAT (lohibition K,)
RMSE, 0.792 1,003 0.911 1,042
0.666 (XGR)
0.646
0.682
0.673
0.665
0.670
PPARa (Aganism EC5a)
RMSE, 0.713 0,870 0,872
0.929
0.823
0.671 (SVM)
0.623
0.634
0.649
0,661
0.616
DOR (Iubibition K.)
RMSE. 0.874 1,259 1.152
1,281
1,179
0,836 (RF)
0,787
0,813
0.796
0,799
0,810
6.2 Results
Main results. We show the main results in Table 3 from which we can make the following observations: (1) The proposed feature mapping method can significantly improve the performance of the deep models molecular datasets; (2) The deep models equipped with the feature mapping method can beat non-deep counterparts in most cases, verifying the effectiveness of our method.
Results on activity cliff's. We also employ the proposed feature mapping method on activity cliff datasets where the target functions are less smooth. The results shown in Table f indicate that IFM improves the deep models by significant margins. Moreover, nearly all the deep models with IFM method can beat traditional methods, confirming that our method can help neural networks learn non-smooth target functions.
Table 5: Comparisons with SM (Sinusoidal Map- Table 6: Pre-training on molecules with encoders equipped with IFM. GIN: Graph Isomorphism Network [78].
ping 44 and GM (Gaussian Mapping 62).
Methods
BACE
HIV
BBBP
ESOL
Lipop
FreeSolv
Mothod (Encoder)
Tox21
ToxCast
Sider
ClisTox
BBBP
Bice
MIP+ SM
8890
0.798
0.925
0.640
0.613
1,833
AmMMsking (GUN)
0.745
0.626
0.598
0.55
MLP & GM
0.896
0,806
0.923
0.625
0.591
0.992
AuzMasking (LFM GCN)
0.758
0.639
0.612
0.741
IFM-MLP
0.915
0.16
0.937
0.587
0.556
0.862
MGSSL 83 (INN)
0.752
0.633
0616
0.771
0.88
GCN SM
0.894
0.762
1.204
MGSSL MI GIN)
0.764
0.639
0.623
0.783
0.704
0.796
GCN + GM
0.892
0.847
0.926
0.766
0.609
1.193
Mole DERT
26 (GIN)
0.568
0.643
0.628
0.789
0.719
0.808
JFM-GCN
0.903
0.862
0.915
0.728
0,577
0.916
Mole-RERT (IMM-GIN)
0.775
0.649
0.627
0.796
0.724
0.813
Comparisons with other feature mapping methods (ablation study). We compare the proposed IFM with the previous feature mapping methods. The results shown in Table 5 indicate that the proposed feature mapping method is superior to the previous method, which verifies that mixing different dimensions of molecular features as GM 62 would degrade the performance.
Pre-training molecules with IFM. Compared with non-deep models, deep ones can be com- bined with the prevalent 'Pretraining and Finetuning" paradigm to exploit large-scale unlabeled molecules [75]. This motivates us to develop more powerful neural encoders for this paradigm. Specifically, we pre-train the deep models equipped with IFM and report the fine-tuning results in T Our experimental settings are the same the pioneering work (27 where the datasets split with scaffold splitting, differing from the random splitting in Table 3 As can be observed, our method can boost various pre-training strategies to advance their performance in downstream tasks.
9

---

7 Discussion and Conclusion
In this paper, we perform comprehensive benchmark of representative models on molecular property prediction. Our results reveal that traditional machine learning models, especially tree models, can easily outperform well-designed deep models in most cases. These phenomena can attributed the unique patterns of molecular data and different inductive biases of various models. Specifically, the target function mapping molecules to properties are non-smooth, and some small changes can incur significant property variance. Deep models struggle to learn such patterns. Additionally, molecular features carry meanings individually and deep models would undesirably mix different dimensions of molecular features. These findings stimulate us develop simple-yct-effeettee method for molecule data that can help deep models learn non-smooth target functions with theoretical guarantees. Extensive experiments verify the effectiveness of the proposed method. Our study leaves an open question for future research: Can our findings and methods be generalized to other AIDD tasks including wrug-target interactions (DTIs) prediction, drug-drug interactions (DDIs) prediction?
feature mapping
Acknowledgements
We thank the anonymous the area chairs and reviewers for their constructive and helpful reviews. This work was supported by the National Key R&D Program of China (Project 20222I011 15100), the National Natural Science Foundation of China (Project U21A20427), the Research Center for Industries of the Future (Project WU2022C043), and the Competitive Research Fund (Project WU2022A009) from the Westlake Center for Synthetic Biology and Integrated Bioengineering.
References
[1] Tox21 challenge. https:/trrooddnii.gg//////lllllf 2017.
[2] Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. ICML, 2019.
[3] Kenneth Atz, Francesca Grisoni, et al. Geometric deep learning molecular representations. Nature Machine Intelligence, 2021.
[4] Haim Avron, Vikas Sindhwani. Jiyan Yang, and Michael W. Mahoney, Quasi-monte carlo feature maps for shift-invariant kernels. J. Mach. Learn. Res., 17: 17:12011-220:1: 2014.
[5] Seth D Axen, Xi-Ping Huang, Elena L Cáceres, Leo Gendelev, Bryan L Roth, and Michael Keiser. A simple representation of dimensiime molecular structure. Journal of medicinal chemistry, 60(11):7393-740) 2017.
[6] Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritch- man. Frequency bias in neural networks for input of non-uniform density, arXiv preprint arXiv:2003.04566 2020.
[7] Kyle R. Brimacombe, Tongan Zhao, Richard T. Eastman, Xin Hu, Ke Wang, Mark Backus, Bolormaa Baljinnyam, Catherine Z. Chen, Lu Chen, Tara Ficher, Marc Ferrer, Ying Fu, Kirill Gorshkov, Hui Guo, Quinlin M. Hanson, Zina Itkin, Stephen C. Kales, Carleen Slumpp-Thoman Emily M. Lee, Sam Michael, Tim Mierzwa, Andrew Patt, Manisha Pradhan, Alex Renn, Paul Shinn, Jonathan H. Shrimp, Amit Viraktamath, Kelli M. Wilson, Miao Xu, Alexey V. Zakharov, Wei Zhu, Wei Zheng, Anton Simeonov, Ewy A. Mathé, Donald C. Lo, Matthew D. Hall, and Min Shen. An opendata portal share covid- 19 drug repurposing data in real time. bioRxiv, 2020.
[8] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd aem sigkdd international conference knowledge discovery and data mining, pages 785-794, 2016.
[9] Seyone Chithrananda, Gabriel Grand, et ChemBERTa: Large-Scale Sel--Superviee Pre- training for Molecular Property Prediction. arXiv.org, October 2020.
10

---

[10] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rmn encoder- decoder for statistical machine translation. arXiv preprint 1406. 1078, 2014.
[11] Connor W Coley, Regina Barzilay, et al. Convolutional embedding of attributed molecular graphs for physical property prediction. Journal of chemical information and modeling. 2017.
[12] John S Delaney. Esol: estimating aqueous solubility directly from molecular structure. Journal of chemical information and computer sciences, 2004.
[13] Jianyuan Deng, Zhibo Yang, Hehe Wang. Iwao Ojima, Dimitris Samaras, and Fusheng Wang. Taking a respite from representation learning for molecular property prediction. arXiv preprint 2022.
[14] Weitao Du, He Zhang, et al. SE(3) Equivariant Graph Neural Networks with Complete Local Frames. In ICMI,, 2022.
[15] David Duvenaud, Dougal Maclaurin, and others. Convolutional networks on graphs for learning molecular fingerprints. NIPS, 2015.
[16] Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 34:6790-6802, 2021.
[17] Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional message passing for molecular graphs. In International Conference on Learning Representations, 2020.
[18] Anna Gaulton, Louisa J. Bellis, et al. ChEMBL: A Large-Scale Bioactivity Database for Drug Discovery. Nucleic Acids Res., 2012.
[19] Kaitlyn M Gayvert, Neel Madhukar, and Olivier Elemento. A data-driven approach to predicting successes and failures of clinical trials. Cell chemical biology, 2016.
[20] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning, pages 1263-1272, 2017.
[21] Garrett B Goh, Nathan O Hodas, and Abhinav Vishnu. Deep learning for computational chemistry. Journal computational chemistry, 38 12916::121113 2017.
[22] Jiuxiang Gu, Zhenhua Wang, Jason Kuen, Lianyang Ma, Amir Shahroudy, Bing Shuai, Ting Liu, Xingxing Wang, Gang Wang, Jianfci Cai, et al. Recent advances in convolutional neural networks. Pattern recognition, 77:354 07744-37,222 2018.
[23] Raffay Hamid, Ying Xiao, Alex Gittens, and Dennis DeCoste. Compact random feature maps. In International Conference Machine Learning, 2013.
[24] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning large graphs. Advances in neural information processing systems, 30, 2017.
(25) Shion Honda, Shoi Shi, and Hiroki R Ueda. Smiles transformer: Pre-trained molecular fingerprint for low data drug discovery. arXiv preprint arXiv: 1tXiv:1911.047:: 2019.
[26] Zhenyu Hou, Xiao Liu, et al. GraphMAE: Supervieed Masked Graph Autoencoders. In KDD, 2022.
[27] Weihua Hu, Bowen Liu, and others. Strategies for pre-training graph neural networks. ICLR, 2020.
[28] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural Tangent Kernel: Convergence and generalization in neural networks. NeurIPS, 2018.
[29] Tiago Janela and Jürgen Bajorath. Simple nearest-neighhour analysis meets the accuracy of compound potency predictions using complex machine learning models. Nature Machine Intelligence, pages 1-10, 2022.

---

[30] Dejun Jiang, Zhenxing Wu, Chang- Yu Hsich, Guangyong Chen, Ben Liao, Zhe Wang, Chao Shen, Dongsheng Cao, Jian Wu, and Tingjun Hou. Could graph neural networks learn better molecular representation for drug discovery? comparison study of descriptor-based and graph-based models. Journal of cheminformatics, 13(1):1-23,20) 2021.
[31] Steven Kearnes, Kevin McCloskey, and others. Molecular graph convolutions: Moving beyond fingerprints. J Comput Aid Mol Des, 2016.
[32] Talia B Kimber, Maxime Gagnebin, and Andrea Volkamer. Maxsmi: maximizing molecular property prediction performance with confidence estimation using smiles augmentation and deep learning. Artificial Intelligence in the Life Sciences, 1:100014, 2021.
[33] N. Thomas Kipf. and M. Welling. Semi-supervised classification with graph convolutional networks. ICLR, 2017.
[34] Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alan Aspuru-Guzia Self-referencing embedded strings (selfies): 100% robust molecular string representation. Machine Learning: Science and Technology, 1(4):045024, 2020.
[35] Greg Landrum. RDKit: software suite for cheminformatics, computational chemistry, and predictive modeling, 2013.
[36] Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. NeurlPS, 2019.
[37] Yi Liu, Limei Wang, et Spherical message passing for 3d molecular graphs. Iclr, 2021.
[38] Yi Liu, Limei Wang, Meng Liu, Yuchao Lin, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d molecular graphs. In International Conference Learning Representations (ICLR), 2022.
[39] Chengqiang Lu, Qi Liu, Chao Wang, Zhenya Huang, Peize Lin, and Lixin He. Molecular property prediction: multilevel quantum interactions modeling perspective. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 1052- 1060, 2019.
[40] Gerald M Maggiora. On outliers and activity cliffs why qsar often disappoints, 2006.
[41] Andreas Mayr, Günter Klambauer, Thomas Unterthiner, Marvin Steijaert, Jörg K. Wegner, Hugo Ceulemans, Djork-Arné Clevert, and Sepp Hochreiter. Large-scale comparison of machine learning methods for drug target prediction on chembl. Chemical science, 9(24):54415541,, 2018.
[42] Eukasz Maziarka, Tomasz Danel, Slawomir Mucha, Krzysztof Rataj, Jacek Tabor, and Stanislaw Jastrzębski. Molecule attention transformer. arXiv preprint arXiv:2002.08264, 2020.
[43] Larry Medsker and Lakhmi C Jain. Recurrent neural networks: design and applications. CRC press, 1999.
[44] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[45] Erxue Min, Runfa Chen, Yatao Bian, Tingyang Xu, Kangfei Zhao, Wenbing Huang, Peilin Zhao, Junzhou Huang. Sophia Ananiadou, and Yu Rong. Transformer for graphs: An overview from architecture perspective. arXiv preprint arXiv:2202.08445. 2022.
[46] Wim De Mulder, Steven Bethard, and Marie- Francine Moens. A survey on the application of recurrent neural networks to statistical language modeling. Comput. Speech Lang., 30:61-98, 2015.
[47] Christopher W Murray and David C Rees. The rise of fragment-based drug discovery. Nature chemistry, 13)) 2009.
12

---

[48] Cuong Q Nguyen, Constantine Kreatsoulas, and Kim M Branson. Meta-learning gnn ini- tializations for low-resource molecular property prediction. arXiv preprint arXiv:2003.05996 2020.
[49] Aneesh Pappu and Brooks Paige. Making graph neural networks worth it for low-data molecular machine learning. arXiv preprint arXiv:201 12203, 2020.
[50] Lagnajit Pattanaik and Connor W Coley. Molecular representation: going long on fingerprints. Chem, 6(6):1204-1207 2020.
[51] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301-5310. PMLR, 2019.
[52] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In NIPS, 2007.
[53] David Rogers and Mathew. Hahn. Extended-connecttvvty fingerprints. J chem inf, 2010.
[54] Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang. and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems, 33:12559-12571, 2020.
[55] Jerret Ross, Brian Belgodere, et al. Molformer: Large Scale Chemical Language Representations Capture Molecular Structure and Properties. Nat. Mach. Intell., 2022.
[56] Victor Garcia Satorras, Emiel Hoogeboom, et al. E(n) Equivariant Graph Neural Networks. In ICML,, 2021.
[57] T. K. Schuett, Kindermans, and others. Schnet: A continuous -ilter convolutional neural network for modeling quantum interactions. NIPS, 2017.
[58] Ying Song, Shuangjia Zheng, Zhangming Niu, Zhang- Hua Fu, Yutong Lu, and Yuedong Yang. Communicative representation learning on attributed molecular graphs. In IJAAI volume 2020, 2020.
pages
[59] Megan Stanley, John F Bronskill, Krzysztof Maziarz, Hubert Misztela, Jessica Lanini, Marwin Segler, Nadine Schneider, and Mare Brockschmidt. FS-mol: few-shot learning dataset of molecules. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021.
[60] Dagmar Stumpfe and Jurgen Bajorath. Exploring activity cliffs in medicinal chemistry: miniper- spective. Journal of medicinal chemistry, 55(7):2932-2942 29422 2012.
[61] Vladimir Svetnik, Andy Liaw, Christopher Tong. J Christopher Culberson, Robert P Sheridan, and Bradley P Feuston. Random forest: classification and regression tool for compound classification and qsar modeling. Journal of chemical information and computer sciences, 1947-11947 2003.
[62] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. ArXiv, abs/2006.107:: 2020.
[63] Hao Tian, Rajas Ketkar, and Peng Tao. Admetboost: web server for accurate admet prediction. Journal of Molecular Modeling, 28(12):1-6, 2022.
[64] Roberto Todeschini and Viviana Consonni. Molecular descriptors. Recent Advances in QSAR Studies, pages 29-102, 2010.
[65] Cecile Valsecchi, Magda Collarile, Francesca Grisoni, Roberto Todeschini, Davide Ballabio, and Viviana Consonni. Predicting molecular activity on nuclear receptors by multitask neural networks. Journal of Chemometries, 36(2): 2022.
13

---

[66] Derek van Tilborg, Alisa Alenicheva, and Francesca Grisoni. Exposing the limitations of molecular machine learning with activity cliffs. Journal of Chemical Information and Modeling. (((23):5938-5511 2022.
[67] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[68] Petar Velickovic, Guillem Cucurull, al. Graph Attention Networks. In ICLR, 2018.
[69] Sheng Wang, Yuzhi Guo, and others. Smiles-bert large scale unsupervised pre-training for molecular property prediction. BCB, 2019.
[70] Yanli Wang, Stephen H. Bryant, et al. Pubchem Bioassay: 2017 Update. Nucleic Acids Res., 2017.
[71] David Weininger, Arthur Weininger, and others. Smiles. 2. algorithm for generation of unique smiles notation. CHEM INF MODEL, 1989.
[72] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9(2):113-330, 2018
[73] Jun Xia, Lirong Wu, Jintao Chen, Bozhen Hu, and Stan Z. Li. SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation. In Proceedings of The Web Conference 2022. Association for Computing Machinery, 2022.
[74] Jun Xia, Chengshuai Zhao, et al. Mole-BERT: Rethinking Pre-training Graph Neural Networks for Molecules. In ICLR, 2023.
[75] Jun Xia, Yanqiao Zhu, Yuanqi Du, Yue Liu, and Stan Li. A systematic survey of chemical pre-trained models. [JCAL, 2023.
[76] Zhaoping Xiong, Dingyan Wang, Xiaohong Liu, Feisheng Zhong, Xiaozhe Wan, Xutong Li, Zhaojun Li, Xiaomin Luo, Kaixian Chen, Hualiang Jiang, et al. Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism. Journal of medicinal chemistry, 63(16):8749-87660 2019.
[77] Zhaoping Xiong, Dingyan Wang, and others. Pushing the boundaries of molecular representation for drug discovery with graph attention mechanism. J Med Chem, 2020.
[78] Keyulu Xu, Weihua Hu, et al. How Powerful are Graph Neural Networks? In ICLR, 2019.
[79] Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et al. Analyzing learned molec- ular representations for property prediction. Journal of chemical information and modeling, 59(8)33370-3388 2019.
[80] Chun Wei Yap. deser-de:::,ptor:: An open source software to calculate molecular descriptors and fingerprints. Journal of computational chemistry, 32(7): (177):466611777 1474, 2011.
[81] Chengxuan Ying, Tianle Cai, et al. Do Transformers Really Perform Badly for Graph Represen- tation? In NeurlPS, 2021.
[82] Atakan Yüksel, Erva Ulusoy, Atabey Ünlü, Gamze Deniz, and Tunca Dogan. Selformer: Molec- ular representation learning via selfies language models. arXiv preprint arXiv:2304.04662. 2023.
[83] Matthew D. Zeiler. Adadelta: An adaptive learning rate method. ArXiv, abs//212.5701, 2012.
[84] Vladimir Zernov, Konstantin V Balakin, Andrey Ivaschenko, Nikolay P Savchuk, and Igor V Pletnev. Drug discovery using support vector machines. the case studies of drug-likeness, agrocheemicalllkknnss,, and enzyme inhibition predictions. Journal of chemical information and computer sciences, ((66))244-2055 2003.
14

---

[85] Zaixi Zhang. Qi Liu, and others. Motif- graph self- self-supprvised learning for molecular property prediction. NeurIPS, 2021.
[86] Shuangjia Zheng, Xin Yan, Yuedong Yang, Jun Xu. Identifying structure-property rela- tionships through smiles syntax analysis with self-attention mechanism. Journal of chemical information and modeling, 59(2)914-923, 2019.
[87] Ellen D. Zhong, Tristan Bepler, Joseph Davis, and Bonnie Berger. Reconstructing continuous distributions of 3d protein structure from cryo-em images. In International Conference Learning Representations, 2019.
A Datasets
Table 7: Summary for the molecule datasets in the benchmark.
Dataset
Task
# Tasks
Molecules
BBBP
Classification
1
2,039
Tox21
Classification
7,831
ToxCast
Classification
617
8,576
Sider
Classification
27
1,427
ClinTox
Classification
2
1,478
MUV
Classification
93,087
HIV
Classification
41.127
Bace
Classification
1,513
SARS-CoV-2
Classification
14,332 13
Lipop
Regression
4,200
FreeSolv
Regression
642
ESOL
Regression
1,1128
QM7
Regression
7.160
QM8
Regression
16
21,786
QM9
Regression
12
133,885
In this section, we provide detailed information on the molecular datasets used for downstream tasks. Molecular Property: Pharmacology The Blood-Brain Barrier Penetration (BBBP) dataset mea- sures whether molecule will penetrate the central nervous system. All three datasets, Tox21 []], ToxCast 72), and Clin'ox [9 are related to the toxicity of molecular compounds. The Side Effect marketed drug database.
Resource (SIDER) dataset stores the adverse drug reactions
Molecular Property: Physical Chemistry Dataset proposed in [12 measures aqucous solubility of the molecular compounds. Lipophilicity (Lipo) dataset is subset of ChEMBL [18] measuring the molecule octanol/wate distribution coefficient. CEP dataset is subset of the Havard Clean Energy Project (CEP), which estimates the organic photovoltaic efficiency.
Molecular Property: Biophysics Maximum Unbiased Validation (MUV) another sub-database from PCBA, and is obtained by applying refined nearest neighbor analysis. HIV is from the Drug Therapeutics Program (DTP) AIDS Antiviral Screen, and aims at predicting inhibit HIV replication. BACE measures the binding results for set of inhibitors of f-secretase (BACE-1) and is gathered in MoleculeNet (72).
Molecular Property: Quantum Mechanics QM7, QM8 and QM9 contain stable organic molecules with to 7, or 9 heavy atoms. 3D atomic coordinates as well as electrical prop- erties of molecules were calculated with initio Density Functional Theory (DFT).
Molecular Property: COVID-19 SARS-CoV-2 [7] is a collection of datasets generated by screening a panel of SARS-CoV-2-related assays against approved drugs. 13 assays 14,332 drugs were used in our experiments.
Statistics of the datasets are in Table
"The datasets are available at https//opendaat..cco. www.s.nna..icccc... www..ho//oovil1// (CC BY 4,0 license) and are continu- ously extended. The data used in experiments downloaded February 16th, 2021.
15

---

B Results on QM9 dataset with 12 tasks
We show the results of each task of QM9 dataset in Table As can be observed, the proposed IFM module can consistently improve the performance base models and can even outperform the SPN with 3D geometry information as inputs in most tasks.
Table Comparison of MAE on QM9, The best result highlighted with 'bold'.
Target
GLUMO
Ac
R²
ZPVE
Uo
U
H
G
eV
eV
bohr
meV
meV
V
meV
cal/ /mollK
SVM
0.016
0.038
0.040
1.19
6.35
0.026
3.22
XGB
0.039
0.040
1.19
6.36
0.026
3.22
RF
0.016
0.038
0.039
1.19
6.34
0.020
3.22
CNN
0.018
0.019
0.032
1.11
1,55
99.27
0,073
3.52
4.09
3.81
4.03
1,08
RNN
0.014
0.028
0.028
3.53
03
0.015
14.83
14.93
15.66
14.84
1.47
TRSF
0.015
0.017
1.46
96.67
0.057
1.97
2.95
1.90
1.73
0.92
GCN
0.030
0.015
0.019
0.69
0.77
27.49
0.014
0.55
0.56
0.55
0.55
0.34
MPNN
0,100
0.440
0.410
0.95
1.47
38.96
0.410
2.15
2.19
2.16
2.18
0.93
GAT
0.060
0.012
0.85
24.85
0.057
0.79
0.81
0.80
0.43
AFP
0.089
0.029
0.073
0.74
27.35
0.110
1.20
1.18
1.19
0.51
SPN
0.024
0.019
0.032
0.31
0.47
29.26
0.011
0.63
0.73
0.64
0.82
0.20
IFM-MLP
0.013
0.022
0.032
1.04
125.47
0.015
16.19
16.23
16.20
16.22
0.97
IFM-GCN
0.019
0.011
0.014
0.63
0.53
15.18
0.012
0.46
0.43
0.47
0.44
0.25
IFM-MPNN
0.076
0.371
0.358
0.77
0.94
19.82
0.252
1.88
1.86
1.86
1.87
0.75
IFM-GAT
0.032
0.009
0.010
0.69
0.038
0.62
0.63
0.29
IFM-AFP
0.009
0.017
0.575
0.48
14.69
0.096
0.86
0.88
0.87
0.33
C The influence of the hyperparameters k and a in IFM
Table The influence of the hyper-paramette (with 6).
Models / k
4
8
16
32
64
Toxcast (IFM-MLP)
0.782
0.795
0.797
0.789
0.792
Toxcast (IFM-GCN)
0.771
0.780
0.783
0.785
0.774
Sider (IFM-MLP)
0.625
0.631
0.626
0.635
0.636
Sider (IFM-GCN)
0.635
0.638
0.639
0.633
0.632
Table 10: The influence of the hyper-parameter (with & 8).
Models /
12
15
Toxcast (IFM-MLP)
0.767
0.782
0.795
0.791
0.785
0.772
Toxcast (IFM-GCN)
0.763
0.775
0.780
0.790
0.788
0.776
Sider (IFM-MLP)
0.611
0.626
0.631
0.646
0.639
0.613
Sider (IFM-GCN)
0.620
0.633
0.638
0.649
0.643
0.627
I and a are hyper-parameters parameters in IFM and they are tuned for the optimal performance using the validation set. We show their tunning space and corresponding results in Table / and Table TO respectively. As can be observed, the hyper-paramnn parameerree & barely impacts the performance of IFM. Larger & would undesirably incur more computational overhead. In contrast, smaller larger o would degrade the performance of IFM dramatically. In other words, the hyper-parameter o is more important and we are encouraged to set larger tuning space a to pick the optimal value for each dataset using the validation set.
16

---

D Performance of Non-deep models (SVM, RE, and XGB) equipped with IFM
In Table 11 also show the results of non-deep models (SVM, RF, and XGB) equipped with IFM. As can be observed, IFM barely impacts the performance of non-deep models.
Table 11: Performance of Non-deep models (SVM, RF, and XGB) equipped with IFM
Models
Tox21
ToxCast
Sider
ClinTox
BBBP
Bace
SVM
0.820
0.725
0.626
0.879
0.913
0.886
IFM-SVM
0.822
0.724
0.629
0.883
0.907
0.883
RF 0.838
0.778
0.644
0.933
0,923
0,890
IFM-RF
0.843
0.785
0.643
0.935
0,929
0.883
XGB 0,837
0.785
0,638
0,919
0,926
0.896
IFM-XGB
0.833
0.780
0.636
0.911
0.933
0.894
E
The performance of CNN, RNN, and TRSF on smoothed molecular datasets
Due the limited space in the main text, report the results of CNN, RNN, and TRSF on smoothed molecular datasets in Table 12 here.
Table 12: The performance of CNN, RNN, and TRSF on smoothed molecular datasets.
ESOL
Lipop
FreeSalv
Smoothing level
0-smooth
10-smooth
20-smooth
0-smooth
10-smooth
20-smooth
0-smooth
10-smooth
20-smooth
CNN
2.569
1.937
1.334
1.016
0.524
0.409
2.275
1.592
0.986
RNN
1.511
1.04
0,862
1.207
0.770
0.687
2.205
1,570
1.028
TRSF
0.718
0.667
0,564
0.947
0.549
0.435
1.504
1.175
0.954
F Standard deviations of the benchmarking results and main results
For the limited space in the main text, report the standard deviations of the benchmarking results and main results here in Table [3 and Table 14 respectively.
G Discussions on the previous feature mapping methods
Gaussian Feature Mapping 62 (GM): Given the molccular feature as r E Rd. fz =
(cos(2xBx)) (in sin(2nBr))) where each entry in B E Rmxd is sampled from / (0, In experiments, we use an isotropic Gaussian distribution.
Sinusoidal Feature Mapping (SM): The general form of Sinusoidal Feature Mapping in previous
works 44 87 can be formulated as fa = [..., cos (2ma)//²a) sin (2mo5//²)) ,...] T for = O.... 1.
The superiority of IFM over SM and GM: different dimensions of molecular features, our IFM avoids this issue. Moreover, the parameters c in IFM is learnable while the parameters B are sampled from fixed Gaussian distribution. Additionally, the feature mapping in SM is deterministic and only contains on-axis frequencies, making it naturally biased towards data that has more frequency content along the axes. contrast, our IFM allows all directions to share the same frequency content.
The matrix-vector vector product in GM would undesirably mix
17

---

Table 13: The standard deviations of main results.
Dataset (No.)
Metric
IFM-MLP IEM-GCN IFM-MPNN IFM-GAT IEM-AFP
BACE (1,513)
AUC_ROC
0.024
0.028
0.022
0.023
0.025
HIV (40.748)
AUC_ROC
0.022
0,027
0.029
0,033
0.032
BBBP (2,035)
AUC_RO RCC
0.026
0.029
0.022
0.032
0.035
ClinTox (1,475)
AUC_ROC
0.049
0,042
0.040
0,047
0.046
SIDER (1,366)
AUC_ROO ROC
0.025
0.027
0.028
0.025
0.029
Tox21 (7,811)
AUC_ RCC
0.017
0.014
0.015
0.012
0.011
ToxCast (8,539)
AUC_ROC ROC
0.017
0.016
0.020
0.019
0.017
MUV (93,087)
AUC_RRC
0.028
0.032
0.018
0.019
0.023
SARS-CoV-2 (14,332)
AUC_ROC
0.027
0.025
0.020
0.018
0.023
ESOL (1,127)
RMSE
0.082
0,063
0.088
0.065
0.063
Lipop (4,200)
RMSE
0.036
0.073
0.051
0.038
0.046
FreeSoly (639)
RMSE
0.207
0.247
0.258
0.261
0.259
QM7 (6.830)
MAE
0.025
0.028
0.022
0.013
0.024
QM8 (21,786)
MAE
0,0002
0.0003
0.0012
0.0003
0.0003
Table 14: The standard deviations of benchmarking results.
Dataset (No.)
Metric
SVM
XGB
RF
CNN
RNN
TRSF
MLP
GCN
MPNN
GAT
AFP
SPN
BACE (1,513)
AUC_ROC 0.018
0.020
0.023
0,030
0.009
0.007
0,025
0,021
0,020
0,026
0.022
0.021
HIV (40.748)
AUC_ROC 0.019
0.020
0.017
0.009
0.029
0.011
0.019
0.024
0.029
0.033
0.028
0.030
BBBP (2,035)
AUC_ROC 0.029
0.025
0.027
0,026
0.019
0.014
0.028
0.026
0.032
0.034
0.030
0.033
ClinTox (1,475)
AUC_ROC 0.041
0,042
0.039
0.008
0.098
0.007
0.048
0,045
0,043
0.053
0,042
0.046
SIDER (1,366)
AUC_ROC: 0.020
0.019
0.017
0.012
0.015
0.009
0.026
0.027
0.024
0.029
0.025
0.024
Tox21 (7,811)
AUC_ROC 0.008
0.011
0.013
0,004
0.007
0.004
0.015
0.018
0.014
0.013
0,012
0.014
ToxCast (8,539)
AUC_ROC 0.007
0.005
0.007
0.006
0.010
0.005
0.018
0,017
0.020
0.021
0.018
0.019
MUV (93,087)
AUC_PRC 0.039
0.026
0.030
0.014
0.017
0.060
0.027
0.035
0.030
0.015
0.022
0.026
SARS-CoV-2 (14.332)
AUC_ROC 0.110
0,010
0.013
0.007
0.023
0.015
0.028
0,028
0.019
0.007
0,025
0.017
ESOL (1,127)
RMSE
0.049
0.051
0.073
0.107
0.043
0.048
0.089
0.057
0.092
0.068
0.067
0.073
Lipop (4,200)
RMSE
0.037
0.032
0.029
0.016
0.009
0.020
0.033
0.080
0.053
0.041
0.039
0.035
FreeSoly (639)
RMSE
0.165
0.180
0.236
0.117
0.114
0.098
0.212
0.255
0.263
0.269
0.271
0.259
QM7 (6,830)
MAE
0.110
0.010
0.013
0.023
0.033
0.035
0.028
0.028
0.019
0.007
0.025
0.027
QM8 (21,786)
MAE
0,0001
0.0002
0.0002
0.0004
0.0001
0.0004
0.0002
0.0005
0.0014
0.0004
0.0002
0.0003
H Hardware
For all our benchmarks and experiments, we use the GPUs including NVIDIA V100, and NVIDIA A100 GPUs based availability.
I Hyper-parameters tuning space
For SVM, we adopt the widely-used radial basis function (RBF) as the kernel and we optimize the hyper-parameters within the following ranges: C (0.1 100) and gamma values (0 to 0.2).
For XGB, we penalize the complexity of the traditional gradient boosting model and perform shrink- age and column subsampling to prevent over-fitting, Also, employ sparsity-aware split finding technique for efficient training on sparse data, etc. In the experiments, we tune the hyper-parametent within the following ranges: learning patein (0.01 to 0.2), gamma (0 to 0.2), min_child weight (1 to 6), subsample (0.7 to 1.0), colsample_byy bytree (0.7 to 1.0), max_ depth (3 to 10) and n estimators (50, 100, 200, 300, 400, 500, 1000).
For RF, in the implementation of RF algorithm, sample perturbation via bootstrap sampling of the training data and feature perturbation via random feature subset selection are introduced to improve the diversity of base learners (decision trees), which corrects for the overftting habit of decision trees and subsequently enhances the generalization ability of RF, In the experiments, we optimize the
18

---

hyper-pane parametessssss within the following ranges: estimators (50, 100, 200, 300, 400, 500, 1000), max depth (3 to 12). min samples Ieaf (1.3.5,00,20.50,, min impurity deccreaee (0 0.01) and max mex_feetures ( ('sqrt', 'log2', 0.7, 0.8, 0.9).
For 1D CNN, optimize the hyper-parameters within the following ranges: learning rate 10-1, 5 x 10-5, 5 x 10- the convolution kernel size (4, 8, 10), and the number of the hidden features in the fully connected layer (128, 256, 512, 1024).
For RNN, we optimize the hyper-parameters within the following ranges: the hidden dimensions of each layer ([64, 128, 256, 512)), L2 regularization (0 to 0.01), dropout rate (0 to 0.5). The other hyperparameters are the same as the previous work [25]
For TRSF, we optimize the nyper-parameters within the following ranges: the hidden dimensions of each layer (664, 128, 256, 512)), L-2 regularization to 0.01), dropout rate (0 to 0.5). The other
hyperparameters are the same as the previous work 25
For MLP, we optimize the hyper-parameters within the following ranges: L2 regularization to 0.01), dropout rate (0.0 to 0.5), and neurons for each hidden layer (64, 128, 256, 512). The other important hyper-parameters were fixed: ReLU function that has been recommended by many previous studies was used as the activation function, and the optimizer was set to an adaptive learning rate algorithm: Adadelta (83.
For MPNN, we optimize the hyper-parametee parameters within the following ranges: L2 regularization (0, 10 10-8, 10 x 10-6, 10 x 10 10-1), learning rate (10 10-2.5 , 10 10-3.5 10 10-1.5), dimension of node feature in hidden layers (64, 32, 16), dimension edge feature in hidden layers (64, 32, 16), and the number of set2set layers (2,3,4). The number of message passing steps and set2set steps were fixed to 6.
For GCN, we optimize the pyraeeeassmeters within the following ranges: L2 regularization (0, 10 x 10-8 80-8, 10 x 10-6 10 x 10-1) learning rate (10 10-2.5, 2.5 10 1053.5 10 x 10 10-1.5), dimension of the FNN classifier (64, 128, 256), and the dimension of GCN hidden layers ([128, 128], [256, 256], [128, 64], [256, 128]).
For GAT, we optimize the hyper-parameters within the following ranges: L2 regularization (0, 10 x 10-8, 10 x 10-6, 10 x 10-1), learning rate (10 10 10-2.5, 10 10-3.5 10 10-1.5), dimension of GAT hidden layers ([128, 128), [256, 256], [128, 64), [256, 128]), dimension of FNN classifier (64, 128, 256), and the number attention heads ([2,2], [3,3], [4.4], [3,4], [2.3]).
For AFP, we optimize the hyper-parameters within the following ranges: L2 regularization (0, 10 10-8 10 x 10-6, 10 40, learning rate (10 x 10-2.5 10 10-3.5 10 10 10-1.3), the number of attentive layers for atom embedding (2,3,4,5,6), the number of attentive layers for molecule embedding (1.2,3,4.5), dropout rate (0, 0.1, 0.3, 0.5), and fingerprint dimension (50, 100, 200, 300).
For SPN, we optimize the pureeeparameters within the following ranges: the distance LB2 intermedi- size (4, 8, 16), the angle I.2 intermediate size (4, 8, 16), the torsion LB2 intermediate size (4, 8. 16), the number interaction blocks (3, 4, 5), the number of spherical harmonics (3, 5, 7), the cutoff distance (4, 5, 6), the batch size (32, 64), the initial learning rate (1 10 4 5 x 10-4 -, 1 x 10 10-3), the learning rate step size (50, 100, 150), the learning rate decay ratio (0.4, 0.5, 0.6). The other settings the same the original paper (38).
19


Reviews:
[{"review_id": "Nrh55peOu8", "reviewer": "Reviewer_zBbT", "summary": "The authors compared various deep learning and traditional non-deep machine learning model performance on multiple molecular property prediction tasks. Their findings indicate that deep models tend to underperform non-deep ones due to the irregular patterns of molecular data and relatively small data size. To that end, they proposed to apply a feature mapping technique to help deep models capture high frequency patterns, and demonstrated the effectiveness on benchmark datasets.\n\nI find this work more like a technical report providing comprehensive benchmark results. My main concerns include (1) the conclusions are somewhat expected and probably already well-known in the community (e.g., the sharp vs smooth decision boundary issue); (2) it seems that the main focus of this work is about feature engineering, while the proposed methodology (i.e., Fourier-type feature mapping) exhibits insufficient novelty. \n\nTherefore, I do not recommend publishing this manuscript to NeurIPS.", "questions": "- The molecular similarity from a visual perspective (e.g., Figure 3) is quite intuitive for us, but does it necessarily mean deep models cannot find an appropriate latent space where these \"cliff molecules\" are far apart?\n- With increasing data size, what is the trend of deep vs non-deep model performance?\n- Since the conclusion is that tree-based models tend to outperform others, does it mean rule-based models are superior to deep models?\n- I think Figure 2 and 4 are a bit misleading. The scale of y-axis has been engineered to enhance the numerical difference, imo this is not appropriate, especially for AUROC plots (i.e., baseline 0.5).\n- Line 245, the authors mentioned \"convex combination\", yet my understanding is that $\\hat{x}_i$ underwent orthogonal transformation Q from $x$, which is not necessarily a convex combination of $x$?", "limitations": "- The over-smoothing issue of GNNs could be discussed in the manuscript to further support their claims.\n- For molecules without available 3D structures, RDKit-generated conformers are used. However, these generated structures (probably relaxed with MLFF) are not as accurate as DFT-relaxed ones -- which might confuse 3D structure-based deep models.\n", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- The authors provided comprehensive benchmark results on many commonly-used molecular property prediction datasets.\n- Comparing the performance amongst non-deep and deep learning models on molecular property prediction is a significant research topic.\n", "weaknesses": "- Insufficient novelty. The proposed methodology is well-known (e.g., NeRF, Fourier features, etc.).\n- The main idea is about feature engineering, while the set of optimal features cannot be theoretically guaranteed.\n- The comparison between deep and non-deep models on these molecular datasets might be unfair for deep models due to the data size. Although the authors claimed that irregular data patterns is another important factor explaining their predictive power, larger data size (i.e., the scaling law) indeed tends to improve deep model generalizability (e.g., see [pvd](https://arxiv.org/abs/2206.00133)).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "1dDuAKbWpH", "reviewer": "Reviewer_1yt1", "summary": "The authors demonstrate, through a large number of experiments across many datasets and tasks, that tree-based models on molecular fingerprints consistently outperform deep learning models in molecular property prediction, and attribute this to 1) the \"non-smooth\" nature of molecular labels (e.g. the existence of \"activity cliffs\"), and 2) the manner in which deep learning models mix molecular features. Given these conclusions, the authors propose a novel molecular feature pre-processing method called Independent Feature Mapping (IFM) and show that using IFM-transformed features as inputs to deep learning models allows them to outperform tree-based models.", "questions": "- Why is batch size not treated as a hyperparameter? Section 3.1 suggests it is kept fixed at 128.\n- The molecules in Figure 3 would have very similar structural fingerprints, so why would tree-based methods on these fingerprints do well on this example? Is there a better example that could illustrate the point being made?", "limitations": "No mention of limitations is made.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "### Originality\n\nThe authors are certainly not the first to point out that deep learning methods are often outperformed by non-deep models, especially tree-based models. However, the non-trivial explanation for this phenomenon in terms of the \"non-smooth\", high-frequency nature of molecular labels and the spectral bias of deep learning methods, as well as the introduction of IFM to deal with these issues, are important contributions to the understanding of how to improve molecular property prediction generally.\n\n### Quality\n\nThe quality of the paper is high, containing a large number of relevant experiments across various datasets, tasks, and models. The appendix suggests that a suitable amount of hyperparameter tuning was performed for each model.\n\n### Clarity\n\nThe paper is mostly clear, though a number of improvements can be made (see Weaknesses)\n\n### Significance\n\nAccurate OOD molecular property prediction is an incredibly valuable tool for drug discovery and material design, and this paper makes an important contribution to this area of research.", "weaknesses": "- A number of the tables and figures could be significantly improved to make them more easily comprehended. For example:\n  - the lack of standard deviations in Table 1 makes the conclusions less significant, please move from appendix to main text.\n  - Figures 2 and 4 should be summarized as tables and only the percent difference from 0-smooth or Original, respectively, displayed\n  - Tables 5 and 6 could be moved to the appendix to make room.\n- The caption of Figure 2 claims that the smoothing experiments were only done for regression tasks because smoothing is unsuitable for classification, but this is not true - cross-entropy loss can be applied to both one-hot and dense labels. Please provide a classification example.\n- In Explanation 2, the following claim is made without evidence: \"The learning style of tree models is more suitable for molecular data because only a handful of features (e.g., certain substructures) are most indicative of molecular properties.\" Please provide evidence for this claim.\n- In Section 6.2, the following statement is made: \"mixing different dimensions of molecular features as SM [33] and GM [48] would degrade the performance.\" However, only GM mixes features. Please fix.\n- The Discussion section is short and weak. Possible limitations of the proposed approach are not discussed. Please update.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ncDXjYTpNY", "reviewer": "Reviewer_ah6B", "summary": "This paper investigates why classical ML models (i.e. anything except neural networks) often outperform neural networks on molecular property prediction tasks.\n\nThe paper first presents a large study evaluating many deep and non-deep methods on a variety of datasets (seems to be mostly MoleculeNet but I was unsure of this detail). The conclusions are:\n\n1. Classical models do tend to perform better\n2. Dataset size is not the only factor responsible for low performance of deep methods\n3. Tree methods (e.g. random forest + XGBoost) do particularly well\n\nThe authors speculate that this is due to molecular functions not being smooth and mixing of input features. They do some experiments which provide some evidence for this (although I don't think they are conclusive, more on this below).\n\nThe paper then proposes to fix this by transforming the input features with a sinusoidal transformation, and show experimentally that this improves performance.\n\n**EDIT**: I read the rebuttal and it improved my opinion of the paper, but only slightly. I still believe the significance/novelty of the findings is not super high.", "questions": "See weaknesses section. I would mainly like the authors to respond to my questions about whether their experiments support their conclusions.", "limitations": "The main limitation of the study is that the conclusions are not well-supported in my opinion.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "I think the biggest strength of this paper is the problem choice: not many works study this question and I think it is arguably *the* most important question in research on ML for molecules.\n\nI also think this paper has some good ideas for studying this problem: e.g:\n- looking at performance on activity cliff vs non-activity cliff molecules.\n- doing experiments on an artificially smoothed version of the dataset\n- performing experiments with artificially mixed input features\n\nUnfortunately I don't think the experiments fully support the authors' conclusions, but I will discuss that further in weaknesses.\n\nThe idea of feature transformation is interesting but not very well-explained.\n\n**Originality / quality / clarity / significance**\n\nMy explicit opinion on these aspects of the paper (also accounting for the weaknesses in the next section) are:\n\n- *Originality*: although this is the only paper I know of which specifically addresses deep vs non-deep models for molecules, this question has been previously studied in more general settings (e.g. https://arxiv.org/abs/2108.13637) and countless papers in cheminformatics have compared NNs and classical methods. The idea of studying activity cliffs was studied of course in the paper introducing the activity cliff dataset. Therefore I think the originality of the paper is not super high, although there is some originality.\n- *Quality*: I think the key analysis of the paper is not very thorough, so I must unfortunately rate this as medium/low.\n- *Clarity*: I thought this was ok, but the paper was made less clear because many important details were relegated to the appendix without any indication of where this information could be found.\n- *Significance*: I could imagine some ideas from this paper being referenced in future work, but I think the conclusions are not fully supported by the experiments (more next section) so I think the significance of the paper is fairly limited.", "weaknesses": "There are a few weaknesses I would like to point out in this paper, which I will list roughly in order of importance:\n\n**Experiments don't seem to support conclusions**\n\nSpecifically:\n- In section 3, the authors conclude that \"irregular data patterns, NOT solely the small size of molecular datasets [are] to blame for the failure of deep models\". However, \"irregular data patterns\" is not really defined: what makes a molecule pattern \"irregular\"? This seemed to just be giving a name to whatever phenomenon causes dataset size to not totally predict performance. The other 2 conclusions from this section are supported but just confirm what is in numerous previous works.\n- In section 4, \"smoothness\" is not defined precisely. Although deep models perform better than non-deep models in Figure 2, the error for all methods decreases sharply, and the differences between all methods are small compared to the overall change in performance, so the statistical significance of this result is questionable. My conclusion would just be that the smoothed tasks are much easier! Also, the number of datasets is very limited. Perhaps the authors could extend this to the classification dataset by using a mixture of labels, e.g. like MIXUP (https://arxiv.org/abs/1710.09412)\n- Similarly, in section 4 the authors conclude from their experiments on \"activity cliff\" and \"non-activity cliff\" molecules that deep models are \"less sensitive to subtle structural changes and struggle to learn non-smooth target functions\" because the performance difference between activity cliff and non-activity cliff molecules is small. I think this cannot be concluded by looking at error rates. You would need to look at how the predictions change as the input structure is perturbed. The overall error rate is high for both cliff and non-cliff molecules, so I would instead conclude that deep model just fit poorly to both the cliff and non-cliff datasets, and that smoothness is *not* affecting the quality of the fit.\n- At the end of section 4 the authors conclude that molecular features should not be mixed, even though this does not affect the performance of deep models. I agree that one would expect this to worsen the performance of tree models, but as the authors themselves note a deep model could learn the inverse transformation and therefore be less affected. It seemed like the authors concluded the opposite of what I expected?\n- Experiments with the sinusoidal featurization in section 6 are not comprehensive: the authors don't seem to control for many other factors including regularization, early stopping, etc. There are too many factors going into neural network training to conclude that their featurization is effective from such a simple study.\n- The tables do not contain error bars, so the statistical significance of performance differences are unclear\n\n**Clarity**\n\nThe paper was not very clear, with many details omitted or put into unspecified areas of the appendix. Some key thing which were unclear are:\n\n- The fingerprint featurization in section 3.1: how many dimensions were dropped? Are the vectors no longer sparse?\n- Neural network training: how was regularization handled? Was the number of parameters varied to avoid overfitting?\n- The featurization in section 5 is barely described...\n\n**Related work**\n\nThe authors did not comment on numerous other works which compare deep and non-deep methods in some capacity, when this is arguably the most important related work. The related work section only introduced the descriptors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "SRAiouXXku", "reviewer": "Reviewer_rZmN", "summary": "- The paper evaluates various (non-)deep models for molecular property prediction. Tree models using molecular fingerprints as inputs are shown to perform better than other competitors.\n- The paper claims also to contain an extensive empirical investigation into the unique patterns of molecule data and inductive biases of various models underlying these phenomena. \n- It also presents a feature mapping method for molecule data prior to feeding them into deep models.\n", "questions": "Just the above-mentioned question about the feature mapping.", "limitations": "-", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The topic certainly offers various interesting research directions and the paper mentions open questions.\n\n- The study considers datasets beyond Moleculenet, namely, datasets related to COVID-19 and activity cliff datasets.\n\n- The analysis in terms of smoothness is interesting, even if we know it. Basically, the paper shows that the molecular data often does not match this model bias of deep learning.\n\n- The feature mapping is an interesting novelty but needs some clarification (see below).\n", "weaknesses": "- I do not fully understand why this paper was not submitted to the datasets & benchmark track, the research part seems minor (large parts are in the appendix), l67ff.:\nWe provide the most comprehensive benchmark on MPP tasks to date and expose the limitations of deep models on molecule datasets. Our findings offer new and valuable insights for the fast-growing AIDD community.\n\n\n- There are important related works the paper is missing, which contain similar results, e.g.\n  - Tom et al. Calibration and generalizability of probabilistic models on low-data chemical datasets with DIONYSUS, RSC Digital Discovery 2023\n  - Deng et al. Taking a Respite from Representation Learning for\nMolecular Property Prediction, arXiv preprint, arXiv:2209.13492, 2022\n\n- Experimental Setting\n  - The 2D baselines considered besides MPNN seem inappropriate to me. The study applies GCN and GAT, which do not even consider edge features. Models accepted by the community would more fitting, e.g., D-MPNN (Yang et al, 2019).\n  - The study focuses on random splits while the scaffold split setting was suggested, e.g., with Moleculenet or the foundational study on GNN SSL by Hu et al. 2019, and is standard in all SOTA works. \n\n- l. 174, Observation 2. It is irregular data patterns, NOT solely the small size of molecular datasets to blame for the failure of deep models! - as mentioned above, exactly because of this, the scaffold splitting has been accepted by the community. Either the authors missed that or I am misunderstanding the point.\n\n- l. 183, Observation 3. Tree models (XGB and RF) exhibit a particular advantage over other models.\nGiven the above-mentioned works, I assume every researcher in the field runs some simple fingerprint-based baselines for comparison. Hence I do not think this is a novel finding for the community.\n\n- To me, it's not clear in how far the feature mapping is better than the commonly used mapping to dense features (i.e., using torch.Embedding). Is the latter mapping used in the baselines? I would also suggest to move the theoretical justification from the appendix to the main text.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "46AhubE6w6", "reviewer": "Reviewer_SHsm", "summary": "The authors systematically study the performance of deep vs. non-deep machine-learned models in the setting of molecular property prediction, assessing the impact of input representations and models on 15 representative datasets taken from MoleculeNet, a standard benchmark in molecular property prediction. \n\nThe authors undertake a study into why non-deep models perform well on these molecular benchmark datasets, including experimentation with smoothing molecular data, and using activity cliff pairs as an example of non-smooth data. There is discussion of two possible explanations as to why non-deep models perform well in molecular property prediction, focusing on smoothness of data and mixing of input features.", "questions": "As stated above, the authors mention pretraining on molecules with IFM and note that it boosts the performance of deep models to use a pretraining and finetuning and pretraining paradigm. However it is not clear how this fits with the explanations given for the poor performance of such models in non-smooth datasets. Do the authors claim that this pretraining is somehow reducing the requirement for smooth data? Clarification and experimentation around this point would be useful.", "limitations": "Limitations of the work are not discussed.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The strength of this paper lies in its exploration of possible explanations for poor performance of deep-learning approaches in molecular property prediction, as this has not been systematically addressed previously. The exploration of the smoothness of the target function is a potentially valuable addition to the understanding of the field. Discussion is clear and concise, and experiments useful. The explanations for poor performance of non-deep models are plausible and well-supported by the experiments. It is helpful that the authors focused on the MoleculeNet benchmark as this is a good standard in the community.", "weaknesses": "In the related work section, there is a good description of the different approaches to representation and modelling of molecular properties. However, this might be better termed \"background\" or introduction necessary to understand the methods benchmarked rather than related work.  There is no discussion of works actually related to the topic of the paper -- that is, previous work that attempts to benchmark a variety of methods -- such as the MoleculeNet paper itself (Wu et al., 2018) and works building on that which include a variety of non-deep methods, and the range of few-shot learning works in molecular property prediction that also explore the MoleculeNet benchmark datasets (Nguyen et al. 2020 ICML GRL+, Pappu et al. 2020 CoRR), as well as similar works exploring performance of non-deep methods on small property prediction datasets (eg. Stanley et al. 2021). \n\nTherefore, the authors must include a thorough discussion of these works, and the current results, including those reported on the MoleculeNet website for a range of non-deep techniques such as xgb. \n\nAt the beginning of section 4, the authors assert that \"unlike image data, molecular data patterns are non-smooth\" -- this is a somewhat strong assertion to make without evidence, particular as the authors then go on to smooth data with regression labels, which inherently images are not. This statement needs some clarification; overall, it would be useful to more tightly define what is meant by \"smooth\" data in this section. \n\nIn addition, the authors mention pretraining on molecules with IFM and note that it boosts the performance of deep models to use a pretraining and finetuning and pretraining paradigm. However it is not clear how this fits with the explanations given for the poor performance of such models in non-smooth datasets. Do the authors claim that this pretraining is somehow reducing the requirement for smooth data? Clarification and experimentation around this point would be useful. \n\nOverall while the paper attempts to address the possible explanations for poor performance of deep models in molecular property prediction, a major weakness is that it is hard to conclusively state that these reasons are in fact absolute truth, especially given that there are a number of other issues in that domain that are equally pressing (eg. the small size of datasets from a deep-learning perspective, which the authors do not address). The ideas are helpful and explored well but inconclusive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper studies the phenomenon that deep models are not as good as non-deep ML methods on molecule property prediction tasks. 

The paper present empirical evidence for this, although this phenomenon is known before. The authors give plausible explanation: (1) non-smoothness nature of molecules; (2) better decision needs to be done for individual molecule features before mix. 

The paper then proposes a method to improve the performance of all existing deep models (Independent Feature Mapping). This is a small extension of Tancik et al, but Tancik was never used for molecule tasks. 

There are divided reviews for this paper. So making a decision is extremely difficult in this case. It can certainly go either way. Both sides have valid points which I would summarize below: 

**Reason to accept**

S1. Understanding the behavior of deep models comparing to non-deep models for molecule property prediction tasks is very important and timely. 

S2. The author’s contributions are therefore important since they provide a plausible explanation backed by numerous well-designed demonstrations as well as a proposed method based on the new understanding, again backed by convincing experiments.

S3. Additional experiments provided in the response are adequate to address major concerns of reviewers. 

**Reason to Reject**

W1. It is not fully confirmed that in general case non-smoothness of data cause the issue of deep models comparing to non-deep models. 

W2. The phenomenon is well known. Therefore the paper over-claim on this. 

W3. The method is a straightforward extension to Tancik et al, and it is just feature engineering, therefore it lacks novelty. 

W4. It is better suited for benchmark and data track. 

W5. The authors did not provide official rebuttal in time, instead they provided comments during discussion period therefore reducing time to discuss. 

W6. Too much significant results in author response so it will change the paper dramatically

I spend additional time to go through the paper, all the reviews, author responses and discussion. I agree with reviews on the above S1-3, W1 and W2. But W3-W5 are invalid or unsupported. 
- Regarding W1. The paper does not provide a explanation why non-smoothness of data causes trouble for general deep models versus non-deep models. But it might be a too broad request for authors and the author did not claim that for general tasks as well. They only study molecule tasks. 

- Regarding W3, the paper's IMF method is still different from Tancik et al, as explained by authors. The Tancik et al was for graphics/cv but not tested on Molecules. It is a good contribution even to demonstrate the effectiveness of Tancik et al on molecule tasks and this paper goes beyond that. 

- W4 (provided by Reviewer rZmN, Reviewer ah6B) is invalid. This paper provides explanation of deep models and a new method to improve. It does not provide a benchmark or an evaluation metric. Reviewer rZmN, ah6B either misunderstood the main content (but from the review it seems they understand well) or misunderstood the Benchmark track. 

- W5. The authors' response came late. It is not ideal and should be discouraged in the future. But we managed to have intensive discussion among reviewers. 

- W6. It seems ok since these experiments are requested by reviewers. It simply does not make sense to reject a paper because author responded with the information requested by reviewers. It could be better if the original paper already included all these results. 

Overall, the strengths outweigh the weakness. I support inclusion of the paper in the conference. It would be beneficial for the researchers studying this topic. 

Suggestion for authors:
1. You may want to add the additional explanation and experimental results in the revision. 
2. Please add more details about IMF method. You may shrink a bit in Section 1, 2, 3. 
3. Please tune down your claims and avoid exaggeration  (the findings in section 3, which was discovered before. Non-smoothness explanation, please restrict scope, novelty of IMF and relation with Tancik et al.)
Decision: Accept (poster)