Title: Understanding the Limitations of Deep Models for Molecular property prediction: Insights and Solutions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph Neural Networks
Abstract: Molecular Property Prediction (MPP) is a crucial task in the AI-driven Drug Discovery (AIDD) pipeline, which has recently gained considerable attention thanks to advancements in deep learning. However, recent research has revealed that deep models struggle to beat traditional non-deep ones on MPP. In this study, we benchmark 12 representative models (3 non-deep models and 9 deep models) on 15 molecule datasets. Through the most comprehensive study to date, we make the following key observations: \textbf{(\romannumeral 1)} Deep models are generally unable to outperform non-deep ones; \textbf{(\romannumeral 2)} The failure of deep models on MPP cannot be solely attributed to the small size of molecular datasets; \textbf{(\romannumeral 3)} In particular, some traditional models including XGB and RF that use molecular fingerprints as inputs tend to perform better than other competitors. Furthermore, we conduct extensive empirical investigations into the unique patterns of molecule data and inductive biases of various models underlying these phenomena. These findings stimulate us to develop a simple-yet-effective feature mapping method for molecule data prior to feeding them into deep models. Empirically, deep models equipped with this mapping method can beat non-deep ones in most MoleculeNet datasets. Notably, the effectiveness is further corroborated by extensive experiments on cutting-edge dataset related to COVID-19 and activity cliff datasets.

Understanding the Limitations of Deep Models for Molecular property prediction: Insights and Solutions

Abstract

Molecular Property Prediction (MPP) is a crucial task in the AI-driven Drug Discovery (AIDD) pipeline, which has recently gained considerable attention thanks to advancements in deep learning. However, recent research has revealed that deep models struggle to beat traditional non-deep ones on MPP. In this study, we benchmark 12 representative models (3 non-deep models and 9 deep models) on 15 molecule datasets. Through the most comprehensive study to date, we make the following key observations: (i) Deep models are generally unable to outperform non-deep ones; (ii) The failure of deep models on MPP cannot be solely attributed to the small size of molecular datasets; (iii) In particular, some traditional models including XGB and RF that use molecular fingerprints as inputs tend to perform better than other competitors. Furthermore, we conduct extensive empirical investigations into the unique patterns of molecule data and inductive biases of various models underlying these phenomena. These findings stimulate us to develop a simple-yet-effective feature mapping method for molecule data prior to feeding them into deep models. Empirically, deep models equipped with this mapping method can beat non-deep ones in most MoleculeNet datasets. Notably, the effectiveness is further corroborated by extensive experiments on cutting-edge dataset related to COVID-19 and activity cliff datasets.

1 Introduction

Molecular Property Prediction (MPP) is a critical task in computational drug discovery, aimed at identifying molecules with desirable pharmacological and ADMET (absorption, distribution, metabolism, excretion, and toxicity) properties. Machine learning models have been widely used in this fast-growing field, with two types of models being commonly employed: traditional nondeep models and deep models. In non-deep models, molecules are fed into traditional machine learning models such as random forest and support vector machines in the format of computed or handcrafted molecular fingerprints [64]. The other group utilizes deep models to extract expressive representations for molecules in a data-driven manner. Specifically, the Multi-Layer Perceptron (MLP) could be applied to computed or handcrafted molecular fingerprints; Sequence-based neural architectures including Recurrent Neural Networks (RNNs) [43], 1D Convolutional Neural Networks (1D CNNs) [22], and Transformers [25, 54] are exploited to encode molecules represented in Simplified Molecular-Input Line-Entry System (SMILES) strings [71]. Additionally, molecules can be naturally represented as graphs with atoms as nodes and bonds as edges, inspiring a line of works to leverage such structured inductive bias for better molecular representations [20, 76, 79, 58]. The key advancements underneath these approaches are Graph Neural Networks (GNNs), which

Nc1ccc(cc1)C(=O)O

CNN

RNN

TRSF

(a) Fingerprints (FPs)

SVM

RF

XGB

MLP

1

0

1

1

0

0

0

0

0

(d) 3D Graph

(c) 2D Graph

GCN

MPNN

GAT

AFP

0

7

1

0

0

Atoms features

Bonds features

0

1

3

Models

Descriptors

SPN

consider graph structures and attributive features simultaneously in the learning process [33, 68, 24]. More recently, researchers incorporate 3D conformations of molecules into their representations for better performance, whereas pragmatic considerations such as calculation cost, alignment invariance, uncertainty in conformation generation, and unavailable conformations of target molecules limited the practical applicability of these models [5, 17, 57, 16, 38]. We summarize the widely-used molecular descriptors and their corresponding models in our benchmark, as shown in Figure 1. Despite the fruitful progress, previous studies [41, 29, 79, 65, 30, 13, 66] have observed that deep models struggled to outperform non-deep ones on molecular datasets. However, these studies neither consider the emerging powerful deep models (e.g., Transformer [25], SphereNet [37]) nor explore various molecular descriptors (e.g., 3D molecular graph). Also, they did not investigate the reasons why deep models often fail on molecules.

To narrow this gap, we present the most comprehensive benchmark study on molecular property prediction to date, with a precise methodology for dataset inclusion and hyperparameter tuning. Our empirical results confirm the observations of previous studies, namely that deep models generally struggle to outperform traditional non-deep counterparts, even without accounting for the slower training of deep learning algorithms. Moreover, we observe several interesting phenomena that challenge the prevailing beliefs of the community, which can guide optimal methodology design for future studies.

Furthermore, we aim to understand why deep models often underperform non-deep ones in MPP. Specifically, we transform the original molecular data to observe the performance changes of various models, uncovering the unique patterns of molecular data and the differing inductive biases of various models. These in-depth empirical studies shed light on the benchmarking results: Deep models struggle to learn non-smooth target functions that map molecular data to labels, while the target functions are often non-smooth in MPP. This means that small changes in the chemical structure of a molecule may result in large changes in molecular properties. Additionally, deep models tend to attend to molecule features as a whole, especially handling the molecular fingerprints, while partial substructures known as functional groups are the most informative for molecules. On the other hand, XGB and random forest are well-suited for molecules because they make decisions based on each dimension of molecular features separately. Based on these phenomena and analyses, we develop a novel feature mapping method for molecule data before feeding them into models. Theoretically, we show that our method can help deep models learn non-smooth target functions that map molecules to properties. Moreover, our method is readily pluggable into various deep methods for performance improvement.

We highlight the following contributions: (I) We provide the most comprehensive benchmark on MPP tasks to date and expose the limitations of deep models on molecule datasets. Our findings offer new and valuable insights for the fast-growing AIDD community. (II) We empirically investigate the unique patterns of molecular data and inductive biases of various models, providing explanations for why deep models often cannot beat non-deep ones on MPP tasks. (III) We develop a simple-yeteffective feature mapping method to help deep models learn the non-smooth target functions with theoretical guarantees. (IV) We verify the effectiveness of our method through extensive experiments on MoleculeNet datasets, a cutting-edge dataset related to COVID-19 and activity cliff datasets.

2 Related Work

In this section, we elaborate on various molecular descriptors and their respective learning models.

2.1 Fingerprints-based Molecular Descriptors

Molecular fingerprints (FPs) serve as one of the most important descriptors for molecules. Typical examples include Extended-Connectivity Fingerprints (ECFP) [53] and PubChemFP [70]. These fingerprints encode the neighboring environments of heavy atoms in a molecule into a fixed bit string with a hash function, where each bit indicates whether a certain substructure is present in the molecule. Traditional models and MLPs can take these fingerprints as 'raw' input. However, the high-dimensional and sparse nature of FPs introduces additional efforts for feature selection when they are fed into certain models. Additionally, it is difficult to interpret the relationship between properties and structures because the hash functions are non-invertible.

2.2 Linear Notation-based Molecular Descriptors

Another option for molecules is linear notations, among which SMILES [71] is the most frequentlyused one owing to its versatility and interpretability. In SMILES, each atom is represented as a respective ASCII symbol; Chemical bonds, branching, and stereochemistry are denoted by specific symbols. However, a significant fraction of SMILES strings does not correspond to chemically valid molecules. As a remedy, a new language named SELF-referencIng Embedded Strings (SELFIES) for molecules was introduced in 2020 [34]. Every SELFIES string corresponds to a valid molecule, and SELFIES can represent every molecule. Naturally, RNNs, 1D CNN, and Transformers are powerful deep models for processing such sequences [69, 86, 25, 55, 82]. However, the poor scalability of the sequential notations and the loss of spatial information limit the performances of these approaches.

2.3 2D and 3D Graph-based Molecular Descriptors

Molecules can be represented with graphs naturally, with nodes as atoms and edges as chemical bonds. Initially, [15] first adopted convolutional layers to encode molecular graphs to neural fingerprints. Following this work, [11] employs the atom-based message-passing scheme to learn expressive molecular graph representations. To complement the atom's information, [31] utilized both the atom's and bonds' attributes, and MPNN [20] generalized it to a unified framework. Also, multiple variants of the MPNN framework are developed to avoid unnecessary loops (DMPNN [79]), to strengthen the message interactions between nodes and edges (CMPNN [58]), to capture the complex inherent quantum interactions of molecules (MGCN [39]), or take the longer-range dependencies (Attentive FP [76]). More recently, some hybrid architectures [54, 81, 42, 45] of GNNs and transformers are emerging to capture the topological structures of molecular graphs.

The 3D molecular graph is composed of nodes (atoms), and their positions in 3D space and edges (bonds). The advantage of using 3D geometry is that the conformer information is critical to many molecular properties, especially quantum properties. In addition, it is also possible to directly leverage stereochemistry information such as chirality given the 3D geometries. Recently, multiple works [57, 56, 14, 38, 3] have developed message-passing mechanisms tailored for 3D geometries, which enable the learned molecular representations to follow certain physical symmetries, such as equivariance to translations and rotations. However, the calculation cost, alignment invariance, uncertainty in conformation generation, and unavailable conformations of target molecules limited the applicability of these models in practice.

3 Benchmark Representative Models on Multiple Molecular Datasets.

In this section, we present a benchmark on 15 molecular datasets with 12 representative models.

3.1 Experimental Setups

Fingerprints ↦-→ SVM, XGB, RF, and MLP. Following the common practice [30, 63, 50], we feed the concatenation of various molecular fingerprints including 881 PubChem fingerprints (PubchemFP), 307 substructure fingerprints (SubFP), and 206 MOE 1-D and 2-D descriptors [80] to

SVM, XGB, RF, and MLP models to comprehensively represent molecular structures, with some pre-processing procedures to remove features (1) with missing values; (2) with extremely low variance (variance < 0.05); (3) have a high correlation (pearson correlation coefficient > 0.95) with another feature. The retained features are normalized to the mean value of 0 and variance of 1. Additionally, considering that traditional machine models (SVM, RF, XGB) cannot be directly applied in the multi-task molecular datasets, we split the multi-task dataset into multiple single-task datasets and use each of them to train the models. Finally, we report the average performance of these single tasks.

SMILES strings ↦-→ CNN, RNN, and TRSF. We adopt the 1D CNNs from a recent study [32], which include a single 1D convolutional layer with a step size equal to 1, followed by a fully connected layer. As for the RNN, we use a 3-layer bidirectional gated recurrent units (GRUs) [10] with 256 hidden vector dimensions. Additionally, we use the pre-trained SMILES transformer [25] with 4 basic blocks and each block has 4-head attentions with 256 embedding dimensions and 2 linear layers. The SMILES are split into symbols (e.g., 'Br', 'C', '=', '(','2') and then fed into the transformer together with the positional encoding [67].

2D Graphs ↦-→ GCN, MPNN, GAT, and AFP. As in previous studies [76], we exhaustively utilized all readily available atom/bond features in our 2D graph-based descriptors. Specifically, we have incorporated 9 atom features, including atom symbol, degree, and formal charge, using a one-hot encoding scheme. In addition, we included 4 bond features, such as type, conjugation, ring, and stereo. The resulting encoded graphs were then fed into GCN, MPNN, GAT, and AFP models. Further details on the graph descriptors used in our experiments can be found in [76].

3D Graphs ↦-→ SPN. We employ the recently proposed SphereNet [38] for molecules with 3D geometry. Specifically, for quantum mechanics datasets (QM7, QM8, and QM9) that contain 3D atomic coordinates calculated with ab initio Density Functional Theory (DFT), we feed them into SphereNet directly. For other datasets without labeled conformations, we used RDKit [35]-generated conformations to satisfy the request of SphereNet.

Datasets splits, evaluation protocols and metrics, hyper-parameters tuning. Firstly, we randomly split the training, validation, and test sets at a ratio of 8:1:1. And then, we tune the hyper-parameters based on the performance of the validation set. Due to the heavy computational overhead, GNNsbased models on the HIV and MUV datasets are in 30 evaluations; all the models on the QM7 and QM8 are in 10 evaluations; all the models on the QM9 dataset are in one evaluation. And then, we conduct 50 independent runs with different random seeds for dataset splitting to obtain more reliable results, using the optimal hyper-parameters determined before. Following MoleculeNet benchmark [72], we evaluate the classification tasks using the area under the receiver operating characteristic curve (AUC-ROC), except the area under the precision curve (AUC-PRC) on MUV dataset due to its extreme biased data distribution. The performance on the regression task are reported using root mean square error (RMSE) or mean absolute error (MAE). Kindly note that we report the average performance across multi-tasks on some datasets because they contain more than one task. Additionally, to avoid the overfitting issue, all the deep models are trained with an early stopping scheme if no validation performance improvement is observed in successive 50 epochs. We set the maximal epoch as 300 and the batch-size as 128. We provide more details including hyper-parameters tuning space for each model in the appendix.

3.2 Observations

Table 1 documents the benchmark results for various models and datasets, from which we can make the following Observations :

Observation 1. Deep models underperform non-deep counterparts in most cases.

As can be observed in Table 1, non-deep models rank as the top one on 11/15 datasets. Kindly note that we report the results of each task in the QM9 dataset in the appendix. On some datasets such as MUV, QM7, and BACE, three non-deep models can even beat any deep models.

Observation 2. The failure of deep models should not be solely attributed to the limited size of molecular datasets.

Intuitively, many previous works [21, 79, 59] pointed out that the small size of molecular datasets is a bottleneck for deep learning models and propose various strategies accordingly [49, 48]. Here, we complement this pre-dominant belief with a new opinion with empirical evidence. As shown in Table 1, all the non-deep models can outperform any deep ones on some larger-scale datasets (e.g.,

0-smooth

10-smooth

20-smooth

0.4

0.5

0.6

0.7

0.8

RMSE

SVM

XGB

RF

MLP

GCN

MPNN

GAT

AFP

0-smooth

10-smooth

20-smooth

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

RMSE

0-smooth

10-smooth

20-smooth

20

40

60

80

100

120

MAE

MUVand QM 7). However, in some small datasets (e.g., ClinTox and ESOL), some deep models can beat partial non-deep ones. Therefore, we argue that there could be other reasons for the failure of deep models, not solely the dataset size. We will provide further analysis in Sec. 4.

Observation 3. XGB and RF exhibit a particular advantage over other models.

In the experiments shown in Table 1, we can see that the XGB and RF models consistently rank among the top three on each dataset. Additionally, tree models rank as the top one on 8/15 datasets. Next, we will explore why tree models are well-suited for molecular fingerprints in Sec. 4.

4 Empirical Study: Why above phenomena would occur?

In this section, we attempt to understand which characteristics of molecular data lead to the failure of powerful deep models. Also, we aim to understand the inductive biases of XGB and RF that make them well-suited for molecules, and how they differ from the inductive biases of deep models. The details of the experiments in this section can be found in the appendix .

Explanation 1. Unlike image data, molecular data patterns are non-smooth. Deep models struggle to learn non-smooth target functions that map molecules to properties.

We design two experiments to verify the above explanation, i.e., increasing or decreasing the level

of data smoothing in the molecular datasets. Firstly, we 'increase' the molecular data smoothing level by smoothing the labels based on similarities between molecules. Specifically, let D denote the molecular dataset and ( x i , y i ) ∈ D be i -th molecule and its label, we smooth the target function as follows,

where s ( · , · ) denotes the Tanimoto coefficient of the extended connectivity fingerprints (ECFP) between two molecules that can be considered as their structural similarity. N x i is the k -nearest neighbor set of x i (including x i ) picked from the whole dataset based on the structural similarities. ̂ y i denotes the label after smoothing. We smooth all the molecules in the dataset in this way and use the smoothed label ̂ y i to train the models. The results are shown in Figure 2, where '0-smooth' denotes the original datasets. '10-smooth' and '20-smooth' mean k = 10 and k = 20 , respectively. As can be observed, the performance of deep models improves dramatically as the level of dataset smoothing increases, and many deep models including MLP, GCN, and AFP can even beat non-deep ones after smoothing. These phenomena indicate that deep models are more suitable for the smoothed datasets.

Secondly, we 'decrease' the level of data smoothing using the concept of activity cliff [40, 60] from chemistry, which means a situation where small changes in the chemical structure of a drug lead to significant changes in its bioactivity. We provide an example activity cliff pairs in Figure 3. Apparently, the target function of activity cliffs that map molecules to the activity values is less smoothing than normal molecular datasets. We then evaluate the models on the activity cliff datasets [66]. The test set contains molecules that are chemically similar to those in the training set but exhibit either a large difference in bioactivity (cliff molecules) or similar bioactivity (non-cliff molecules). As shown in Table 2, the non-deep models consistently outper-

form deep ones on these activity cliff datasets. Furthermore, the deep models exhibit less significant prediction change rates compared to the non-deep ones. This observation suggests that deep models are indeed less sensitive to subtle structural changes compared to non-deep ones. Our explanation is consistent with the conclusions in deep learning theory [51], i.e., deep models struggle to learn high-frequency components of the target functions. However, traditional models such as XGB and RF can learn piece-wise target functions, and do not exhibit such bias. Our explorations uncover several promising avenues to enhance deep models' performance on molecules: smoothing the target functions or improving deep models' ability to learn the non-smooth target functions.

Original

Transformed

0.8

1.0

1.2

1.4

1.6

RMSE

SVM

XGB

RF

MLP

GCN

MPNN

GAT

AFP

Original

Transformed

0.80

0.82

0.84

0.86

0.88

0.90

0.92

0.94

0.96

AUC_ROC

Original

Transformed

0.78

0.79

0.80

0.81

0.82

0.83

0.84

0.85

0.86

0.87

AUC_ROC

Typically, features in molecular data carry meanings individually. As we elaborated in Sec. 2.1, each dimension of molecular fingerprints often indicates whether a certain substructure is present in the molecule; each dimension of nodes/edges features in molecular graph data indicates a specific characteristic of the atoms/bonds (e.g., atom/bond type, atom degree). To verify the above explanation, we mix the different dimensions of molecular features x i ∈ R d using an orthogonal transformation before feeding them into various models,

where Q ∈ R d × d is the orthogonal matrix and ̂ x i is the molecular feature after transformation. Kindly note that the meaning of x i depends on the input molecular descriptors in the experiments. Specifically, for SVM, XGB, RF, and MLP, x i denotes the molecular fingerprints; for GNN models, x i can denote the atom features and bond features in the molecular graphs, i.e., we apply orthogonal transformations to both the atom features and bond features. As can be observed in Figure 4, the performance of tree models deteriorates dramatically and falls behind most deep models after the orthogonal transformation. It is because each dimension of ̂ x i is a linear combination of all the dimensions of x i according to the matrix-vector product rule. In other words, the molecular features after orthogonal transformation no longer carry meanings individually, accounting for the failure of tree models that make decisions based on each dimension of the features separately. The learning style of tree models is more suitable for molecular data because only a handful of features (e.g., certain substructures) are most indicative of molecular properties [47]. On the other hand, the performance decreases of deep models are less significant, and most deep models can beat tree models after the transformations. We explain this observation as follows. Without the loss of generality, we assume that a linear layer of deep models can map the original molecular feature x i to the label y i ,

where W and b denote the parameter matrix and the bias term of the linear layer, respectively. And then, we aim to learn a new linear layer mapping the transformed model feature ̂ x i to label y i ,

where ̂ W and ˆ b denote the parameter matrix and the bias term of the new linear layer, respectively. Apparently, to achieve the same results as the original feature, we only have to learn ̂ W so that ̂ W = Q W because Q -1 = Q ⊤ as an orthogonal matrix, and also ˆ b = b . Therefore, applying the orthogonal transformation to molecular features barely impacts the performance of deep models. The empirical results in Figure 4 confirm this point although some performance changes are observable due to uncontrollable random factors.

5 Methodology

Although we have shown and explained the superiority of non-deep models on molecular data, deep models have numerous advantages over the traditional approaches: (i) GNNs can exploit the structural/geometrical inductive biases of 2D/3D molecular graphs, alleviating the manual efforts to capture the topology of the networks; (ii) The pre-trained representations with deep models are beneficial to various downstream tasks, showing promising values in drug discovery [75, 74, 9, 73]. Therefore, we aim to empower the deep models to beat the non-deep ones on MPP tasks including activity cliff cases.

As we explained before, deep models struggle to learn the non-smooth target functions of molecular data, a phenomenon referred to 'spectral bias' in literature [51]. To overcome such bias, prior works [44, 87] have experimentally found that a heuristic sinusoidal mapping of the input features allows MLPs to learn the non-smooth target functions. However, these mapping methods would undesirably mix the original features. Please refer to the appendix for detailed discussions due to the limited space. As a remedy, we introduce a new method named Independent Feature Mapping (IFM) that embeds each dimension of molecular features separately before feeding them into models. Denoting a molecular feature as x ∈ R d , we formulate IFM as,

where || denotes the concatenation of two vectors, c = [ c 1 , c 2 , · · · , c k ] are the learnable parameters initialized from N (0 , σ ) and f x ∈ R 2 k × d . We study the influence of the hyperparameters k and σ in the appendix. Since cos( a -b ) = cos a cos b +sin a sin b , we have,

where · is the dot product and x ′ is another molecular feature. Therefore, IFM can map data points to a vector space so that their dot product achieves a certain distance metric, which is an expected characteristic for feature mapping methods [52, 4, 23]. In what follows, we will provide theoretical justifications on the effectiveness of our IFM following a previous study [62]. As revealed in previous works, deep models can be approximated with Neural Tangent Kernel (NTK) [28, 2, 6, 36, 62]. Specifically, let I be a fully-connected deep network with weights θ initialized from a Gaussian distribution N , the NTK theory shows that as the width of the layers in I becomes infinite and the learning rate for stochastic gradient descent (SGD) approaches zero, the function I ( x ; θ ) converges during training to the kernel regression solution using the neural tangent kernel (NTK), which is:

When the inputs are limited to a hypersphere, the NTK for an MLP can be expressed as a dot product kernel (a kernel in the form h NTK ( x · x ′ ) for a scalar function h NTK : R → R ). In our cases, the input to the deep models would be f x , the composed kernel of IFM and NTK can be formulated as,

thus, training deep models on these mapped molecular features corresponds to kernel regression with the stationary composed NTK function h NTK ◦ g c . Considering that the parameters c are tunable, IFM creates a composed NTK that is not only stationary but also tunable. It enables us to dramatically control the range of frequencies that can be learned via manipulating the parameters c .

6 Experiments

6.1 Experimental Settings

In our experiments, we equip various deep models with IFM. Specifically, for MLPs with fingerprints as inputs, we employ the proposed feature mapping method to the fingerprints (after feature selection and standardization) directly; For molecular graphs, we map both the features of atoms and bonds before feeding them into the GNNs. The other settings are the same as the benchmarking experiments. If a deep model named ' z ' (e.g., MLP) is equipped with IFM, we re-name it as 'IFMz ' (e.g., IFM-MLP) in our results. Also, we evaluate the non-deep models equipped with IFM in the appendix.

6.2 Results

Main results. We show the main results in Table 3, from which we can make the following observations: (1) The proposed feature mapping method can significantly improve the performance of the deep models on molecular datasets; (2) The deep models equipped with the feature mapping method can beat non-deep counterparts in most cases, verifying the effectiveness of our method.

Results on activity cliffs. We also employ the proposed feature mapping method on activity cliff datasets where the target functions are less smooth. The results shown in Table 4 indicate that IFM improves the deep models by significant margins. Moreover, nearly all the deep models with IFM method can beat traditional methods, confirming that our method can help neural networks learn non-smooth target functions.

Comparisons with other feature mapping methods (ablation study). We compare the proposed IFM with the previous feature mapping methods. The results shown in Table 5 indicate that the proposed feature mapping method is superior to the previous method, which verifies that mixing different dimensions of molecular features as GM [62] would degrade the performance.

Pre-training on molecules with IFM. Compared with non-deep models, deep ones can be combined with the prevalent 'Pretraining and Finetuning' paradigm to exploit large-scale unlabeled molecules [75]. This motivates us to develop more powerful neural encoders for this paradigm. Specifically, we pre-train the deep models equipped with IFM and report the fine-tuning results in Table 6. Our experimental settings are the same as the pioneering work [27] where the datasets are split with scaffold splitting, differing from the random splitting in Table 3. As can be observed, our method can boost various pre-training strategies to advance their performance in downstream tasks.

7 Discussion and Conclusion

In this paper, we perform a comprehensive benchmark of representative models on molecular property prediction. Our results reveal that traditional machine learning models, especially tree models, can easily outperform well-designed deep models in most cases. These phenomena can be attributed to the unique patterns of molecular data and different inductive biases of various models. Specifically, the target function mapping molecules to properties are non-smooth, and some small changes can incur significant property variance. Deep models struggle to learn such patterns. Additionally, molecular features carry meanings individually and deep models would undesirably mix different dimensions of molecular features. These findings stimulate us to develop a simple-yet-effective feature mapping method for molecule data that can help deep models learn non-smooth target functions with theoretical guarantees. Extensive experiments verify the effectiveness of the proposed method. Our study leaves an open question for future research: Can our findings and methods be generalized to other AIDD tasks including drug-target interactions (DTIs) prediction, drug-drug interactions (DDIs) prediction?

References

Tox21 challenge. https://tripod.nih.gov/tox21/challenge/, 2017.

Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. ICML, 2019.

Kenneth Atz, Francesca Grisoni, et al. Geometric deep learning on molecular representations. Nature Machine Intelligence, 2021.

Haim Avron, Vikas Sindhwani, Jiyan Yang, and Michael W. Mahoney. Quasi-monte carlo feature maps for shift-invariant kernels. J. Mach. Learn. Res., 17:120:1-120:38, 2014.

Seth D Axen, Xi-Ping Huang, Elena L Cáceres, Leo Gendelev, Bryan L Roth, and Michael J Keiser. A simple representation of three-dimensional molecular structure. Journal of medicinal chemistry, 60(17):7393-7409, 2017.

Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non-uniform density. arXiv preprint arXiv:2003.04560, 2020.

Kyle R. Brimacombe, Tongan Zhao, Richard T. Eastman, Xin Hu, Ke Wang, Mark Backus, Bolormaa Baljinnyam, Catherine Z. Chen, Lu Chen, Tara Eicher, Marc Ferrer, Ying Fu, Kirill Gorshkov, Hui Guo, Quinlin M. Hanson, Zina Itkin, Stephen C. Kales, Carleen Klumpp-Thomas, Emily M. Lee, Sam Michael, Tim Mierzwa, Andrew Patt, Manisha Pradhan, Alex Renn, Paul Shinn, Jonathan H. Shrimp, Amit Viraktamath, Kelli M. Wilson, Miao Xu, Alexey V. Zakharov, Wei Zhu, Wei Zheng, Anton Simeonov, Ewy A. Mathé, Donald C. Lo, Matthew D. Hall, and Min Shen. An opendata portal to share covid-19 drug repurposing data in real time. bioRxiv, 2020.

Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.

Seyone Chithrananda, Gabriel Grand, et al. ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction. arXiv.org, October 2020.

Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoderdecoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.

Connor W Coley, Regina Barzilay, et al. Convolutional embedding of attributed molecular graphs for physical property prediction. Journal of chemical information and modeling, 2017.

John S Delaney. Esol: estimating aqueous solubility directly from molecular structure. Journal of chemical information and computer sciences, 2004.

Jianyuan Deng, Zhibo Yang, Hehe Wang, Iwao Ojima, Dimitris Samaras, and Fusheng Wang. Taking a respite from representation learning for molecular property prediction. arXiv preprint arXiv:2209.13492, 2022.

Weitao Du, He Zhang, et al. SE(3) Equivariant Graph Neural Networks with Complete Local Frames. In ICML, 2022.

David Duvenaud, Dougal Maclaurin, and others. Convolutional networks on graphs for learning molecular fingerprints. NIPS, 2015.

Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 34:6790-6802, 2021.

Johannes Gasteiger, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. In International Conference on Learning Representations, 2020.

Anna Gaulton, Louisa J. Bellis, et al. ChEMBL: A Large-Scale Bioactivity Database for Drug Discovery. Nucleic Acids Res., 2012.

Kaitlyn M Gayvert, Neel S Madhukar, and Olivier Elemento. A data-driven approach to predicting successes and failures of clinical trials. Cell chemical biology, 2016.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning, pages 1263-1272, 2017.

Garrett B Goh, Nathan O Hodas, and Abhinav Vishnu. Deep learning for computational chemistry. Journal of computational chemistry, 38(16):1291-1307, 2017.

Jiuxiang Gu, Zhenhua Wang, Jason Kuen, Lianyang Ma, Amir Shahroudy, Bing Shuai, Ting Liu, Xingxing Wang, Gang Wang, Jianfei Cai, et al. Recent advances in convolutional neural networks. Pattern recognition, 77:354-377, 2018.

Raffay Hamid, Ying Xiao, Alex Gittens, and Dennis DeCoste. Compact random feature maps. In International Conference on Machine Learning, 2013.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.

Shion Honda, Shoi Shi, and Hiroki R Ueda. Smiles transformer: Pre-trained molecular fingerprint for low data drug discovery. arXiv preprint arXiv:1911.04738, 2019.

Zhenyu Hou, Xiao Liu, et al. GraphMAE: Self-Supervised Masked Graph Autoencoders. In KDD, 2022.

Weihua Hu, Bowen Liu, and others. Strategies for pre-training graph neural networks. ICLR, 2020.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural Tangent Kernel: Convergence and generalization in neural networks. NeurIPS, 2018.

Tiago Janela and Jürgen Bajorath. Simple nearest-neighbour analysis meets the accuracy of compound potency predictions using complex machine learning models. Nature Machine Intelligence, pages 1-10, 2022.

Dejun Jiang, Zhenxing Wu, Chang-Yu Hsieh, Guangyong Chen, Ben Liao, Zhe Wang, Chao Shen, Dongsheng Cao, Jian Wu, and Tingjun Hou. Could graph neural networks learn better molecular representation for drug discovery? a comparison study of descriptor-based and graph-based models. Journal of cheminformatics, 13(1):1-23, 2021.

Steven Kearnes, Kevin McCloskey, and others. Molecular graph convolutions: Moving beyond fingerprints. J Comput Aid Mol Des, 2016.

Talia B Kimber, Maxime Gagnebin, and Andrea Volkamer. Maxsmi: maximizing molecular property prediction performance with confidence estimation using smiles augmentation and deep learning. Artificial Intelligence in the Life Sciences, 1:100014, 2021.

N. Thomas Kipf. and M. Welling. Semi-supervised classification with graph convolutional networks. ICLR, 2017.

Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alan Aspuru-Guzik. Self-referencing embedded strings (selfies): A 100% robust molecular string representation. Machine Learning: Science and Technology, 1(4):045024, 2020.

Greg Landrum. RDKit: A software suite for cheminformatics, computational chemistry, and predictive modeling, 2013.

Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha SohlDickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. NeurIPS, 2019.

Yi Liu, Limei Wang, et al. Spherical message passing for 3d molecular graphs. In Iclr, 2021.

Yi Liu, Limei Wang, Meng Liu, Yuchao Lin, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d molecular graphs. In International Conference on Learning Representations (ICLR), 2022.

Chengqiang Lu, Qi Liu, Chao Wang, Zhenya Huang, Peize Lin, and Lixin He. Molecular property prediction: A multilevel quantum interactions modeling perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1052-1060, 2019.

Gerald M Maggiora. On outliers and activity cliffs why qsar often disappoints, 2006.

Andreas Mayr, Günter Klambauer, Thomas Unterthiner, Marvin Steijaert, Jörg K Wegner, Hugo Ceulemans, Djork-Arné Clevert, and Sepp Hochreiter. Large-scale comparison of machine learning methods for drug target prediction on chembl. Chemical science, 9(24):5441-5451, 2018.

Łukasz Maziarka, Tomasz Danel, Sławomir Mucha, Krzysztof Rataj, Jacek Tabor, and Stanisław Jastrz˛ ebski. Molecule attention transformer. arXiv preprint arXiv:2002.08264, 2020.

Larry Medsker and Lakhmi C Jain. Recurrent neural networks: design and applications. CRC press, 1999.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.

Erxue Min, Runfa Chen, Yatao Bian, Tingyang Xu, Kangfei Zhao, Wenbing Huang, Peilin Zhao, Junzhou Huang, Sophia Ananiadou, and Yu Rong. Transformer for graphs: An overview from architecture perspective. arXiv preprint arXiv:2202.08455, 2022.

Wim De Mulder, Steven Bethard, and Marie-Francine Moens. A survey on the application of recurrent neural networks to statistical language modeling. Comput. Speech Lang., 30:61-98, 2015.

Christopher W Murray and David C Rees. The rise of fragment-based drug discovery. Nature chemistry, 1(3):187-192, 2009.

Cuong Q Nguyen, Constantine Kreatsoulas, and Kim M Branson. Meta-learning gnn initializations for low-resource molecular property prediction. arXiv preprint arXiv:2003.05996, 2020.

Aneesh Pappu and Brooks Paige. Making graph neural networks worth it for low-data molecular machine learning. arXiv preprint arXiv:2011.12203, 2020.

Lagnajit Pattanaik and Connor W Coley. Molecular representation: going long on fingerprints. Chem, 6(6):1204-1207, 2020.

Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301-5310. PMLR, 2019.

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In NIPS, 2007.

David Rogers and Mathew. Hahn. Extended-connectivity fingerprints. J chem inf, 2010.

Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. Advances in Neural Information Processing Systems, 33:12559-12571, 2020.

Jerret Ross, Brian Belgodere, et al. Molformer: Large Scale Chemical Language Representations Capture Molecular Structure and Properties. Nat. Mach. Intell., 2022.

Victor Garcia Satorras, Emiel Hoogeboom, et al. E(n) Equivariant Graph Neural Networks. In ICML, 2021.

T. K. Schuett, Kindermans, and others. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. NIPS, 2017.

Ying Song, Shuangjia Zheng, Zhangming Niu, Zhang-Hua Fu, Yutong Lu, and Yuedong Yang. Communicative representation learning on attributed molecular graphs. In IJCAI, volume 2020, pages 2831-2838, 2020.

Megan Stanley, John F Bronskill, Krzysztof Maziarz, Hubert Misztela, Jessica Lanini, Marwin Segler, Nadine Schneider, and Marc Brockschmidt. FS-mol: A few-shot learning dataset of molecules. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.

Dagmar Stumpfe and Jurgen Bajorath. Exploring activity cliffs in medicinal chemistry: miniperspective. Journal of medicinal chemistry, 55(7):2932-2942, 2012.

Vladimir Svetnik, Andy Liaw, Christopher Tong, J Christopher Culberson, Robert P Sheridan, and Bradley P Feuston. Random forest: a classification and regression tool for compound classification and qsar modeling. Journal of chemical information and computer sciences, 43(6):1947-1958, 2003.

Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. ArXiv, abs/2006.10739, 2020.

Hao Tian, Rajas Ketkar, and Peng Tao. Admetboost: a web server for accurate admet prediction. Journal of Molecular Modeling, 28(12):1-6, 2022.

Roberto Todeschini and Viviana Consonni. Molecular descriptors. Recent Advances in QSAR Studies, pages 29-102, 2010.

Cecile Valsecchi, Magda Collarile, Francesca Grisoni, Roberto Todeschini, Davide Ballabio, and Viviana Consonni. Predicting molecular activity on nuclear receptors by multitask neural networks. Journal of Chemometrics, 36(2):e3325, 2022.

Derek van Tilborg, Alisa Alenicheva, and Francesca Grisoni. Exposing the limitations of molecular machine learning with activity cliffs. Journal of Chemical Information and Modeling, 62(23):5938-5951, 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.

Petar Velickovic, Guillem Cucurull, et al. Graph Attention Networks. In ICLR, 2018.

Sheng Wang, Yuzhi Guo, and others. Smiles-bert - large scale unsupervised pre-training for molecular property prediction. BCB, 2019.

Yanli Wang, Stephen H. Bryant, et al. Pubchem Bioassay: 2017 Update. Nucleic Acids Res., 2017.

David Weininger, Arthur Weininger, and others. Smiles. 2. algorithm for generation of unique smiles notation. J CHEM INF MODEL, 1989.

Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9(2):513-530, 2018.

Jun Xia, Lirong Wu, , Jintao Chen, Bozhen Hu, and Stan Z. Li. SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation. In Proceedings of The Web Conference 2022. Association for Computing Machinery, 2022.

Jun Xia, Chengshuai Zhao, et al. Mole-BERT: Rethinking Pre-training Graph Neural Networks for Molecules. In ICLR, 2023.

Jun Xia, Yanqiao Zhu, Yuanqi Du, Yue Liu, and Stan Z Li. A systematic survey of chemical pre-trained models. IJCAI, 2023.

Zhaoping Xiong, Dingyan Wang, Xiaohong Liu, Feisheng Zhong, Xiaozhe Wan, Xutong Li, Zhaojun Li, Xiaomin Luo, Kaixian Chen, Hualiang Jiang, et al. Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism. Journal of medicinal chemistry, 63(16):8749-8760, 2019.

Zhaoping Xiong, Dingyan Wang, and others. Pushing the boundaries of molecular representation for drug discovery with graph attention mechanism. J Med Chem, 2020.

Keyulu Xu, Weihua Hu, et al. How Powerful are Graph Neural Networks? In ICLR, 2019.

Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et al. Analyzing learned molecular representations for property prediction. Journal of chemical information and modeling, 59(8):3370-3388, 2019.

Chun Wei Yap. Padel-descriptor: An open source software to calculate molecular descriptors and fingerprints. Journal of computational chemistry, 32(7):1466-1474, 2011.

Chengxuan Ying, Tianle Cai, et al. Do Transformers Really Perform Badly for Graph Representation? In NeurIPS, 2021.

Atakan Yüksel, Erva Ulusoy, Atabey Ünlü, Gamze Deniz, and Tunca Do˘ gan. Selformer: Molecular representation learning via selfies language models. arXiv preprint arXiv:2304.04662, 2023.

Matthew D. Zeiler. Adadelta: An adaptive learning rate method. ArXiv, abs/1212.5701, 2012.

Vladimir V Zernov, Konstantin V Balakin, Andrey A Ivaschenko, Nikolay P Savchuk, and Igor V Pletnev. Drug discovery using support vector machines. the case studies of drug-likeness, agrochemical-likeness, and enzyme inhibition predictions. Journal of chemical information and computer sciences, 43(6):2048-2056, 2003.

Zaixi Zhang, Qi Liu, and others. Motif-based graph self-supervised learning for molecular property prediction. NeurIPS, 2021.

Shuangjia Zheng, Xin Yan, Yuedong Yang, and Jun Xu. Identifying structure-property relationships through smiles syntax analysis with self-attention mechanism. Journal of chemical information and modeling, 59(2):914-923, 2019.

Ellen D. Zhong, Tristan Bepler, Joseph H. Davis, and Bonnie Berger. Reconstructing continuous distributions of 3d protein structure from cryo-em images. In International Conference on Learning Representations, 2019.

A Datasets

In this section, we provide detailed information on the molecular datasets used for downstream tasks. Molecular Property: Pharmacology The Blood-Brain Barrier Penetration (BBBP) dataset measures whether a molecule will penetrate the central nervous system. All three datasets, Tox21 [1], ToxCast [72], and ClinTox [19] are related to the toxicity of molecular compounds. The Side Effect Resource (SIDER) dataset stores the adverse drug reactions on a marketed drug database.

Molecular Property: Physical Chemistry Dataset proposed in [12] measures aqueous solubility of the molecular compounds. Lipophilicity (Lipo) dataset is a subset of ChEMBL [18] measuring the molecule octanol/water distribution coefficient. CEP dataset is a subset of the Havard Clean Energy Project (CEP), which estimates the organic photovoltaic efficiency.

Molecular Property: Biophysics Maximum Unbiased Validation (MUV) is another sub-database from PCBA, and is obtained by applying a refined nearest neighbor analysis. HIV is from the Drug Therapeutics Program (DTP) AIDS Antiviral Screen, and it aims at predicting inhibit HIV replication. BACE measures the binding results for a set of inhibitors of β -secretase 1 (BACE-1) and is gathered in MoleculeNet [72].

Molecular Property: Quantum Mechanics QM7, QM8 and QM9 contain stable organic molecules with up to 7, 8 or 9 heavy atoms. 3D atomic coordinates as well as electrical properties of molecules were calculated with ab initio Density Functional Theory (DFT).

Molecular Property: COVID-19 SARS-CoV-2 [7] is a collection of datasets 3 generated by screening a panel of SARS-CoV-2-related assays against approved drugs. 13 assays of 14,332 drugs were used in our experiments.

Statistics of the datasets are in Table 7.

B Results on QM9 dataset with 12 tasks

We show the results of each task of QM9 dataset in Table 1. As can be observed, the proposed IFM module can consistently improve the performance of base models and can even outperform the SPN with 3D geometry information as inputs in most tasks.

C The influence of the hyperparameters k and σ in IFM

k and σ are hyper-parameters in IFM and they are tuned for the optimal performance using the validation set. We show their tunning space and corresponding results in Table 9 and Table 10, respectively. As can be observed, the hyper-parameter k barely impacts the performance of IFM. Larger k would undesirably incur more computational overhead. In contrast, smaller or larger σ would degrade the performance of IFM dramatically. In other words, the hyper-parameter σ is more important and we are encouraged to set larger tuning space for σ to pick the optimal value for each dataset using the validation set.

D Performance of Non-deep models (SVM, RF, and XGB) equipped with IFM

In Table 11, we also show the results of non-deep models (SVM, RF, and XGB) equipped with IFM. As can be observed, IFM barely impacts the performance of non-deep models.

E The performance of CNN, RNN, and TRSF on smoothed molecular datasets

Due to the limited space in the main text, we report the results of CNN, RNN, and TRSF on smoothed molecular datasets in Table 12 here.

F Standard deviations of the benchmarking results and main results

For the limited space in the main text, we report the standard deviations of the benchmarking results and main results here in Table 13 and Table 14, respectively.

G Discussions on the previous feature mapping methods

Gaussian Feature Mapping [62] (GM): Given the molecular feature as x ∈ R d , f x = [cos(2 π B x ) , sin(2 π B x )] T , where each entry in B ∈ R m × d is sampled from N ( 0 , σ 2 ) . In our experiments, we use an isotropic Gaussian distribution.

Sinusoidal Feature Mapping (SM): The general form of Sinusoidal Feature Mapping in previous works [44, 87] can be formulated as f x = [ . . . , cos ( 2 πσ j/m x ) , sin ( 2 πσ j/m x ) , . . . ] T for j = 0 , · · · , m -1 .

The superiority of IFM over SM and GM: The matrix-vector product in GM would undesirably mix different dimensions of molecular features, our IFM avoids this issue. Moreover, the parameters c in IFM is learnable while the parameters B are sampled from a fixed Gaussian distribution. Additionally, the feature mapping in SM is deterministic and only contains on-axis frequencies, making it naturally biased towards data that has more frequency content along the axes. In contrast, our IFM allows all directions to share the same frequency content.

H Hardware

For all our benchmarks and experiments, we use the GPUs including NVIDIA V100, and NVIDIA A100 GPUs based on availability.

I Hyper-parameters tuning space

For SVM , we adopt the widely-used radial basis function (RBF) as the kernel and we optimize the hyper-parameters within the following ranges: C (0.1 to 100) and gamma values (0 to 0.2).

For XGB , we penalize the complexity of the traditional gradient boosting model and perform shrinkage and column subsampling to prevent over-fitting. Also, we employ sparsity-aware split finding technique for efficient training on sparse data, etc. In the experiments, we tune the hyper-parameters within the following ranges: learning_rate (0.01 to 0.2), gamma (0 to 0.2), min_child_weight (1 to 6), subsample (0.7 to 1.0), colsample_bytree (0.7 to 1.0), max_depth (3 to 10) and n_estimators (50, 100, 200, 300, 400, 500, 1000).

For RF , in the implementation of RF algorithm, sample perturbation via bootstrap sampling of the training data and feature perturbation via random feature subset selection are introduced to improve the diversity of base learners (decision trees), which corrects for the overftting habit of decision trees and subsequently enhances the generalization ability of RF. In the experiments, we optimize the

hyper-parameters within the following ranges: n_estimators (50, 100, 200, 300, 400, 500, 1000), max_depth (3 to 12), min_samples_leaf (1,3,5,10,20,50), min_impurity_decrease (0 to 0.01) and max_features ('sqrt', 'log2', 0.7, 0.8, 0.9).

For 1D CNN , we optimize the hyper-parameters within the following ranges: learning rate ( 5 × 10 -4 , 5 × 10 -5 , 5 × 10 -6 ), the convolution kernel size (4, 8, 10), and the number of the hidden features in the fully connected layer (128, 256, 512, 1024).

For RNN , we optimize the hyper-parameters within the following ranges: the hidden dimensions of each layer ([64, 128, 256, 512]), L 2 regularization (0 to 0.01), dropout rate (0 to 0.5). The other hyperparameters are the same as the previous work [25].

For TRSF , we optimize the hyper-parameters within the following ranges: the hidden dimensions of each layer ([64, 128, 256, 512]), L 2 regularization (0 to 0.01), dropout rate (0 to 0.5). The other hyperparameters are the same as the previous work [25].

For MLP , we optimize the hyper-parameters within the following ranges: L 2 regularization (0 to 0.01), dropout rate (0.0 to 0.5), and neurons for each hidden layer (64, 128, 256, 512). The other important hyper-parameters were fixed: ReLU function that has been recommended by many previous studies was used as the activation function, and the optimizer was set to an adaptive learning rate algorithm: Adadelta [83].

For MPNN , we optimize the hyper-parameters within the following ranges: L 2 regularization (0, 10 × 10 -8 , 10 × 10 -6 , 10 × 10 -4 ), learning rate ( 10 × 10 -2 . 5 , 10 × 10 -3 . 5 , 10 × 10 -1 . 5 ), dimension of node feature in hidden layers (64, 32, 16), dimension of edge feature in hidden layers (64, 32, 16), and the number of set2set layers (2,3,4). The number of message passing steps and set2set steps were fixed to 6.

For GCN , we optimize the hyper-parameters within the following ranges: L 2 regularization (0, 10 × 10 -8 , 10 × 10 -6 , 10 × 10 -4 ), learning rate ( 10 × 10 -2 . 5 , 10 × 10 -3 . 5 , 10 × 10 -1 . 5 ), dimension of the FNN classifier (64, 128, 256), and the dimension of GCN hidden layers ([128, 128], [256, 256], [128, 64], [256, 128]).

For GAT , we optimize the hyper-parameters within the following ranges: L 2 regularization (0, 10 × 10 -8 , 10 × 10 -6 , 10 × 10 -4 ), learning rate ( 10 × 10 -2 . 5 , 10 × 10 -3 . 5 , 10 × 10 -1 . 5 ), dimension of GAT hidden layers ([128, 128], [256, 256], [128, 64], [256, 128]), dimension of FNN classifier (64, 128, 256), and the number of attention heads ([2,2], [3,3], [4,4], [3,4], [2,3]).

For AFP , we optimize the hyper-parameters within the following ranges: L 2 regularization (0, 10 × 10 -8 , 10 × 10 -6 , 10 × 10 -4 ), learning rate ( 10 × 10 -2 . 5 , 10 × 10 -3 . 5 , 10 × 10 -1 . 5 ), the number of attentive layers for atom embedding (2,3,4,5,6), the number of attentive layers for molecule embedding (1,2,3,4,5), dropout rate (0, 0.1, 0.3, 0.5), and fingerprint dimension (50, 100, 200, 300).

For SPN , we optimize the hyper-parameters within the following ranges: the distance LB2 intermediate size (4, 8, 16), the angle LB2 intermediate size (4, 8, 16), the torsion LB2 intermediate size (4, 8, 16), the number of interaction blocks (3, 4, 5), the number of spherical harmonics (3, 5, 7), the cutoff distance (4, 5, 6), the batch size (32, 64), the initial learning rate ( 1 × 10 -4 , 5 × 10 -4 , 1 × 10 -3 ), the learning rate step size (50, 100, 150), the learning rate decay ratio (0.4, 0.5, 0.6). The other settings are the same as the original paper [38].

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors systematically study the performance of deep vs. non-deep machine-learned models in the setting of molecular property prediction, assessing the impact of input representations and models on 15 representative datasets taken from MoleculeNet, a standard benchmark in molecular property prediction. \n\nThe authors undertake a study into why non-deep models perform well on these molecular benchmark datasets, including experimentation with smoothing molecular data, and using activity cliff pairs as an example of non-smooth data. There is discussion of two possible explanations as to why non-deep models perform well in molecular property prediction, focusing on smoothness of data and mixing of input features.\n\nSTRENGTHS:\nThe strength of this paper lies in its exploration of possible explanations for poor performance of deep-learning approaches in molecular property prediction, as this has not been systematically addressed previously. The exploration of the smoothness of the target function is a potentially valuable addition to the understanding of the field. Discussion is clear and concise, and experiments useful. The explanations for poor performance of non-deep models are plausible and well-supported by the experiments. It is helpful that the authors focused on the MoleculeNet benchmark as this is a good standard in the community.\n\nWEAKNESSES:\nIn the related work section, there is a good description of the different approaches to representation and modelling of molecular properties. However, this might be better termed \"background\" or introduction necessary to understand the methods benchmarked rather than related work.  There is no discussion of works actually related to the topic of the paper -- that is, previous work that attempts to benchmark a variety of methods -- such as the MoleculeNet paper itself (Wu et al., 2018) and works building on that which include a variety of non-deep methods, and the range of few-shot learning works in molecular property prediction that also explore the MoleculeNet benchmark datasets (Nguyen et al. 2020 ICML GRL+, Pappu et al. 2020 CoRR), as well as similar works exploring performance of non-deep methods on small property prediction datasets (eg. Stanley et al. 2021). \n\nTherefore, the authors must include a thorough discussion of these works, and the current results, including those reported on the MoleculeNet website for a range of non-deep techniques such as xgb. \n\nAt the beginning of section 4, the authors assert that \"unlike image data, molecular data patterns are non-smooth\" -- this is a somewhat strong assertion to make without evidence, particular as the authors then go on to smooth data with regression labels, which inherently images are not. This statement needs some clarification; overall, it would be useful to more tightly define what is meant by \"smooth\" data in this section. \n\nIn addition, the authors mention pretraining on molecules with IFM and note that it boosts the performance of deep models to use a pretraining and finetuning and pretraining paradigm. However it is not clear how this fits with the explanations given for the poor performance of such models in non-smooth datasets. Do the authors claim that this pretraining is somehow reducing the requirement for smooth data? Clarification and experimentation around this point would be useful. \n\nOverall while the paper attempts to address the possible explanations for poor performance of deep models in molecular property prediction, a major weakness is that it is hard to conclusively state that these reasons are in fact absolute truth, especially given that there are a number of other issues in that domain that are equally pressing (eg. the small size of datasets from a deep-learning perspective, which the authors do not address). The ideas are helpful and explored well but inconclusive.\n\nQUESTIONS:\nAs stated above, the authors mention pretraining on molecules with IFM and note that it boosts the performance of deep models to use a pretraining and finetuning and pretraining paradigm. However it is not clear how this fits with the explanations given for the poor performance of such models in non-smooth datasets. Do the authors claim that this pretraining is somehow reducing the requirement for smooth data? Clarification and experimentation around this point would be useful.\n\nLIMITATIONS:\nLimitations of the work are not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The paper evaluates various (non-)deep models for molecular property prediction. Tree models using molecular fingerprints as inputs are shown to perform better than other competitors.\n- The paper claims also to contain an extensive empirical investigation into the unique patterns of molecule data and inductive biases of various models underlying these phenomena. \n- It also presents a feature mapping method for molecule data prior to feeding them into deep models.\n\nSTRENGTHS:\n- The topic certainly offers various interesting research directions and the paper mentions open questions.\n\n- The study considers datasets beyond Moleculenet, namely, datasets related to COVID-19 and activity cliff datasets.\n\n- The analysis in terms of smoothness is interesting, even if we know it. Basically, the paper shows that the molecular data often does not match this model bias of deep learning.\n\n- The feature mapping is an interesting novelty but needs some clarification (see below).\n\nWEAKNESSES:\n- I do not fully understand why this paper was not submitted to the datasets & benchmark track, the research part seems minor (large parts are in the appendix), l67ff.:\nWe provide the most comprehensive benchmark on MPP tasks to date and expose the limitations of deep models on molecule datasets. Our findings offer new and valuable insights for the fast-growing AIDD community.\n\n\n- There are important related works the paper is missing, which contain similar results, e.g.\n  - Tom et al. Calibration and generalizability of probabilistic models on low-data chemical datasets with DIONYSUS, RSC Digital Discovery 2023\n  - Deng et al. Taking a Respite from Representation Learning for\nMolecular Property Prediction, arXiv preprint, arXiv:2209.13492, 2022\n\n- Experimental Setting\n  - The 2D baselines considered besides MPNN seem inappropriate to me. The study applies GCN and GAT, which do not even consider edge features. Models accepted by the community would more fitting, e.g., D-MPNN (Yang et al, 2019).\n  - The study focuses on random splits while the scaffold split setting was suggested, e.g., with Moleculenet or the foundational study on GNN SSL by Hu et al. 2019, and is standard in all SOTA works. \n\n- l. 174, Observation 2. It is irregular data patterns, NOT solely the small size of molecular datasets to blame for the failure of deep models! - as mentioned above, exactly because of this, the scaffold splitting has been accepted by the community. Either the authors missed that or I am misunderstanding the point.\n\n- l. 183, Observation 3. Tree models (XGB and RF) exhibit a particular advantage over other models.\nGiven the above-mentioned works, I assume every researcher in the field runs some simple fingerprint-based baselines for comparison. Hence I do not think this is a novel finding for the community.\n\n- To me, it's not clear in how far the feature mapping is better than the commonly used mapping to dense features (i.e., using torch.Embedding). Is the latter mapping used in the baselines? I would also suggest to move the theoretical justification from the appendix to the main text.\n\nQUESTIONS:\nJust the above-mentioned question about the feature mapping.\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates why classical ML models (i.e. anything except neural networks) often outperform neural networks on molecular property prediction tasks.\n\nThe paper first presents a large study evaluating many deep and non-deep methods on a variety of datasets (seems to be mostly MoleculeNet but I was unsure of this detail). The conclusions are:\n\n1. Classical models do tend to perform better\n2. Dataset size is not the only factor responsible for low performance of deep methods\n3. Tree methods (e.g. random forest + XGBoost) do particularly well\n\nThe authors speculate that this is due to molecular functions not being smooth and mixing of input features. They do some experiments which provide some evidence for this (although I don't think they are conclusive, more on this below).\n\nThe paper then proposes to fix this by transforming the input features with a sinusoidal transformation, and show experimentally that this improves performance.\n\n**EDIT**: I read the rebuttal and it improved my opinion of the paper, but only slightly. I still believe the significance/novelty of the findings is not super high.\n\nSTRENGTHS:\nI think the biggest strength of this paper is the problem choice: not many works study this question and I think it is arguably *the* most important question in research on ML for molecules.\n\nI also think this paper has some good ideas for studying this problem: e.g:\n- looking at performance on activity cliff vs non-activity cliff molecules.\n- doing experiments on an artificially smoothed version of the dataset\n- performing experiments with artificially mixed input features\n\nUnfortunately I don't think the experiments fully support the authors' conclusions, but I will discuss that further in weaknesses.\n\nThe idea of feature transformation is interesting but not very well-explained.\n\n**Originality / quality / clarity / significance**\n\nMy explicit opinion on these aspects of the paper (also accounting for the weaknesses in the next section) are:\n\n- *Originality*: although this is the only paper I know of which specifically addresses deep vs non-deep models for molecules, this question has been previously studied in more general settings (e.g. https://arxiv.org/abs/2108.13637) and countless papers in cheminformatics have compared NNs and classical methods. The idea of studying activity cliffs was studied of course in the paper introducing the activity cliff dataset. Therefore I think the originality of the paper is not super high, although there is some originality.\n- *Quality*: I think the key analysis of the paper is not very thorough, so I must unfortunately rate this as medium/low.\n- *Clarity*: I thought this was ok, but the paper was made less clear because many important details were relegated to the appendix without any indication of where this information could be found.\n- *Significance*: I could imagine some ideas from this paper being referenced in future work, but I think the conclusions are not fully supported by the experiments (more next section) so I think the significance of the paper is fairly limited.\n\nWEAKNESSES:\nThere are a few weaknesses I would like to point out in this paper, which I will list roughly in order of importance:\n\n**Experiments don't seem to support conclusions**\n\nSpecifically:\n- In section 3, the authors conclude that \"irregular data patterns, NOT solely the small size of molecular datasets [are] to blame for the failure of deep models\". However, \"irregular data patterns\" is not really defined: what makes a molecule pattern \"irregular\"? This seemed to just be giving a name to whatever phenomenon causes dataset size to not totally predict performance. The other 2 conclusions from this section are supported but just confirm what is in numerous previous works.\n- In section 4, \"smoothness\" is not defined precisely. Although deep models perform better than non-deep models in Figure 2, the error for all methods decreases sharply, and the differences between all methods are small compared to the overall change in performance, so the statistical significance of this result is questionable. My conclusion would just be that the smoothed tasks are much easier! Also, the number of datasets is very limited. Perhaps the authors could extend this to the classification dataset by using a mixture of labels, e.g. like MIXUP (https://arxiv.org/abs/1710.09412)\n- Similarly, in section 4 the authors conclude from their experiments on \"activity cliff\" and \"non-activity cliff\" molecules that deep models are \"less sensitive to subtle structural changes and struggle to learn non-smooth target functions\" because the performance difference between activity cliff and non-activity cliff molecules is small. I think this cannot be concluded by looking at error rates. You would need to look at how the predictions change as the input structure is perturbed. The overall error rate is high for both cliff and non-cliff molecules, so I would instead conclude that deep model just fit poorly to both the cliff and non-cliff datasets, and that smoothness is *not* affecting the quality of the fit.\n- At the end of section 4 the authors conclude that molecular features should not be mixed, even though this does not affect the performance of deep models. I agree that one would expect this to worsen the performance of tree models, but as the authors themselves note a deep model could learn the inverse transformation and therefore be less affected. It seemed like the authors concluded the opposite of what I expected?\n- Experiments with the sinusoidal featurization in section 6 are not comprehensive: the authors don't seem to control for many other factors including regularization, early stopping, etc. There are too many factors going into neural network training to conclude that their featurization is effective from such a simple study.\n- The tables do not contain error bars, so the statistical significance of performance differences are unclear\n\n**Clarity**\n\nThe paper was not very clear, with many details omitted or put into unspecified areas of the appendix. Some key thing which were unclear are:\n\n- The fingerprint featurization in section 3.1: how many dimensions were dropped? Are the vectors no longer sparse?\n- Neural network training: how was regularization handled? Was the number of parameters varied to avoid overfitting?\n- The featurization in section 5 is barely described...\n\n**Related work**\n\nThe authors did not comment on numerous other works which compare deep and non-deep methods in some capacity, when this is arguably the most important related work. The related work section only introduced the descriptors.\n\nQUESTIONS:\nSee weaknesses section. I would mainly like the authors to respond to my questions about whether their experiments support their conclusions.\n\nLIMITATIONS:\nThe main limitation of the study is that the conclusions are not well-supported in my opinion.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors demonstrate, through a large number of experiments across many datasets and tasks, that tree-based models on molecular fingerprints consistently outperform deep learning models in molecular property prediction, and attribute this to 1) the \"non-smooth\" nature of molecular labels (e.g. the existence of \"activity cliffs\"), and 2) the manner in which deep learning models mix molecular features. Given these conclusions, the authors propose a novel molecular feature pre-processing method called Independent Feature Mapping (IFM) and show that using IFM-transformed features as inputs to deep learning models allows them to outperform tree-based models.\n\nSTRENGTHS:\n### Originality\n\nThe authors are certainly not the first to point out that deep learning methods are often outperformed by non-deep models, especially tree-based models. However, the non-trivial explanation for this phenomenon in terms of the \"non-smooth\", high-frequency nature of molecular labels and the spectral bias of deep learning methods, as well as the introduction of IFM to deal with these issues, are important contributions to the understanding of how to improve molecular property prediction generally.\n\n### Quality\n\nThe quality of the paper is high, containing a large number of relevant experiments across various datasets, tasks, and models. The appendix suggests that a suitable amount of hyperparameter tuning was performed for each model.\n\n### Clarity\n\nThe paper is mostly clear, though a number of improvements can be made (see Weaknesses)\n\n### Significance\n\nAccurate OOD molecular property prediction is an incredibly valuable tool for drug discovery and material design, and this paper makes an important contribution to this area of research.\n\nWEAKNESSES:\n- A number of the tables and figures could be significantly improved to make them more easily comprehended. For example:\n  - the lack of standard deviations in Table 1 makes the conclusions less significant, please move from appendix to main text.\n  - Figures 2 and 4 should be summarized as tables and only the percent difference from 0-smooth or Original, respectively, displayed\n  - Tables 5 and 6 could be moved to the appendix to make room.\n- The caption of Figure 2 claims that the smoothing experiments were only done for regression tasks because smoothing is unsuitable for classification, but this is not true - cross-entropy loss can be applied to both one-hot and dense labels. Please provide a classification example.\n- In Explanation 2, the following claim is made without evidence: \"The learning style of tree models is more suitable for molecular data because only a handful of features (e.g., certain substructures) are most indicative of molecular properties.\" Please provide evidence for this claim.\n- In Section 6.2, the following statement is made: \"mixing different dimensions of molecular features as SM [33] and GM [48] would degrade the performance.\" However, only GM mixes features. Please fix.\n- The Discussion section is short and weak. Possible limitations of the proposed approach are not discussed. Please update.\n\nQUESTIONS:\n- Why is batch size not treated as a hyperparameter? Section 3.1 suggests it is kept fixed at 128.\n- The molecules in Figure 3 would have very similar structural fingerprints, so why would tree-based methods on these fingerprints do well on this example? Is there a better example that could illustrate the point being made?\n\nLIMITATIONS:\nNo mention of limitations is made.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors compared various deep learning and traditional non-deep machine learning model performance on multiple molecular property prediction tasks. Their findings indicate that deep models tend to underperform non-deep ones due to the irregular patterns of molecular data and relatively small data size. To that end, they proposed to apply a feature mapping technique to help deep models capture high frequency patterns, and demonstrated the effectiveness on benchmark datasets.\n\nI find this work more like a technical report providing comprehensive benchmark results. My main concerns include (1) the conclusions are somewhat expected and probably already well-known in the community (e.g., the sharp vs smooth decision boundary issue); (2) it seems that the main focus of this work is about feature engineering, while the proposed methodology (i.e., Fourier-type feature mapping) exhibits insufficient novelty. \n\nTherefore, I do not recommend publishing this manuscript to NeurIPS.\n\nSTRENGTHS:\n- The authors provided comprehensive benchmark results on many commonly-used molecular property prediction datasets.\n- Comparing the performance amongst non-deep and deep learning models on molecular property prediction is a significant research topic.\n\nWEAKNESSES:\n- Insufficient novelty. The proposed methodology is well-known (e.g., NeRF, Fourier features, etc.).\n- The main idea is about feature engineering, while the set of optimal features cannot be theoretically guaranteed.\n- The comparison between deep and non-deep models on these molecular datasets might be unfair for deep models due to the data size. Although the authors claimed that irregular data patterns is another important factor explaining their predictive power, larger data size (i.e., the scaling law) indeed tends to improve deep model generalizability (e.g., see [pvd](https://arxiv.org/abs/2206.00133)).\n\nQUESTIONS:\n- The molecular similarity from a visual perspective (e.g., Figure 3) is quite intuitive for us, but does it necessarily mean deep models cannot find an appropriate latent space where these \"cliff molecules\" are far apart?\n- With increasing data size, what is the trend of deep vs non-deep model performance?\n- Since the conclusion is that tree-based models tend to outperform others, does it mean rule-based models are superior to deep models?\n- I think Figure 2 and 4 are a bit misleading. The scale of y-axis has been engineered to enhance the numerical difference, imo this is not appropriate, especially for AUROC plots (i.e., baseline 0.5).\n- Line 245, the authors mentioned \"convex combination\", yet my understanding is that $\\hat{x}_i$ underwent orthogonal transformation Q from $x$, which is not necessarily a convex combination of $x$?\n\nLIMITATIONS:\n- The over-smoothing issue of GNNs could be discussed in the manuscript to further support their claims.\n- For molecules without available 3D structures, RDKit-generated conformers are used. However, these generated structures (probably relaxed with MLFF) are not as accurate as DFT-relaxed ones -- which might confuse 3D structure-based deep models.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}