SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Explanation 2. Deep models undesirably mix different dimensions of molecular features, whereas tree models make decisions based on each dimension of the features separately.
Figure 4: The performance of various models on the orthogonally transformed datasets. Left: FreeSolv (Regression); Middle: ClinTox (Classification); Right: Tox21 (Classification). Kindly note that we did not evaluate CNN, RNN, and TRSF on the transformed datasets because we cannot apply the orthogonal transformations to the input SMILES strings.
Typically, features in molecular data carry meanings individually. As we elaborated in Sec. 2.1, each dimension of molecular fingerprints often indicates whether a certain substructure is present in the molecule; each dimension of nodes/edges features in molecular graph data indicates a specific characteristic of the atoms/bonds (e.g., atom/bond type, atom degree). To verify the above explanation, we mix the different dimensions of molecular features  $x_{i} \in \mathbb{R}^{d}$  using an orthogonal transformation before feeding them into various models,
$$
\widehat {x _ {i}} = \mathcal {Q} x _ {i}, \tag {2}
$$
where  $\mathcal{Q} \in \mathbb{R}^{d \times d}$  is the orthogonal matrix and  $\widehat{x_i}$  is the molecular feature after transformation. Kindly note that the meaning of  $x_i$  depends on the input molecular descriptors in the experiments. Specifically, for SVM, XGB, RF, and MLP,  $x_i$  denotes the molecular fingerprints; for GNN models,  $x_i$  can denote the atom features and bond features in the molecular graphs, i.e., we apply orthogonal transformations to both the atom features and bond features. As can be observed in Figure 4, the performance of tree models deteriorates dramatically and falls behind most deep models after the orthogonal transformation. It is because each dimension of  $\widehat{x_i}$  is a linear combination of all the dimensions of  $x_i$  according to the matrix-vector product rule. In other words, the molecular features after orthogonal transformation no longer carry meanings individually, accounting for the failure of tree models that make decisions based on each dimension of the features separately. The learning style of tree models is more suitable for molecular data because only a handful of features (e.g., certain substructures) are most indicative of molecular properties [47]. On the other hand, the performance decreases of deep models are less significant, and most deep models can beat tree models after the transformations. We explain this observation as follows. Without the loss of generality, we assume that a linear layer of deep models can map the original molecular feature  $x_i$  to the label  $y_i$ ,
$$
y _ {i} = W ^ {\top} x _ {i} + b, \tag {3}
$$
where  $W$  and  $b$  denote the parameter matrix and the bias term of the linear layer, respectively. And then, we aim to learn a new linear layer mapping the transformed model feature  $\hat{x}_i$  to label  $y_i$ ,
$$
y _ {i} = \widehat {W} ^ {\top} \widehat {x} _ {i} + b = \widehat {W} ^ {\top} Q x _ {i} + \hat {b}, \tag {4}
$$
where  $\widehat{W}$  and  $\hat{b}$  denote the parameter matrix and the bias term of the new linear layer, respectively. Apparently, to achieve the same results as the original feature, we only have to learn  $\widehat{W}$  so that  $\widehat{W} = QW$  because  $\mathcal{Q}^{-1} = \mathcal{Q}^{\top}$  as an orthogonal matrix, and also  $\hat{b} = b$ . Therefore, applying the orthogonal transformation to molecular features barely impacts the performance of deep models. The empirical results in Figure 4 confirm this point although some performance changes are observable due to uncontrollable random factors.
Although we have shown and explained the superiority of non-deep models on molecular data, deep models have numerous advantages over the traditional approaches: (i) GNNs can exploit the structural/geometrical inductive biases of 2D/3D molecular graphs, alleviating the manual efforts to capture the topology of the networks; (ii) The pre-trained representations with deep models are beneficial to various downstream tasks, showing promising values in drug discovery [75, 74, 9, 73]. Therefore, we aim to empower the deep models to beat the non-deep ones on MPP tasks including activity cliff cases.
As we explained before, deep models struggle to learn the non-smooth target functions of molecular data, a phenomenon referred to "spectral bias" in literature [51]. To overcome such bias, prior works [44, 87] have experimentally found that a heuristic sinusoidal mapping of the input features allows MLPs to learn the non-smooth target functions. However, these mapping methods would undesirably mix the original features. Please refer to the appendix for detailed discussions due to the limited space. As a remedy, we introduce a new method named Independent Feature Mapping (IFM) that embeds each dimension of molecular features separately before feeding them into models. Denoting a molecular feature as  $x \in \mathbb{R}^d$ , we formulate IFM as,
$$
f _ {x} = [ \sin (v) | | \cos (v) ], \quad v = [ 2 \pi c _ {1} x, \dots , 2 \pi c _ {k} x ], \tag {5}
$$
where  $||$  denotes the concatenation of two vectors,  $\pmb{c} = [c_1, c_2, \dots, c_k]$  are the learnable parameters initialized from  $\mathcal{N}(0, \sigma)$  and  $f_x \in \mathbb{R}^{2k \times d}$ . We study the influence of the hyperparameters  $k$  and  $\sigma$  in the appendix. Since  $\cos(a - b) = \cos a\cos b + \sin a\sin b$ , we have,
$$
f _ {x} \cdot f _ {x ^ {\prime}} = \sum_ {i = 1} ^ {k} \cos \left(2 \pi c _ {i} \left(x - x ^ {\prime}\right)\right) := g _ {\mathbf {c}} \left(x - x ^ {\prime}\right), \tag {6}
$$
where  $\cdot$  is the dot product and  $x^{\prime}$  is another molecular feature. Therefore, IFM can map data points to a vector space so that their dot product achieves a certain distance metric, which is an expected characteristic for feature mapping methods [52, 4, 23]. In what follows, we will provide theoretical justifications on the effectiveness of our IFM following a previous study [62]. As revealed in previous works, deep models can be approximated with Neural Tangent Kernel (NTK) [28, 2, 6, 36, 62]. Specifically, let  $I$  be a fully-connected deep network with weights  $\theta$  initialized from a Gaussian distribution  $\mathcal{N}$ , the NTK theory shows that as the width of the layers in  $I$  becomes infinite and the learning rate for stochastic gradient descent (SGD) approaches zero, the function  $I(x;\theta)$  converges during training to the kernel regression solution using the neural tangent kernel (NTK), which is:
$$
h _ {\mathrm {N T K}} \left(x, x ^ {\prime}\right) = \mathbb {E} _ {\theta \sim \mathcal {N}} \left\langle \frac {\partial I (x ; \theta)}{\partial \theta}, \frac {\partial I \left(x ^ {\prime} ; \theta\right)}{\partial \theta} \right\rangle \tag {7}
$$
When the inputs are limited to a hypersphere, the NTK for an MLP can be expressed as a dot product kernel (a kernel in the form  $h_{\mathrm{NTK}}(x \cdot x')$  for a scalar function  $h_{\mathrm{NTK}}: \mathbb{R} \to \mathbb{R}$ ). In our cases, the input to the deep models would be  $f_x$ , the composed kernel of IFM and NTK can be formulated as,
$$
h _ {\mathrm {N T K}} \left(f _ {x} \cdot f _ {x ^ {\prime}}\right) = h _ {\mathrm {N T K}} \left(g _ {c} \left(x - x ^ {\prime}\right)\right) = \left(h _ {\mathrm {N T K}} \circ g _ {c}\right) \left(x - x ^ {\prime}\right), \tag {8}
$$
thus, training deep models on these mapped molecular features corresponds to kernel regression with the stationary composed NTK function  $h_{\mathrm{NTK}} \circ g_c$ . Considering that the parameters  $c$  are tunable, IFM creates a composed NTK that is not only stationary but also tunable. It enables us to dramatically control the range of frequencies that can be learned via manipulating the parameters  $c$ .
In our experiments, we equip various deep models with IFM. Specifically, for MLPs with fingerprints as inputs, we employ the proposed feature mapping method to the fingerprints (after feature selection and standardization) directly; For molecular graphs, we map both the features of atoms and bonds before feeding them into the GNNs. The other settings are the same as the benchmarking experiments. If a deep model named  $\mathbf{z}$  (e.g., MLP) is equipped with IFM, we re-name it as 'IFM-z' (e.g., IFM-MLP) in our results. Also, we evaluate the non-deep models equipped with IFM in the appendix.
Table 3: The performance comparison on multiple molecular datasets. The best performance on each dataset is highlighted with bold. The 'P-Best (Model)' denotes the best result in Table 1 and its corresponding model name. The results of 12 tasks on QM 9 can be seen in the appendix.
Dataset (No.)	Metric	MLP	GCN	MPNN	GAT	AFP	P-Best (Model)	IFM-MLP	IFM-GCN	IFM-MPNN	IFM-GAT	IFM-AFP
BACE (1,513)	AUC_ROC	0.887	0.880	0.846	0.886	0.879	0.896 (XGB)	0.915	0.903	0.866	0.894	0.907
HIV (4,0748)	AUC_ROC	0.791	0.834	0.814	0.812	0.819	0.834 (GCN)	0.816	0.862	0.846	0.838	0.849
BBBP (2,035)	AUC_ROC	0.918	0.915	0.872	0.902	0.893	0.926 (XGB)	0.937	0.945	0.908	0.933	0.940
ClinTox (1,475)	AUC_ROC	0.890	0.889	0.868	0.891	0.907	0.963 (TRSF)	0.941	0.938	0.929	0.953	0.959
SIDER (1,366)	AUC_ROC	0.617	0.633	0.603	0.614	0.620	0.644 (RF)	0.646	0.649	0.638	0.647	0.652
Tox21 (7,811)	AUC_ROC	0.834	0.830	0.816	0.829	0.845	0.845 (AFP)	0.842	0.839	0.837	0.849	0.853
ToxCast (8,539)	AUC_ROC	0.781	0.767	0.736	0.768	0.788	0.788 (AFP)	0.795	0.790	0.772	0.797	0.806
MUV (93,087)	AUC_PRC	0.018	0.056	0.019	0.055	0.044	0.093 (SVM)	0.052	0.113	0.068	0.124	0.097
SARS-CoV-2 (14,332)	AUC_ROC	0.638	0.646	0.640	0.683	0.651	0.700 (XGB)	0.675	0.682	0.686	0.716	0.704
ESOL (1,127)	RMSE	0.653	0.773	0.695	0.661	0.594	0.583 (XGB)	0.587	0.728	0.673	0.566	0.561
Lipop (4,200)	RMSE	0.633	0.665	0.669	0.680	0.664	0.585 (XGB)	0.556	0.577	0.568	0.584	0.578
FreeSolv (639)	RMSE	1.046	1.316	1.327	1.304	1.139	0.715 (XGB)	0.862	0.916	0.911	0.908	0.883
QM7 (6,830)	MAE	86.060	64.530	107.013	78.217	59.973	42.814 (SVM)	66.570	38.793	84.918	59.595	33.775
QM8 (21,786)	MAE	0.0104	0.0154	0.0109	0.0187	0.0098	0.0098 (AFP)	0.0091	0.0114	0.0085	0.0139	0.0079
Table 4: The results on activity cliff datasets. The best result for each dataset is highlighted in 'bold'.
Target name (Response type)	Metric	MLP	GCN	MPNN	GAT	AFP	P-Best (Model)	IFM-MLP	IFM-GCN	IFM-MPNN	IFM-GAT	IFM-AFP
CB1 (Agonism EC50)	RMSEc	0.807	0.992	0.989	0.975	0.967	0.767 (XGB)	0.715	0.748	0.756	0.741	0.746
DAT (Inhibition Ki)	RMSEc	0.792	1.003	0.921	1.042	0.995	0.696 (XGB)	0.646	0.682	0.673	0.665	0.670
PPARα (Agonism EC50)	RMSEc	0.713	0.870	0.872	0.929	0.823	0.671 (SVM)	0.623	0.634	0.649	0.661	0.616
DOR (Inhibition Ki)	RMSEc	0.874	1.259	1.152	1.281	1.179	0.836 (RF)	0.787	0.813	0.796	0.799	0.810
Main results. We show the main results in Table 3, from which we can make the following observations: (1) The proposed feature mapping method can significantly improve the performance of the deep models on molecular datasets; (2) The deep models equipped with the feature mapping method can beat non-deep counterparts in most cases, verifying the effectiveness of our method.
Results on activity cliffs. We also employ the proposed feature mapping method on activity cliff datasets where the target functions are less smooth. The results shown in Table 4 indicate that IFM improves the deep models by significant margins. Moreover, nearly all the deep models with IFM method can beat traditional methods, confirming that our method can help neural networks learn non-smooth target functions.
Table 5: Comparisons with SM (Sinusoidal Map-Table 6: Pre-training on molecules with encoders equipped ping [44]) and GM (Gaussian Mapping [62]). with IFM. GIN: Graph Isomorphism Network [78].
Methods	BACE	HIV	BBBP	ESOL	Lipop	FreeSolv	Method (Encoder)	Tox21	ToxCast	Sider	ClinTox	BBBP	Bace
MLP + SM	0.890	0.798	0.925	0.640	0.613	1.033	AttrMasking [27] (GCN)	0.745	0.626	0.598	0.724	0.653	0.773
MLP + GM	0.896	0.806	0.923	0.625	0.591	0.992	AttrMasking (IFM-GCN)	0.758	0.639	0.612	0.741	0.667	0.782
IFM-MLP	0.915	0.816	0.937	0.587	0.556	0.862	MGSSL [85] (GIN)	0.752	0.633	0.616	0.771	0.688	0.788
MGSSL (IFM-GIN)	0.764	0.639	0.623	0.783	0.704	0.796
GCN + SM	0.894	0.838	0.920	0.762	0.635	1.204
GCN + GM	0.892	0.847	0.926	0.766	0.609	1.193	Mole-BERT [26] (GIN)	0.768	0.643	0.628	0.789	0.719	0.808
IFM-GCN	0.903	0.862	0.945	0.728	0.577	0.916	Mole-BERT (IFM-GIN)	0.775	0.649	0.627	0.796	0.724	0.813
Comparisons with other feature mapping methods (ablation study). We compare the proposed IFM with the previous feature mapping methods. The results shown in Table 5 indicate that the proposed feature mapping method is superior to the previous method, which verifies that mixing different dimensions of molecular features as GM [62] would degrade the performance.
Pre-training on molecules with IFM. Compared with non-deep models, deep ones can be combined with the prevalent 'Pretraining and Finetuning' paradigm to exploit large-scale unlabeled molecules [75]. This motivates us to develop more powerful neural encoders for this paradigm. Specifically, we pre-train the deep models equipped with IFM and report the fine-tuning results in Table 6. Our experimental settings are the same as the pioneering work [27] where the datasets are split with scaffold splitting, differing from the random splitting in Table 3. As can be observed, our method can boost various pre-training strategies to advance their performance in downstream tasks.
In this paper, we perform a comprehensive benchmark of representative models on molecular property prediction. Our results reveal that traditional machine learning models, especially tree models, can easily outperform well-designed deep models in most cases. These phenomena can be attributed to the unique patterns of molecular data and different inductive biases of various models. Specifically, the target function mapping molecules to properties are non-smooth, and some small changes can incur significant property variance. Deep models struggle to learn such patterns. Additionally, molecular features carry meanings individually and deep models would undesirably mix different dimensions of molecular features. These findings stimulate us to develop a simple-yet-effective feature mapping method for molecule data that can help deep models learn non-smooth target functions with theoretical guarantees. Extensive experiments verify the effectiveness of the proposed method. Our study leaves an open question for future research: Can our findings and methods be generalized to other AIDD tasks including drug-target interactions (DTIs) prediction, drug-drug interactions (DDIs) prediction?
We thank the anonymous the area chairs and reviewers for their constructive and helpful reviews. This work was supported by the National Key R&D Program of China (Project 2022ZD0115100), the National Natural Science Foundation of China (Project U21A20427), the Research Center for Industries of the Future (Project WU2022C043), and the Competitive Research Fund (Project WU2022A009) from the Westlake Center for Synthetic Biology and Integrated Bioengineering.
[1] Tox21 challenge. https://tripod.nih.gov/tox21/challenge/, 2017.
[2] Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. ICML, 2019.
[3] Kenneth Atz, Francesca Grisoni, et al. Geometric deep learning on molecular representations. Nature Machine Intelligence, 2021.
[4] Haim Avron, Vikas Sindhwani, Jiyan Yang, and Michael W. Mahoney. Quasi-monte carlo feature maps for shift-invariant kernels. J. Mach. Learn. Res., 17:120:1-120:38, 2014.
[5] Seth D Axen, Xi-Ping Huang, Elena L Cáceres, Leo Gendelev, Bryan L Roth, and Michael J Keiser. A simple representation of three-dimensional molecular structure. Journal of medicinal chemistry, 60(17):7393-7409, 2017.
[6] Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non-uniform density. arXiv preprint arXiv:2003.04560, 2020.
[7] Kyle R. Brimacombe, Tongan Zhao, Richard T. Eastman, Xin Hu, Ke Wang, Mark Backus, Bolormaa Baljinnyam, Catherine Z. Chen, Lu Chen, Tara Eicher, Marc Ferrer, Ying Fu, Kirill Gorshkov, Hui Guo, Quinlin M. Hanson, Zina Itkin, Stephen C. Kales, Carleen Klumpp-Thomas, Emily M. Lee, Sam Michael, Tim Mierzwa, Andrew Patt, Manisha Pradhan, Alex Renn, Paul Shinn, Jonathan H. Shrimp, Amit Viraktamath, Kelli M. Wilson, Miao Xu, Alexey V. Zakharov, Wei Zhu, Wei Zheng, Anton Simeonov, Ewy A. Mathé, Donald C. Lo, Matthew D. Hall, and Min Shen. An opendata portal to share pandemic data in real time. bioRxiv, 2020.
[8] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm SIGkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
[9] Seyone Chithrananda, Gabriel Grand, et al. ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction. arXiv.org, October 2020.
[85] Zaixi Zhang, Qi Liu, and others. Motif-based graph self-supervised learning for molecular property prediction. NeurIPS, 2021.
[86] Shuangjia Zheng, Xin Yan, Yuedong Yang, and Jun Xu. Identifying structure-property relationships through smiles syntax analysis with self-attention mechanism. Journal of chemical information and modeling, 59(2):914–923, 2019.
[87] Ellen D. Zhong, Tristan Bepler, Joseph H. Davis, and Bonnie Berger. Reconstructing continuous distributions of 3d protein structure from cryo-em images. In International Conference on Learning Representations, 2019.
Table 7: Summary for the molecule datasets in the benchmark.
Dataset	Task	# Tasks	# Molecules
BBBP	Classification	1	2,039
Tox21	Classification	12	7,831
ToxCast	Classification	617	8,576
Sider	Classification	27	1,427
ClinTox	Classification	2	1,478
MUV	Classification	17	93,087
HIV	Classification	1	41,127
Bace	Classification	1	1,513
SARS-CoV-2	Classification	13	14,332
Lipop	Regression	1	4,200
FreeSolv	Regression	1	642
ESOL	Regression	1	1,1128
QM7	Regression	1	7,160
QM8	Regression	16	21,786
QM9	Regression	12	133,885
In this section, we provide detailed information on the molecular datasets used for downstream tasks. Molecular Property: Pharmacology The Blood-Brain Barrier Penetration (BBBP) dataset measures whether a molecule will penetrate the central nervous system. All three datasets, Tox21 [1], ToxCast [72], and ClinTox [19] are related to the toxicity of molecular compounds. The Side Effect Resource (SIDER) dataset stores the adverse drug reactions on a marketed drug database.
Molecular Property: Physical Chemistry Dataset proposed in [12] measures aqueous solubility of the molecular compounds. Lipophilicity (Lipo) dataset is a subset of ChEMBL [18] measuring the molecule octanol/water distribution coefficient. CEP dataset is a subset of the Havard Clean Energy Project (CEP), which estimates the organic photovoltaic efficiency.
Molecular Property: Biophysics Maximum Unbiased Validation (MUV) is another sub-database from PCBA, and is obtained by applying a refined nearest neighbor analysis. HIV is from the Drug Therapeutics Program (DTP) AIDS Antiviral Screen, and it aims at predicting inhibit HIV replication. BACE measures the binding results for a set of inhibitors of  $\beta$ -secretase 1 (BACE-1) and is gathered in MoleculeNet [72].
Molecular Property: Quantum Mechanics QM7, QM8 and QM9 contain stable organic molecules with up to 7, 8 or 9 heavy atoms. 3D atomic coordinates as well as electrical properties of molecules were calculated with ab initio Density Functional Theory (DFT).
Molecular Property: COVID-19 SARS-CoV-2 [7] is a collection of datasets<sup>3</sup> generated by screening a panel of SARS-CoV-2-related assays against approved drugs. 13 assays of 14,332 drugs were used in our experiments.
Statistics of the datasets are in Table 7.
<sup>3</sup>The datasets are available at https://opendata.ncats.nih.gov/covid19/ (CC BY 4.0 license) and are continuously extended. The data used in our experiments were downloaded on February 16th, 2021.
B Results on QM9 dataset with 12 tasks
We show the results of each task of QM9 dataset in Table 1. As can be observed, the proposed IFM module can consistently improve the performance of base models and can even outperform the SPN with 3D geometry information as inputs in most tasks.
Table 8: Comparison of MAE on QM9. The best result is highlighted with 'bold'.
Target Unit	εHOMO eV	εLUMO eV	Δε eV	μ D	α bohr3	R2a02	ZPVE meV	U0meV	U meV	H meV	G meV	cv cal/molK
SVM	0.016	0.038	0.040	1.19	6.35	-	0.026	-	-	-	-	3.22
XGB	0.016	0.039	0.040	1.19	6.36	-	0.026	-	-	-	-	3.22
RF	0.016	0.038	0.039	1.19	6.34	-	0.020	-	-	-	-	3.22
CNN	0.018	0.019	0.032	1.11	1.55	99.27	0.073	3.52	4.09	3.81	4.03	1.08
RNN	0.014	0.028	0.028	0.91	3.53	91.03	0.015	14.83	14.93	15.66	14.84	1.47
TRSF	0.011	0.015	0.017	0.83	1.46	96.67	0.057	1.97	2.95	1.90	1.73	0.92
GCN	0.030	0.015	0.019	0.69	0.77	27.49	0.014	0.55	0.56	0.55	0.55	0.34
MPNN	0.100	0.440	0.410	0.95	1.47	38.96	0.410	2.15	2.19	2.16	2.18	0.93
GAT	0.060	0.012	0.016	0.69	0.85	24.85	0.057	0.79	0.81	0.80	0.80	0.43
AFP	0.089	0.029	0.073	0.90	0.74	27.35	0.110	1.20	1.18	1.19	1.19	0.51
SPN	0.024	0.019	0.032	0.31	0.47	29.26	0.011	0.63	0.73	0.64	0.82	0.20
IFM-MLP	0.013	0.022	0.032	0.87	1.04	125.47	0.015	16.19	16.23	16.20	16.22	0.97
IFM-GCN	0.019	0.011	0.014	0.63	0.53	15.18	0.012	0.46	0.43	0.47	0.44	0.25
IFM-MPNN	0.076	0.371	0.358	0.77	0.94	19.82	0.252	1.88	1.86	1.86	1.87	0.75
IFM-GAT	0.032	0.009	0.010	0.53	0.69	20.15	0.038	0.62	0.63	0.62	0.61	0.29
IFM-AFP	0.009	0.017	0.575	0.74	0.48	14.69	0.096	0.86	0.88	0.87	0.86	0.33
C The influence of the hyperparameters  $k$  and  $\sigma$  in IFM
Table 9: The influence of the hyper-parameter  $k$  (with  $\sigma = 6$ ).
Models / k	4	8	16	32	64
Toxcast (IFM-MLP)	0.782	0.795	0.797	0.789	0.792
Toxcast (IFM-GCN)	0.771	0.780	0.783	0.785	0.774
Sider (IFM-MLP)	0.625	0.631	0.626	0.635	0.636
Sider (IFM-GCN)	0.635	0.638	0.639	0.633	0.632
Table 10: The influence of the hyper-parameter  $\sigma$  (with  $k = 8$ ).
Models / σ	1	3	6	9	12	15
Toxcast (IFM-MLP)	0.767	0.782	0.795	0.791	0.785	0.772
Toxcast (IFM-GCN)	0.763	0.775	0.780	0.790	0.788	0.776
Sider (IFM-MLP)	0.611	0.626	0.631	0.646	0.639	0.613
Sider (IFM-GCN)	0.620	0.633	0.638	0.649	0.643	0.627
$k$  and  $\sigma$  are hyper-parameters in IFM and they are tuned for the optimal performance using the validation set. We show their tuning space and corresponding results in Table 9 and Table 10, respectively. As can be observed, the hyper-parameter  $k$  barely impacts the performance of IFM. Larger  $k$  would undesirably incur more computational overhead. In contrast, smaller or larger  $\sigma$  would degrade the performance of IFM dramatically. In other words, the hyper-parameter  $\sigma$  is more important and we are encouraged to set larger tuning space for  $\sigma$  to pick the optimal value for each dataset using the validation set.
D Performance of Non-deep models (SVM, RF, and XGB) equipped with IFM
In Table 11, we also show the results of non-deep models (SVM, RF, and XGB) equipped with IFM. As can be observed, IFM barely impacts the performance of non-deep models.
Table 11: Performance of Non-deep models (SVM, RF, and XGB) equipped with IFM
Models	Tox21	ToxCast	Sider	ClinTox	BBBP	Bace
SVM	0.820	0.725	0.626	0.879	0.913	0.886
IFM-SVM	0.822	0.724	0.629	0.883	0.907	0.883
RF	0.838	0.778	0.644	0.933	0.923	0.890
IFM-RF	0.843	0.785	0.643	0.935	0.929	0.883
XGB	0.837	0.785	0.638	0.919	0.926	0.896
IFM-XGB	0.833	0.780	0.636	0.911	0.933	0.894
E The performance of CNN, RNN, and TRSF on smoothed molecular datasets
Due to the limited space in the main text, we report the results of CNN, RNN, and TRSF on smoothed molecular datasets in Table 12 here.
Table 12: The performance of CNN, RNN, and TRSF on smoothed molecular datasets.
Smoothing level	ESOL	Lipop	FreeSolv
0-smooth	10-smooth	20-smooth	0-smooth	10-smooth	20-smooth	0-smooth	10-smooth	20-smooth
CNN	2.569	1.937	1.334	1.016	0.524	0.409	2.275	1.592	0.986
RNN	1.511	1.074	0.862	1.207	0.770	0.687	2.205	1.570	1.028
TRSF	0.718	0.667	0.564	0.947	0.549	0.435	1.504	1.175	0.954
F Standard deviations of the benchmarking results and main results