SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For the limited space in the main text, we report the standard deviations of the benchmarking results and main results here in Table 13 and Table 14, respectively.
G Discussions on the previous feature mapping methods
Gaussian Feature Mapping [62] (GM): Given the molecular feature as  $x \in \mathbb{R}^d$ ,  $f_x = [\cos(2\pi \mathbf{B}x), \sin(2\pi \mathbf{B}x)]^{\mathrm{T}}$ , where each entry in  $\mathbf{B} \in \mathbb{R}^{m \times d}$  is sampled from  $\mathcal{N}(0, \sigma^2)$ . In our experiments, we use an isotropic Gaussian distribution.
Sinusoidal Feature Mapping (SM): The general form of Sinusoidal Feature Mapping in previous works [44, 87] can be formulated as  $f_{x} = \left[\dots ,\cos \left(2\pi \sigma^{j / m}x\right),\sin \left(2\pi \sigma^{j / m}x\right),\dots \right]^{\mathrm{T}}$  for  $j = 0,\dots ,m - 1$ .
The superiority of IFM over SM and GM: The matrix-vector product in GM would undesirably mix different dimensions of molecular features, our IFM avoids this issue. Moreover, the parameters c in IFM is learnable while the parameters B are sampled from a fixed Gaussian distribution. Additionally, the feature mapping in SM is deterministic and only contains on-axis frequencies, making it naturally biased towards data that has more frequency content along the axes. In contrast, our IFM allows all directions to share the same frequency content.
Table 13: The standard deviations of main results.
Dataset (No.)	Metric	IFM-MLP	IFM-GCN	IFM-MPNN	IFM-GAT	IFM-AFP
BACE (1,513)	AUC_ROC	0.024	0.028	0.022	0.023	0.025
HIV (40,748)	AUC_ROC	0.022	0.027	0.029	0.033	0.032
BBBP (2,035)	AUC_ROC	0.026	0.029	0.022	0.032	0.035
ClinTox (1,475)	AUC_ROC	0.049	0.042	0.040	0.047	0.046
SIDER (1,366)	AUC_ROC	0.025	0.027	0.028	0.025	0.029
Tox21 (7,811)	AUC_ROC	0.017	0.014	0.015	0.012	0.011
ToxCast (8,539)	AUC_ROC	0.017	0.016	0.020	0.019	0.017
MUV (93,087)	AUC_PRC	0.028	0.032	0.018	0.019	0.023
SARS-CoV-2 (14,332)	AUC_ROC	0.027	0.025	0.020	0.018	0.023
ESOL (1,127)	RMSE	0.082	0.063	0.088	0.065	0.063
Lipop (4,200)	RMSE	0.036	0.073	0.051	0.038	0.046
FreeSolv (639)	RMSE	0.207	0.247	0.258	0.261	0.259
QM7 (6,830)	MAE	0.025	0.028	0.022	0.013	0.024
QM8 (21,786)	MAE	0.0002	0.0003	0.0012	0.0003	0.0003
Table 14: The standard deviations of benchmarking results.
Dataset (No.)	Metric	SVM	XGB	RF	CNN	RNN	TRSF	MLP	GCN	MPNN	GAT	AFP	SPN
BACE (1,513)	AUC_ROC	0.018	0.020	0.023	0.030	0.009	0.007	0.025	0.021	0.020	0.026	0.022	0.021
HIV (40,748)	AUC_ROC	0.019	0.020	0.017	0.009	0.029	0.011	0.019	0.024	0.029	0.033	0.028	0.030
BBBP (2,035)	AUC_ROC	0.029	0.025	0.027	0.026	0.019	0.014	0.028	0.026	0.032	0.034	0.030	0.033
ClinTox (1,475)	AUC_ROC	0.041	0.042	0.039	0.008	0.098	0.007	0.048	0.045	0.043	0.053	0.042	0.046
SIDER (1,366)	AUC_ROC	0.020	0.019	0.017	0.012	0.015	0.009	0.026	0.027	0.024	0.029	0.025	0.024
Tox21 (7,811)	AUC_ROC	0.008	0.011	0.013	0.004	0.007	0.004	0.015	0.018	0.014	0.013	0.012	0.014
ToxCast (8,539)	AUC_ROC	0.007	0.005	0.007	0.006	0.010	0.005	0.018	0.017	0.020	0.021	0.018	0.019
MUV (93,087)	AUC_PRC	0.039	0.026	0.030	0.014	0.017	0.060	0.027	0.035	0.030	0.015	0.022	0.026
SARS-CoV-2 (14,332)	AUC_ROC	0.110	0.010	0.013	0.007	0.023	0.015	0.028	0.028	0.019	0.007	0.025	0.017
ESOL (1,127)	RMSE	0.049	0.051	0.073	0.107	0.043	0.048	0.089	0.057	0.092	0.068	0.067	0.073
Lipop (4,200)	RMSE	0.037	0.032	0.029	0.016	0.009	0.020	0.033	0.080	0.053	0.041	0.039	0.035
FreeSolv (639)	RMSE	0.165	0.180	0.236	0.117	0.114	0.098	0.212	0.255	0.263	0.269	0.271	0.259
QM7 (6,830)	MAE	0.110	0.010	0.013	0.023	0.033	0.035	0.028	0.028	0.019	0.007	0.025	0.027
QM8 (21,786)	MAE	0.0001	0.0002	0.0002	0.0004	0.0001	0.0004	0.0002	0.0005	0.0014	0.0004	0.0002	0.0003
H Hardware
For all our benchmarks and experiments, we use the GPUs including NVIDIA V100, and NVIDIA A100 GPUs based on availability.
I Hyper-parameters tuning space
For SVM, we adopt the widely-used radial basis function (RBF) as the kernel and we optimize the hyper-parameters within the following ranges:  $C$  (0.1 to 100) and gamma values (0 to 0.2).
For XGB, we penalize the complexity of the traditional gradient boosting model and perform shrinkage and column subsampling to prevent over-fitting. Also, we employ sparsity-aware split finding technique for efficient training on sparse data, etc. In the experiments, we tune the hyper-parameters within the following ranges: learning_rate (0.01 to 0.2), gamma (0 to 0.2), min_child_weight (1 to 6), subsample (0.7 to 1.0), colsample_bytree (0.7 to 1.0), max_depth (3 to 10) and n_estimators (50, 100, 200, 300, 400, 500, 1000).
For RF, in the implementation of RF algorithm, sample perturbation via bootstrap sampling of the training data and feature perturbation via random feature subset selection are introduced to improve the diversity of base learners (decision trees), which corrects for the overfitting habit of decision trees and subsequently enhances the generalization ability of RF. In the experiments, we optimize the
hyper-parameters within the following ranges: n_estimators (50, 100, 200, 300, 400, 500, 1000), max_depth (3 to 12), min_samples_leaf (1,3,5,10,20,50), min_impurity_decrease (0 to 0.01) and max_features ('sqrt', 'log2', 0.7, 0.8, 0.9).
For 1D CNN, we optimize the hyper-parameters within the following ranges: learning rate  $(5 \times 10^{-4}, 5 \times 10^{-5}, 5 \times 10^{-6})$ , the convolution kernel size (4, 8, 10), and the number of the hidden features in the fully connected layer (128, 256, 512, 1024).
For RNN, we optimize the hyper-parameters within the following ranges: the hidden dimensions of each layer ([64, 128, 256, 512]),  $\mathrm{L}_2$  regularization (0 to 0.01), dropout rate (0 to 0.5). The other hyperparameters are the same as the previous work [25].
For TRSF, we optimize the hyper-parameters within the following ranges: the hidden dimensions of each layer ([64, 128, 256, 512]),  $\mathrm{L}_2$  regularization (0 to 0.01), dropout rate (0 to 0.5). The other hyperparameters are the same as the previous work [25].
For MLP, we optimize the hyper-parameters within the following ranges:  $\mathrm{L}_2$  regularization (0 to 0.01), dropout rate (0.0 to 0.5), and neurons for each hidden layer (64, 128, 256, 512). The other important hyper-parameters were fixed: ReLU function that has been recommended by many previous studies was used as the activation function, and the optimizer was set to an adaptive learning rate algorithm: Adadelta [83].
For MPNN, we optimize the hyper-parameters within the following ranges:  $\mathrm{L}_2$  regularization (0,  $10 \times 10^{-8}$ ,  $10 \times 10^{-6}$ ,  $10 \times 10^{-4}$ ), learning rate  $(10 \times 10^{-2.5}$ ,  $10 \times 10^{-3.5}$ ,  $10 \times 10^{-1.5}$ ), dimension of node feature in hidden layers (64, 32, 16), dimension of edge feature in hidden layers (64, 32, 16), and the number of set2set layers (2,3,4). The number of message passing steps and set2set steps were fixed to 6.
For GCN, we optimize the hyper-parameters within the following ranges:  $\mathbf{L}_2$  regularization (0,  $10\times 10^{-8}$ $10\times 10^{-6}$ $10\times 10^{-4}$ ), learning rate  $(10\times 10^{-2.5}$ $10\times 10^{-3.5}$ $10\times 10^{-1.5}$ ), dimension of the FNN classifier (64, 128, 256), and the dimension of GCN hidden layers ([128, 128], [256, 256], [128, 64], [256, 128]).
For GAT, we optimize the hyper-parameters within the following ranges:  $\mathrm{L}_2$  regularization (0,  $10\times 10^{-8}$ ,  $10\times 10^{-6}$ ,  $10\times 10^{-4}$ ), learning rate ( $10\times 10^{-2.5}$ ,  $10\times 10^{-3.5}$ ,  $10\times 10^{-1.5}$ ), dimension of GAT hidden layers ([128, 128], [256, 256], [128, 64], [256, 128]), dimension of FNN classifier (64, 128, 256), and the number of attention heads ([2,2], [3,3], [4,4], [3,4], [2,3]).
For AFP, we optimize the hyper-parameters within the following ranges:  $\mathrm{L}_2$  regularization (0,  $10\times 10^{-8}$ $10\times 10^{-6}$ $10\times 10^{-4}$ ), learning rate  $(10\times 10^{-2.5}$ $10\times 10^{-3.5}$ $10\times 10^{-1.5}$ ), the number of attentive layers for atom embedding (2,3,4,5,6), the number of attentive layers for molecule embedding (1,2,3,4,5), dropout rate (0, 0.1, 0.3, 0.5), and fingerprint dimension (50, 100, 200, 300).
For SPN, we optimize the hyper-parameters within the following ranges: the distance LB2 intermediate size (4, 8, 16), the angle LB2 intermediate size (4, 8, 16), the torsion LB2 intermediate size (4, 8, 16), the number of interaction blocks (3, 4, 5), the number of spherical harmonics (3, 5, 7), the cutoff distance (4, 5, 6), the batch size (32, 64), the initial learning rate  $(1 \times 10^{-4}, 5 \times 10^{-4}, 1 \times 10^{-3})$ , the learning rate step size (50, 100, 150), the learning rate decay ratio (0.4, 0.5, 0.6). The other settings are the same as the original paper [38].