SlowGuess/NeurIPS_2023_OCR huggingface dataset:
C.3 More Results of the Effect of Data Split
Figure 9: The effect of data split on the scaling law with fingerprint modality.
To enhance the robustness of our findings in Section 3.4, in addition to the comparison of the 2D graph modality across various data splits provided in the main text, we also conduct analogous experiments on the fingerprint and SMILES modalities. The empirical results, illustrated in Figure 9 and Figure 10, align with our observations in the Section 3.4: (1) Most performance within these modalities adhere to the power law relationship. (2) The SMILES modality still exhibits a performance drop phenomenon in the MUV dataset. The only deviation lies in the smallest dataset, HIV, where the performance variation under imbalanced/scaffold split settings does not conform to the power law. This deviation might be attributed to the small scale of the HIV dataset, posing challenges for models to generalize and extrapolate effectively.
C.4 Comparison of Different Fingerprint Encoders
Given the absence of established encoder design for fingerprints in the field, we conduct a comparison between two classic encoders, a single-layer MLP and a single-layer Transformer, in terms of their performance variation. The results indicate that both encoders exhibit a strong adherence to the
Figure 10: The effect of data split on the scaling law with SMILES modality.
Figure 11: The Effect of Model Architecture on the Scaling Law with fingerprint Modality.
power law relationship in their scaling behavior. As such, the choice of encoder does not impact the robustness of our conclusions.
C.5 Single-property performance of MUV dataset with SMILES modality
In order to gain a more detailed understanding of the performance degradation phenomenon in the multitask scenario of MUV, we specifically demonstrate the neural scaling behavior of the SMILES modalities in single-property ROC-AUC, as shown in Figure 12. Despite some ascending behavior, the bulk of properties exhibit varying degrees of performance drop in the last few proportions, which accounts for the overall performance drop in the multi-task setting.
C.6 Results of different-layers Transformer with SMILES modality.
Figure 13 presents the scaling behavior of transformers with different numbers of layers in terms of their performance. It can be observed that RoBERTa achieves the overall best performance, followed by a 1-layer Transformer, and the worst performance is exhibited by a 3-layer one. In contrast to fingerprint, the SMILES modality could experience a performance drop on some datasets (HIV and MUV) in the high-data regime. Additionally, the varying numbers of Transformer layers do not affect our conclusions in Section 3.2 regarding modality comparison, as even the superior RoBERTa model overall does not surpass the performance of graph and fingerprint modalities.
Figure 12: The neural scaling law of single-property performance (ROC-AUC) of MUV dataset with SMILES modality.
Figure 13: The neural scaling law of different-layer model (Transformer) performance with SMILES modality.
C.7 Data pruning strategies and additional results
Problem statement. Consider a learning scenario where we have a large training set denoted as  $\mathcal{T} = (x_i, y_i)_{i=1}^{|\mathcal{T}|}$ , consisting of input-output pairs  $(x_i, y_i)$ , where  $x_i \in \mathcal{X}$  represents the input and  $y_i \in \mathcal{Y}$  denotes the ground-truth label corresponding to  $x_i$ . Here,  $\mathcal{X}$  and  $\mathcal{Y}$  refer to the input and output spaces, respectively. The objective of data pruning is to identify a subset  $\mathcal{S} \subset \mathcal{T}$ , satisfying the constraint  $|\mathcal{S}| < |\mathcal{T}|$ , that captures the most informative instances. This subset, when used to train a model denoted as  $\theta^{\mathcal{S}}$ , should yield a similar or better generalization performance to that of the model  $\theta^{\mathcal{T}}$ , which is trained on the entire training set  $\mathcal{T}$ .
Data pruning strategies. In our data pruning experiments, we implement a total of seven data pruning (or coreset selection) strategies: Herding [56], Entropy [57], Least Confidence [57], Forgetting [58], GraNd [59], K-means [60] and we additionally include random pruning as a baseline method. These seven strategies are widely used in the field of CV [67] and have potential in mitigating the issue of data redundancy in large-scale datasets, thereby saving computational and storage resources. Here we provide a brief overview to each of them.
- Herding [56] operates by selecting data points in the feature space based on the distance between the coreset center and the original center. It follows an incremental and greedy approach, adding one sample at a time to the coreset in order to minimize the distance between the centers.
Table 3: Empirical performance of data pruning strategies on eight subsets of the MUV dataset in terms of ROC-AUC (\%,  $\uparrow$ ). Results that are significantly higher or lower than random pruning (with a  $p$ -value of less than  $5\%$  in the significance test) are highlighted.
Uniform	1%	5%	10%	20%	30%	40%	60%	80%
Random	47.4±3.5	58.3±3.4	59.6±4.7	68.9±2.1	71.2±3.4	70.2±3.0	72.9±1.7	77.4±2.8
Herding	48.4±4.7	51.9±4.8	53.1±10.3	61.5±4.9	68.5±7.3	65.1±5.6	71.7±6.4	78.7±5.0
Entropy	48.9±5.5	58.5±5.1	62.8±4.0	63.8±4.2	68.7±3.4	71.0±3.7	75.0±2.5	79.0±3.8
Least Confidence	52.0±7.9	57.4±5.7	61.1±3.3	66.3±2.1	67.6±4.8	70.0±3.6	75.1±2.5	78.5±2.2
Forgetting	47.1±2.1	58.6±2.1	60.9±5.2	63.3±1.8	67.1±3.0	70.5±2.8	74.5±2.6	76.9±3.8
GraNd	47.3±3.4	52.2±7.1	64.7±5.1	65.9±3.1	64.4±5.9	71.4±4.4	72.5±2.5	76.2±3.2
k-means	49.9±4.7	60.5±7.8	65.5±3.7	68.0±4.2	65.5±3.7	67.1±2.8	72.1±1.4	76.9±4.0
Imbalanced	1%	5%	10%	20%	30%	40%	60%	80%
Random	47.7±3.5	58.3±3.4	60.4±4.7	64.1±2.1	66.8±3.4	67.4±3.0	72.4±1.7	66.3±2.8
Herding	50.2±4.7	55.8±4.8	59.3±10.3	58.8±4.9	64.5±7.3	65.6±5.6	65.8±6.4	69.6±5.0
Entropy	47.7±5.5	57.5±5.1	61.0±4.0	68.0±4.2	64.4±3.4	67.2±3.7	68.1±2.5	69.5±3.8
Least Confidence	50.4±7.9	52.9±5.7	60.5±3.3	64.2±2.1	65.5±4.8	68.3±3.6	69.4±2.5	66.4±2.2
Forgetting	49.3±2.1	51.4±2.1	58.7±5.2	64.0±1.8	65.7±3.0	66.5±2.8	67.7±2.6	68.8±3.8
GraNd	52.0±3.4	55.3±7.1	63.8±5.1	63.4±3.1	67.0±5.9	68.6±4.4	70.3±2.5	69.0±3.2
k-means	52.7±4.7	56.0±7.8	58.5±3.7	61.0±4.2	63.4±3.7	63.4±2.8	67.1±1.4	70.0±4.0
- Entropy and Least Confidence [57] iteratively select samples with lower entropy and least confidence, respectively. These methods identify informative samples by considering that lower uncertainty can provide more information gain, thereby benefiting model training and reducing data redundancy.
- Forgetting [58] calculates the frequency of forgetting that occurs during the training process, which refers to the number of times the samples correctly classified in the previous epoch are misclassified in the current epoch. Those unforgettable samples, exhibiting robust performance across epochs, have minimal impact on model performance when removed.
- GraNd [59] measures the average impact of each sample on the reduction of training loss during the initial epochs. Training samples are more important if they contribute more to the error or loss when training neural networks.
-  $k$ -means [60] employs the application of  $k$ -means clustering in the latent space to define the difficulty of each data point based on its Euclidean distance to its nearest cluster centroid. Simple samples (with low difficulty) are considered for removal to reduce data redundancy. It is noteworthy that this method, unlike the aforementioned approaches, does not require any label information or training and can be directly applied to the dataset.
Additional data pruning results. Here we include additional results of data pruning performance on MUV and HIV datasets, as shown in Table 3 and Table 4. Observing the performance of these data pruning strategies, none of these methods consistently outperform random sampling, which aligns with our observation in Section 3.6. Note that in the case of MUV with an imbalanced data split, several data pruning methods such as Random, GraNd, and Least Confidence even demonstrate a performance decline when the subset scale surpasses  $60\%$ .
The following section provides a more broad literature review across the spectrum of molecular representation learning and neural scaling law.
D.1 Molecular representation learning
The past decade has seen remarkable success in the application of deep learning in a variety of biochemical tasks, spanning from virtual screening [68] to molecular property prediction [6]. Within
Table 4: Empirical performance of data pruning strategies on eight subsets of the HIV dataset in terms of ROC-AUC (\%, ↑). Results that are significantly higher or lower than random pruning (with a  $p$ -value of less than  $5\%$  in the significance test) are highlighted.
Uniform	1%	5%	10%	20%	30%	40%	60%	80%
Random	63.4±2.8	69.8±2.2	75.0±2.7	78.5±1.2	79.0±2.2	79.0±1.3	81.5±1.7	83.8±0.8
Herding	60.2±3.9	63.3±3.8	64.7±5.0	69.5±5.4	71.8±7.0	75.8±6.6	80.0±2.8	82.6±1.2
Entropy	67.9±2.2	71.1±3.7	74.2±1.6	76.2±1.2	77.0±2.0	79.2±1.8	81.4±1.9	83.2±1.4
Least Confidence	66.2±4.0	70.4±2.1	72.8±3.9	76.7±2.3	78.0±1.0	81.0±1.4	81.6±1.6	83.3±0.6
Forgetting	67.7±1.2	75.2±1.3	75.1±1.9	76.2±1.7	80.0±1.8	79.8±1.6	82.8±1.0	83.7±1.4
GraNd	66.2±4.0	69.3±2.6	73.6±2.0	78.1±1.1	78.1±1.6	78.6±1.0	82.3±0.8	83.2±1.4
k-means	63.8±4.8	64.4±3.4	65.7±1.8	68.1±1.6	71.5±1.4	72.5±3.5	79.2±0.5	82.3±2.2
Imbalanced	1%	5%	10%	20%	30%	40%	60%	80%
Random	66.6±1.7	68.6±3.1	69.9±3.7	70.9±1.4	70.7±4.1	72.1±3.0	74.1±1.1	74.4±1.1
Herding	57.1±3.0	63.0±3.8	64.9±3.6	65.8±5.9	67.3±6.0	72.6±1.8	73.3±2.2	73.7±0.6
Entropy	67.7±7.5	71.5±2.8	70.1±1.1	71.2±2.1	73.2±2.3	71.7±2.6	74.7±1.3	74.8±1.0
Least Confidence	66.8±5.2	71.4±1.0	71.3±2.6	71.8±2.7	69.5±2.8	73.7±3.4	73.4±2.6	73.8±1.8
Forgetting	66.1±3.1	69.7±5.8	70.2±3.6	71.9±1.9	71.6±1.8	71.4±2.0	73.9±1.4	74.2±2.3
GraNd	62.7±4.5	71.0±2.6	69.2±3.6	73.1±1.9	70.0±3.4	72.9±3.0	74.4±1.8	75.9±1.2
k-means	67.9±1.8	65.4±3.2	65.0±1.9	67.1±4.3	69.1±4.0	68.5±4.3	72.8±1.2	74.4±1.7
this context, molecular representation learning (MRL) serves as a pivotal link between the molecular modalities and the target tasks, efficiently capturing and encoding rich chemical semantic information into vector representations.
One of the mainstream research approaches in MRL is based on  $2D$  topology graphs. The advancements in Graph Neural Networks (GNNs) have enabled the application of more powerful GNN models in the field of molecular chemistry [5, 8, 11, 9, 10, 14], which has proven effective in enhancing the discriminability between representations and capturing underlying chemical semantics. The study of the expressive power of GNNs using the Weisfeiler-Lehman graph isomorphism test has been widely applied in MRL. GIN [5], as one of the most representative works, develops a simple and effective architecture based on a multi-perceptron layer (MLP) that has been proven to be as powerful as the WL test. Some works propose improvements in the expressive power of GNNs to address issues related to long-range interactions [11, 10], higher-order structures [9, 8] and substructure recognition [14] from different perspectives. Unlike traditional message passing mechanisms, Graphormer [12] have explored the direct application of Transformers [45] to graph representation with tailor-made positional encoding. A few research [13, 32] focus more on the ad-hoc model design for biochemical tasks, incorporating constraints based on molecular physics and chemical properties.
More recently, 3D graph representation for molecules has been largely explored thanks to its importance in modeling the dynamical behaviors of molecular structures [42, 69]. Different from 2D graph representation, euclidean group symmetry (E(3)/SE(3)) including translation, rotation and reflection needs to be baked into the design of the models. Specifically, SchNet [43] is one of the earliest work that incorporates euclidean distances between each pair of nodes as features which make the model E(3)-invariant. In addition to pairwise distance features, angle (between three atoms) and dihedral angle (between four atoms) are later on introduced as features into E(3)/SE(3)-invariant models [52, 70, 71]. Instead of invariant models, another branch of work explore equivariant design of 3D graph neural networks. EGNN [72] devises a simple equivariant update layer such that only linear transformation is applied to vector input to achieve equivariance in which similar ideas have also been applied to GVP [73] and PaiNN [51]. ClofNet [74] and LEFTNet [75] on the other side improves the expressiveness of such design by building local frames to scalarize equivariant inputs from there any neural architecture could be applied then transform back to vectors through a vectorization block without information loss. Another noticeable line of work leverage spherical harmonics and tensor product to construct equivariant layers [76-80].
In the advancements of supervised MRL, there has been limited progress in the model designs specifically tailored to the SMILES string and fingerprint modalities. It is worth noting that the
early benchmark models proposed in MoleculeNet [38] have maintained their competitiveness over time. With the rise of pre-training research paradigms, there has been promising progress in recent years towards pre-training approaches based on these two modalities [27, 28, 26, 16] as well as the former two. By employing contrastive [29-32] and generative [33, 30, 34] self-supervised strategies, molecular pre-training approaches guide the model training and subsequently facilitate positive transfer to downstream tasks. However, as mentioned in Section 3.3, existing molecular pre-training still suffer from issues such as parameter ossification [54], necessitating further exploration for more data-efficient and training-efficient models.
D.2 Neural scaling law
The study of neural scaling law can be traced back to early theoretical analyses of bounding generalization error [81-84]. These works, based on assumptions about model capacity and data volume, reveal power-law relationships between the bounds of model generalization error and the amount of data. However, the conclusions drawn from these theoretical studies often yield loose or even vacuous bounds, leading to a disconnection between the theoretical findings and the empirical results of generalization error.
Early follow-on research have investigated empirical generalization error scaling, which represents an initial attempt at exploring the neural scaling law. Bango and Bill [85] conduct experiments on a language modeling problem called confusion set disambiguation, using subsets of a large-scale text corpus containing billions of words. Their findings suggest a power-law relationship between the average disambiguation validation error and the size of the training data. Similarly, Sun et al. [21] demonstrate that the accuracy of image classification models improves with larger data sizes and conclude that the accuracy increases logarithmically based on the volume of the training data size.
Hestness et al. [18] empirically validate that model accuracy improves as a power-law as growing training sets in various domains, which exhibit consistent learning behavior across model architectures, optimizers and loss functions. However, there exists generalization error plateau in small data region and irreducible error region. With a broader coverage, Michael et al. [22] present findings that consistently show the scaling behavior of language model log-likelihood loss in relation to non-embedding parameter count, dataset size, and optimized training computation. They leverage these relationships to derive insights into compute scaling, the extent of overfitting, early stopping step, and data requirements in the training of large language models.
In recent years, several investigations of neural scaling laws specific to particular tasks have been conducted [86, 19, 87, 60]. Unlike previous research, while power-law relationships hold within specific ranges of data size or model parameter count, certain tasks exhibit unique and uncommon learning behaviors. For instance, only marginal performance gains are expected beyond a few thousand examples in image reconstruction [20]. Moreover, there is a relevant study sharing a common spirit with our work in analyzing MRL from a scaling behavior viewpoint [88]. However, our research diverges in the following aspects: (1) Scope of Neural Scaling Law Investigation: The study of Frey et al. primarily considers large language models for generative chemical modeling and quantum property tasks. In contrast, our work encompasses a broader range of dimensions, such as modalities, data distributions, pre-training and model capacity, to investigate the influence of various factors on neural scaling laws. (2) Different contributions: Their contribution primarily revolves around developing strategies for scaling deep chemical models, notably large language models, and introducing novel model architectures. In contrast, our focus lies in exploring the impact of different dimensions on data utilization efficiency through a data-centric perspective and benchmarking existing data pruning strategies on MRL tasks.