Uncovering Neural Scaling Laws in Molecular Representation Learning
Authors: Dingshuo Chen
Yanqiao Zhu
Jieyu Zhang
Yuanqi Du
Zhixun Li
Qiang Liu
Shu Wu
Liang Wang
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: molecular representation learning
neural scaling law
data-centric AI

Abstract: Molecular Representation Learning (MRL) has emerged as a powerful tool for drug and materials discovery in a variety of tasks such as virtual screening and inverse design. While there has been a surge of interest in advancing model-centric techniques, the influence of both data quantity and quality on molecular representations is not yet clearly understood within this field. 
In this paper, we delve into the neural scaling behaviors of MRL from a data-centric viewpoint, examining four key dimensions: (1) data modalities, (2) dataset splitting, (3) the role of pre-training, and (4) model capacity.
Our empirical studies confirm a consistent power-law relationship between data volume and MRL performance across these dimensions. Additionally, through detailed analysis, we identify potential avenues for improving learning efficiency.
To challenge these scaling laws, we adapt seven popular data pruning strategies to molecular data and benchmark their performance. Our findings underline the importance of data-centric MRL and highlight possible directions for future research.

Uncovering Neural Scaling Laws in Molecular Representation Learning
Dingshuo Chen'*
Yanqiao Zhu Jieyu Zhang?
Yuanqi Du
Zhixun Li Qiang Liu' Shu Wu" Liang Wang'
Center for Research Intelligent Perception and Computing Institute of Automation, Chinese Academy of Sciences
*Department of Computer Science. University of California, Los Angeles
'The Paul Allen School of Computer Science and Engineering, University Washington *Department of Computer Science, Cornell University
'Department of Systems Engineering and Engineering Management
The Chinese University of Hong Kong
Primary contact: dingshuo. wwwnwcripac. aa.ac.aa........................ ac cn
Abstract
Molecular Representation Learning (MRL) has emerged as powerful tool for drug and materials discovery in variety of tasks such virtual screening and inverse design. While there has been a surge of interest in advancing model- centric techniques, the influence of both data quantity and quality molecular representations is not yet clearly understood within this field. In this paper. we delve into the neural scaling behaviors of MRL from data-centric viewpoint, examining four key dimensions: (1) data modalities, (2) dataset splitting, (3) the role pre-training, and (4) model capacity. Our empirical studies confirm consistent power-law relationship between data volume and MRL performance across these dimensions. Additionally, through detailed analysis, for improving learning efficiency. To challenge these scaling laws, we adapt seven popular data pruning strategies molecular data and benchmark their performance. Our findings underline the importance of data-centric MRL and highlight possible directions for future research.
identify potential avenues
Introduction
The research enthusiasm for Molecular Representation Learning (MRL) is steadily increasing, attributed to its potential expediting drug and materials discovery processes compared with conventional in vitro and in vivo experiments [1-4). Within the context of MRL, the central objective to leverage specific featurizations, or modalities, of molecules in order to learn continuous vector representations. These representations aim capture comprehensive chemical semanties and exhibit high expressiveness, thereby effectively addressing various downstream tasks [5-17].
A trend in the field is developing neural architectures and training strategies to improve the ex- pressiveness of the learned representations. However, the influence of varying data scales on the performance of MRL under different learning scenarios is yet to be fully understood. To fill this gap, we draw attention to the following questions: What are the neural scaling behaviors of molecular representation learning? Do they align with the previous scaling laws such the power-law, es- tablished in other domains [1-20]? Beyond common research objects in neural scaling laws such the impact of pre-training and model parameter sizes [21, 18, 22], MRL further presents unique
'These authors made equal contribution research. Corresponning author: Shu Wu (shu. wu@nlpr ia. ac. en).
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

data-orie data-oriented challenges. These include the selection of appropriate modality [23] and issues related to Out-Of- out-Of-Distriuttion (OOD) generalization [24].
To provide a comprehensive understanding of these complexities, we investigate the impact of various design dimensions MRL from data-centric perspective. Specifically, our exploration spans four core design dimensions: (1) data modalities (molecular fcaturizations), (2) data splitting, (3) the role of pre-training, and (4) model capacity. We identify five scientific questions and outline our key observations as follows. A succinct summary of contribution of this work is present in Figure 1.
(Section 3.1) How does performance scale with data quantities? We conduct extensive experiments on four large-scale molecular property prediction datasets. These datasets contain a number of classifi- cation and regression tasks, both in single-tast task and multi-task settings, focusing on properties ranging from quantum mechanical properties to macroscopic influence on human body. The experimental results indicate that the model performance generally follows power-law relationship with data quantities. Compared with the neural scaling laws in Natural Language Processing (NLP) and Com- puter Vision (CV) domains, there is no apparent performance plateau [21] within the range of datasets we explored, regardless of low-data and high-data regimes, which implies that the performance improvement with increasing dataset size in MRL highly predictable.
(Section 3.2) How do different molecular modalities influence scaling laws? Since distinct molecular featurizations might carry different semantic meanings and their corresponding neural encoders have different inductive bias, the selection of appropriate modalities (molecular featuriza- tions) in MRL remains open question. In our investigation, we specifically choose three commonly used modalities (2D) graphs [6, 8, 24], SMILES strings [25-28], and Morgan fingerprints [23]) and compare their performance on three classification tasks. We find that different modalities exhibit distinct learning behaviors in MRL; graphs and fingerprints are identified as the most data-efficient modalities, exhibiting the largest power-law exponent. In comparison, SMILES strings demonstrate lower performance improvement with the same data increment.
Does pre-training consistently result in positive transfer across all data scales in MRL? While previous studies tend to argue that molecular pre-training can improve performance on downstream tasks, these conclusions often drawn based evaluations the entirety of available datasets [29-33, 30, 34]. Our work further probes into the effects of graph- pre-training across varying scales of downstream datasets. The results show that pre-training indeed brings improvements in low-data regimes. However, the power-law exponent of the performance curve, reflecting the rate of growth with incremental data sizes, is smaller with pre-training compared training from scratch. We suppose that pre-training only delivers stable gains when the downstream dataset is relatively small-under 40K samples, for instance. As the downstream dataset scales up, this positive gains diminish and might even revert to a negative transfer in the high-data regime.
(Section 3.3)
(Section 3.4) What influence do dataset splits exert on scaling laws? -iittr--titiitiionn (OOD) generalization poses a great challenge in MRL [24]. Given that the training distributions, which typically encompass known compounds, often differ from the test distributions containing unknown compounds, this divergence is particularly prevalent in drug discovery. However, the impact of dataset splitting strategies on neural scaling laws remains largely unknown. To bridge this gap, we study three splitting strategies: random, imbalanced, and scaffold splitting. These strategies differ primarily in the degree of overlap between their respective training and test sets. Our empirical findings indicate that while all three splitting schemes adhere to the scaling laws, the exponent associated with random splitting is the smallest, suggesting that as the divergence between training and test distributions increases, the efficiency with which the model utilizes data samples decreases. Such results accentuate the inherent challenges of MRL in practical scenarios, which face with ubiquitous problems of distribution shifts.
(Section 3.5) How does the model capacity affect the scaling laws? The model capacity stands as another crucial factor impacting the performance of MRL models [18, 22] as it significantly influences the requisite amount of training data. To study the effect of model capacity in MRL, we vary the parameter size of Graph Isomorphism Networks (GIN) [5], a widely adopted graph neural network architecture and examine its performance on three classification tasks. We observe that, while model capacity does affect data utilization efficiency, there is not a clear correlation between the dataset size and the optimal model capacity required obtain saturation performance. Unexpectedly. we find that the optimal model capacity for a task with a smaller training set could be larger than that of a task with a larger training set, which contradicts our intuition that larger models usually need larger
2

---

obuemntter A: the prdtrm prenidy
dosmatie Higornt waikhe Oelomiken,
A Aupqpiiia aaaeted aed ln not cito (Piict
sodu pubiin you ad polindly
imprler Noh
Chao
SIELD
Somert
Fature Work
Oleers stam
preiie sateel jus of
Blereatie Dem sembboe capust repand to akdz aceetio prfveceos
Obervation 4: Deckst quitin geu geijishl udea loaet of sing cadd moniy
Hitter dreeiing straloatt ondh consutisi and pastoes braaghe
for ivlecarrdo dats
Mir Sas-embic explassioon
153
327 2011
Figure 1: Summary of contributions of this work. We conduct a comprehensive data-centric study to examine the neural scaling laws in molecular representation learning. We explore four dimensions affecting the data utilization efficiency: (1) data modalities, (2) the role of pre-training, (3) dataset splitting, and (4) model capacity. Additionally, we study the subset selection problem by adapting seven data pruning strategies to molecular graphs.
training datasets. It suggests the need for further exploration of the interplay between model capacity and data scale, rather than applying a one-size-fits-al approach.
(Section 3.6) Can a curated subset from the full dataset yield comparable or even superior results? In CV and NLP domains, the utility of data pruning has been extensively explored due to the computational burden imposed by increasingly large models and massive amounts of data [35-37]. In MRL, however, the extend of data redundancy and the potential of pruning strategies to alleviate computational burden remain largely unexplored. To address this gap, benchmark seven data pruning strategies originally proposed for image data and adapt them to molecular graph models on three classification tasks. The results show that these data pruning methods do not significantly outperform random selection, which highlights the need for developing data pruning strategies specifically tailored to molecular data.
Based on our empirical analysis, we identify several factors that can enhance data utilizaiton efficiency of MRL. These include the utilization of graph and fingerprint modalities as input data, the application of pre-trained models on small-scale downstream datasets, and the design of data pruming strategies specifically tailored to molecular data. To the best of our knowledge, our study is the first to approach MRL from a data-centric perspective. We envision that this work could provide valuable insights that facilitate future explorations in the field.
Experimental Settings
In order answer the six scientific questions through empirical investigation, we conduct series of experiments on the neural scaling behaviors between model performance and varied data quantities. Specifically, we divide the whole dataset into nine proportional subsets: [1 5%, 10%, 20%, 30%, 40%, 60%, 80%. 100%], and for each configuration. we randomly select five seeds and report the mean performance. Then, we study how each design dimension of interest in MRL models influences the scaling laws. To demonstrate the trend of neural scaling laws, we employ the least square model to fit the performance curve.
2.1 Datasets and Tasks
Given the potential issues of over-fitting and spurious correlations that may arise with limited samples, we focus on relatively large-scale datasets containing more than 40K molecules in MoleculeNet [38]. Also, these datasets should cover both classification and regression tasks, are diverse in task settings (i.e. single- and multi-task settings), and focus on important biophysics and quantum mechanics properties. As result, we choose four datasets ranging from molecular-lvvel properties to macro- scopie influences on human body for experimental investigation: HIV [39], MUV [40], PCBA [41) and QM9 [42]. Readers can refer to Appendix B for more detailed discussion datasets and tasks.
3

---

2.2 Implementation Details
In the following, we briefty introduce the implementation details of our experiments. Note that our experiments cover multiple dimensions, and thus all experimental settings will remain consistent except for the design dimension to be studied. Please refer to Appendix A for detailed introduction of implementation details.
Modalities. Molecular featurizations translate chemical information of molecules into representations that can be understood by machine learning algorithms. Each featurization can thus be regarded as a modality of the molecular data. Here, we consider the following four molecular modalities: (1) 2D topology graphs model atoms and bonds as nodes and edges respectively, (2) 3D geometric graphs include coordinates of atoms into their representation to depict how atoms positioned relative to each other in 3D space, (3) Morgan fingerprints [23] encode molecules into fixed-length bit vectors which map certain structures of the molecule within certain radius of molecular bonds, and (4) SMILES strings [25] represent chemical structures in a linear notation using ASCII characters, with explicit information about atoms, bonds, rings, connectivity, aromaticity, and stereochemistry.
MRL models. Since our experiments involve four different data modalities, each modality is modeled with its corresponding encoders.
For 2D graphs, we utilize the Graph Isomorphism Network (GIN) [5] as the encoder. To ensure the generalizability of our research findings, we adopt the commonly recognized experimental settings proposed by Hu al. [24], with layers, 300 hidden units each of layer, and 50% dropout ratio.
For 3D geometries, we employ the widely-used SchNet model [43] as the encoder. We set the hidden dimension and the number of filters in continuous- convolution to 128. The interatomic distances are measured with 50 radial basis functions, and we stack 6 interaction layers in SchNet.
For fingerprints, first RDKit [44] to generate 1024-bit molecular fingerprints with a radius R = 2, which is roughly equivalent to the ECFP4 scheme [7]. Given the lack of established en- coders for fingerprints, conduct comparison between two classic encoders, single-layer MLP and a single-layer Transformer. Please refer to Appendix C.4 for the detailed results. According the experiments, we choose the Transformer model [45] with 8 attention heads for modeling fingerprints. The bit embedding dimension is set to 64, and the hidden dimension is set to 300.
For SMILES strings, we employ the same model architecture as the fingerprints to ensure a fair comparison. The only difference is the dictionary size, which will be discussed in Section 3.2.
Training details. We follow the settings proposed by Hu al. [24]. The dataset split randomly. For classification tasks, we employ an 80%/10%//0%% partition for the training, validation, and test sets, respectively. Meanwhile, for regression tasks, the dataset is split into 110K molecules for training, 10K for validation, and another 10K for testing. All models are initialized using the Glorot initialization [46]. The Adam optimizer [47] is employed for training with a batch size of 256. For classification tasks, the learning rate is set at 0.001 and we opt against using a scheduler. For regression tasks, we align with the original experimental settings of SchNet, setting the learning rate to 5 x 10 and incorporating cosine annealing scheduler.
Evaluation metrics. For HIV and MUV datasets, performance is measured using the Area Under the ROC-Curve (ROC-AUC), while report the performance on PCBA in terms of Average Precision (AP) higher values in both metrics indicate better performance. When assessing quantum property predictions, the Mean Absolute Error (MAE) is used as the performance metric, with lower values indicating better accuracy,
3 Empirical Studies
In this section, we summarize our empirical studies on the aforementioned dimensions of MRL. Firstly, we show the neural scaling laws between model performance and data quantities. Then, we demonstrate the impact of four design dimensions of MRL:: data modalities, the role of pretraining, data splits, and model capacity. Lastly, we study data pruning strategies for molecular graphs.

---

BOC ROC-AN
88
ROC-AMe
07-
6.605
AP
MAD
MAB
EX43
110
16
110
60
(A)
U0
SOPVIEE
0.90
10C AUC
BOO-afic
XAA
0.70
6.
03303
4.7
0.77
EXCL
(4) IIV (1og log)
(b) M0VV (Jog log)
02 PCIM [log log)
(1 ppp Dlgg log)
(3) D0 (lop log)
03 2PVE (log log)
Figure 2: The general neural scaling laws of molecular representation learning. The first row displays the relationship between model performance with respect to varied data sizes in linear coordinates and the second row shows the same on a logarithmic scale.
3.1 General Neural Scaling Laws for Molecular Data
Early studies of both classical learning theory and neural scaling laws [48, 18] show that the test performance [nn) increases polynomially with the training data size n::
L(n) 8- n²"
(I)
where is the coefficient and a is the exponent of the power law. We start our investigation on whether MRL also adheres to this power law relationship with the most common setting of graph-based MRL. Specifically, on the QM9 dataset, we study GINs on 2D graphs for classification tasks and SchNet on 3D geometric graphs for regression tasks. Figure 2 demonstrates the relationship of model performance with respect to the data size across all datasets, where the first row displays the curves in linear coordinates and the second row shows the same logarithmic scale.
Observation 1. It can be observed that figures in the second row generally exhibit a linear relationship, which suggests alignment with the aforementioned power law. Unlike previous findings in the NLP and CV domains [18], there is no apparent performance plateau within the range of datasets we explored. This pattern remains consistent in both low- and high-data regimes, which suggests that the performance supervised MRL highly predictable and consistently improves with increasing data quantity. We note that we observe the same phenomenon consistently when applying other modalities, as we will show later in Figure 3. Furthermore, we also validate the neural scaling laws by examining them large-scal dataset PCQM4Mv2 [49] from Open Graph Benchmark (OGB) [50], well as by employing two advanced 3D GNN encoders PaiNN [51] and SphereNet [52] on QM9. Readers of interest may refer to Appendix C.2 for detailed results and analysis.
3.2 Scaling Laws with Different Molecular Modalities
The choice of the most appropriate modalities in MRL remains a topic of continuous discussion. Also, there is lack of fair comparisons regarding data efficiency across these modalities. In this section, we compare the learning behaviors of the three modalities: 2D graphs, SMILES strings, and fingerprints. To ensure a fair comparison, we maintain roughly equivalent parameter sizes across the modality encoders. Such configurations are selected to maximize the expressiveness of each model. Specifically, we employ S-layer GIN model for the 2D graph modality and I I-layer transformer model for both SMILES strings and fingerprints. is noting that transformers used for SMILES and fingerprints differ in vocabulary sizes: while the former has a size of 2 for fingerprints due to the binary nature of fingerprint strings, the latter adopts a more expansive vocabulary size of 7,924 for SMILES, following ChemBERTa [26]. We present the results illustrating model performance in relation to different data sizes for the three encoders in Figure 3, with anomalous performance explicitly marked in red.
Observation In general, we observe that different modalities exhibit distinct learning behaviors. Among the three modalities, the graph modality consistently demonstrates superior data utilization efficiency across all three tasks, emerging as the most efficient modality for MRI. Fingerprints, char- acterized with lower exponent, deliver slightly lower performance improvement with equivalent
5

---

ROC-ATC
ROC-AUC
0.-
07
9-6-
0.30
Performance drop
0.6-
60
110
30
60 PCBA
100
HIV
MUV
8000
0.01
1.30
RDCC -00C
0.50
ROC-AUC
8.00
0.70-
3.00
:
0.50
2.0
Power low brokocn
10
10
10
10
HTV (log-log)
MUV (logslog)
(C) PCRA (logslog)
Graph
SMILES
Fingerprins
Figure 3: Neural scaling laws with three encoders: 2D graphs, SMILES strings, and fingerprints. Anomalous performance is explicitly marked in red.
data increments. SMILES strings, on the contrary, are the least data efficient modalities. On the MUV dataset, we even notice counter-intuitive performance degradation that breaks the power law However, it is important to recognize that several studies have underscored the efficacy of language models pre-trained on large-scale SMILES string datasets, which exhibit remarkable performance in the downstream tasks [28, 26]. This points towards the superior transferability of the SMILES-basee modality and we will leave this performance gap between pre-trained and those trained from scratch for future research.
3.3 Inftuence of Pre-training on the Scaling Laws
Molecular pre-training studies how to leverage unlabeled molecular data to improve the learned representations [53]. Previous studies have generally suggested that pre- pra-training can induce a positive transfer to downstream tasks [33, 30, 29]. However, we argue that the existing evaluations rely on the entirety of available datasets. We hypothesize that when fine-tuning with datasets of varied sizes, the extent of this positive transfer might fluctuate. Therefore, in this section, we explore whether pre-training consistently leads to positive transfer with different sizes of fine-tuning datasets.
In the experiments, we focus on the state-of-the-art molecular pre-training model GraphMAE [33], which involves masking the atom type of partial atoms in the molecules, re-masking the encoded atom representations from the backbone model, and eventually using a decoder model to reconstruct the original atom features. The GraphMAE model first pre-trained on PCQM4Mv2 and then fine-tuned on three classification tasks. The results in Figure 4 compares the performance curve with and without pre- teeiiiriiing on downstream datasets of varied sizes.
Observation 3. Our empirical results reveal that, while pre-trained models still adhere to a power law their downstream data size varies, they differ from models trained from scratch by having a higher intercept and a lower exponent in the logarithm-scaled plots. This implies that while pre-trained models benefit from better initializations, they demonstrate reduced data efficiency when fine-tuned on downstream datasets.
Our findings also indicate that the benefits of pre-training are particularly pronounced when fine- tuning with smaller datasets and such advantages start to diminish as the size of downstream datasets increases. This is evidenced by the PCBA dataset, where observe crossover in performance at data scale of 40K. Beyond this threshold, the performance of pre-trained models starts to fall behind its non-pretrained counterpart. This behavior suggests the phenomenon of parameter ossification [54] in pre-trained models, suggesting that pre-training can inadvertently "freeze" the model weights in a
We confirm that the performance degradation is a common phenomenon rather than an outcome of limited model capacity by further altering the number of model layers. Readers can refer to Appendix C for more discussions.

---

0.58
0.25
ROCC.M CCC.ACC
0.50
ROC-AUC
0.15
0.70-
010
100
100
10
109
(b) MUV
(c) PCTA
gb
0.50
1O0C-AR
0.000
0:50
10
UP
10
10
10
10
(d) HTV (logslog)
MUV (log-log)
(f) PCBA (log-log)
No pre training
Pre-traiaing
Figure 4: Comparison of scaling laws with and without pre-training varied downstream data sizes.
way that limits their adaptability to the fine-tuning distribution, especially in large-data scenarios. Consequently, we believe the role and implications of pre-training in MRL warrant careful evaluation.
3.4 Influence of Dataset Splits on the Scaling Laws
The practice of dataset splitting in MRL is usually around two methodologies: random and scaffold splitting [38]. Random splitting ensures that both training and test samples are sourced from the same distribution, representing uniform distribution setting. Conversely, scaffold splitting simulates the Out-Of-Distrbuute (OOD) scenario, wherein the training and test distributions are entirely distinct, pushing the boundaries of OOD generalization capabilities of MRL models. However, the realities of drug discovery often diverge from these two extremes. It is frequent for test compounds to possess substructures already encountered in the training samples [55]. To address this more realistic situation, we introduce imbalanced splitting, which generates the training, validation, and test sets based on scaffolds at first and then moves a fraction of the samples (5% in our experiments) from both test and validation sets to the training set. This approach serves trade-off between random and scaffold splitting, offering a more realistic reflection of real-world scenarios.
In this section, we still focus on 2D graph modalities and the learning behaviors on these three splitting methods by changing the scale of downstream datasets are shown in Figure For readers interested in a deeper exploration, we append additional experiments in Appendix C.3 that analyze the neural scaling laws of two other modalities fingerprints and SMILES strings.
Observation 4. Our empirical observations validate that regardless of the dataset splitting methods, model performance conforms to a power law. Note that the performance curve with random splitting has the highest exponent. which suggests that this uniform setting has a higher data utilization efficiency compared the other two OOD settings. We also observe outlier performance in the imbalanced splitting of the MUV dataset, where certain data scales (e.g.. 80% and 100%) break the the power laws, which we suspect is due to the examples selected outside the training set could mislead the model to capture spurious relationship between label and features. Importantly, these findings underscore the inherent challenges of MRL in real-world scenarios, which often face with the pervasive issues of distribution shifts.
3.5 Influence of the Model Capacity on the Scaling Laws
This section we study the relationship between model performance and capacity. In our MRL models, the capacity is largely determined by the number of layers (depth D) and hidden units (width W). Therefore, we characterize model capacity as D x W, since the parameters of the embedding layer and output layer are the same across different models and hence can be sidelined from our evaluation. In the experiments, we focus on 2D topology graphs with GIN as encoders. Due to the inherit drawbacks of gessage-passing networks, such as as over-smoothing and over-squashing. we carefully select four experimental configurations with reasonable increments: (64 x 2, 100 3, 300 x 5, 600 x 10).
7

---

ROCC CCA.ACC
1.50
ROC-AUC AUC
67-
0.26
0:75
8
Inflection Point
0.15
010
10
60
100
100
(6)
(b) MUV
(6)
0.40
ROCC-AR ACC-
RDC AtC
0.70
0.70
0:90
Inflection Point
0.100
0.30
10
10
10
HTV (logslog)
MUV (log-log)
PCBA (log-log)
Random
ScaHiold
Imbalanced
Outlier
Figure 5: Neural scaling laws with three dataset splitting schemes: random, scaffold, and imbalanced splitting Arrows indicate inflection points that deviate significantly from the fitted curves
0.31
ROC-ATC
ROC-AUC
6--
0.91
0.75
0.15
0.0-
00
100
10
30
60
300
10
30
100
IIIV
MUV
(e) PCDA
1090
0.50
ROC-AUI
ROC-AIR
0.70
to
10
(4) HTV (log-log)
(o) MUV (oog-log)
(f) PCBA (log-log)
128
300
1.5K
6K
Figure 6: Neural scaling laws with the model capacity. We study four configurations of GIN model with 128, 300, 1.5K, and 6K parameters respectively.
The performance with these four model configurations across different datasets is summarized in Figure 6.
Observation 5. The experimental results demonstrate that the relationship between model perfor- mance and capacity also adheres the power law However, it is observed that the optimal model capacity varies across different tasks and that a smaller model could be sufficient to achieve the optimal performance for some tasks. For example, for HIV and PCBA datasets, the model with capacity of 1.5K parameters achieves the best performance. On the contrary, the smallest model with capacity of 128 parameters achieves optimal performance observation is that, on the PCBA dataset with over 400K data samples, there is significant difference in power law exponents among different model capacity. The models with lower capacity achieve saturated performance quickly as the data scales up. whereas this phenomenon is not evident in HIV and MUV datasets. summary, we believe is important carefully select the appropriate model capacity according to the characteristics of datasets and tasks.
the MUV dataset. An interesting
3.6 Data Pruning for Molecular Representation Learning
Lastly, we investigate the data pruning strategies in MRI, which identify representative subset of the complete dataset. Such strategies have been proved effective to improving training efficiency in

---

Table 1: Empirical performance of data pruning strategies on eight subsets of the PCBA dataset in terms of ROC-AUC (%, 1). Results that are significantly higher or lower than random pruning (with a p-value of less than 5% in the significance test) are highlighted.
Uniform
1%
5%
10%
20%
30%
40%
60%
80%
Random
5.2e0.1
9.6sa2
13.2+0.2
17.7
20.0aas
22.2+0.6
24, 94.9..5
26,8+04
Herding
3.7=1.
9.7
10.2+3.
15.8+3.7
20.7
22.6H0.6
25. 25.4
24.5+s4
Entropy
5,4g02
10.0
13.7,0.6
17.6.0.1
20.2a06
20.4
25 O2n.3
26.6204
Least Confidence
5,320.1
9.7x03
13.7
17.6.0.4
20.3
22.020.5
25 25.010.3
26.5402
Forgetting
5.5.02
9,820.4
13.5a04
17.5.0.3
20.4
12.2
24.7
26.5a02
GraNd
5.5+0.3
9.7-03
13.3+0.
17.5+0.4
20.3=0s
22.
24.9+0.
26.8+a.3
k-means
4.1+0a
8.1
11.6+0.3
16.5+0.2
20.4-04
22.6+0,3
24.9+0.4
26.3+a.1
Imbalanced
1%
5%
10%
20%
30%
40%
60%
80%
Random
6.1202
9.9a02
12.4
14.920.3
16.5
17.7,0.2
19.2.0.1
20.4g02
Herding
4.7
8,5
11.5
13.4+16
14.7=28
16.5+2.
16.8+36
17.9+3.1
Entropy
6,0ea3
9,9s04
12.3+0.4
15.
8
17.8.0.2
19.3+0.4
20.5+0.3
Least Confidence
6. 3,03
9,9202
12.320.4
15.320.3
17. 0.0.5
17.9
19.420.3
20.520.1
Forgetting
5.7203
9.9202
12.2s0.3
14.7a0.2
15.8
17.8
19.320.5
20.3203
GraNd
5.9204
9.7203
12.220.5
15.020.6
16.8203
17.8
19.420.2
20.720.5
k-means
4.6+0.
8.1
10.7+0.
13.9-0.2
16,4-04
17.2+0.3
19.6+0.5
20.4+0.1
CV and NLP domains [35-37] but their roles in MRL remain under-explored. In our experiments, we benchmark seven data pruning strategies originally designed for image data and adapt them for 2D molecular graphs: Herding [56]. Entropy [57]. Least Confidence [57], Forgetting [58], GraNd [59], and k-means [60]. We also include random pruning as baseline method. Please refer to Appendix C.7 for a detailed description of these adopted data pruning strategies
For all experiments, we employ each data pruning strategy to curate eight subsets, which are constituted of the following percentages of the whole dataset: [1%, 5%, 10%, 20%, 30%, 40%, 60%. 80%]. For comprehensive investigation, we compare the results with two data splitting schemes to discern if performance of pruning strategies might be influenced by data distribution disparities. Table 1 summarizes the results on PCBA, one of our largest datasets, where we highlight significant results with a p-value less than 5% in the significance test (1-test) compared with random pruning.
Observation 6. Our empirical analysis reveals that no adopted pruning strategy consistently outper- forms random pruning across various data subsets, irrespective of the dataset splitting methodology. The performance difference among these pruning strategies also remain narrow. These trends are con- sistently observed across the MUV and HIV datasets as well, with details provided in Appendix C.7. This leads us to postulate two potential hypotheses. First, the current molecular encoders might be adept at extracting the information from the tested datasets, thereby suggesting a minimal presence of data redundancy. This suggests a potential need for significantly larger datasets to showcase the effi- cacy of data pruning methods in MRL. Second, existing data pruning strategies, originally proposed for image data, may not transfer their effectiveness molecular data. For example, similarity-based approaches like Herding might need the infusion of domain-specifie knowledge when measuring the similarity of molecular graphs.
4 Conclusions
In this paper, we conduct a series of data-centric experiments on Molecular Representation Learning (MRL) spanning four dimensions, including (1) data modality, (2) dataset splitting, (3) the role of pre-training and (4) model capacity, to investigate the model performance under different data scales. Our empirical results demonstrate that the scaling behaviors generally conform to a power law, implying the marginal effect of adding more molecular data. Moreover, we observe that there is no one-size-fiis-all configuration to different MRL datasets, which calls for task-specific analysis in real-world application. We also benchmark seven data pruning strategies and observe that none of these methods outperform random sampling. Our analysis underscores the unique challenges inherent in MRL, such as distribution shift in molecular discovery and the non-E Euclidean nature of molecular

---

datasets. We anticipate that our findings will catalyze further research, driving more data-efficien methodologies for molecular representation learning.
Limitations. We conduct our experiments using widely-adopted model architectures in the field, such as GIN for the 21) topology graph, SchNet for the 3D geometry graph, and Transformer for SMILES string and Fingerprint. However, the rapid development of molecular representation learning in recent years has led the emergence of models with improved expressiveness. The neural scaling laws on these models is still to be explored. Moreover, our focus is primarily on investigating the impact of various dimensions on supervised molecular representation learning. Despite that we have considered the effect of pre-tratee pre-training, we do not explore the neural scaling behaviors between data scales in pre-training and the corresponding performance on the downstream tasks.
Future work. In closing, we point out several prospective avenues for further exploration. (1) Neural scaling laws in molecular generative tasks. In this paper, our exploration has concentrated solely on predictive tasks, leaving an unexplored domain of generative tasks. Potential research might investigate neural scaling laws on tasks such generation of drugs, conformations, and docking poses, to name a few. (2) Refinement of pre-training strategies. In our experiments, we discover that pre-training on molecular data does not improve efficiency of leveraging downstream data. Future work could consider to address the issues of parameter ossification to improve data efficiency in pre-trained MRL models. (3) In-depth analysis of data pruning strategies for molecular data. We observe that the existing image pruning methods do not significantly improve MRL performance. Such observations call for rigorous evaluation of data pruning methodologies, potentially on more larger-scale datasets, to ascertain the potential advantages of data pruning techniques. Moreover, the design of efficient pruning strategies specifically tailored molecular data is also promising direction, which yet remain largely unexplored.
Acknowledgements
The authors would like to thank anonymous reviewers for their helpful feedback. This work is supported by National Natural Science Foundation of China (62141608, 62206291, 62372454).
References
[1] Yue Liu, Tianlu Zhao, Wangwei Ju, and Siqi Shi. Materials discovery and design using machine learning. Journal of Materiomics. 3(3):199- 177, 2017. 1
[2] Shen and Christos Nicolaou. Molecular property prediction: recent trends the era artificial intelligence. Drug Discovery Today: Technologies, 32:29-36, 33, 2019.
[3] Oliver Wieder, Stefan Kohlbacher, Mélaine Kuenemann, Arthur Garon, Pierre Ducrot, Thomas Seidel, and Thierry Langer. A compact review of molecular property prediction with graph neural networks. Drug Discovery Today: Technologies, 37:1-12, 2020.
[4] Xuan Zhang, Limei Wang, Jacob Helwig, Youzhi Luo,, Cong Fu, Yaochen Xie, Meng Liu, Yuchao Lin, Zhao Xu, Keqiang Yan, et al. Artificial intelligence for science in quantum, atomistic, and continuum systems. arXiv preprint arXiv:2307.0842, 2023.
[5] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv: 1 1810,00826, 2018. 1, 4, 16, 23
[6] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263-1272 PMLR, 2017. 2, 22
[7] David Rogers and Mathew Hahn. and modeling, 50(5):742-755 754, 2010. 4
fingerprints. Journal of chemical information
[8] Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. Principal neighbour- hood aggregation for graph nets. Advances in Neural Information Processing Systems, 18:132200-3227 2020. 2, 23
[9] Guohao Li, Chenxin Xiong, Ali Thabet, and Bernard Ghanem. Deepergen: All you need to train deeper gens. arXiv preprint arxiv:2006.0770 07739, 2020. 23
10

---

[10] Dominique Beaini, Saro Passaro, Vincent Létourneau, Will Hamilton, Gabriele Corso, and Pietro Liò. Directional graph networks. In International Conference Machine Learning, pages 748-758. PMLR, 2021. 23
[11] Cristian Bodnar, Fabrizio Frasca, Nina Otter, Yuguang Wang, Pietro Lio, Guido Montufar, and Michael Bronstein. Weisfeiler and lehman go cellular: Cw networks. Advances in Neural Information Processing Systems, 34:2625 2640, 2021. 23
[12] Chengxuan Ying. Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 34:28877-28888, 2021. 23
[13] Shuwen Yang. Ziyao Li, Guojie Song, and Lingsheng Cai. Deep molecular representation learning via fusing physical and chemical information. Advances in Neural Information Processing Systems, 34: 16346-16357, 2021. 23
[14] Giorgos Bouritsas, Fabrizio Frasca, Stefanos Zafciriou, and Michael M Bronstein. Improving graph neural network expressivity via subgraph isomorphism counting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):657-668, 2022. 23
[15] Zhaoning Yu and Hongyang Gao. Molecular representation learning via heterogeneous motif graph neural networks. In International Conference on Machine Learning, pages 25581-25594, PMLR, 2022.
[16] Yangiao Zhu, Dingshuo Chen, Yuanqi Du, Yingze Wang, Qiang Liu, and Shu Wu. Featurizations matter: A multiview contrastive learning approach to molecular pretraining. In /CML 2022 2nd Al for Science Workshop.
24
[17] Yuanqi Du, Tianfan Fu, Jimeng Sun, and Shengchao Liu. Molgensurvey: A systematic survey in machine learning models for molecule design. arXiv preprint arXiv:2203. 14500, 2022.
[18] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint urXiv:1712.00444 2017. 2, 2,
[19] Oren Neumann and Claudius Gros. Scaling laws for multi-agent reinforcement learning model. arXiv preprint (=Xxx:221000044) 2022. 24
[20] Tobit Klug and Reinhard Heckel. Scaling laws for deep learning based image reconstruction. arXiv preprint Xrx: 13435, 2022. 2,
[21] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinay Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference computer vision, pages 843-852, 2017. 1, 2,
[22] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, arXiv preprint arXiv:200J.
2020. 2, 24
[23] Harry L Morgan. The generation of a unique machine description for chemical structures-a technique developed chemical abstracts service. Journal of chemical documentation, 5(2): 113, 1965. 4
[24] Weihua Hu, Bowen Liu, Joseph Gomes. Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXiv preprint arXiv:1905.1220 12265. 2019. 2,
[25] David Weininger. Smiles, chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36, 1988. 2,
[26] Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta: Large-scale self-supervised pretraining for molecular property prediction. arXiv preprint arXiv:20/0.09885, 2020. 5, 6, 24
[27] Sheng Wang, Yuzhi Guo, Yuhong Wang, Hongmao Sun, and Junzhou Huang. Smiles-bert: large scale unsupervised pre-training for molecular property prediction. Proceedings of the 10th ACM international conference on bioinformatics, computational biology and health informatics, pages 429-436. 2019. 24
[28] Jerret Ross, Brian Belgodere, Vijil Chenthamarakshan, Inkit Padhi, Youssef Mroueh, and Payel Das. Large-scale chemical language representations capture molecular structure and properties. Nature Machine Intelligence, (211)::256-1266 2022. 2. 6, 24

---

[29] Yuyang Wang, Jianren Wang. Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4(3):77--877, 2022. 24
[30] Shengchao Liu, Hanchen Wang. Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre-training molecular graph representation with 3d geometry. arXiv preprint I 2021. 2, 6, 24
[31] Xiaomin Fang, Lihang Liu, Jicqiong Lei, Donglong Hc, Shanzhuo Zhang. Jingho Zhou, Fan Wang, Hua Wu, and Haifeng Wang. feeometry-enhancee molecular representation learning for property prediction. Nature Machine Intelligence, 4(2):127-134) 2022.
[32] Shuangli Li, Jingho Zhou, Tong Xu, Dejing Dou, and Hui Xiong. Geomgel: Geometric graph contrastive learning for molecular property prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 154114549, 2022. 23, 24
[33] Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Duta Mining, pages 594-604, 2022. 6, 24
[34] Jinhua Zhu, Yingce Xia, Lijun Wu, Shufang Xie, Tao Qin, Wengang Zhou, Houqiang Li, and Tie- Yan Liu. Unified 2d and 3d pre-training of molecular representations. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2626-2636, 2022. 2, 24
[35] Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine lcarning models. In International Conference on Machine Learning, pages 6950-6960. PMLR, 2020. 3, 9
[36] Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection for high pruning rates. In The Eleventh International Conference on Learning Representutions, 2022.
[37] Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for roal-world data-efficient deep learning. In The Eleventh International Conference on Learning Representations, 2022. 3,
[38] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 92)):513-330 2018. 3, 7, 24
[39] AIDS Antiviral Screen Data. URL WtLLLttp:://idd............ htt.aa/.......g.//////////ttttt. Sviral=Screen+Data. 3,
[40] Sebastian G. Rohrer and Knut Baumann. Maximum Unbiased Validation (MUV) Data Sets for Virtual Screening Based on PubChem Bioactivity Data. J. Chem. Inf. Model., 496 184, 2009. 3, 17
[41] Yanli Wang, Jewen Xiao, Tugba O Suzek, Jian Zhang, Jiyao Wang. Zhigang Zhou, Lianyi Han, Karen Karapetyan. Svetlana Dracheva, Benjamin A Shoemaker, et al. Pubchem's 's bioassay database. Nucleic acids research, 40(D1):D400-D44 D412. 2012. 3. 17
[42] Lars Ruddigkeit, Ruud Van Deursen, Lorenz C Blum, and Jean-Louis Reymond. Enumeration of 166 billion organic small molecules the chemical universe database gdb-17. Journal of chemical information and modeling, 52(11): 2875, 2012. 3, 17, 18, 23
[43] Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Elaus-Robert Müller. Schnet: A continuous- filter convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017. 4. 16, 23
[44] Greg Landrum, Paolo Tosco, Brian Kelley, Ric, sriniker, gedeck. Riccardo Vianello, NadineSchneiderr Eisuke Kawashima, Andrew Dalke, Dan N, David Cosgrove, Brian Cole, Matt Swain, Samo Turk, AlexanderSavelyev. Gareth Jones, Alain Vaucher, Maciej Wojcikowski, Ichiru Take, Daniel Probst, Kazuya Ujihara. Vincent F. Scalfani, guillaume godin, Axel Pahl, Francois Berenger, JLVarjo, strets123, JP, and DoliathGavid. rdkit/rdkit: 2022_03 (ql 2022) release, 2022. 4
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention you need. Advances neural information processing systems, 30, 2017. 4, 23
[46] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics. pages JMLR Workshop and Conference Proceedings, 2010. 4
12

---

[47] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv: 1412.6980, 2014. 4
arXiv preprint
[48] Shun-ichi Amari, Naotake Fujita, and Shigeru Shinomoto. Four types of learning curves. Computation, 4(4):605 618, 1992. 5
Neural
[49] Maho Nakata and Tomomi Shimazaki. Pubchemqe project: large-scale first-principles electronic structure database for data-driven chemistry. Journal of chemical information and modeling, 57(6):1300-1308 2017. 5, 18
[50] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong. Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information procesing systems, 33:22118- 18-22133, 2020. 5, 18
[51] Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference Machine Learning, pages 9377-9388. PMLR, 2021. 5, 23
[52] Yi Liu, Limei Wang, Meng Liu. Xuan Zhang. Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d graph networks. arXiv preprint arXiv:2102.0500 2021. 5, 23
[53] Jun Xia, Yanqiao Zhu, Yuanqi Du, and Stan Z Li. A systematic survey of chemical pre-trained models.
[54] Danny Hemaandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint wrXiv:2102.0129)) 2021. 6, 24
[55] Bharath Ramsundar, Peter Eastman, Pat Walters, and Vijay Pande. Deep learning for the life sciences: applying deep learning to genomics, microscopy, drug discovery, and more. "O'Reilly Media, Inc.", 2019.
[56] Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. arXiv preprint arXiv: 1203.32203.342,.00 2012. 9, 21
[57] David D. Lewis and William A. Gale. A sequential algorithm for training text classifiers. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development Information Retrieval, 1994. 9, 21,
[58] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv: 182.05579, 2018. 9, 2,21,22 22
[59] Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34:20596- 20607, 2021. 9, 21, 22
[60] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Procexsing Systems, 35: 19523-19536, 2022. 9, 21, 22, 24
[61] Boris Weisfeiler and Andrei Leman. A Reduction of a Graph to a Canonical Form and an Algebra Arising During This Reduction. Shuccmo--echmicheskee Informatsia, 129):12-16, 1968. 16
[62] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019. 17
[63] Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint 110iv:1903.0000 024288 2019. 17
[64] Greg Landrum et al. Rdkit: Open- source cheminformatics software. 2016. URL http:///ww. rdkit, org/. https://githubb com/rdkit/vdkit, 1496150)6500, 2016. 17
[65] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of-the-art natural language processing. arXiv preprini arXiv:1910.0377) 2019. 17
[66] Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp. and O Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 2014. 18
13

---

[67] Chengcheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selection in deep learning. In Databuse and Expert Systems Applications: 33rd International Conference, DEXA 2022, Vienna, Austria, August 22-24, 2022, Proceedings, Part I. pages 181-195. Springer, 2022.
[68] Talia B Kimber, Yonghui Chen, and Andrea Volkamer. Deep learning in virtual screening: recent applications and developments. International Journal of Molecular Sciences, 22(())4435, 2021,
[69] Stefan Chmiela, Alexandre Tkatchenko, Huziel Sauceda, Igor Poltavsky, Kristof Schütt, and Klaus- Robert Müller. Machine learning of accurate energy-conserving molecular force fields. Science advances,
2017. 23
[70] Johannes Gasteiger, Janek GroB, and Stephan Günnemann. Directional message passing for molecular graphs. arXiv preprint 2xXxv:203.03//22 2020. 23
[71] Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 34:6790-6802,, 2021. 23
[72] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (a) equivariant graph neural networks. In International conference on machine learning. pages 9323-9332. PMLR, 2021. 23
[73] Bowen Jing. Stephan Eismann, Pratham N Soni, and Ron Dror. Equivariant graph neural networks for 2021. 23
3d macromolecular structure. arXiv preprint arXiv:2106.08:
[74] Weitao Du, He Zhang. Yuanqi Du, Qi Meng, Wei Chen, Nanning Zheng, Bin Shao, and Tie- Han Liu. Se (3) equivariant graph neural networks with complete local frames. In International Conference on Machine Learning, pages 5583-5608 PMLR. 2022. 23
[75] Weitao Du, Yuanqi Du, Limei Wang. Dieqiao Feng, Guifeng Wang. Shuiwang Ji, Carla Gomes, and Zhi thig Ma. A new perspective on building efficient and expressive 3d equivariant graph neural networks. arXiv preprint arXiv:2304.0477 04757, 2023. 23
[76] Mario Geiger and Tess Smidt. e3nn: Euclidean neural networks. arXiv preprint arXiv:2207.09453 2022.
23
[77] Adrien Poulenard and Leonidas Guibas. A functional approach to rotation equivariant non- linearities for tensor field networks. In Proceedings the Recognition, pages 13174-13183, 2021.
the IEEE/CVF Conference on Computer Vision and Pattern
[78] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling, Se (3)-transformers: 3d roto-tramslation equivariant attention networks. Advances in neural information processing systems, 33:1970-1981, 2020.
[79] Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs arXiv preprint 1xXi:220061199000 2022.
[80] llee Batatia, David P Kovacs, Gregor Simm, Christoph Ortner, and Gábor Csányi. Mace: Higher order equivariant message passing neural networks for fast and accurate force fields. Advances in Neural Information Processing Systems, 35:1 1423-11436, 2022.
[81] David Haussler. Quantifying inductive bias: Ai learning algorithms and valiant's learning framework. Artificial intelligence, 36(2):177-221, 1988. 24
[82] Andrzcj Ehrenfeucht, David Haussler, Michael Kearns, and Leslic Valiant, A general lower bound on the number of examples needed for learning. Information and Computation, 82(3):247-22 2611, 1989.
[83] Anselm Blumer, Andrzej Fhrenfeucht, David Haussler, and Manfred K. Warmuth, Learaailiiyy and the vapnik-chervonenkis dimension. Journal of the ACM (JACM), (6(4):929-965 1989.
[84] David Haussler, H Sebastian Scung, Michael Kearns, and Naftali Tishby. Rigorous learning curve bounds from statistical mechanies. In Proceedings of the seventh annual conference on Computational learning theory, pages 87, 1994. 24
[85] Michele Banko and Eric Brill. Scaling to very very large corpora for natural language disambiguation. In Proceedings of the 39th annual meeting of the Association for Computational Linguistics, pages 26-33, 2001. 24
[86] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. arXiv preprint 2rX 2022. 24
14

---

[87] Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. urXiv preprint urXiv:2210.1488), 2022. 24
[88] Nathan Frey, Ryan Soklaski, Simon Axelrod, Siddharth Samsi, Rafael Gomez-Bombarelli, Connor Coley, and Vijay Gadepally. Neural scaling of deep chemical models. 2022. 24
15

---

A Implementation Details
A.1 Molecular Encoders
In this section, we detail the implementation of each encoder. We denote the representation for node (atom) U2 as h; and the representation at the graph (molecule) level as 2.
Embedding 2D graphs. Graph Isomorphism Network (GIN) [5] is simple and effective model to learn discriminative graph representations, which is proved to have the same representational power as the Priifeeler-Lehmane test [61]. Recall that each molecule is represented as G (A, X E), where A is the adjacency matrix, and are features for atoms and bonds respectively. The layer-wise propagation rule of GIN can be written
h [k+1) p(k+1)
(8) h
Σ JEN(()
(n1b)
Solart Esj))
where the input features h (0) wi, (i) is the neighborhood set of atom us, and Jaom, Ahond are two Multilayer Perceptron (MI.)) layers for transforming atoms and bonds features, respectively. By stacking K layers, we can incorporate K K-hop neighborhood information into each center atom in the molecular graph. Then, we take the output of the last layer as the atom representations and further use the mean pooling to get the graph-level molecular representation:
2D
Σ 1EV
h (K)
N
Embedding 3D graphs. We use the SchNet [43] the encoder for the 3D geometry graphs. SchNet models message passing in the 3D space as continuous-filter convolutions, which is composed of series of hidden layers, given as follows:
h (l+1) -MMP
(6) Σ JFa ( m.rr)) + h/
where the input h."0 a, is embedding dependent on the type of atom U. fFa(-) denotes the filter-gendn getter-gnnrattig network. To ensure rotational invariance of a predicted property, the message passing function is restricted to depend only rotationally invariant inputs such as distances, which satisfying the energy properties rotational equivariance by construction. Moreover, SchNet adopts radial basis functions to avoid highly correlated filters. The lltte-ggneraating network is defined follow:
frG j Ti, r3) .ÿ (r.r, r,) = x exp(->ille ry =l1l))
(5)
Similarly, for non-quantum properties prediction concerned in this work, we take the average of the node representations as the 3D molecular embedding:
3D
h (K)
(6)
N
iEV
where is the number hidden layers.
Embedding fingerprints & SMILES strings. Due to the discrete and extremely sparse nature of fingerprint vectors, we first transform all F binary feature fields into a dense embedding matrix FÍP E f DDr via embedding lookup, while we transform SMILES tokens in the same way via another embedding lookup Fumm E R Fr DD; Then, we introduce a positional embedding matrix P Fx to capture the positional relationship among bits in the fingerprint vector, which is defined as:
Pp.2: = sin(p/10000
(7)
P PA2i11 = cos(p/10000///10)
(8)
16

---

where denotes the corresponding bit position and is corresponds to the i-th embedding dimension. The positional embedding matrix will be added the transformed embedding matrix:
+.P
Thereafter, multihead Transformer [45] model the interaction among those feature fields. Specifically, we first transform each feature into a new embedding space as:
Q (k) FW.() (h)
(10)
K(
FW (h)
(11)
(h) FW (h)
(12)
where the three linear transformation matrices (h WK W E RDxx parameterize the query, key, and value transformations for the h-th attention head, respectively. Following that, compute the attention scores among all feature pairs and then linearly combine the value matrix from all H attention heads:
Q (Q() (Q()( T
w(h) (h) softmax
(13)
DDH
2 = (
W v
W (1) v((]]
(14)
Finally, we perform sum pooling on the resulting embedding matrix Ż E R Fx and use linear model fLIN to obtain the final fingerprint or SMILES string embedding R.:
Dr Σ
2 = =fLIN
Za
(15)
A.2 Computing infrastructures
Software infrastructures. All of the experiments are implemented in Python 3.7, with the following supporting libraries: PyTorch 1.10.2 [62], PyG 2.0.3 [63], RDKit 2022.03.1 [64] and HuggingFace 's Transformers 4.17.0 [65].
Hardware infrastructures. We conduct all experiments on computer server with 8 NVIDIA GeForce RTX 3090 GPUs (with 24GB memory each) and 256 AMD EPYC 7742 CPUs.
A.3 Code availability
The source code of our empirical implementation can be accessed at https: //gtthub/ heweitthb.cooeeeeeeeeeeeeeeeeeee com/Dat a M-reindeer/NSL_MM
B Datasets and Tasks
In the following, we will elaborate on the adopted datasets and the statistics are summarized in Table 2.
Datasets. We consider four datasets ranging from molecular-Ievel properties to macroscopic influ- ences on human body for experimental investigation: HIV [39], MUV [40], PCBA [41] and QM9 [42].
HIV (AIDS Antiviral Screen) was developed by the Drug Therapeutics Program (DTP) [39], which is designed to evaluate the ability of molecular compounds to inhibit HIV replication.
Maximum Unbiased Validation (MUV) group was selected from PubChem BioAssay via a refined nearest neighbor analysis approach, which is specifically designed for validation of virtual screening techniques [40].
PubChem BioAssay (PCBA) is database consisting of biological activities of small molecules generated by tigh-throughput screening [41].
17

---

Table 2: Statistics of datasets used in experiments.
Dataset
Data Type
#Molecules
Avg. fatoms
Avg. #boond
Tasks
Avg. degree
Pre-training PCQM4Mv2
SMILES
3.746.620
14.14
14.56
2.06
MUV
SMILES
93,087
24.23
26.28
2.17
Classification
HIV
SMILES
41,127
25.51
27.47
2.15
PCBA
SMILES
437,929
25.96
28.09
92
2.16
QM9-<pp
SMILES 3D
130,831
18.03
18.65
2.07
Regression
QM9-U0
SMILES, 3D
130,831
18.03
18.65
2.07
QM9-ZPVE
SMILES, 3D
130,831
18.03
18.65
1
2.07
QM9 is a comprehensive dataset that provides geometric, energetic, electronic and thermodynamic properties for subset of GDB-17 database, comprising 134 thousand stable organic molecules with up to nine heavy atoms [42]. In our experiments, we delete 3,054 uncharacterized molecules which failed the geometry consistency check [66]. We include the Egup. UO, and ZPVE in our experiment, which cover properties related to electronic structure, stability, and thermodynamies. These properties collectively capture important aspects of molecular behavior and can effectively represent various energetic and structural characteristics within the QM9 dataset.
PCQM4Mv2 is quantum chemistry dataset originally curated under the PubChemQC project [49]. Based on the PubChemQC, Hu et al. [50] define meaningful ML task predicting DFT-calculated HOMO-LUMO energy gap of molecules given their 2D molecular graphs. The HOMO-LUMO gap is one of the most practically-releve related to reactivity, photoexcitation, and charge transport.
quantum chemical properties of molecules since
C
Additional Experimental Results
C.1 The Neural Scaling Law on PCQM4Mv2 Dataset
Following the experimental settings Section 3.1, we conduct additional experiments on the PCQM4Mv2, large-scale challenge dataset provided by OGB 4. To the best of our knowledge, it currently the largest labeled molecular dataset available, containing over 3.7 million molccular samples. It is worth noting that due the unavailability of official test labels, selected 50% from the validation set as the test data and report the results in Figure 7.
PCQM4Mv2
PCQM4Mv2 (log-log)
0.40
MAE
0.35
MAE
0.30
1% 10%
30%
60%
100%
10
10-3
10a
Select Ratio
Select Ratio
Figure 7: The neural scaling law of molecular representation learning PCQM4Mv2 dataset.
C.2 The Effect of Equivariant Representations on the Scaling Law
To further explore the scaling laws of more powerful 3D graph representations, we have extended our investigation to include SE(3) invariant model (SphereNet) and E(3) equivariant model (PaiNN) in the context Section 3.1. The empirical results are demonstrated Figure It can be observed that most of the performances generally adhere to the pawer-law relationship and the difference lies in the coefficients and exponents of the power law. Furthermore, the relative differences performance
'nttps://obb.s/t stanford...
18

---

(a) QM9 UD
(b) QMB
(c) QMB zpve
0.5
3--
0.3
0.00
02
0.04
1--
011
1961.5
192.0%
194.0%
35%
(di UO (lng-log)
(e) gaip (log-log)
zpve (log-log)
10a
MAIE (o0g)
MAI
MAR
1011
PWNN
Spheesliet
SchWel
Figure 8: The neural scaling law of equivariant representations with 3D graph modality.
between the three models tend to vary with distinct predictive tasks. For instance, in the UO prediction task, the power law relationships of the three models exhibit noticeable parallel pattern in the log-log plot, whereas in the other two tasks, a crossing effect becomes evident.
C.3 More Results of the Effect Data Split
ROCAUC
0.80
0.65
0.70
0.10
C-
0.35
1 10
30
60
160
1 10
30
60
100
1.0
30
60
100
(alHlv
(6) BUV
1c1 POBA
2x
6x10
10
10
10P
10mM
10
10"
10aH
10tt
10"
(d) HIV (og logo
MUV 0og log)
PCBA Dog log)
Figure 9: The effect of data split on the scaling law with fingerprint modality.
To enhance the robustness of our findings in Section 3.4, addition to the comparison of the 2D graph modality across various data splits provided in the main text, we also conduct analogous experiments on the fingerprint and SMILES modalities. The empirical results, illustrated Figure 9 and Figure 10, align with our observations in the Section 3.4: (1) Most performance within these modalities adhere to (2) The SMILES modality still exhibits a performance drop phenomenon in the MUV datasct. The only deviation lies in the smallest dataset, HIV, where the performance variation under imbalanced/sccffoll split settings does not conform the power law. This deviation might be attributed to the small scale of the HIV dataset, posing challenges for models to generalize and extrapolate effectively.
the power law relationship.
C.4 Comparison of Different Fingerprint Encoders
Given the absence of established encoder design for fingerprints in the field, we conduct comparison between two classic encoders, a single-layer MLP and single-layer Transformer, in terms of their performance variation. The results indicate that both encoders exhibit strong adherence to the
19

---

ROC ROCAUG
ROc AUC
0.65
0.80
0.50
10
30
100
30
60
100
121HV
(3) MUV
KT POSA
6x101
4x
HOL-AUE
6x10-1.
5x10-1,
2x
10
10t1
10
10
10t1
10'
10
10t1
10
(d) HIV (kiglog)
(e) MUV (egkog))
PCRA (egglog)
random
Imba inbaldnred
scaffold
Figure 10: The effect of data split on the scaling law with SMILES modality.
0.20
ROC AAC
0.65
0.10
0.05
1
10
20
60
100
10
30
100
10
30
66
100
(6) MUV
POBA
2x10-
xx10-
10-1
60.10-
6x10
(ngglog)
(nggog) Ologle layer)
(og-log)
Figure 11: The Effect of Model Architecture on the Scaling Law with fingerprint Modality.
power law relationship in their scaling behavior. As such, the choice of encoder does not impact the robustness of our conclusions.
C.5 Single-property performance of MUV dataset with SMILES modality
In order to gain a more detailed understanding of the performance degradation phenomenon in the multitask scenario MUV, we specifically demonstrate the neural scaling behavior of the SMILES modalities in single-property ROC-AUC, as shown in Figure 12. Despite some ascending behavior, the bulk of properties exhibit varying degrees of performance drop in the last few proportions, which accounts for the overall performance drop in the multi-task setting.
C.6 Results of different-layers Transformer with SMILES modality.
Figure 13 presents the scaling behavior of transformers with different numbers of layers in terms of their performance. It can be observed that RoBERTa achieves the overall best performance, followed by 1-layer Transformer, and the worst performance is exhibited by 3-layer one. In contrast to fingerprint, SMILES modality could experience a performance drop on some datasets (HIV and MUV) in the high-data regime. Additionally, the varying numbers of Transformer layers do not affect our conclusions Section 3.2 regarding modality comparison, as even the superior RoBERTa model overall does not surpass the performance of graph and fingerprint modalities.
20

---

14
14
Figure 12: The neural scaling law single-propeny performance (ROC-AUC) of MUV dataset with SMILES modality.
0.25
80C AUC
0.70
ROC AUC
0.15
0.65
0.55
0.60
0.50
15510%
30%
60%
100%
15410%
30%
60%
100%
15400%
30%
60%
100%
(d) (log-log)
(log-log)
PCBA (log-log)
10m1
10-1
HOC-AUE
10-
10-1
10-2
10-1
layor Transtomer
Figure 13: The neural scaling law of different-layer model (Transformer) performance with SMILES modality.
C.7 Data pruning strategies and additional results
Problem statement. Consider learning scenario where we have large training set denoted as
T = (æ;, y/ [2]] consisting of input-output pairs (æ;, where æ; E X' represents the input and V: )' denotes the ground-truth label corresponding ø;. Here, A' and )' refer to the input and output spaces, respectively. The objective of data pruning is to identify subset S C T, satisfying the constraint [S] [7]. that captures the most informative instances. This subset, when used to train a model denoted as 0 should yield similar or better generalization performance to that of the model 07, which is trained on the entire training set T.
Data pruning strategies. In data pruning experiments, we implement total of seven data pruning (or coreset selection) strategies: Herding [56], Entropy [57]. Least Confidence [57], For- getting [58], GraNd [59], K-means [60] and additionally include random pruning baseline method. These seven strategies are widely used in the field of CV [67] and have potential in mitigat- ing the issue of data redundancy in large-scale datasets, thereby saving computational and storage resources. Here we provide a brief overview to each of them.
Herding [56] operates by selecting data points in the feature space based on the distance between the coreset center and the original center. follows an incremental and greedy approach, adding sample at time to the coreset in order minimize the distance between the centers.
21

---

Table 3: Empirical performance of data pruning strategies on eight subsets the MUV dataset in terms of ROC-AUC (%, 1). Results that are significantly higher or lower than random pruning (with a p-value less than 5% in the significance test) are highlighted.
Uniform
1%
5%
10%
20%
30%
40%
60%
80%
Random
47.4
58,3
59.6+4.
68.9-2.1
71.2-a.
70.2+3.
72.9x1.7
77.4a2x
Herding
48.4s47
51.9s48
53. +10.3
61.5a49
68.5=7.3
65.1+56
71.7
78.7+50
Entropy
48 48.925.
58.5
62.8s4.0
63 13.8242
68,7224
71.023.
75.012.5
79.0238
Least Confidence
52,0279
57 57.425.7
61.1433
66.322.1
67. 07.6.48
70.0s3.6
75.1
78,5s22
Forgetting
47.1
58.6111
60.925.2
63.3a18
67.1
70.5
74.5
76,9238
GraNd
47.3-34
52.2+7.
64.7+5.1
65.943.1
64,4-59
71.4+44
72.5+2.5
76.2+3.2
k-means
49.9-47
60,5-78
65.5+3.7
68.0s42
65.5-37
67.1+28
72.1414
76.9-4n
Imbalanced
5%
10%
20%
30%
40%
60%
80%
Random
47.7435
58.3214
60.414.7
64. 122.1
66,8.34
67.413.0
72.4
66.3a28
Herding
50.2s47
55.8
59.3e10.3
58.8s49
64.5-73
65.6e5.6
65.846.4
69.6.50
Entropy
47.7=s.
57.5es.1
61.0s4.0
68.0 28.004.
64,4
67.2+3.7
1
69.5+3x
Least Confidence
50.4279
52.925.7
60.5
64.2a2.1
65.5248
68.323.
69. 41..5
66.4122
Forgetting
49.3,2.1
51.4.2.1
38.72
64.021.
65.7=3.0
66.5228
67.7a2.6
68, 81388
GraNd
52.0234
55.327.1
63.815.1
63.423.1
67.0=5.9
68.624.4
70.3
99.0233
k-means
52.7-47
56.0+78
58.5+37
61.0+42
63.4-37
63.4+28
67.1+14
70.0<4
Entropy and Least Confidence [57] iteratively select samples with lower entropy and least confidence, respectively. These methods identify informative samples by considering that lower uncertainty can provide more information gain, thereby benefiting model training and reducing data redundancy.
Forgetting [58] calculates the frequency of forgetting that occurs during the training process, which refers to the number of times the samples correctly classified in the previous epoch misclassified the current epoch. Those unforgettable samples, exhibiting robust performance across epochs, have minimal impact model performance when removed.
GraNd [59] measures the average impact of each sample the reduction of training loss during the initial epochs. Training samples are more important if they contribute more to the error or loss when training neural networks.
k-means [60] employs the application of k-means clustering in the latent space define the difficulty of each data point based on its Euclidean distance to its nearest cluster centroid. Simple samples (with low difficulty) are considered for removal reduce data redundancy. It noteworthy that this method, unlike the aforementioned approaches, does not require any label information or training and can be directly applied the dataset.
Additional data pruning results. Here include additional results of data pruning performance on MUV and HIV datasets, as shown in Table and Table 4. Observing the performance of these data pruning strategies, none of these methods consistently outperform random sampling, which aligns with our observation in Section 3.6. Note that in the case of MUV with an imbalanced data split, several data pruning methods such as Random, GraNd, and Least Confidence even demonstrate performance decline when the subset scale surpasses 60%.
D Related Work
The following section provides more broad literature review across the spectrum of molecular representation learning and neural scaling law.
D.1 Molecular representation learning
The past decade has seen remarkable success the application of deep learning in a variety of biochemical tasks, spanning from virtual screening [68] to molecular property prediction [6]. Within
22

---

Table 4: Empirical performance of data pruning strategies on eight subsets of the HIV dataset in terms of ROC-AUC (%, 1). Results that are significantly higher or lower than random pruning (with a p-value of less than 5% in the significance test) are highlighted.
Uniform
1%
5%
10%
20%
30%
40%
60%
80%
Random
63.4=28
69.8+22
75.0
78.5+1.
79.0x22
79.0st.3
81.5+1.7
83.8s0s
Herding
60.2=39
63,3
64. 745.0
69.5+x4
71.8-70
75.8
80.0+2.8
82. 02.6
Entropy
67.9.22
71.1
74.2
76.2a12
77.0a2.
79.2a1.8
81 4119
83.2a1.
Least Confidence
66.2240
70.422.1
72.8
76.7
78.021.
81.02t.4
81.6115
83.3
Forgetting
67.7.12
75.2aL.
15.
76.2a1.7
80.0118
79,8at.6
82.8a1.
83.721.4
GraNd
66,2-40
69.3+26
73.6+2.
18.1
78.1-16
78.6-10
82.3+08
83.2+1
k-means
63.8-48
64,4+14
65.7+1.8
68.1+16
71.5-14
72.5+1.5
79.2+n.5
82.3+a2
Imbulanced
1%
5%
10%
20%
30%
40%
60%
80%
Random
66.6.177
68.613.1
69.913.7
70.9a14
70.724.1
72.1
74.1
74.4
Herding
57.1=30
63.0aa8
64.943.6
65.8+5.9
67.3a6.
72.6+1.8
73.3+2.2
73.7=06
Entropy
67.7=75
71.5+28
1
71.2+2.1
73.2+23
71.7+26
74.7=1.3
74.8=10
Least Confidence
66.8252
71.42L.
71.322.
71.8227
69.5
73.723.
73.4226
73.8218
Forgetting
66.1
69.7a5.8
70.243.6
71.9219
71.6118
71.422.0
73.9214
74.222.
GraNd
62.7.45
71.0226
69. 213.6
73.1
70.0234
72.9230
74.4
75.9212
k-means
67.9-18
65.4+1.2
65.0+1.9
67.1+4.3
69.1
68.5+4.
72.8+12
74.4+1.
this context, molecular representation learning (MRL) serves as a pivotal link between the molecular modalities and the target tasks, efficiently capturing and encoding rich chemical semantic information into vector representations.
One of the mainstream research approaches in MRL is based on 2D topology graphs. The advance- ments in Graph Neural Networks (GNNs) have enabled the application of more powerful GNN models in the field of molecular chemistry [5, 8, 11, 9, 10. 14), which has proven effective in en- hancing the discriminabilityy between representations and capturing underlying chemical semantics. The study of the expressive power of GNNs using the Weisfeiler- has been widely applied in MRL. GIN [5], one of the most representative works, develops a simple and effective architecture based on to be as powerful as the WL test. Some works propose improvements in the expressive power of GNNs to address issues related to long-range interactions [11, 10], higher-order structures [9, 8] and substructure recognition [14] from different perspectives. Unlike traditional message passing mechanisms, Graphormer [12] have explored the direct application of Transformers [45] to graph representation with tailor-made positional encoding. few research [13, 32] focus more on the ad-hoc model design for biochemical tasks, incorporating constraints based on molecular physics and chemical properties.
graph isomorphism test
layer (MLP) that has been proven
More recently, 3D graph representation for molecules has been largely explored thanks to its im- portance in modeling the dynamical behaviors of molecular structures [42, 69]. Different from 2D graph representation, euclidean group symmetry (E(3)/EE(3) including translation, rotation and reflection needs to be baked into the design of the models. Specifically, SchNet [43] is one of the earliest work that incorporates euclidean distances between each pair of nodes as features which make the model B(3)-invariant. In addition to pairwise distance features, angle (between three atoms) and dihedral angle (between four atoms) are later introduced as features into (0t//S(((--inariin)) models [52, 70, 71]. Instead of invariant models, another branch of work explore equivariant design of 3D graph neural networks. EGNN [72] devises simple equivariant update layer such that only linear transformation is applied to vector input to achieve equivariance in which similar ideas have also been applied to GVP [73] and PaiNN [51]. ClofNet [74] and LEFTNet [75] on the other side improves the expressiveness of such design by building local frams to scalarize equivariant inputs from there any neural architecture could be applied then transform back to vectors through vectorization block without information loss. Another noticeable line of work leverage spherical harmonics and tensor product construct equivariant layers [76-80].
In the advancements of supervised MRI., there has been limited progress in the model designs specifically tailored to the SMILES string and fingerprint modalities. It is worth noting that the
23

---

early benchmark models proposed in MoleculeNNet [38] have maintained their competitiveness over time. With the rise of pre-training research paradigms, there has been promising progress in recent years towards pre-training approaches based on these two modalities [27, 28, 26, 16] as well as the former two. By employing contrastive [29-32] and generative [33, 30, 34] self-supervised strategies, molecular pre-training approaches guide the model training and subsequently facilitate positíve transfer to downstream tasks. However, as mentioned in Section 3.3, existing molecular pre-training still suffer from issues such parameter ossification [54], necessitating further exploration for more data-efficient and training-efficieen models.
D.2
Neural scaling law
The study of neural scaling law can be traced back to early theoretical analyses of bounding general- ization error [81-84]. These works, based on assumptions about model capacity and data volume, reveal power-law relationships between the bounds of model generalization error and the amount of data. However, the conclusions drawn from these theoretical studies often yield loose or even vacuous bounds, leading disconnection between the theoretical findings and the empirical results of generalization error.
Early follow-on research have investigated empirical generalization error scaling, which represents an initial attempt at exploring the neural scaling law. Bango and Bill [85] conduct experiments on language modeling problem called confusion set disambiguation, using subsets of a large-scale text corpus containing billions of words. Their findings suggest a power-law relationship between the average disambiguation validation error and the size of the training data. Similarly, Sun et al. [21] demonstrate that the accuracy of image classification models improves with larger data sizes and conclude that the accuracy increases logarithmically based on the volume of the training data size.
Hestness et al. [18] empirically validate that model accuracy improves as a power-law as growing training sets in various domains, which exhibit consistent learning behavior across model architectures, optimizers and loss functions. However, there exists generalization error plateau in small data region and irreducible error region. With broader coverage, Michael et al. [22] present findings that consistently show the scaling behavior of language model log- likellhoooo loss in relation non- embedding parameter count, dataset size, and optimized training computation. They leverage these relationships to derive insights into compute scaling, the extent of overfitting, early stopping step, and data requirements in the training of large language models.
In recent years, several investigations of neural scaling laws specific to particular tasks have been conducted [86, 19, 87, 60]. Unlike previous research, while power-law relationships hold within specific ranges of data size model parameter count, certain tasks exhibit unique and uncommon learning behaviors. For instance, only marginal performance gains are expected beyond a few thousand examples in image reconstruction [20]. Moreover, there relevant study sharing a common spirit with our work in analyzing MRL from a scaling behavior viewpoint [88]. However, our research diverges in the following aspects: (1) Scope of Neural Scaling Law Investigation: The study of Frey et al. primarily considers large language models for generative chemical modeling and quantum property tasks. In contrast, our work encompasses a broader range of dimensions, such as modalities, data distributions, pre-training and model capacity, to investigate the influence of various factors on neural scaling laws. (2) Different contributions: Their contribution primarily revolves around developing strategies for scaling deep chemical models, notably large language models, and introducing novel model architectures. In contrast, our focus lies in exploring the impact of different dimensions data utilization efficiency through data-centric perspective and benchmarking existing data pruning strategies on MRL tasks.
24


Reviews:
[{"review_id": "np8KeDLjwv", "reviewer": "Reviewer_pZ6H", "summary": "", "questions": "", "limitations": "- Authors may provide some additional information on the limitations of their study.\n- I do not see any  negative societal impact of this work.", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Interesting empirical study investigating the performance of Molecular Representation Learning\n- Extensive experiments and detailed results, well documented\n- Interesting to see empirical evidence that MRL performace exhibits a power-law relationship with data quantity", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "JQ8IdpdMSN", "reviewer": "Reviewer_rEJP", "summary": "", "questions": "", "limitations": "* The authors only investigate invariant graph representation learning methods for their study's proposed graph modality. Equivariant graph representation learning methods have recently been shown to be theoretically and empirically more powerful methods for encoding 3D molecular graph inputs. As such, the effects of neural molecular scaling laws on equivariant representations remain uninvestigated.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "* The authors investigate a variety of molecular data modalities, highlighting the strengths (and weaknesses) of each.\n* The authors perform their experiments using a variety of molecular datasets of various sizes and compositions.\n* The authors succinctly highlight for readers the main takeaways of this work, section-by-section, allowing readers to closely follow the authors' line of reasoning behind their successive experiments.\n* The authors provide well-documented source code on GitHub to rerun their experiments.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "CvnSOTjhxP", "reviewer": "Reviewer_5fTN", "summary": "", "questions": "", "limitations": "The authors have included discussions regarding the limitations of the works. For more potential limitations, please refer to \"Opportunities For Improvement\".", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The topic is valuable and underexplored. How deep learning models scale in molecular property predictions can provide helpful guidance in building larger and better molecular datasets. \n2. The authors propose several interesting aspects of scaling law in MRL, including data quantity, modality, pre-training, etc.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "4GOD5O44rl", "reviewer": "Reviewer_weCW", "summary": "", "questions": "", "limitations": "Limitations and societal consequences are sufficiently discussed.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "(S1): The work investigates important research questions, outlines them neatly, and shows experiments which are (for the most part) relatively convincing. The general findings are practical, and certainly useful for practitioners working with molecular models. \n\n(S2): In particular, the analysis comparing the scaling behaviour of different modalities (graph, SMILES, fingerprint) is interesting and timely.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "jZkwrT8Wty", "reviewer": "Reviewer_jova", "summary": "", "questions": "", "limitations": "None", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. While research on molecular representation learning (MRL) is progressing at an amazing pace, the neural scaling laws specific to MRL have not received the same level of attention as other domains such as computer vision (CV) and natural language processing (NLP). The empirical results presented in this paper fill this gap and contribute significantly to the research of MRL. Specifically, the paper comprehensively explores how model performance changes across various dimensions, which is highly valuable to me.\n\n2. The paper demonstrates a comprehensive consideration of settings for most dimensions and presents reasonable result analysis. When the observed trends align with those in other domains, it confirms the applicability of neural scaling laws across diverse fields. For dimensions where unexpected observations or unexplainable phenomena arise, intriguing future research directions are motivated.\n\n3. Considering the promising applications of MRL in virtual screening and other aspects of drug design, this contribution has significant societal value, making it beneficial to society at large.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: ## Summary

The paper computes empirical scaling laws for molecular property prediction with a focus on how data scale and distribution affects test metrics. 

## Strengths

- [quality, clarity, significance] The work investigates important research questions, outlines them neatly, and shows convincing experiments. The general findings are practical, and certainly useful for practitioners working with molecular models.
- [significance, originality] The topic is valuable and underexplored. How deep learning models scale in molecular property predictions can provide helpful guidance in building larger and better molecular datasets.
- [quality] The paper considers a variety of prediction tasks and models. 

## Weaknesses

- Compared to the datasets used to compute scaling laws in CV and NLP, all the datasets used here are quite small. 
- Due to mostly focusing on GNNs, the paper is unable to compute scaling laws on model capacity over a wider range of capacities. 
- The pretraining section only considers one pre-training method, weakening those conclusions. 

## Explanation of recommendation

Although most reviewers gave the paper a 6, I do not see any major unaddressed concerns. On my reading, while the paper could make broader statements with bigger datasets or models, the work presented is well-done and has interesting conclusions for the community.
Decision: Accept (Poster)