SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The practice of dataset splitting in MRL is usually around two methodologies: random and scaffold splitting [38]. Random splitting ensures that both training and test samples are sourced from the same distribution, representing a uniform distribution setting. Conversely, scaffold splitting simulates the Out-Of-Distribution (OOD) scenario, wherein the training and test distributions are entirely distinct, pushing the boundaries of OOD generalization capabilities of MRL models. However, the realities of drug discovery often diverge from these two extremes. It is frequent for test compounds to possess substructures already encountered in the training samples [55]. To address this more realistic situation, we introduce imbalanced splitting, which generates the training, validation, and test sets based on scaffolds at first and then moves a fraction of the samples (5% in our experiments) from both test and validation sets to the training set. This approach serves as a trade-off between random and scaffold splitting, offering a more realistic reflection of real-world scenarios.
In this section, we still focus on 2D graph modalities and the learning behaviors on these three splitting methods by changing the scale of downstream datasets are shown in Figure 5. For readers interested in a deeper exploration, we append additional experiments in Appendix C.3 that analyze the neural scaling laws of two other modalities fingerprints and SMILES strings.
Observation 4. Our empirical observations validate that regardless of the dataset splitting methods, model performance conforms to a power law. Note that the performance curve with random splitting has the highest exponent, which suggests that this uniform setting has a higher data utilization efficiency compared to the other two OOD settings. We also observe outlier performance in the imbalanced splitting of the MUV dataset, where certain data scales (e.g.,  $80\%$  and  $100\%$ ) break the power laws, which we suspect is due to the examples selected outside the training set could mislead the model to capture spurious relationship between label and features. Importantly, these findings underscore the inherent challenges of MRL in real-world scenarios, which often face with the pervasive issues of distribution shifts.
3.5 Influence of the Model Capacity on the Scaling Laws
This section we study the relationship between model performance and capacity. In our MRL models, the capacity is largely determined by the number of layers (depth  $D$ ) and hidden units (width  $W$ ). Therefore, we characterize model capacity as  $D \times W$ , since the parameters of the embedding layer and output layer are the same across different models and hence can be sidelined from our evaluation. In the experiments, we focus on 2D topology graphs with GIN as encoders. Due to the inherit drawbacks of message-passing networks, such as over-smoothing and over-squashing, we carefully select four experimental configurations with reasonable increments:  $[64 \times 2, 100 \times 3, 300 \times 5, 600 \times 10]$ .
Figure 5: Neural scaling laws with three dataset splitting schemes: random, scaffold, and imbalanced splitting. Arrows indicate inflection points that deviate significantly from the fitted curves.
Figure 6: Neural scaling laws with the model capacity. We study four configurations of a GIN model with 128, 300, 1.5K, and 6K parameters respectively.
The performance with these four model configurations across different datasets is summarized in Figure 6.
**Observation 5.** The experimental results demonstrate that the relationship between model performance and capacity also adheres to the power law. However, it is observed that the optimal model capacity varies across different tasks and that a smaller model could be sufficient to achieve the optimal performance for some tasks. For example, for HIV and PCBA datasets, the model with a capacity of  $1.5\mathrm{K}$  parameters achieves the best performance. On the contrary, the smallest model with a capacity of 128 parameters achieves optimal performance on the MUV dataset. An interesting observation is that, on the PCBA dataset with over  $400\mathrm{K}$  data samples, there is significant difference in power law exponents among different model capacity. The models with lower capacity achieve saturated performance quickly as the data scales up, whereas this phenomenon is not evident in HIV and MUV datasets. In summary, we believe it is important to carefully select the appropriate model capacity according to the characteristics of datasets and tasks.
3.6 Data Pruning for Molecular Representation Learning
Lastly, we investigate the data pruning strategies in MRL, which identify a representative subset of the complete dataset. Such strategies have been proved effective to improving training efficiency in
Table 1: Empirical performance of data pruning strategies on eight subsets of the PCBA dataset in terms of ROC-AUC (\%,  $\uparrow$ ). Results that are significantly higher or lower than random pruning (with a  $p$ -value of less than  $5\%$  in the significance test) are highlighted.
Uniform	1%	5%	10%	20%	30%	40%	60%	80%
Random	5.2±0.1	9.6±0.2	13.2±0.2	17.7±0.5	20.0±0.6	22.2±0.6	24.9±0.5	26.8±0.4
Herding	3.7±1.5	9.7±0.6	10.2±3.6	15.8±3.7	20.7±0.7	22.6±0.6	25.4±0.8	24.5±3.4
Entropy	5.4±0.2	10.0±0.4	13.7±0.6	17.6±0.1	20.2±0.6	22.1±0.4	25.0±0.3	26.6±0.4
Least Confidence	5.3±0.1	9.7±0.3	13.7±0.5	17.6±0.4	20.3±0.3	22.0±0.5	25.0±0.3	26.5±0.2
Forgetting	5.5±0.2	9.8±0.4	13.5±0.4	17.5±0.3	20.4±0.3	22.1±0.2	24.7±0.6	26.5±0.2
GraNd	5.5±0.3	9.7±0.3	13.3±0.4	17.5±0.4	20.3±0.5	22.1±0.5	24.9±0.3	26.8±0.3
k-means	4.1±0.2	8.1±0.3	11.6±0.3	16.5±0.2	20.4±0.4	22.6±0.3	24.9±0.4	26.3±0.1
Imbalanced	1%	5%	10%	20%	30%	40%	60%	80%
Random	6.1±0.2	9.9±0.2	12.4±0.5	14.9±0.3	16.5±0.2	17.7±0.2	19.2±0.1	20.4±0.2
Herding	4.7±0.6	8.5±0.5	11.5±0.3	13.4±1.6	14.7±2.8	16.5±2.1	16.8±3.6	17.9±3.1
Entropy	6.0±0.3	9.9±0.4	12.3±0.4	15.1±0.5	16.8±0.5	17.8±0.2	19.3±0.4	20.5±0.3
Least Confidence	6.0±0.3	9.9±0.2	12.3±0.4	15.3±0.3	17.0±0.5	17.9±0.3	19.4±0.3	20.5±0.1
Forgetting	5.7±0.3	9.9±0.2	12.2±0.3	14.7±0.2	15.8±0.3	17.8±0.2	19.3±0.5	20.3±0.3
GraNd	5.9±0.4	9.7±0.3	12.2±0.5	15.0±0.6	16.8±0.3	17.8±0.4	19.4±0.2	20.7±0.5
k-means	4.6±0.3	8.1±0.3	10.7±0.3	13.9±0.2	16.4±0.4	17.2±0.3	19.6±0.5	20.4±0.1
CV and NLP domains [35-37] but their roles in MRL remain under-explored. In our experiments, we benchmark seven data pruning strategies originally designed for image data and adapt them for 2D molecular graphs: Herding [56], Entropy [57], Least Confidence [57], Forgetting [58], GraNd [59], and  $k$ -means [60]. We also include random pruning as a baseline method. Please refer to Appendix C.7 for a detailed description of these adopted data pruning strategies.
For all experiments, we employ each data pruning strategy to curate eight subsets, which are constituted of the following percentages of the whole dataset:  $[1\%, 5\%, 10\%, 20\%, 30\%, 40\%, 60\%, 80\%]$ . For a comprehensive investigation, we compare the results with two data splitting schemes to discern if performance of pruning strategies might be influenced by data distribution disparities. Table 1 summarizes the results on PCBA, one of our largest datasets, where we highlight significant results with a  $p$ -value less than  $5\%$  in the significance test (t-test) compared with random pruning.
Observation 6. Our empirical analysis reveals that no adopted pruning strategy consistently outperforms random pruning across various data subsets, irrespective of the dataset splitting methodology. The performance difference among these pruning strategies also remain narrow. These trends are consistently observed across the MUV and HIV datasets as well, with details provided in Appendix C.7. This leads us to postulate two potential hypotheses. First, the current molecular encoders might be adept at extracting the information from the tested datasets, thereby suggesting a minimal presence of data redundancy. This suggests a potential need for significantly larger datasets to showcase the efficacy of data pruning methods in MRL. Second, existing data pruning strategies, originally proposed for image data, may not transfer their effectiveness to molecular data. For example, similarity-based approaches like Herding might need the infusion of domain-specific knowledge when measuring the similarity of molecular graphs.
In this paper, we conduct a series of data-centric experiments on Molecular Representation Learning (MRL) spanning four dimensions, including (1) data modality, (2) dataset splitting, (3) the role of pre-training and (4) model capacity, to investigate the model performance under different data scales. Our empirical results demonstrate that the scaling behaviors generally conform to a power law, implying the marginal effect of adding more molecular data. Moreover, we observe that there is no one-size-fits-all configuration to different MRL datasets, which calls for task-specific analysis in real-world application. We also benchmark seven data pruning strategies and observe that none of these methods outperform random sampling. Our analysis underscores the unique challenges inherent in MRL, such as distribution shift in molecular discovery and the non-Euclidean nature of molecular
datasets. We anticipate that our findings will catalyze further research, driving more data-efficient methodologies for molecular representation learning.
Limitations. We conduct our experiments using widely-adopted model architectures in the field, such as GIN for the 2D topology graph, SchNet for the 3D geometry graph, and Transformer for SMILES string and Fingerprint. However, the rapid development of molecular representation learning in recent years has led to the emergence of models with improved expressiveness. The neural scaling laws on these models is still to be explored. Moreover, our focus is primarily on investigating the impact of various dimensions on supervised molecular representation learning. Despite that we have considered the effect of pre-training, we do not explore the neural scaling behaviors between data scales in pre-training and the corresponding performance on the downstream tasks.
Future work. In closing, we point out several prospective avenues for further exploration. (1) Neural scaling laws in molecular generative tasks. In this paper, our exploration has concentrated solely on predictive tasks, leaving an unexplored domain of generative tasks. Potential research might investigate neural scaling laws on tasks such as generation of drugs, conformations, and docking poses, to name a few. (2) Refinement of pre-training strategies. In our experiments, we discover that pre-training on molecular data does not improve efficiency of leveraging downstream data. Future work could consider to address the issues of parameter ossification to improve data efficiency in pre-trained MRL models. (3) In-depth analysis of data pruning strategies for molecular data. We observe that the existing image pruning methods do not significantly improve MRL performance. Such observations call for a rigorous evaluation of data pruning methodologies, potentially on more larger-scale datasets, to ascertain the potential advantages of data pruning techniques. Moreover, the design of efficient pruning strategies specifically tailored to molecular data is also a promising direction, which yet remain largely unexplored.
The authors would like to thank anonymous reviewers for their helpful feedback. This work is supported by National Natural Science Foundation of China (62141608, 62206291, 62372454).
[1] Yue Liu, Tianlu Zhao, Wangwei Ju, and Siqi Shi. Materials discovery and design using machine learning. Journal of Materiomics, 3(3):159-177, 2017. 1
[2] Jie Shen and Christos A Nicolaou. Molecular property prediction: recent trends in the era of artificial intelligence. *Drug Discovery Today: Technologies*, 32:29–36, 2019.
[3] Oliver Wieder, Stefan Kohlbacher, Méline Kuenemann, Arthur Garon, Pierre Ducrot, Thomas Seidel, and Thierry Langer. A compact review of molecular property prediction with graph neural networks. *Drug Discovery Today: Technologies*, 37:1-12, 2020.
[4] Xuan Zhang, Limei Wang, Jacob Helwig, Youzhi Luo, Cong Fu, Yaochen Xie, Meng Liu, Yuchao Lin, Zhao Xu, Keqiang Yan, et al. Artificial intelligence for science in quantum, atomistic, and continuum systems. arXiv preprint arXiv:2307.08423, 2023. 1
[5] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018. 1, 2, 4, 16, 23
[6] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263-1272. PMLR, 2017. 2, 22
[7] David Rogers and Mathew Hahn. Extended-connectivity fingerprints. Journal of chemical information and modeling, 50(5):742-754, 2010. 4
[8] Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Lio, and Petar Velicković. Principal neighbourhood aggregation for graph nets. Advances in Neural Information Processing Systems, 33:13260-13271, 2020. 2, 23
[9] Guohao Li, Chenxin Xiong, Ali Thabet, and Bernard Ghanem. Deepergen: All you need to train deeper gcns. arXiv preprint arXiv:2006.07739, 2020. 23
[87] Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. arXiv preprint arXiv:2210.14891, 2022. 24
[88] Nathan Frey, Ryan Soklaski, Simon Axelrod, Siddharth Samsi, Rafael Gomez-Bombarelli, Connor Coley, and Vijay Gadepally. Neural scaling of deep chemical models. 2022. 24
A.1 Molecular Encoders
In this section, we detail the implementation of each encoder. We denote the representation for node (atom)  $v_{i}$  as  $h_{i}$  and the representation at the graph (molecule) level as  $z$ .
**Embedding 2D graphs.** Graph Isomorphism Network (GIN) [5] is a simple and effective model to learn discriminative graph representations, which is proved to have the same representational power as the Weisfeiler-Lehman test [61]. Recall that each molecule is represented as  $\mathcal{G} = (\mathbf{A}, \mathbf{X}, \mathbf{E})$ , where  $\mathbf{A}$  is the adjacency matrix,  $\mathbf{X}$  and  $\mathbf{E}$  are features for atoms and bonds respectively. The layer-wise propagation rule of GIN can be written as:
$$
\boldsymbol {h} _ {i} ^ {(k + 1)} = f _ {\text {a t o m}} ^ {(k + 1)} \left(\boldsymbol {h} _ {i} ^ {(k)} + \sum_ {j \in \mathcal {N} (i)} \left(\boldsymbol {h} _ {j} ^ {(k)} + f _ {\text {b o n d}} ^ {(k + 1)} \left(E _ {i j}\right)\right)\right), \tag {2}
$$
where the input features  $\pmb{h}_i^{(0)} = \pmb{x}_i, \mathcal{N}(i)$  is the neighborhood set of atom  $v_i$ , and  $f_{\mathrm{atom}}, f_{\mathrm{bond}}$  are two MultiLayer Perceptron (MLP) layers for transforming atoms and bonds features, respectively. By stacking  $K$  layers, we can incorporate  $K$ -hop neighborhood information into each center atom in the molecular graph. Then, we take the output of the last layer as the atom representations and further use the mean pooling to get the graph-level molecular representation:
$$
\boldsymbol {z} ^ {\mathrm {2 D}} = \frac {1}{N} \sum_ {i \in \mathcal {V}} \boldsymbol {h} _ {i} ^ {(K)}. \tag {3}
$$
**Embedding 3D graphs.** We use the SchNet [43] as the encoder for the 3D geometry graphs. SchNet models message passing in the 3D space as continuous-filter convolutions, which is composed of a series of hidden layers, given as follows:
$$
\boldsymbol {h} _ {i} ^ {(k + 1)} = f _ {\mathrm {M L P}} \left(\sum_ {j = 1} ^ {N} f _ {\mathrm {F G}} \left(\boldsymbol {h} _ {j} ^ {(t)}, \boldsymbol {r} _ {i}, \boldsymbol {r} _ {j}\right)\right) + \boldsymbol {h} _ {i} ^ {(t)}, \tag {4}
$$
where the input  $\pmb{h}_i^{(0)} = \pmb{a}_i$  is an embedding dependent on the type of atom  $v_i$ ,  $f_{\mathrm{FG}}(\cdot)$  denotes the filter-generating network. To ensure rotational invariance of a predicted property, the message passing function is restricted to depend only on rotationally invariant inputs such as distances, which satisfying the energy properties of rotational equivariance by construction. Moreover, SchNet adopts radial basis functions to avoid highly correlated filters. The filter-generating network is defined as follow:
$$
f _ {\mathrm {F G}} \left(\boldsymbol {x} _ {j}, \boldsymbol {r} _ {i}, \boldsymbol {r} _ {j}\right) = \boldsymbol {x} _ {j} \cdot e _ {k} \left(\boldsymbol {r} _ {i} - \boldsymbol {r} _ {j}\right) = \boldsymbol {x} _ {j} \cdot \exp \left(- \gamma \| \left\| \boldsymbol {r} _ {i} - \boldsymbol {r} _ {j} \right\| _ {2} - \mu \| _ {2} ^ {2}\right). \tag {5}
$$
Similarly, for non-quantum properties prediction concerned in this work, we take the average of the node representations as the 3D molecular embedding:
$$
z ^ {3 \mathrm {D}} = \frac {1}{N} \sum_ {i \in \mathcal {V}} h _ {i} ^ {(K)}, \tag {6}
$$
where  $K$  is the number of hidden layers.
**Embedding fingerprints & SMILES strings.** Due to the discrete and extremely sparse nature of fingerprint vectors, we first transform all  $F$  binary feature fields into a dense embedding matrix  $\pmb{F}^{fp} \in \mathbb{R}^{F^{fp} \times D_{\mathrm{F}}}$  via embedding lookup, while we transform SMILES tokens in the same way via another embedding lookup  $\pmb{F}^{sm} \in \mathbb{R}^{F^{sm} \times D_{\mathrm{F}}}$ . Then, we introduce a positional embedding matrix  $\pmb{P} \in \mathbb{R}^{F \times D_{\mathrm{F}}}$  to capture the positional relationship among bits in the fingerprint vector, which is defined as:
$$
\boldsymbol {P} _ {p, 2 i} = \sin \left(p / 1 0 0 0 0 ^ {2 i / D _ {F}}\right), \tag {7}
$$
$$
\boldsymbol {P} _ {p, 2 i + 1} = \cos (p / 1 0 0 0 0 ^ {2 i / D _ {F}}), \tag {8}
$$
where  $p$  denotes the corresponding bit position and  $i$  is corresponds to the  $i$ -th embedding dimension. The positional embedding matrix will be added to the transformed embedding matrix:
$$
\boldsymbol {F} = \boldsymbol {F} ^ {\prime} + \boldsymbol {P}. \tag {9}
$$
Thereafter, we use a multihead Transformer [45] to model the interaction among those feature fields. Specifically, we first transform each feature into a new embedding space as:
$$
\boldsymbol {Q} ^ {(h)} = \boldsymbol {F} \boldsymbol {W} _ {\mathrm {Q}} ^ {(h)}, \tag {10}
$$
$$
\boldsymbol {K} ^ {(h)} = \boldsymbol {F} \boldsymbol {W} _ {\mathrm {K}} ^ {(h)}, \tag {11}
$$
$$
\boldsymbol {V} ^ {(h)} = \boldsymbol {F} \boldsymbol {W} _ {\mathrm {V}} ^ {(h)}, \tag {12}
$$
where the three linear transformation matrices  $\mathbf{W}_{\mathrm{Q}}^{(h)}, \mathbf{W}_{\mathrm{K}}^{(h)}, \mathbf{W}_{\mathrm{V}}^{(h)} \in \mathbb{R}^{D_{\mathrm{F}} \times D / H}$  parameterize the query, key, and value transformations for the  $h$ -th attention head, respectively. Following that, we compute the attention scores among all feature pairs and then linearly combine the value matrix from all  $H$  attention heads:
$$
\boldsymbol {W} _ {\mathrm {A}} ^ {(h)} = \operatorname {s o f t m a x} \left(\frac {\boldsymbol {Q} ^ {(h)} \left(\boldsymbol {K} ^ {(h)}\right) ^ {\top}}{\sqrt {D _ {\mathrm {H}}}}\right), \tag {13}
$$
$$
\widehat {\boldsymbol {Z}} = \left[ \boldsymbol {W} _ {\mathrm {A}} ^ {(1)} \boldsymbol {V} ^ {(1)}; \boldsymbol {W} _ {\mathrm {A}} ^ {(2)} \boldsymbol {V} ^ {(2)}; \dots ; \boldsymbol {W} _ {\mathrm {A}} ^ {(H)} \boldsymbol {V} ^ {(H)} \right], \tag {14}
$$
Finally, we perform sum pooling on the resulting embedding matrix  $\widehat{Z} \in \mathbb{R}^{F \times D_{\mathrm{F}}}$  and use a linear model  $f_{\mathrm{LIN}}$  to obtain the final fingerprint or SMILES string embedding  $z \in \mathbb{R}^{D}$ :
$$
\boldsymbol {z} = f _ {\mathrm {L I N}} \left(\sum_ {d = 1} ^ {D _ {\mathrm {F}}} \widehat {\boldsymbol {Z}} _ {d}\right). \tag {15}
$$
A.2 Computing infrastructures
Software infrastructures. All of the experiments are implemented in Python 3.7, with the following supporting libraries: PyTorch 1.10.2 [62], PyG 2.0.3 [63], RDKit 2022.03.1 [64] and HuggingFace's Transformers 4.17.0 [65].
Hardware infrastructures. We conduct all experiments on a computer server with 8 NVIDIA GeForce RTX 3090 GPUs (with 24GB memory each) and 256 AMD EPYC 7742 CPUs.
A.3 Code availability
The source code of our empirical implementation can be accessed at https://github.com/Dataa-reindeer/NSL_MRL.
B Datasets and Tasks
In the following, we will elaborate on the adopted datasets and the statistics are summarized in Table 2.
Datasets. We consider four datasets ranging from molecular-level properties to macroscopic influences on human body for experimental investigation: HIV [39], MUV [40], PCBA [41] and QM9 [42].
- HIV (AIDS Antiviral Screen) was developed by the Drug Therapeutics Program (DTP) [39], which is designed to evaluate the ability of molecular compounds to inhibit HIV replication.
- Maximum Unbiased Validation (MUV) group was selected from PubChem BioAssay via a refined nearest neighbor analysis approach, which is specifically designed for validation of virtual screening techniques [40].
- PubChem BioAssay (PCBA) is a database consisting of biological activities of small molecules generated by high-throughput screening [41].
Table 2: Statistics of datasets used in experiments.
Dataset	Data Type	#Molecules	Avg. #atoms	Avg. #bonds	#Tasks	Avg. degree
Pre-training	PCQM4Mv2	SMILES	3,746,620	14.14	14.56	-	2.06
Classification	MUV	SMILES	93,087	24.23	26.28	17	2.17
HIV	SMILES	41,127	25.51	27.47	1	2.15
PCBA	SMILES	437,929	25.96	28.09	92	2.16
Regression	QM9-εgap	SMILES, 3D	130,831	18.03	18.65	1	2.07
QM9-U0	SMILES, 3D	130,831	18.03	18.65	1	2.07
QM9-ZPVE	SMILES, 3D	130,831	18.03	18.65	1	2.07
- QM9 is a comprehensive dataset that provides geometric, energetic, electronic and thermodynamic properties for a subset of GDB-17 database, comprising 134 thousand stable organic molecules with up to nine heavy atoms [42]. In our experiments, we delete 3,054 uncharacterized molecules which failed the geometry consistency check [66]. We include the  $\epsilon_{\mathrm{gap}}$ , U0, and ZPVE in our experiment, which cover properties related to electronic structure, stability, and thermodynamics. These properties collectively capture important aspects of molecular behavior and can effectively represent various energetic and structural characteristics within the QM9 dataset.
- PCQM4Mv2 is a quantum chemistry dataset originally curated under the PubChemQC project [49]. Based on the PubChemQC, Hu et al. [50] define a meaningful ML task of predicting DFT-calculated HOMO-LUMO energy gap of molecules given their 2D molecular graphs. The HOMO-LUMO gap is one of the most practically-relevant quantum chemical properties of molecules since it is related to reactivity, photoexcitation, and charge transport.
C.1 The Neural Scaling Law on PCQM4Mv2 Dataset
Following the experimental settings of Section 3.1, we conduct additional experiments on the PCQM4Mv2, a large-scale challenge dataset provided by OGB  $^{4}$ . To the best of our knowledge, it is currently the largest labeled molecular dataset available, containing over 3.7 million molecular samples. It is worth noting that due to the unavailability of official test labels, we selected  $50\%$  from the validation set as the test data and report the results in Figure 7.
Figure 7: The neural scaling law of molecular representation learning on PCQM4Mv2 dataset.
C.2 The Effect of Equivariant Representations on the Scaling Law
To further explore the scaling laws of more powerful 3D graph representations, we have extended our investigation to include SE(3) invariant model (SphereNet) and E(3) equivariant model (PaiNN) in the context of Section 3.1. The empirical results are demonstrated in Figure 8. It can be observed that most of the performances generally adhere to the power-law relationship and the difference lies in the coefficients and exponents of the power law. Furthermore, the relative differences in performance
<sup>4</sup>https://ogb.stanford.edu/docs/lsc/pcqm4mv2
Figure 8: The neural scaling law of equivariant representations with 3D graph modality.
between the three models tend to vary with distinct predictive tasks. For instance, in the U0 prediction task, the power law relationships of the three models exhibit a noticeable parallel pattern in the log-log plot, whereas in the other two tasks, a crossing effect becomes evident.