On the Scalability of GNNs for Molecular Graphs
Authors: Maciej Sypetkowski
Frederik Wenkel
Farimah Poursafaei
Nia Dickson
Karush Suri
Philip Fradkin
Dominique Beaini
Venue: NeurIPS 2024 poster (2024)
Keywords: Molecular Biology
Graph Neural Networks
Graph Transformers
Scaling Laws
Pretraining
Finetuning
Primary area: graph_neural_networks
TL;DR: We study scaling laws of Graph Neural Networks on 2D molecular graphs during pretraining as well as finetuning.

Abstract: Scaling deep learning models has been at the heart of recent revolutions in language modelling and image generation. Practitioners have observed a strong relationship between model size, dataset size, and performance. However, structure-based architectures such as Graph Neural Networks (GNNs) are yet to show the benefits of scale mainly due to lower efficiency of sparse operations, large data requirements, and lack of clarity about the effectiveness of various architectures. We address this drawback of GNNs by studying their scaling behavior. Specifically, we analyze message-passing networks, graph Transformers, and hybrid architectures on the largest public collection of 2D molecular graphs for supervised pretraining. For the first time, we observe that GNNs benefit tremendously from the increasing scale of depth, width, number of molecules and associated labels. A major factor is the diversity of the pretraining data that comprises thousands of labels per molecule derived from bio-assays, quantum simulations, transcriptomics and phenomic imaging. We further demonstrate strong finetuning scaling behavior on 38 highly competitive downstream tasks, outclassing previous large models. This gives rise to MolGPS, a new graph foundation model that allows to navigate the chemical space, outperforming the previous state-of-the-arts on 26 out the 38 downstream tasks. We hope that our work paves the way for an era where foundational GNNs drive pharmaceutical drug discovery.

On the Scalability of GNNs for Molecular Graphs
Maciej Sypetkowski
Frederik Wenkel Valence Labs, Montreal Université de Montréal, Mila Quebec fredeerie@valencelaaee com
Valence Labs, Montreal naciej@valencee
Farimah Poursafaei
Nia Dickson
Karush Suri
Valence Labs, Montreal McGill University, Mila Quebec
NVIDIA Corporation
Valence Labs, Montreal
Philip Fradkin
Dominique Beaini
Valence Labs, Montreal
Valence Labs, Montreal Université de Montréal, Mila Quebec
University of Toronto, Vector Institute
Abstract
Scaling deep learning models has been the heart of recent revolutions in language modelling and image generation. Practitioners have observed a strong relationship between model size, dataset size, and performance. However, structure-baad based architectures such as Graph Neural Networks (GNNs) are yet to show the benefits of scale mainly due to lower efficiency of sparse operations, large data requirements, and lack of clarity about the effectiveness of various architectures. We address this drawback of GNNs by studying their scaling behavior. Specifically, we analyze messagge-asssing networks, graph Transformers, and hybrid architectures on the largest public collection of 2D molecular graphs for supervised pretraining. For the first time, we observe that GNNs benefit tremendously from the increasing scale of depth, width, number of molecules and associated labels. A major factor is the diversity of the pretraining data that comprises thousands of labels per molecule derived from bio-assays, quantum simulations, transcriptomics and phenomic imaging. We further demonstrate strong finetuning scaling behavior on 38 highly competitive downstream tasks, outclassing previous large models. This gives rise to MolGPS, a new graph foundation model that allows to navigate the chemical space, outperforming the previous state-of-the-arts on 26 out the downstream tasks. We hope that our work paves the way for an era where foundational GNNs drive pharmaceutical drug discovery.
1 Introduction
Recent successes in language modelling the increasing amount of training data and computational resources. Across different domains, practitioners have observed direct relationship between model parameter count and performance on novel tasks (27]. In natural language processing, large Transformer-based models have demonstrated impressive generalization capabilities utilizing causal autoregressive objective [5]] In the meantime. image generation has undergone incredible leaps with large models trained utilizing pixel level unsupervised objectives.
47
65 and image generation
52
55 are driven by
'Equal contribution; order determined alphabetically.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

While data power law scaling behav- ior has been tremendously beneficial in language and image domains, its practical impact on molecular reason- ing and drug díscovery has remained limited. This is a direct consequence of complex scientific tasks requiring reasoning regarding the underlying structure of the data [IO]. In the past, molecular property prediction approaches have made use of graph- based methods, as these allow us to reason about the structure and inter- action of different components of a molecule. Molecules naturally rep- resented as graphs, where the nodes represent atoms and edges correspond to covalent bonds between the atoms.
More width?
More depth?
More labels?
MPNN
More molecules?
Transformer?
More datasets?
Figure Summary our GNN scaling hypotheses stud- ied in the present work. The baseline model is presented in dark grey. followed by different scaling hypotheses illus- trated in lighter colors. We analyze the scaling behavior of cressage-passingg networks, graph Transformers and hybrid architectures with respect to the increasing scale of width, depth, number of molecules, number of labels, and diversity of datasets.
Graph Neural Networks (GNNs) have emerged as promising way of learn- ing molecular representations 33 [7]. GNN architectures are equipped with the flexibility of learning molecular structure while building general, powerful representations over graphs utilizing backpropagation. These representations have been utilized in various paradigms such reasoning about drug properties [61]. target binding interactions [62], retrosynthesis of reagents 32] ligand-based docking [13 and
in-silico experiments (70]
Despite the growing applicability of GNNs in molecular tasks, the lack of supervised data has significantly hindered our ability to proportionally scale model sizes. It remains unclear whether graph-based architectures hold the promise of scale, similar to the paradigms of language and unsupervised image generation.
Learning molecular properties with GNNs presents its own set of unique challenges. First, multiple different GNN architectures are being actively researched. These include graph-convolutions 20)
message passing architectures (6), graph Transformers [80] and hybrid architectures [53] 339. These
approaches have shown recent progress, but their applicability practical applications remains an open question [56].
Second, commonly used self-supervised training techniques do not transfer well when applied to molecular graphs; e.g., retrieving masked bonds and atoms is not informative. This is primarily due to large data requirements and the fact that graphs are limited in capturing domain-specific aspects such as chemical interactions and biological compositions [36]. Other methods such as GPSE 34 solely learn the graph structure, thus providing a better positional encoding for another GNN.
Lastly, public datasets have insufficient high-quality data for effective GNN architecture training. While recent attempts have been made to expand open-source datasets [7], their extensions towards multi-task settings remain an open question.
We aim to address these limitations and provide a concrete understanding of the required data and architectures to build foundational models for molecular graphs. Specifically, want to answer the question: How do graph-based based architectures scale in supervised multi task training on molecular graphs?
As summarized in Figure [ we aim to answer the above question by studying the scaling behavior of 2D molecular GNNs under different settings of width, depth, number of molecules, number of labels, and the diversity datasets. We analyze message-passingg networks, graph Transformers, and hybrid architectures on the largest public collection of 2D molecular graphs. The models are tested in 2 different settings: (1) randomly split train and test sets for pretraining and (2) finetuning/probing of pretrained models on standard benchmarks.

---

Our work aims to provide a proper understanding of how different GNN architectures scale for molecular GNNs and its affects on performance in various settings. Our main contributions are:
We study the scaling behavior of 2D molecular GNNs under varied settings of depth, width, number of molecules, number of labels, the diversity in dataset, and the architectural choice.
We show that our largest billion parameter models continue to scale with constant gains in molecular property prediction. To the best of our knowledge, this is the first work to demonstrate the continuous scaling behavior of molecular GNNs.
We show that supervised pretraining over molecular graphs provides a rich fingerprint embedding, useful for MLP probing, and more expressive as we scale the model and datasets.
We provide an in-depth analysis of scaling trends across different probing and finetuning strategies. Specifically, we observe that model width and number (and quality) of labels are the most important factors driving finetuning performance.
Finally, we propose MolGPS, a foundation model derived from our findings on how to best scale molecular GNNs. MolGPS constitutes the most dominant model across the presented benchmarks to date, establishing state-of-the-art (SOTA) on 26/38 highly competitive downstream tasks.
Preliminaries
2.1 Graph Neural Networks
Our problem setting consists of graphs of the form - where V denotes the set of nodes and
E denotes the set of edges. Each node i V indicates the atom and each edge (u, v) & denotes chemical bond between two atoms u, v. Total number of atoms in the molecule are = | while
total number of edges are M = E Node features layer I are denoted by x E Rd and are
concatenated into an N x dn representation matrix X [x]: xhi
...x. T Edge features el Rdd
are concatenated into the M x de edge feature matrix E [e (u,v) E
2.2 Scaling Laws
We denote the parameters of a model as with the total number of trainable parameters being |0]_ We consider training dataset 'D consisting of labeled data samples (9, y) D. Here, 9 indicates the input graph and y E TRN denotes the categorical or continuous label. Total size of the dataset is denoted as [D]. Given the study of large foundational models, we note that [0] is large in size and lies on a high dimensional manifold such that 0 E IR where B > [D]. Recent work has shown that increasing the size of dataset [D] or the number of trainable parameters |0) has a direct power law relationship on the loss function Le ((D)) [27]. Mathematically, we have the following,
Le x (10c1/10))
(1)
Equation denotes the plwer-law relationship between the number of trainable parameters and the loss obtained when utilizing the parameters 8. Further, tc denotes the critical parameters and E IR a scalar constant. Intuitively, as the number of parameters approaches a critical value, with every gradient step, the test loss decreases at power-law with constant rate. A similar relationship holds for the size of datasets. Mathematically, we have the following,
Lo x (/Dcl/(D))
Equation 2 describes the power-law relationship between the size of dataset and loss when training the model on D. Here. /Del denotes the critical size of the dataset and a scalar constant.
3 How Do Molecular GNNs Scale?
Our study aims to answer the question: How do molecular GNNs scale? We begin by studying GNNs the multi-task supervised pretraining setup. Since our analysis consists multiple tasks on large scale, we utilize the Graphium library 77. Due to the absence of a unified consensus on the best architecture for molecular GNNs, we focus our efforts on three specific models. We select MPNN+ [39 which improves quantum prediction over the MPNN [8]. Graph Transformers [44] and Hybrid
3

---

GPS++ [39 along with the use of positional encodings. Finally, we evaluate our models on range of public benchmarks with 38 datasets from TDC [25], Polari and MoleculeNet [7T]. We evaluate our models in both finetuning and probing (fingerprinting) settings.
We begin by providing detailed description of the datasets and benchmarks. We then elaborate on the choice of architectures. Finally, we discuss finetuning and probing strategies along with the results of our analysis.
3.1 Datasets
We study the scaling behavior of GNNs primarily on the LargeMix dataset mixture 77. These datasets cover different types of molecules exhibiting variable properties. Thus, the training is done a multi-task setting consisting of thousands of labels. This is a challenging approach towards learning representations with GNNs, especially as some labels are very imbalanced and sparse. While most of our experiments are conducted with LargeMix, we later also explore an additional data derived from phenomic cell imaging that highlights the potential of this emerging data modality.
LaxgeMix. This dataset mixture consists of 5 million molecules grouped into 5 different tasks at different graph levels, with each task having multiple labels. The diversity of this mixture of data makes this dataset suitable for pretraining large GNNs. Below is a description of the individual tasks contained within the LargeMix.
L1000_ VCAP and MON00 are two datasets of 16k and 20k molecules, respectively, with 998 graph-level classification labels corresponding to transcriptomics changes in the cell when exposed to drugs.
PCBA_132 1328 is a dataset of 1.6M molecules with 1,328 binary classification labels. Each label corresponds to the activity tags of the molecules in a bioassay reported PubChem (28)
PCQM4M G25 and PCQM4M N4 are two datasets of 3.8M molecules with 25 graph-level labels and 4 node-level labels. Labels are obtained using density functional theory (DFT) simulations, highly accurate quantum simulation method [57]
Phenomics. To exemplify how incorporating additional data can enhance LargeMix and improve the obtained pretrained models, we experimented with an additional data type that comes from phenomic imaging. The dataset contains ~-6k labels for ~550k molecules that were derived from phenomic imaging [ of cells perturbed with either dose of compound or gene knockout. We conducted a similarity analysis between the obtained images (represented by vector embeddings Kraus et al. side, and images subject to gene perturbation the other side. The pretraining task is to predict for each compound it has a phonemically visible similarity to gene knockout (indicating biological relationship).
BO)) subject to compound perturbation
3.2 Finetuning and Probing Benchmarks
A major benefit of foundational models is that they allow to easily generalize to unseen downstream tasks through approaches like finetuning or (linear) probing. In this work want also study the effect of scaling of the pretrained models on the performance on downstream tasks. For downstream task evaluation we use open-source therapeutic benchmarks, For fair and comprehensive evaluation. all models are first pretrained using common supervised learning strategy and then finetuned (or probed) for molecular property prediction. The benchmarks used for evaluating are listed below.
TDC. Therapeutics Data Commons 25 is one of the common benchmarks for drug discovery. Our study focuses on 22 ADMET (Absorption, Distribution, Metabolism, Excretion and Toxicity) tasks. While TDC serves as the bedrock for open-source drug discovery evaluation, we note that it suffers from data collection and processing biases across dissimilar molecules [68]
Polaris. This is a recent collection of benchmarks addressing concerns over previous datasets. Developed by an industry consortium of various biotech and pharmaceutical organizations, it provides access to high-quality molccular samples across various tasks. Our analysis considers 12 of the top tasks from either ADME (Absorption, Distribution, Metabolism, and Excretion) or DTI (Drug- Target Interaction) group for molecular property prediction.
PollarisHu: https:/pollarshubb.o/

---

MoleculeNet. This is a benchmark dataset for molecular machine learning that is built upon public datasets [7T] It consists of various datasets covering different levels of molecular properties spanning from properties at the molecular level to broader impacts on the human body. There are different categories of properties including quantum mechanies, physical chemistry, biophysics, and physiology. We investigate 4 datasets that are commonly used in similar studies such as 35 74.34.
3.3 Architectures
We broadly study three types of architectures: (1) message- paassagg-gssinggggg networks, (2) graph Transformers, and (3) hybrid models. In the case of message-passing networks, we focus on the MPNN++ model as provides suitable testbed for evaluating molecular graphs while maintaining performance across various tasks. Our graph Transformer and hybrid models make use of GPS++ model, which is known for its scalable nature on quantum property predictions. In addition to GNN models, we make use of Positional and Structural Encodings (PSEs) to improve the expressivity of MPNNs and introduce soft bias into the Transformer. We discuss architectures and their design aspects below.
+ This is variation of the neural message passing architecture with edge and global features [8 g Choosing the MPNN++ allows us to maximize architecture expressivity while
minimizing the risk of overfitting on larger datasets (39]. Each MPNN block makes use of sequential
Dropout [59], MLP and LayerNorm [
modules followed by a skip connection [33 60 across node
and edge features:
E X Dropout (MLP (E'))
X (ayeeroor((rropuu((XX)) + + =
GPS++ This is a hybrid model leveraging the MPNN+ inductive bias while providing the flexibility of self-attention--aase modules (79) to allow for a rich feature extraction scheme across nodes and edges, and was empirically proven very successful (39). Here, the standard self-attention weights are biased by structural prior B from the input graph. Mathematically, the GPS++ module carries out the following computation:
X g++1= M E'); Z' =BiasedAttn(X** 1+1 ,B); X MLP(X 1+1 + zz'
Transformer. This model is identical to GPS++, but without the MPNN++ module and concatenation. Instead, it relies solely on positional and structural encodings (PSEs) for structural bias.
PSEs. These are important design choices when training GNN architectures [53] 34]. as they allow each node to understand its position and surroundings within graph. This is essential for any graph Transformer, but it was also shown to improve the expressivity of molecular GNNs. Specifically, we use three PSE schemes. First, we use random- -wakk diagonals [15] to allow one to decouple structural and positional representations. Learned positional encodings are used to tackle isomorphic nodes. Second, we use Laplacian eigenvectors 66] these form expressive way to encode node geometries and positions. Laplacian eigenvectors provide strong theoretical guarantees with respect to the expressivity of the Weisfeiler-Lehman test, useful insight when evaluating GNNs scale. Last, we use the Laplacian eigenvalues [31] as a suitable PSE scheme to fully leverage the Laplacian spectrum. Additionally, they provide global structural information about the graph.
3.4 Finetuning and Probing
Following pretraining, we finetune our pretrained models on range of unseen downstream tasks. While there exist no clear guidelines for finetuning GNNs, this aspect is extensively explored in this work. Notably, our evaluation considers two strategies (finetuning and probing), which both significantly benefit from increased scale of the pretrained model.
Finetuning, Since our training setup consists of multiple tasks and our architectures incorporate multiple task heads, we need to identify a finetuming module after which the remaining pretraining architecture is removed and replaced by a newly initialized MLP: the finetuning head. As all down- stream tasks discussed above reside on the graph level, our main choice is the graph output network, the MLP that processes features after being aggregated from the node to the graph level, and further feeds into the task heads for graph-level tasks. Intuitively, this layer's output representations have benefited the most from pretraining on diverse data and tasks, as feeds directly into the various task heads. We further investigate the effeet of choosing layers of the tasks heads as finetuning module to
5

---

potentially leverage synergies between specific pretraining and downstream tasks. As all downstream tasks are on the graph level, we trim the architecture by removing parts related node level tasks and unused task heads.
Probing and Fingerprinting. Similar finetuning, we employ probing using MLP strategy for solving new downstream tasks. For probing, the base model is kept frozen and only the new layers trained. This allows the training procedure focus the gradient on newly added parameters, resulting in task-specific probing head layers. In the case of large model sizes, extracting embeddings (so-called fingerprints) from the frozen base model expensive with respect to memory consumption and compute. We tackle this bottleneck by caching fingerprints on disk and reusing them during probing. Since the gradient does not impact parameters of the base model, fingerprints remain unchanged yielding an optimized strategy for downstream tasks capable of parallelization across multiple inexpensive devices. this work, similar to the finetuning setup, we extract fingerprints from the task head MLPs of graph level tasks, and from the graph output network, the MLP that directly feeds into the task heads.
4 Experiments
4.1 Scaling Trends for Pretraining
Widsh soaling
Label
aoiling
scaling
alurtion
secling
In this section, evaluate the scaling be- haviour our models according to various parameters summarized in Figure [ namely the architectural choice, width, depth, num- ber of molecules, labels, and different datasets. We analyze our models on datasets from LargeMix described in Section [.1 For detailed results and experiments of our study, please refer the supplementary ma- terial.
1-18-
MPSNN
Overview. Figure 2 presents the variation of architectural choices between MPNN++ Transformer and GPS++, with training curves in Appendix [D] and full finetuning (and probing) results in Appendix [ No- tably, all models scale favourably with the increasing scale of width (number of param-
0.00
0.700
POOHAUU 0P)
804
eters per neuron), depth (number of GNN layers) and number of molecules (dataset size).
Hosteddos tomal dnastolleed (ee)))
Tolass hrvidee
MPNN+ Transformer. MPNN++ mod- els are more parameter efficient as they per- form better with small width and depth com- pared to Transformers. They are also data efficient as they perform significantly better for the quantum PCQM4M_ tasks when sub-sampling the datasets, although smaller differences are observed for the remain- ing (biological) datasets. Transformers be- ing "data- dlatahhungry" is consistent with recent works in domains such as natural language and computer vision [50]] I7]. The hybrid GPS++ benefits from the MPNN+ expres- sivity in now-parameter regimes, while also exhibiting a similar molecular scaling to the Transformer in low-data regimes. Finally, we notice that MPNN+ models are more affected by depth, which unsurprising considering that, contrarily to Transformers, their receptive field depends the number of layers.
Madd
Mikeda Eacilo
So LUBOL
No PORALIT 110- NO
Caled
Figure Effect of scaling different scaling types (columns) test performance (rows). The standard- ized mean is calculated as mean of standardized scores for every task in dataset group, i.e.. a mean and stan- dard deviation per task were calculated based on all models in this study (signs of tasks with lower is better metrics were flipped).

---

Width scaling. As seen in the first column of Figure 2 increasing the width has a significant impact on model performance across all tasks. Further, we trained larger models for fewer epochs as the loss converged faster and more likely to exhibit overfitting on the PCQM4M tasks.
Depth scaling. Similar to width, depth of GNN models plays important role in the dataset fit in test time. Deeper models with larger layers capture intricate aspects of the data resulting in 12.5% improvement in test error. However, performance plateaus at around 8-16 layers for quantum datasets. For PCBA_ the performance continues to increase.
Molecule scaling. Unsurprisingly, the number of molecules in the training set correlates strongly with the performance of all models. Contrary to width and depth, molecule scaling is consistent across all models and test sets, with GPS++ models and Transformer benefiting more than MPNN++ on quantum tasks. For instance, increasing the dataset size by eight-fold (12.5% to 100%) yields a significant 33.33% improvement in model performance in the case of the hybrid GPS++ model.
Detailed scaling law analysis. We provide detailed analysis of the observed scaling trends in terms of Equations] and 2 in Appendix G observing scaling laws similar to those in other domains [27).
4.2 Scaling Trends Downstream Tasks
We now evaluate scaling of models when finetuning and probing on downstream tasks. As detailed in Section 3.4 all weights are tuned in the case of finetuning. while the pretrained model is frozen when fingerprinting followed by probing.
Due to the large number of downstream tasks spread across 38 tasks, we limit our evaluation to probing for most experiments, except for MPNN++ where we also finetune the model.
MPNN-+
Pazson www.da 401-
Widd Silii
Datesets
To summarize scaling trends, we compute the Spear- man's rank correlation coefficient 58 between model performance on given metric and the scale of model/data used. The correlation is given by a value in the range [-1,1] 1] with a value of indicating perfect scaling (i.e., larger model or dataset yields better downstream task performance), -1 indicating imperfect scaling (i.e., a smaller model or dataset would be preferred) and O indicating no correlation. We note that this evaluation scheme, although statisti- cal, aims to answer an important question: What kind of design decisions are necessary to build a founda- tional model for molecular representations?
AUNIC
AUNIC
Metric
3M
45
.3
56
55
42
31
59
Froblig
.66
58
20
30M4
A9
8:3
2.1
211
100M
65
.75
74
56
64
37
36
.26
111
66
275
-10
-26
24
:24
Sprarss
115
1M
44
Fle-using
70M
25
AD
12
22
SM
55
55
25
300M
.1
.00
35
77
74
47
25
A4
.57
30
86
86
LM
96
MPNN++ vs. Transformer. For probing on down- stream tasks, we study the effect of architecture choices of width, depth, and number of molecules. We find that Transformers benefit more from in- creased width on downstream tasks compared to GPS++ and MPNN+ as seen in Figure 2 (bottom two columns). Despite the number of molecules hav- ing a stronger impact on all model's performance, it only slightly impacts the downstream performance of all models, with a small benefit for MPNN++. Finally. Transformer is the only model with small positive trend for depth scaling, while GPS++ and MPNN+ show close to no trend (Figure 2
Finetuning and probing perfor- mance of pretrained MPNN+ models of different width on the Polaris benchmark. Darker green shades denote better metric values. Larger models tend to perform bet- ter on unseen tasks. Spearman correlation values closer to indicate that predictive per- formance correlates with larger model sizes.
Figure 3:
Width scaling, We evaluate width scaling on Polaris and TDC datasets in Figure 3 (and Figure 9 in the appendix). We observe linear scaling trends for MPNN+ on all Polaris datasets, with an average Spearman correlation of 0.91 for probing and 0.85 for finetuning. On TDC, similar trends are observed (on average 0.69 for probing and 0.72 for finetuning) with a strong correlation of > 0.75 for 15/22 datasets during probing and 17/22 during finetuning. These results strongly indicate the benefits of larger pretrained GNNs for downstream tasks, a result consistent with prior findings scaling studies [27]. Similarly, Transformer and GPS++ show strong positive width scaling trends.
7

---

1.00
0.90
3B MatePs
0.80
50TAA
0.60
B Malt
0:50
0.40
0.30
0.10
0.00
The spontaiitt Aarahoene
fanod Worg
rocioe thu roe debabiyyyy Aet-ee
Tecerssa Al
462 Marten
THE PIcco Superrtt Gertondan tongen
You Hos
IDC/PE Proccredd
TOC ievorriaiiiiyy....
TOE Lostaats
RE-- 186 Chat
The mi-omee coat
raan adiva SQLL
roc EYP22 Pelo
Plaar adres Dover
returts offern Zurt
Petetic light sasterti Mape ddunter Cleets Mestee Side
estur Mattten 6417
you creaa Saimaas Tac
pebsen phii
Figure 4: Comparison of our MolGPS foundation model (that combines fingerprints from the MPNN+ Transformer and hybrid GPS++ model) to the SOTA across TDC, Polaris, and MoleculeNet benchmarks. SOTA refers to the maximum value for each dataset. MolGPS establishes new SOTA on 11/22 TDC tasks and on all but one task among Polaris and MoleculeNet.
Depth scaling. We evaluate the scaling of depth of MPNN++ GPS++ and Transformer models on the Polaris and TDC benchmarks in Figures I0 and IT II (Appendix E.2). For probing on Polaris, we observe weak positive trends, with average scaling Spearman correlations of 0.47, 0.55, and 0.50, respectively. We see weaker average correlations on TDC, being slightly negative for MPNNN+ and GPS++ and best for Transformer with 0.27. However, finetuning MPNN+ achieves a respectable correlation of 0.33. While some datasets strongly benefit from deeper networks, others strongly deteriorate with no clear pattern observable for the TDC datasets. We conjecture that degradation with depth is related to the oversmoothing issue described in Appendix Certain molecular properties can be well predicted only from small local substructures, hence eliminating the need for long-range interactions that deeper networks enable.
Molecule scaling, In this setting. we randomly sub-sample a number of molecules in the training set by 12.5%, 25% and 50% to study their effect on downstream tasks. Surprisingly, probing and finetuning performance does not correlate strongly with the amount of molecules in the training set, as reported in Figures 12 and 13 (Appendix E.3) .. For MPNN++ we observe average Spearman correlations of 0.28 and 0.32 when probing and finetuning on TDC, respectively. Contrarily their stronger trends on the pretraining tasks, Transformer and GPS++ have lower correlations during probing O. and 0.15. In the case of Polaris, only average correlation of Transformers stands out at 0.73, however reaching worse peak performance per task compared to the less correlated MPNN++ and GPS++ The globally weak positive trends come from the variation across the downstream tasks, with many strong correlations and a few strong negative correlations.
Label sealing. We now study the effect of target labels by randomly sub-sampling the number of labels of each dataset in the training set by 12.5%, 25% and 50%. In Figures 14]ad I3 (Appendix [E.4). we observe a large Spearman correlation of 0.57 on Polaris and 0.54 on TDC between the ratio of training labels and the performance, with only few negative correlations. In the finetuning regime, this number lowers to 0.37 on TDC. These stronger correlations put label scaling as the second-best strategy for improving the model's downstream performance.
Dataset ablation. We further conducted study determine the importance of the pretraining data in two ways. Firstly, we repeat pretraining of the models without specific pretraining datasets (dataset ablation). Secondly, we probe models specifically from certain task head MLPs compared to the base GNN models (task-head ablations).
Observing the dataset ablations in Figure 16 and (Appendix E.5 we see that PCBA_1333 is the most important pretraining dataset for downstream task performance while L1000 * actually hurts the performance certain tasks. It will therefore prove beneficial later pretrain without L1000_* *
Task-head ablations. We further tested the effect of probing from different layers of the task heads rather than the graph output network. Results are shown in Figure T8 (Appendix (.6). While overall, the graph output network leads to best performance and correlation, the representation after the first layer of the PCBA 1328 task head performs strikingly well for some tasks, possibly due synergies
8

---

from pretraining on bioassays. This suggests probing approaches using combinations of fingerprints could further improve results. On the other hand, the layers from the PCQM4M G25 dataset perform poorly, which is intuitive as this pretraining task is dissimilar to the downstream task.
Probing vs. finetuning. So far, we have considered finetuning and probing side-by-side, establishing both as effective strategies. However, given the relatively similar performance and the significantly higher computational cost of finetuning, we find probing be more advantageous. Another major benefit of probing is the ability to leverage multi-level information from the pretrained GNN as investigated in our task head ablation study above. We recall that our pretraining is based on a supervised multi Multii-tsk//a learning approach. As result, different task heads capture task-specific information, while earlier layers that feed into the task heads carry more general information. When combining several fingerprints, we can think of taking into account knowledge from several "experts".
4.3 Towards a Final Foundation Model
We now explain how the above findings can be pieced together to develop MolGPS, a powerful graph foundation model. Apart from scaling the model width, we found two other design choices with major impact on the performance for the various downstream tasks, We report results on the MoleculeNet benchmark [7] here in addition the previously used TDC and Polaris benchmarks.
Multi-fingerprinn probing. Our previ- ous task ablation study suggested that different fingerprints may be opti- mal for probing depending the down- stream task. As a result, we choose probing (instead of finetuning) and fur- ther experiment with combinations of multiple fingerprints extracted differ- ent layers of a pretrained model, which improves performance on downstream tasks. Moreover, performance can be further enhanced by combining finger- prints from multiple pretrained models.
TDC ADMET benchmark aggregated over 22 tasks
1.50
1.25
Normalized performance
0.50
0.25
0.00
NPNN++
12524
Mapi ijft + GAN sisteingle model (024))
-025
Hal:
Molt (any se Fouperiiilll
facelin (2221)
Pretraining without L1000. Addi- tionally, based on our observation in the dataset ablation, we pretrained new models without the L1000_* * tasks, which leads to performance improve- ments across all scales. We hypothesize this is due to the challenging signal-to- noise ratio for those particular tasks, as also pointed out in the literature [64].
-050
10a
Number of paramoters
Figure 5: Comparison of our MPNN+ probing (that lever- ages multiple fingerprints; with and without additional phenomies pretraining) and MolGPS (that leverages finger- prints from MPNN++ Transformer and GPS++) to various baselines across TDC benchmark collection.
we present multi-fingerprint probing results of the MPNN+ model. We report the normalized performance across the TDC benchmark collection. We observe a strictly positive scaling trend up to 1B parameters, clearly outperforming TDC Baseline (the normalized score across the best models per task reported in (25)) and the MolE foundational model 443 a gold standard for molecular property prediction (including model variant with only self-supervised pretraining). The IB MPNN+ only slightly trails MaNLight+GNN [46], the best-performing single model on the TDC benchmark. Surprisingly, we were unable to further scale the model to the 3B parameter regime, likely due to limitations of our pretraining data mix.
In Figure 5
Integrating phenomics into pretraining. To improve our pretraining data mix, we experimented with an additional data type for model pretraining. The dataset contains 6k labels for 500k molecules that were derived from phenomic imaging [ of cells perturbed with either a dose of a compound or a gene knockout. The pretraining task is to predict for each compound if it has a phonemically visible similarity to a gene knockout (indicating biological relationship).
'The normalized performance of model on a benchmark collection calculated by first deriving the Z- z-scores of the model's performance per task relative to the leaderboard. We flip the sign of z-scores for tasks that are "the lower the better" and then average the z-score across the tasks.

---

Adding phenomics data to LargeMix (without L1000), improved our downstream task performance across the board. Comparing scaling trends in Figure 5 MPNN++ with phenomics exhibits significant vertical upwards shift compared to the original MPNN+ Notably, we were also able to extend our scaling study to the 3B parameter regime. While we were previously unable to extend the scaling trend, MPNN++ with phenomics maintains a positive scaling trend. These findings highlight the vast potential of our method as more data from different modalities becomes available.
MoIGPS. We introduce a final graph foundation model that leverages the various findings of this paper. MolGPS inherits many architecture design choices from the General, Powerful, Scalable Graph Transformer method [53] and can be used to navigate the molecular space. Our 1B and 3B MolGPS combines fingerprints from MPNN++ Transformer, and GPS++ models (of scale IB and 3B, respectively) that have been pretrained with phenomics data and without L1000, followed by specialized probing MLP. Figure compares this model across the TDC. Polaris and MoleculeNet benchmark collections to the current SOTA for each task and to MolE. MolGPS yields by far the strongest downstream task results, outperforming MolE in 21 21/22 TDC tasks and establishing SOTA performance 11/22 TDC tasks. This makes MolGPS the model with the most SOTA entries in the TDC leaderboard followed by GNN (46) that established SOTA on TDC tasks and 7 other methods that are SOTA for at least one TDC task Similarly, compared to previous best methods on the Polaris and MoleculeNet benchmarks, we observe that our model is significantly better (often by large margins) for all but one downstream task. We primarily attribute the large- scale success to width scaling up to 3B parameters and the integration of phenomics data for pretraining. Figure 55 shows the normalized performance of MolGPS for the TDC benchmark. where it performs comparable to the best model per task for 1B parameters and clearly outperforms that baseline for 3B parameters. This is remarkable recalling that this score is derived from the best scoring method per task of the benchmark collection, while we use single method for all tasks. Further comparison of MolGPS to foundation models that rely on supervipe pretraining can be found in Appendix
5 Conclusion
In this paper, we studied the scalability of GNN models including message-passing networks, graph Transformers and hybrid architectures on the largest public collection of 2D molecules for the tasks of molecular property prediction. We showed major performance gains from the growing amount of parameters, data and compute, both on the original test set and downstream finetuning. Importantly, our models benefit tremendously from the increasing scale of width, number of molecules, and number of labels. Our largest 3B parameter models, including MPNN++, Transformer, and GPS++ continue to scale favourably. More importantly. we demonstrate a consistent performance improvement on downstream property prediction tasks via finetuning and probing as we scale model and data size. Finally, we derive MolGPS, a powerful foundational model based on a multi-fingerprin probing approach that can be used to navigate the chemical space, establishing saate-of-the-ae on 26 out of 38 highly competitive downstream tasks. We hope that our work paves the way for the development of foundational GNNs and new architectures with applications in pharmaceutical drug discovery.
Future Work. While our study demonstrates the benefits of increasing number of parameters far greater than prior work, there are still orders of magnitude before we reach a general-purye purpose foundational model of molecules. Our analysis is restricted to the effeett of number of parameters and molecules during pretraining and finetuning stages. Future work would aim to uncover additional aspects of GNN training such the increasing complexity of aggregation functions and their effect on scaling properties. It will also be important to bridge current limitations for training large GNNs for molecules related to the expensive graph featurization and fast data loading.
Broader Impact. We foresee positive impacts of GNNs in areas of drug discovery, pharmaceutical advancements and tackling rare discases by studying their molecular configurations. On the other hand, such models could also be used for harmful purposes such as developing chemical weapons and biohazards. We note that the usage of GNN models for such applications is less likely.
*e report scores from the TDC leaderboards extracted on March 15, 2024. SOTA on TDC is established by a group of 8 different models, namely Chemprop-RDD RDKit (78), MapLight, MapLight+GNN (46). BaseBoosting (26). CFA [4), SimGCN, ZairaChem (48) and ContextPred [85). For MoleculeNet, we use the same split as GraphMVP (35). thus report the best results from their table.
10

---

References
[1] A. Aghajanyan, L. Yu, A. Conncau, W.-N. Hsu, K. Hambardzumyan, S. Zhang, S. Roller, N. Goyal, O. Levy, and L. Zettlemoyer. Scaling laws for generative mixed-modal language models. arXiv preprint arXiv:2301.03728, 2023.
[2] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Millican, M. Reynolds, al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 16-2336,,, 2022.
[3] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization, arXiv preprint arXiv:1607.06450 2016.
[4] Y. Bahri, E. Dyer, J. Kaplan, J. Lee, and U. Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2102.06701, 2021.
[5] W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzan V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1806.0/806.000 2018.
[6] D. Beaini, S. Passaro, V. Létourneau, W. Hamilton, G. Corso, and Liô. Directional graph networks. In International Conference Machine Learning, pages 748-758. PMLR, 2021.
[7] D. Beaini, S. Huang, J. A. Cunha, G. Shisescu-Parejee O. Dymov, S. Maddrell-Mander C. McLean, F. Wenkel, L. Müller, J. H. Mohamud, et al. Towards foundational models for molecular learning on large-scale multi-task datasets. ICLR 2024, 2024.
[8] M.-A. Bray, S. Singh, H. Han, C. T. Davis, B. Borgeson, C. Hartland, M. Kost-Alimova, S. M. Gustafsdottir, C. C. Gibson, and A. E. Carpenter. Cell painting, a high-content image-based assay for morphological profiling using multiplexed fluorescent dyes. Nature protocols, 11(9): 1757-1774, 2016.
[9] M. M. Bronstein, J. Bruna, T. Cohen, and P. Veličković. Geometric deep learning: Grids, groups, graphs geodesics, and gauges. arXiv preprint arXiv:2104. 100.14488 2021.
[10] S. Bubeck, V. Chandrasekaran,
Eldan, Gehrke, E. Horvitz, Kamar, Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. 12712, 2023.
[11] K. Cao, P. M. Phothilimthana, Hbija, D. Zelle, Y. Zhou, C. Mendis, J. Leskovee, and B. Perozzi. Learning large graph property prediction via graph segment training. arXiv preprint 2rXiv:2305.12322 2023.
[12] D. Chen, Y. Zhu, Zhang, Y. Du, Z. Li, Q. Liu, Wu, and L. Wang. Uncovering neural scaling laws in molecular representation learning. Advances in Neural Information Processing Systems, 36, 2024.
[13] G. Corso, H. Stärk, B. Jing, Barzilay, and T. Jaakkola. Diffdock: Diffusion steps, twists, and turns for molecular docking, 2023.
[14] G. J. Dovonon, M. M. Bronstein, and M. Kusner. Setting the record straight on transformer oversmoothing. arXiv preprint arXiv:2401.00 240.00400, 2024.
[15] V. P. Dwivedi, A. T. Luu, T. Laurent, Y. Bengio, and X. Bresson. Graph neural networks with learnable structural and positional representations. CoRR, abs/21 10.07875, 2021.
[16] C. Fifty, J. Leskovec, and S. Thrun. In-context learning for few-shot molecular property prediction. arXiv preprint arXiv:2310.08863 2023.
[17] M. Galkin, X. Yuan, H. Mostafa, J. Tang, and 7. Zhu. Towards foundation models for knowledge graph reasoning. arXiv preprint
2023.
[18] J. for quantum chemistry. International conference on machine learning, pages 1263-1272 PMLR, 2017.
Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. F. Dahl. Neural message passing

---

[19] J. Godwin, M. Schaarschmidt, A. Gaunt, A.
Y. Rubanova, P. Veličkovié, J. Kirkpatrick, and P. Battaglia. Simple gmn regularisation for 3d molecular property prediction & beyond. arXiv preprint arXiv:2106.07977 2021.
[20] Z. Guo, C. Zhang, W. Yu, J. Herr, O. Wiest, M. Jiang, and N. V. Chawla. Few-shot graph learning for molecular property prediction. In Proceedings of the web conference 2021, pages 2559-2567, 2021.
[21] W. L. Hamilton. Graph representation learning. Morgan & Claypool Publishers, 2020.
[22] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[23] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778. 2016.
[24] D. Hemanndez, J. Kaplan, T. Henighan, and S. McCandlish. Scaling laws for transfer, 2021.
[25] K. Huang, T. Fu, W. Gao, Y. Zhao, Y. Roohani, J. Leskovec, C. W. Coley, C. Xiao, J. Sun, and M. Zitnik. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. Proceedings of Neural Information Processing Systems, NeurIPS Datasets and Benchmarks, 2021.
[26] D. Huang12, S. R. Chowdhuri13, 13, A. Li13, A. Agrawal14, K. Gano15, and A. Zhu. A unified system for molecular property predictions: Oloren chemengine and its applications.
[27] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford,
J. Wu, and D. Amodei. Scaling laws for neural language models, 2020.
2023. [28] S. Kim, J. Chen, T. Cheng, A. Gindulyte, J. He, S. He, Q. Li, B. A. Shoemaker, P. A. Thiessen,
B. Yu, al. Pubchem 2023 update. Nucleic acids research, 51 (((D)):D1333-DDI
[29] N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[30] O. Kraus, K. Kenyon-Dean, S. Saberian, M. Fallah, P. McLean, J. Leung, V. Sharma, A. Khan, J. Balakrishnan, S. Celik, al. Masked autoencoders for microscopy are scalable learners of cellular biology. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11757-11766 11768, 2024.
[31] D. Kreuzer, D. Beaini, W. Hamilton, V. Létourneau, and P. Tossou. Rethinking graph trans- formers with spectral attention. Advances in Neural Information Processing Systems, 34: 21618-21629, 2021.
[32] C.-H. Liu, M. Korablyov, S. Jastrzębski, P. Wlodarczyk-Pruszyyie Y. Bengio, and M. H. Segler. Retrogmn:
Approximating retrosynthesis by graph neural networks for de novo drug design. arXiv preprint S..Y::2011.13044 2020.
[33] Liu, C. Yang, Z. Lu, Chen, Y. Li, M. Zhang, T. Bai, Fang, L. Sun, S. Yu, et al. Towards graph foundation models: A survey and beyond. arXiv preprint arXiv:2310. /829, 2023.
[34] R. Liu, Cantürk, O. Lapointe-Gagné, V. Létourneau, G. Wolf, D. Beaini, and L. Rampášek. Graph positional and structural encoder, 2023.
[35] S. Liu, H. Wang, W. Liu, J. Lasenby, H. Guo, and J. Tang. Pre-training molecular graph representation with 3d geometry. arXiv preprint arXiv:2 10.07728, 2021.
[36] Y. Liu, M. Jin, S. Pan, C. Zhou, Y. Zheng, F. Xia, and S. Y. Philip. Graph self-supervised learning: A survey. IEEE Transactions on Knowledge and Data Engineering, 35(6):587-55900, 2022.
12

---

[37] R. Luo, L. Sun, Y. Xia. T. Qin, S. Zhang, H. Poon, and T.-Y. Liu. Biogpt: generative pre- -reitrained transformer for biomedical text generation and mining, Briefings in Bioinformatics, 23(6). Sept. 2022. ISSN 1477-4054. doi: 10. (0.1093/bib/bbac409)
[38] A. Madani, B. Krause, E. R. Greene, S. Subramanian, B. P. Mohr, J. M. Holton, J. L. Olmos Jr, C. Xiong, Z. Z. Sun, R. Socher, et al. Large language models generate functional protein sequences across diverse families. Nature Biotechnology, pages 1-8, 2023.
[39] D. Masters, J. Dean, K. Klaser, Z. Li, S. Maddrell-Mander, A. Sanders, H. Helal, D. Beker, I.. Rampášek, and D). Beaini. Gps++: An optimised hybrid mpnn/transformer for molecular property prediction. arXiv preprint arXiv:2212.02222, 2022.
[40] M. Moret, L. Friedrich, F. Grisoni, D. Merk, and G. Schneider. Generative molecular design in low data regimes. Nature Machine Intelligence, 2(3):171-180, 2020.
[41] M. Moret, I. Pachon Angona, L. Cotos, S. Yan, K. Atz, C. Brunner, M. Baumgartner, F. Grisoni, and G. Schneider. Leveraging molecular structure and bioactivity with chemical language models for de novo drug design. Nature Communications, 14(1):114, 2023.
[42] C. Morris, M. Ritzert, M. Fey, W. Hamilton, J. Lenssen, G. Rattan, and M. Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 4602-4609, 2019.
[43] O. Méndez-Lucte C. Nicolaou, and B. Earnshaw. Mole: molecular foundation model for drug discovery, 2022.
[44] L. Müller, M. Galkin, C. Morris, and L. Rampášek. Attending to graph transformers. arXiv preprint arXiv:2302.04413 2023.
[45] E. Nijkamp. J. A. Ruffolo, E. N. Weinstein, N. Naik, and A. Madani. Progen2: exploring the boundaries of protein language models. Cell Systems, 14(11):968 07788,9774478 2023.
[46] H. Notwell and M. W. Wood. Admet property prediction through combinations of molecular fingerprints. arXiv preprint arXiv:2310.00172 2023.
[47] OpenAI. Gpt-4 technical report, 2023.
[48] S. O. Oselusi, P. Dube, A. I. Odugbemi, K. A. Akinyede, T. L. Ilori, E. Egieyeh, N. R. Sibuyi, M. Meyer, A. M. Madiehe, G. J. Wyckoff, al. The role and potential of computer-aided drug discovery strategies in the discovery of novel antimicrobials. Computers in biology and medicine, page 107927, 2024.
[49] M. Quazi, C. Schweikert, D. F. Hsu, T. Oprea, et al. Enhancing admet property models performance through combinatorial fusion analysis. 2023.
[50] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training, 2018.
[51] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019.
[52] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss. A. Radford, M. Chen, and Sutskever. Zero-shot text-to-image generation. ICML 2021, 2021.
[53] I.. Rampášek, M. Galkin, V. P. Dwivedi, A. T. Luu, G. Wolf, and D. Beaini. Recipe for general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems, 35: 14501-14515, 2022.
[54] R. M. Rao, J. Meier, T. Sercu, S. Ovchinnikov, and A. Rives. Transformer protein language models unsupervised structure learners. bioRxiv, 2020. doi: 10000122200.1.15.222220.
[55] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. CVPR 2022, 2022.
13

---

[56] Y. Rong, Y. Bian, T. Xu, W. Xie, Y. Wei, W. Huang, and J. Huang. Self-supervised graph transformer on large-scale molecular data, Advances in Neural Information Processing Systems, 1511259--1271,, 2020.
[57] E. Saal, S. Kirklin, M. Aykol. B. Meredig, and Wolverton. Materials design and discovery with high-throughput density functional theory: the open quantum materials database (oqmd). Jom, 65:1501-1500 2013.
[58] P. Schober, C. Boer, and L. A. Schwarte. Correlation coefficients: appropriate use and interpre- tation. Anesthesia & analgesia, 126(5):1763-1766) 2018.
[59] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15 (1):1929-1958, 2014.
[60] R. Srivastava, K. Greff, and Schmidhuber. Highway networks. arXiv preprint arXiv: 1505.00387 2015.
[61] H. Stärk, D. Beaini, G. Corso, Tossou, C. Dallago, Günnemann, and P. Liò. 3d infomax improves gnns for molecular property prediction. ICML. 2022, 2022.
[62] H. Stärk, O.-E. Ganea, L. Pattanaik, Barzilay, and T. Jaakkola. Equibind: Geometric deep learning for drug binding structure prediction. ICML 2022, 2022.
[63] S. Thakoor, C. Tallec, M. G. Azar, M. Azabou, E. L. Dyer. Munos, Veličković, and M. Valko. Large-scale representation learning on graphs via bootstrapping. arXiv preprint arXiv:2102.0653 2021.
[64] X. Tong. N. Qu, X. Kong, S. Ni, K. Wang, L. Zhang. Y. Wen, S. Zhang, X. Li, and M. Zheng. Transigen: Deep representation learning of themical-induced transcriptional profile. bioRxiv. pages 2023-11, 2023.
[65] H. Touvron, Martin, K. Stone, Albert, A. Almahairi. Y. Babaei, N. Bashlykov, Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, Fu, W. Fu, B. Fuller, C. Gao, Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Mao, X. Martinet, Mihaylov, Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[66] H. Veith, N. Southall, R. Huang, T. James, D. Fayne, N. Artemenko, M. Shen, J. Inglese, C. P. Austin, D. G. Lloyd, et al. Comprehensive characterization cytochrome p450 isozyme selectivity across chemical libraries. Nature biotechnology, 27(11):05001155 2009.
[67] Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio. Graph attention networks. ICLR 2018, 2017.
[68] Walters. We need better benchmarks for machine learning drug discovery, 2023. 2024-01- 18.
[69] H. Wang, Fu, Y. Du, W. Gao, K. Huang, Z. Liu, P. Chandak, S. Liu, Van Katwyk, A. Deac, et al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47-60, 2023.
[70] Y. Wang, Z. Li, and A. Barati Farimani. Graph Neural Networks for Molecules, page 21-66. Springer International Publishing, 2023. ISBN 9783031371967. doi: 1007/ 978-3-031-37199-72
[71] Z. Wu, B. Ramsundar, E. N. Feinberg, J. Gomes, C. Geniesse, A. S. Pappu, K. Leswing, and Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9(): 513-530, 2018.
14

---

[72] K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? ICLR, 2019.
[73] K. Xu, M. Zhang, S. S. Du, K.-i. Kawarabayashi, and Jegelka. How neural networks extrapolate: From feedforward to graph neural networks. ICLR, 2021.
[74] M. Xu, H. Wang, B. Ni, H. Guo, and J. Tang. Self-supervised graph-level representation learning with local and global structure. In International Conference on Machine Learning, pages 11548 11 11558. PMLR, 2021.
[75] M. Xu, M. Liu, W. Jin, S. Ji, Leskovee, and S. Ermon. Graph and geometry generative modeling for drug discovery. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5833-5834, 2023.
[76] G. Yang. E. Hu, I. Babuschkin, S. Sidor, X. Liu, Farhi, N. Ryder, J. Pachocki, W. Chen, and J. Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer, 2022.
[77] G. Yang, D. Yu, C. Zhu, and S. Hayou. Tensor programs vi: Feature learning in infinite-depth neural networks, 2023.
[78] K. Yang, K. Swanson, W. Jin, C. Coley, Eiden, H. Gao, A. Guzman-Perez, T. Hopper, B. Kelley, M. Mathea, et al. Analyzing learned molecular representations for property prediction. Journal of chemical information and modeling, 3908)33370-3388 2019.
[79] C. Ying, T. Cai, Luo, Zheng, G. Ke, D. He, Y. Shen, and T.-Y. Liu. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 34:28877-28888, 2021.
[80] C. Ying. Т. Cai, S. Luo, Zheng, G. Ke, D. He, Y. Shen, and T.-Y. Liu. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 34:28877-28888, 2021.
[81] Z. Ying. J. You, C. Morris, X. Ren, W. Hamilton, and J. Leskovec. Hierarchical graph representation learning with differentiable pooling. Advances in neural information processing systems, 31, 2018.
[82] Y. Yuan. On the power of foundation models. In International Conference on Machine Learning, pages 40519-40530. PMLR, 2023.
[83] S. Yun, M. Jeong, R. Kim, Kang, and H. Kim. Graph transformer networks. Advances in neural information processing systems, 32, 2019.
[84] X. Zhang, L. Wang, J. Helwig, Y. Luo, C. Fu, Y. Xie, M. Liu, Y. Lin, Z. Xu, K. Yan, et al. Artificial intelligence for science in quantum, atomistic, and continuum systems. arXiv preprins 20X1v:2307.08423, 2023.
[85] J. Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun. Graph neural networks: A review of methods and applications. Al open, 1:57-81, 2020.
15

---

A Related Work
Foundation Models for Molecules. Recent work has relied on foundation models as a generalist class of models for sequential modelling [82 33
well knowledge retrieval [17. Within molecular
40 Luo al. [37 and Moret drug discovery, recent works rely on structured models of ligands
et al. [4]] study a general model for protein synthesis. Rao et al. (54] construct self-attention driven architecture for contact prediction. Madani et al. [38] learn to generate a family of functional proteins. 45] present class of protein-pretrainee language models. Similarly, Méndez-Lucio interaction level. While many models focus on the design of molecules, recent class of methods has also focused on properties of molecules 77]. Our study follows up by exploring similar molecular tasks for property prediction.
Nijkamp et al.
et al. 43 study binding interactions between different assays at the molecule-moleen
We elaborate more on additional advancements in foundational models making use of molecular graphs. Recent works have argued that the use of high-capacity models will be a significant boon to scientific discovery tasks [69] Of specific interest are tasks in the quantum and molecular discovery paradigms [84] which demand domain -peaif- expertise such as knowledge of structure, provision of additional inductive biases and large data requirements. Towards this hypothesis, Fifty et al. [16] present an in-context learning framework for molecular property prediction without explicitly using a meta learning procedure. This leads to a general algorithm capable of discovering high-level structures from a pretraining sample set. Guo et al. [20] propose similar framework making use of few-shot learning techniques resulting in a sample-efficiee lcarning procedure. Xu et al. (75) present an alternative approach by modelling the full graphical structure of molecules across different property prediction tasks. Although effective, modelling the entire graph results in a computationally intensive learning procedure. Finally, Cao et al. [II] scale up learning to larger graph sizes by segmenting graph neighborhoods on the fly. An ad-hoc partitioning procedure is employed and interleaved with the learning phase in order to accelerate learning on larger and dense graphical clusters.
Architecture Design. Recent methods in graph architecture design focus attending to structural information across nodes [44]. Of specific interest are graph Transformer networks which extract node as well as edge information by composing sequential attention modules over graph readouts 83 In paraliel, graph attention networks model attention weights across edges of a graph [67].
While attention mechanisms have demonstrated modern progress, traditional architectures such as neural message passing [18. On one hand, message passing provides a rich and expressive framework for constructing representations. Godwin et al. [19] study regularization based on noisy nodes for the task of molecular property prediction. Provision of noise imputation node-level features leads to simple and expressive method for tackling sparse molecular graphs. Graph bootstrapping [63 allows prior architectures to scale up to larger and complex graphs for representation learning. Our exploration of different architectures is aligned with the aforesaid works in literature, and with recent trends towards Transformers in related machine learning fields.
Scaling Laws. Recent work in model scaling has demonstrated that performant models follow a power law relationship between their parameter sizes and performance on new data samples (27]. Additionally, this relationship holds during the finetuning stage [24]. thus indicating a strong reliance on model parameters. Bahri et al. explain this power law fit by observing learning as moving on a smooth data manifold. Aghajanyan et al. [ study the power law fit for mixed modality generative models, indicating that the scaling behavior modality agnostic across various datasets. The result hints at the generality of scaling across different domains and applications, which can be extended to the study of scaling laws towards different training regimes (such as finetuning, downstream transfer and inference) and different problem domains (vision, language, audio, generative modelling, etc.) Our exploration of scaling behaviors in graph networks is motivated by the aforesaid directions.
Expressivity of GNNs. Prior work highlights that GNN architectures are limited in their expressivity to distinguish between graphs of similar node arrangements but different geometrical structures [2]. Various works indicate this as consequence of aggregation functions and other design factors involved in GNN training 21). On the other hand, recent work argues that only specific architectures are found robust to over-smoothing when building latent representations [1] For instance, graph Transformers exhibit over-smoothing robustness as they utilize strong inductive biases such as
attention. Xu et al. [33 connect the limited expressivity of GNNs with their ability to extrapolate on
16

---

simpler tasks. Contrary to multi-layer networks. GNNs struggle to extrapolate on simpler tasks but show promise for improvement. Morris al. 42 aim tackle seo-hhng by building higher- order GNN architectures capable of capturing intricate node characteristics in their deeper layers. Finally, Ying et al. [8]] present the differentiable pooling module capable of pooling neighboring node features which aid in reducing noise across layer representations.
B Experimental Details
B.1 Pretraining
All models use 2-layer MLPs to encode node and edge features, respectively, followed by the core model of 16 layers of the MPNN++, Transformer or GPS++ (except for when scaling depth). To be able to tackle graph-level tasks, the outputs are aggregated to the graph level, c.g., by summing them up across the atoms of a molecule. Then, node and graph level representations go through separate 2-layer ML.ss. Finally, representations are processed by separate task heads (2-layer MLPs) specific to each pretraining task. Further, all layers use layer norm and dropout with p = 0.1. The encoder and model core additional have residual connections similar to the design in He et al. [22
Our hyperparameter search for all base models was conducted on all oberserved data samples with constant model size of 10M + 0.1M parameters. For scaling on width, zero-shot scaling from IP (76) was used. For other scaling results, pP was used to scale the model with 10M parameters used as the base model. In the case of depth scaling, we adjusted the learning rate as suggested by depth-uP (77). We did not consider adjusting the residual connections.
Our base M Transformer and hybrid GPS++ models are trained using Adam with a base learning rate of 0.003, 0.001, and 0.001, respectively. We use 5 warm-up epochs followed by linear learning rate decay. All pretraining has been conducted with batch size of 1024. Scaled version of the used models require advanced training strategies due to the large model size. We used multi multi-gpu training (with up 8 NVIDIA A A100-SXM4-40GB GPUs) and gradient accumulation, while adjusting batch size to keep the effective batch size constant. Most models were trained on sigle gpus but our 300M and IB parameter models used 4 and 8 gpus, respectively.
B.2 Finetuning and Probing
finetuning module is selected from one of the layers of the pretraining architecture and a newly initialized MLP is appended to that layer. Here, we use 2-layer MLPs with a hidden dimension of 256. For each experiment, when retraining this model, we set the dropout rate to zero and train for 40 epochs using batch size of 256 and constant learning rate of 0.0001. To first adjust the finetuning head - the newly initialized MLP after the finetuning module freeze the remaining architecture for the first 10 epochs. To find unified finetuning strategy for each pretrained model/downstream task combination, we select the best epoch where validation performance was maximized across all seeded runs of the experiment.
Finetuning. As outlined in Section
Probing. Similar to finetuning, we apply 2-layer MLP to the fingerprints derived from the pretrained model. We choose hidden dimension of 128 and train for 30 epochs with batch size of 128 and a constant learning rate of 0.0001. Further, we use the same approach as for finetuning to select unified number of epochs for each pretrained model/downstream task combination based on validation.
We finally note that all downstream task experiments were conducted on single cpus.
B.3 Performance Metrics
We provide detailed explanations for the metrics used for evaluating the performance of different tasks throughout this work.
The Pearson correlation coefficient measures the strength and direction of the linear relationship between two datasets. It ranges from -1 to +1, where 0 indicates no correlation, -1 indicates a perfect negative linear relationship, and +1 indicates perfect positive linear relationship.
Pearson.
17

---

The Spearman correlation coefficient is nonparametric measure of the monotonic relationship between two datasets. Similar to other correlation coefficients. Spearman's 's correlation varies between -1 and +1, with 0 indicating no monotonic relationship. Correlations of -1 or ++ imply a perfect monotonie relationship between the two datasets.
Spearman.
AUROC. The ROC curve demonstrates the trade-off between the grue-positive rate and the false- positive rate at different thresholds. The area under the ROC curve (AUROC) expresses how good a model is regardless of the chosen threshold.
AUPRC. This metric is useful particularly when dealing with imbalanced datasets and it summa- rizes the area under the precision- pecalll curve at various thresholds.
MAE. Absolute error the magnitude the difference between a prediction and the true value of observation. Mean Absolute Error (MAE) calculates the average of these absolute errors for group of predictions and observations, providing measure of the overall error magnitude for the group.
Average Standardized Score. The standardized score of model M for specific task T is derived follows. Let sy (m) denote the score of model m for task T and let Sy = {sr(m) where My denotes the set of all methods that have been applied task T. The aggregated score is defined as (MM) = sgn(sT) (sT(M) (M) mean mean(Sr)) /std (Sr), where sgn(sy) is the polarity of the metric, i... positive higher is better" and negative if "lower better".
Mrh.
C Trade-Off Between Over-smoothing and Depth
We note that GNN architectures exhibit over-smoothing phenomenon, which implies that latent representations of network become similar and coarser as the network grows in size. Prior evidence suggests that over-smoothing occurs linearly with the increasing depth of GNN networks 21 72). observed similar behaviors for MPNN architectures during pretraining where the performance for node-level tasks degrades significantly with very deep networks. However, it is difficult to determine without any doubt that over- smoothing is the culprit.
On another hand, over-smoothing is believed to be aleviated by graph Transformers. Recent works argue that Transformers present favorable properties which make them robust towards over-smoothing, such as the provision of embeddings and the inductive bias of attention [14). However, we still observe a degradation of performance with depth of our Transformer models, contradiction with this hypothesis. Its theoretical understanding and empirical analysis remains an open question for future work.
18

---

D Training Curves of Pretraining Models
(ANPR)
(AWPR)
(2P)
0.30
0.05
0.44
0.00
0.70
013
0.00
033
0.50
0.00
0.44
0.50
0.33
0.01
0.33
10.00
0.00
0.38
0.42
0.30
0.97
0.17
0.41
0.30
0.16
1..00
0.12
0.30
0.70
0.00
0.92
-34
132
0.58
0.11
16
10
20'
Gradizat atup
Gandian slge
Ccedkan Mtge
Gzadkaz stepe
Grailaxa vieae
Figure 6: Model performance on the test set throughout training for MPNN++ Transformer, and GPS++ architectures with width scaling. Different colors represent models with varying number of parameters.
1,1000" (AVPR)
(AVPR)
POQMMM (87)
POGMEM_N (R))
10.0
0.90
0.41
0.56
0.58
0.50
0.140
0.288
0.31
0.70
0.42
0:31
0.40
01140
0.00
0.008
0.33
0.00
14.00
170
0.42
0.100
0.168
0.00
0.40
0.54
0.00
138
0.58
Grafber dtrps
Grarlinnt
Grotect
Gradliee
Craffiee
Figure 7: Model performance on the test set throughout training for MPNN++ Transformer, and GPS++ architectures with depth scaling. Different colors represent models with varying number of network layers.
19

---

Loss
L1000 (AVPR)
PCDA 1325 (AVPR)
POQMIS G2R (AP)
PCQSSAM N4 (AP)
0.11
0.42
0.91
0.07
0.40
0.33
0.50
0.16
0.33
6.30
0.58
0.32
0.01
0.38
0.18
6030
0.50
0.17
0.43
1.33
0.42
0.30
0..00
0.06
0.16
0.30
0.48
0.553
0.55
0.12
6.70
17.35
0.31
0.00
20.2
30005
10P
Figure 8: Model performance on the test set throughout training for MPNN++ Transformer, and GPS++ architectures with molecule scaling. Different colors represent models with varying fraction of molecules used for training.
E Additional Results on Downstream Tasks
20

---

E.1 Width Scaling
Trandarus
AUPC: C172R SOMEI Chrtoaa
CYPA CYPIA SAMIII www.oootaattttt
Growe Oanie Jtg.eeeegae... AM
Epppkta Anazaans
Chemsn Meputegtt AZ
Chenac Nevmee AZ
TDC
Onl Wan
FTER AZ
BIL
Pat Besemask
MENG
DEI
1he TanTaanhun
C1733
CYAM
C1733 Y-4h
Datlacts
TDC
Cand
Saaaii Jupatnn
1PP0 AZ
800 Mentie
Pas Beeends
Benndlidn Me
AM8S
1Ds. Jenheeds
Bad UBs Olach
CYPIK
C3P23
CYP23 1elt
VAn
C3PXS
W310 Suileg
TDC
Upypkliitt
Caed Weas
min Martee
HIA Jite
Pby Breecuch
Bonddallly As
LERG
DLUI
lok L.R Olach
CYPL SANINI
OPP YeI
CYPICE
AXEC
MES
ANOX
ADOX
ACNOC
AXOXC
ADRC
ADWC
MA-
MAAT
AUNDC
AUROC
ADRGE
AUFIC
AOPRC
MAX
AME
MAA
AUROC
AUBOCC
Sporeet
APPNC
AUTAC
AURNC
Manc
R
1M
8
3M
ICM
30M
300M
I0EM
768M
Sperraa
21
20M
300M
16
08
42
29
13
39 35
31
Comparison of probing and finetuning for MPNN+ (left), Transformer (center), and hybrid GPS++ (right) across different model sizes on the TDC benchmark. Darker green shades denote higher/desirable metric values. Average Spearman correlations for MPNN++ Transformer and GPS++ models show improving scaling behavior with increasing number of parameters across the TDC benchmark. The average Spearman correlation between width and performance for probing is 0.69, 0.82 and 0.73, respectively, and 0.72 when finetuning MPNN+ effectively showing that model size plays an important role in predictive performance.
Figure 9: Width Scaling:

---

E.2 Depth Scaling
MPNN+
Transformer
GPS++
adme- adme-fange fe-fangge SOLUU1
l'euraon adme-fanm fang- RPPB-1
Pairain adme-fang fan--an----- Saae-fang--PP---- HPPB-1
Peurson adme- fang- PERM-
adme- fang- RC Lint-1
Paurain adme- fang- HCLint-]
adme-fang-- fesf-nn---e Mime-fang-SOLU SOL U-1
Penroon udme- fang- lang-RPPI RPPB.1
Perrson adme- HPPB-1
adme- Mame-fan--22 fang-ng----- weaff--ggPPPNOO PERM-1
Penronn adme- fang-fng---- RCI int-
Swa.fang--CCLAA HCI HCLint- Lit--1
Peusson adme- fan--ang--- Tmme-fang-SOLUU SOLU-
Panron adme- fang- R-PB-P
Petroon adme- fow--ang--HBP PR.1
Pensonn adme- fang- PERM-1
adme- fang- RCI int-1
Penron adme- Comn-HCLLMn intt-
Depth scaling
pkis2- pkis2-kit Sa--kit--n
-
1Mi22-kitwttrr11
pkis2-ret-wt-c-
pkis2-ret-wt-r--
pkin2-e2 Miiz--gfr-wt----
pkis2 pkis2-ogfi-wtail
Depth scaling
adme-fang: fanf-nng--ee
pAis2-kit-wt-e-1
pkis2-kit-ww-rd
pkis2-ret-wte4
pkis2-ret- ret--ttr-11
pkis2-eght -rt-e-1--
pkis2-egfr-wte
Depth scaling
pkis2-kit -itst-kff-wt-e-11
pkis2- kit-w-r-t
nkisl-ret-wt-ee
pkis2-ret-wt-rr-
pkix2 wgittwe-f--t---111
pkis2-eathe fllz2-egff-wt---11
Polaris
Polaris
Poluris
Datasets
Datasets
Datusets
Pearson
Peuraon
AUPRC
Pearson
APPRC
Patrson
UPPRC
Pearson
Peurson
Pesrsin
LPPR
Penrow
UPPC
UPRC
Pausson
AUPRE
PewNon
AUPRC
Pounson
Peuron
Metrie
Metrie
Metric
59
30
82
.57
31
61
.37
.2
72
66
.19
.61
73
66
68
40
Probing
.5
38
69
26
63
66
32
40
28
74
.19
Probing
.64
.21
76
.7
AM
52
52
.25
407
.61
.57
300
75
68
67
54
33
39
31
23
.25
.66
66
72
.79
69
.5
24
40
.27
.36
13
Probing
66
.3
.22
62
.58
33
80
40
40
.65
75
75
67
.7
53
29
28
63
66
71
64
67
.1
54
32
72
22
Speurmon
.90
-.4
.20
.90
90
770
.30
330
.60
Speamman
.90
-700
.5
80
40
.70
50
1.00
.00
32
67
63
70
74
67
66
39
37
62
43
31
27
Spearman
Figure 10: Depth Scaling: Comparison of probing for different model depths on the Polaris benchmark with MPNN++ (left), Transformer (center), and hybrid GPS+ (right). Darker green shades denote higher/desirable metric values. Average Spearman correlation between depth and performance is 0.47, 0.55, and 0.50, respectively. Probing shows positive scaling trend with increasing depth across the Polaris benchmark.

---

CYPTAL SdMiie Cabahtigg
CYPAN Caleeltage
CITN SUNEBLL CoR. catullago
Chemen, Jegahees MM
Marona AZ
Supt ScllEe
Dipk
Cer Ming
LIM the
YPAR,
BIA
Ressee barelaally.... No
KING
BILL
CYPIAL CHD
Can Cano
LOO
DIR
BIA
Dyg Brecuess
3t
AMES
MBBG
YDu Lankurk
Ohuk
CYPSR
CYAC
CY735
CYPA4 VOa
CYASE
Coac Woree
LIVE JM
TPAR XX
Thu Leckekk
CXTHL VoI
RAXT
BOOK
GOOO
BODX
BOOK
GOOK
MAC
MAE
ADOOC
ADOOC
ALYIC
A/C
VON
Natri
23
Tritbs
Sparnia
Spcrre
12
Spuree
Figure 11: Depth Scaling: Comparison of probing and finetuning for MPNN+ (left), Transformer (center), and GPS++ (right) models across different model depths on the TDC benchmark. Darker green shades denote higher/desirable metric values. Average Spearman correlation between depth and probing performance -0.11. 1. 0.27 and -0.03, respectively, and 0.33 for finetuned MPNN++. While performance mostly increases with depths up larger depths often show signs of saturation.

---

E.3
Molecule Scaling
MPNN+
Tramsformer
GPS++
Mokecule Scaling
alme-fan-SSOLL
Peanoo adme- fang Maee--ann-RPPiin RPPB-
Peuooo attme- Sane--nng---e SPnng-PPPB-11
Peamon adme fung- PERM-
ad sdt---fag-RCCiin -CCng--C Lin--
adme- admee-fang--i adtt-fann---inann HCLnnt-
AUPRC pkin2- pkis--kitwtee kit-wt----
pkin2 phin2-kit-wtt kit--t----
AUPRC pkin2-rot- 1o2--rot-tt-e-11
Penninn pkän2-ret-nn ret-wt----
Peunou pkiz2 -ef---fff------
Molseale Scaling
Pewson adme-- Saagg-ggg RPPPPPPPPP 8-1
adme- fang 1II PB-1
Panrson adme-- fo-g--g--- PPR---PMMM-
Pennsnn adme- fang LinLin--
Pannoon adme- fang BC Lin-in-s
AUPRC pla2--it-wt-e-1
Pannsn Nase2liitowtee
AUPRC pkik2-nut-nt-e-1
Pauson pläs2 1o2---tt-tt---
AUPRC www.k-gfi--o..ccpp 1ak2-gfr-wt-e-1 -ot--wtt---
pkin2-gff weff-wt-r-l
Mokeust Scoliing
Pwotrow adme Lang, SOLU-1
Peurzon adme- adiee-fang- Fang REPP--PP--
Paaron adme-f fang HIPPB-1
adme-f Fang CIN-RRCLL RCCiin-))
adme- fange HCLint-1
APPCC pkis--Ait-wt-e
pkin pkis--ltt-vee 1ha2-llttwt-nd
pkin2-ret-wt-o
pkiz agfeettet
mane-ffng----- fangg
Peurows admo- fangang--
AUPRC pkic2-ret--nt
pkic2 waf--erfrrr
Polaris
Pearson adme-
Polaris
Polaris
Datasets
Datusets
Pegrron
Peunoe
Pennow
Pennor
Peunnee
APPRC
Metrie
Metric
Metrie
12.50%
12.50%
.88
63
32
Probing
2.5%
.57
.7
.72
84
66
47
JJ8
28
Prehing
2.5
76
Probing
25%
63
.67
.55
66
34
47
37
50%
.4
.64
72
.7
.67
AP
.37
35
25
50%
40
20
70
81
11
100%
82
76
51
84
91
20
100%
.66
.50
82
.79
70
69
54
26
.44
35
.35
23
100%
.65
.75
66
64
.61
37
36
30
Spearman
1.00
Spearmnn
-80
-.60
.80
.80
-1.00
1.00
1.00
-80
..20
Molecule Scaling: Scaling behavior of probed 100M parameter models across different dataset molecule fractions the Polaris benchmark with MPNN++ (left), Transformer (center), and hybrid GPS++ (right). Darker green shades denote higher/desirable metric values. Average Spearman correlation between molecule fraction and probing performance is 0.30, 0.73, and 0.40, respectively, Models show consistent improvement in performance with the increasing size of datasets.
Figure 12:
24

---

Trandamar
G75
OYPA Satera Carhar
CITIN Storac CaPhaMtaaps
C17101 Satverr www.gaaeeeeeeeeeeeeeeeooo Cochheeeegggnn
O1T20 Saburan Coneatloge
CYPEN Sewewrr Www.essseeeer.eeeettte
Lopph Joteitaee
Gurano Degunee
Onner Www.wwe.
Mimiin
Ohe Miess
LD0 Jh1
B0S Marthe
Aarataaly No
1De, Lontonk
CY Vala
C1PX9 Tata
LIOE Jte
PHBA
BORL Mentee
AMDS
MCNG
VTu Joabertee
BM NELUROOO Olrh
C1206
Malcalo
TDC
One WERE
BRS Martte
RIN Hee
Bonndalts
ATRG
DIU
OTFE 2ett
TBC
DEL
Dotaets
Dusteots
CDOOC
AUBOC
ABBCC
ADNC
ADDOC
APMCC
APPIC
AUPNE
SAAT
LOOK
LASOC
LOOK
LOOCC
LOOK
LOME
M415
Metrk
12.54
25
24%
10%
70%
Spcarsm
Figure 13: Molecule Scaling: Comparison of probing and finetuning for MPNN++ (left), Transformer (center) and GPS++ (right) models across different dataset sizes the TDC benchmark. Darker green shades denote higher/desirable metrie values. Average Spearman correlation between molecule fraction and probing performance 0.28, 0.13 and 0.15, respectively, and 0.32 for finetuned MPNN+ Finctuned models scale better when compared to probed models. However, increasing the size of finetuning datasets leads minor improvements beyond the 50% dataset size fraction.

---

E,4 Label Scaling
MPNN+
adme- fang-SOLU--
Peatson adme fang- RPPB-1
adme e- fang- faee-afngg-PPaaa HPPB-1
adm e- fang-------- tee--annn----s- PERM-1
Pearson adme e- fang--a----- RCLint-1
Pearson adme- wer--fang---iiimn
pkis2-egfr-wt-r-1
AUPRC pkis2-kit-wt-el
Pearson pkis2-kit-wt-r-i
AUPRC pkis2-ret-wt-c-
Peatson pkis2-ret-wt-r-I
AUPRC pkis2-egfr-wt-cr
Label Scaling
Polaris
Datasets
Pearson
Pearson
Pearson
Peanson
Metric
12.50%
50
65
55
.33
.37
.42
30
Probing
25%
.61
.41
.49
.33
.22
50%
.45
.39
.29
100%
.65
.57
.75
744
.66
64
.61
37
.53
36
.30
.26
Spearman
1.00
1.00
-.60
-.40
.40
Figure 14: Label Scaling: Performance of MPNN++ probed models across different label fractions in the Polaris benchmark. Darker green shades denote higher/desirable metric values. Average Spearman correlation between label fraction and probing performance 0.57.
MPXN-+
AUPAC CYPZO Subarate, Mangee
Lipophilleity Astraanus Zanas
Microomic,
Salabilty, Aqualne
Labd
Croo,, Wing
BIB Mariis
Broccondi
Hnuvailablllyy
VDoc, Lambarde
Hel lfe Ohauh
AUPAC CYP206, Vellh
CXP3AL Veltn
CYP2C9 Velth
PPAR AZ
MBG
TDC
HIA
DILI
Datanet
AUROC
AUROC
AUROC
AUROC
AUROC
Stomen
ALPAC
AUPAC
MAEE
12.9%
S
Prohing
35%
A2
20
S4
RT
39
A2
50%
100%%
Spuarman
74
1000%
35
66
22
2.5
%5
J6
30
39
50
68
Spearman
.80
L.50 -90
-20
29
-20
-40
46
1.90
-1.0
20
-48
1.00
1.00
Figure 15: Label Scaling: Comparison of MPNN+ probing and finetuning across different label fractions on the TDC benchmark. Darker green shades denote higher/desirable metric values. Average Spearman correlation between label fraction and performance is 0.54 for probed MPNN+ and 0.37 for finetuned MPNN++ Finetuned models scale better when compared to probed models. Increasing label fractions do not deteriorate model performance.
26

---

E,5
Dataset Ablation
MPNN+
adme-fang fang- SOLUU1
ndme-fang- RP-B-1
Pearson udme-fang fang- HPPPB11
Peurson adme adme-fang fang- PERM-I
Pearson adme fang--n--gg- RCLint-I
Pearson adme-fang fane--fng--e HCLint-
pkis2-kit-wt-e-ll
Pearson pkin2-kit-wt-r-1
pkinz-rettwf-e-l
Pearson pkin2-eetwwt-r-1
pkiss-egfr-nn-eel
Pearson pkis2- eefr-wt-r-d
Scaling
Polaris Datasets
Pearson
Pearson
AUPRC
AUPRC
AUPRC
Metric
No L1000
.67
.25
.75
.76
.68
.68
.50
.51
Probing
No PCBA_II2 1328
56
.42
No POQM4M
.75
66
Bascline
.37
Figure 16: Dataset Ablation: Comparison of probed 100M parameter MPNN+ models on the Polaris benchmark tasks (in columns) after pretrained without certain pretraining datasets (in rows). Darker green shades denote higher/desirable metric values. We observe that removing PCBA_1328 significantly hurts downstream performance across almost all tasks, while removing the L1000 leads to noticeable improvements on most tasks.
MPNN+
CYPID6, Sabstate, Curbon Curbon Mungels
JUPAC CVP2C9 Sahsrate, Mangas
Lipophlicity Artiizneea
Clorrace Micosome. AZ
Clearane Hopatneyt,
Sotabilley AgSolDa
tynaiiiaahii Ms
Psp Broosstell
Vhee Lombardo
CYP3M Voth
CYP2C9 Voll
RBB Martins
LD5O Zhe
Croa
PPBR
AUROC AMES
JUROC DILI
Spcurna
AUROC
JUROC
AUROC
JUPRC
AUPRC
APPRC
MAA
MAR
No LOOXU
12
22
R
:11
AB
Probing
No PCDA, L328
9:36
x7
89
66
NaPDQM4M N4
.55
.00
10:33
.5
.30
.1
80
87
39
.3
A1
Buseline
0.00
a5
Figure 17: Dataset Ablation: Performance of probed 100M parameter MPNN+ models TDC benchmark tasks (in columns) after pretrained without certain datasets (in rows). Darker green shades denote higher/desirable metrie values. We observe that removing PCBA 1328 significantly hurts downstream performance across almost all tasks, while removing the L1000 leads to noticeable improvements most tasks.
27

---

E.6
Task-Head Ablation
CYPOO Sahumie, Cwt.ee
AUAAC CYPIS Jahama Cathe
Sabatiny Mysion
Homalattty Xb
Vbu Jasturb
Vearaa Halt Lis Obad
themaos Biqukon.
Naras alne annebrg
Parros alie fang NPPD:
afree Oag LIPPSS
afmocda ICCL Lana
Caod Woar
LINE LIW8AA Au
Bnn Nante
E3PA4 Vann
C Vilh
CYPNN
PeDLL
AMES
NERG
AUBOL
AUBOC
LUME
AUBOC
AUROC
AUNIC
AUNC
AUNIC
AUYIC
SPAT
MAB
Maric
FORA
LIBRU MCFT Hand-
72
883
1.90
913
82
83
52
-02
23
73
LIB90_ VCAP Bind-1
72
1:91
73
POOMIM GI5 Head-4
3.66
72
10L8
A4
LUBBU, VCAP Blad-2
12
A3
12.0
.80
22
J9
88
88
.37
33
82
.7
57
200
.22
28
.33
POONAN G23 Heal22
264
Figure 18: Task-Head Ablation: Performance of probed 100M parameter MPNN+ models on TDC benchmark (left) and Polaris benchmark (right), with the tasks in columns, when probing from different task heads (in rows). Darker green shades denote higher/desirable metric values, and bold/underline indicates the best value in given column. We observe that the graph_output output (e.g. the hidden representation that fed to all task-heads) is overall the best choice for probing. We hypothesize this is because it captures and compresses the combined information from the various pretraining tasks. The PCBA 1328 task-head is also good choice due to its proximity to the considered downstream tasks. The PCQM4M G25 task-head is the least useful the tasks fundamentally different from the downstream tasks.
E.7
The TDC Benchmark Data Leakage
Considering that the pretraining dataset is supervised, it is important to consider data-lcakage source of experimental error. This is especially the case for the PCBA_132 dataset.
PCBA 1328 contains only classification assays with more than 6000 molecules each, which au- tomatically disqualifies most of TDC and all of Polaris. The TDC datasets remaining after this constraint are the 3 CYP* Veith datasets, and the AMES dataset. The AMES dataset is not present in PubChem 288, excluding from the of potential leaks. Regarding the CYP* CYP*_Veith Veith datasets,
they represent inibition assays against recombinant enzymes [66]. The three assays from TDC, and two others from the paper, are present and aggregated under assayID-1851, Therefore, whenever molecule is active against any of the enzyme, the value is 1, otherwise it is 0. Therefore, there is a minor leak, although the datasets are not identical. Further, no evidence of leak was observed in terms of abnormally high performance of the model on these assays, which is expected considering that the model is learning more than 3000 labels simultaneously.
28

---

Table 1: Comparison of MolGPS variants the self-supervised GraphMVP model on the Molecu- leNet dataset (test AUROC). Note that we only consider the datasets that were not part of our pretraining.
Method
BACE
BBBP
Clintox
Sider
GraphMVP
0.812
0.724
0.775
0.639
IB MolGPS w/o Phenomics
0,806
0.802
0.797
0.649
IB MolGP
0.828
0.8
0.809
0.67
3B MolGPS
0.832
0.809
0.807
0.666
F
Comparison of MolGPS to Unsupervised Methods
Compared to unsupervised pretraining approaches, is important to remark that the scale of pre- training data is not directly comparable that of supervised pretraining approach, i.,, billions of molecules in some cases. We note that previous works in the GNN literature [12] found interesting scaling trends despite scaling to less than the 5M graphs that have aggregated in the LargeMix dataset.
during In our supervised context, the data scale does not only depend on the number of molecules pretraining. Instead, each molecule should be considered in conjunction with the set of pretraining labels. PCBA_ 1328 considers more than Ik different labels per molecule (albeit with high sparsity)
and PCQM4M comes with 25 graph-level tasks and node-level tasks (for each node of the 4M molecules). We point to label scaling study that shows the impact of reduced data diversity by removing labels (e.g.. Figure 2 for convincing evidence for this trend. The performance gains from incorporating Phenomics data into the pretraining data mix also support this claim, where the addition of 500k molecules with 6k highly informative labels per graph (Figure 5) leads to strong performance gains.
Here, we provide further comparison to unsupervised pretraining approaches (Table and and 2). We observe that the unsupervised MolE model variants [4 clearly underperform the standard MolE model that leverages both unsupervised and supervised pretraining (Table 2 which is turn outperformed by the listed MolGPS variants by large margins. Even smaller 100M MPNN+ models (that are of comparable parametric size to MolE) both outperform the self-supervised variants (see Figure 5).
model on MoleculeNet. MolGPS without Phenomics pretraining outperforms GraphMVP on 3/4 tasks, while the model variants that were pretrained with additional Phenomics data outperform across all tasks.
In Table T we compare to the self-supervised GraphMVP
Table 2: Comparison of MolGPS variants to the self-supervised variants of MolE and the standard MolE (supervised + self-supervised pretraining) on TDC benchmark collection. Supervised pre- training significantly improves MolE performance as seen in the normalized performance (left-most column). MolGPS models outperform all MolE models by large margin, exhibiting the best perfor- mance in all but two tasks.
EVPREN Sabetor, wwt.wwwwaaaaaaaaaoooo
Racenaallly 38
VIO Lnneuant
MOND EXPIRK Yuak
LOCE 301
Le
AMES
AE500
(osesperreed
0.72
01477
0.6
3.77
0183
1005
11..0
1963
1371
1190
1.66
11396
6175
MuxcisS Phmundo
1355
0.4
130
0:45
6314
624
0003
6984
234
E381
0.6
8.65
8.28
8.50
0312
0.00
8747
0985
6371
Nakutt
1486
8.56
E32
9522
1.2
646
9U81
0.6
1.58
9131
068
BA5T
0364
8.52
0549
0631
883
6.15
0.64
0.5
63
612
29

---

G
Scaling Law Details
We explore the power law fit which governs the scaling behavior of our GNNs using Equations Tand to compute the power law constants and identify data and parameter requirements. For metrics with higher desirable values (such as AUROC or R2), fractions inside the exponents are reversed.
Table 3 presents values of a (Equation I based on model parameters and tasks considered in our downstream task experiments the Polaris benchmark (Figure 3 We choose 00. 1B and fix final training error values corresponding to this model's performance. On average, we see a R3 0.081 for probing and 0.098 for finetuning. These relationships hold across orders of magnitude in [0] indicating that finetuning behavior scales logarithmically with the number of trainable parameters. is also worth noting that Kaplan et [27] obtain similar 0.076 indicating that our power law fit lies within the same parameter budget.
Table compares values of (Equation 2 for pretraining models presented in Figure We choose [D. = 5M to be 100% of molecular data. As observed, 3 z 0.110 for MPNN++ / = 0.106 for Transformer and [ O. 106 for GPS++ for the overall test error. This holds for datasets up to 5M molecules. Notably, all 3 > 0 indicate that, given computational budget, larger GNNs scale favorably. We again note that Kaplan et al. 27 have similar power law fits of 0.095 albeit with a 10a3 token corpus. This allows us to draw two conclusions. Firstly, GNNs continue to demonstrate optimal scaling with limited datasets. Secondly, our 5M molecules (10M node features) dataset is sufficient to demonstrate scaling behavior equivalent to 10 token language corpus, up to the IB parameters regime.
Table 3: Power law constants (a in Equation 1 for different downsttean tasks from Polaris benchmark when varying the number parameters (Figure) 3)
Method T1
T2
T3
T4
T5
T6
17
T8
T9
T10
T11
T12
Prohing 0.055
0.053
0.041
0.034
0.049
0.053
0.083
0.147
0.064
0.078
0.112
0.206
Finetuning
0.055
0.300
0.057
0.028
0.039
0.047
0.053
0.212
0.037
0.115
0.062
0.179
Table 4: Power law constants (3 in Equation for pretraining different architectures when varying
dataset sizes (Figure 2
Model
Global Loss
L1000 (AVPR)
PCBA (AVPR)
PCQM4M G25 (R2)
PCQM4M (R2)
MPNN+ 0.110
0.047
0.061
0.011
0.002
Transformer
0.106
0.047
0.067
0.012
0.001
GPS++
0.
0.047
0.067
0.012
0.001
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main contributions introduced in the Abstract and Section Introduction, are investigated and discussed throughout the paper. In short, we investigate how GNNs benefit from the increasing scale of depth, width, number of molecules, number of labels, and diversity in the pretraining datasets. These scaling trends are extensively investigated and the outcomes reflected in Section 3 Additionally, presented MolGPS, graph foundation model, whose details and performance are fully presented in Section 4.3
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss some of the limitations of this work as potential directions for the future research in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
31

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: This paper primarily presents empirical results. The observed scaling trends of our work in the theoretical context of prior work in Section G with references providing the theoretical context.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: For pretraining, we use datasets and code from the literature 77. The code can be found at htttp:://gitth wow/daaamol-ioo http///dtttaooooooo/eooiiiioo graphium while the data can be We further provide the experimen- tal details throughout the main text and in Section B covering both pretraining and finetun- ing/probing.
com
found ptttps://zenodee org/ httg://ecord/1110101 10797799
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
32

---

(d)
The authors should provide scripts to reproduce all experimental results for the new reproducible, they We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use the LargeMix dataset 77. which can be found at hhttps://zeno:n org/records/10791 10797799 Links the datasets for finetuning and probing can be found referring to the corresponding references in the paper. The experiments can be reproduced by using the code associated 7
com/datamol- h///grphhium to- gether with the instructions in the main text and Section of the appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips..cc/publcc/uuut
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
the train/test proposed in 7 and similar follow the splitting conventions the benchmark collections for finetuning/probing Hyperparameters are discussed in the main text and in Section B of the Appendix.
Justification:
For pretraining,
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
33

---

Answer: [Yes]
Justification: We conducted several iterations of each experiments and reported the average results over all runs. In most cases, we have also demonstrated the standard deviations (for example, Figure (Bottom)) or as the correlations across the performance of different models. However, there readability of the figures and tables. Moreover, we mainly study the trends where correlation metrics do integrate naturally with error bars.
cases that we decided omit error bars for improving the
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots. The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: The omputational resources that were used are reported in Section B in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://pww..aaaee..ccc///aaaaaaa
Answer: [Yes]
Justification:
We have reviewed the NeurIPS Code of Ethics and made sure to adhere to
them.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
34

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss both positive and negative impacts of this work in Section 5
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not propose new data. We have discussed the impact of our work in Section [5] We believe that our proposed model and approach for harmful purposes is considerably less likely and it is not direct outcome of our work.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
35

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have publicly available data [7] that are available here are shared under Creative Commons Attribution Non Commercial Share Alike
These data
4.0 Internationss license. In addition, built upon the Graphium library 7 whose license is based on Apache 2.0 (The Graphium License can be found here).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
We do propose any new asset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This question is not applicable to this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
36

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This question is not applicable to this work.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "4Yy3RGUYaP", "reviewer": "Reviewer_YrHv", "summary": "The paper investigates the scalability of GNNs in molecular graph applications. It highlights the relationship between model size, dataset size, and performance upon message-passing networks, graph Transformers, and hybrid architectures using a large dataset of 2D molecular graphs. It shows that supervised pretraining on molecular graphs provides rich embeddings beneficial for downstream tasks. And it finds that the number of labels is crucial for fine-tuning performance.", "questions": "See weaknesses.", "limitations": "The authors claim that this work uncovers additional aspects of GNN training such as the increasing complexity of aggregation functions and their effect on scaling properties.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The experiment of MPNN and Transformer on pretrained and downstream tasks is very thorough and provides ample evidence to support the claim. The ablation studies are detailed.", "weaknesses": "Many scaling results are unsurprising, such as the consistent improvement with increased depth and width during pretraining. However, the thorough experiments compensate for this predictability. I am curious if these trends will also occur for 3D graph datasets like QM9 or others. Conducting a few additional experiments on these datasets could strengthen this paper, though this isn't necessarily required in this rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing detailed feedback on the paper which is of utmost value to our work. Below we address your concerns point-by-point. We also kindly invite the reviewer to refer to the general rebuttal where we share further information and a summary of the feedback we received.\n\n>**Many scaling results are unsurprising, such as the consistent improvement with increased depth and width during pretraining. However, the thorough experiments compensate for this predictability.**\n\nThank you for appreciating the quality of our empirical analysis and finding our findings interesting. \n\nWe agree that, especially given the unprecedented benefits of scale in other domains such as natural language, some results of this work may seem unsurprising at first. In the context of GNNs however, our work explores largely uncharted territory in terms of parametric scale [1,2,3] and the possible implications on biochemistry and drug discovery in particular. We recall  the finetuning and probing performance on downstream tasks, establishing new state-of-the-art on 12/22 tasks on highly-competitive TDC benchmark and all but two tasks on Polaris and MoleculeNet benchmarks with our proposed foundation model (MolGPS). In our rebuttal we have expanded our empirical study, adding model variants of MolGPS that were scaled to the 3B parameter regime andpretrained on an improved data mix that further enhances our performance across the downstream task benchmarks. We kindly refer to the general rebuttal for more details.\n\n>**I am curious if these trends will also occur for 3D graph datasets like QM9 or others. Conducting a few additional experiments on these datasets could strengthen this paper, though this isn't necessarily required in this rebuttal.**\n\nThank you for the suggestion. We note that our proposed MolGPS model solely relies on learning from 2D graphs, while tasks in QM9 depend on molecules with 3D positions. This is one of the reasons why we limit our analysis of downstream tasks to 2D molecular tasks.\n\nIt is important to highlight the importance of publicly available labeled data for our supervised pretraining setup. Our label scaling experiments (e.g., Figure 2a of submission) highlight the importance of many diverse labels for each molecule. Current databases for 3D molecules are still fairly small and would hardly allow for the required data diversity present in our 2D pretraining data collection. However, we note that scaling design decisions uncovered in our empirical analysis may prove useful to construct and train foundational 3D GNNs in the future, when databases reach adequate scales. These models would further require the use of sophisticated training strategies and aggregation functions, a future direction which we explicitly mention in our limitations section.\n\nKindly let us know if our response above addressed your concerns. We will be happy to further discuss and answer any questions you may have.\n\n[1]. Sun et al, Does GNN Pretraining Help Molecular Representation?, NeurIPS 2022.\n\n[2]. Sun et al, MoCL: Data-driven Molecular Fingerprint via Knowledge-aware Contrastive Learning from Molecular Graph, ACM SIGKDD conference on knowledge discovery & data mining 2021.\n\n[3]. Liu et al., Neural Scaling Laws on Graphs, arxiv 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rUd6xqFHBn", "reviewer": "Reviewer_kaW1", "summary": "The paper examines the scaling behavior of GNNs in the context of molecular graph representation and prediction. The authors analyze various GNN architectures, including message-passing networks, graph Transformers, and hybrid models, on a large dataset of 2D molecular graphs. They find that increasing the scale of model depth, width, dataset size, and diversity significantly enhances performance on downstream tasks. They introduces MolGPS, a new graph foundation model, which demonstrates superior performance across numerous tasks in molecular property prediction.", "questions": "1. How well do these insights generalize to other types of graph-structured data beyond molecular graphs?", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. Comprehensive experiments were performed on large scale datasets to demonstrate the scaling law.\n3. The empirical results of MolGPS on various dataset is promising.\n4. The insights on how to scale GNN models is meaningful to the community.", "weaknesses": "1. Another important aspect of scaling the parameters is regularization [1]. The authors did not explicitly discuss / validate the regularization methods they are using.\n2. The authors did not control the compute used to train the models. It would nice to see some efficiency comparison as well.\n\n[1] How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers https://arxiv.org/abs/2106.10270", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback on the paper which is of utmost value to our work. Below we address your concerns point-by-point. We also kindly invite the reviewer to refer to the general rebuttal where we share further information and a summary of the feedback we received.\n\n>**Another important aspect of scaling the parameters is regularization [1]. The authors did not explicitly discuss / validate the regularization methods they are using.**\n\nOur model indeed uses regularization techniques, e.g., dropout in each GNN layer. We will include a detailed discussion of the used methods and their parameterization in the appendix of the revised paper. The need of more sophisticated regularization techniques is largely avoided through the use of rich pretraining data that prevents the models from overfitting (except for the MPNN++ model in a few instances).\n\n>**The authors did not control the compute used to train the models. It would nice to see some efficiency comparison as well.**\n\nThank you very much for the suggestion. Model performance tables with the training compute time will be added to the appendix. Training time of a single model varies from 24 GPU hours for the smallest models to ~1120 GPU hours for the largest models. In total ~13400 GPU hours were used for computation in this paper.\n\nThe work of [1] pointed out by the reviewer presents a compelling analysis of the used computational resources compared between finetuning to a downstream task and training a model of the same size from scratch. \n\nWe note that such a study may be difficult to replicate in our case due to the high parametric scale of the GNNs in contrast to the low-data regimes for the downstream tasks with only a few 100's or 1,000's of molecules per task and almost exclusively a single target label, which would cause overfitting when trained from scratch.\n\n>**How well do these insights generalize to other types of graph-structured data beyond molecular graphs?**\n\nWe thank the reviewer for their question on potential applications outside the biochemical domain. It is highly likely that models similar to MolGPS could be pretrained for other applications and domains given adequate pretaining data. However, the present work would not be suited for downstream tasks outside the biochemical domain. To obtain a domain-agnostic graph-foundation model, unsupervised approaches that primarily learn from the graph structure may be a more natural choice.\n\nThat said, the proposed foundation model MolGPS is specifically targeted towards biochemistry and drug discovery in particular, with the pretraining data chosen accordingly to learn domain-specific information. For comparison, we provide some comparison to unsupervised pretraining approaches in the rebuttal doc (Tables 1 & 2) that indicate our supervised pretraining strategy is to be preferred for a “molecular” graph foundation model.\n\nKindly let us know if our response above addressed your concerns. We will be happy to further discuss and answer any questions you may have.\n\n[1] Steiner et, How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers, arxiv 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W47X2qtsVt", "reviewer": "Reviewer_FQ72", "summary": "This paper investigates the scaling of GNNs on molecular tasks. In their setting, they pre-train a GNN on multiple molecule dataset and then fine-tune these GNNs on different datasets (“down-stream tasks”). They investigate how the performance of different GNNs changes with parameter count, depth, training samples and so on.", "questions": "- **(Q1)** How can you train a single model on multiple datasets if these datasets might have different features? Are all datasets pre-processed to have the same features?\n\n- **(Q2)** Figure 2 - probing on Polaris / TDC: it seems that most models do not profit from an increase in the number of molecules. Does this not contradict the idea of foundation models?  It seems to me like this indicates that when pre-training GNNs the size of the original dataset does not matter much to the downstream task.\n\n- **(Q3)** In a similar vein, you notice that you can get better results when pretraining without the L1000 dataset. Does this imply that for graphs it is more important to have the right kind of data compared to having a lot of data (as for example LLMs or diffusion models)?\n\n- **(Q4)** From (W2): how does fine-tuning compare to probing? Is one clearly better than the other?\n\n\n**To sum up,** this is a good experimental paper that should be accepted.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- **(S1 - Clarity)** The paper is extremely well written and easy to follow.\n\n- **(S2 - Significance)** Both foundation models for graphs and ML on molecules are very active research fields. As such this paper can be important to a large number of people.\n\n- **(S3 - Novelty and Significance)** To me, there are four interesting and novel contributions / observations in this paper:\n   1) Scaling laws for GNNs. \n   2) The investigation of how to train a pre-trained GNN on a downstream task. Here they investigate two options: fine-tuning the entire model and training only the final MLP (probing).\n   3) A comparison of three different types of GNNs: MPNNs, Transformers and MPNNs+Transformers. It is particularly interesting that GNNs scale better with parameters than transformers.\n   4) The observation that removing a dataset from the training data can improve performance on downstream tasks.\n\n- **(S4 - Quality)** The evaluation seems solid, the authors compare three different models and observe mostly similar behavior for all models across datasets.", "weaknesses": "- **(W1 - Clarity)** The presentation of the results in the main paper is done poorly. Figure 2 is too small and difficult to read / understand. I understand that this paper presents a lot of results but putting all of them into a single figure is in my opinion a bad choice.\n\n- **(W2)** Fine tuning vs probing: you have experiments to compare these two approaches. However, I could not find an analysis of the results in the paper (maybe I have missed it).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing detailed feedback on the paper which is of utmost value to our work. Below we address your concerns point-by-point. We also kindly invite the reviewer to refer to the general rebuttal where we share further information and a summary of the feedback we received.\n\n>**(W1) Figure 2 is too small and difficult to read**\n\nWe thank the reviewer for the comment and agree that Figure 2 can be better represented when split up into multiple larger figures. We will apply those changes in a revised version of our paper.\n\n>**(W2 & Q4)  Fine tuning vs probing: How does fine-tuning compare to probing?**\n\nWe thank the reviewer for this helpful comment. We will further clarify this point when revising the paper.\n\nSection 4.2 of the submission studies finetuning and probing side-by-side, establishing both as effective strategies for tackling downstream tasks, without conducting a direct comparison between them.\n\nHowever, when deriving the foundation model MolGPS in Section 4.3, we find probing to be the overall stronger approach. The major advantage of probing is the ability to leverage multi-level information from the pretrained GNN. We recall that our pretraining is based on a supervised multi-task learning approach. As a result, different task heads capture task-specific information, while earlier layers that feed into the task heads carry more general information. When we combine fingerprints from various layers, we can think of aggregating knowledge from several “experts”. Our multi-fingerprint probing suggests this knowledge is additive as it clearly outperforms probing of any single fingerprint. Our foundation model MolGPS doubles down on this idea, combining fingerprints from multiple layers and various pretrained models. For finetuning, there is no straightforward way of taking advantage of this multi-level information, making probing the preferred approach.\n\n>**(Q1) How can you train a single model on multiple datasets if these datasets might have different features? Are all datasets pre-processed to have the same features?**\n\nAll datasets are indeed pre-processed to have the same features. Each molecule is initially represented by a SMILES string [1], that our pipeline features into a molecular graph (i.e., atoms and bonds as nodes and edges, respectively). The (node) features are also agnostic to the data source, e.g., using atom type, row/col in periodic table, etc.. We will add more context on the pre-processing to the appendix.\n\n>**(Q2) Figure 2 - probing on Polaris / TDC: it seems that most models do not profit from an increase in the number of molecules. Does this not contradict the idea of foundation models?**\n\nWe indeed found no consistent trend on downstream task performance for increased number of molecules in the pretraining dataset (Figure 2(a) and Appendix E.3 of our submission). That said, no molecular scale (except for the lowest 12.5%) had an outstanding negative effect on downstream tasks. We therefore hypothesize that our approach is robust to a smaller number of molecules during pretraining when the highly important label diversity per molecule is maintained that can be observed in our label scaling study (e.g., in Figure 2(a) and Appendix E.4 of our submission).\n\nWe would like to point out the importance of characterizing the diversity of the data not only by the number of molecules, but also paired with the number of pretraining labels/tasks per molecule. PCBA_1328 considers more than 1k different labels per molecule (albeit with high sparsity) and PCQM4M comes with 25 graph-level tasks and 4 node-level tasks (that are learned for each node of the ~4M molecules). This perspective is reinforced by our label scaling experiments that show the impact of reducing the data diversity by removing labels (e.g., Figure 2(a) of submission). Performance gains from recently incorporating Phenomics data into the pretraining data mix further support this claim, adding ~500k of molecules with a highly informative set of ~6k labels per graph (Figure 2 of rebuttal).\n\n>**(Q3) In a similar vein, you notice that you can get better results when pretraining without the L1000 dataset. Does this imply that for graphs it is more important to have the right kind of data compared to having a lot of data (as for example LLMs or diffusion models)?**\n\nThis is an accurate observation, which relates back to the importance of the number of molecules and the number (and quality of the labels). L1000 is exceptionally small (only 20k molecules) but features almost 1k labels for each molecule. However, in contrast to, e.g., PCBA_1328 (1328 binary classification tasks), it is a regression task that suffers from low signal-to-noise ratio. Despite trying to stabilize learning by transforming it into a classification task via binning of regression targets, the overall impact on downstream task performance was negative. Notably, the small amount of molecules may lead to the absence of positive samples in most batches, as the batches are dominated by the larger PCBA_1328 and PCQM4M molecules.\n\nAs pointed out by the reviewer, it is the quality (and label diversity) of the data that is highly relevant. The perfect example for this is the massively improved downstream task performance we report in the rebuttal doc after recently adding Phenomics data (with ~500k molecules and ~6k highly informative labels) to our previously use pretraining data mix (excluding L1000). We kindly refer the reviewer to the general rebuttal doc for more details.\n\nKindly let us know if our response above addressed your concerns. We will be happy to further discuss and answer any questions you may have.\n\n[1] Weininger D, SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules. Journal of chemical information and computer sciences. 1988", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WhdkUal7gb", "reviewer": "Reviewer_Lxfs", "summary": "This paper investigates the scaling of GNNS across various settings including width, depth, number of molecules, number of labels, and diversity in pretraining across 12 datasets. Different conclusions were drawn from experiments conducted during both pretraining and fine-tuning stages. Finally, the authors introduce a foundational model named MolGPS which integrates various findings.", "questions": "See weakness", "limitations": "The paper attempts to uncover the significant scaling law challenges in molecular representation learning. However, it is limited by its focus on a single supervised strategy and constrained data scaling. Despite conducting extensive experiments, these efforts fall short of fully addressing the issue and providing actionable conclusions to the community.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The experiments are conducted extensively, taking into account various variables, resulting in some interesting findings. \n2. The problem addressed in this paper—the scaling law problem in the molecular domain—remains unresolved but is crucial for advancing molecular representation learning.", "weaknesses": "1. The pre-training strategy employed in this paper is a supervised method, overlooking a range of existing self-supervised approaches in molecular representation learning [1][2][3]. It is unconventional to explore scaling laws without considering established pre-training strategies like masking and denoising.\n2. The pre-training data in this paper consists of only 5 million molecules. While this scale may be constrained by the label requirements in supervised pre-training, it is insufficient for exploring scaling laws in the molecular domain. Many works use significantly larger datasets, often involving hundreds of millions or even billions of molecules for pre-training their models.\n3. The scaling of molecular data appears to perform poorly under the downstream fine-tuning and probing testing protocols. Are there any insights or analyses regarding this phenomenon?\n4. From the foundational model perspective, it is also important to evaluate quantum molecular properties such as QM9 and MD17 beyond tasks related to biological ADME properties.\n\n[1]:Uni-Mol: a universal 3d molecular representation learning framework\n\n[2]:Pre-training Molecular Graph Representation with 3D Geometry\n\n[3]:Fractional Denoising for 3D Molecular Pre-training", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing detailed feedback on the paper which is of utmost value to our work. Below we address your concerns point-by-point. We also kindly invite the reviewer to refer to the general rebuttal where we share further information and a summary of the feedback we received.\n\n>**The pre-training strategy [is] supervised [...], overlooking a range of existing self-supervised approaches [...]**\n\nThank you for your valuable feedback. We agree that our work utilizes a lesser explored strategy of supervised pretraining. Unsupervised molecular models are built on years of cutting-edge pretraining research [2,3,4] with vast empirical promises [6]. In fact, our work, to our best knowledge, is the first to successfully scale supervised pretraining for GNNs to the multi-billion parameter regime, setting new downstream task performance standards for the multiple benchmarks considered in our submission.\n\nIn Table 1 and 2 of the rebuttal doc, we compare our approach to self-supervised strategies and find that our method maintains the clear upper hand. The unsupervised MolE [10] model variants clearly underperform the standard MolE model that leverages both unsupervised and supervised pretraining (Table 2), which is in turn outperformed by the listed MolGPS variants by large margins. Even our smaller MPNN++ models of comparable size to MolE (~100M parameters) both outperform the self-supervised variant.\n\nIn Table 1, we compare to the unsupervised GraphMVP [11] on MoleculeNet (only on tasks that were not part of our pretraining). Our MolGPS from the original submission outperforms [11] on 3/4 tasks, while the model variants that we recently pretrained with additional Phenomics data outperform across all tasks. We acknowledge the results reported by Uni-Mol [12] slightly surpass our score, but note that [12] uses a different data splitting approach compared to GraphMVP, which can have a significant impact in low-data regiemes. We have checked the paper and code, but we were unable to find the recipe so far.\n\nLastly, the mentioned reference [13] focuses on 3D molecular modeling, which is outside the scope of the present work (as discussed below).\n\n>**The pretraining data [...] of only 5 million molecules.**\n\nWe would like to point out that due to the supervised pretraining, the scale of the dataset is not directly comparable. We also note that previous works in GNN literature scale to less than 5M graphs [14] and show interesting scaling trends.\n\nIn our case, it is important to characterize the diversity not only by the number of molecules, but instead paired with the number of labels per molecule. We recall our label scaling experiments that show the impact of reducing the data diversity by removing labels. The performance gains from incorporating Phenomics data further support this hypothesis (Figure 2 of rebuttal).\n\n>**The scaling of molecular data appears to perform poorly [for] the downstream [...].**\n\nWe indeed found no consistent trend on downstream task performance for increased number of molecules in the pretraining dataset (Figure 2(a) and Appendix E.3 of our submission). That said, no molecular scale (except for the lowest 12.5%) had an outstanding negative effect on downstream tasks. We therefore hypothesize that our approach is robust to a smaller number of molecules during pretraining when the highly important label diversity per molecule is maintained (as discussed above).\n\n>**[I]t is also important to evaluate quantum molecular properties such as QM9 and MD17.**\n\nThank you for the suggestion. However, datasets such as QM9 and MD17 require the model to reason over graph geometry in 3D coordinate spaces. Evaluating on such tasks would be an uneven evaluation of a 2D model such as MolGPS being used to learn 3D downstream tasks.\n\nWe would like to also highlight the strong performance of MolGPS on the complementary gene inhibition tasks (e.g., the pkis2-* task series of Polaris benchmark) and the significant improvements of our most recent model variants that have been pretrained with additional Phenomics data (Figure 1 in rebuttal pdf).\n\nFurthermore, ADME(T) tasks capture the efficacy and toxicity of bioassays which play a crucial role in screening and evaluation of likely drug candidates. These tasks continue to remain the pinnacle of industrial pharmaceutical evaluation [15,16]. We thus prioritize our empirical evaluation by keeping their real-world applications and biological importance in mind.\n\n[2]. Li et al, A knowledge-guided pre-training framework for improving molecular representation learning, Nature 2023\n\n[3]. Xia et al, Pre-training Graph Neural Networks for Molecular Representations: Retrospect and Prospect, ICML AI4Science Workshop 2022\n\n[4]. Li et al, KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction, ACM SIGKDD 2022\n\n[6]. Lu et al, Learning to Pre-train Graph Neural Networks, AAAI 2021\n\n[7]. Sun et al, Does GNN Pretraining Help Molecular Representation?, NeurIPS 2022\n\n[8]. Sun et al, MoCL: Data-driven Molecular Fingerprint via Knowledge-aware Contrastive Learning from Molecular Graph, ACM SIGKDD 2021\n\n[10] Méndez-Lucio et al, MolE: a molecular foundation model for drug discovery, arxiv 2022\n\n[11] Liu et al, Pre-training molecular graph representation with 3d geometry. arXiv preprint arxiv 2021\n\n[12] Zhou et al, Uni-mol: A universal 3d molecular representation learning framework, ChemRxiv 2023\n\n[13] Feng et al, Fractional denoising for 3d molecular pre-training, ICML 2023\n\n[14]. Chen et al, Uncovering neural scaling laws in molecular representation learning, NeurIPS, 2024\n\n[15]. Shi et al, Fine-tuning BERT for automatic ADME semantic labeling in FDA drug labeling to enhance product-specific guidance assessment, Journal of biomedical informatics, 2023\n\n[16]. Walter et al, Multi-task ADME/PK Prediction at Industrial Scale: Leveraging Large and Diverse Experimental Datasets, Molecular Informatics, 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for providing detailed feedback on the paper and appreciating its presentation (YrHv, kaW1, FQ72, Lxfs), organization (kaW1, FQ72) and scientific contribution (FQ72, Lxfst). Overall, we have updated the paper and responded to the reviewer’s concerns in the individual rebuttals. In the following, we discuss our updates and reviewer comments that might be relevant for all reviewers.

## Incorporating Phenomics data
We further integrated an additional data type into our pretraining data mix. The added Phenomics dataset contains ~6k labels for ~500k molecules (compounds) that were derived from phenomic imaging [1] of cells perturbed with either a dose of a compound or a gene knockout. We conducted a similarity analysis between the obtained images (represented by vector embeddings, e.g., similar to [2]) subject to a compound perturbation on one side, and images subject to a gene perturbation on the other side. The pretraining task is to predict for each compound if it has a phonemically visible similarity to a gene knockout (indicating a biological relationship).

Adding Phenomics data to our pretraining data mix (i.e., PCBA_1328 and PCQM4M), improved our downstream task performance across the board (Figures 1 and 2 of rebuttal doc). Comparing the scaling trends in Figure 2 of the rebuttal doc, MPNN++ w/ Phenomics pretraining exhibits a significant vertical upwards shift compared to the original MPNN++. Notably, we were also able to extend our scaling study to the 3B parameter regime. While we were previously unable to extend the scaling trend (Figure 2 of the rebuttal doc), MPNN++ w/ Phenomics maintains a positive scaling trend. We note that, in a slight deviation from the figure shown in our submission, Figure 2 of the rebuttal doc shows scaling trends for MPNN++ instead of GPS++. This is because our recent experiments with added Phenomics data were only conducted with MPNN++ for parameter scales <1B due to the availability of compute.

To better visualize the impact of our results on the TDC benchmark collection [3], we have renamed and added few baselines compared to Figure 2(b) of the original submission. We renamed TDC SOTA to “Best model per task” (a collection of 8 different models that together establish SOTA across the benchmark collection) and added MapLight + GNN [1] (the best single model out of those 8 methods) evaluated on the benchmark collection, which falls significantly short of the newly added MPNN++ variant w/ Phenomics pretraining. We are also thrilled to report that our MolGPS w/ Phenomics even outperforms the best model per task at parameter scale 3B and performs on par at scale 1B. Lastly, we added a purely self-supervised MolE variant [4] to represent unsupervised pretraining strategies (which is further discussed below).

## Comparison with unsupervised methods
In response to the feedback of Reviewer Lxfs, we added further comparison to unsupervised pretraining approaches (Table 1 and 2 of rebuttal doc). We observe that the unsupervised MolE model variants [4] clearly underperform the standard MolE model that leverages both unsupervised and supervised pretraining (Table 2), which is in turn outperformed by the listed MolGPS variants by large margins. Even our smaller MPNN++ models of comparable size to MolE (with ~100M parameters; see Figure 2 of rebuttal doc at the comparable parametric scale of 100M MPNN++) both outperform the self-supervised variant.

In Table 1, we compare to the self-supervised GraphMVP [11] model on MoleculeNet (which was already featured in the submission; we only consider the tasks that were not part of our pretraining). TheMolGPS model from the original submission outperforms GraphMVP on 3/4 tasks, while the model variants that we recently pretrained with additional Phenomics data outperform across all tasks.
 
We overall conclude that our pretraining strategy, is a promising avenue of molecular pretraining and novel at the parametric scale of the present work.

## Dataset scale in the context of supervised molecular pretraining
We would like to point out that due to our supervised pretraining approach, the scale of the dataset is not directly comparable to that of self-supervised methods, i.e, billions of molecules in some cases. We note however that previous works in GNN literature scaled to even less than 5M graphs [5] and observe interesting scaling trends.

In our case, it is important to characterize data scale not only by the number of molecules, but instead paired with the number of pretraining labels per molecule. PCBA_1328 considers more than 1k different labels per molecule (albeit with high sparsity) and PCQM4M comes with 25 graph-level tasks and 4 node-level tasks (for each node of the ~4M molecules). This is further confirmed by our label scaling study that shows the impact of reduced data diversity by removing labels (e.g., Figure 2(a) of submission). The performance gains from incorporating Phenomics data into the pretraining data mix also support this, adding ~500k of molecules with ~6k highly informative labels per graph (Figure 2 of rebuttal).

## Applicability to downstream tasks with 3D molecules like QM9 and MD17
As downstream tasks from 3D molecular modeling were mentioned by 2 reviewers, we would like to clarify that our modeling approach operates on 2D molecules, while datasets such as QM9 and MD17 require the model to reason over graph geometry in 3D coordinate spaces.

[1] Bray et al, Cell Painting, a high-content image-based assay for morphological profiling using multiplexed fluorescent dyes, Nat Protoc 2016

[2] He et al, Masked autoencoders are scalable vision learners, CVPR 2022

[3] Notwell et al, Admet property prediction through combinations of molecular fingerprints, arxiv 2023

[4] Méndez-Lucio et al, MolE: a molecular foundation model for drug discovery, arxiv 2022

[5] Chen et al, Uncovering neural scaling laws in molecular representation learning, NeurIPS, 2024

Meta-review: The paper investigates scaling laws for GNNs on molecular graphs through an extensive set of experiments. They also present a new GNN architecture called MolGPS based on the insights from these experiments that outperform prior models on several tasks.

Strengths:
1. Comprehensive set of experimental results.
2. New insights into the scaling properties of GNNs that leads to a new architecture.
3. Strong performance of MolGPS on multiple tasks.

Weaknesses:
1. The main weakness of the paper is the small scale of the datasets used -- scaling behavior cannot be fully ascertained at such small scales, and it is still unclear how the scaling properties hold at larger scales.
2. Focus on supervised learning, which is uncommon for studying scaling behaviors.
3. Focus limited to 2D graphs, which again makes it unclear how generalizable the observed scaling properties are.

Recommendation:
In spite of some weaknesses, the paper sheds light on some valuable insights into the scaling behavior of GNNs, and presents a new architecture that exhibits strong performance on multiple tasks. Overall, the strengths outweigh the limitations, and the paper is likely to be of strong interest to researchers applying GNNs to molecular tasks.
Decision: Accept (poster)