Does GNN Pretraining Help Molecular Representation?
Authors: Ruoxi Sun
Hanjun Dai
Adams Wei Yu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Graph Pretraining
Molecular Representation
Self Supervised Learning

Abstract: Extracting informative representations of molecules using Graph neural networks (GNNs) is crucial in AI-driven drug discovery. Recently, the graph research community has been trying to replicate the success of self-supervised pretraining in natural language processing, with several successes claimed. However, we find the benefit brought by self-supervised pretraining on small molecular data can be negligible in many cases. We conduct thorough ablation studies on the key components of GNN pretraining, including pretraining objectives, data splitting methods, input features, pretraining dataset scales, and GNN architectures, to see how they affect the accuracy of the downstream tasks. Our first important finding is, self-supervised graph pretraining do not always have statistically significant advantages over non-pretraining methods in many settings. Secondly, although noticeable improvement can be observed with additional supervised pretraining, the improvement may diminish with richer features or more balanced data splits. Thirdly, hyper-parameters could have larger impacts on accuracy of downstream tasks than the choice of pretraining tasks, especially when the scales of downstream tasks are small. Finally, we provide our conjectures where the complexity of some pretraining methods on small molecules might be insufficient, followed by empirical evidences on different pretraining datasets.

Does GNN Pretraining Help Molecular Representation?
Ruoxi Sun
Hanjun Dai Google Research, Brain Team hadai@google. com
Adams Wei Yu Google Research, Brain Team adamsyuxei@goggle com
Google Cloud AI Research ruoxis@google.com com
Abstract
Extracting informative representations of molecules using Graph neural networks (GNNs) is crucial in Al-driven drug discovery. Recently, the graph research community has been trying to replicate the success of self-supervised pretraining in natural language processing, with several successes claimed. However, find the benefit brought by self-supervised pretraining on small molecular data can be negligible in many cases. We conduct thorough ablation studies the key components of GNN pretraining, including pretraining objectives, data splitting methods, input features, pretraining dataset scales, and GNN architectures, how they affect the accuracy of the downstream tasks. Our first important finding is, self-supe graph pretraining do not always have statistically significant advantages over non-pretraining methods in many settings. Secondly, although noticeable improvement can be observed with additional supervised pretraining, the improvement may diminish with richer features or more balanced data splits. Thirdly,
could have larger impacts on accuracy of downstream tasks than the choice of pretraining tasks, especially when the scales of downstream tasks are small. Finally, we provide our conjectures where the complexity of some pretraining methods on small molecules might be insufficient, followed by empirical evidences on different pretraining datasets.
Introduction
Graph neural networks (GNNs). due their effectiveness, have been adopted model a wide range of structured data, such as social networks, road graphs, citation networks, etc. Molecule modeling is one of these important applications, where it serves as the foundation of biomedicine and nurturing techniques like novel drug discovery. However, labeling biomedical data are usually time-consuming and expensive and thus task-specific labels are extremely inadequate. This poses a big challenge to the field. Recently, inspired by the remarkable success of self-supervised pretraining from natural language processing [6, 2, 28] and computer vision domains [11, 5], researchers start trying to apply the pretrain-finetune paradigm to molecule modeling with GNN, hoping boost the performance of various molecular tasks by pretraining the model on the enormous unlabeled data. For instance, many methods have been proposed [32, 29]. where significant performance improvements pretraining on large scale datasets [12, 22, 35-37J. Despite of the promising results, we find that reproducing some of these outstanding gains via graph pretraining the improvement largely relies on the experimental setup and the extensive hyper-parameter
claimed by
be non-trivial, and sometimes parameter tuning of downstream tasks, rather than the design of pretraining objectives. These observations motivate to rethink the effectiveness of graph pretraining with unsupervised or self-supervised objectives, and investigate what factors would influence the effectiveness of self-supervised graph pretraining.
In this paper, we perform systematic studies to assess the performance of popular graph pretraining objectives different types of datasets, and exploit various confounding components in experimental
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Groph Features
GNN Booibione
Datasot
Chjective
Data Sp
Confog
Modecude Gimpph
HLL based ONN
mankee node
SLb predict
Soathar
Balanoat
1e3
SUBSCE
1eid
SAM
Contrnlioee
GIN
64 120
Pinckll
GOND_TVR
GraphSage
Suporissd
Other panamoi
Protroliig
Fine
Figure 1: A typical pipeline for graph pretraining and deployment for downstream applications.
setup in deciding the performance of downstream tasks with or without pretraining. Here, we restrict our studies to small molecular graphs, as opposed to other application domains, such as social networks or citation graphs. The key insights and take-aways of this paper are:
Among the pretraining tasks we evaluated, the self-supervised pretraining alone does not provide statistically significant improvements over non-pretrained methods on downstream tasks.
When additional supervised pretraining step is conducted after self-supervised pretraining, we observe statistically significant improvements. However, the gain becomes marginal on some specific data splits or diminishes if richer features are introduced.
Beyond data splits and hand-crafted features, the usefulness of graph pretraining is also sensitive to the experimental hyperparameters, such learning rates and number of study repeats. Different setups can lead to opposite conclusions.
In conclusion, different from the previous works, we do not observe clear and unconditional gains achieved by graph pretraining on molecular representation, indicating is still too early to conclude graph pretraining is effective in molecular domain.
We investigate the reason of above and hypothesize that the complexity of some pretraining methods on molecules is insufficient, leading to less transferable knowledge for downstream tasks.
Despite the overall negative results we obtained, the main goal of this paper is not to discourage the pretraining research for small molecules. Instead, we hope to raise the attention on different aspects of experiments and the role of simple hand-crafted features, designing better pretraining approaches. Below we first introduce the background of GNN and its pretraining in Section 2, and then experimental design and results in Section 3 and Section 4, respectively. Finally we conclude with our findings and the limitations in Section and Section 6.
to provide useful information for
2 Preliminary
Table 1: Summary of Experiments. Table 12 and Table 13 are deferred to appendix due to space limit.
Pretrain Objective
Graph Features Rich
Downstream Splits
Arch
Pretruin Datasel
Sell Superviedd
Supervised
Basi
Balanced
Scallodd
GIN
GraphSage
ZINCT3
SAVI
Table 2
Table 3
Tabble 4
Table 5
"Table's
Table 7
Table
Table 9
Table 10
Table 11
Table 12
Table 15
Graph Neural Networks (GNNs). Let G = {V. E} denote molecule graph with V as the set of nodes and E. as the set of edges. Given the node features X;, most GNNs learn an embedding representation h; for every node E V by aggregating representations from connected nodes and

---

edges, denoted as graph convolution. These procedure repeats for K times with the update equation as follows:
h UPDATE(M AGGREGATE((( ,115" eist Vj E N()))
where (Ni is the set of neighbor nodes of i and No X, The representation for en- tire graph G is then obtained by permutation-invarte transformation on node representation, ha READOUT E V). In this paper we mainly study the GNNs that belong to this family, namely the GNNS.
Finetune. After the pretraining, the pretrained model is used to finetune on the downstream tasks. For molecule property prediction tasks, the graph-level representation obtained from the pretrained model is connected to linear classifiers predict downstream task labels. The fine-tuning performed in an end-to-end manner, where both the pretrained GNN and the linear classifiers are trainable.
Graph pretraining objectives. The primary goal of pretraining to learn representations with robust transferable knowledge of graphs from the abundant pretraining data and then generalize to downstream tasks with usually different supervision signals. Generally the pretraining objectives can be categorized into self-superrised and supervised ones. We present brief overview of some representative objectives in the following sections.
2.1 Self-supervised (unsupervised) pretraining
In self- supervieed pretraining, the pretraining objective is designed to learn self-generated targets from the structure of the molecules, such as the type of nodes and edges, prediction of local context, graph partition, node clustering, occurrence of some functional groups, and etc. The predictive target can be node/edge level or entire graph level. We present some representative ones below:
2.1.1 Node Prediction
Node prediction is node-level classification task given the masked context of entire graph. Similar to Devlin et al. [6], some portion of node attributes are masked and replaced with mask-specified indicators in the node input feature. After graph convolution, the embedding output from GNN is used to predict the true attribute of the node, e.g. atom type in molecular graphs, through a linear classifier top of the node embedding.
2.1.2 Context Prediction
Context prediction task is sub-graph level task aiming at learning embedding that can represent the local subgraph surrounding a node. Generally it can be viewed as masked task for substructure. Since it is essentially structured prediction which can be difficult in general, Hu et al. [12] leverages the adversarial learning to teach the model to distinguish the positive sub-graph embedding from negative ones. Rong et al. [22] instead builds a dictionary of structures that captures the property of sub-graphs (e.g. type and quantity of neighbour nodes and bonds), and turns into multi-class classification problem.
2.1.3 Motif Prediction
Motif prediction [22] is to predict the existence functional groups, such as benzene ring or hydroxyl. The motifs extracted automatically from RDKit [15]. The motif prediction task is formulated as graph-level multi-label binary classification task, where the graph embedding is used to jointly predict the occurrence of these semantic functional motifs.
2.1.4 Contrastive learning
Graph contrastive learning is to maximize the agreement of two augmented views of the same graph, and minimize agreement different graphs. The optimization is conducted using contrastive loss in the latent embedding space [35, 10, 37, 29]. The augmentation function needs to transform graphs into realistic and novel augmentations without affecting semantie labels of the graphs. For example, the transformation be small perturbations modifications node/edge embedding, drop of

---

Table 2: Self-supervised Rich feature Balanced Scaffold Split. No pretrain has average value of 78.0% over all 5 datasets.
Methods
ВВEP
BACE
TOX21
TOXCAST
SIDER
AVE GAIN
No pretrain
92.25(133007)
79.20(+1.99) 1..3))
69.13(40.55)
61.92(+0.89)
0[F-1626)
Node Prediction
92.24(4-2.76)
87.32(++1.67)
79.57(42.003
69.77(10013)
61.62(#1.12)
0.122(
Context Prediction
92.68(#1.19)
86.98(+1.26)
79.05(+2.51)
70.18(+:0.4)
61.65(#0.77)
0.126(+1..544
Motif Prediction
1.193(++.11)
1..16(+1.66)
79.22(+2.38)
69.09(+0.07)
1.245(+1.255)
0.128(+1.310)
Contrastive learning
92.31(##1.58)
86.67(+2.40)
78.45(42.44)
68.37(+0.80)
61.22(#1.20)
(0.788++1.684)
few nodes or edges, and so on. These transformations enforce an underlying prior for contrastive learning, that is, local transformation does not change the semantic meaning of graph.
2.2 Supervised pretraining
Supervised pretraining aims learn domain-spectie graph-level knowledge from specifically de- signed pretraining tasks. For molecular application, the supervised labels are generated from diverse set of functional studies like biochemical assays. The pretrainning task to perform multiple binary classification and jointly learn the supervised labels. Although the pretraining mainly refers to unsupervised or supervieee methods as they are not limited by the requirement of supervised labels, supervised pretraining is still great source to investigate the graph pretraining in general.
Table 3: Supervised + Rich feature + Balanced Scaffold. No pretrain has an average AUC of 78.0%.
Methods
BRBP
BACE
TOX21
TOXCAST
SIDER
AVE GAIN
pretrain
79.2(11.99) 17.33(1.633) 92.23(13.07)
69.13(10.55)
(6166
Supervised
86.91(+-186) 91.65(++2.1)
81.13(+2.39)
71.64(10046)
(2.14(-113))
0.712(++1590)
Masking Supervised
93.43(+-2.50)
86.90(=2.04)
11.93(+1.79)
71.66(+0.73)
62.6((=188)
1.386+11,766
Context Prediction Supervised
92.27(++1.57)
38.72(+1.6)
81.71(+1.9)
02.19(:0.79)
63.210 1.49)
1,638++14464
3 Experiment framework
To investigate pretraining on graphs for molecule representations, we first revisit the typical eetraining-finetunnii pipeline used in the literature. Figure 1 shows the overall procedure of deployment, with several design choices presented at each stage the pipeline. Since different choices each stage can lead different performances on the downstream tasks, investigate them one at a time while keeping others the unchanged. The design principle of our experiment framework is analyze the effect of every stage in the pipeline comprehensive as possible, while also keeping it tractable to avoid exponentially many experiments.
3.1 Design choices
We consider the design choices for the four pretraining objectives.
Pretraining objective In Section 2 we have provided a brief literature review over the pretraining methods for molecule representation. Here we categorize those pretraining by different principles, and present one well-recognized representative each category. The representatives are selected because they have more desired properties, such as better performance, compared with their counterparts.
Masking. We leverage the node prediction objective, which randomly masks 155% of the nodes' feature and then ask GNN to make prediction on the node attributes of the masked ones. This strategy resembles the BERT pretraining [6] in natural language processing.
Structured. Unlike text data where the topology is a sequence, the graph has rich structure information. Following Hu et al. [12], we use context prediction objective, which masks the context from Lj -h-s to k -o-hops and leverages adversarial training to predict the true context embeddings from the random context embeddings.
Graph-level self-supervised. Following [22], GNN is asked to predict whether a motif is con- tained molecule. The motif can be extracted from the molecule with RDKit [15]. The motifs are 85 motifs for multi-label classification.
hattp://rdkite org/ poocs//oorce/rdke. Chem. Fragments html

---

Table 4: Self-supervised Rich feature Scaffold. No pretrain has average ROC-AUC of 71.8%
over all benckmark datasets.
Methods
ВВEP
BACE
TOX21
TOXCAST
STDER
AVE GAIN
No pretrain
74.33(H0.73)
75.86(+0058)
65.95(+0.15)
11.30)
0(H0.579)
Node Prediction
73.45(1-0.27)
83.66(10.75)
75.30(1-0.7))
66.50(+00066)
65.08(1-.122)
0.990(1-0.233)
Context Prediction
74.10(=0.22)
81.87(+0:49)
75.37(++.11)
66.86(++0.07)
62.84(#0.46)
0.400(+0.280)
Motif Prediction
73.65(+0.36)
74.55(+0.79) 80.58(++2004
65.63(+0.07)
64.05(+0.23)
-0.116(H0.766)
Contrastive learning
73.32(+2.38)
80.51(=0.80)
74.55(+0.22))
65.70(#0.09)
64.39(=0.63)
-0.1 -0.114(++.513)
Table 5: Supervised Self-supervised + Rich feature Scaffold. No pretrain get 71.8% average ROC-AUC.
Methods
BBBP
BACE
TOX21
TOXCAST
SIDER
AVE GAIN
No pretrain
7..33220773
40.10(=0.42)
75.86(20.58)
63.95(00.13)
62.30(21.14)
0(20.604)
Supervised
72.7%
83,23(20.67)
11.66(10.08)
67.72(-0.13)
65.34(10.17)
1.500(103500
Masking Node + Supervised
73.38(=0.555)
84.42(=0.27)
67.141-0.28) 77.85(+0.24)
64.06(+0.28)
(.50((+0.324)
Context Prediction Supervised
73.81(1:0.52)
84.3((20.93)
77.10(0.14)
67.87(00.08)
65.19(10.17)
1858(10.368)
Contrastive. We generate two views of the same graph by corrupting the input node features with Gaussian noise. We leverage the contrastive learning loss proposed in [35]: we maximize the consistency between positive pairs (from same graphs) and minimize that between negative pairs (from different graphs). In this paper, we restrict ourselves to this specific contrastive training method, however, various contrastive learning methods can be further explored.
Graph-level supervised. Finally when applicable, we use the ChEMBL dataset with graph-level labels for graph-level supervised pretraining as Hu al. [12].
The above are the design choices for pretraining objectives. Next, we consider other factors that influence graph- pretranntraiiing performance.
Graph Features Each molecule is represented by a graph with atoms as nodes and bonds as edges. In this paper mainly consider the graph representations without the 3D information. For each molecule graph, chemical properties of nodes and edges are extracted to serve as node and edge features for the graph neural networks. Depending on how rich the features are, categorize the design choices into two categories:
Basic features. The basic set of features are the ones used in Hu et [12]. Specifically, the node features contain the atom type and the derived features, such as formal charge list, chirality list, etc. The edge features contain the bond types and the bond directions. These features are categorical, and thus will be encoded in one-hot vector individually and then concatenated together to form the feature vector for node/edge representation.
Rich features. The rich feature is a superset of the basic features. In addition the basic ones mentioned above, it comes with the additional node features such as hydrogen acceptor match, acidic match and bond features such as ring information. This set of features are used in Rong et al. [22]. Additionally and importantly, we follow their setting to incorporate additional 2d normalized vdNormaaizeeDeseripoos features 2, which used in the downstream tasks only and not in pretraining.
Please refer to the original papers for the full set of basic [12] and rich [22] features, respectively.
GNN Backbone The GNN architecture also plays role in graph pretraining. In Hu et al. [12], the results show that pretraining on GNN variants like GIN [33] would improve the performance on downstream tasks, while the performance with architectures like GAT [27] would actually get worse performance with pretraining. As the GNNs based on I-Weisfeiler-Lehmee isomorphism check [23],
(WL) test have
similar representation power [33] bounded by the Weisfeiler- consider the two representative GNN architectures, namely the GIN [33] and GraphSage [9]. They have shown benefits with graph pretraining in Hu et al. [12].
Pretraining dataset In natural language pretraining, researchers observed a significant performance boost due to self-supervised pretraining on large-scale data, that is, the larger the pretraining dataset
Phttps://githin http:////gggtt.b.coo/t//l/lllll/ wwtp//p-eeelell//dssssssepppepee
for the feature descriptor.
5

---

Table 6: Self-supervised + Basic feature + Balanced Scaffold. No pretrain has average AUC of 76.7% over all 5 datasets.
Methods
BRBP
BACE
TOX21
TOXCAST
STDER
GAIN
No prefrain
91.46C 0.85)
84,29(+ 3.80)
78.35(E 0.95)
6 1.61)
61.15(+ 2.46)
0(+1.934)
Node Prediction
91.23 1.51)
84.97(= 1.55)
77.77(# 1.23)
68. 68.98(= 1.11)
61.20(# 0.41)
0.11((+1.122) 0.560(22.194)
Context Prediction
92.13 1.04)
84.83(2 3.19)
78.79( 2.52)
68.29(= 1.23)
62.32(4 2.99)
Table 7: Supervised + Self-supervised + Basic feature + Balanced Scaffold. No pretrain has an
average AUC of 76.7%.
Methods
BBBP
BACE
TOX21
TOXCAST
SIDER
AVIN
No pretrain
91.46. 0.85)
84.29k 3,80)
78.35(=
0.95)
68.31(1661]
61.15 2,46]
01 1,954)
Supervised
90.70C+ 0.74)
84.22C= 2.69)
80.45(= 1.47)
69,47(+ 1.06)
63.38(+ 1.44)
0.932 1480)
Musking Node + Supervised
91. 2.88)
85.54( 4.57)
81.49( 1.52)
70.77(+ 1.00)
62.81
261)
1.630 2.516)
Context Prediction Supervised
91.54 3.52)
85.71
81,23(2 1.94)
71.361 1.5)
62.75(1 2.27)
1806(= 2.340)
is, the better the downstream performance it is [20]. Inspired by this success in natural language processing, test the algorithms two unlabeled pretraining datasets with different scales.
ZINC15 [25]: ZINCI5 contains 2 million molecules. This dataset was preprocessed following Hu al. [12].
SAVI [19]: The SAVI dataset contains about 1 billion molecules, which are significantly larger than ZINCI5. To the best our knowledge, it has never been used for pretraining tasks before. This dataset contains drug-like molecules synthesized by computer simulated reactions.
Additionaly, we used ChEMBL [8] as the supervised datasets. Different from the above ZINC15 and SAVI dataset which are only used for self-supervised pretraining, this dataset contains 500k drug-able molecules with 1,3 10 prediction target labels from bio-activity assays for drug discovery. Thus like Hu al. [12] we only leverage it for supervised pretraining.
Data split on downstream tasks The downstream tasks for molecular domain we used are 5 benchmark datasets from MoleculeNet [30] (See Appendix A.5 for more details). The tali//s sets are split with ratio 8:1:1. For molecule domain, the random split is not the most meaningful way to assess the performance, because the real-world scenarios often require generalization ability on out-of of-diitiibttionn samples. So we consider the following ways to split the data:
Scaffold Split [12, 21] This strategy first sorts the molecules according to the scaffold (e.g. molecule structure), and then partition the sorted list into train/valid/test splits consecutively. Therefore, the molecules train and test sets are most different ones according to their molecule structure. Note this strategy would yield deterministic data splits.
Balanced Scaffold Split [1, 22] This strategy introduces the randomness in the sorting and splitting stages above, thus one can run on splits with different random seeds and report the average performance to lower the evaluation variance.
We choose balanced scaffold as our major evaluation configuration, because it allows us to evaluate the algorithm multiple data splits while maintaining the ability evaluate out of distribution samples (e.g. assess generalization ability). Evaluating on one single split (such as scaffold split) can be subject bias due to specific split, leading to higher variance in evaluation.
3.2 Experiment protocol
As the total number of configurations for the entire pipeline can combinatorially large which is not practical for us to exhaustively experiment with all of them, design protocol with pairwise comparison principle. Specifically, we first anchor vanilla configuration with a certain design choice of combination for each stage. To study the effect of each stage the pretraining effectiveness, we vary the design choice one stage at time compared to the vanilla configuration.
For all these experiments, to assess the effectiveness of graph pretraining, we report the ROC-AUC on downstream tasks as well as the relative average gain over all downstream datasets with and without pretraining. For each setting we will report the mean and standard deviation (in parenthesis) over three runs with different random seeds. We tune the model on downstream tasks with the validation set, and report the evaluation metric the test set using the model with best validation

---

Table 8: Unsupervised + Basic feature Scaffold. No pretrain has average accuracy of 68.7%
over all benckmark datasets.
Methods
BBBP
BACE
TOX21
TOXCAST
SIDER
AVE AVE GAIN
No pretrain
69.62( 1.05)
29.77(+-429)
75.52(+0.67)
63.67(+0.32)
19.07(+1113)
0C+1.492)
Node Prediction
68. (00222116)
76.95(10.12)
75.88(10060)
64.11(10.38)
0.656(1:0822) (1.27()..87)
Context Prediction
69.4((211.4) 44)
81.96(110.72)
75.49(11.775)
63.48(10.31)
62.27(110.9)
1.792(1:0.824)
Table 9: Supervised Basic feature + Scaffold. No pretrain has an average accuracy of 68.7% over all benckmark datasets.
Methods
ВВBP
BACE
TOX21
TOXCAST
SIDER
AVE GAIN
prefrain
69.62( 1.05)
75.77(+4.29)
75.521 -0667)
63.67(
59.07(+1.13)
Supervised
68.96(110.64
76.30t 11,30)
76.64(_0.39
66.07(10.22
61.97(10.96)
2258(10.700
Masking Node Supervised
7.677(+0.67)
79.13 0.299)
6..32(
62.12((0.19)
+0.374
Context Prediction Supervised
69.25()0.25)
83.34 10.67)
78.11(_0.28)
66.15)
10.43)
3.412(10.422)
performance. For each setup, we report the average performance obtained with three random seeds. We tune the learning rate in {le-1, -le-,,5e 5e 1e-s, 5e -8, lel2, -e,e,, lel1e for each setup individually and select the one with best validation performance. For GNNs fix the hidden dimension to 300 and number of layers to 5.
Results
In this section, we present the results and discussions for set of experiments designed with the protocols in Section 3.2. Table summarizes the experimental configurations for each following table. We will elaborate on them in the following sections. Due to space limit, defer investigation different GNN architectures to appendix (Section A.
4.1 Vanilla configuration
We choose the vanilla configuration with the settings from existing works [12, 22]. Specifically, we use the rich feature with GIN backbone, pretrained on ZINC15 when pretraining is applied, and evaluate on the Balanced Scaffold Split for downstream tasks. One important baseline is without pretraining. For the ease of comparison, we include the results without pretraining in each table.
4.2 Self-supervised pretraining objectives
We compare the results pretrained with different self-supervised pretraining objectives. As is presented in Section 3.1, we consider four representative types of pretraining objectives. For the ease of comparing the performance, we only consider one objective at a time, instead of mixing different pretraining objectives obtain a multi-task pretrained model. Table shows the performance on downstream molecule property prediction benchmarks with models initialized from different pretraining objectives. The relative average gain compared the one without pretraining is not statistically significant, i.e., not larger than the standard deviations of multiple runs. All the four different objectives obtain similar gains/loses regardless of very different designs. To fully understand the effect of self-supervised pretraining molecule representation, we further investigate the performance of different pretraining objectives in combination with other factors, such as input features or data splits, as described in the following sections.
4.3 Supervised pretraining objectives
In addition the self-supervised objectives, we study the potential benefits with supervised pretrain- ing. Unlike the self- supervise setting where the molecule graphs themselves are used for pretraining, the supervised pretraining requires extra cost of data labeling, and thus pretraining. In this paper, we present the results with supervised pretraining alone, as well as the joint pretraining. e.g. pretrain with self-supervised objective and followed by supervised pretraining, in Table 3. We can see with the supervised pretraining, one can improve the downstream performance, which aligns with the observation from Hu [12]. Our hypothesis that, supervised pretraining is helpful when the pretraining tasks are closely aligned with the downstream tasks. In particular,
scalable for large scale
7

---

Table 10: Large scale pretraining data with balanced scaffold split. No pretraining gets average
AUC of 78.0%.
Methods
BBBP
BACE
TOX21
TOXCAST
SIDER
AVE GAIN
No pretrain
92.22(+3.07)
1.633
79.2t= .99
69. (9.13((0:55)
61.9((0899)
Node Prediction
92.33(+22088)
87.22(=1.79)
19.12(+11.2)
69.47(+0.65)
61.24(++1.94)
-0.106(=1.616
Context Prediction
03.32(+-0.53)
79.18(42488 87.77(+-2.4))
48)
70. 50)13((0.56)
62.24(4-2.5)
0.546(+1 8322)
Masking Node SSpeerissd
93.2M 1.02)
86.39( 1.67)
81899
63.73(1 (1.77(10.50
1.200X 797)
Context Prediction Supervised
92.55(++2.33)
87.76(+1887)
12.19(+1.58)
72.91(+0.71)
62.44(+0.4)
I 1,588(+1 508)
the bio-activity labels provided by ChEMBL is highly related to the drug discovery purpose and
drug discovery properties are the major topics evaluated in the downstream tasks. Therefore, the positive correlation between the pretraining supervision and downstream tasks contribute the most to the performance improvement of downstream tasks.
4.4 Data split on downstream tasks
Molecular data is usually diverse and limited, so chemists are particularly interested in the generaliza- tion ability of GNNs on out of distribution data. Also due to the same reason (i.e. limited and diverse data). the variance in performance of different splits is significant, which poses challenges on robust evaluation. In vanilla configuration we use the balanced scaffold split, and here we show additional results with the scaffold split, which is deterministic data split that makes the train/valid/test set differ from each other the most. Table 4 and Table 5 respectively present the results using scaffold split with self-supervised without and with additional supervised pretraining. Compared with Table and Table 3, it is clear to see that Table 4 and Table 5 have significantly lower ROC-AUC. Specifically the AUC drops 6.2% on average for all benchmarks without pretraining. On the other hand, we can see if we compare Table with Table 2, or Table 5 with Table respectively, the gain of pretraining is more significant on the scaffold split. We speculate the reason for the improvement of scaffold split is that the initialization of neural network parameters (e.g. from pretraining) are typically critical for the -i-fr-idittiitution generalization (e.g. scaffold split). Similar observations have also been studied in the meta-learning literature [7]. Although the gain with supervised pretraining is significant in Table 5, the effect self-supervised pretraining mixed in Table This indicates the effectiveness of self-supervised pretraining on scaffold split is not significant enough to claim "very helpful".
4.5 Graph features
So far we have presented the results with rich features. Now we want to see how those basic features used in Hu al. [12] affect the outcome. Table 6 and Table show the test ROC-AUC (%) performance with features on the balanced scaffold splits using self-supervised or supervised pretraining objectives, respectively. Table and Table show the same results but on scaffold split.
In a nutshell, without pretraining, rich features lead to an average gain of 1.3% and 3.1% over basic features using balanced scaffold split and scaffold split, respectively. Specifically, it achieves 76.7% vs 78.0% for balanced scaffold split, and 78% vs 71.8% on scaffold split. The gain brought by the rich features are more significant than the ones with different self supe-supereieed pretraining objectives. Table to Table 9 show that pretraining has more positive impact when basic features are used. In particular, the self- pettrarettann with context prediction shows significant gains especially in the scaffold split setting. However, the gain diminishes when careful feature engineering are applied to the downstream tasks (use rich feature in vanilla configuration). The supervised pretraining continues the significant gain under these settings, which shows the consistency and reliability of the situation supervisions.
with the labeled and downstream-taskkale
4.6 Pretraining datasets
As observed in natural language processing domain, more text pretraining data lead better down- stream performance. Intuitively this can be true for molecule representation domain well, so we run a new set of experiments with the model pretrained on SAVI dataset, which is about 500 times larger than the ZINC15 dataset we used in the above result sections. We present the results pretrained on SAVI dataset using balanced scaffold split or scaffold split in Table 10 and Table 11, respectively. Other configurations are the same as the vanilla configuration.
8

---

Table 11: Large scale pretraining data with scaffold split. No pretraining gets an average AUC of 71.8%.
Methods
BEBP
BACE
TOX21
TOXCAST
SIDER
AVE GAIN
No pretrain
74.83(+0073)
80. 00.10
75.86(+0.58)
65.95(=0.15)
62.30(+1.14))
604)
Node Prediction
81.90(=1.59) 13.81(#1882)
66.95(=0.12) 74.94(40.05)
62.93(=0.34)
0
Context Prediction
74.32(10088)
83.93(20.24)
14.42(10.19)
67.01(00229)
1094(10.404) 64.83220.45)
Masking Node + +Sppervised
13.32(=00.00)
18.59(+0.009 13.38)=1.05)
67.01(0118)
05.40(=0.12)
4732
Context Prediction Supervised
74.38(
86.33(=0.16)
78.16(40.25)
68.71(0007))
62.22(20.48)
2.152(20.378)
Compared with the performance on ZINC15, the SAVI pretraining data does not lead to a significant improvement either on balanced scaffold split (Table 2 vs Table 10) scaffold split (Table 4 or Table 11). Similarly, the self-supervised pretraining objectives lead to negligible gain on downstream task performance, while the supervised one still achieves clear gain.
As the result is counterintuitive, we further investigate the reason behind it by inspecting the pretrain- ing performances with different training objectives on both ZINCI5 and SAVI datasets. We plot the curve of accuracy growth with the number of training steps iterated. We can see from Figure 2 that in all settings the pretraining accuracy grows above 90% quickly after only 0.1 to 0.2M steps and also converges quickly. Given that the model gets very high accuracy without even going through I epoch of the SAVI dataset, it is expected that the larger training data like SAVI may not provide more learning signals for the model, and partially explains why more molecules wouldn't help significantly in this case. Furthermore, these figures might suggest several reasons of why the self-supervised pretraining may not be very effective in some situations:
Tasks are easy. Some of the pelf-rrrtg tasks for molecules might be easy, so that model learns less useful information from pretraining. For example in the masked node prediction case, the model is expected to predict the atomic number from a vocabulary with less than 100 candidate atoms. Furthermore, due the valence constraints, the graph topology may already exclude most of the wrong atoms. As a comparison, the vocabulary size for text pretraining may be 100k or even higher. Some structured prediction tasks like context prediction might be hard, but due the difficulty of structured prediction itself and the proposal for high quality negative examples for contrastive learning, it can still be challenging for downstream task improvements. Other strategies like motif prediction can be achieved by subgraph matching, which can be easy for GNN that intrinsically does the graph isomorphism test.
Data lacks diversity. Due to biophysical and functional requirements, molecules share many common sub-structures, e.g., functional motifs. Hence, molecules may not be as diversified text data. This is why the model learns to generalize quickly within the training distribution.
2D Structure not enough to infer functionality, Some important biophysical properties (such as 3D structure, chirality) are barely reflected in the 2D-feature-bas or 2D graph features). For example, the molecules with the same chemical formula and 2D feature, can have very different chirality, which leads to quite different toxicity [24] (e.g. flipped toxicity labels). This is not captured the current GNN pretraining frameworks that we considered.
pretraining (e.g., using smiles
4.7 prpee-ppraaettrs for downstream tasks
(5AVI] Mode Predidion
DAVVI Centont Prodictin
We also find that hyper-parameters for downstream tasks are critical for the their performance that their choices may change the conclusion of the effec- tiveness of pretraining in some settings. We can take the learning rate as an ex- ample. As the models initialized from scratch and pretraining may have dif- ferent scales, the most suitable learning rate required for downstream tasks may also be different. Without tuning learn- ing rate extensively, we may reach a misleading conclusion. In particular,
45% 110 125
4% 100 14 18
240
Figure 2: Pretraining accuracy on ZINC15 or SAVI datasets with node prediction or context prediction objectives.

---

when we adopt the default learning rate for reproducing the existing success of pretraining in Table 17 of Appendix A.4, we indeed observe the advantage of pretraining. However, if we follow our proce- dures (e.g. extensive search learning rate and averaging over three splits), the resulting Table 2 and Table 3 indicate no performance gain by pretraining. So we suggest that the evaluation of pretraining should consider the hypeer-arammeer tuning and averaging over different splits.
5
Summary and takeaways
Based on our experiments in Section 4, we present our takeaways by empirically summarizing our conjectures when the pretraining would/would not help the molecular representation learning.
When pretraining might help We find it typically helps 1) if we can have the supervised pretrain- ing with target labels that are aligned with the downstream tasks. However, getting large amount of high-quality and relevant supervision is not always feasible; 2) if the high quality hand-crafted features are absent. However, it seems that the gain obtained by self-supervised pretraining is not as significant as these high quality hand-crafted features based on our current studies; 3) if the downstream train, valid and test dataset distributions are substantially different.
When the gain diminishes? In some situations the gain of pretraining might diminish 1) if we already have the high quality hand-crafted features (e.g. rich features described in Section 3.1); 2) we don't have the highly relevant supervisions. As shown in Section 4.6, many self-supervised pretraining tasks might be too easy for the model to learn meaningful embedding; 3) if the downstream data splits are balanced; 4) if the self-supervised learning dataset lacks diversity, despite its scale.
Why pretraining may not help in some cases? In our paper we pretrained GNN on much larger dataset (SAVI) than before, hoping to replicate gain of pretraining like in NLP domain. However, we do not obtain the expected gain. The pretraining accuracy curve (Figure 2) provides some potential explanations of why pretraining may not work: some of the pretraining accuracy curve grows above 95%+ quickly and converges fast, unlike pretraining in NLP which keeps growing to 70% and hardly plateaus. This suggests that some of the pretraining methods like masked node label prediction might be easy (as the vocabulary size is much smaller compared to NLP) and therefore transfer less knowledge for downstream tasks.
6 Limitations of current study
Although we have tried our best to design a comprehensive study on the effectiveness of graph pretraining for molecular representation, there are still limitations want to point out. Due the limited time and resources we have, we are not able to fully cover the whole picture of the current pretraining paradigm in graph neural networks. Nevertheless, we list them here in hope of preventing the sver-generalization of our conclusion.
Distribution of graphs. Our study focuses on pretraining for small molecule graph inductive representation learning. Recently there are works on pretraining transductive representation learning [36] on large graphs [13], where our conclusion may not be directly extended to these cases.
Graph architectures. GNN is a popular research field where many new architectures with probable expressiveness are/will be proposed. The results we have shown are on two representative 1-WL GNNs. It can be possible that the latest advances of deep GNN [17] and Transformer-based
GNN [34, 3. 14] might yield different results.
Learning objectives. Although we have presented results with different types of self-supervised losses, there are still many variants of each type that we did not explore, like different variants [26, 31] of contrastive learning. Also, multi-task learning of different self-supervised objectives might be another direction for further exploration.
Downstream datasets. We obtained our conclusion mainly on the datasets from MoleculeNet [30]. Datasets like Alchemy [4] and drug-target Interaction [18] may show different results.
10

---

References
[1] Guy W Bemis and Mark A Murcko. The propertics of known drugs. 1, molecular frameworks. Journal of medicinal chemistry, 59(15):2887-2893 1996.
[2] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv: 2020.
[3] Dexiong Chen, Leslie O'Bray, and Karsten Borgwardt. Structure-aware transformer for graph representation learning. In International Conference Machine Learning, pages 3469-3489. PMI.R 2022.
[4] Guangyong Chen, Pengfei Chen, Chang- Yu Hsieh, Chee- Kong Lee, Benben Liao, Renjie Liao, Weiwen Liu, Jiezhong Qiu, Qiming Sun, Jie Tang, et al. Alchemy: A quantum chemistry dataset for benchmarking ai models. arXiv preprint arXiv: / /906/09477007 2019.
[5] Ting Chen, Simon Korbblihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint urXiv:1810.. 048050 2018.
[7] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adapta- tion of deep networks. In International Conference on Machine Learning, pages 1126-1 135. PMLR, 2017.
[8] Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, et al. Chembl: a large-scale bioactivity database for drug discovery. Nucleic acids research, 10(D D1107, 2012.
[9] William L Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. arXiv preprint arXiv: 2017.
[10] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International Conference on Machine Learning, pages 41 4116-4126. PMLR, 2020.
[11] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pages 9726-9735, 2020.
[12] Weihua Hu. Bowen Liu, Joseph Gomes, Marinka Zimik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXiv preprint 1e...1905..... 2019.
[13] Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1857-1867, 2020.
[14] Jinwoo Kim, Tien Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, and Se- unghoon Hong. Pure transformers are powerful graph learners. arXiv preprint a7X:5.220..025055 2022.
[15] G Landrum. Rdkit: Open-source cheminformatics software, 2016.
[16] P. Langley. Crafting papers on machine learning. In Pat Langley, editor, Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pages 1207-1216, Stanford, CA, 2000. Morgan Kaufmann.
[17] Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards deeper graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 338-348, 2020.
11

---

[18] Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby. Hongyu Guo, and Jian Tang. Pre-training molecular graph representation with 3d geometry, arXiv preprint 1rX0::2110.07728, 2021.
[19] Hitesh Patel, Wolf Ihlenfeldt, Philip Judson, Yurii S Moroz, Yuri Pevzner, Megan Peach, Nadya Tarasova, and Mare Nicklaus. Synthetically accessible virtual inventory (savi). 2020.
[20] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint 19X0v:190..100
2019.
[21] Bharath Ramsundar, Peter Eastman, Patrick Walters, and Vijay Pande. Deep learning for the life sciences: applying deep learning to genomics, microscopy, drug discovery, and more. " O'Reilly Media, Inc.", 2019.
[22] Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. arXiv preprint arXiv:2007.02835, 2020.
[23] Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-lemman graph kernels. Journal of Machine Learning Research, 12(9), 2011.
[24] Silas W Smith. Chiral toxicology: it's the same thing... only different. Toxicological sciences, 110(1):4-30, 2009.
[25] Teague Sterling and John J Irwin. Zinc 15-lin discovery for everyone. Journal of chemical information and modeling,
2337, 2015.
[26] Mengying Sun, Jing Xing, Huijun Wang, Bin Chen, and Jiayu Zhou. Moel: data-driven molecular fingerprint via knowledge-aware contrastive learning from molecular graph. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 3585-3594, 2021.
[27] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint 1710. 1710.1003,, 2017.
[28] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2/09.01665, 2021.
[29] Lirong Wu, Haitao Lin, Zhangyang Gao, Cheng Tan, Stan Li, et al. Self-supervised on graphs: Contrastive, generative, or predictive. arXiv preprint arXiv:2105.07342, 2021.
[30] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9():513-5530 2018.
[31] Jun Xia, Lirong Wu, Jintao Chen, Bozhen Hu, and Stan Z Li. Simgrace: A simple framework for graph contrastive learning without data augmentation. In Proceedings of the ACM Web Conference 2022, pages 1070-1079, 2022.
[32] Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji. Sulf-supervised learning of graph neural networks: A unified review. arXiv preprint arXiv:2102. 10757, 2021.
[33] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.000
2018.
[34] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform bad for graph representation? arXiv preprint 2tt1:21000052334 2021.
[35] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances Neural Information Processing Systems, 33:5812-5823, 2020.
12

---

[36] Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. When does self-supervision help graph convolutional networks? In International Conference on Machine Learning, pages 10871-10880. PMLR, 2020.
[37] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In Proceedings of the Web Conference 2021, pages 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]. Though really apply.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as URL)? [No]. We will prepare code soon.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're you're using/curating? [Yes]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "xHBi5HNpwX", "reviewer": "Reviewer_GwSv", "summary": "This paper benchmarks the performance of graph pretraining under different settings. In particular, the paper performs thorough ablation studies on the key components of GNN pretraining, including pretraining objectives, data splitting methods, input features, pretraining dataset scales, and GNN architectures. They find that self-supervised graph pretraining does not have statistically significant advantages over non-pretraining methods in many settings. Although improvement can be observed with additional supervised pretraining, the improvement may diminish with richer features or more balanced data splits. The authors hypothesize the complexity of pretraining on molecules is insufficient, leading to less transferable knowledge for downstream tasks.", "questions": "1. Why the five downstream datasets are used? Why others are not used?\n\n2. What the reason behind the success and failure of graph pretraining? ", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The idea of benchmarking graph pretraining under different settings is original and deserves a thorough study. \n2. This paper considers many different settings including pretraining objectives, data splitting methods, input features, pretraining dataset scales, and GNN architectures. The settings are extensive and well studied in the experiments. \n3. The organization and writing are sound and satisfying. \n\nWeakness:\n1. The authors use ZINC/SAVI for self-supervised learning, ChEMBL for supervised pretraining, and evaluate the performance on BBBP, BACE, TOX21, TOXCAST, and SIDER. I am not sure whether the five datasets BBBP, BACE, TOX21, TOXCAST, and SIDER are representative enough. It seems that the choice of downstream tasks is a bit ad-hoc. For example, these three datasets ClinTox, MUV, and HIV are also widely used for evaluating graph pre-training. As far as I know, the performance of graph pretraining varies on different datasets. The results on some datasets are insignificant and may be worse than random initialization. The authors may need to provide more evidence about why these five datasets are chosen and why other datasets are not considered. Otherwise, the evaluation is not comprehensive enough.\n2. The authors show that graph pretraining does not bring significant improvements in many settings. Then the authors hypothesize that the complexity of pretraining on molecules is insufficient, leading to less transferable knowledge for downstream tasks. However, the paper does not provide any insights into why graph pretraining fails in many different settings. On the other hand, graph pretraining does improve the performance in some settings. Although the paper hypothesizes that the similarity between downstream tasks and pretraining tasks might play a role here, the paper does not provide significant insights into why pretraining can help in these settings. Some empirical analysis might be needed to improve the paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "iBxqJ5l0NyX", "reviewer": "Reviewer_gddu", "summary": "In this paper, the authors perform a comprehensive empirical study on GNN pre-training for molecular representations from the perspectives of pretraining objectives, data splitting methods, input features, pretraining dataset scales, and GNN architectures. They find that self-supervised pretraining alone does not provide statistically significant improvements over non-pretrained methods on downstream tasks. Also, data splitting methods, input features, hyperparameters may be more important than the choice of pre-training strategies. Additionally, they think the insufficient complexity of pre-training on molecular graphs is the key factor for the poor performance.", "questions": "See the weakness above.", "limitations": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is well-presented and easy to follow.\n- The experimental results are meaningful to the community.\n\n**Weakness**\n- The authors only evaluate the performance on the downstream task of molecular property prediction. However, their used MoleculeNet benchmark is potentially brittle because the performance of pre-trained models on these datasets is extremely unsteady with diverse random seeds, which may bias their conclusions. I suggest the authors to try more recent benchmarks such as Alchemy [1] for property prediction. Additionally, the learned molecular representations can also be applied in more molecule-related downstream tasks such as drug-target Interaction [2]. Above-mentioned benchmarks or tasks may help avoid the instability of MoleculeNet. \n- The conclusion of existing pre-training strategy cannot work in molecular representations may not be convincing. Firstly, the authors only evaluate some naïve strategies which pre-train GNNs at the level of either entire graphs or individual nodes. As illustrated in a representative work [3], these tasks often give limited improvement and can even lead to negative transfer on many downstream tasks. Moreover, the authors show that 'Masking Node + Supervised' or 'Context Prediction + Supervised' pre-training tasks can bring notable improvements over 'no pre-training', which aligns the conclusions of [3] that combining node-level and graph-level pre-training will be conducive indeed. Additionally, the authors only try the vanilla molecular graph augmentations (adding Gaussian noise to input node features) in contrastive learning, which may alter the semantics of molecules dramatically due to their discrete nature and thus incur negative transfer. I would suggest the authors to try more advanced molecular augmentations such as automated augmentations [4], hidden feature augmentations [5] and knowledge-guided augmentations [6].\n- The discussions of the reasons why pretraining does not help are neither adequate nor convincing. Moreover, how can we make the molecular pretraining more difficult?\n\n[1] Alchemy: A Quantum Chemistry Dataset for Benchmarking AI Models\n\n[2] Pre-training Molecular Graph Representation with 3D Geometry (ICLR 2022)\n\n[3] Strategies for Pre-training Graph Neural Networks (ICLR 2020)\n\n[4] Graph Contrastive Learning Automated (ICML 2021)\n\n[5] SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation (WWW 2022)\n\n[6] MoCL: Data-driven Molecular Fingerprint via Knowledge-aware Contrastive Learning from Molecular Graph (KDD 2021)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dASVYFAF_pJ", "reviewer": "Reviewer_LNeY", "summary": "This paper presents a comprehensive empirical study on pretraining GNNs for downstream molecule property prediction tasks, investigating aspects including features, GNN architectures, pretraining datasets, pretraining objectives, downstream task data split, and fine-tuning hyperparameters etc. It shows that self-supervised pretraining alone generally bring non-significant improvement in performance alone, while supervised pretraining helps when downstream tasks do not have richer features or more balanced data splits.", "questions": "It would be beneficial to also include \n1) another contrastive learning baseline with node dropping and subgraph augmentation\n2) the performance when using several or all the self-supervised pretraining objectives\n3) some more scalable architectures.", "limitations": "The limitations are adequately addressed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#Strengths\n- The empirical study is novel, systematic and comprehensive, probing a range of design choices made by previous works. \n- The central claims, especially the set of observations in section 5, are valuable summaries for future works and well supported by the experiments.\n- The paper is clearly written and easy to follow\n\n#Weaknesses\n- The specific contrastive method evaluated is not the known best fit. As shown in GraphCL(https://arxiv.org/pdf/2010.13902.pdf), the choice of data augmentation is crucial. Specifically, this work shows that the suitable augmentations for biochemical molecules should be node dropping and subgraph, instead of attribute masking used in this empirical study. \n- The experiments do not consider combined self-supervision objectives.\n- The paper does not include more scalable architectures (e.g. deep GNN and various transformer-based GNNs)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_oJjKxJWkWi", "reviewer": "Reviewer_qMss", "summary": "The paper looks into the effect of self-supervised GNN pre-training in graphs used for molecular representation. The paper assesses the performance of various popular graph pre-training objectives and finds that the benefit is negligible in many cases. The paper also finds that when improvement is observed with supervised pre-training, the improvement may diminish with richer features or more balanced data splits. Lastly, the paper shows that hyper-parameters can have a larger impact on accuracy of downstream tasks than the pretraining tasks. Paper offers examples of when pre-training might and might not be useful, based on experimental findings. Takeaways in this paper are contradictory to some related work in the field of GNNs which claims that, similarly to NLP models, GNNs might benefit from self-supervised pre-training. ", "questions": "- Have the authors thought about whether or not findings in this paper would translate to GNNs pretrained on graphs coming from different distributions - e.g. more general domain?\n\n- Minor Miscellaneous Edits\n- Line 18: extra space (GNNs) , \n- Line 21: typo: nurturing?\n- Line 184: typo:  scenarios\n\n", "limitations": "Authors adequate address limitations in their paper under ‘Limitations of current study’\n\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The research question is interesting: does pretraining help in GNNs?\n- Paper is sound & authors design and perform an extensive number of experiments to support their claims\n- The main question of assessing the value of pretraining is conceptually significant. The findings that pretraining might have negligible gains is interesting. However, one of the takeaways of the paper is that benefit of pretraining diminishes with richer/more high quality features. I believe this is to be expected - particularly on small datasets, as the ones used in this paper (authors also mention this under Limitations). If you have a small graph and rich enough features, then there probably isn’t much additional information to get from pretraining. Moreover, one of the benefits of GNNs is being able to learn complex representations from the graph structure itself, rather than having to use rich features. For small enough graphs, there probably isn’t that much representational information to learn from. Authors acknowledge this under Section 4.6, but I think it should be made clear that the findings in the paper apply to small graphs\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_TBHVScy6Tw", "reviewer": "Reviewer_tcsP", "summary": "Self-supervised pre-training of GNNs on molecule data can be very useful for tasks in computational chemistry, e.g., drug discovery. This has motivated several works proposing pretraining methods that allegedly lead to higher downstream fine-tuning performance over non-pretrained (i.e. randomly initialized) models.\n\nThe authors of the reviewed paper challenge this claim and conduct systematic experiments revealing that the aforementioned SSL methods do not have statistically significant advantages over non-pretrained methods in many settings. Their main conclusion is that current SSL methods produce \"less transferable knowledge for downstream tasks.\"  \n", "questions": "* Are the two pre-training datasets you're using (ZINC15,SAVI) commonly used by the pre-training methods you investigate? For example, [1,2] use NCI1, and [3] uses the GEOM dataset. While the two datasets you use seem to be appropriate based on your description, one possibility leading to negligible pre-training gains might still be that they are simply not well-suited for pre-training?\n\n[1] Graph Contrastive Learning with Augmentations, You et al, Neurips 2020.\n[2] Graph Contrastive Learning Automated, You et al, ICML 2021. \n[3] Evaluating Self-Supervised Learning for Molecular Graph Embeddings, Wang et al, arXiv:2206.08005.", "limitations": "Yes, the authors describe two relevant limitations of their work. ", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\nThe paper \n* systematically investigates the sensitivity of choices in different stages of the pre-training pipeline by changing one factor at a time. \n* proposes to split downstream task data not randomly like previous works but based on scaffolds. This allows for better OoD generalization evaluation since evaluating on a single split can be subject to biases. \n* further examines the impact of using rich features [1], i.e., additional node features not commonly used in many previous papers. \n* The paper reveals several surprising results, e.g., Table 2 shows that no pretraining method performs significantly better than non-pre-trained models.  \n* The paper complements other recent works reporting positive results for GNN pretraining on molecules with some negative results [1,2,3,4]. This is not to say that one must be right and the other wrong. More diversity in terms of investigated settings or experimental configurations (data split, hyper-parameters, etc.) is useful for the community overall. \n\n# Weaknesses\nAssuming that the authors conducted experiments properly and with genuine efforts to reproduce previous work's results (see my questions below), I don't see any major weaknesses in this empirical study. \n\nSome minor writing suggestions which do not significantly impact my scores:\n* tables and figures are quite misplaced; e.g., Table 3 is shown on Page 4 but eventually discussed on Page 7. Figure 2 is shown right next to section 4.7; it isn't discussed there but in section 6. \n* The paper looks pretty crammed. For example, section 4.4. should be split up into multiple paragraphs.  I also see that the authors removed vertical space/margins of the section titles. I understand that this is likely due to the page limit. However, I suspect that some sentences might not add much information and can be removed to make the key messages stand out more clearly. Alternatively, maybe more details can be moved to the appendix. \n\n[1] Self-Supervised Graph Transformer on Large-Scale Molecular Data, Rong et al, NeurIPS 2020. \n[2] Graph Contrastive Learning with Augmentations, You et al, Neurips 2020.\n[3] Graph Contrastive Learning Automated, You et al, ICML 2021. \n[4] Evaluating Self-Supervised Learning for Molecular Graph Embeddings, Wang et al, arXiv:2206.08005.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept