Strategies for Pre-training Graph Neural Networks
Authors: Weihua Hu*
Bowen Liu*
Joseph Gomes
Marinka Zitnik
Percy Liang
Vijay Pande
Jure Leskovec
Venue:  (2020)
Keywords: Pre-training
Transfer learning
Graph Neural Networks

Abstract: Many applications of machine learning require a model to make accurate pre-dictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training. An effective approach to this challenge is to pre-train a model on related tasks where data is abundant, and then fine-tune it on a downstream task of interest. While pre-training has been effective in many language and vision domains, it remains an open question how to effectively use pre-training on graph datasets. In this paper, we develop a new strategy and self-supervised methods for pre-training Graph Neural Networks (GNNs). The key to the success of our strategy is to pre-train an expressive GNN at the level of individual nodes as well as entire graphs so that the GNN can learn useful local and global representations simultaneously. We systematically study pre-training on multiple graph classification datasets. We find that naïve strategies, which pre-train GNNs at the level of either entire graphs or individual nodes, give limited improvement and can even lead to negative transfer on many downstream tasks. In contrast, our strategy avoids negative transfer and improves generalization significantly across downstream tasks, leading up to 9.4% absolute improvements in ROC-AUC over non-pre-trained models and achieving state-of-the-art performance for molecular property prediction and protein function prediction.

Published as a conference paper at ICLR 2020
STRATEGIES FOR PRE-TRAINING GRAPH NEURAL NETWORKS
Weihua Hu'; Bowen Liu Joseph Gomes Marinka Zitnik Percy Liang', Vijay Pande", Jure Leskovec!
'Department of Computer Science, *Chemistry, *Bionngineering. Stanford University, *Department of Chemical and Biochemical Engineering. The University of lowa, 5 *Department of Biomedical Informatics, Harvard University
(weihuahu, liubowen, pliang, jure (es stanford.edu edu, joe Loe-gomes seooons@ edu, marinka harvard. edu, pandelstan edu
ABSTRACT
Many applications of machine learning require model make accurate pre- dictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training. An effective approach to this challenge is to pre-train a model on related tasks where data is abundant, and then fine-tune it on downstream task interest. While pre-training has been effective in many language and vision domains, it remains an open question how to effec- tively use pre-training on graph datasets. In this paper, we develop a new strategy and sule-sup methods for pre-training Graph Neural Networks (GNNs). The key to the success of our strategy to pre-train an expressive GNN the level individual nodes as well as entire graphs so that the GNN can learn useful local and global representations simultaneously. We systematically study pre-training multiple graph classification datasets. We find that naïve strategies, which pre-train GNNs at the level of either entire graphs or individual nodes, give limited improvement and can even lead to negative transfer many downstream tasks. contrast, our strategy avoids negative transfer and improves generalization sig- nificantly across downstream tasks, leading up to 9.4% absolute improvements ROC-AUC over non-pre-trained models and achieving state-of-the-art performance for molecular property prediction and protein function prediction.
INTRODUCTION
Transfer learning refers to the setting where model, initially trained on some tasks, is re-purposed on different but related tasks. Deep transfer learning has been immensely successful in computer vision (Donahue et al., 2014; Girshick et al., 2014; Zeiler & Fergus, 2014) and natural language processing (Devlin et al., 2019; Peters et al., 2018: Mikolov et al., 2013). Despite being an effective approach to transfer learning, few studies have generalized pre-training to graph data.
Pre-training has the potential provide attractive solution to the following two fundamental challenges with learning on graph datasets (Pan & Yang, 2009; Hendrycks et al., 2019): First, task-specific labeled data be extremely scarce. This problem is exacerbated in important graph datasets from scientific domains, such as chemistry and biology, where data labeling (e.g.. biological experiments in a wet laboratory) is resource- and time-intensive (Zitnik et al., 2018). Second, graph applications often contain out-of distriibubttnn samples, meaning that graphs in
data from the training set are structurally very different from graphs in the test set. nut-of-distribution prediction is common in real-world graph datasets, for example, when one wants to predict chemical properties of a brand-new, just synthesized molecule, which is different from molecules synthesized so far, and thereby different from all molecules in the training set.
However, pre-training on graph datasets remains a hard challenge. Several key studies (Xu al., 2017; Ching et al., 2018; Wang et al., 2019) have shown that successful transfer learning is not only
"Equal contrihution. Project website, data and code: http::/snne stanford. iaa.norrrredggnnnpeerrrraaaa

---

Published as a conference paper at ICLR 2020
(a.i)
Node-level pre-tralning only
(a.il)
Graph-level pre-training only
(a.ii) Node-level
(b) Categorization of our pre-training methods
Graph-level pre-training
Node space
Node-level
Graph-level
Supervised Attribute
Pooling
Pooling
Pooling
Attribute prediction
Attribute Masking
Graph space
Prediction
Structural
Structure
Context Prediction
Similarity
prediction
Prediction
Graph embeddings
Node embeedoings
Linear classifier
Figure I: (a.i) When only node-level pre-training used, nodes of different shapes (semantically different nodes) can be well separated, however, node embeddings are not composable, and thus resulting graph embeddings (denoted by their classes, + and -) that are created by pooling node-level embeddings are not separable. (a.ii) With graph-level pre- pre-training only, graph embeddings are well separated, however the embeddings of individual nodes do not necessarily capture their domain- specific semantics. (a.iii) High-quality node embeddings well separated, while at the same time, the embedding space is also composable. This allows for accurate and robust representations of entire graphs and enables robust transfer of pre-trained models to variety of downstream tasks. (b) Categorization of pre-training methods for GNNs. Crucially, our methods, i.e., Context Prediction, Attribute Masking, and graph-level supervised pre-training (Supervised Attribute Prediction) enable both node-level and graph-level pre-training.
such that nodes of different types
matter of increasing the number of labeled pre-training datasets that are from the same domain as the downstream task. Instead, it requires substantial domain expertise carefully select examples and target labels that are correlated with the downstream task of interest. Otherwise, the transfer of knowledge from related pre-training tasks to new downstream task can harm generalization. which is known as negative transfer (Rosenstein et al., 2005) and significantly limits the applicability and reliability of pre-trained models.
Present work. Here, we focus on pre-training as an approach to transfer learning in Graph Neural Networks (GNNs) (Kipf& Welling, 2017; Hamilton et al., 2017a; Ying et al., 2018b; Xu et al., 2019; 2018) for graph-level property prediction. Our work presents two key contributions. (1) We conduct the first systematic large-scale investigation of strategies for pre-training GNNs. For that, we build two large new pre-training datasets, which share with the community: chemistry dataset with sppciffcc
2 million graphs and a biology dataset with 395K graphs. We also show that large datasets are crucial to investigate pre-trale pre-training and that existing downstream benchmark datasets are too small to evaluate models in a statistically reliable way. (2) We develop an effective pre- training strategy for GNNs and demonstrate its effectiveness and its ability for out-of-distribution generalization on hard transfer-learning problems.
In our systematic study, we show that pre-training GNNs does always help. Naïve pre-training strategies can lead to negative transfer on many downstream tasks. Strikingly, a seemingly strong pre-training strategy (i.e., graph-level multi-task supervised pre-training using a state-of-the-as graph neural network architecture for graph-level prediction tasks) only gives marginal performance gains. Furthermore, this strategy even leads to negative transfer on many downstream tasks (2 out of molecular datasets and 13 out of 40 protein prediction tasks).
We develop an effective strategy for pre-training GNNs. node-level information and encourage GNNs to capture domain-specion edges, in addition to graph-level knowledge. This helps the GNN to learn useful representations at both global and local levels (Figure 1 (a.iii)), and is crucial to be able to generate graph-level representations (which are obtained by pooling node representations) that are robust and transferable to diverse downstream tasks (Figure 1). Our strategy is in contrast naïve strategies that either leverage only at graph-level properties (Figure (a.ii)) or node-level properties (Figure (a.i)).
The key idea to use easily accessible
knowledge about nodes and
Empirically, our pre-training strategy used together with the most expressive GNN architecture, GIN (Xu et al., 2019), yields ofte-of-ttheart results on benchmark datasets and avoids negative transfer across downstream tasks we tested. It significantly improves generalization performance
2

---

Published as a conference paper at ICLR 2020
across downstream tasks, yielding up 9.4% higher average ROC-AUC than non-pre-trainee tre-pre-triiedd GNNs, and up to 5.2% higher average ROC-AUC compared to GNNs with the extensive graph-level multi- task supervised pre- pre-training. Furthermore, we find that the most expressive architecture, GIN, benefits more from pre-training compared to those with less expressive power (e.g., GCN (Kipf & Welling, 2017), GraphSAGE (Hamilton et al., 2017b) and GAT' (Velickovic et al., 2018)), and that pre-training GNNs leads to orders-of-magniuude faster training and convergence in the fine-tuning stage.
PRELIMINARIES OF GRAPH NEURAL NETWORKS
We first formalize supervised learning of graphs and provide overview of GNNs (Gilmer et al., 2017). Then, we briefly review methods for unsupervised graph representation learning.
Supervised learning of graphs. Let G (V, E) denote graph with node attributes X, for E V and edge attributes Cue for (u, v) E E. Given a set of graphs [Gi...., GN } and their labels (1), yv 1, the task of graph supervised learning is to learn representation vector ha that helps predict the label of an entire graph G, yG g(hc). For example, in molecular property prediction, G is molecular graph, where nodes represent atoms and edges represent chemical bonds, and the label to be predicted can be toxicity or enzyme binding.
Graph Neural Networks (GNNs). GNNs use the graph connectivity as well node and edge features to learn representation vector (i.e., embedding) h, for every node E G and vector hG for the entire graph G. Modern GNNs use a neighborhood aggregation approach, where repre- sentation of node is iteratively updated by aggregating representations of v's neighboring nodes and edges (Gilmer et al., 2017). After iterations of aggregation, v's representation captures the structural information within its k-hop network neighborhood. Formally, the k-th layer of GNN is:
h COMBINE(A)
(1(- ((-11) (ed=",,e)))) u E E((i)))) (2.1)
where (R) is the representation of node u at the k-th iteration/layer, Cuv is the feature vector of edge between u and and (we) is set neighbors of v. We initialize 1. X
Graph representation learning. To obtain the entire graph's representation his the READOUT function pools node features from the final iteration K,
hg = READOUT (K) GEE
(2.2)
READOUT is a crmmutation-invariann function, such as averaging or a more sophisticated graph-level pooling function (Ying et al., 2018b; Zhang al., 2018).
3 STRATEGIES FOR PRE-TRAINING GRAPH NEURAL NETWORKS
At the technical core of our pre-training strategy is the notion to pre-train& a GNN both at the level of individual nodes well as entire graphs. This notion encourages the GNN to capture domain-specific semantics at both levels, as illustrated in Figure (a.iii). This is in contrast to straightforward but limited pre-training strategies that either only use pre-training predict properties of entire graphs (Figure (a.ii) or only use pre-training to predict properties of individual nodes (Figure (a.i)).
In the following, we first describe our node-level pre-training approach (Section 3.1) and then graph-level pre-training approach (Section 3.2). Finally, we describe the full pre-training strategy in Section 3.3.
3.1 NODE-LEVEL PRE- TRAINING
For node-level pre- pre-training of GNNs, approach is casily accessibce unlabeled data to capture sppeiffc knowledge/regularitis in the graph. Here we propose two self-supervised methods, Context Prediction and Attribute Masking.
3

---

Published as a conference paper at ICLR 2020
Input graph
(a) Context Prediction
(b) Attribute Masking
hop neighbodhood
ONN
IC, N S.
ONN
Context graph
Cemer made
CNN
(C, N,O,O, Scee? Marked mode
Figure 2: Illustration of our node-Ievel methods, Context Prediction and Attribute Masking for pre- training GNNs. (a) In Context Prediction, the subgraph is a K K-hop neighborhood around selected center node, where K is the number of GNN layers and is set to 2 in the figure. The context defined as the surrounding graph structure that is between 71- and r2-hop from the center node, where use ri and 72 = in the figure. (b) In Attribute Masking, the input node/edge attributes (e.g., atom type in the molecular graph) randomly masked, and the GNN is asked to predict them.
3.1.1 CONTEXT PREDICTION: EXPLOITING DISTRIBUTION OF GRAPH STRUCTURE
In Context Prediction, we use subgraphs to predict their surrounding graph structures. Our goal is to pre-train a GNN so that it maps nodes appearing in similar structural contexts to nearby embeddings (Rubenstein & Goodenough, 1965; Mikolov et al., 2013).
Neighborhood and context graphs. For every node e, we define v's neighborhood and context graphs as follows. K Khhop neighborhood of v contains all nodes and edges that are at most K-hops away from v in the graph. This is motivated by the fact that a K-layer GNN aggregates information
across the K-th order neighborhood of u, and thus node embedding (( depends nodes that at most K--hops away from v. We define context graph of node v as graph structure that surrounds e's neighborhood. The context graph described by two hyperparameters, r'1 and r2, and represents a subgraph that is between r1 ri-hops and r2-hops away from v (i.e., it is a ring of width r2 - ri). Examples of neighborhood and context graphs are shown in Figure (a). We require 71 < K so that some nodes are shared between the neighborhood and the context graph, and we refer to those nodes as context anchor nodes. These anchor nodes provide information about how the neighborhood and context graphs are connected with each other.
Encoding context into a fixed vector using an auxiliary GNN. Directly predicting the context graph is intractable due to the combinatorial nature of graphs. This is different from natural language processing, where words come from a fixed and finite vocabulary. To enable context prediction, we encode context graphs as fixed-length vectors. To this end, we use auxiliary GNN, which we refer to as the context GNN. As depicted in Figure (a), we first apply the context GNN (denoted as GNN' in Figure 2 (a)) to obtain node embeddings in the context graph. We then average embeddings of context anchor nodes to obtain a fixed-length context embedding. For node v in graph G, we denote its corresponding context embedding as cy
Learning via negative sampling, We then use negative sampling (Mikolov et al., 2013; Ying et al., 2018a) jointly learn the main GNN and the context GNN. The main GNN encodes neighborhoods to obtain node embeddings. The context GNN encodes context graphs to obtain context embeddings. In particular, the learning objective of Context Prediction is a binary classification of whether a particular neighborhood and particular context graph belong to the same node:
(KO C.' Rd 1 1 and of are the same nodes),
(3.1)
where (-)) is the sigmoid function, and 1(-) is the indicator function. We either let o' and G' = (i.e., a positive peighborhood-conter pair), or we randomly sample o' from a randomly chosen graph G' (i.e., negative weighborhood-comter pair). We use negative sampling ratio of 1 (one negative pair per one positive pair), and use the negative log likelihood as the loss function. After pre-training, the main GNN is retained as our pre-trained model
4

---

Published as a conference paper at ICLR 2020
3.1.2 ATTRIBUTE MASKING: EXPLOITING DISTRIBUTION OF GRAPH ATTRIBUTES
In Attribute Masking, we aim to capture domain knowledge by learning the regularities of the node/edge attributes distributed over graph structure.
Masking node and edges attributes. Attribute Masking pre-training works as follows: We mask node/edge attributes and then we let GNNs predict those attributes (Devlin et al., 2019) based on neighboring structure. Figure (b) illustrates our proposed method when applied to molecular graph. Specifically, We randomly mask input node/edge attributes, for example atom types in molecular graphs, by replacing them with special masked indicators. We then apply GNNs to obtain the corresponding node/edge embeddings (edge embeddings can be obtained embeddings of the edge's end nodes). Finally, a linear model is applied on top of embeddings to predict masked node/edge attribute. Different from Devlin et al. (2019) that operates sentences and applies message passing over the fully-connected graph of tokens, we operate on non-fully- connected graphs and aim capture the regularities of node/edge attributes distributed over different graph structures. Furthermore, we allow masking edge attributes, going beyond masking node attributes.
sum of node
Our node and edge attribute masking method is especially beneficial for richly-annotated graphs from scientific domains. For example, (1) in molecular graphs, the node attributes correspond to atom types, and capturing how they are distributed over the graphs enables GNNs to learn simple chemistry rules such as valency, as well as potentially more complex chemistry phenomenon such as the electronic or steric properties of functional groups. Similarly, (2) in protein-protein interaction (PPI) graphs, the edge attributes correspond to different kinds of interactions between pair of proteins. Capturing how these attributes distribute across the PPI graphs enables GNNs to learn how different interactions relate and correlate with each other.
3.2 GRAPH-LEVEL PRE-TRAINING
We aim to pre-train GNNs to generate useful graph embeddings composed of the meaningful node embeddings obtained by methods in Section 3.1. Our goal is to ensure both node and graph embed- dings are of high-quality so that graph embeddings are robust and transferable across downstream tasks, illustrated in Figure (a.iii). Additionally, there are two options for graph-level pre-training, as shown in Figure (b): making predictions about domain-specific attributes of entire graphs (e.g., supervised labels), or making predictions about graph structure.
3.2. SUPERVISED GRAPII-LVEL PROPERTY PREDICTION
As the graph-level representation hG is directly used for fine-tuning on downstream prediction tasks, it is desirable to directly encode domain-specific information into hG.
We inject graph-level domain-specifie knowledge into our pretrained embeddings by defining su- pervised graph-level prediction tasks. In particular, consider practical method to pre-train graph representations: graph-level multi-task supervised pre-training to jointly predict a diverse set of supervised labels of individual graphs. For example, in molecular property prediction, we can pre-train GNNs to predict essentially all the properties of molecules that have been experimentally measured so far. In protein function prediction, where the goal is predict whether a given protein has a given functionality, we can pre-train GNNs to predict the existence of diverse protein functions that have been validated far. In our experiments in Section 5, we prepare a diverse set of supervised tasks (up to 5000 tasks) to simulate these practical scenarios. Further details of the supervised tasks and datasets are described in Section 1. To jointly predict many graph properties, where each property corresponds to binary classification task, we apply linear classifiers on top of graph representations.
Importantly, naïvely performing the extensive multi-task graph-level pre-training alone can fail to give transferable graph-level representations, as empirically demonstrated Section 5. This is because some supervised pre-training tasks might be unrelated to the downstream task of interest and can even hurt the downstream performance (negative transfer). One solution would be to select "truly-relevant supervised pre-training tasks and pre-train GNNs only on those tasks. However, such a solution is extremely costly since selecting the relevant tasks requires significant domain expertise and pre-training needs to be performed separately for different downstream tasks.
5

---

Published as a conference paper at ICLR 2020
To alleviate this issue, our key insight is that the multi-task supervised pre- pre-training only provides graph-level supervision; thus, local node embeddings from which the graph-level embeddings are created may not be meaningful, as illustrated in Figure (a.ii). Such non-useful node embeddings can exacerbate the problem of negative transfer because many different pre-training tasks can more easily interfere with each other in the node embedding space. Motivated by this, our pre-training strategy is to first regularize GNNs at the level of individual nodes via node-level pre-training methods described in Section 1.1, before performing graph-level pre-training. As we demonstrate empirically, the combined strategy produces much more transferable graph representations and robustly improves downstream performance without expert selection of supervised pre-training tasks.
3.2.2 STRUCTURAL SIMILARITY PREDICTION
A second approach is to define a graph-level predictive task where the goal would be to model the structural similarity of two graphs. Examples of such tasks include modeling the graph edit distance (Bai et al., 2019) predicting graph structure similarity (Navarin et al., 2018). However, finding the ground truth graph distance values is a difficult problem, and in large datasets there is a quadratic number of graph pairs consider. Therefore, while this type of pre-training is also very natural, it is beyond the scope of this paper and we leave its investigation for future work.
3.3 OVERVIEW: PRE-TRAINING GNNS AND FINE-TUNING FOR DOWNSTREAM TASKS
Altogether, our pre-training strategy is to first perform node-level self-supervised pre-training (Sec- tion 3.1) and then graph-level multi-task supervised pre-training (Section 3.2). When the GNN pre-training finished, we fine-tune the pre-trained GNN model on downstream tasks. Specifically, we add linear classifiers top of graph-level representations to predict downstream graph labels. The full model, i.e,, the pre-trained GNN and downstream linear classifiers, is subsequently fine-tuned in an end-to-end manner. Time-compleste analysis is provided in Appendix F, where show that our pre-training methods incur little computational overhead to forward computation in GNNs.
FURTHER RELATED WORK
There is rich literature on unsupervised representation learning of individual nodes within graphs, which broadly falls into two categories. In the first category are methods that use local random walk-based objectives (Grover & Leskovec, 2016; Perozzi et al., 2014; Tang et al., 2015) and methods that reconstruct graph's 's adjacency matrix, e.g.. by predicting edge existence (Hamilton al., 2017a; 7a; Kipf & Welling, 2016). In the second category are methods, such as Deep Graph Infomax (Veličković al., 2019), that train a node encoder that maximizes mutual information between local node representations and pooled global graph representation. All these methods encourage nearby nodes to have similar embeddings and were originally proposed and evaluated for node classification and link prediction. This, however, can be sub-optimal for graph-level prediction tasks, where capturing structural similarity of local neighborhoods is often more important than capturing the positional information of nodes within graph (You et al., 2019; Rogers & Hahn, 2010; Yang et al., 2014). Our approach thus considers both the node-level as well as graph-level pretraining tasks and as we show in our experiments, it is essential to use both types of tasks in order for pretrained models to achieve good performance.
A number of recent works have also explored how node embeddings generalize across tasks (Jaeger et al., 2018; Zhou et al., 2018: Chakravarti, 2018; Narayanan et al., 2016). However, all of these methods use distinct node embeddings for different substructures and do not share any parameters. Thus, they are inherently transductive, cannot transfer between datasets, cannot be fine-tuned in an und-to-end manner, and cannot capture large and diverse neighborhoodd/ccmneets due to data sparsity. Our approach addresses all these challenges by developing pre-training methods for GNNs that use shared parameters to encode the the graph-level as well as node-level dependencies and structures.

---

Published as a conference paper at ICLR 2020
5 EXPERIMENTS
5.1 DATASETS
We consider two domains; molecular property prediction in chemistry and protein function prediction in biology. We release the new datasets at: http:://saee
stanford. edu/ http/gggnnnngaaeeeeeeeeeeeeeeeee rain.
Pre-training datasets. For the chemistry domain, we use 2 million unlabeled molecules sampled from the ZINC15 database (Sterling & Irwin, 2015) for node-level self-supervvsed pre-training, For graph-level multi-task supervised pre-training, we use a preprocessed ChEMBL dataset (Mayr et al., 2018: Gaulton et al., 201 1), containing 456K molecules with 1310 kinds of diverse and extensive biochemical assays. For the biology domain, we use 395K unlabeled protein ego-networks derived from PPI networks of 50 species (e.g. humans, yeast, zebra fish) for node-level self-supervised pre- training. For graph-level multi-task supervised pre-training, we use 88K labeled protein ego-networks to jointly predict 5000 coarse- coarse-grained biological functions (e.g., cell apoptosis, cell proliferation)
Downstream classification datasets. For the chemistry domain, considered classical graph classification benchmarks (MUTAG, PTC molecule datasets) (Kersting et al., 2016; Xu et al., 2019) as downstream tasks, but found that they too small (188 and 344 examples for MUTAG and PTC) to evaluate different methods in a statistically meaningful way (see Appendix B for the results and discussion). Because of this, as our downstream tasks, we decided use 8 larger binary classification datasets contained in MoleculeNet (Wu et al., 2018), a recently-curated benchmark for molecular property prediction. The dataset statisties are summarized in Table For the biology domain, we compose our PPI networks from Zitnik et al. (2019), consisting of 88K proteins from 8 different species, where the subgraphs centered at protein of interest (i.e., ego-networks) are used to predict their biological functions. Our downstream task is to predict 40 fine-grained biological functions' that correspond to 40 binary classification tasks. In contrast to existing PPI datasets (Hamilton et al., 2017a), our dataset is larger and spans multiple species (i.c., not only humans), which makes it a suitable benchmark for evaluating out-o--distribution prediction. Additional details about datasets and features of molecule/PPI graphs are in Appendices C and D.
Dataset splitting. In many applications, conventional random split is overly optimistic and does not simulate the real-world use case, where test graphs can be structurally different from training graphs (Wu et al., 2018; Zitnik et al., 2019). To reflect the actual use case, we split the downstream data in the following ways to evaluate the models' domain, we use scaffold split (Ramsundar et al., 2019), where we split molecules according to their scaffold (molecular substructure). In the biology domain, we use species split, where we predict functions of proteins from new species. Details are in Appendix E. Furthermore, to prevent data leakage, all test graphs used for performance evaluation are removed from the graph-level supervised pre-training datasets.
dittr--dittribttionn generalization. In the chemistry
5.2 EXPERIMENTAL SETUP
We thoroughly compare our pre-training strategy with two naive baseline strategies: (i) extensive supervised multi-task pre-training on relevant graph-level tasks, and (ii) node-level self-supervised pre-training.
GNN architectures. We mainly study Graph Isomorphism Networks (GINs) (Xu et al., 2019), the most expressive and ofate-of-the-art GNN architecture for graph-level prediction tasks. We also experimented with other popular architectures that are less expressive: GCN (Kipf & Welling, 2016), GAT (Veličković al., 2019), and GraphS (with mean neighborhood aggregation) (Hamilton et al., 2017b). We select the following hyper-parameters that performed well across all downstream tasks in the validation sets: 300 dimensional hidden units, 5 GNN layers (K = 5), and average pooling for the READOUT function. Additional details can be found in Appendix A.
Pre-training. For Context Prediction illustrated in Figure 2 (a), on molecular graphs, we define context graphs setting inner radius r 4. On PPI networks whose diameters are often smaller than 5, we use r1 = 1, which works well empirically despite the large overlap between the neighborhood and context subgraphs. For both molecular and PPI graphs, we let outer radius r2 = ri + and
Fine--gral grained labels harder obtain than coarse-grained labels; the latter used for pre-training.
7

---

Published as a conference paper at ICLR 2020
Dataset
BBBP
Tox21
ToxCast
SIDER
ClinTox
MUV
HIV
BACE
Average
Molocules
2039
1427
41127
Binary prediction tasks
12
617
27
17
Pre-training strategy
Out-of of disttribbtttionn prediction (scuffold split)
Graph-level
Node-level
65.824 =4.
74.0
20.8
20.6
57.3 11.6
718
75.3 21.9
70.1 =3.4
67.0
Infomax
68.8 00.8
75.3
+0.5
62.7+00.4
+0.4
58.4
+0,8
69.9
75.3 +25
76.0
+0.7
75.9 +1.
70.3
EdgePred
76.0
10.6
121
79.9
109
AurMasking
64.3 222
76.7 10.4
642
10.5
60.
71. 14.
74.7
71.2 E1.1
79.3 2166
71.1
ContextPred
+20
75.7 +0.7
63.9 +06
60.9 406
+3.
75.8
77.3 +1.0
79.6 +1.
70.9
Supervised
68.3 =0.7
77.0
#0.3
644 =04
62.1 #0.
S22
194
74.4 #12
769=10
70.0
Supervised
Infomax
68.0
77.8
10.3
64.9 :0.7
60,6
71.2
128
81.
77.8 10.9
80.1 =099
72.8
Supervised
EdgePred
66.6222
78.3
+0.3
66.5 =0.3
70,9
78.5
71.5 +08
79.1
72.6
Supervied
AnrMasking
66.5 555
779"+00
63. F6.3
6339
73.
8122-19
77.1 FTT
80.3 -0.9
73.55
Supervised
ContextPred
78.1
10.6
84.5
Table I: Test ROC-AUC performance on molecular prediction benchmarks using different pre-training strategies with GIN. The rightmost column averages the mean of test performance across the 8 datasets. The best result for each dataset and comparable results (i.e., results within one standard deviation from the best result) are bolded. The shaded cells indicate negative transfer, i.e. ROC-AUC of pre-trained model is worse than that of tretteeed model. Notice that node- as well as graph-level pretraining are essential for good performance.
Chemistry
Biology
Non-pu con-prettrained
Pre-trained
Gain
Non-pre-trainet
Pre-trained
Gain
GIN
67.0
74.2
+7.2
64.8 + 1.0
74.2 + 1.5
+9.4
GCN
68.9
72.2
+3.4
63.2 1.0
70.9 + 1.7
+7.7
GraphSAGE 68.3
70.3
+2.0
65.7
1.2
68.5 + 1.5
+2.8
GAT
66.8
60.3
-6.5
68.2 1.1
67.8 + 3.6
-0.4
Table 2: Test ROC-AUC (%) performance of different GNN architectures with and without pre-training. Without pre-training. the less expressive GNNs give slightly better performance than the most expressive GIN because of their smaller model complexity in low data regime. However, with pre-training. the most expressive GIN properly regularized and dominates the other architectures. For results split by chemistry datasets, see Table 4 in Appendix H. Pre-training strategy for chemistry data: Context Prediction + Graph-level supervised pre-training; pre-training strategy for biology data: Attribute Masking Graph-level supervised pre-training.
3-layer GNN encode the context structure. For Attribute Masking shown in Figure (b), we randomly mask 15% of node (for molecular graphs) or edge attributes (for PPI networks) for prediction. As baselines for node-level self-supervised pre-training, we adopt the original Edge Prediction (denoted by EdgePred) (Hamilton et al., 2017a and Deep Graph Infomax (denoted by Infomax) (Veličković et al., 2019) implementations. Further details are provided in Appendix G.
5.3 RESULTS
We report results for molecular property prediction and protein function prediction in Tables 2 and I and Figure 3. Our systematic study suggests the following trends:
Observation (1): Table shows that the most expressive GNN architecture (GIN), when pre-trained, achieves the best performance across domains and datasets. Compared with gains of pre-training achieved by GIN architecture, gains of pre- pre-training training using less expressive GNNs (GCN, GraphSAGE, and GAT) are smaller and can sometimes even be negative (Table 2). This finding confirms previous observations (e.g., Erhan et al. (2010)) that using an expressive model is crucial fully utilize pre-training, and that pre-training expressive power, such as GCN, GraphSAGE, and GAT.
even hurt performance when used on models with limited
Observation (2): As seen from the shaded cells of Table and highlighted region in the middle panel of Figure 3, the strong baseline strategy that performs extensive graph-level multi-task supervised pre-training of GNNs gives surprisingly limited performance gain and yields negative transfer many downstream tasks (2 out of 8 datasets molecular prediction, and out of 40 tasks protein function prediction).
8

---

Published as a conference paper at ICLR 2020
Observation (3): From the upper half of Table 1 and the left panel of Figure that another baseline strategy, which only performs node-level self-supervised pre-training, also gives limited performance improvement and is comparable to the graph-level multi-task supervised pre-training baseline.
Observation (4): From the lower half of Table and the right panel of Figure 3, pre-training strategy of combining graph-level multi-task supervised and node-level self-supervised pre-training avoids negative transfer across downstream datasets and achieves best performance.
that
Observation (5): Furthermore, from Table and the left panel of Figure 3, we see that our strategy gives significantly better predictive performance than the baseline pre-training strategies well as non-pre-traie preettrinned models, achieving state-of-the of-the-art performance.
Specifically, in the chemistry datasets, we see from Table that our Context Prediction + Graph-level multi-task supervised pre-training strategy gives the most promising performance, leading to an increase in average ROC-AUC of 7.2% over non-pre-traie treitrri baseline and 4.2% over graph-level multi-task supervised pre-trained baseline. On the HIV dataset, where number of recent works (Wu al., 2018; et al., 2017; Ishiguro al., 2019) have reported performance on the same scaffold split and using the same protocol, our best pre-trained model (ContextPred Supervised) achieves state-of- the-art performance. In particular, we achieved ROC-AUC score of 79. 79.9%, while best-performing graph models in Wu et al. (2018), Li et al. (2017), and Ishiguro et al. (2019) had ROC-AUC scores of 76.3%, 77.6%, and 76.2%, respectively.
Also, in the biology datasets, which we have built in this work, we see from the left panel of Figure 3 that our Attribute Masking Graph-level multi-task supervised pre-training strategy achieves the best predictive performance compared to other baseline strategies across almost all 40 downstream prediction tasks (the right panel of Figure 3). On average, our strategy improves ROC-AUC by 9.4% over traapee baseline and 5.2% over graph-level multi-task supervised pre-trained baseline, again achieving state-of-the-ee performance.
Observation (6): In the chemistry domain, we also report performance on classic benchmarks (MUTAG, PTC molecule datasets) in Appendix B. However, as mentioned in Section 5.1, the extremely small dataset sizes make these benchmarks unsuitable to compare different methods in statistically reliable way.
Observation (7): Beyond predictive performance improvement, Figure shows that our pre-trained models achieve orders-of ma-gmniude faster training and validation convergence than non-pre-trained models. For example, on the MUV dataset, took hour for the non-pre-traine trained GNN to get 74.9% validation ROC-AUC, while it took only 5 minutes for our pre-trained GNN to get 85.3% validation ROC-AUC, The same trend holds across the downstream datasets we used, as shown in Figure 5 in Appendix I. We emphasize that pre-training is a one-time-effort. Once the model is pre-trained, it can be used for any number of downstream tasks to improve performance with little training time.
As final remark, in preliminary experiments, we performed the Attribute Masking and Context Prediction simultaneously to pre-train GNNs. That approach did not improve performance in our experiments. We leave thorough analysis of the approach for future work.
6 CONCLUSIONS AND FUTURE WORK
We developed a novel strategy for pre-training GNNs. Crucial to the success of our strategy is to consider both node-level and graph-level pre-training in combination with an expressive GNN. This ensures that node embeddings capture local neighborhood semantics that are pooled together to obtain meaningful graph-level representations, which, in turn, are used for downstream tasks. Experiments on multiple datasets, diverse downstream tasks and different GNN architectures show that the new pre-training strategy achieves consistently better diitrr-distribution generalization than non-pre- non-pre-trained trained models.
Our work makes an important step toward transfer learning graphs and addresses the issue of negative transfer observed in prior studies. There are many interesting avenues for future work. For example, further increasing generalization by improving GNN architectures as well as pre-training and fine-tuning approaches, is a fruitful direction. Investigating what pre-trained models have learned would also be useful to aid scientific discovery (Tshitoyan et al., 2019). Finally, it would be interesting

---

Published as a conference paper at ICLR 2020
Pre traiiing stalley
Out
Altii Mhookking Cheep deee---eess appien ann
Grapt-lovel
Node-level
ispecies spli))
rered p Priiiing
Infommx
64.1 1:15
EdgaPrud
65.7 1:33
ContextPred
27.6
AlrMasking
Supervised
69.0 =24
hagatine zaraer
eodting regaihv
Ad Masiing iratem renesiaadion taking screas tukke
Supervised
Infomax
72.8 1:15
o4d o4
Supervised
EdgePred
11.4
taata.
Supervised
ConeexPrrdd
1.0
de Gisph ceoph-iiielleo supervloed pre-traking only
Supervised
Amsaskinn
74.2 11.5
pre-triningg
Figure 3: Test ROC-AUC of protein function prediction using different pre-training strategies with GIN. (Left) Test ROC-AUC scores (%) obtained by different pre-training strategies, where the scores are averaged over the 40 fine-grained prediction tasks. (Middle and right): Scatter plot comparisons of ROC-AUC scores for pair of pre-training strategies on the 40 individual downstream tasks. Each point represents particular individual downstream task. (Middle): There are many individual downstream tasks where graph-level multi-task supervised pre-trained model performs worse than non-pre-tree con-pre-trained model, indicating negative transfer. (Right): When the graph-level multi- task supervised pre-training and Attribute Masking are combined, negative transfer is avoided across downstream tasks. The performance also improves over pure graph-level supervised pre- tre-training.
Chemistry: MUV
Biology: PPI prediction
Non-pre-trained
Random
Training aue
3.8-
Training AVC
0.5
Pre-trained
Graph-level supervised pre-treining Masking
6.7-
0.a
0.6-
Graph-level supervised pre-training only
4.-
Masking
20
62
60
100
10
20
30
40
50
Epoch
Epoch
Figure 4: Training and validation curves of different pre-training strategies on GINs. Solid and dashed lines indicate training and validation curves, respectively.
to apply our methods to other domains, e.g., physics, material science, and structural biology. where many problems are defined over graphs representing interactions of e.g., atoms, particles, and amino acids.
ACKNOWLEDGMENTS
We thank Camilo Ruiz, Rex Ying, Zhenqin Wu, Shantao Li, Srijan Kumar, Hongwei Wang, and Robin Jia for their helpful discussion. W.H is supported by Funai Overseas Scholarship and Masason Foundation Fellowship. J.L is a Chan Zuckerberg Biohub investigator. We gratefully acknowledge the support of DARPA under Nos. FA865018C7888 (ASED). N66001 1924033 (MCS): ARO under (MURI), 1911NF-16-1-017 (DURIP): NSF under Nos. OAC-1835598 (CINES), OAC- 19345994778 (HDR); Stanford Data Science Initiative, Wu Tsai Neurosciences Institute, Chan Zuckerberg Biohub, JD.com, Amazon, Boeing, Docomo, Huawei, Hitachi, Observe, Siemens, UST Global.
Nos. W99INF-16-1-02
The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or rec- ommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of DARPA, NIH, ARO, or the U.S. Government.
The Pande Group acknowledges the generous support of Dr. Anders G. Frøseth and Mr. Christian Sundt for work machine learning. The Pande Group is broadly supported by grants from the NIH (R01 GM062868 and U19 A1109662) as well as gift funds and contributions from Folding@home donors.
10

---

Published as a conference paper at ICLR 2020
V.S.P is consultant & SAB member of Schrodinger, LLC and Globavir, sits the Board of Directors of Apeel Sciences, Asimov, BioAge Labs, Citizen, Devoted Health, Freenome, Insitro, Omada Health, PatientPing, and is General Partner at Andreessen Horowitz.
REFERENCES
AACT. AACT database, Jan 2017. URL http::/www..cci--clnncalllllllll
Michael Ashburner, Catherine A Ball, Judith A Blake, David Botstein, Heather Butler, J Michael Cherry, Allan Davis, Kara Dolinski, Selina Dwight, Janan Eppig, al. Gene ontology: tool for the unification of biology. Nature Genetics, 25(1):25, 2000.
Yunsheng Bai, Hao Ding, Yang Qiao, Agustin Marinovic, Ken Gu, Ting Chen, Yizhou Sun, and Wei Wang. Unsupervised inductive whole- grapll embedding by preserving graph proximity. In International Joint Conference Artificial Intelligence (IJCAI), 2019.
Guy W. Bemis and Mark A. Mureko. The properties of known drugs. molecular frameworks. Journal of Medicinal Chemistry, 39(15):2887-288) 1996. doi: 10.1021/jm6022288. PMID: 8709122.
Andrew Bradley. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 301): 1997.
Suman K Chakravarti. Distributed representation of chemical fragments. ACS Omega, 3(3):2825- 2836, 2018.
Bin Chen, Robert P. Sheridan, Viktor Hornak, and Johannes H. Voigt. Comparison of random forest and pipeline pilot naïve bayes prospective QSAR predictions. Journal of Chemical Information and Modeling, 52(3):792-80) 803, 2012.
doi: 10. 00.102/cc200615. PMID: 22360769.
Travers Ching. Daniel S Himmelstein, Brett K Beaulieu-Jones, Alexandr A Kalinin, Brian T Do, Gregory P Way, Enrico Ferrero, Paul-Michael Agapow, Michael Zietz, Michael Hoffman, et al. Opportunities and obstacles for deep learning in biology and medicine. Journal of The Royal Society Interface, 15(141):2017038) 2018.
Gene Ontology Consortium. The gene ontology resource: 20 years and still going strong. Nucleic Acids Research, 47DD1):D330-D333 2018.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding- In Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International Conference on Machine Learning (ICML), pp. 647-655, 2014.
Brendan L Douglas. The Weisfeiler-Lehmar method and graph isomorphism testing, arXiv preprint 1rX 2011.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoin Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, (1(eeb):25-660) 660, 2010.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with Pytorch Geometric, In International Conference Learning Representations (ICLR), RLGM Workshop, 2019.
Fleanor Gardiner, John D Holliday, Caroline O' OoDow,, and Peter Willett. Effectiveness of 2d fingerprints for scaffold hopping. Future Medicinal Chemistry, 404 2011.
Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, et al. ChEMBL: a large-scale bioactivity database for drug discovery. Nucleic Acids Research, 40(D1):D1100-D) 107, 2011.
11

---

Published as a conference paper at ICLR 2020
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning (ICML), pp. 1273-1272, 2017.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 580-587, 2014.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pp. 855-864. ACM, 2016.
William I. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances Neural Information Processing Systems (NeurIPS), pp. 1025-1035, 20174
William I. Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. IEEE Data Engineering Bulletin, 40(3):52-74, 2017b.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In International Conference on Machine Learning (ICML), 2019.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), pp. 448-456, 2015.
Katsuhiko Ishiguro, Shin-ichi Maeda, and Masanori Koyama. Graph warp module: An auxiliary module for boosting the power of graph neural networks. arXiv preprint arXiv:1902.000 000000000000,...... 2019.
Sabrina Jaeger, Simone Fulle, and Samo Turk. Mol2vec: unsupervised machine learning approach with chemical intuition. Journal of chemical information and modeling, 58(1):27-35, 2018.
Kristian Kersting, Nils M Kriege, Christopher Morris, Petra Mutzel, and Marion Neumann. Bench- mark data sets for graph kernels, 2016. URL http://graphhkrnnls. cs. tu-dortmund. de, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Thomas N Kipf and Max Welling, Variational graph auto-encoders. In Advances in Neural Information Processing Systems (NeurIPS), Bayesian Deep Learning Workshop, 2016.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
DV Klopfenstein, Liangsheng Zhang, Brent S Pedersen, Fidel Ramírez, Alex Warwick Vesztrocy, Aurélien Naldi, Christopher J Mungall, Jeffrey M Yunes, Olga Botvinnik, Mark Weigel, et al. Goatools: A Python library for gene ontology analyses. Scientific Reports, 8(1):10872, 2018.
Michael Kuhn, Ivica Letunic, Lars Juhl Jensen, and Peer Bork. The sider database of drugs and side effects. Nucleic Acids Research, 44D11):D0775-D00) 2015.
Greg Landrum et al. RDKit: Open-source cheminformatics, 2006.
Junying Li, Deng Cai, and Xiaofei He. Learning graph-level representation for drug discovery. arXiv preprint arXiv: 1709. 037411, 2017.
Ines Filipa Martins, Ana L Teixeira, Luis Pinheiro, and Andre Falcao. A bayesian approach to in silico blood-brain barrier penetration modeling- Journal of Chemical Information and Modeling, 5266):168611697, 2012
Andreas Mayr, Günter Klambauer, Thomas Unterthiner, Marvin Steijaert, Jörg K Wegner, Hugo Ceulemans, Djork-Amé Clevert, and Sepp Hochreiter. Large-scale comparison of machine learning methods for drug target prediction ChEMBL. Chemical Science, 9424):54415545 2018.
12

---

Published as a conference paper at ICLR 2020
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems (NeurIPS), pp. 3111-3119, 2013.
Annamalai Narayanan, Mahinthan Chandramohan, Lihui Chen, Yang Liu, and Santhoshkumar Saminathan. subgraph2vec: Learnnng distributed representations of rooted sub- sub-graphs from large graphs. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), MI.G workshop, 2016.
Nicolò Navarin, Dinh V Tran, and Alessandro Sperduti. Pre-training graph neural networks with kernels. arXiv preprint arXiv: 1.06930, 2018.
Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural networks for graphs. In International Conference on Machine Learning (ICML), pp. 2014-2023, 2016.
Paul A. Novick, Oscar F. Ortiz, Jared Poelman, Amir Y. Abdulhay, and Vijay S. Pande. SWEETLEAD: in silico database of approved drugs, regulated chemicals, and herbal isolates for computer- aided drug discovery. PLOS ONE, 8(11), 11 2013. doi: 1371/ 10w.1/3ur//aaaaa/ pone. 0079568.
URL
https:: / /oi..oi..////ooo www.orggg/..... 1tt./11.1111111
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345-13
2009.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito,,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In Advances in Neural Information Processing Systems (NeurIPS), Workshop, 2017.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social represen- tations. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pp. 701-710. ACM, 2014.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2018
Bharath Ramsundar, Peter Eastman, Patrick Walters, and Vijay Pande. Deep Learn- ing for the Life Sciences. O O'Reilly Media, 2019. https: /wwww aww.on.aaoo..................... com/
Ann M. Richard, Richard S. Judson, Keith A. Houck, Christopher M. Grulke, Patra Volarath, Inthirany Thillaindaarjjah, Chihae Yang, James Rathman, Matthew T. Martin, John F. Wambaugh, Thomas B. Knudsen, Jayaram Kancherla, Kamel Mansouri, Grace Patlewicz, Antony J. Williams, Stephen B. Liute, Kevin M. Crofton, and Russell S. Thomas. Toxcast chemical landscape: Paving the road to 21st century toxicology. Chemical Research in Toxicology, 1225-1225-1 2016.
David Rogers and Mathew Hahn. fion and Modeling, 50(5):742-754, 2010. doi: 00.1021/ci100055
fingerprints. Journal of Chemical Informa-
Michael T Rosenstein, Zvika Marx, Leslie Pack Kaelbling, and Thomas G Dietterich. To transfer or not to transfer. In Advances in Neural Information Processing Systems (NeurIPS), Workshop on fransfer learning, volume 898, pp. 1-4, 2005.
Herbert Rubenstein and John B Goodenough. Contextual correlates of synonymy. Communications of the ACM, 8(10):277633, 1965.
Robert P. Sheridan. Time-split cross-validatee as a method for estimating the goodness of prospective prediction. Journal of Chemical Information and Modeling, 53(4).783-790, 2013. doi: 10.1021/ ci400084k. PMID: 23521722.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(9):1929-1958 2014.
13

---

Published as a conference paper at ICLR 2020
Teague Sterling and John J. Irwin. Zine 15 ligand discovery for everyone. Journal of Chemical Information and Modeling, 55(11):2324 23374, 2015. doi: 10.1022//cs,jiimm5500033 PMID: 26479676.
Govindan Subramanian, Bharath Ramsundar, Vijay Pande, and Rajiah Aldrin Denny. Computational modeling of 3-secretase (BACE-I) 1) inhibitors using ligand based approaches. Journal of Chemical Information and Modeling, 56( 1931))):93699990 2016.
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Large-scale information network embedding. In Proceedings of the International World Wide Web Conference
(WWW), pp. 1067-1077, 2015.
Tox21. Tox21 data challenge 2014, 2014. URL httpss//ttripcd..iiit chall challenge/.
Vahe Tshitoyan, John Dagdelen, Leigh Weston, Alexander Dunn, Ziqin Rong, Olga Kononova, Kristin A Persson, Gerbrand Ceder, and Anubhav Jain. Unsupervised word embeddings capture latent knowledge from materials science literature. Nature, 571(7763):95, 2019.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations (ICLR), 2018.
Petar Veličković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In International Conference on Learning Representations (ICLR), 2019.
Jingshu Wang, Divyansh Agarwal, Mo Huang, Gang Hu, Zilu Zhou, Chengzhong Ye, and Nancy R Zhang. Data denoising with transfer learning in single-cell transcriptomics. Nature Methods, 16 (9):875-877) 2019.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. MoleculeNet: benchmark for molecular machine learning. Chemical science, 9():5133-530, 2018.
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning graphs with jumping knowledge networks. In International Conference on Machine Leurning (ICML), pp. 5453-5462, 2018.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? International Conference on Learning Representations (ICLR), 2019.
Yuting Xu, Junshui Ma, Andy Liaw, Robert P Sheridan, and Vladimir Svetnik. Demystifying multi- task deep neural networks for quantitative structure-activity relationships. Journal of Chemical Information and Modeling, 57(10) -504,, 2017.
Rendong Yang, Yun Bai, Zhaohui Qin, and Tianwei Yu. EgoNet: identification of human disease ego-network modules. BMC Genomics, 15(1):314, 2014.
Rex Ying. Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pp. 974-983, 2018a.
Rex Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. In Advances in Neural Information Processing Systems (NeurIPS), 2018b
Jiaxuan You, Rex Ying, and Jure Leskovec. Position aware graph neural networks. In International Conference Machine Learning (ICML), 2019.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision (ECCV), pp. 818-833, Springer, 2014.
14

---

Published as a conference paper at ICLR 2020
Muhan Zhang. Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In AAAI Conference on Artificial Intelligence, pp. 1138-4445. 2018.
Quan Zhou, Peizhe Tang, Shenxiu Liu, Jinbo Pan, Qimin Yan, and Shou-Cheng Zhang. Learning atoms for materials discovery. Proceedings of the National Academy of Sciences, 115(28):E641 1- E6417, 2018.
Marinka Zitnik, Rok Sosic, and Jure Leskovec. Prioritizing network communities. Nature Communi- cations, 9()):2544, 2018.
Marinka Zitnik, Rok Sosič, Marcus W. Feldman, and Jure Leskovec. Evolution of resilience in protein interactomes across the tree of life. Proceedings of the National Academy of Sciences, 116(10):4426 4433, 2019. ISSN 0027-8424. doi: 10.1073/pnas. 1818013116. URL https:
pwww.ppaa. wwwg/ooggeeeeeeooooooooooooooooe argg/content/tien /11 /twpe//nnn/////////////
15

---

Published as a conference paper at ICLR 2020
A DETAILS OF GNN ARCHITECTURES
Here we describe GNN architectures used in our molecular property and protein function prediction experiments. For both domains we use the GIN architecture (Xu et al., 2019) with some minor modifications to include edge features, as well as center node information in the protein ego-networks.
As our primary goal is systematically compare our pre-training strategy the strong baseline strategies, we fix all of these hyper-parameters in our experiments and focus on relative improvement directly caused by the difference in pre-training strategies.
Molecular property prediction. In molecular property prediction, the raw node features and edge features are both 2-dimensional categorical vectors (see Appendix C for details), denoted (i.,1, (v.2) and : for node and edge respectively. Note that we also introduce unique categories to indicate masked node/edges as well as self-loop edges. As input features to GNNs, we first embed the categorical vectors by
h EmbNode; (fx,1) + EmbNodoc((u..))
K h (k) EmbEdge] (Je.) + EmbEdge, (k) (je,2) for k= k 1-
where EmbNode (.), EmbNodee2.)). EmbEdgee (8) (-), and EmbNode (x) represent embedding operations that map integer indices to d-dimensional real vectors, and l: represents the index of GNN layers. At the k-th layer, GNNs update node representations by
ReLU
MLP(3)
Σ
E
h h(--1)
(A.1)
HEA(O)
where N(v) a of nodes adjacent to u, and (v, v) represents the self-loop edge. Note that for the final layer, i.e., h = K, we removed the ReLU from Eq. (A.1) so that h can take negative values. This crucial for pre-training methods based on the dot product, e.g., Context Prediction and Edge Prediction, as otherwise, the dot product between two vectors would be always positive.
The graph-level representation lGg is obtained by averaging the node embeddings at the final layer, i.e.,
hg MEAN((M( G
(A.2)
The label prediction is made by linear model on top hG.
In our experiments, we set the embedding dimension to 300. For MLPs in Eq. (A.1), we use the ReLU activation with 600 hidden units. We apply batch normalization (Ioffe & Szegedy, 2015) right before the ReLU in Eq. (A.1) and apply dropout (Srivastava et al., 2014) to (.()) at all the layers except the input layer.
Protein function prediction. The GNN architecture used for protein function prediction is similar to the one used for molecular property prediction except for few differences. First, the raw input node features are uniform (denoted as X here) and second, the raw input edge features are binary vectors (see Appendix D for the detail), which we denote as c. E {0, 10 As input features GNNs, we first embed the raw features by
(((0) X
h (k) Wc, c, + for à = 0,1,...,
where W Raxdo and & Rd are learnable parameters, and (1 ,)(()) E Rd. At each layer, GNNs update node representations by
h(t) ReLU
MLP(2)
CONCAT
(kk-))
Σ
(((--1)
WEN())U(E}
(A.3)
16

---

Published as a conference paper at ICLR 2020
Dataset
MUTAG 188
PTC 344
# Molecules
# Binary prediction tasks
1
Prvious results
Cross validation split
WL substree (Douglas, 2011)
90,4 + 5.7
59.9 + 4.3
Patchysan (Niepert et al., 2016)
92.6 + 4.2
60.0 + 4.8
GIN (Xu et al., 2019)
89.4 = 5.6
64.6 + 7.0
Pre-training strategy
Cross validation split
Graph-level
Node-level
89.3 7.4
62.4 6.3
Infomax
89.8 -
65.9 +
EdgePred
91.977
66.5 5.7
Masking
9
6..4
ContextPred
9.1
68.3
Supervised
9..9
64.7 7.9
Supervised
Infomax
90.9
63.0
Supervised
EdgePred
91.9
63,5 + 8.2
Supervised
"Masking"
90.3
60.9 -9.1
Supervised
ContextPred
92.5 5.0
66.5 5.2
Table 3: 10-fold cross validation accuracy (% on classic graph classification benchmarks using different pre-fraining strategies with GIN. All the previous results excerpted from Xu al. (2019).
where CONCAT(..)) takes two vectors as input and concatenates them. Since the downstream task ego-network classification, we use the embedding the center node Ucenter together with embedding of the entire ego-network. More specifically, we obtain graph-level representation lig by
hg CONCAT (MEAN(()(K) | E G)), (K) Vicettrr
(A.4)
Other GNN architectures. For GCN, GraphSAGE, and GAT, we adopt the implementation in the Pytorch Geometric library (Fey & Lenssen, 2019), where we set the number of GAT attention heads to be 2. The dimensionality of node embeddings as well as the number of GNN layers are kept the same as GIN. These models do originally handle edge features. We incorporate edge features into these models similarly to how we do it for the GIN; we add edge embeddings into node embeddings, and perform the GNN cessage-passingg on the obtained node embeddings.
B EXPERIMENTS ON CLASSIC GRAPH CLASSIFICATION BENCHMARKS
In Table we report our experiments the commonly-used classic graph classification benchmarks (Kersting et al., 2016). Among the datasets Xu et al. (2019) used, MUTAG, PTC, and NCII are molecule datasets for binary classification. Out of these three, we excluded the NCII dataset, because it misses edge information (i.e., bond type) and therefore, we cannot recover the original molecule information, which is necessary construct our input representations described in Appendix C.
For fair comparison, we used exactly the same evaluation protocol as Xu et al. (2019), i.e., report 10-fold valssvvaiittion accuracy. All the hyper-parameters in our experiments are kept the same in the main experiments except that we additionally tuned dropout rate from {0, 0.2, 0.5) and the batch size from {8, 64) at the fine-tuning stage.
While the pre-trained GNNs (especially those with Context Prediction) give competent performance, all the accuracies (including all the previous methods) are within standard deviation with each other, making it hard to reliably compare different methods. As Xu et al. (2019) has pointed out, this is due to the extremely small dataset size; validation set at each fold only contains around 19 to 35 molecules for MUTAG and PTC, respectively. Given these results, argue that is necessary to larger datasets to make reliable comparison, so we mainly focus on MoleculeNet (Wu et al., 2018) in this work.
17

---

Published as a conference paper at ICLR 2020
C DETAILS OF MOLECULAR DATASETS
Input graph representation. For simplicity, we use minimal set of node and bond features that unambiguously describe the two-dimensional structure of molecules, We use RDKit (Landrum et al., 2006) to obtain these features.
Node features:
Atom number: [1, 118]
Chirality tag: (unspecified,, tetrahedral ow, tetrahedral cow, other)
Edge features:
Bond type: (single, double, triple, aromatic)
Bond direction: {-, endupright, enddownright)
Downstream task datasets. 8 binary graph classification datasets from Moleculenct (Wu al., 2018) are used evaluate model performance.
BBBP. Blood-brain barrier penetration (membrane permeability) (Martins et al., 2012).
Tox21. Toxicity data on 12 biological targets, including nuclear receptors and stress response pathways (Tox21).
ToxCast. Toxicology measurements based on over 600 in vitro high-throughput screenings (Richard et al., 2016).
SIDER. Database of marketed drugs and adverse drug reactions (ADR), grouped into 27 system organ classes (Kuhn et al., 2015).
Clin'oxx Qualitative data classifying drugs approved by the FDA and those that have failed clinical trials for toxicity reasons (Novick et al., 2013; AACT).
MUV. Subset of PubChem BioAssay by applying refined nearest neighbor analysis, designed for validation of virtual screening techniques (Gardiner al., 2011).
HIV. Experimentally measured abilities inhibit HIV replication (?).
BACE. Qualitative binding results for of inhibitors of human 8-secretase (Subrama- nian et al., 2016).
D DETAILS OF PROTEIN DATASETS
Input graph representation. The protein subgraphs only have edge features.
Edge features:
Neighbourhood: (True, False)
Fusion: (True, False]
Co-occurrence: (True, False)
Co-expression: (True, False)
Experiment: (rue, False)
Database: (Tuee, False]
Text: (True, False)
These edge features indicate whether particular type of relationship exists between pair of proteins:
Neighbourhood: if pair of genes consistently observed in each other's genome neigh- bourhood
Fusion: pair of proteins have their respective orthologs fused into a single protein-coding gene in another organism
Co-occurrence: if a pair of proteins tend to be observed either as present or absent in the same subset of organisms
Co-expression: if a pair of proteins share similar expression patterns
18

---

Published as a conference paper at ICLR 2020
Experiment: if a pair of proteins are experimentally observed to physically interact with each other
if a pair of proteins belong to the same pathway, based on assessments by a human curator
Database:
Text mining: if a pair of proteins are mentioned together in PubMed abstracts
Datasets. A dataset containing protein subgraphs from 50 species is used (Zitnik et al., 2019). The original PPI networks do not have node attributes, but contain edge attributes that correspond the degree of confidence for 7 different types of peltein-protein relationships. The edge weights range from O, which indicates no evidence for the specific relationship, to 1000, which indicates the highest confidence. The weighted edges of the PPI networks are thresholded such that the distribution of edge types across the 50 PPI networks are uniform. Then, for every node in the PPI networks, subgraphs centered each node were generated by: (1) performing breadth first search to select the subgraph nodes, with a search depth limit of 2 and a maximum number of 10 neighbors randomly expanded per node, (2) including the selected subgraph nodes and all the edges between those nodes to form the resulting subgraph.
The entire dataset contains 394,925 protein subgraphs derived from 50 species. Out of these 50 species, 8 species (arabidopsis, celegans, ecoli, fly, human, mouse, yeast, zebrafish) have proteins with GO protein annotations. The dataset contains 88,000 protein subgraphs from these species, of which 57,448 proteins have at least one positive coarse- -erainggidddd GO protein annotation and 22,876 proteins have at least one positive fine-gran fine-grained GO protein annotation. For the self self-supervised pre-training dataset, we use all 394,925 protein subgraphs.
We define fine- graineg protein functions as Gene Ontology (GO) annotations that are leaves in the GO hierarchy, and define coarse-gres maase-grained protein functions as GO annotations that are the immediate parents of leaves (Ashburner et al., 2000; Consortium, 2018). For example, a fine-grained protein function is "Factor XII activation", while coarse-grained function is "positive regulation of protein". The former is a specific type of the latter, and is much harder to derive experimentally. The GO hierarchy information is obtained using GOATOOLS (Klopfenstein et al., 2018). The supervised pre-training dataset and the downstream evaluation dataset are derived from the 8 labeled species, as described in Appendix E. The 40-th most common, fine- ane-grained protein label only has 121 positively annotated proteins, while the 40-th most common coarse-graine protein label has 9386 positively annotated proteins. This illustrates the extreme label scarcity of our downstream tasks.
For supervised pre-training, we combine the train, validation, and prior sets described previously, with the 5,000 most common coarse-grained protein function annotations as binary labels. For our downstream task, we predict the 40 most common fine-gre tne-ggained protein function annotations, to ensure that each protein function has least 10 positive labels in our test set.
E DETAILS OF DATASET SPLITTING
For molecular prediction tasks, following Ramsundar al. (2019), we cluster molecules by scaffold (molecular graph substructure) (Bemis & Murcko, 1996), and recombine the clusters by placing the most common scaffolds in the training set, producing validation and test sets that contain structurally different molecules. Prior work has shown that this scaffold split provides a more realistic estimate of model performance in prospective evaluation compared random split (Chen et al.. 2012; Sheridan, 2013). The split for train/validation/tesl sets is 80%: 109:100:000
In the PPI network, species split simulates a scenario where we have only high-level coarse- grained knowledge on a subset of proteins (prior set) in a species of interest (human in our experiments). and want to predict fine- fieinnd biological functions for the rest of the proteins in that species (test set). For species split, we use 50% of the protein subgraphs from human as test set, and 50% prior set containing only coarse- coarse-grained protein annotations. The protein subgraphs from 7 other labelled species (arabidopsis, celegans, ecoli, fly, mouse, yeast, zebrafish) are used as train and validation sets, which are split 85%: : 15%. The effective split ratio for the train/ mnin/validation/ 12%: 9.5% 9.5%.
palidatiioo/rioorttsss prior//estttttttoooee sets is 69%
19

---

Published as a conference paper at ICLR 2020
F TIME COMPLEXITY OF PRE-TRAINING
Here we analyze the time complexity for processing graphs in Attribute Masking and Context Prediction. First, the time complexity for Attribute Masking is linear with respect to the number of edges/nodes as it only involves sampling nodes/edges to be masked. Second, the time complexity for Context Prediction is again linear with respect to the number of edges/nodes, because it involves sampling center node per graph plus extracting K -hop neighborhood and context graph. Extracting neighborhoodd/conext graphs is performed by the breadth-fing search, which takes at most linear time with respect to the number of edges in the graph. In summary, the time complexity for both of our pre-training methods are at most linear with respect to the number of edges, which as efficient message-parn message-passing computation in GNNs, and thus, is efficient as the ordinary supervised learning using GNNs. Also, there is almost no memory overhead as we transform data (e.g., mask input node/edge features, sample the context graphs) on-the-fly.
G FURTHER DETAILS OF THE EXPERIMENTAL, SETUP
Optimization. All models are trained with Adam optimizer (Kingma & Ba, 2015) with a learning rate of 0.001. We use Pytorch (Paszke et al., 2017) and Pytorch Geometric (Fey & Lenssen, 2019) for all of our implementation. We run all pre-training methods for 100 epochs. For self-supervised pre-training, we use batch size of 256, while for supervised pre-training, we use a batch size of 32 with dropout rate of 20%.
Fine-tuning, After pre-training, we follow the procedure in Section 3.3 to fine-tune the models on the training sets of the downstream datasets. We use a batch size of 32 and dropout rate of 50%. Datasets with multiple prediction tasks are fit jointly. On the molecular property prediction datasets, we train models for 100 epochs, while on the protein function prediction dataset (with the 40 binary prediction tasks), we train models for 50 epochs.
Evaluation. We evaluate test performance on downstream tasks using ROC-AUC (Bradley, 1997) with the validation early stopping protocol, i.e., test ROC-AUC at the best validation epoch is reported. For datasets with multiple prediction tasks, we take the average ROC-AUC across all their tasks. The downstream experiments are run with 10 random seeds, and we report mean ROC-AUC and standard deviation.
Computation time for pre-training. The computation time for the two stages of our pre-training is reported below. Chemistry: Self-supervised pre-training takes about 24 hours, while supervised pre-training takes about 11 hours. Biology: Self-super pre-training takes about 3.8 hours, while supervised pre-training takes about 2.5 hours.
H COMPARISON OF PRE-TRAINING WITH DIFFERENT GNN ARCHITECTURES
Table 4 shows the detailed comparison of different GNN architectures on the chemistry datasets. We that the most expressive GIN architectures benefit most from pre-training compared to the other less expressive models.
I ADDITIONAL TRAINING AND VALIDATION CURVES
Training and validation curves. In Figure 5, we plot training and validation curves for all the datasets used in the molecular property prediction experiments.
Additional scatter plot comparisons of ROC-AUCs. In Figure 6, we compare our Context Predic- tion + graph-level supervised pre-training with non-pre-trained model and a graph-level supervised pre-trained model. We see from the left plot that the combined strategy again completely avoids negative transfer across all the 40 downstream tasks. Furthermore, we see from the right plot that ad- ditionally adding our nove-level Context Prediction pre-training almost always improves ROC-AUC scores of supervised pre-trainee trained models across the 40 downstream tasks.
20

---

Published as a conference paper at ICLR 2020
Dataset
Tox21 12
Tox
SIDER
Clin Tox
MOV
11V
BACE
Average
Molecules
1427
93087
41127
1513
Binary prediction tasks
617
27
17
Configuration
dustri-diittibuution prediction (scallold
Architscture
Pre-train?
GIN
No
65.8 1:..5
74.0 10.8
63.4
10.6
57.3 11.6
58.0 =4.4
71.8
225
75.3 #1.9
70.1 15.4
67.0
GIN
Yes
68.7 11.3
78.1 10.6
65.7 10.6
62.7 10.8
72.6 1.5
81.3 1211
79.9 10.7
84.5 10.7
74.2
GCN
No
64.9113.0
74.910.8
63,31:0.
60.021.0
65.8:445
73.2=1.4
75.7=1.1
73.6=3.0
68.9
OCN
Yes
70.6+:1.6
75.8+0.3
65.3=0011
62.4+0.5
63.6=1.7
79.4=13
78.2=066
82.3=3,4
72.2
GmphSAGE
No
74,7+00.7
59.2444
72.7-14
AGG
Yes
63.912.1
76.8.10.3
64.91.022
60.71.0.5
60.71200
78.41:2
76.2 11
80.7 0.9
70.3
GAT
Yes
59.440.55
68.1
59.34-0.7
56.0+:0.5
47.6
65.4-008
62.5-1.6
64.3-11
60.3
Table 4: Test ROC-AUC (%) performance molecular prediction benchmarks with different GNN architectures. The rightmost column averages the mean of test performance across the datasets. For pre-training, applied Context Prediction graph-level supervised pre-training.
BBBP
Tox21
Non-pre-trained
intaalzztion
8.5-
0.
Pre-trained
Trawing ANC
2.3
3.8
Art buta Masking - Graph- loval suparvised pre-training
9.-
Graph-level supervised
pre-Itaining
Masking
40
60
100
20
60
100
Epoch
Epoch
ToxCast
SIDER
ClinTox
3.2
3.9
0.0
0.8
0.2
20
40
62
120
69
60
300
Epoch
Epoch
Epach
MUV
HIV
BACE
0.5
0.0
0.-
0.--
0.7
6.7
20
40
Epoch
42
02
120
Epoch
100
Epoch
100
Figure 5: Training and validation curves of different pre-training strategies. The solid and dashed lines indicate the training and validation curves, respectively.
21

---

Published as a conference paper at ICLR 2020
ContextPred Graph-level supervised pre-training
Context Prediction helps avoiding negative transfer across tasks.
Context Prediction improves generalization over pure graph-level pre- training across tasks.
02-
0.2
6.2
0.4
0.6
0.8
0.0
0a Graph-level supervised pre-training only
6.2
0.4
No pre-training
Figure 6: Scatter plot comparisons of ROC-AUC scores of our Context Prediction + graph-level supervised pre-training strategy versus the two baseline strategies mion-pre-trained and graph-level supervised pre-trained) on the 40 individual downstream tasks of predicting different fine-grained protein function labels.
22


Reviews:
[{"review_id": "SkxAD0ECtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes new pre-training strategies for GNN with both a node-level and a graph-level pretraining. For the node-level pretraining, the goal is to map nodes with similar surrounding structures to nearby context (similarly to word2vec). The main problem is that directly predicting the context is intractable because of combinatorial explosion. The main idea is then to use an additional GNN to encode the context and to learn simultaneously the main GNN and the context GNN via negative sampling. Another method used is attribute masking where some masked node and edge attributes need to be predicted by the GNN. For graph-level pretraining, some general graph properties need to be predicted by the graph.\nExperiments are conducted on datasets in the chemistry domain and the biology domain showing the benefit of the pre-training.\n\nThe paper addresses an important and timely problem. It is a pity that the code is not provided. In particular, the node-level pretraining described in section 3.1.1. seems rather complicated to implement as a context graph needs to be computed for each node in the graph. In particular I do not think the satement 'all the pre-training methods are at most linear with respect to the number of edges' made in appendix F is correct.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkxPCGTX9B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce strategies for pre-training graph neural networks. Pre-training is done at the node level as well as at the graph level. They evaluate their approaches on two domains, biology and chemistry on a number of downstream tasks. They find that not all pre-training strategies work well and can in fact lead to negative transfer. However, they find that pre-training in general helps over non pre-training.\n\nOverall, this paper was well written with useful illustrations and clear motivations. The authors evaluate their models over a number of datasets. Experimental construction and analysis also seems sound.\n\nI would have liked to see a bit more analysis as to why some pre-training strategies work over others. However, the authors mention that this is in their planned future work.\n\nAlso, in figure 4, the authors mention that their pre-trained models tend to converge faster. However, this does not take into account the time already spent on pre-training. Perhaps the authors can include some results as to the total time taken as well as amortized total time over a number of different downstream tasks.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJeDySijKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes pre-training strategies (PT) for graph neural networks (GNN) from both node and graph levels. Two new large-scale pre-training datasets are created and extensive experiments are conducted to demonstrate the benefits of PT upon different GNN architectures. I am relative positive for this work. Detail review of different aspects and questions are as follows. \n\nNovelty: As far as I know, this work is among the earliest works to think about GNN pre-training. The most similar paper at the same period is [Z Hu, arXiv:1905.13728]. I read both papers and found they have similar idea about PT although they have different designs. This paper leverages graph structure (e.g., context neighbors) and supervised labels/attributes (e.g., node attributes, graph labels) for PT. These strategies are not surprising for me and the novelty is incremental. \n\nExperiment: The experiments are overall good. The authors created two new large scale pre-training graph datasets. Experimental results of different GNN architectures w/o different PT for different tasks are provided. Comparing to non-pretraining GNN, the improvements are significant for most cases. \n\nWriting: The writing is good and easy to follow. \n\nQuestions: I would like to see more discussion about difference between this work and [Z Hu, arXiv:1905.13728]. Comparing to the other work, what are strengths of this work? In addition, have the authors compared the performances of their work and [Z Hu, arXiv:1905.13728] using the same data? ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers are consistently positive on this paper. Thus an accept is recommended.
Decision: Accept (Spotlight)