Self-supervised Graph-level Representation Learning with Local and Global Structure
Authors: Minghao Xu
Hang Wang
Bingbing Ni
Hongyu Guo
Jian Tang
Venue:  (2021)
Keywords: Self-supervised Representation Learning
Graph Representation Learning
Hierarchical Semantic Learning

Abstract: This paper focuses on unsupervised/self-supervised whole-graph representation learning, which is critical in many tasks including drug and material discovery. Current methods can effectively model the local structure between different graph instances, but they fail to discover the global semantic structure of the entire dataset. In this work, we propose a unified framework called Local-instance and Global-semantic Learning (GraphLoG) for self-supervised whole-graph representation learning. Specifically, besides preserving the local instance-level structure, GraphLoG leverages a nonparametric strategy to learn hierarchical prototypes of the data. These prototypes capture the semantic clusters in the latent space, and the number of prototypes can automatically adapt to different feature distributions. We evaluate GraphLoG by pre-training it on massive unlabeled graphs followed by fine-tuning on downstream tasks. Extensive experiments on both chemical and biological benchmark datasets demonstrate the effectiveness of our approach. 

Under review as a conference paper at ICLR 2021
SELF-SUPERVISED GRAPH-LEVEL REPRESENTATION LEARNING WITH LOCAL AND GLOBAL STRUCTURE
Anonymous authors Paper under double-blind review
ABSTRACT
This paper focuses learning, which is critical in many tasks including drug and material discovery. Current methods can effectively model the local structure between different graph instances, but they fail to discover the global semantic structure of the entire dataset. this work, we propose a unified framework called Local-instance and Global-semantie Learning (GraphLoG) for self-supervised whole-gree mhole-graph represen- tation learning. Specifically, besides preserving the local instance-level structure, GraphLoG leverages nonparametric strategy to learn hierarchical prototypes of the data. These prototypes capture the semantic clusters in the latent space, and the number of prototypes can automatically adapt to different feature distributions. We evaluate GraphLoG by pre-training it on massive unlabeled graphs followed by fine-tuning downstream tasks. Extensive experiments on both chemical and biological benchmark datasets demonstrate the effectiveness of our approach.
seterrviseedsslf-suurrriiiie whole-graph representation
INTRODUCTION
Learning informative representations of whole graphs is a fundamental problem in a variety of do- mains and tasks, such as molecule properties prediction in drug and material discovery (Gilmer al., 2017: Wu et al., 2018), protein function forecast in biological networks (Alvarez & Yan, 2012; Jiang al., 2017), and predicting the properties of circuits in circuit design (Zhang al., 2019). Recently, Graph Neural Networks (GNNs) have attracted surge of interest and showed the effectiveness in learning graph representations. These methods are usually trained supervised fashion, which requires large number of labeled data. Nevertheless, in many scientific domains, labeled data very limited and expensive to obtain. Therefore, it is becoming increasingly important to learn the representations of graphs in unsupervised or self-supervised fashion.
Self-supervised learning has recently achieved profound success for both natural language process- ing, e.g. GPT (Radford al., 2018) and BERT (Devlin et al., 2019), and image understanding, e.g. MoCo (He et al., 2019) and SimCLR (Chen et al., 2020). However, how to effectively learn the rep- resentations of graphs in self-supervised way is still open problem. Intuitively, desirable graph representation should be able to preserve the docal-instane structure, so that similar graphs are em- bedded close to each other and dissimilar ones stay far apart. In addition, the representations of set of graphs should also reflect the flobal-semantte structure of the data, so that the graphs with similar semantic properties are compactly embedded, which benefits various downstream tasks, e.g. graph classification or regression. Such structure can be sufficiently captured by semantic clusters (Caron et al., 20:8;11 al., 2019), especially in a hierarchical fashion (Li et al., 2020).
There are some recent works that learn graph representation in a self-superviséd manner, such as local-glohal mutual information maximization (Velickovic al., 2019; Sun al., 2019), structural- similarity/context prediction (Navarin et al., 2018; Hu et al.. 2019; You et al.. 2020) and contrastive multi-view learning (Hassani Ahmadi, 2020). However, all these methods are capable of modeling only the local structure between different graph instances but fail to discover the global-semantic structure. To address this shortcoming, we are seeking for an approach that is sufficient to model both the local and global structure of given set of graphs.
To attain this goal, we propose a Local-instance and Global-semantic Learning (GraphLoG) frame- work for self self-supervieed graph representation learning. In specific, for preserving the local similarity

---

Under review as a conference paper at ICLR 2021
x
X
graph embbolding
x
* layer prototype
layer prototype
geaph
Hierarchical prototypes
X
comolated graph G''
x
(a) local-instance structure
(b) global-semantic structure
Figure 1: Illustration of GraphLoG. (a) Correlated graphs are constrained to be adjacently embed- ded to pursue the local-instance structure of the data. (b) Hierarchical prototypes employed to discover and refine the global-semantic structure of the data.
between various graph instances, we first align the embeddings of correlated graphs/subgraphs by maximizing their mutual information. In this locally smooth embedding space, we further represent the distribution of different graph embeddings with hierarchical prototypes" whose number adap- tively determined by the data in a nonparametric fashion. During training, these prototypes guide feature cluster, and, simultaneously, the prototypes
each graph to map to the remantically-simile are maintained by online-updated graph embeddings. In this process, the global-semanee structure of the data is gradually discovered and refined. The whole model is pre-trained with a large number of unlabeled graphs, and then fine-tuned and evaluated on some downstream tasks.
We summarize our contributions as follows:
We contribute unified framework called Local-instanee and Global-semantic Learning (GraphLoG) for sulf--uue graph representation learning, which is able model the structure of set of graphs both locally and globally.
We novelly propose to infer the global-semanti structure underlying the unlabeled graphs by learning hierarchical prototypes via nonparametric strategy.
We empirically verify our framework's superior performance different GNN architec- tures through pre-training on a large-scale unlabeled dataset and fine-tuning on benchmark tasks both the chemistry and biology domains.
PROBLEM DEFINITION AND PRELIMINARIES
2.1 PROBLEM DEFINITION
An ideal representation should preserve the local structure among the data instances. More specifi- cally, we define it as follows:
Definition 1 (Local-instance Structure). The local-instance structure refers to the local pairwise similarity between different instances (Roweis & Saul, 2000; Belkin & Niyogi, 2002). To preserve the local-instance structure of graph-structured data, a pair of similar graphs/subgraphs, G and g', expected to be mapped to the nearby positions of embedding space, illustrated in Fig. 1(a), while the dissimilar pairs should be mapped to far apart.
The pursuit of local-instance structure is usually insufficient capture the semanties underlying the entire dataset. It is therefore important to discover the global-semantic structure of the data, which coneretely defined as follows:
Definition 2 (Global-semantic Structure). A real-world dataset is usually distributed as different semantic clusters (Furnas et al., 2017: Ji al., 2019). Therefore, we define the global-semantic structure of a dataset as the distribution of its semantic clusters, and each cluster is represented by a prototype (i.e. a representative cluster embedding). Since the semantics of a of graphs can be structured in a hierarchical way (Ashburner et al., 2000: Chen et al., 2012), we represent the whole dataset with hierarchical prototypes. detailed example can be seen in Fig. 1(b).
In our method, we obtain correlated graphs/subgraphs via minor modification on node/edge attributes. Hierarchical prototypes representative cluster emheddings organized in hierarchical way.
2

---

Under review as a conference paper at ICLR 2021
Problem Definition, For self-suuervisee graph representation learning, set of unlabeled graphs G = {9,, G2. ................................ .. are given, and we aim to learn a low-dimensional vector hi, E R for each graph Gj E G under the guidance of the data itself. In specific, expect the derived graph embeddings H E RMx follow both the local-instance and global-semantic structure.
2.2 PRELIMINARIES
Graph Neural Networks (GNNs), Given graph G (V, E) with node attributes Xy = {X,l V} and edge attributes Xg = (u, v) E J. GNN aims to learn an embedding vector he for each node v E V and also vector hig for the entire graph G. For an L-layer GNN, a neighborhood aggregation scheme is performed to capture the T--hop information surrounding each node. The 1-th
layer of a GNN can be formalized as follows:
(1) COMBINE (1.6-1), AGGREGATEE (((RN-1), -..)) : :
(I) where (Nv) is the neighborhood set v, A denotes the representation of node v at the I-th layer, and hg is initialized as the node attribute X,- Since h, summarizes the information of patch centered around node v, we will refer to hy as patch embedding to underscore this point. The entire graph's 's embedding can be derived by a permutationninnaariat readout function:
hg READOUT((hhl) E V)).
Mutual Information Estimation. Mutual information (MI) can measure both the linear and non- linear dependency between two random variables. Some recent works (Belghazi al., 2018; Hjelm et al., 2019) employed neural networks to estimate the lower bound of MI. Among which, InfoNCE loss (van den Oord et al., 2018 8) has been introduced to maximize a lower bound of MI by minimizing itself, and we also adopt it in this work for its simplicity and effectiveness. In practice, given query q, InfoNCE loss is optimized score the positive sample 2+ higher than of distractors {
exp (T(g, 2+))
LNCE (g, 24 { - -log
(3)
exp (T(q, Σ exp (T(q, =))
where T(--) is a parameterized discriminator function which maps two representation vectors to a scalar value, whose architecture is detailed in Sec. 6.1.
Rival Penalized Competitive Learning (RPCL.. The RPCL method (Xu et al., 1993) is a variant of classical competitive learning approaches, e.g. K-means clustering. Concretely, given a sample for update, RPCL- clustering not only pulls the winning cluster center (i.e. the closest onc) towards the sample, but also pushes the rival cluster center (i.e. the second closest one) away from the sample. We adopt this clustering algorithm for its strong capability of discovering feature clusters without specifying the number of clusters beforehand (i.e. in a nonparametric fashion), which is critical in the self-supervised learning scenarios where the number of semantic categories is not given.
3 LOCAL-INSTANCE AND GLOBAL-SEMANTIC LEARNING
3.1 LEARNING LOCAL STRUCTURE OF GRAPH REPRESENTATIONS
We first define the correlated graphs that are expected to be embedded close to each other in the embedding space. Since the graphs from dataset lie in highly discrete space, it is hard to seek out the correlated counterpart of each graph from the dataset. To tackle this limitation, we propose to construct pairs of correlated graphs via the attribute masking strategy (Hu al., 2019) which randomly masks a part of node/edge attributes in a graph (theoretical analysis is stated Sec. A).
Through applying this technique to randomly sampled mini-batch B3 {G, (V,, E with N graphs, the correlated counterpart of each graph can be obtained, which forms another mini- batch By (9/- = (V/, N (G, and 9j are deemed as input, the corresponding patch and graph embeddings are derived as follows:
pair of correlated graphs). Taking both
by, (h./v V,} GNN(X v,, Xg,), hy; (4.// E V2) GNN(X Xe;),
(4)
hg (hy)
hgj READOUT((w))
(5)
3

---

Under review as a conference paper at ICLR 2021
where Ay, (hy) is the set of patch embeddings for graph 93 (9)), and hig (hry)) denotes the em- bedding of entire graph. With these ingredients, design the learning objective for local-instance structure based on two desiderata: (1) similar subgraphs (i.e. patches) have similar feature represen- tations; (2) graphs with set of similar patches are embedded close each other. To attain these goals, we propose to maximize the mutual information (i.e. minimize the InfoNCE loss) between correlated patches/graphs, which derives two constraints for the local-instanee structure:
Cpatch
Σ 1=
E
Σ .EV;
Lues LNCE he, {helö Vj, ē # u}),
(6)
N Σj=1 DI
V'EV'
ggapph N
LNCE (hoy, hog, (hea 11 & k N j});
Liocal = pasch Leraph:
where LNCE(")'.) is the InfoNCE loss function defined in Eq. 3, and 1 weys denotes the indicator function judging whether v and v' are the corresponding nodes in a pair of correlated graphs. Note that, masking node/edge attributes doesn't 't change the topology of a graph, which makes it casy to determine these corresponding nodes in our method.
3.2 LEARNING GLOBAL- SEMANICC STRUCTURE OF GRAPH REPRESENTATIONS
It is worth noticing that the graphs in dataset may possess hierarchical semantic information. For example, drugs (i.e. molecular graphs) are represented by five-level hicrarchy in the Anatomical Therapeutic Chemical (ATC) classification system (Chen et al., 2012). Moreover, the biological functions of proteins (i.e. graphs of amino acid residues) can be organized in a hierarchical structure (e.g. Gene Ontology (Ashburner al., 2000) and FunCat (Ruepp al., 2004) protein functional- definition schemes).
Motivated by this fact, we propose the notion of hierarchical prototypes to describe the distributions of graph embeddings. These prototypes are structured as set of trees (Fig. 1(b)), in which each node denotes a prototype (i.e. a representative embedding of feature cluster) and corresponds to an unique parent node unless it is at the top layer. Formally, the hierarchical prototypes can be represented as (d))M) (1 1,2, Lp), where Lp denotes the depth of hierarchical prototypes, and Mi is the number of prototypes at the [-th layer. Except for the leaf nodes, each prototype possesses a set of .... T.p 1). During training, the derivation
child nodes, denoted as C(e() (1 i M, 1 = 1, ....... of these variables can be divided into two stages, i.e. initialization and maintenance.
Initialization of hierarchical prototypes. In order to establish appropriate priors of graph em- beddings, we first pre-train the GNN by minimizing Liecal for one epoch and utilize it to extract the embeddings of all graphs in the training set, denoted {hg, J.N (Nd is the size of training set). These embeddings are used to initialize the bottom layer prototypes (i.e. " RPCL-based clustering algorithm (Sec. 2.2): Mr, (c M RPC J.a)
Map) via the
(9) where RPCL(.) outputs the cluster centers that are assigned with at least one sample. After that, the prototypes of upper layers are initialized by iteratively applying Rasee clustering to the prototypes of the layer below:
M = RPCLLLCC (+2 (c(t)) 1-
Lp
(10)
It is noteworthy that, in the initialization scheme. the number of prototypes is automatically adapted to the distribution of graph embeddings. As result, this scheme is nonparametric and can adapt to different datasets without the prior knowledges about them.
Maintenance of hierarchical prototypes. In the training process, since the embedding of each graph is dynamically changing, we propose strategy maintain hierarchical prototypes with on- line updated graph embeddings. Concretely, under the guidance a similarity measurement (e.g. cosine similarity in implementation), the graph embeddings extracted from mini-batch Bg
4

---

Under review as a conference paper at ICLR 2021
Algorithm Training procedure of Local-instance and
Learning (GraphLoG).
Input:
Training set D = {9, ) No the number of training iterations Nr, hierarchical prototypes' depth Lp and exponential decay rate B. The pre-trained GNN.
Output:
Initialize hierarchical prototypes {c] (1= 1,2, Lp)
for f. = to N7. do
Bg RandomSample())
# Get mini-batch of graphs
B'g - AttrMaskiin(BB3 (gg)
# Get the correlated graphs
hy, ho,, hgg + Eqs. (4, 5) (j 1,2,...,,A
Extract patch and graph embeddings Compute losses
Lioual: Lelahall t- Eqs. (8, 13)
OGNN = -Vas (Clocal Lglohal)
Update GNN's parameters
Or = -oo (Ciccal + Lglobal
# Update discriminator's 's parameters
[cl + Eqs. (11, 12) 1,2,-.. Ipp
Maintain hierarchical prototypes
end for
divided into Mr, groups according their most similar bottom layer prototype, and the mean graph embeddings are computed within each group, denoted { Mr, These mean embeddings employed to update bottom layer prototypes via an exponential moving average scheme:
o + 3C + (1 -) 1 ML.:
(11)
where / is the exponential decay rate. For the prototypes of upper layers, they are updated with the mean of their child prototypes in the corresponding tree:
d
Σ C())
dt,, 1 MI, 1= 1,2, Lpp 1.
(12)
[CC)
Constraint for global-semantic structure. Now that the latent semantic structure of the data has been represented by hierarchical prototypes, we seek to constrain the distributions of graph embed- dings with these prototypes. The major goal here is map correlated graphs the same set of fea- ture clusters. In practice, according to cosine similarity, we first search for the prototypes most simi- lar to the embedding of graph Gj each layer, denoted s(G,) {si (Gş), (2(9)),.. Sr., (G)))- Note that, this search process follows the topology of hierarchical prototypes, which means that: (9,) E C(s (9,))
2. Lp 1). Correspondingly, when using the embedding of the correlated graph 9 for search, we expect an identical searching path, and such objective is pursued by maximizing the mutual information (i.e. minimizing the InfoNCE loss) between graph embed- ding he; and the prototypes in s(G)):
Lglobal Lp
(hce(ho), (c/11 < & Mi siG)))))
(13)
Discussion. A recent work (Li et al., 2020) employed hierarchical prototypes for visual represen- tation learning. The semantic hierarchy established that work is derived from multiple times of clustering with different numbers of clusters, which relies heuristically selected cluster numbers and fails to model the relations between the prototypes from different hierarchies. In contrast, our method is free from pre-defined cluster numbers, and of relational trees are constructed to embody the hierarchical relations between different prototypes.
3.3 MODEL OPTIMIZATION
The training procedure of Local-instance and Elobal-semmante Learning (GraphLoG) is summarized in Algorithm 1. Along training, using the online-updated graph embeddings, the constraints for local-instance and global-semantic structure are derived, and the hierarchical prototypes are main- tained. In each iteration, the parameters of GNN and discriminator are optimized with gradient descent using the following objective:
min Liocal + Lyloball GNN,T
(14)
5

---

Under review as a conference paper at ICLR 2021
4 SUP-GRAPHLOG: A SUPERVISED BASELINE FOR LOCAL-INSTANCE AND GLOBAL-SEMANTIC LEARNING
In order to verify the effectiiveness of local-insse instance and global-semene semannti learning when it is directly applied to supervised downstream tasks, we propose a baseline model, named as sup-GraphL.od which combines a plain GNN and the proposed hierarchical prototypes.
In the training phase, in order to establish appropriate local-instance structure of graph embeddings, the GNN is first pre-trained along with linear classifier to perform graph classification the training set. For the initialization of hierarchical prototypes, the number of bottom layer prototypes set as the class number of the supervised task (e.g. 2KT bottom layer prototypes for a task with Kr binary classification problems), and each bottom layer prototype is the mean embedding of all the training graphs belonging the corresponding class. The upper layer prototypes are initialized as in Eq. 10. For maintenance, given mini-batch of labeled graphs, each bottom layer prototype updated by the mean embedding of all the graphs belonging to the corresponding class using an exponential moving average scheme as in Eq. 11, and the prototypes of upper layers are maintained following Eq. 12.
For constraining the global-semantic structure, compared with the top-down search in the self- supervised model (Sec. 3.2), this supervised setting, we first use the label of graph Gj to ran- domly select matched bottom layer prototype SL, $L,(G)) (Gj) and then obtain the whole searching path sG,) = sz (G)) SL, (Gj)} from bottom to up. Based on this positive search- ing path, we randomly sample negative path (G,) {sT(G)) 83 (G)),.. .8, (Gj)) satisfy- ing that graph 9j does not belong to the corresponding class of s2,p (9,), and 87 (g)) * st(G)), (9,) E (G))) (1 = 1, ... Ipp 1). It is expected that the embedding of graph G₁ is more similar with the prototypes on path s(G)) than the ones on path s" (G)), which defines the loss constraint mini-batel as follows:
N E
Sup globall
Σ jm1
((9)),
(15)
N.
Lp
We further optimize the GNN by minimizing this loss, which refines the global- global-semantic structure in the embedding space.
In the inference phase, given an unlabeled graph, we first compute the similarity between its em- bedding and all the bottom layer prototypes via the cosine similarity function. After that, the task- specific prediction is derived by comparing the similarity scores of the classes corresponding to these prototypes, in which the classes with larger scores serve as the prediction result.
5 RELATED WORK
Graph Neural Networks (GNNs). Recently, following the efforts of learning graph representations via optimizing random walk (Perozzi et al., 2014; Tang et al., 2015: Grover Leskovee, 2016; Narayanan et al., 2017) or matrix factorization (Cao et al., 2015; Wang et al., 2016) objectives, GNNs feature vectors a neighborhood aggregation
are proposed to explicitly derive moxxiitty-preserve manner. As suggested in Gilmer al. (2017), the forward pass of most GNNs can be depicted two phases, Message Passing and Readout phase, and various works (Kipf & Welling, 2017: Hamilton al., 2017: Velickovic al., 2018: Ying al., 2018; Zhang al., 2018; Xu al., 2019) sought to improve the effectiveness of these two phases. Unlike these methods which are mainly trained in a supervised fashion, our approach aims for unsupervised/self-uueee supervviee learning for GNNs.
Self-supervised Learning for GNNs. There are some recent works that explored self-supervised graph representation learning with GNNs. Gareía-Durán Niepert (2017) learned graph repre- sentations by embedding propagation, and Velickovic et al. (2019), Sun et al. (2019) and Hassani & Ahmadi (2020) achieved this goal through mutual information maximization. Also, some self- supervised tasks, e.g. edge prediction (Kipf & Welling, 2016), context prediction (Hu et al., 2019; Rong et al., 2020a), graph partitioning (You et al., 2020) and edge/attribute generation (Hu et al., 2020), have been designed to acquire knowledges from unlabeled graphs. Nevertheless, all these methods are only able to model the local relations between different graph instances. The proposed framework seeks to discover both the local-instance and glohal-semantic structure of set of graphs.

---

Under review as a conference paper at ICLR 2021
Self-supervised Semantic Learning. Clustering-based methods (Xie al., 2016; Yang al., 2016; 2017: Caron et al., 2018; Ji et al., 2019; Li et al., 2020) are commonly used to learn the semantic information of the data in a self-supervised fashion. Among which, DeepCluster (Caron et al., 2018) proved the strong transferability of the visual representations! learnt by clustering prediction to various downstream visual tasks. Prototypical Contrastive Learning (Li et al., 2020) set new state- of-the-art for unsupervised visual representation learning. These methods are mainly developed for images but not for graph-structured data. Furthermore, the hierarchical semantic structure of the data has been less explored in previous works.
6 EXPERIMENTS
6.1 EXPERIMENTAL SETUP
Pre-training details. Following Hu et al. (2019), we adopt a five-layer Graph Isomorphism Network (GIN) (Xu et al., 2019) with 300-dimensional hidden units and mean pooling readout function for performance comparisons (Secs. 6.2 and 6.3). The discriminator for mutual information estimation is formalized as: T( .2) (f (x22), where f(-) is a projection function fitted by two linear layers and a ReLU nonlinearity between them, and .) is a similarity function (e.g. cosine similarity in our method). In all experiments, we use an Adam optimizer (Kingma & Ba, 2015) (learning rate: 1 x 10-$, batch size: 512) to train the model for 20 epochs. Unless otherwise specified, the hierarchical prototypes' depth Lp is set as 3, and the exponential decay rate 3 is set as 0.95. For attribute masking, 30% node attributes in molecular graphs are masked, and 30% edge attributes in Protein-Protein Interaction (PPI) networks are masked. These hyperparameters are selected by the grid search on the validation sets of four downstream molecule datasets (i.e. BBBP, SIDER, ClinTox and BACE), and their sensitivity is analyzed in Secs. 6.4 and F.
downstream task, a linear classifier is appended on the top of pre-trained GNN, and an Adam optimizer (classifier's learning rate: x 10-3 GNN's learning rate: 1 x 10-1, batch size: 32) is employed to train the model for 100 epochs. For sup- GraphLr,,L the GNN is first trained along with a linear classifier for 50 epochs using an Adam optimizer (learning rate: 1 10-3 batch size: 32), and is then fine-tuned under the guidance of hierarchical prototypes by an Adam optimizer (learning rate: 1 1x10 batch size: 32). All reported results are averaged over five independent runs under the same configuration. Our approach is implemented with PyTorch (Paszke et al., 2017), and the source code will be released for reproducibility.
Fine-tuning details. For fine-tuning
Performance comparison. We compare the proposed method with existing self-supervised graph representation learning algorithms (i.e. EdgePred (Kipf & Welling, 2016), InfoGraph (Sun et al., 2019), AttrMasking (Hu et al., 2019), ContextPred (Hu et al., 2019) and GraphPartition (You al., 2020)) to verify its effectiveness. Following the setting in Hu et al. (2019), after pre-training GNN models with self-supervised methods, a graph-level multi-task supervised pre-training is conducted to achieve more transferable graph representations, and the performance on downstream tasks is respectively evaluated before and after this graph-level supervised pre-training.
6.2 EXPERIMENTS ON CHEMISTRY DOMAIN
Datasets. For fair comparison, we use the same datasets as in Hu et al. (2019). specific, a subset of ZINC15 database (Sterling & Irwin, 2015) with 2 million unlabeled molecules is employed for self-supervised pre-training, and a preprocessed ChEMBL dataset (Mayr et al., 2018) with 456K labeled molecules is used for graph-level supervised pre- pre-training. Eight binary classification datasets contained in MoleculeNet (Wu et al., 2018) serve as downstream tasks.
Results. Tab. I reports the performance of proposed GraphLoG method compared with other works.
Among all self-supervised learning strategies, our approach achieves the best performance on seven of eight tasks, and a 3% performance gain is obtained in terms of average ROC-AUC. After applying a subsequent graph-level supervised pre-training, our models' performance is further promoted. In particular, 2.9% increase observed on the SIDER dataset. Also, the comparison between two supervised methods without self-supervised pre-training is presented in the table, the proposed sup- GraphLoG outperforms the vanilla GIN model with random initialization, which demonstrates the benefit of learning global-semantic structure. The training curves of eight downstream tasks are provided in Sec. G.
7

---

Under review as a conference paper at ICLR 2021
Table I: Test ROC-AUC (%) on molecular property prediction benchmarks.
Methods
BBBP
Tox21
ToxCast
SIDER
ClinTox
MUV
HIV
BACE
Avg
Random
65.8 4.5 74.0 = 0.8 6.64
58.0 67.3 7..8
75.3
70.1
67.0
50p-GraphiLoG (ours)
0.3 0.29 638+0.1
614#06 6.0 725=10
0.5
76.5
69,9
EdgePred (2016)
67.3424
76.0
61.10.6
60.1
611
74.12.
1.0
0.9
70,3
InloGraph (2019)
6.7 75.5
63.1
59,4
70. 7.,
77.6
77.6
78.9
1.1
71.1
AttrMasking (2019)
2..
76.7
642
61.0 718
71.7114
77.2
79.3
1.6
71.1
ContextPred (2019)
68.0 75.7 (.
63.9 60.9 65.9
35.8
75.8
77.3
1.2
70.9
GraphiParittion (2020)
0.7 75.2 63.2
610 (12:
04000000000000000000000011111111 0.
75,411.7
0.7
7..6
70.8
GraphLoG (ours)
0.7 76.2 64. 01.00-66....1.. 6.7 786
764
0.6
83.3
1.4
74.1
Supervised
0.7
79.4
1.4
76.0
70,0
EdgePred" (2016)
2.. 0.3
665
63.
70,9
07.5
79.1
72.6
InfoGraph" (2019)
68.4 07..6
73.8
79.
78.0
82.4
1.3
73.4
AnrMasking (2019)
2.5 77.9 9
05.1
63.9
73.7
81.2
77.1
1.2
80.3
0.9
73.2
ContexPree (2019)
68.7 1.3 78.1 +..1
65.
6.66
62.7
726
813
79.9
0.7
84.5
14.2
GraphParttion" (2020)
71.1 0.5 7.
612 03
02 08.
78.6
0.4
80.
73,3
Graphi oG* (ours)
74,0 0.8 78.5 0.2 0.5 64.6 0.8 7.77 79.5
80.1 0.7 85.1 + 0.9
75.9
denotes the model composed of specific self-supervised pre-training and subsequent graph-level supervised pre-training.
Table 2: Performance comparison and ablation study on biological function prediction benchmark.
(a) Test ROC-AUC (%) of different methods.
(b) Ablation study for different loss terms.
Methods
ROC-AUC (%)
Cpascs
gxaph
Legitbak
ROC-AUC (%)
Random
64.8 1.0
70.6 + 0.5
Sup-GraphLoG (ours)
67.6
71.1 0.7
EdgePred (Kipf Welling, 2016)
70.5 0.7
71.4
InfoGraph (Sun et al., 2019)
70.7 0.5
AttMaskknn (Hu et al., 2019)
70.5 0.5
72.0 4 0.3
ContextPred (Hu al., 2019)
69.9 = 0.3
4
72.0 4 0.6
Grapharrittion (You et al., 2020)
71.0 0.2
71.8 2 0.5
GraphLoG (ours)
72.8
V
728
0.4
6.3 EXPERIMENTS ON BIOLOGY DOMAIN
Datasets. For biology domain, following the settings in Hu et al. (2019). 395K unlabeled pro- tein ego-networks are utilized for self-supervised pre-training, and the prediction of 5000 coarse- grained biological functions on 88K labeled protein ego-networks serves as graph-level supervised pre-training. The downstream task is to predict 40 fine-graine fine-grined biological functions of 8 species.
Results. In Tab. 2a, we report the test ROC-AUC of various self-supervised learning techniques, and more results on biology domain can be found in Sec. C. It can be observed that the proposed GraphLoG method outperforms existing approaches with a clear margin, i.e. a 1.8% performance gain. This result illustrates that the proposed scheme is beneficial to fine one-grained downstream tasks. In addition, sup-GraphLoG is able to promote the performance of plain GIN model by 2.8% on this biological downstream task.
6.4 ANALYSIS
Effeet of different loss terms. In Tab. 2b, we analyze the effect of three loss terms on biological function prediction, and we continue using the GIN depicted in Sec. 6.1 in this experiment. When each loss is independently applied (Ist, 2nd and 3rd row), the loss for global-semantic structure performs best, which probably benefits from its exploration of data's 's semantic information. Through combining these losses, the full model (last row) achieves the best performance, which illustrates that the learning of iccal-instance and global-semantic structure are complementary to each other. We provide more ablation studies on different model components in Sec. E.
Results on different GNNs. Fig. 2(a) presents the effect of self-supervised pre-training four kinds of GNNs, GCN (Kipf & Welling, 2017), GraphSAGE (Hamilton et al., 2017), GAT (Velick- ovic et al., 2018) and GIN (Xu et al., 2019). We observe that the proposed GraphLoG scheme outperforms two existing methods, AttrMasking and ContextPred, on all configurations, and avoids the performance decrease relative to random initialization baseline GAT.
8

---

Under review as a conference paper at ICLR 2021
ROC-AUC (50
DripeSAGE
0.05)
Figure 2: (a) Experimental results on different GNNs. (b)&(c) Sensitivity analysis of hierarchical prototypes' depth Lp and exponential decay rate B. (All results are reported on biology domain.)
graph embbeddin
prototypes of bottom layer
prototypes of middle layer
prototypes of
(a) Lpacck
(b) Expcch
gropph
(c) Eipachh + Eirass + Ligabal
Figure 3: The t-SNE (Maaten & Hinton, 2008) visualization of graph embeddings and hierarchical prototypes on ZINC15 database (i.e. the pre-training dataset for chemistry domain).
Sensitivity of hierarchical prototypes' depth Ly. this part, we discuss the selection of parameter Lp which controls the number of discovered semantic hierarchies. In Fig. 2(b), we plot model's performance under different Tp values. It can observed that deeper hierarchical prototypes (i.e. Ip > 3) achieve stable performance gain compared to the shallow ones (i.e. Ly & 2).
Sensitivity of exponential decay rate B. In this experiment, we evaluate our approach's sensitivity to the parameter 3. Fig. 2(c) shows the test ROC-AUC downstream task using different 3 values. From the line chart, we can observe that the proposed model's performance not sensitive to B, which makes the maintenance scheme of hierarchical prototypes easy to tune.
Visualization. Fig. 3, we utilize t-SN (Maaten & Hinton, 2008) to visualize the distributions of graph embeddings and hierarchical prototypes ZINC15 dataset. Compared to the model with only Epcch constraint, some feature clusters are formed after constraining the relations between correlated graphs' embeddings by Lyaph- More obvious feature separation is achieved after applying g which illustrates its effectiveness on discovering the global-semantee structure of the data.
CONCLUSIONS AND FUTURE WORK
We devise a unified framework called Local-instance and Global-semantic Learning (GraphLoG) for self-supervised graph representation learning, which models the structure of a of unlabeled graphs both locally and globally. In this framework, we novelly propose to learn hierarchical proto- types upon graph embeddings infer the global-semantic structure in graphs. Using the benchmark datasets from both chemistry and biology domains, we empirically verify our method's superior performance over state-of-the-art approaches on different GNN architectures.
Our future works will include exploring novel ways to construct correlated graphs, improving self- supervised learning manners, unifying pre-training and fine-tuni -uentun and extending our framework to other domains such sociology, physics and material science.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Bijaya Adhikari, Yao Zhang, Naren Ramakrishnan, and B. Aditya Prakash, Sub2vec: Feature learn- ing for subgraphs. In Advances in Knowledge Discovery and Data Mining, 2018
Marco A. Alvarez and Changhui Yan. A new protein graph model for function prediction. Compu- tational Biology and Chemistry, 37:6-10, 2012.
Michael Ashburner, Catherine A Ball, Judith Blake, David Botstein, Heather Butler, J Michael Cherry, Allan P Davis, Kara Dolinski, Selina S Dwight, Janan T Eppig, et al. Gene ontology: tool for the unification of biology. Nature Genetics, 25(1):25-29, 2000.
Mohamed Ishmael Belghazi. Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, R. De- von Hjelm, and Aaron C. Courville. Mutual information neural estimation. In International Conference on Machine Learning, 2018.
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps and spectral techniques for embedding and clustering. Advances in Neural Information Processing Systems, 2002.
Shaosheng Cao, Wei Lu, and Qiongkai Xu. Grarep: Learning graph representations with global structural information. In ACM International Conference on Information and Knowledge Man- agement, 2015.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In European Conference on Computer Vision, 2018.
Lei Chen, Wei-Ming Zeng, Yu-Dong Cai, Kai- Yan Feng, and Kuo-Chen Chou. Predicting anatom- ical therapeutic chemical (atc) classification of drugs by integrating chemical-chemical interac- tions and similarities. PloS one, 7(4))e35254, 2012.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geolfrey E. Hinton. A simple framework for contrastive learning of visual representations. CoRR, abs/2002.05709, 2020.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. BERT:
pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
George W Furnas, Scott Deerwester, Susan T Durnass, Thomas K Landauer, Richard A Harshman, Lynn A Streeter, and Karen E Lochbaum. Information retrieval using a singular value decompo- sition model of latent semantic structure. In ACM SIGIR Forum, 2017.
Alberto García-Durte and Mathias Niepert. Learning graph representations with embedding propa- gation. In Advances in Neural Information Processing Systems, 2017.
Thomas Gärtner, Peter A. Flach, and Stefan Wrobel. On graph kernels: Hardness results and efficient alternatives. In Computational Learning Theory and Kernel Machines, 2003.
Justin Gilmer, Samuel S. Schoenholz. Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning, 2017.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In ACM
SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In IEEE Conference on Computer Vision and Pattern Recognition, 2006.
William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning large graphs. Advances in Neural Information Processing Systems, 2017.
Kaveh Hassani and Amir Hosein Khas Ahmadi. Contrastive multi-view representation learning on graphs. International Conference on Machine Learning, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu. Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. CoRR, abs/1911 10011100550 1,05222, 2019.
10

---

Under review as a conference paper at ICLR 2021
R. Devon Hjelm, Alex Fedorov, Samuel Srooie-Marchhldon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference Learning Representations, 2019.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitmik. Percy Liang, Vijay S. Pande, and Jure 12265, 2019.
Leskovec. Pre-training graph neural networks. CoRR, abs/1905. 122./9955.2225.,..
Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai- Wei Chang. and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2020.
Xu Ji, Andrea Vedaldi, and João F. Henriques. Invariant information clustering for unsupervised image classification and segmentation. In International Conference on Computer Vision, 2019.
Biaobin Jiang, Kyle Kloster, David F. Gleich, and Michael Gribskov. Aptrank: adaptive pagerank model for protein function prediction on bi-relational graphs. Bioinformatics, 33(12):1829 1829-16822-1886, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Thomas N. Kipf and Max Welling. Variational graph auto-encoders. CoRR, fbs//1611.07306 2016.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In International Conference on Learning Representations, 2017.
Johannes Klicpera, Stefan Weißenberger, and Stephan Günnemann. Diffusion improves graph learn- ing. In Advances in Neural Information Processing Systems, 2019.
Nils M. Kriege, Pierre-Louis Giscard, and Richard C. Wilson. On valid optimal assignment kernels and applications to graph classification. In Advances in Neural Information Processing Systems, 2016.
Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven C. H. Hoi. Prototypical con- trastive learning of unsupervised representations. CoRR, abs/2005.04966 2020.
Laurens Van Der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine
Learning Research, 9(2605):2579-260) 2008
Andreas Mayr, Günter Klambauer, Thomas Unterthiner. Marvin Steijaert, Jörg K Wegner, Hugo Ceulemans, Djork-Arné Clevert, and Sepp Hochreiter. Large-scale comparison of machine learn- ing methods for drug target prediction on chembl. Chemical science, 9(24):5441-5411, 2018.
Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. graph2vec: Learning distributed representations of graphs. CoRR, abs/ 0as/1777.05000 2017.
Nicolò Navarin, Dinh Van Tran, and Alessandro Sperduti. Pre-training graph neural networks with kernels. CoRR, abs/18 11211811.0699 1.06930, 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In Advances in Neural Information Processing Systems Workshop, 2017.
Bryan Perozzi, Rami AI-Rfou, and Steven Skiena. Deepwalk: online learning of social represen- tations. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2014.
Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. GCC: graph contrastive coding for graph neural network pre-training. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2020.

---

Under review as a conference paper at ICLR 2021
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improv- ing language understanding by generative pre-training. 2018. URL https:
//s3-us-west- amazonaws
Yu Rong. Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. GROVER: self-supervised message passing transformer on large-scale molecular data. CoRR, abs/2007.02835, 2020a.
Yu Rong. Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. In International Conference on Learning Repre- sentations, 2020b.
Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embed- ding. Science, 290655000:2323-2325 2000.
Andreas Ruepp, Alfred Zollner, Dieter Maier, Kaj Albermann, Jean Hani, Martin Mokrejs, Igor Tetko, Ulrich Güllener, Gertrud Mannhaupt, Martin Münsterkötter, et al. The funcat, functional annotation scheme for systematic classification of proteins from whole genomes. Nucleic Acids Research, (1(18))5539-55555 2004.
Teague Sterling and John Irwin. Zine ligand discovery for everyone. Journal of Chemical Information and Modeling, (1::))2222233355 2015.
Fan-Yun Sun, Jordan Hoffmann, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. CoRR, abs/1908 01000, 2019.
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. LINE: large-scale information network embedding. In International Conference on World Wide Web, 2015.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, abs/1 807.807000 507.03348, 2018.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
Petar Velickovic, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. In International Conference on Learning Representations, 2019.
Daixin Wang, Peng Cui, and Wenwu Zhu Structural deep network embedding. In ACM SIGKDD International Conference Knowledge Discovery and Data Mining, 2016.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learn- ing. Chemical science, 9(2):513-530 2018
Junyuan Xie, Ross B. Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International Conference on Machine Learning, 2016.
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanic Jegelka. How powerful are graph neural networks? In International Conference Learning Representations, 2019.
Lei Xu, Adam Krzyzak, and Erkki Oja. Rival penalized competitive learning for clustering analysis, rbf net, and curve detection. IEEE Transactions Neural Networks, 4(4):636-649, 1993.
Pinar Yanardag and S. V. N. Vishwanathan. Deep graph kernels. In International Conference on Knowledge Discovery and Data Mining, 2015.
Bo Yang, Xiao Fu. Nicholas D. Sidiropoulos, and Mingyi Hong. Towards k-means friendly spaces: Simultaneous deep learning and clustering. In International Conference on Machine Learning, 2017.
k-means-friendly
12

---

Under review as a conference paper at ICLR 2021
Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In JEEE Conference on Computer Vision and Pattern Recognition, 2016.
Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L. Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. In Advances in Neural Information Processing Systems, 2018.
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. When does self-supervision help graph convolutional networks? In International Conference on Machine Learning, 2020.
Guo Zhang. Hao He, and Dina Katabi. Circuit-gnn: Graph neural networks for distributed circuit design. In International Conference Machine Learning. 2019.
Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In AAAI Conference on Artificial Intelligence, 2018
13

---

Under review as a conference paper at ICLR 2021
A THEORETICAL ANALYSIS OF CORRELATED GRAPH CONSTRUCTION
In our method, we choose the attribute masking (Hu et al., 2019) strategy to generate correlated graph pairs, which is widely used in recent self-supervised graph representation learning algo- rithms (Hu et al., 2019; 2020; Qiu et al., 2020). Since the graph structure has not been changed by the masking operation, the masked node attribute information surrounding neighbors after being fed into a GNN. Therefore, the embeddings of correlated graph pairs can maintain a high degree of consistency in the feature space, which is desirable for the proposed GraphLoG model. We formally elucidate this point as follows.
be partially recovered by its
Given an attributed graph 9 (V, Xy, Xε) (Xy: node attributes, Xg: edge attributes), assume that its correlated graph g' = (y, Xx) is obtained by masking the attribute of node v, i.e. X3 & (v) {
Proposition 1. The I-layer GNN can repair the lost information induced by attribute masking operation by Ixepair 3 I(X,, {Xalō
(Xal E NY)),, where I(; denotes the mutual information, and No. is the h-hop neighborhood set v.
Proof. Before information propagation by GNN, we define the lost information Itsst induced by attribute masking as the conditional entropy of graph 9 conditioned on its correlated graph G':
Tost H(Glg") H(x)).
(16)
After the information propagation by GNN, we have:
Jsst H((gg/hg)) H(hg) I (hg,
(17)
where Tostt is the information lost in the embedding of correlated graph g' compared with the embedding of origin graph G.
According to the neighborhood aggregation scheme in GNN, we can derive:
I(hg, H(hg) H(h,)) + I(X,, {Xal0 j E N.
(18)
where I(X,,, {xalō E J) denotes the recovered information for the masked node v from its L-hop neighbors. Combining Eqs. 17 and 18 leads to:
I H(h,,) I(X,,, E
(19)
We can derive the information repaired by GNN, I Trepair as:
Trepair = -iost Irast I(X,, {xalö NC11 (H(X.) H(hh)).
(20)
Since he is the low-dimensional embedding of node attribute X.. deduce that:
((x)) > H(h.).
(21)
Therefore, combining Eqs. 20 and 21, we can conclude:
Irepair 2 I(X,, { E N.)).
(22)
We also evaluate the GraphLoG model with different correlated graph construction strategies, and the experimental results can be found in Sec. E.1. empirically shows that attribute masking more reliable to our method.
B MORE IMPLEMENTATION DETAILS
Attribute masking strategy. We add an extra dimension in the vector of node/edge attribute and set only that dimension as 1 when the corresponding node/edge is masked. Given a mini-batch of graphs, we mask the same proportion of node/edge attributes in each graph, and, for undirected graphs, the attributes the both directions of an edge are masked/unmasked.
GNN architecture. All the GNNs in our experiments (i.e. GCN (Kipf & Welling, 2017). Graph- SAGE (Hamilton et al., 2017), GAT (Velickovic et al., 2018) and GIN (Xu et al., 2019)) are with 5 layers, 300-dimensional hidden units and a mean pooling readout function. In addition, two attention heads employed in each layer of the GAT model.
14

---

Under review as a conference paper at ICLR 2021
C
MORE RESULTS ON BIOLOGY DOMAIN
Table 3: Test ROC-AUC (%) on biological function prediction benchmark.
Methods
ROC-AUC (%)
Methods
ROC-AUC (%)
Random
64.8 + 1.0
Supervised
72.9 + 0.5
EdgePred (Kipf & Welling, 2016)
70.5 = 0.7
EdgePred* (Kipf & Welling, 2016)
73.1
InfoGraph (Sun al., 2019)
InfoGraph* (Sun et al., 2019)
73.7
AttrMasking (Hu et al., 2019)
10.5
Attraasking" (Hu et al., 2019)
74.2 1.5
ContextPred (Hu et al., 2019)
69.9 + 0.3
ContextPred (Hu et al., 2019)
74.3 1 0.6
GraphPartition (You et al., 2020)
71.0 = 0.2
GraphPartition (You ct al., 2020)
73.5 + 0.1
GraphLo (ours)
72.8 + 0.4
GraphLoG* (ours)
75.7 - 0.6
denotes the model composed of specific self supervised pre training and subsequent graph-level supervised pre- reettraining.
In Tab. 3, we report the performance of different approaches on the downstream task of biology domain, and the results before and after applying subsequent graph-level supervised pre-training are respectively reported. It can be observed that the proposed GraphLoG method outperforms existing approaches with clear margin under both settings, which illustrates the effectiveness of proposed learning scheme with and without the guidance of graph-level supervisory signal.
D MORE RESULTS ON GRAPH CLASSIFICATION BENCHMARKS
Table The 10-fold cross validation accuracy (mean std %) of self-supervised methods on graph classification benchmarks.
Methods
MUTAG
PTC-MR
IMDB IMDB-Binary
IMDB-Mult Multi Reddit Binary
random walk (Gärtner et al., 2003)
1.5 1.3
50.7 0.3
34.7 + 0.2
node2vec (Grover & Leskovec, 2016)
72.6 10.2 8.0
graph2vec (Narayanan al., 2017)
9.6 60.2
0.5
50.4 0.9
75.8 1.0
sub2vec (Adhikari et al., 2018)
61.1 15.8 60.0
1.5
0.8
71.5 044
InfoGraph (Sun al., 2019)
1..
6.47
7.9
0.5
82.5 1.4
Contrastive (Hassani & Ahmadi, 2020)
89.7 1 1.1
6
7.77
51.2
84.5 0.6
GraphLoG (ours)
89.9
1.5 63.8
1.6
76.6
+
85.9 2.9
Setups. In this experiment, we compare GraphLoG with six self-supervised graph representation learning methods, i.e. random walk (Gärtner et al., 2003), node2vec (Grover & Leskovec, 2016), graph2vec (Narayanan et al., 2017), sub2vec (Adhikari al., 2018), InfoGraph (Sun et al., 2019) and Contrastive (Hassani & Ahmadi, 2020). We strictly follow the linear evaluation protocol in Sun al. (2019) and report the mean accuracy of 10-fold cross validation. Five conventional graph classification benchmark datasets, i.e. MUTAG (Kriege et al., 2016), PTC (Kriege et al., 2016), LiDD-Binary (Yanardag & Vishwanathan, 2015), IMDB-Multi (Yanardag & Vishwanathan. 2015) and Reddit -indrry (Yanardag & Vishwanathan, 2015), used for evaluation. The settings of network architecture, optimizer and training parameters follow those in Sec. 6.1.
Results. Tab. presents the comparisons of self-supervised approaches five graph classification benchmark datasets. The proposed GraphLoG model ranks the first place in every task, and, espe- cially, it outperforms recent contrastive- based method (Hassani & Ahmadi, 2020), which demonstrates the effectiveness of learning local-instance and global-semantic structure.
E MORE ABLATION STUDIES
ABLATION STUDY ON CONSTRUCTING CORRELATED GRAPHS
In this part, analyze three ways of constructing correlated graphs, i.e Attr.asking (Hu et al., 2019) (with 30% attribute masking rate), DropEdge (Rong et al., 2020b) (with 10% edges dropped) and GrfphDifffusin (Klicpera al., 2019) (with heat kernel derive denser adjacency matrix),
15

---

Under review as a conference paper at ICLR 2021
Table Ablation study for three model components on biological function prediction benchmark.
Model Components
Methods
ROC-AUC (%)
AttrMasking
72.8 + 0.4
Correlated Graph Construction
DropEdge
71.5 0.5
GraphDiffusion
70.4 0.7
InfoNCE
72.8 0.4
Loss Format
Hinge Loss
0.8
K-means
0.3
Clustering Algorithm
RPCL
72.8 + 0.4
Adaptive-REE RPPpi
72,9 = 0.3
and evaluate them under the proposed GraphLoG framework. As shown in the first segment of Tab. 5, the AttrMasking strategy outperforms other two techniques with a clear margin, which is mainly ascribed to the fact that, compared with dropping adding edges, masking node attributes can preserve the matching degree of correlated graph pairs to greater extent (referring to the theo- retical analysis in Sec. A).
E.2 ABLATION STUDY ON LOSS FORMAT
We investigate the effect of two loss formats, InfoNCE loss (van den Oord al., 2018) and hinge- loss-based contrastive loss (Hadsell et al., 2006), on our model. The conventional contrastive loss uses negative sample for each positive pair and is in hinge loss form, while the InfoNCE loss employs a large number of negative samples and in the form of softmax. We modify the loss format in Eqs. 6, 7 and 13 conduct the comparison. According the second segment Tab. 5, the InfoNCE loss marginally improve model's performance, and both losses can achieve superior performance under the GraphLoG framework.
E.3 ABLATION STUDY ON CLUSTERING ALGORITHM
We employ three clustering algorithms derive hierarchical prototypes in method and evaluate their corresponding pre-trained models on downstream task. For K-means, in the prototype initial- ization stage, we perform clustering hierarchically in Eqs. 9 and 10 and obtain three hierarchies of prototypes with fixed number M1 10, M2 30 and M3 = 100 for each hierarchy. respectively. Also, we design an adaptive variant of RPCL (Xu et al., 1993), named Adaptive-RPCL, which is able to adjust the number of prototypes during training. Specifically, a counter is additionally maintained for each bottom layer prototype to record the number of iterations from the last time that the prototype is updated by graph embeddings. When counter reaches threshold 100, the corresponding bottom layer prototype is removed, and an upper layer prototype is removed if all the bottom layer prototypes in its corresponding tree are eliminated.
In the third segment of Tab. 5, the performance on biological downstream task is reported for the pre-trained models using different clustering algorithms. It can be observed that two RPCL-based clustering methods marginally outperform K-means, and their performance comparable with each other. These results illustrate that the proposed GraphLoG model is not too sensitive to the selection of clustering algorithm.
E.4 RORUSTNESS CLUSTERING AL.GORITHM
In this experiment, we examine the robustness of RPCL clustering algorithm the proposed GraphLoG model. Specifically, conduct clustering-basse hierarchical prototype initialization for six times and obtain six different self-supervised pre-trane tr-iniiiing models based on distinct initializa- tion. Fig. 4 plots the biological downstream task performance of these six models. We observe that they perform comparably with each other, which demonstrates that the GraphLoG model is fairly robust to different clustering outputs.
16

---

Under review as a conference paper at ICLR 2021
73.0
72.9
72.7
ROC-AUC (%6
72.5
72.0
71.0
Model number
Figure 4: Test ROC-AUC (%) of six models derived by different clustering results biological
downstream task.
F SENSITIVITY ANALYSIS OF ATTRIBUTE MASKING
Avenge (%)
ROC-AUI (%)
71
70
A Attibbute masking rate
Attbbute masking rate
(a) Molecular property prediction
(b) Biological function prediction
Figure 5: Sensitivity analysis attribute masking rate on (a) molecular property prediction bench- marks, where the average test ROC-AUC (%) on eight downstream tasks is reported, and (b) biolog- ical function prediction benchmark.
Setups. When varying the attribute masking rate to evaluate its sensitivity, other hyperparameters are fixed as the values depicted in Sec. 6.1. In specific, the hierarchical prototypes' depth Lp is set as 3, and the exponential decay rate 3 equals to 0.95.
Results. Fig. 5, we plot model's performance on the downstream tasks chemistry and biology domains under different masking rates. The highest test ROC-AUC is achieved when attribute mask- ing rate around 30%. which means that, under such settings, the constructed correlated graphs benefit the proposed learning scheme most.
G TRAINING CURVES
In Fig. 6, we plot the training curves of four approaches, i.e. the random initialization baseline, context prediction (Hu et al., 2019), attribute masking (Hu et al., 2019) and the proposed Graphl .oG method, on eight molecular property prediction tasks. From these line charts, we can observe that, through pre-training on large-scale unlabeled dataset by GraphLoG, GNN model is able to con- verge a higher ROC-AUC on the training set compared with other three methods.
17

---

Under review as a conference paper at ICLR 2021
BBBP
Tox21
110
Random
Contexth red
8
AttMaakiing
noc
3
GraphLoG
25
68
100
25
40
100
Epoch
Epach
ToxC ast
SIDER
ClinTox
(60)
ROC-MIC 060
BOC-AUC
100
Spach
Epochh
Epech
MUV
HIV
BACE
D61
ROCAMC D61
ROC-UU
32
100
100
Epsch
Eosch
Essch
Figure 6: Training curves of different methods on eight downstream tasks of chemistry domain. The ROC-AUC (%) on training set recorded along the training process.
18


Reviews:
[{"review_id": "jSgse61B9ao", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n- The paper proposed a novel self-supervised learning method to embed graphs to vector space. Different from previous methods, the method proposed a global-semantic learning strategy to encourage the embeddings to form a hierarchical clustering structure.  Both the embedding network and the hierarchical structure can be jointly learned.\n\n- Authors have provided extensive and convincing comparison results and numerical analysis to show the effectiveness of the method.\n\n- The paper is well-organized and clearly written. To the best of my knowledge, the proposed method is technically feasible.\n\nCons:\n- The number of prototypes is determined by RPCL and can not be adjusted in training.　\n- Clustering algorithms are usually not very robust. Since the prototypes of GraphLoG is initialized by RPCL, is the performance of GraphLoG robust? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "b4uaKv95lCI", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The motivation and novelty of the proposed method are good. However, the validation is kind of\nweak.\nI can understand that this papers follows the validation in Hu et al (2019), however, I feel that\ntwo tasks (one on chemical benchmark and one on biological benchmark) may not be sufficient\nto give a detailed idea of the improvement of the proposed GraphLoG over other baselines. I\nthink 3-5 tasks are much better.\nFor the ablation study in Section 5.4, these ablated items are good. However, I more would like\nto see fluctuated parts in the proposed GraphLoG. One example may be: is there any different\nchoice/option for hierarchical prototype? Which one is good/bad? What is the reason. Or other\npotential and similar examples exist in the proposed GraphLoG. I think this will help us to\nunderstand GraphLoG more.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y38wFBmjJrW", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an unsupervised framework to perform graph representation learning. The local-instance structure is learned by first gets patch-level and graph-level representations for each graph, then maximize the mutual information between both correlated patches and correlated graphs, which are decided by attribute masking strategy. The global-semantic structure is maintained by leveraging RPCL to derive hierarchical prototypes of the representation and maximizing the mutual information between correlated graph representation and the searching path in the prototypes.\n\nStrengths:\n+  This paper presents a framework to jointly consider the local instance structure and global-semantic structure of graphs. It is a meaningful direction and could be beneficial for explainability.\n+  The experimental results are quite thorough with comparisons to several baseline methods. Moreover, the ablation study of different mechanisms is provided in the experiments.\n\nWeaknesses:\n-  The proposed model seems like a simple combination of several existing techniques and thus lacks novelty.\n-  The performance of this model seems to heavily rely on the attribute masking strategy as all the operations are built upon the correlated graph pairing from the attribute masking strategy. But how reliable is this technique? It seems to be a bottleneck of the model, and I think there should be an explanation on this either theoretically or experimentally.\n\nOverall, the proposes a reasonable model for learning hierarchical graph representations. However, the novelty is limited since the proposed method seems like a simple combination of several existing techniques.\n\nQuestions:\n\n1.  As I mentioned earlier, I wonder how reliable the attribute masking strategy is. As graph matching is an extremely hard problem, can this strategy provide a reliable pairing between correlated graphs?\n2.  It is not clear how to leverage prototypes in classification tasks? I understand that the prototypes serve to ensure a better structure of the embeddings, but when classifying graphs, I wonder whether embeddings and prototypes are both used or not?\n3.  In the Constraint for the global-semantic structure part, the loss for a graph embedding includes both the representations for its correlated graph and its searching path consisting of several prototypes. When minimizing the loss, I wonder both representations of the correlated graph and prototypes are updated together? Or the prototypes are only updated in eq. (11) and (12).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AIwlhBeS8Ij", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a method for self-supervised graph-level representation learning. The main idea is to enforce both the instance level smoothness embedding constraints, and a so-called global, semantic grouping structures across all instance graphs in the training data set.  To achieve this goal, the authors have adopted a global clustering framework to encourage the embedding of the graphs belonging to the same clusters to be close to each other, and by using a hierarchically organized set of prototypes. The proposed method is applied to pre-train GNN on massive unlabeled graphs, which is then fine-tuned to downstream learning tasks.\n\nEnforcing a global clustering structure can be useful in capturing the distribution of large number of graphs in the training data set and hopefully carry the learned representations over to other tasks. However, it appears to me that the paper has combined the carefully devised ideas from too many existing work, each of which alone has shown great success in improving the learning performance. Therefore it can be difficult to judge which part of the choices really leads to the final improvement, and in particular whether it is  the local and global structure preserving part, which seems to be the core theme of the paper (with the other theme being sel-supervised learning), that can fully explain the result.\n\nIn more detail, the authors have used (1) masking strategy by Hu et al., 2019 to generate correlated graph pairs, (2) mutual information estimation technique InfoNCE to enforce the correlation between paired graphs, and (3) Rival Penalized Competitive Learning (RPCL) as the main building block for hierarchical prototype-based learning. Therefore, a natural question to ask is, if one uses plain GNN architecture of each graph and plug it in a (hierarchical) clustering framework (without self-supervised learning and RPCL), whether similar improvements in the learning performance can still be obtained? If not, then the gains are merely due to the effectiveness of these specially designed components from the literatures and not by the general idea of local and global structures. \n\nWith regard to this concern, I would suggest the authors to clarify on what is the main theme of the work and demonstrate that the empirical performance gains are truly due to a novel, focused idea they propose rather than by combining some of the  existing algorithms which have shown great impact and performance gains in their respective context. In the current form, the novelrity of the work seems less significant by introducing so many components from other works.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a self-supervised learning method for learning representations for graph-structured data, with both local and global objectives. The local objective aims to maximize the mutual information between two correlated graphs generated with attribute masking [Hu et al. 19], with the InfoNCE loss [van den Oord et al. 18], and the global objective aims to cluster the graphs using the RPCL [Xu et al. 93] objective, which pulls the sample toward the closest cluster while pushing it away from the rival clusters. The proposed method is validated on standard graph classification benchmarks by training a linear classifier on top of the GNN pre-trained with it, and the results show that it largely outperforms existing graph pre-training methods. 

This paper fell into a borderline case, receiving split reviews with two of the reviewers learning toward rejection, and two others proposing to accept. The reviewers in general agreed that the experimental validation is thorough (except for one reviewer), and some of the reviewers mentioned that the proposed idea of performing self-supervised learning at both local and global level makes sense. However, the negative reviewers were concerned with the limited novelty of the proposed method, since the proposed method seems like a simple combination of two objectives each of which are based on existing ideas (although the latter has not been explored for GNN pre-training). The reviewers had interactive discussions with the authors, and the authors provided detailed feedback. Yet, the reviewers were not convinced that the method has sufficient novelty to warrant publication even after the internal discussion period, and decided to keep their negative ratings.

I believe that this is a simple yet effective pre-training method for GNNs on graph-structured data. The proposed method of combining the local and global objective seems like a promising solution to learn a metric space that well-captures the graph-level similarity and also is well-separated for discriminative classification, and it may have some practical impact given its good performance on benchmark datasets. However, as the two negative reviewers mentioned, the paper in its current form is presented as a simple combination of existing approaches. The local objective is a slight modification of attribute masking strategy of [Hu et al. 19], and the global objective of clustering has been explored in self-supervised learning of CNNs for image data [Asano et al. 20]. Thus, I lean toward rejecting the paper, considering its relative novelty and quality. 

However, I find the proposed work highly promising, and encourage the authors to further develop the method while also improving on the paper writing. I suggest the authors to focus more on the main idea of learning with both local and global objectives, without specifically tying each objective to any of the existing methods. The authors may consider various techniques for both local and global objectives (such as hinge loss-based contrastive loss with k-means clustering as shown in the response to R3), and suggest the proposed work as a more general framework.  

[Asano et al. 20] Self-Labeling via Simultaneous Clustering and Representation Learning, ICLR 2020
Decision: Reject