Similarity-aware Positive Instance Sampling for Graph Contrastive Pre-training
Authors: Xueyi Liu
Yu Rong
Tingyang Xu
Fuchun Sun
Wenbing Huang
Junzhou Huang
Venue: NeurIPS 2021 Submitted (2021)
Keywords: graph neural networks
contrastive learning
graph pre-training

Abstract: Graph instance contrastive learning has been proved as an effective task for Graph Neural Network (GNN) pre-training. However, one key issue may seriously impede the representative power in existing works: Positive instances created by current methods often miss  crucial information of graphs or even yield illegal instances (such as non-chemically-aware graphs in molecular generation). To remedy this issue,  we propose to select positive graph instances directly from existing graphs in the training set, which ultimately maintains the legality and similarity to the target graphs. Our selection is based on certain domain-specific pair-wise similarity measurements as well as sampling from a hierarchical graph encoding similarity relations among graphs. Besides, we develop an adaptive node-level pre-training method to dynamically mask nodes to distribute them evenly in the graph. We conduct extensive experiments on $13$ graph classification and node classification benchmark datasets from various domains. The results demonstrate that the GNN models pre-trained by our strategies can outperform those trained-from-scratch models as well as the variants obtained by existing methods. 

Similarity-aware Positive Instance Sampling for Graph Contrastive Pre-training
Anonymous Author(s)
Affiliation Address email
Abstract
Graph instance contrastive learning has been proved effective task for Graph Neural Network (GNN) pre-training. However, one key issue may seriously impede the representative power in existing works: Positive instances created by current methods often miss crucial information of graphs or even yield illegal instances (such non-chemically-awee
2
4
graphs in molecular generation). To remedy this issue, we propose to select positive graph instances directly from existing graphs in the training set, which ultimately maintains the legality and similarity the target graphs. Our selection is hased on certain domain-specific pair-wise similarity measurements as well as sampling from hierarchical graph encoding similarity relations among graphs. Besides, we develop an adaptive node-level pre-training method to dynamically mask nodes distribute them evenly in the graph. We conduct extensive experiments on 13 graph classification and node classification benchmark datasets from various domains. The results demonstrate that the GNN models pre- trained by our strategies can outperform those trained-frr feined-from-seee firom-scrachh models as well as the variants obtained by existing methods.
8
10
11
12
13
14
15
Introduction
16
17
Pre-training on graph data has received wide interests in recent years, with large range of insightful
18. works focused on learning universal graph structural patterns lying in different kinds of graph For instance, Hu
[1]] pre-train graph neural networks on molecules 20 and transfer the learned model to molecular graph classification tasks, while Qiu et al. [25] pioneer
19 data 25
13
40 28
21 pre- rre-training on big graphs. Compared with traditional semi supervised or supervised training methods
for graph neural networks [12 18 38 10 33), pre-training tasks formulate the training objective without the access training labels, and they empower graph neural networks to be generalized to
23
24 unseen graphs or nodes with no or minor fine-tuning training cost. How to define proper pre-training tasks comes as the principal and also the most challenging part in graph self-supervised learning.
25
28
Among current works, graph instance contrastive learning based pre-training tasks have been proved effective learn graph streutrual information 255 40
27
preforms contrast between positive/negative instance pairs extracted from real graphs observed in the dataset. Though positive pairs for graph contrastive learning seems easy to define for those tasks not performed on graph instances, like DeepWalk [22), node2vec I1I, where near node-node pairs are treated as positive pairs and Infomax based models like DGI [34], InfoGraph 31, where node-graph pairs from a same graph are treated as positive pairs, it is not the case for graph instance contrastive learning. Attempts from previous literature mainly focus on devising suitable graph augmentation methods, such as graph sampling [25] node dropping [40]. edge perturbation [40], and diffusion graph [I3] to get positive graph instances from the original graph. Despite the achievements they have made using such graph data augmentation strategies, assume that such perturbation based graph data augmentation methods
28
29
30
31
33
40).
35
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

37 are not universal strategies to get ideal positive samples preserving necessary information for graph contrastive learning for various kinds of graph data such as molecular graphs, social graphs, and academic graphs.
38
39
We make our assumptions the necessary in- formation that should be preserved in positive in- stances in the contrastive learning process, which though have not been proved theoretically, are rea- sonable and are arrived from the re-thinking of the purpose and inherent principle of the contrastive learning and what should positive samples pre- serve to get an effective method. Such unproved but reasonable assumptions for positive instances are as follows:
0.4 PF1
Logal Prob.
D.
Logal Prob. Sim.
41
powisity Siberry
Sim.
Powiiiltty Sumbery
42
0.3
5-2203
agiie
021
43
02
44
6.
45
0.0255
222254
0,2121
2,0224
0.5211
0.024
46
0.10
0.15
0.25
0.10
0.35
0.25
47
Drop Noda Rado
Drop Edge Radio
48
Figure 1: The fingerprint similarity scores (in or- ange) and percent of legal molecular outputs (in blue) generated by graph data augmentation strategies: drop- ping nodes and dropping edges from same molecule w.r.t. ratio of nodes edges being dropped on 1000 molecular graphs. The fingerprint similarity and the percent of legal molecular outputs decreased dra- matically even for the small proportion of nod/edge dropping-
49
50
Positive instances should be semantically sim- ilar with the target instance; Positive samples for the same target instance should also be similar with each other;
51
52
53
54
Positive instances should preserve certain do- main information if necessary.
55
56
Based on such assumptions, we can see that some widely used graph data augmentation strategies cannot always get positive instances with such properties preserved when being applied on different kinds of graph data. As shown in Fig. [ simple edge-perturbation graph contrastive learning strategies can hardly get legal graph instances given the fact that molecules are specifically formulated in accordance strict chemical constraints which will be easily broken if some edges/nodes, even of a very small number, are perturbed. Moreover, subgraph sampling strategy, though effective when applied on graphs without node/edge attributes, may always lead to positive instances that are dissimilar with the target instance when applied on molecular graphs. Statistical results for subgraph sampling and another data augmentation strategy suffering from similar problems attribute masking, are presented in Appendix A.5.I
57
58
node-dropping for molecular
59
80
61
62
63
64
85
Thus, in this paper, we move beyond the widely used graph data augmentation strategies for an effective and more universal method to get positive graph instances for graph instance contrastive learning. We propose simple but effective similarity based positive instances sampling strategy that can be applied on various kinds of graph data. Unlike previous methods that construct contrastive pairs by graph augmentation, our method encodes the pair-wise similarity information, measured by certain domain-specific similarity/proximity. into hierarchical structure and selects positive graph
67
68
69
70
71
72 instances from such a structure which ultimately maintains the legality of the sampled instances
73 and high similarity to the target graphs (see Appendix C for details). Moreover, we also propose
74 an improvement for a widely-used node-level pre-traiming strategy [1)]. which, together with our
75 similarity aware graph positive sampling strategy, brings us an upper strategy design philosophy. That
76 is, the necessary of introducing prior knowledge or bias in random strategies.
77 We conduct extensive experiments on three representative kinds of graph data: molecular graphs, social graphs as well big social and academic graphs where nodes are of interest demonstrate the effectiveness and superiority of our proposed sampling based strategy over previous graph contrastive learning strategies and also some other strategies not based on contrastive learning for different kinds of graph data. Besides, some additional experiments which try to transfer the GNN models pre- trained on molecular graph dataset to downstream social graph classification task let us have a glimpse of the potential possibility of the pre-trained models' ability to capture universal graph structural information underlying different kinds of graph data as well as the possibility to get such 85 a universally transferable pre-trained model. Similar things have been explored in other domains
78
79
80
81
82
83
84
86 such as multi-lingual language models. However, to our best knowledge, we are the first to propose
87 such possibility for pre-trained GNN models, which, though lacks further and thorough exploration
88 in the paper, can probably point out a new possibly meaningful research direction and cast light on successive work.
2

---

2
Related Works
90
81
Graph Representation Learning. How to generate expressive representation vectors for nodes or graphs that can capture both node-level information, like node attributes and node proximities [32 122 as well as graph-level information, like structural proximity between nodes [26] and graph
93
94
property [TO, is vital question and has aroused great interests from graph learning community,
95
Common approaches include unsupervised manners 122
32
31
34
42
24
23 which always
96
adopt a shallow architecture, semi-supervised and supervised approaches 33
10 38 which
97
always leverage expressive graph neural networks to capture critical information from both graph
structure and node/edge attributes. In this work, we adopt graph neural networks our graph encoder to generate expressive representations for nodes or graphs
96
99
100
Contrastive Learning. Contrastive learning has proved its efficiency to learn highly expressive representations in Computer Vision domain [5 [4]. Moreover, contrastive learning has also been
101
102
used in graph learning for long time, like doing contrast between node-node pairs [22] [1] to encode various node proximities into node representations. Recently, there are also efforts focusing
103
104
on using contrastive learning on graph instances to learn instance-level representations that can be
105
aware of critical graph structural information [25] 40 In this work, we also focus on graph instance contrastive learning, but turn to approach this problem in new manner.
106
107
Graph Pre-training, Pre-trained models have proved their highly transferable ability when being applied on downstream datasets in other domains, such as the language models [6] in NI.P domain. Famous pre-training strategies for GNNs on graph data largely fall into two genres: node-level and graph-level strategies. Node-level strategies aim to design proper tasks that can help GNNs learn
108
109
110
node/edge attribute distribution information [5] 28 More universally, graph-level strategies try to
112
learn design tasks that can learn structural information for both nodes and graphs [25] 40). In this work, we aim to design more powerful pre-training strategies for graph data from both graph-level and leve-
114
3 Preliminary
115
116
We denote an attributed graph as E(V,E, where | = n refers to a set of n nodes and |E| = m refers to a set of m edges. We denote D. E Rd as the initial feature of node o and Euv as the initial feature of edge (u, v).
117
119
Graph Neural Networks (GNNs) can be modeled as the messaging passing process, which involves neighborhood aggregation among nodes in graph and message updating to the next layer. Namely, the general message passing process is defined as: AGGREGATE( ((h T euv) E No.),
120
121
(1+1)
h [+1
a W m(+1) + b(9),
(I+1) where h refers to the hidden state of at (2 + 1)-th layer with h x. and m refers to the aggregated message of u at (2+ 1)-th layer. N₁, denotes the neighbor node set of node v. AGGREGATE(() aggregates the hidden states of v's neighbor nodes and edges, such as mean/max
122
123
124
125
pooling and graph attention 38
33 (-)) is the activation function, such as ReLU(.). ( and (!))
are the trainable parameters. If the model Mr contains layers, the output of last layer (h(")) YeEv usually represents the node-level embeddings of input graph. Moreover, the graph-level embedding hG derived by simply applying a READOUT function as hg READOUT((M(")) (eEN.).
126
127
128
129
Representations generated by GNNs over graphs, including node-level and graph-level representa-
130
tions, are meaningful embeddings to perform various downstream graph learning tasks, like node classification 44 graph classification
131
15 and on.
132 4 Similarity-awarg Positive Graph Instance Sampling
133 In this section, we propose our similarity-awar hierarchical graph positive instance sampling method to sample positive graph instances with three kinds of information mentioned in Sec. preserved. We
3

---

30
Stailatty hraction aust..)
Similarity -besed based hienschical graph
Hlerarchicll sampling
Posttive paiss
mhedm
Similarit estimation
grapt
Figure 2: Illustration of the hierarchical graph instance sampling process for molecular graphs.
other graph instances in the pre-training dataset for positive instances and why it may work for graph data. Then we propose our sampling strategies as well as two versions of the sampling process. We also give some further discussions for such two sampling strategies, Moreover, we also propose an improvement of the widely used node-level pre tre-training strategy, which is an additional contribution of our work.
first explain our motivation w.r.t why
136
137
138
139
4.1
Motivation: Sampling Constructing?
141
As discussed in Sec. D it is hard to design a clean and elegant data augmentation strategy universally for various kinds of graph data to get positive instances that similar enough with the target graph instance and can also preserve necessary domain specific information. Since what we care about for positive graph instances
143
144
their similarity with the target graph instance, rather than the way to obtaining them, we move beyond popular graph data augmentation skills and propose to sample positive instances from the pre-training dataset for the target graph instance. Specifically, we propose
145
146
147
to use approximate similarity functions that can reveal the semantic similarity between two graph instances to some extend to estimate the semantic similarity scores between two graph instances. The similarity relations between each pair of graphs are then encoded into a similarity hierarchy, which is
148
149
150
then used for positive instance sampling. We also make some further discussions for the proposed similarity-awe aware sampling process, which may inspire future design for other sampling strategies.
151
152
4.2
Similarity-awan Positive Graph Instance Sampling
Following 11, assume that each graph instance G3 G has its semantic class class c-- Thus, the optimal positive sampling strategy should choose graph instances of the same semantic class with the graph instance G3 as its positive instances. Formally, the rate for sampling graph G, as the positive instance of G is:
154
155
156
If class(G,) =lass((.)=cass((G).))) class(G,),
ps P,,())) = c
otherwise
157
where G = [GxIGa E G, class(Gr)) class(G;)) is the set of graph instances of the same class with graph instance G, We can then assume that there exists a ground-truth semantic similarity function sim.)(..)) which reveals whether two graph instances belong to same semantic class accurately:
158
159
160
{1
If class(G,) = class(G,), otherwise
(G, (G,,G,) (
161
However, we have no knowledge of the such ground-truth semantic similarity function since our pre- pre-training graph datasets are always unlabeled. Thus we propose to use approximate similarity functions that can be obtained from the real-world and applied in practice easily to estimate the similarity between two graph instances. We can make some assumptions for the chosen approximate similarity functions to ensure their good quality, which are deferred to Appendix B.
162
163
164
165
186
Specifically, we choose a similarity score function sim(:,) :) to estimate the semantic similarity between two graphs. To further the similarity measurement to perform flexible positive sampling, propose a two-step approact[] to encode pair-wise similarity into a more abstract and structural hierarchy efficiently similarity-baa based hierarchical graph HGG EH), where G is the set of graphs in our pre-training dataset, E is the edge set. Formally, we introduce a similarity threshold 7(0 <
188
169
170
Please refer to Appendix A.4.1 for details.

---

1), and based on which the edge set is defined as: Eµ = {[G, C,)ssim(G,,,G) C E.G.GGGG G. E
171
172
G}. Many similarity functions are good candidates for sim (..) such as fingerprint similarity [27 for molecular graphs, Weisfeiler-Lehman Graph Kernel (29) normalized similarity for graphs without
178
174
node/edge attributes and node proximity for nodes in a big graph.
175
The constructed hierarchical graph, which encodes more information beyond pair-wise similarity can be used design flexible sampling strategies for positive graph instance selection. We propose two sampling strategies:
178
177
First-order neighbourhood sampling. For each graph G2, sample one-hop neighbour set of fixed size as its positive instances.
179
High-order graph sampling. We perform I-hops random walks starting from graph G2 for kt times and choose positive instances according to their appearance frequencies.
181
182
An illustration for HGC is presented in Fig. 2
We will give some further discussions w.r.t. why we use similarity for positive instance sampling and how would high-order sampling potentially benefit the sampling process and the resulting positive instances in the next section.
183
185
4.3
Further Discussion for Similarity-wware Sampling Strategy
186
Why we still sample positive instances based on approximate pair-wise similarity scores, though it may not be an accurate similarity estimation? Q2: How would high-order sampling potentially benefit the sampling process and the resulting positive instances? Moreover, we also propose some further discussions for the proposed similarity-aware positive instance sampling strategy.
In this section, we want to answer two questions: Q1:
187
188
189
180
To begin with, we propose a property of the contrastive learning that is intuitively correct:
192
Property 1. Avoiding false- false-positives is important in the contrastive learning process.
193
Here, "false-positives" denotes positive instances selected by a non-optimal positive sampling strategy whose semantic classes are not same with the target graph instance. We explain why such a property holds in Appendix B.2 in detail, though it should be correct intuitively.
194
195
Then, Q1 can be answered by proposing the following property of the positive instances sampled according to their similarity scores with the target graph instance:
197
196
Property 2. If the similarity threshold T is changing in proper range, an instance that has a high similarity score with the target instance will also has high probability to be a ground-truth positive instance
200
201
We would explain why this property holds in detail in Appendix B.3 based on our assumptions on good properties of the approximate similarity function (Def. 2 Thus, the answer for Q1 could be: sampling positive instances according to their similarity scores with the target graph instance may help avoid sampling
202
203
204
To answer Q2, pointing out a crucial property of the ground-trun truth similarity function that the approximate similarity functions always fail to preserve the transitivity of the ground-trudt similarity function:
first propose limitation of the first-order similarity sampling strategy by
206
Property 3 (Transitivity of the ground-trute -ruth similarity function). Ground-truth similarity function is transitive: if sim (G., G,) = and Sim G2) = 1, then (G, G&) = =1
209
210
Such transitivity of the ground-truth similarity function ensures the transitivity of the relations between nodes in the hierarchical graph constructed based on the ground-truth similarity measurement. However, is obvious that relations between nodes in our constructed similarity- based hierarchical graph represented by edges, are not fully-transitive. It is because that the approximate similarity function we use in practice is not an optimal one.
211
212
213
214
We introduce the definition of connectivity and connectivity order between nodes in the graph in 215
216
Appendix [..4] The transitivity of the ground-truth similarity function ensures that G,'s positive instances sampled by first-order neighbourhood sampling strategy can have connectivity orders with
217
G ranging from 1 to | 1. It is hard for first-order sampling strategy applied the hierarchical
218
Such information will be discussed in Sec. 4.3
5

---

219
graph constructed in practice to get positive instances that also have high-order connectivity (e.g.. second-order connectivity) with the target graph instance. The reason is that first-order information cannot reveal high-order information (e.g., high-order connectivity with the target graph instance) in the constructed hierarchical graph, while can fully reveal higher-order connectivity in the constructed hierarchical graph based on ground-truth similarity function (i.e., if graph instance G3 is I-connected to G, then it is 2.., | - connected to G2 as well).
220
222
223
225
We can prove that first-order neighbouring positive instances sampled by second-order sampling process are more likely to be connected with each other (see Appendix [B.4 for details). This can remedy the limitation of the first-order sampling strategy, which cannot guarantee the similarity between positive instances. Moreover, it can be empirically verified that positive instances that are
228
227
228
229
both first-order and second-order connected to the target instance are also more similar with the target instance More details are deferred to Appendix [B.4] We also expect that higher-order sampling process can bring more benefit to the resulting positive instances and worth trying in practice.
230
231
232
Additionally, we propose further discussions w.r.t. how would the changing similarity threshold T influence the balance between the increasing sampling rate estimation accuracy for ground-truth positive instances and the risk of sampling more false-positive instances. Detailed discussions are deferred to Appendix B.5
233
234
235
236
4.4
Adaptive Masking for Node-level Pre-training
237
In this section, we propose our improvement of the widely used attribute masking node-level pre- training strategy: Adaptive Masking, which is designed for attributed graphs only. As introduced in [15, altribute masking lask, which is inspired from "masked language model" (MLM) in NLP, helps the model learn node/edge attribute distribution across the graph. Formally, attribute masking task is defined as:
238
239
240
241
242
Definition 1. (Attribute masking task): Given an attributed graph G(V,E, X), a target node v E V and its corresponding feature vector Be. attribute masking task is first mask subset of the features sub C x. in feature vector x. and produce a new feature vector x'g for node v. Then let a model M make the prediction of the masked feature set sub given the new feature vector x' as input.
244
245
246
Hu et al. [15] follows the same protocol in MLM by uniformly selecting the nodes set from graphs to construct the attribute mask task. But, we argue that the uniform selection may break structural relations among nodes in graphs so that the model may miss critical information for node attribute distribution from such relations. We introduce toy example in the Appendix|A.4o
247
248
249
250
Inspired by Kmean++D2 21, which aims to obtain the good initial centroids with widely separated in space, we also adopt the adaptive masking (AdaM) generate the mask node set within less correlations. In particular, we divide the masking process into T steps. At the first step, we uniformly sample small mask set. Secondly, the masking weight of each candidate node is adaptive by function PScore. The detail of PScore is demonstrate in Algorithm 2 (see Appendix (A.42)). In PScore, for the candidate node v, we calculate the similarity of model output between before and after masking. High similarity indicates that node is not influenced by the mask operation at the current step. resulting in the low correlation between node and current mask set Scur- Finally, we randomly sample a node set K with the probability constructed by masking weight. The algorithmic details are provided in the supplementary material.
252
253
254
255
256
257
258
259
280
According to the adaptive masking operation, we can dynamically adjust the importance of nodes during training and obtain more representative mask node set for the attribute masking task. Such intuition is further discussed in Appendix A.7
262
263
5
Experiments
284
5.1
Experimental Configuration
265
Pretraining Data Collection. We conduct the pretraining on four datasets from various domains:
1). academic and purchusing graphs: we collect four data sources from Deep Graph Library (36) and merge them into one pretraining dataset dubbed AP AP_NF. 2). social graphs: we construct two pretraining datasets termed SF and SF.L_NF. SocS_ NF contains five data sources, while
267

---

Table 1: Experimental results (ROC-AUC) on molecular datasets. The numbers in brackets standard deviations. Numbers in gray are the best results achieved by backbone models. Bold numbers represent the best results by difterent backbones. Bold numbers in green represent the best results over all backbones.
Backhone
Strilegy
SIDER 1427
ClinTox
BACE
BBBP
Tox21
ToxCast
#Moleccles
1478
1513
41127
20399
78.31
8575
#Prediction tacks
2
617
GraphCLL
0.7713k roo
0.7755mmmm
0.70660...19
0.7562
0.6289 0.6289 0053
C.Shrraphh
0,5838
0.6390. 00711
0.7736 dua)
0.73
0.6901
0.2521
0 1.6268maaa
Prol
0.7157
0.002300.222
g.
Infomax
0.644
0.757
0.700% (.7653
D-7674 0.6999
0.6302 (o690
Attr Musk
0.504 (003)
0,80K4 (esh
0.6316ga.00m acor)
0.7657 pu
0.544 00229)
Contest Prod
0.6152 mnt0)
0,8055 (115)
0.
0.7022,, 1880
0.7715gn terry
0.6427
HGC
0.6333g..121
0.8 a.
0. 0132
0
U. 0.r21 (a.00a)
0 o
0.0520
AdaM
0.7777
0.8224
0. 0.7273
HGC Adal
0.818
(1784
0,8428 (184))
0.71 0.7177 (.783))))))))
0. 0.7022 (useday:
0.6631
0.6243 d0we)
0.8403 0906)
0.738 1.70Skia
GCN
AdaM
0.8270 0025)
0.855ho
U. (.33
U. 0.7018,0.ae74
0. 0.7653
HGC AdaM
(.
(.8299
a.01))
a.202)
HGC
0.6286
0.7386uuu24)
0.7722mmm) (366)
0.7129
D.7583
0.6505
GraghSAGE
AdaM
0.81 o130)
0.709
0.8212 (019)
U.7730 (067)
0,09822 0r400
0.7643 0011)
0.0492
HGC_AdGM
0.7812/2.00m) (.8127
0. (.7700
(.7187
0.7010/a.oom:
269
SocL_ NF contains 13 data sources collected from TUDataset [19]. 3). molecular graphs: we use
270
the same pretraining dataset with 2 million molecules in [15] and denote it as MolD. The suffix NF indicates "no feature". Since the data sources have different features, we remove all feature and only pretrain these datasets with HGC. The details are presented in Appendix A.I
272
273
Downstream Tasks, We mainly evaluate the peformance on two tasks, node classfication and
graph classification. For the node classification, we conduct the experiments two datasets, Airport (26) and H-index [1 following the same splitting protocol in [25]. For the graph classification, conduct the experiments on 11 datasets from molecular graph (7 datasets from (37)
275
277
and social graphs (4 datasets from (3)). ). Details of those datasets are deferred to Appendix A.
278
For molecular graph classification, we comprehensively compare our pre-training strategies with recent 6 self-supervised learning strategies for graphs. Among them, Edge_ Pred, Infomax, Attr_ MarkM
Baselines.
280
context_Predd are proposed in [15,, all of which are node-level pre-training strategies. GraphCL 40] and C_ C_SSbbrrahh [25] are graph level contrastive pre-training strategies. For
282
node classification and social network graph classification, we compare our model with the best result of GCC (25) and several other models (i.e., ProNE (42) GraphWave 77 DGK [39], graph2vec 20), InfoGraph B1. DGCNN 43 and GIN (38). Details for the implementation, pre-training and fine-tuning settings of baseline models will be discussed in the Appendix A.2 and A.3
204
Pre-training Settings. We use Adam 17 for optimization with the learning rate of 0.001, 31 0.9, 82 = 0.999 and weight decay of O, lcarning rate warms up over the first 10% steps and then decays linearly. Gradient norm clipping is applied with range [-1,1]. The temperature is set to 0.07 in HGC pre-training stage. The batch size of MolD pre-training is 256. For SocL_NF and mean-pooling
287
288
289
290
SocS_No pre-training, the batch size is 32. For the graph classification task,
281
to get graph-level representations following [15). More pre-training details, including backbones, hyper-parameters and training steps are deferred to Appendix A.2.2
282
Fine-tuning Settings. For each fine-tuning task, we train models for 100 epochs. For graph classification tasks (whether social graphs or molecular graphs), we select the best model by their corresponding validation metrics, while the last model after 100 epochs training on downstream training sets are used for further evaluation on downstream evaluation sets, the same with (55. We adopt micro Fl-score and ROC-AUC as the evaluation measures for differrntt tasks. For molecular dataset, as suggested by (37), we apply three independent randomly initialized runs on each dataset and report the mean and standard deviation. More details are are deferred to Appendix A.2.2
294
295
296
297
298
299
300
5.2
Results of Downstream Tasks
301
5.2.1
Graph Classification
302
We evaluate both HGC and AdaM on 7 popular molecular graph classification datasets and HGC on social network graph classification datasets.
7

---

304
The result of molecular graph classification. For molecular graph classification datasets, we report our pre-training strategies on different backbones, including GIN [8). GCN [88] Graph- SAGE [1]. Meanwhile, since only MolD contain node features, we apply both HGC and AdaM
305
306
307
strategies on the molecular datasets. WIGC_AdaM indicates the combination of two strategies. As shown in Table [ we have the following observations: (1). GIN model pre-trained by our pre-training strategies can consistently outperform those pre-trained by other existing strate- gies, with large margin on most of them. The overall absolute improvement is 2.98% in av- erage. (2). Specially, HGC can consistently outperform those
308
309
311
312
contrastive learning strategies (i.e., GraphCL and C_ Subgraph) It verifies stand point that the graph data augmentation will lose some crucial domain information and compromise the fi- nal performance, while HGC dose not lose such information and leads to better performance. (3). Even though GCN/GraphSAAGE can not
313
315
Table 2: Results on graph classification datasets. The
316
surpass the our pre-trained model on GIN pre- trained model, they still outperform the other pretraining strategy, which reaffirms the effec- liveness of our pre-training strategies, (4). The combined strategy HGC_Adal achieve more
evaluation metric is micro Fl-score. Stratogy
IMDB-B IMDB-M
RDT-B
RDT-M
317
graphs
5000
318
classes
2
2.
3
319
DGK
0.670
0.413
graph2vee
0,711
0.504
0.758
0.479
320
InfoGraph
0.730
0.497
0.825
0,535
321
benefits on GCN and GraphSAGE than that of
DGCNN
0.700
0.478
322
GIN. We conjecture that GIN encodes the addi- tional noise which is introduced by this simple combination due to its strong expressive power.
GIN. GCC (Bess)
GINGNo-PreL
0.756
0.734
0.509
0.433
0.898
0.885
0.530
0.635
323
GIN GIN_HGCISocC HGCISeeS_NFF
0.765
0,474
0,913
0.657
324
GIN_ GIN_ HGCCHHCFSScL
0.756
0.490
0.914
0.652
325
The result of social graph classification. To check the transferability of HGC, we conduct the finetune experiments on two models pretrained by SocL_NF and SocS NF. SocL_NF contains the unlabeled data set used in finetune while SocS NF does not. Table 2 documents the performance
328
327
328
of GIN model pre-trained by HGC on SocL_NF NN and SocS_Ne datasets. Such results show that
329
GIN model pre-trained by HGC achieves the best performance on three out of four datasets. The comparison between GIN HGC and GIN(No-Pret.) also confirms the benefits of HGC. Another
330
331
interesting observation is that the pretrain model based on SocS _NF can obtain the better performance
than than So on two out of four datasets. It implies that HGC dose not just memorize the training samples. can encode the latent structural information from unseen graphs and transfer the knowledge to the downstream tasks.
333
335
5.2.2
Node Classification.
Table 3: Results on node classi- fication datasets. The evaluation
336
We evaluate our model pre-trained by HGC on AP_NF on two down-
metric is micro Fl-score.
337
stream node classification datasets and summarize the results in
Datavets
ASiprrp
H-index
338
Table 3 Among different versions of GCC, the best ones are pre-
V
1190
5000
339
sented. From Table 3 the model pre-trained by our HGC strategy
[
13599
44020
340
can outperform the best GCC model on both datasets. It is worth
GraphWave
ProNE
0.623
0.602
0.691
0.703
341
noting that the pre-training dataset AP a__NF contains only 70k graphs,
Stree
0.602
342
which is much smaller than that of GCC(9M graphs). This verifies the efficiency of HGC in the information extraction.
GCC (Best)
0.683
0,806
HGC(AP 2__F
0.706
0.824
343
5.3
Ablation Study
345
How useful are the proposed self-supervised tasks?
346 To evaluate the contribution of our pre-training strategies, we compare the the performance of the pre-trained model by HGC and AdaM, with the model without any pre-training, each of which shares the same hyper-parameter setting. Results are summarized in Tar backbone GIN. It can be seen clearly
Table 4:
Effectiveness of the pre-
347
training on GIN. Bold numbers for ab- solute improvements larger than 0.05.
348
3/9
No-Pret.
Pret.
Abs. Imp.
350
SIDER ClinTox
0.5637
0.6333
+0,0696
351
that all GIN models benefit from self supervieed pre-training tasks on all datasets. To be more specific. for GIN, absolute
0.6480
0.8134
+0.1654
BACE
0.6653
0,8442
+0.1789
352
HIV
0.7475
0.7853
+0.0378
353
17.9% ROC-AUC increase is observed on the dataset BACE,
BBBP
0.6939
0.7273
+0.0334
16.5% on Clin'Tox, and 6.96% on SIDER, leading to 7.53% on
Tox21
0.7580
0.7770
+0.0190
ToxCast
0.6370
0.6603
+0.0233
355
average. Furthermore, pre-trained models gain larger improve-
ment on datasets of relatively small size (c.g., BACE, ClinTox and SIDER), which is also observed
8

---

357
in [28]. It indicates that sele-vuuer pre- training helps GNN models learn more inherent graph properties, thus getting better performance in small downstream datasets where labeled graphs are scarce.
358
380
Can we transfer pre-trained mod- els downstream datasets that are dramatically different from the pre-training one? It has long been known that the pre-trained model can be generalized to unseen data in pre- training dataset 25
361
Table 5: Results for pretraining transferability on graph classifica- tion datasets. Numbers in red are the negative transfer cases.
362
363
Pretraming Type
Strategy
IMDB-B
IMDB-M
RDT-B
RDT-M
None
GIN.No Pret.)
0.734
0.435
0.885
0.635
365
GIN_GCC (best)
0.756
0.509
0.898
0.530
15
6
28
40)
Social
HGC[SorS
0.474
0.657
367
However, previous literature [25
15
Context Pred (MalD)
HGC(Socl_ NF)
0.756
0.734
0.473
0.490
0.914
0.875
0,635
0.652
40 largely focuses on transferring
S S_Context_ Pred (MoiD)
0,763
0,460
0.818
0.625
389
the pre-trained model to downstream
Molecular
AdaMiMoiD)
HGC(MolD)
0.740
0.768
0.486
0,504
0,880
0.912
0.654
0.656
datasets with similar type of data. Here, what we afC in A6ling 46
371
pre- pre-trained model to the downstream datasets with clearly different type of graphs compared to the
372
ones in the pre-trate mre-trainng dataset? To show this, demonstrate the case from molecular graph to
373
social network graph classification. We pre-train GIN in two different ways: one is pretrained by
374
HGC on two social network graph datasets: SocS_ _NF and SocL_NF, _NF, the other is by HGC, AdaM or as well as Context_Pree Pred or S __Conextt_Pee [5) on the molecular dataset MolD.
375
The results are summarized in Table 5 which offers the following observations: (1). Perhaps surprisingly, our methods including HGC and HGC_Adall enable the models pre-trained on molec- ular graphs to even outperform those pre-trained
377
social graphs. For example, the accuracy of HGC AdaM on IMDB-M (0.509) and RDT-M (0.665) is much better than that of HGC(SocS_ _NF) and HGC(SocL_NF) Apart from the universal graph-level properties, the results also inform that larger pre-training datasets can help the model learn such inherent properties better. (2). Different pre- training strategies could deliver different performance. Models pre-trained by graph-level pre-training strategies or combined strategies (i.e., HGC(M)lD) and HGC HGC_Ada AdaM(M)lD) can always get better results than those pre-trained by node-level strategies (i.e., Mdo and Context_ (eedPPredMMooD which indicates that graph-level pre-training strategies can help the model learn global graph-level properties that can be easily transferred to other domains. (3). We also observe the negative transfer
379
381
382
383
384
385
386
brought by the supervised pretraining in some cases. For instance, S S_Context Pred (MolD I get worse performance than its no supervised trained version Context_
387
388
(MoID) on two datasets: RDT-B and RDT-M. It indicates that simple efforts to learn graph-level properties, such as training with labeled graphs, is probable to be limited in the certain domain, thus performing bad in such cross- domai- transfer tasks. Despite this, our HGC and HGC_AdaM still consistently lead to better performance compared to other pretraining strategies, which, once again, versifies our assumption that our proposed graph contrastive learning strategy can learn more universal, even cross-domain, greel patterns.
389
390
381
392
394
6 Conclusion
395
In this work, we focus developing an effective, efficient and more universal positive instances sampling method that can be applied on many different kinds of graph data for graph instance contrastive learning. We also propose an improvement for widely used node-level pre-training strategy to adaptively select nodes to mask for an even distribution (AdaM). Moreover, we also discover the potential cross-domain transferring ability for the pre-trained GNN models. However, there are still some limitations in our work: 1). Though high-order graph sampling can get positive instances of better quality than those obtained by first-order sampling in analysis, it cannot always outperform the model pre-trained by first-order sampling process. We guess that it is relevant with the pre- training dataset. 2). Just combining HGC and AdaM in simple manner leads to significant improvement. Though we make no further investigation into more effective combination method since it is not the keypoint of the paper, it is a meaningful research direction. 3). We discover the
397
399
401
402
403
404
405
406
407
potential cross-domain transferring ability for pre-trained GNN models. It is an interesting point but
408
no further discussion is made in this paper. However, further relevant investigation interesting and meaningful.
409
Supervised trained by labeled graph dataset after unsupervised pre-training. See 15 for details.

---

References
411
[1] Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09922, 2019.
413
[2] David Arthur and Sergei Vassilvitskii. k-means++: The advantages of careful seeding. Technical report, Stanford, 2006.
415
[3] Guy W Bemis and Mark A Murcko. The properties of known drugs. 1. molecular frameworks. Journal of medicinal chemistry, 39(15):88772289,, 1996.
417
[4]
Smriti Bhagat, Graham Cormode, and S Muthukrishnan. Node classification in social networks In Social network data analytics. 2011.
419
420
[5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607. PMLR, 2020.
421
422
423
[6] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04580 2018.
424
425
428
[7] Claire Donnat, Marinka Zitnik, David Hallac, and Jure Leskovec. Learning structural node embeddings via diffusion wavelets. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1320-1329, 2018.
427
428
[8] Greg Landrum al. Rdkit: Open-source cheminformaties. 2006.
430
[9]
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning Graphs and Manifolds, 2019.
432
[10] Justin Gilmer, Samuel S Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning, pages 1263-1272. PMLR, 2017.
434
[11] Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 855-864, 2016.
436
438
[12] William Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. arXiv preprint arXiv: 2xXvv:700.022) 027006022166.. 2017.
440
[13] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International Conference on Machine Learning, pages 4116-4126. PMLR, 2020.
[14] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
443
444
445
(15) Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks, arXiv preprint arXiv: 1905. 12265, 2019.
447
[16] Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojeiech M Czameekki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint 171 2017.
449
451
[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
453
[18] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint
2016.
10

---

455
[19] Christopher Morris, Nils M. Kriege. Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. In ICML
456
457
2020 Workshop Graph Representation Learning and Beyond (GRL+ 2020), 2020. URL www. www.graye www.grrppheeaning... 1o
458
459
[20]
Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. graph2vec: Learning distributed representations of graphs. arXiv preprint arXiv:1707, 05000, 2017.
480
461
[21] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high perfoommeoee deep learning library. arXiv preprint arXiv:1912.01 7033, 2019.
463
464
[22] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk:
Online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 701-710, 2014.
486
[23] Jiezhong Qiu, Yuxiao Dong, Hao Ma, Jian Li, Kuansan Wang. and Jie Tang. Network embedding as matrix factorization: Unifying deepwalk, line, pte, and node2vec. In Proceedings of the eleventh ACM international conference web search and data mining, pages 459-467, 2015 8.
469
471
[24] Jiezhong Qiu, Yuxiao Dong, Hao Ma, Jian Li, Chi Wang, Kuansan Wang, and Jie Tang. Netsmf: Large-scale network embedding as sparse matrix factorization. In The World Wide Web Conference, pages 1509-1520, 2019.
472
473
474
[25] Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang. Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gee: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1150-1160, 2020.
475
478
477
[26] Leonardo FR Ribeiro, Pedro HP Saverese, and Daniel R Figueiredo. struc2vec: Learning node representations from structural identity. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 385-394, 2017.
479
480
[27] David Rogers and Mathew Hahn. Extended-connectivvn fingerprints. Journal of chemical information and modeling, 50(5):742-754 2010.
482
483
[28]
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Grover: Self-supervised message passing transformer on large-scale molecular data. arXiv preprint arXiv:2007.02: 00835, 2020.
484
485
486
[29]
Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-lhmman graph kernels. Journal of Machine Learning Research, 12(9), 2011.
487
488
[30] Teague Sterling and John J Irwin. Zine 15 ligand discovery for everyone. Journal of chemical information and modeling, 55(1 (1:2(11):2223333) 2015.
490
491
[31] Fan- Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. arXiv preprint arXiv: 1900.000000 01000, 2019.
493
494
[32] Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Large- scale information network embedding. In Proceedings of the 24th international conference on world wide web, pages 1067-1077, 2015.
495
496
487
[33] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 17100.10000. 2017.
498
499
[34] Petar Velickovic, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In ICLR (Poster), 2019.
500
[35] Minjie Wang, Lingfan Yu, Da Zheng, Quan Gan, Yu Gai, Zihao Ye, Mufei Li, Jinjing Zhou, Qi Huang, Chao Ma, et al. Deep graph library: Towards efficient and scalable deep learning on graphs. 2019.
502
11

---

504
[36] Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. Deep graph library: A graph-centric, highly-performant package for graph neural networks. arXiv preprint arXiv: 1, 2019.
505
507
508
[37] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9(2):513-330, 2018.
510
[38] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural 2018.
512
networks? arXiv preprini
513
[39]
Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 1365 1374, 2015.
514
515
516
[40] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang. and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems, 33, 2020.
518
519 [41] Fanjin Zhang, Xiao Liu, Jie Tang, Yuxiao Dong, Peiran Yao, Jie Zhang, Xiaotao Gu, Yan Wang. Bin Shao, Rui Li, et al. Oag: Toward linking large-scale heterogeneous entity graphs. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2585-2595, 2019.
521
523
[42] Jie Zhang, Yuxiao Dong, Yan Wang. Jie Tang, and Ming Ding. Prone: Fast and scalable network representation learning. In IJCAI, volume 19, pages 4278-4284, 2019.
[43] Muhan Zhang. Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
526
527
[44] Shenghuo Zhu, Kai Yu, Yun Chi, and Yihong Gong. Combining content and link for classifica- tion using matrix factorization. In SIGIR, 2007.
529
12

---

Checklist
530
531
1. For all authors...
532
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope? [Yes] The main contribution of this paper is the proposal of an effective and more universal positive instance selection strategy that can be applied on various kinds of graph data in the contrastive learning process. We also propose an improvement of the a widely used node-level pre- vre-tranning strategy to adaptively choose nodes to make them distributed evenly in the graph. Morcover, we discover the potential possibility of the cross-doman transferable ability of the pre-trained GNN models.
534
535
536
538
540
(b) Did you describe the limitations of your work? [Yes] See Sec. c
541
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Sec.
542
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
543
544
2. If you are including theoretical results...
545
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Sec. A.7 We make reasonable assumptions on the possibility density function of the approximate similarity function on the ground-truth positive graph set and negative graph set. Some reasonable approximations are made in the derivation process.
546
547
549
(b) Did you include complete proofs of all theoretical results? [Yes] See Sec. A.7
550
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See Sec. A for descriptions and download links for datasets. Download links for our pre-processed data are shared along with the code. Code is provided with supplemental material. Instructions for reproduction are stated in README.md file in the supplemental material.
552
558
554
555
556
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec.
558
[A.2 for implementation details, including pre-training and fine-tuning configuration and hyper-parameter selection. See Sec. A. for descriptions for datasets and the splitting methods.
560
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We report mean and std values for 3 independently random initialized run for each evaluation process on molecular graph datasets. See
562
563
564
Table and Table 12 for details.
565
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] See Sec. A.2 for hardware configurations.
566
587
588
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
589
(a) If your work uses existing assets, did you citep the creators? [Yes] We provide links for data and code that are from public respiratory we used in our project. We also cite related papers. See Sec. A.T and Sec. A.3
570
571
(b) Did you mention the license of the assets? [Yes] Datasets obtained from published
573
works are with related papers cited. See Sec. Sec.A.I
574
(c) Did you include any new assets either in the supplemental material or as a URL? [No] No new datasets are proposed.
575
576
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [Yes] Download links for public datasets we used are provided. Datasets obtained from published works are with related papers cited. See Sec. Sec.
577
578
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] Datasets we use are obtained from public datasets, containing such information. We provide links for them in Sec. A.1
580
5. If you used crowdsourcing conducted research with human subjects...
13

---

583
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
585
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount
588
spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "zIBK9biYSdK", "reviewer": "Reviewer_qU31", "summary": "In this paper, the authors argue that existing graph unsupervised pre-training models may miss some important information when selecting positive samples, or the reconstructed graphs do not meet some specifications. Based on this motivation, the authors design a similarity-aware sampler to automatically select positive samples from the existing samples for graph contrastive learning. Experiments are also conducted to demonstrate the effectiveness of the method.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: The problem faced by the paper is interesting and timely and the proposed approach seems reasonable. The article is well written, the method is clearly described, and the overall quality is good. The authors also provide the source code to facilitate experimental replication.\n\nWeakness: \n1.\tRegarding the adaptive masking part, the authors' work is incremental, and there have been many papers on how to do feature augmentation, such as GraphCL[1], GCA[2]. The authors do not experiment with widely used datasets such as Cora, Citeseer, ArXiv, etc. And they did not compare with better baselines for node classification, such as GRACE[3], GCA[2], MVGRL[4], etc. I think this part of the work is shallow and not enough to constitute a contribution. The authors should focus on the main contribution, i.e., graph-level contrastive learning, and need to improve the node-level augmentation scheme.\n2.\tIn the graph classification task, the compared baseline is not sufficient, such as MVGRL[4], gpt-gnn[5] are missing. I hope the authors could add more baselines of graph contrastive learning and test them on some common datasets.\n3.\tI am concerned whether the similarity-aware positive sample selection will accelerate GNN-based encoder over-smoothing, i.e., similar nodes or graphs will be trained with features that converge excessively and discard their own unique features. In addition, whether selecting positive samples in the same dataset without introducing some perturbation noise would lead to lower generalization performance. The authors experimented with the transfer performance of the model on the graph classification task, though it still did not allay my concerns about the model generalization. I hope there will be more experiments on different downstream tasks and across different domains.\nRemarks: 1. The authors seem to have over-compressed the line spacing and abused vspace. \n2. Table 5 is collapsed.\n\n\n[1] Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen, “Graph contrastive learning with augmentations,” Advances in Neural Information Processing Systems, vol. 33, 2020.\n[2] Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang, “Graph contrastive learning with adaptive augmentation,” arXiv preprint arXiv:2010.14945, 2020.\n[3] Y. Zhu, Y. Xu, F. Yu, Q. Liu, S. Wu, and L. Wang, “Deep graph contrastive representation learning,” arXiv preprint arXiv:2006.04131, 2020.\n[4] Hassani, Kaveh, and Amir Hosein Khasahmadi. \"Contrastive multi-view representation learning on graphs.\" International Conference on Machine Learning. PMLR, 2020.\n[5] Hu, Ziniu, et al. \"Gpt-gnn: Generative pre-training of graph neural networks.\" Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors elaborate on the article's limitations and social impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "spUu744aHn5", "reviewer": "Reviewer_6nrq", "summary": "The authors of the paper proposed to select positive graph instances (in graph pre-training) directly from existing graphs\nin the training set in order to make them more legitimate.  The proposed scheme is based on pair-wise similarity\nmeasurements as well as sampling from a hierarchical graph encoding similarity relations among graph data sets. An adaptive node-level pre-training method is used to for noe-;evel pretraining tasks as well. Extensive experiments on 13 graph classification and node classification benchmark datasets from various domains are reported to demonstrate the usefulness of the methods. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors proposed to use exisent samples in the graph data sets to select positive samples, by carefully designing a similarity-based sampling strategy, in order to improve the resultant pre-training step.  The idea is also used in node-level pre-training tasks. The idea is simple and makes some sense, but there are a numbe rof concerns\n\n(1) There are a number of grammar mistakes and the writing needs to be improved. Sentences are too long to understand quickly and precisely.\n\n(2) Is there any advantage of perturbation-based method for existing method in generating positive samples? For example, is the proposed method for generating positive pairs limited by the size of the graph data set?\n\n(3) The proposed method can be deemed as applying node-level contrastive learning methods (such as deep walk, t-SNE, etc.) in graph-level pre-training tasks; the only difference is that  rather than using proximity relation (such as spatial distance between two nodes in a graph, or co-occurrence of two image patches in a single image), the similarity metric has to be used, since there is no clear prosximity relation defined among graph-level objects. But this is exactly what makes the proposed method very similar to existing embedding methods like t-SNE, and the only difference  is that how to exploit the similarities are defined in more refined manner. Therefore, the novelty of the proposed method is limited. Please note that proximity-based positive/negative sample pairs are very much different from similariy-based ones; in the former case, proximity and similarity can be consistent or conflictive, and they interact with each other; but in the latter case, there is only similarity to be discussed and its scope is therefore not much different than embedding method that relies solely on similarities. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As mentioned above, the proposed idea is more or less a reuse of similarity-based embedding methods. Although authors claim that it is a contrastive learning framework, it is indeed lacking the important interaction between proximity and similarity relations, therefore its close resemblance with embedding methods such as t-SNE, deep walk makes the novelty of the paper very limited.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "9MZ5ejWGCYw", "reviewer": "Reviewer_y4eS", "summary": "- The position of this work relates to graph-level contrastive learning. The paper aims to introduce a better way to sample the positive instances, avoiding illegal or poor-quality positive instances.\n\n- The authors propose to select positive graph instances directly from existing graphs in the training set, which can maintain the legality and similarity to the target graphs. \n\n- The positive sample selection is done on domain-specific pair-wise similarity measurements, sampling from a hierarchical graph encoding similarity relations among graphs. \n\n- A side idea using adaptively select nodes to mask for even distribution has been proposed.\n\n- Extensive experiments on various graph classification and node classification benchmark datasets have shown the effectiveness of this simple design’s impact on the final pre-train performance. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Interesting point\n\nThis paper mainly aim to discuss the impact of the quality of the positive sample when using contrastive objective to pre-train GNN models. This research angel is interesting and allows the pre-train GNN step to learn more effectively with the quality improved positive samples. \n\n### Main concern\n\n**Methodology aspect**:\n- The approach proposed in the paper seems to be a small incremental change on top of the previous GNN pre-train work. The novelty aspect is low.\n\n- A lot of important points only discussed in the appendix. Going back and forth between the main paper and the appendix makes the reader a bit hard to read through the paper.\n- Fails to discuss a easy fix to the poor positive sample quality. If we increase the positive sample number, will it be able to match the performance with the proposed high quality (similarity aware) positive instance sampling strategy.\n\n**Experimental result**:\n\n- Fail to discuss the trade off between the performance improvement gain vs. the introduced computation complexity by computing the similarity  measurement between the positive sampled vs. the original graph. \n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- The authors have discussed the limitation of their work to some degree. But the authors fail to discuss one big limitation with is their limitation in terms of the computation complexity. I think the authors should address the trade-off between the performance improvement gain vs. the introduced computation complexity by computing the similarity measurements. \n\n- I do not believe this work has a negative social impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "1eeSyHy9Azi", "reviewer": "Reviewer_bttS", "summary": "The authors propose a new method for graph pre-training based on instance contrastive learning. They exploit graphs in the training set, to select positive instances that are similar to target graphs. They also propose a new strategy for node-level pre-training. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution is the novel pre-training strategy introduced. The work has potential high impact in the research area, given that the issue of selecting appropriate graphs and datasets for pre-training is still an open and relevant question. The proposed strategy ensured that structurally similar graphs are used for pre-training. An important experimental contribution is also the transfer across domains. \n\nOverall, while technical details are properly provided, I found the general structure of the paper difficult to follow. As a general comment, I would recommend to make the text more smooth and highlight the key steps from sampling to pre-training. \n\nDetailed comments\n\n1) The sampling strategy and concept of the neighbour graph should be better described. The relations with similarity score and specific pre-training strategy should be highlighted clearly. \n\n2) It would also be useful to describe in more details, how the sampling strategy is practically used in pre-training. Maybe a schematic figure would help. \n\n3) I do not see the need of Table 4. The non pre-trained model can be included in table 1. This would also make easier for the reader to validate the effectiveness of the method. \n\n4) The results for no pre-training of table 2 and 3 seem to be missing\n\n---------------------------------------------------------\n\nAfter authors response\n\nThank you for your response. I still think the paper needs some restructuring in order to better describe the contribution and improve the clarity of the text. I will keep my score as it is.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitation and societal (broader) impact are adequately addressed", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to select positive graph instances directly from existing graphs in the training set for constrastive learning, using pair-wise similarity measurements as well as sampling from a hierarchical graph encoding similarity relations. It also develops an adaptive node-level pre-training method. Experimental results show that the GNN models pre-trained by the proposed strategies can outperform baselines. 

The reviewers pointed several concerns/aspects for improvement, mainly:
- relative contribution over existing work
- discussion about the baseline of using more positive instances
- discussion on the tradeoff between the gain and the computational overhead
- clarity of presentation
- experiments: more baselines and datasets

The authors have made good responses that significantly strengthened the work. After the response, the consensus is:
- The authors have provided detailed and meaningful discussions on the baseline of using more positive instances, and the tradeoff between the gain and the computational overhead
- The presentation needs improvement
- The authors have provided results of two more baselines that are helpful, while more results will further strengthen the work.
- The novelty/relative contribution of the work remains a concern for the reviewers. Better presentation (importance of the positive samples' quality,relation with existing work) and more thorough and organized experimental evaluation will help illustrate the contribution. 


Decision: Reject