Similarity-aware Positive Instance Sampling for Graph Contrastive Pre-training

Graph instance contrastive learning has been proved as an effective task for Graph1 Neural Network ( GNN ) pre-training . However , one key issue may seriously impede2 the representative power in existing works : Positive instances created by current3 methods often miss crucial information of graphs or even yield illegal instances4 ( such as non-chemically-aware graphs in molecular generation ) . To remedy this5 issue , we propose to select positive graph instances directly from existing graphs6 in the training set , which ultimately maintains the legality and similarity to the7 target graphs . Our selection is based on certain domain-specific pair-wise similarity8 measurements as well as sampling from a hierarchical graph encoding similarity9 relations among graphs . Besides , we develop an adaptive node-level pre-training10 method to dynamically mask nodes to distribute them evenly in the graph . We11 conduct extensive experiments on 13 graph classification and node classification12 benchmark datasets from various domains . The results demonstrate that the GNN13 models pre-trained by our strategies can outperform those trained-from-scratch14 models as well as the variants obtained by existing methods.15 1 Introduction16 Pre-training on graph data has received wide interests in recent years , with a large range of insightful17 works focused on learning universal graph structural patterns lying in different kinds of graph18 data [ 25 , 15 , 40 , 28 ] . For instance , Hu et al . [ 15 ] pre-train graph neural networks on molecules19 and transfer the learned model to molecular graph classification tasks , while Qiu et al . [ 25 ] pioneer20 pre-training on big graphs . Compared with traditional semi-supervised or supervised training methods21 for graph neural networks [ 12 , 18 , 38 , 10 , 33 ] , pre-training tasks formulate the training objective22 without the access of training labels , and they empower graph neural networks to be generalized to23 unseen graphs or nodes with no or minor fine-tuning training cost . How to define proper pre-training24 tasks comes as the principal and also the most challenging part in graph self-supervised learning.25 Among current works , graph instance contrastive learning based pre-training tasks have been proved26 effective to learn graph strcutrual information [ 25 , 40 ] . It preforms contrast between positive/negative27 instance pairs extracted from real graphs observed in the dataset . Though positive pairs for graph28 contrastive learning seems easy to define for those tasks not performed on graph instances , like29 DeepWalk [ 22 ] , node2vec [ 11 ] , where near node-node pairs are treated as positive pairs and Infomax30 based models like DGI [ 34 ] , InfoGraph [ 31 ] , where node-graph pairs from a same graph are treated31 as positive pairs , it is not the case for graph instance contrastive learning . Attempts from previous32 literature mainly focus on devising suitable graph augmentation methods , such as graph sampling [ 25,33 40 ] , node dropping [ 40 ] , edge perturbation [ 40 ] , and diffusion graph [ 13 ] to get positive graph34 instances from the original graph . Despite the achievements they have made using such graph data35 augmentation strategies , we assume that such perturbation based graph data augmentation methods36 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . are not universal strategies to get ideal positive samples preserving necessary information for graph37 contrastive learning for various kinds of graph data such as molecular graphs , social graphs , and38 academic graphs.39 0.10 0.15 0.20 0.25 Drop Edge Ratio 0.0 0.1 0.2 0.3 0.4 0.5 Po ss ib ilit y / S im ila rit y 0.0218 0.0224 0.0211 0.0224 0.4890 0.3890 0.3163 0.2736 Legal Prob . Avg . Sim . 0.10 0.15 0.20 0.25 Drop Node Ratio 0.1 0.2 0.3 0.4 Po ss ib ilit y / S im ila rit y 0.0265 0.0254 0.0291 0.0345 0.4008 0.3030 0.2425 0.2118 Legal Prob . Avg . Sim . We make our assumptions on the necessary in-40 formation that should be preserved in positive in-41 stances in the contrastive learning process , which42 though have not been proved theoretically , are rea-43 sonable and are arrived from the re-thinking of the44 purpose and inherent principle of the contrastive45 learning and what should positive samples pre-46 serve to get an effective method . Such unproved47 but reasonable assumptions for positive instances48 are as follows:49 • Positive instances should be semantically sim-50 ilar with the target instance ; 51 • Positive samples for the same target instance52 should also be similar with each other ; 53 • Positive instances should preserve certain do-54 main information if necessary.55 2 Related Works90 Graph Representation Learning . How to generate expressive representation vectors for nodes or91 graphs that can capture both node-level information , like node attributes and node proximities [ 32,92 22 , 11 ] , as well as graph-level information , like structural proximity between nodes [ 26 ] and graph93 property [ 10 ] , is a vital question and has aroused great interests from graph learning community.94 Common approaches include unsupervised manners [ 22 , 11 , 32 , 31 , 34 , 42 , 24 , 23 ] , which always95 adopt a shallow architecture , semi-supervised and supervised approaches [ 33 , 18 , 12 , 10 , 38 ] , which96 always leverage expressive graph neural networks to capture critical information from both graph97 structure and node/edge attributes . In this work , we adopt graph neural networks as our graph encoder98 to generate expressive representations for nodes or graphs.99 Contrastive Learning . Contrastive learning has proved its efficiency to learn highly expressive100 representations in Computer Vision domain [ 5 , 14 ] . Moreover , contrastive learning has also been101 used in graph learning for a long time , like doing contrast between node-node pairs [ 22 , 11 ] to102 encode various node proximities into node representations . Recently , there are also efforts focusing103 on using contrastive learning on graph instances to learn instance-level representations that can be104 aware of critical graph structural information [ 25 , 40 ] . In this work , we also focus on graph instance105 contrastive learning , but turn to approach this problem in a new manner.106 Graph Pre-training . Pre-trained models have proved their highly transferable ability when being107 applied on downstream datasets in other domains , such as the language models [ 6 ] in NLP domain.108 Famous pre-training strategies for GNNs on graph data largely fall into two genres : node-level and109 graph-level strategies . Node-level strategies aim to design proper tasks that can help GNNs learn110 node/edge attribute distribution information [ 15 , 28 ] . More universally , graph-level strategies try to111 learn design tasks that can learn structural information for both nodes and graphs [ 25 , 40 ] . In this112 work , we aim to design more powerful pre-training strategies for graph data from both graph-level113 and node-level.114 3 Preliminary115 We denote an attributed graph as G ( V , E , X ) , where |V| = n refers to a set of n nodes and |E| = m116 refers to a set of m edges . We denote xv ∈ Rd as the initial feature of node v and euv as the initial117 feature of edge ( u , v ) .118 Graph Neural Networks ( GNNs ) can be modeled as the a messaging passing process , which involves119 neighborhood aggregation among nodes in graph and message updating to the next layer . Namely,120 the general message passing process is defined as:121 m ( l+1 ) v = AGGREGATE ( { ( h ( l ) v , h ( l ) u , euv ) |u ∈ Nv } ) , hl+1v = σ ( W ( l ) m ( l+1 ) v + b ( l ) ) , where hl+1v refers to the hidden state of v at ( l + 1 ) -th layer with h ( 0 ) v = xv and m ( l+1 ) v refers122 to the aggregated message of v at ( l + 1 ) -th layer . Nv denotes the neighbor node set of node v.123 AGGREGATE ( · ) aggregates the hidden states of v ’ s neighbor nodes and edges , such as mean/max124 pooling and graph attention [ 38 , 33 ] . σ ( · ) is the activation function , such as ReLU ( · ) . W ( l ) and b ( l ) 125 are the trainable parameters . If the modelML contains L layers , the output of last layer { h ( L ) v } v∈v126 usually represents the node-level embeddings of input graph . Moreover , the graph-level embedding127 hG is derived by simply applying a READOUT function as128 hG = READOUT ( { h ( L ) v } v∈Nv ) . Representations generated by GNNs over graphs , including node-level and graph-level representa-129 tions , are meaningful embeddings to perform various downstream graph learning tasks , like node130 classification [ 44 , 4 ] , graph classification [ 31 , 15 , 28 ] , and so on.131 4 Similarity-aware Positive Graph Instance Sampling132 In this section , we propose our similarity-aware hierarchical graph positive instance sampling method133 to sample positive graph instances with three kinds of information mentioned in Sec . 1 preserved . We134 first explain our motivation w.r.t . why we turn to other graph instances in the pre-training dataset135 for positive instances and why it may work for graph data . Then we propose our sampling strategies136 as well as two versions of the sampling process . We also give some further discussions for such137 two sampling strategies . Moreover , we also propose an improvement of the widely used node-level138 pre-training strategy , which is an additional contribution of our work.139 4.1 Motivation : Sampling or Constructing ? 140 As discussed in Sec . 1 , it is hard to design a clean and elegant data augmentation strategy universally141 for various kinds of graph data to get positive instances that are similar enough with the target graph142 instance and can also preserve necessary domain specific information . Since what we care about143 for positive graph instances are their similarity with the target graph instance , rather than the way144 to obtaining them , we move beyond popular graph data augmentation skills and propose to sample145 positive instances from the pre-training dataset for the target graph instance . Specifically , we propose146 to use approximate similarity functions that can reveal the semantic similarity between two graph147 instances to some extend to estimate the semantic similarity scores between two graph instances . The148 similarity relations between each pair of graphs are then encoded into a similarity hierarchy , which is149 then used for positive instance sampling . We also make some further discussions for the proposed150 similarity-aware sampling process , which may inspire future design for other sampling strategies.151 4.2 Similarity-aware Positive Graph Instance Sampling152 Following [ 1 ] , we assume that each graph instance Gi ∈ G has its semantic class class ( Gi ) = ci.153 Thus , the optimal positive sampling strategy should choose graph instances of the same semantic154 class with the graph instance Gi as its positive instances . Formally , the rate for sampling graph Gj as155 the positive instance of Gi is:156 P+i ( Gj ) = { 1 |G+i | If class ( Gi ) = class ( Gj ) , 0 otherwise ( 1 ) where G+i = { Gk|Gk ∈ G , class ( Gk ) = class ( Gi ) } is the set of graph instances of the same class157 with graph instance Gi . We can then assume that there exists a ground-truth semantic similarity158 function simgt ( · , · ) which reveals whether two graph instances belong to a same semantic class159 accurately:160 simgt ( Gi , Gj ) = { 1 If class ( Gi ) = class ( Gj ) , 0 otherwise ( 2 ) However , we have no knowledge of the such ground-truth semantic similarity function since our161 pre-training graph datasets are always unlabeled . Thus we propose to use approximate similarity162 functions that can be obtained from the real-world and applied in practice easily to estimate the163 similarity between two graph instances . We can make some assumptions for the chosen approximate164 similarity functions to ensure their good quality , which are deferred to Appendix B.1.165 Specifically , we choose a similarity score function sim ( · , · ) to estimate the semantic similarity between166 two graphs . To further use the similarity measurement to perform flexible positive sampling , we167 propose a two-step approach1 to encode pair-wise similarity into a more abstract and structural168 hierarchy efficiently – a similarity-based hierarchical graphH ( G , EH ) , where G is the set of graphs in169 our pre-training dataset , EH is the edge set . Formally , we introduce a similarity threshold τ ( 0 < τ < 170 1Please refer to Appendix A.4.1 for details . 1 ) , and based on which the edge set is defined as : EH = { ( Gi , Gj ) |sim ( Gi , Gj ) ≥ τ , Gi ∈ G , Gj ∈171 G } . Many similarity functions are good candidates for sim ( · , · ) such as fingerprint similarity [ 27 ] for172 molecular graphs , Weisfeiler-Lehman Graph Kernel [ 29 ] normalized similarity for graphs without173 node/edge attributes and node proximity for nodes in a big graph.174 The constructed hierarchical graph , which encodes more information beyond pair-wise similarity2,175 can be used to design flexible sampling strategies for positive graph instance selection . We propose176 two sampling strategies:177 • First-order neighbourhood sampling . For each graph Gi , sample a one-hop neighbour set of a178 fixed size as its positive instances.179 • High-order graph sampling . We perform l-hops random walks starting from graph Gi for k180 times and choose positive instances according to their appearance frequencies.181 An illustration for HGC is presented in Fig . 2 . We will give some further discussions w.r.t . why we182 use similarity for positive instance sampling and how would high-order sampling potentially benefit183 the sampling process and the resulting positive instances in the next section.184 4.3 Further Discussion for Similarity-aware Sampling Strategy185 In this section , we want to answer two questions : Q1 : Why we still sample positive instances based on186 approximate pair-wise similarity scores , though it may not be an accurate similarity estimation ? Q2:187 How would high-order sampling potentially benefit the sampling process and the resulting positive188 instances ? Moreover , we also propose some further discussions for the proposed similarity-aware189 positive instance sampling strategy.190 To begin with , we propose a property of the contrastive learning that is intuitively correct:191 Property 1 . Avoiding false-positives is important in the contrastive learning process.192 Here , “ false-positives ” denotes positive instances selected by a non-optimal positive sampling strategy193 whose semantic classes are not same with the target graph instance . We explain why such a property194 holds in Appendix B.2 in detail , though it should be correct intuitively.195 Then , Q1 can be answered by proposing the following property of the positive instances sampled196 according to their similarity scores with the target graph instance:197 Property 2 . If the similarity threshold τ is changing in a proper range , an instance that has a high198 similarity score with the target instance will also has a high probability to be a ground-truth positive199 instance.200 We would explain why this property holds in detail in Appendix B.3 , based on our assumptions on201 good properties of the approximate similarity function ( Def . 2 ) . Thus , the answer for Q1 could be:202 sampling positive instances according to their similarity scores with the target graph instance may203 help avoid sampling false-positives.204 To answer Q2 , we first propose one limitation of the first-order similarity sampling strategy by205 pointing out a crucial property of the ground-truth similarity function that the approximate similarity206 functions always fail to preserve – the transitivity of the ground-truth similarity function:207 Property 3 ( Transitivity of the ground-truth similarity function ) . Ground-truth similarity function is208 transitive : if simgt ( Gi , Gj ) = 1 and simgt ( Gi , Gk ) = 1 , then simgt ( Gj , Gk ) = 1.209 Such transitivity of the ground-truth similarity function ensures the transitivity of the relations210 between nodes in the hierarchical graph constructed based on the ground-truth similarity measurement.211 However , it is obvious that relations between nodes in our constructed similarity-based hierarchical212 graph – represented by edges , are not fully-transitive . It is because that the approximate similarity213 function we use in practice is not an optimal one.214 We introduce the definition of connectivity and connectivity order between nodes in the graph in215 Appendix B.4 . The transitivity of the ground-truth similarity function ensures that Gi ’ s positive216 instances sampled by first-order neighbourhood sampling strategy can have connectivity orders with217 Gi ranging from 1 to |G+i | − 1 . It is hard for first-order sampling strategy applied on the hierarchical218 2Such information will be discussed in Sec . 4.3. graph constructed in practice to get positive instances that also have high-order connectivity ( e.g.,219 second-order connectivity ) with the target graph instance . The reason is that first-order information220 can not reveal high-order information ( e.g. , high-order connectivity with the target graph instance ) 221 in the constructed hierarchical graph , while it can fully reveal higher-order connectivity in the222 constructed hierarchical graph based on ground-truth similarity function ( i.e. , if a graph instance Gj223 is 1-connected to Gi , then it is 2 , 3 , ... , |G+i | − 1 connected to Gi as well ) .224 We can prove that first-order neighbouring positive instances sampled by second-order sampling225 process are more likely to be connected with each other ( see Appendix B.4 for details ) . This can226 remedy the limitation of the first-order sampling strategy , which can not guarantee the similarity227 between positive instances . Moreover , it can be empirically verified that positive instances that are228 both first-order and second-order connected to the target instance are also more similar with the target229 instance . More details are deferred to Appendix B.4 . We also expect that higher-order sampling230 process can bring more benefit to the resulting positive instances and worth trying in practice.231 Additionally , we propose further discussions w.r.t . how would the changing similarity threshold τ232 influence the balance between the increasing sampling rate estimation accuracy for ground-truth233 positive instances and the risk of sampling more false-positive instances . Detailed discussions are234 deferred to Appendix B.5.235 4.4 Adaptive Masking for Node-level Pre-training236 In this section , we propose our improvement of the widely used attribute masking node-level pre-237 training strategy : Adaptive Masking , which is designed for attributed graphs only . As introduced238 in [ 15 ] , attribute masking task , which is inspired from “ masked language model ” ( MLM ) in NLP,239 helps the model learn node/edge attribute distribution across the graph . Formally , attribute masking240 task is defined as:241 Definition 1 . ( Attribute masking task ) : Given an attributed graph G ( V , E , X ) , a target node v ∈ V242 and its corresponding feature vector xv , attribute masking task is first to mask a subset of the features243 xsub ⊆ xv in feature vector xv and produce a new feature vector x′v for node v. Then let a modelM244 to make the prediction of the masked feature set xsub given the new feature vector x′v as input.245 Hu et al . [ 15 ] follows the same protocol in MLM by uniformly selecting the nodes set from graphs246 to construct the attribute mask task . But , we argue that the uniform selection may break structural247 relations among nodes in graphs so that the model may miss critical information for node attribute248 distribution from such relations . We introduce a toy example in the Appendix A.4.2.249 Inspired by Kmean++ [ 2 ] , which aims to obtain the good initial centroids with widely separated250 in space , we also adopt the adaptive masking ( AdaM ) to generate the mask node set within less251 correlations . In particular , we divide the masking process into T steps . At the first step , we uniformly252 sample a small mask set . Secondly , the masking weight of each candidate node is adaptive by function253 PScore . The detail of PScore is demonstrate in Algorithm 2 ( see Appendix A.4.2 ) . In PScore , for the254 candidate node v , we calculate the similarity of model output between before and after masking . High255 similarity indicates that node v is not influenced by the mask operation at the current step , resulting256 in the low correlation between node v and current mask set Scur . Finally , we randomly sample a node257 set K with the probability constructed by masking weight . The algorithmic details are provided in the258 supplementary material.259 According to the adaptive masking operation , we can dynamically adjust the importance of nodes260 during training and obtain a more representative mask node set for the attribute masking task . Such261 intuition is further discussed in Appendix A.7.262 5 Experiments263 5.1 Experimental Configuration264 Pretraining Data Collection . We conduct the pretraining on four datasets from various domains:265 1 ) . academic and purchasing graphs : we collect four data sources from Deep Graph Library [ 36 ] 266 and merge them into one pretraining dataset dubbed AP_NF . 2 ) . social graphs : we construct two267 pretraining datasets termed SocS_NF and SocL_NF . SocS_NF contains five data sources , while268 SocL_NF contains 13 data sources collected from TUDataset [ 19 ] . 3 ) . molecular graphs : we use269 the same pretraining dataset with 2 million molecules in [ 15 ] and denote it as MolD . The suffix NF270 indicates “ no feature ” . Since the data sources have different features , we remove all feature and only271 pretrain these datasets with HGC . The details are presented in Appendix A.1.272 Downstream Tasks . We mainly evaluate the peformance on two tasks , node classfication and273 graph classification . For the node classification , we conduct the experiments on two datasets,274 US-Airport [ 26 ] and H-index [ 41 ] following the same splitting protocol in [ 25 ] . For the graph275 classification , we conduct the experiments on 11 datasets from molecular graph ( 7 datasets from [ 37 ] ) 276 and social graphs ( 4 datasets from [ 39 ] ) . Details of those datasets are deferred to Appendix A.1.277 Baselines . For molecular graph classification , we comprehensively compare our pre-training278 strategies with recent 6 self-supervised learning strategies for graphs . Among them , Edge_Pred,279 Infomax , Attr_Mask , Context_Pred , are proposed in [ 15 ] , all of which are node-level pre-training280 strategies . GraphCL [ 40 ] and C_Subgraph [ 25 ] are graph level contrastive pre-training strategies . For281 node classification and social network graph classification , we compare our model with the best result282 of GCC [ 25 ] and several other models ( i.e. , ProNE [ 42 ] , GraphWave [ 7 ] , DGK [ 39 ] , graph2vec [ 20 ] ,283 InfoGraph [ 31 ] , DGCNN [ 43 ] and GIN [ 38 ] ) . Details for the implementation , pre-training and284 fine-tuning settings of baseline models will be discussed in the Appendix A.2 and A.3.285 Pre-training Settings . We use Adam [ 17 ] for optimization with the learning rate of 0.001 , β1 =286 0.9 , β2 = 0.999 and weight decay of 0 , learning rate warms up over the first 10 % steps and then287 decays linearly . Gradient norm clipping is applied with range [ −1 , 1 ] . The temperature τ is set288 to 0.07 in HGC pre-training stage . The batch size of MolD pre-training is 256 . For SocL_NF and289 SocS_NF pre-training , the batch size is 32 . For the graph classification task , we use mean-pooling290 to get graph-level representations following [ 15 ] . More pre-training details , including backbones,291 hyper-parameters and training steps are deferred to Appendix A.2.2.292 Fine-tuning Settings . For each fine-tuning task , we train models for 100 epochs . For graph293 classification tasks ( whether social graphs or molecular graphs ) , we select the best model by their294 corresponding validation metrics , while the last model after 100 epochs training on downstream295 training sets are used for further evaluation on downstream evaluation sets , the same with [ 25 ] . We296 adopt micro F1-score and ROC-AUC as the evaluation measures for different tasks . For molecular297 dataset , as suggested by [ 37 ] , we apply three independent randomly initialized runs on each dataset298 and report the mean and standard deviation . More details are are deferred to Appendix A.2.2.299 5.2 Results of Downstream Tasks300 5.2.1 Graph Classification301 We evaluate both HGC and AdaM on 7 popular molecular graph classification datasets and HGC on 4302 social network graph classification datasets.303 The result of molecular graph classification . For molecular graph classification datasets , we304 report our pre-training strategies on different backbones , including GIN [ 38 ] , GCN [ 18 ] , Graph-305 SAGE [ 12 ] . Meanwhile , since only MolD contain node features , we apply both HGC and AdaM306 strategies on the molecular datasets . HGC_AdaM indicates the combination of two strategies.307 As shown in Table 1 , we have the following observations : ( 1 ) . GIN model pre-trained by308 our pre-training strategies can consistently outperform those pre-trained by other existing strate-309 gies , with large margin on most of them . The overall absolute improvement is 2.98 % in av-310 erage . ( 2 ) . Specially , HGC can consistently outperform those graph-data-augmentation-based311 contrastive learning strategies ( i.e. , GraphCL and C_Subgraph ) . It verifies our stand point that312 the graph data augmentation will lose some crucial domain information and compromise the fi-313 nal performance , while HGC dose not lose such information and leads to better performance.314 ( 3 ) . Even though GCN/GraphSAGE can not315 surpass the our pre-trained model on GIN pre-316 trained model , they still outperform the other317 pretraining strategy , which reaffirms the effec-318 tiveness of our pre-training strategies . ( 4 ) . The319 combined strategy HGC_AdaM achieve more320 benefits on GCN and GraphSAGE than that of321 GIN . We conjecture that GIN encodes the addi-322 tional noise which is introduced by this simple323 combination due to its strong expressive power.324 The result of social graph classification . To check the transferability of HGC , we conduct the325 finetune experiments on two models pretrained by SocL_NF and SocS_NF . SocL_NF contains the326 unlabeled data set used in finetune while SocS_NF does not . Table 2 documents the performance327 of GIN model pre-trained by HGC on SocL_NF and SocS_NF datasets . Such results show that328 GIN model pre-trained by HGC achieves the best performance on three out of four datasets . The329 comparison between GIN_HGC and GIN ( No-Pret . ) also confirms the benefits of HGC . Another330 interesting observation is that the pretrain model based on SocS_NF can obtain the better performance331 than than SocL_NF on two out of four datasets . It implies that HGC dose not just memorize the332 training samples . It can encode the latent structural information from unseen graphs and transfer the333 knowledge to the downstream tasks.334 5.2.2 Node Classification.335 Table 3 : Results on node classification datasets . The evaluation metric is micro F1-score . Datasets US-Ariport H-index |V | 1190 5000 |E| 13599 44020 ProNE 0.623 0.691 GraphWave 0.602 0.703 Struc2vec 0.662 - GCC ( Best ) 0.683 0.806 HGC ( AP_NF ) 0.706 0.824 We evaluate our model pre-trained by HGC on AP_NF on two down-336 stream node classification datasets and summarize the results in337 Table 3 . Among different versions of GCC , the best ones are pre-338 sented . From Table 3 , the model pre-trained by our HGC strategy339 can outperform the best GCC model on both datasets . It is worth340 noting that the pre-training dataset AP_NF contains only 70k graphs,341 which is much smaller than that of GCC ( 9M graphs ) . This verifies342 the efficiency of HGC in the information extraction.343 5.3 Ablation Study344 How useful are the proposed self-supervised tasks ? 345 To evaluate the contribution of our pre-training strategies , we346 compare the the performance of the pre-trained model by HGC347 and AdaM , with the model without any pre-training , each of348 which shares the same hyper-parameter setting . Results are349 summarized in Table 4 for backbone GIN . It can be seen clearly350 that all GIN models benefit from self-supervised pre-training351 tasks on all datasets . To be more specific , for GIN , absolute352 17.9 % ROC-AUC increase is observed on the dataset BACE,353 16.5 % on ClinTox , and 6.96 % on SIDER , leading to 7.53 % on354 average . Furthermore , pre-trained models gain larger improve-355 ment on datasets of relatively small size ( e.g. , BACE , ClinTox and SIDER ) , which is also observed356 in [ 28 ] . It indicates that self-supervised pre-training helps GNN models learn more inherent graph357 properties , thus getting better performance in small downstream datasets where labeled graphs are358 scarce.359 Table 5 : Results for pretraining transferability on graph classification datasets . Numbers in red are the negative transfer cases . Pretraining Type Strategy IMDB-B IMDB-M RDT-B RDT-M None GIN ( No-Pret . ) 0.734 0.433 0.885 0.635 GIN_GCC ( best ) 0.756 0.509 0.898 0.530 HGC ( SocS_NF ) 0.765 0.474 0.913 0.657Social HGC ( SocL_NF ) 0.756 0.490 0.914 0.652 Context_Pred ( MolD ) 0.734 0.473 0.875 0.635 S_Context_Pred ( MolD ) 0.763 0.460 0.818 0.625 HGC ( MolD ) 0.768 0.504 0.912 0.656 AdaM ( MolD ) 0.740 0.486 0.880 0.654 Molecular HGC_AdaM ( MolD ) 0.743 0.509 0.896 0.665 Can we transfer pre-trained mod-360 els to downstream datasets that361 are dramatically different from the362 pre-training one ? It has long been363 known that the pre-trained model can364 be generalized to unseen data in pre-365 training dataset [ 25 , 15 , 6 , 28 , 40 ] .366 However , previous literature [ 25 , 15,367 40 ] largely focuses on transferring368 the pre-trained model to downstream369 datasets with similar type of data . Here , what we are interested in asking is can we transfer the370 pre-trained model to the downstream datasets with clearly different type of graphs compared to the371 ones in the pre-training dataset ? To show this , we demonstrate the case from molecular graph to372 social network graph classification . We pre-train GIN in two different ways : one is pretrained by373 HGC on two social network graph datasets : SocS_NF and SocL_NF , the other is by HGC , AdaM or374 HGC_AdaM as well as Context_Pred or S_Context_Pred [ 15 ] on the molecular dataset MolD.375 The results are summarized in Table 5 , which offers the following observations : ( 1 ) . Perhaps376 surprisingly , our methods including HGC and HGC_AdaM enable the models pre-trained on molec-377 ular graphs to even outperform those pre-trained on social graphs . For example , the accuracy of378 HGC_AdaM on IMDB-M ( 0.509 ) and RDT-M ( 0.665 ) is much better than that of HGC ( SocS_NF ) 379 and HGC ( SocL_NF ) . Apart from the universal graph-level properties , the results also inform that380 larger pre-training datasets can help the model learn such inherent properties better . ( 2 ) . Different pre-381 training strategies could deliver different performance . Models pre-trained by graph-level pre-training382 strategies or combined strategies ( i.e. , HGC ( MolD ) and HGC_AdaM ( MolD ) ) can always get better383 results than those pre-trained by node-level strategies ( i.e. , AdaM ( MolD ) and Context_Pred ( MolD ) ) ,384 which indicates that graph-level pre-training strategies can help the model learn global graph-level385 properties that can be easily transferred to other domains . ( 3 ) . We also observe the negative transfer386 brought by the supervised pretraining in some cases . For instance , S_Context_Pred ( MolD ) 3 get387 worse performance than its no supervised trained version Context_Pred ( MolD ) on two datasets:388 RDT-B and RDT-M . It indicates that simple efforts to learn graph-level properties , such as training389 with labeled graphs , is probable to be limited in the certain domain , thus performing bad in such390 cross-domain transfer tasks . Despite this , our HGC and HGC_AdaM still consistently lead to better391 performance compared to other pretraining strategies , which , once again , versifies our assumption392 that our proposed graph contrastive learning strategy can learn more universal , even cross-domain,393 graph-level patterns.394 6 Conclusion395 In this work , we focus on developing an effective , efficient and more universal positive instances396 sampling method that can be applied on many different kinds of graph data for graph instance397 contrastive learning . We also propose an improvement for a widely used node-level pre-training398 strategy to adaptively select nodes to mask for an even distribution ( AdaM ) . Moreover , we also399 discover the potential cross-domain transferring ability for the pre-trained GNN models . However,400 there are still some limitations in our work : 1 ) . Though high-order graph sampling can get positive401 instances of better quality than those obtained by first-order sampling in our analysis , it can not always402 outperform the model pre-trained by first-order sampling process . We guess that it is relevant with the403 pre-training dataset . 2 ) . Just combining HGC and AdaM in a simple manner leads to no significant404 improvement . Though we make no further investigation into a more effective combination method405 since it is not the keypoint of the paper , it is a meaningful research direction . 3 ) . We discover the406 potential cross-domain transferring ability for pre-trained GNN models . It is an interesting point but407 no further discussion is made in this paper . However , further relevant investigation is interesting and408 meaningful.409 3Supervised trained by a labeled graph dataset after unsupervised pre-training . See [ 15 ] for details . References410 [ 1 ] Sanjeev Arora , Hrishikesh Khandeparkar , Mikhail Khodak , Orestis Plevrakis , and Nikunj411 Saunshi . A theoretical analysis of contrastive unsupervised representation learning . arXiv412 preprint arXiv:1902.09229 , 2019.413 [ 2 ] David Arthur and Sergei Vassilvitskii . k-means++ : The advantages of careful seeding . Technical414 report , Stanford , 2006.415 [ 3 ] Guy W Bemis and Mark A Murcko . The properties of known drugs . 1. molecular frameworks.416 Journal of medicinal chemistry , 39 ( 15 ) :2887–2893 , 1996.417 [ 4 ] Smriti Bhagat , Graham Cormode , and S Muthukrishnan . Node classification in social networks.418 In Social network data analytics . 2011.419 [ 5 ] Ting Chen , Simon Kornblith , Mohammad Norouzi , and Geoffrey Hinton . A simple framework420 for contrastive learning of visual representations . In International conference on machine421 learning , pages 1597–1607 . PMLR , 2020.422 [ 6 ] Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . Bert : Pre-training of423 deep bidirectional transformers for language understanding . arXiv preprint arXiv:1810.04805,424 2018.425 [ 7 ] Claire Donnat , Marinka Zitnik , David Hallac , and Jure Leskovec . Learning structural node426 embeddings via diffusion wavelets . In Proceedings of the 24th ACM SIGKDD International427 Conference on Knowledge Discovery & Data Mining , pages 1320–1329 , 2018.428 [ 8 ] Greg Landrum et al . Rdkit : Open-source cheminformatics . 2006.429 [ 9 ] Matthias Fey and Jan E. Lenssen . Fast graph representation learning with PyTorch Geometric.430 In ICLR Workshop on Representation Learning on Graphs and Manifolds , 2019.431 [ 10 ] Justin Gilmer , Samuel S Schoenholz , Patrick F Riley , Oriol Vinyals , and George E Dahl . Neural432 message passing for quantum chemistry . In International Conference on Machine Learning,433 pages 1263–1272 . PMLR , 2017.434 [ 11 ] Aditya Grover and Jure Leskovec . node2vec : Scalable feature learning for networks . In435 Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and436 data mining , pages 855–864 , 2016.437 [ 12 ] William L Hamilton , Rex Ying , and Jure Leskovec . Inductive representation learning on large438 graphs . arXiv preprint arXiv:1706.02216 , 2017.439 [ 13 ] Kaveh Hassani and Amir Hosein Khasahmadi . Contrastive multi-view representation learning440 on graphs . In International Conference on Machine Learning , pages 4116–4126 . PMLR , 2020.441 [ 14 ] Kaiming He , Haoqi Fan , Yuxin Wu , Saining Xie , and Ross Girshick . Momentum contrast for442 unsupervised visual representation learning . In Proceedings of the IEEE/CVF Conference on443 Computer Vision and Pattern Recognition , pages 9729–9738 , 2020.444 [ 15 ] Weihua Hu , Bowen Liu , Joseph Gomes , Marinka Zitnik , Percy Liang , Vijay Pande , and Jure445 Leskovec . Strategies for pre-training graph neural networks . arXiv preprint arXiv:1905.12265,446 2019.447 [ 16 ] Max Jaderberg , Valentin Dalibard , Simon Osindero , Wojciech M Czarnecki , Jeff Donahue , Ali448 Razavi , Oriol Vinyals , Tim Green , Iain Dunning , Karen Simonyan , et al . Population based449 training of neural networks . arXiv preprint arXiv:1711.09846 , 2017.450 [ 17 ] Diederik P Kingma and Jimmy Ba . Adam : A method for stochastic optimization . arXiv preprint451 arXiv:1412.6980 , 2014.452 [ 18 ] Thomas N Kipf and Max Welling . Semi-supervised classification with graph convolutional453 networks . arXiv preprint arXiv:1609.02907 , 2016.454 [ 19 ] Christopher Morris , Nils M. Kriege , Franka Bause , Kristian Kersting , Petra Mutzel , and Marion455 Neumann . Tudataset : A collection of benchmark datasets for learning with graphs . In ICML456 2020 Workshop on Graph Representation Learning and Beyond ( GRL+ 2020 ) , 2020 . URL457 www.graphlearning.io.458 [ 20 ] Annamalai Narayanan , Mahinthan Chandramohan , Rajasekar Venkatesan , Lihui Chen , Yang459 Liu , and Shantanu Jaiswal . graph2vec : Learning distributed representations of graphs . arXiv460 preprint arXiv:1707.05005 , 2017.461 [ 21 ] Adam Paszke , Sam Gross , Francisco Massa , Adam Lerer , James Bradbury , Gregory Chanan,462 Trevor Killeen , Zeming Lin , Natalia Gimelshein , Luca Antiga , et al . Pytorch : An imperative463 style , high-performance deep learning library . arXiv preprint arXiv:1912.01703 , 2019.464 [ 22 ] Bryan Perozzi , Rami Al-Rfou , and Steven Skiena . Deepwalk : Online learning of social repre-465 sentations . In Proceedings of the 20th ACM SIGKDD international conference on Knowledge466 discovery and data mining , pages 701–710 , 2014.467 [ 23 ] Jiezhong Qiu , Yuxiao Dong , Hao Ma , Jian Li , Kuansan Wang , and Jie Tang . Network embedding468 as matrix factorization : Unifying deepwalk , line , pte , and node2vec . In Proceedings of the469 eleventh ACM international conference on web search and data mining , pages 459–467 , 2018.470 [ 24 ] Jiezhong Qiu , Yuxiao Dong , Hao Ma , Jian Li , Chi Wang , Kuansan Wang , and Jie Tang.471 Netsmf : Large-scale network embedding as sparse matrix factorization . In The World Wide Web472 Conference , pages 1509–1520 , 2019.473 [ 25 ] Jiezhong Qiu , Qibin Chen , Yuxiao Dong , Jing Zhang , Hongxia Yang , Ming Ding , Kuansan474 Wang , and Jie Tang . Gcc : Graph contrastive coding for graph neural network pre-training . In475 Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & 476 Data Mining , pages 1150–1160 , 2020.477 [ 26 ] Leonardo FR Ribeiro , Pedro HP Saverese , and Daniel R Figueiredo . struc2vec : Learning node478 representations from structural identity . In Proceedings of the 23rd ACM SIGKDD international479 conference on knowledge discovery and data mining , pages 385–394 , 2017.480 [ 27 ] David Rogers and Mathew Hahn . Extended-connectivity fingerprints . Journal of chemical481 information and modeling , 50 ( 5 ) :742–754 , 2010.482 [ 28 ] Yu Rong , Yatao Bian , Tingyang Xu , Weiyang Xie , Ying Wei , Wenbing Huang , and Junzhou483 Huang . Grover : Self-supervised message passing transformer on large-scale molecular data.484 arXiv preprint arXiv:2007.02835 , 2020.485 [ 29 ] Nino Shervashidze , Pascal Schweitzer , Erik Jan Van Leeuwen , Kurt Mehlhorn , and Karsten M486 Borgwardt . Weisfeiler-lehman graph kernels . Journal of Machine Learning Research , 12 ( 9 ) ,487 2011.488 [ 30 ] Teague Sterling and John J Irwin . Zinc 15–ligand discovery for everyone . Journal of chemical489 information and modeling , 55 ( 11 ) :2324–2337 , 2015.490 [ 31 ] Fan-Yun Sun , Jordan Hoffmann , Vikas Verma , and Jian Tang . Infograph : Unsupervised and491 semi-supervised graph-level representation learning via mutual information maximization . arXiv492 preprint arXiv:1908.01000 , 2019.493 [ 32 ] Jian Tang , Meng Qu , Mingzhe Wang , Ming Zhang , Jun Yan , and Qiaozhu Mei . Line : Large-494 scale information network embedding . In Proceedings of the 24th international conference on495 world wide web , pages 1067–1077 , 2015.496 [ 33 ] Petar Veličković , Guillem Cucurull , Arantxa Casanova , Adriana Romero , Pietro Lio , and Yoshua497 Bengio . Graph attention networks . arXiv preprint arXiv:1710.10903 , 2017.498 [ 34 ] Petar Velickovic , William Fedus , William L Hamilton , Pietro Liò , Yoshua Bengio , and R Devon499 Hjelm . Deep graph infomax . In ICLR ( Poster ) , 2019.500 [ 35 ] Minjie Wang , Lingfan Yu , Da Zheng , Quan Gan , Yu Gai , Zihao Ye , Mufei Li , Jinjing Zhou,501 Qi Huang , Chao Ma , et al . Deep graph library : Towards efficient and scalable deep learning on502 graphs . 2019.503 [ 36 ] Minjie Wang , Da Zheng , Zihao Ye , Quan Gan , Mufei Li , Xiang Song , Jinjing Zhou , Chao Ma,504 Lingfan Yu , Yu Gai , Tianjun Xiao , Tong He , George Karypis , Jinyang Li , and Zheng Zhang.505 Deep graph library : A graph-centric , highly-performant package for graph neural networks.506 arXiv preprint arXiv:1909.01315 , 2019.507 [ 37 ] Zhenqin Wu , Bharath Ramsundar , Evan N Feinberg , Joseph Gomes , Caleb Geniesse , Aneesh S508 Pappu , Karl Leswing , and Vijay Pande . Moleculenet : a benchmark for molecular machine509 learning . Chemical science , 9 ( 2 ) :513–530 , 2018.510 [ 38 ] Keyulu Xu , Weihua Hu , Jure Leskovec , and Stefanie Jegelka . How powerful are graph neural511 networks ? arXiv preprint arXiv:1810.00826 , 2018.512 [ 39 ] Pinar Yanardag and SVN Vishwanathan . Deep graph kernels . In Proceedings of the 21th ACM513 SIGKDD international conference on knowledge discovery and data mining , pages 1365–1374,514 2015.515 [ 40 ] Yuning You , Tianlong Chen , Yongduo Sui , Ting Chen , Zhangyang Wang , and Yang Shen.516 Graph contrastive learning with augmentations . Advances in Neural Information Processing517 Systems , 33 , 2020.518 [ 41 ] Fanjin Zhang , Xiao Liu , Jie Tang , Yuxiao Dong , Peiran Yao , Jie Zhang , Xiaotao Gu , Yan Wang,519 Bin Shao , Rui Li , et al . Oag : Toward linking large-scale heterogeneous entity graphs . In520 Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & 521 Data Mining , pages 2585–2595 , 2019.522 [ 42 ] Jie Zhang , Yuxiao Dong , Yan Wang , Jie Tang , and Ming Ding . Prone : Fast and scalable network523 representation learning . In IJCAI , volume 19 , pages 4278–4284 , 2019.524 [ 43 ] Muhan Zhang , Zhicheng Cui , Marion Neumann , and Yixin Chen . An end-to-end deep learning525 architecture for graph classification . In Proceedings of the AAAI Conference on Artificial526 Intelligence , volume 32 , 2018.527 [ 44 ] Shenghuo Zhu , Kai Yu , Yun Chi , and Yihong Gong . Combining content and link for classifica-528 tion using matrix factorization . In SIGIR , 2007.529 Checklist530 1 . For all authors ... 531 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s532 contributions and scope ? [ Yes ] The main contribution of this paper is the proposal of533 an effective and more universal positive instance selection strategy that can be applied534 on various kinds of graph data in the contrastive learning process . We also propose535 an improvement of the a widely used node-level pre-training strategy to adaptively536 choose nodes to make them distributed evenly in the graph . Moreover , we discover the537 potential possibility of the cross-domain transferable ability of the pre-trained GNN538 models.539 ( b ) Did you describe the limitations of your work ? [ Yes ] See Sec . 6.540 ( c ) Did you discuss any potential negative societal impacts of your work ? [ Yes ] See Sec . C.541 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to542 them ? [ Yes ] 543 2 . If you are including theoretical results ... 544 ( a ) Did you state the full set of assumptions of all theoretical results ? [ Yes ] See Sec . A.7.545 We make reasonable assumptions on the possibility density function of the approximate546 similarity function on the ground-truth positive graph set and negative graph set . Some547 reasonable approximations are made in the derivation process.548 ( b ) Did you include complete proofs of all theoretical results ? [ Yes ] See Sec . A.7.549 3 . If you ran experiments ... 550 ( a ) Did you include the code , data , and instructions needed to reproduce the main experi-551 mental results ( either in the supplemental material or as a URL ) ? [ Yes ] See Sec . A.1552 for descriptions and download links for datasets . Download links for our pre-processed553 data are shared along with the code . Code is provided with supplemental material.554 Instructions for reproduction are stated in README.md file in the supplemental555 material.556 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they557 were chosen ) ? [ Yes ] See Sec . A.2 for implementation details , including pre-training and558 fine-tuning configuration and hyper-parameter selection . See Sec . A.1 for descriptions559 for datasets and the splitting methods.560 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running exper-561 iments multiple times ) ? [ Yes ] We report mean and std values for 3 independently562 random initialized run for each evaluation process on molecular graph datasets . See563 Table 1 and Table 12 for details.564 ( d ) Did you include the total amount of compute and the type of resources used ( e.g.,565 type of GPUs , internal cluster , or cloud provider ) ? [ Yes ] See Sec . A.2 for hardware566 configurations.567 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 568 ( a ) If your work uses existing assets , did you citep the creators ? [ Yes ] We provide links569 for data and code that are from public respiratory we used in our project . We also cite570 related papers . See Sec . A.1 and Sec . A.3.571 ( b ) Did you mention the license of the assets ? [ Yes ] Datasets obtained from published572 works are with related papers cited . See Sec . A.1573 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ No ] 574 No new datasets are proposed.575 ( d ) Did you discuss whether and how consent was obtained from people whose data576 you ’ re using/curating ? [ Yes ] Download links for public datasets we used are provided.577 Datasets obtained from published works are with related papers cited . See Sec . A.1578 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable579 information or offensive content ? [ Yes ] Datasets we use are obtained from public580 datasets , containing no such information . We provide links for them in Sec . A.1 . .581 5 . If you used crowdsourcing or conducted research with human subjects ... 582 ( a ) Did you include the full text of instructions given to participants and screenshots , if583 applicable ? [ N/A ] 584 ( b ) Did you describe any potential participant risks , with links to Institutional Review585 Board ( IRB ) approvals , if applicable ? [ N/A ] 586 ( c ) Did you include the estimated hourly wage paid to participants and the total amount587 spent on participant compensation ? [ N/A ] 588

Summary: The authors propose a new method for graph pre-training based on instance contrastive learning. They exploit graphs in the training set, to select positive instances that are similar to target graphs. They also propose a new strategy for node-level pre-training.