Self-supervised Graph-level Representation Learning with Local and Global Structure

1 INTRODUCTION . Learning informative representations of whole graphs is a fundamental problem in a variety of domains and tasks , such as molecule properties prediction in drug and material discovery ( Gilmer et al. , 2017 ; Wu et al. , 2018 ) , protein function forecast in biological networks ( Alvarez & Yan , 2012 ; Jiang et al. , 2017 ) , and predicting the properties of circuits in circuit design ( Zhang et al. , 2019 ) . Recently , Graph Neural Networks ( GNNs ) have attracted a surge of interest and showed the effectiveness in learning graph representations . These methods are usually trained in a supervised fashion , which requires a large number of labeled data . Nevertheless , in many scientific domains , labeled data are very limited and expensive to obtain . Therefore , it is becoming increasingly important to learn the representations of graphs in an unsupervised or self-supervised fashion . Self-supervised learning has recently achieved profound success for both natural language processing , e.g . GPT ( Radford et al. , 2018 ) and BERT ( Devlin et al. , 2019 ) , and image understanding , e.g . MoCo ( He et al. , 2019 ) and SimCLR ( Chen et al. , 2020 ) . However , how to effectively learn the representations of graphs in a self-supervised way is still an open problem . Intuitively , a desirable graph representation should be able to preserve the local-instance structure , so that similar graphs are embedded close to each other and dissimilar ones stay far apart . In addition , the representations of a set of graphs should also reflect the global-semantic structure of the data , so that the graphs with similar semantic properties are compactly embedded , which benefits various downstream tasks , e.g . graph classification or regression . Such structure can be sufficiently captured by semantic clusters ( Caron et al. , 2018 ; Ji et al. , 2019 ) , especially in a hierarchical fashion ( Li et al. , 2020 ) . There are some recent works that learn graph representation in a self-supervised manner , such as local-global mutual information maximization ( Velickovic et al. , 2019 ; Sun et al. , 2019 ) , structuralsimilarity/context prediction ( Navarin et al. , 2018 ; Hu et al. , 2019 ; You et al. , 2020 ) and contrastive multi-view learning ( Hassani & Ahmadi , 2020 ) . However , all these methods are capable of modeling only the local structure between different graph instances but fail to discover the global-semantic structure . To address this shortcoming , we are seeking for an approach that is sufficient to model both the local and global structure of a given set of graphs . To attain this goal , we propose a Local-instance and Global-semantic Learning ( GraphLoG ) framework for self-supervised graph representation learning . In specific , for preserving the local similarity between various graph instances , we first align the embeddings of correlated graphs/subgraphs1 by maximizing their mutual information . In this locally smooth embedding space , we further represent the distribution of different graph embeddings with hierarchical prototypes2 whose number is adaptively determined by the data in a nonparametric fashion . During training , these prototypes guide each graph to map to the semantically-similar feature cluster , and , simultaneously , the prototypes are maintained by online-updated graph embeddings . In this process , the global-semantic structure of the data is gradually discovered and refined . The whole model is pre-trained with a large number of unlabeled graphs , and then fine-tuned and evaluated on some downstream tasks . We summarize our contributions as follows : • We contribute a unified framework called Local-instance and Global-semantic Learning ( GraphLoG ) for self-supervised graph representation learning , which is able to model the structure of a set of graphs both locally and globally . • We novelly propose to infer the global-semantic structure underlying the unlabeled graphs by learning hierarchical prototypes via a nonparametric strategy . • We empirically verify our framework ’ s superior performance on different GNN architectures through pre-training on a large-scale unlabeled dataset and fine-tuning on benchmark tasks in both the chemistry and biology domains . 2 PROBLEM DEFINITION AND PRELIMINARIES . 2.1 PROBLEM DEFINITION . An ideal representation should preserve the local structure among the data instances . More specifically , we define it as follows : Definition 1 ( Local-instance Structure ) . The local-instance structure refers to the local pairwise similarity between different instances ( Roweis & Saul , 2000 ; Belkin & Niyogi , 2002 ) . To preserve the local-instance structure of graph-structured data , a pair of similar graphs/subgraphs , G and G′ , are expected to be mapped to the nearby positions of embedding space , as illustrated in Fig . 1 ( a ) , while the dissimilar pairs should be mapped to far apart . The pursuit of local-instance structure is usually insufficient to capture the semantics underlying the entire dataset . It is therefore important to discover the global-semantic structure of the data , which is concretely defined as follows : Definition 2 ( Global-semantic Structure ) . A real-world dataset is usually distributed as different semantic clusters ( Furnas et al. , 2017 ; Ji et al. , 2019 ) . Therefore , we define the global-semantic structure of a dataset as the distribution of its semantic clusters , and each cluster is represented by a prototype ( i.e . a representative cluster embedding ) . Since the semantics of a set of graphs can be structured in a hierarchical way ( Ashburner et al. , 2000 ; Chen et al. , 2012 ) , we represent the whole dataset with hierarchical prototypes . A detailed example can be seen in Fig . 1 ( b ) . 1In our method , we obtain correlated graphs/subgraphs via minor modification on node/edge attributes . 2Hierarchical prototypes are representative cluster embeddings organized in a hierarchical way . Problem Definition . For self-supervised graph representation learning , a set of unlabeled graphs G = { G1 , G2 , · · · , GM } are given , and we aim to learn a low-dimensional vector hGi ∈ Rδ for each graph Gi ∈ G under the guidance of the data itself . In specific , we expect the derived graph embeddings H ∈ RM×δ follow both the local-instance and global-semantic structure . 2.2 PRELIMINARIES . Graph Neural Networks ( GNNs ) . Given a graph G = ( V , E ) with node attributes XV = { Xv|v ∈ V } and edge attributes XE = { Xuv| ( u , v ) ∈ E } , a GNN aims to learn an embedding vector hv for each node v ∈ V and also a vector hG for the entire graph G. For an L-layer GNN , a neighborhood aggregation scheme is performed to capture the L-hop information surrounding each node . The l-th layer of a GNN can be formalized as follows : h ( l ) v = COMBINE ( l ) ( h ( l−1 ) v , AGGREGATE ( l ) ( { ( h ( l−1 ) v , h ( l−1 ) u , Xuv ) : u ∈ N ( v ) } ) ) , ( 1 ) where N ( v ) is the neighborhood set of v , h ( l ) v denotes the representation of node v at the l-th layer , and h ( 0 ) v is initialized as the node attribute Xv . Since hv summarizes the information of a patch centered around node v , we will refer to hv as patch embedding to underscore this point . The entire graph ’ s embedding can be derived by a permutation-invariant readout function : hG = READOUT ( { hv|v ∈ V } ) . ( 2 ) Mutual Information Estimation . Mutual information ( MI ) can measure both the linear and nonlinear dependency between two random variables . Some recent works ( Belghazi et al. , 2018 ; Hjelm et al. , 2019 ) employed neural networks to estimate the lower bound of MI . Among which , InfoNCE loss ( van den Oord et al. , 2018 ) has been introduced to maximize a lower bound of MI by minimizing itself , and we also adopt it in this work for its simplicity and effectiveness . In practice , given a query q , InfoNCE loss is optimized to score the positive sample z+ higher than a set of distractors { zi } Ki=1 : LNCE ( q , z+ , { zi } Ki=1 ) = − log exp ( T ( q , z+ ) ) exp ( T ( q , z+ ) ) + ∑K i=1 exp ( T ( q , zi ) ) , ( 3 ) where T ( · , · ) is a parameterized discriminator function which maps two representation vectors to a scalar value , whose architecture is detailed in Sec . 6.1 . Rival Penalized Competitive Learning ( RPCL ) . The RPCL method ( Xu et al. , 1993 ) is a variant of classical competitive learning approaches , e.g . K-means clustering . Concretely , given a sample for update , RPCL-based clustering not only pulls the winning cluster center ( i.e . the closest one ) towards the sample , but also pushes the rival cluster center ( i.e . the second closest one ) away from the sample . We adopt this clustering algorithm for its strong capability of discovering feature clusters without specifying the number of clusters beforehand ( i.e . in a nonparametric fashion ) , which is critical in the self-supervised learning scenarios where the number of semantic categories is not given . 3 LOCAL-INSTANCE AND GLOBAL-SEMANTIC LEARNING . 3.1 LEARNING LOCAL-INSTANCE STRUCTURE OF GRAPH REPRESENTATIONS . We first define the correlated graphs that are expected to be embedded close to each other in the embedding space . Since the graphs from a dataset lie in a highly discrete space , it is hard to seek out the correlated counterpart of each graph from the dataset . To tackle this limitation , we propose to construct pairs of correlated graphs via the attribute masking strategy ( Hu et al. , 2019 ) which randomly masks a part of node/edge attributes in a graph ( theoretical analysis is stated in Sec . A ) . Through applying this technique to a randomly sampled mini-batch BG = { Gj = ( Vj , Ej ) } Nj=1 with N graphs , the correlated counterpart of each graph can be obtained , which forms another minibatch B′G = { G′j = ( V ′j , E ′j ) } Nj=1 ( Gj and G′j are deemed as a pair of correlated graphs ) . Taking both mini-batches as input , the corresponding patch and graph embeddings are derived as follows : hVj = { hv|v ∈ Vj } = GNN ( XVj , XEj ) , hV′j = { hv|v ∈ V ′ j } = GNN ( XV′j , XE′j ) , ( 4 ) hGj = READOUT ( hVj ) , hG′j = READOUT ( hV′j ) , ( 5 ) where hVj ( hV′j ) is the set of patch embeddings for graph Gj ( G ′ j ) , and hGj ( hG′j ) denotes the embedding of entire graph . With these ingredients , we design the learning objective for local-instance structure based on two desiderata : ( 1 ) similar subgraphs ( i.e . patches ) have similar feature representations ; ( 2 ) graphs with a set of similar patches are embedded close to each other . To attain these goals , we propose to maximize the mutual information ( i.e . minimize the InfoNCE loss ) between correlated patches/graphs , which derives two constraints for the local-instance structure : Lpatch = 1∑N j=1 |V ′j | N∑ j=1 ∑ v′∈V′j ∑ v∈Vj 1v↔v′ · LNCE ( hv′ , hv , { hṽ|ṽ ∈ Vj , ṽ 6= v } ) , ( 6 ) Lgraph = 1 N N∑ j=1 LNCE ( hG′j , hGj , { hGk |1 6 k 6 N , k 6= j } ) , ( 7 ) Llocal = Lpatch + Lgraph , ( 8 ) where LNCE ( · , · , · ) is the InfoNCE loss function defined in Eq . 3 , and 1v↔v′ denotes the indicator function judging whether v and v′ are the corresponding nodes in a pair of correlated graphs . Note that , masking node/edge attributes doesn ’ t change the topology of a graph , which makes it easy to determine these corresponding nodes in our method .

Summary: This paper proposed a method for self-supervised graph-level representation learning. The main idea is to enforce both the instance level smoothness embedding constraints, and a so-called global, semantic grouping structures across all instance graphs in the training data set.  To achieve this goal, the authors have adopted a global clustering framework to encourage the embedding of the graphs belonging to the same clusters to be close to each other, and by using a hierarchically organized set of prototypes. The proposed method is applied to pre-train GNN on massive unlabeled graphs, which is then fine-tuned to downstream learning tasks.