GL-Disen: Global-Local disentanglement for unsupervised learning of graph-level representations

Graph-level representation learning plays a crucial role in a variety of tasks such as molecular property prediction and community analysis . Currently , several models based on mutual information maximization have shown strong performance on the task of unsupervised graph representation learning . In this paper , instead , we consider a disentanglement approach to learn graph-level representations in the unsupervised setting . Our work is the first to study disentanglement learning for graph-level representations . Our key observation is that the formation of many real-world graphs is a complex process with global and local generative factors . We hypothesize that disentangled representations which capture these global and local generative factors into independent latent units can be highly beneficial . Specifically , for graph-level representation learning , our disentanglement approach can alleviate distraction due to local variations of individual nodes or individual local neighbourhoods . We propose a VAE based learning algorithm to disentangle the global graph-level information , which is common across the entire graph , and local patch-level information , which varies across individual patches ( the local subgraphs centered around the nodes ) . Through extensive experiments and analysis , we show that our method achieves the state-of-the-art performance on the task of unsupervised graph representation learning . 1 INTRODUCTION . Graph structured data has been very useful in representing a variety of data types including social networks ( Newman & Girvan , 2004 ) , protein-protein interactions Krogan et al . ( 2006 ) , scene graphs ( Krishna et al. , 2016 ) , customer purchasing patterns ( Bhatia et al. , 2016 ) and many more . Graph Neural Networks ( GNNs ) have recently become the prominent approach for representing graph structured data ( Li et al. , 2016 ; Gilmer et al. , 2017 ; Kipf & Welling , 2017 ; Velickovic et al. , 2018 ; Xu et al. , 2019 ) . GNNs are capable of representing graphs in a permutation invariant manner , enabling information propagation among neighbours and mapping graphs to low dimensional spaces . In this work , we focus on graph-level representation learning . Graph-level representation learning is crucial for tasks like molecular property identification ( Duvenaud et al. , 2015 ) and community classification based on the patterns of discussion threads ( Yanardag & Vishwanathan , 2015 ) , and they are useful for applications such as drug discovery and recommendation systems . Availability of task specific labels plays a significant role in graph representation learning as much as its role in other domains such as images , text and speech . However , due to many specialized fields which graphs are utilized ( e.g. , biological sciences , quantum mechanics ) , collecting labels has become very expensive as it needs expert knowledge ( Sun et al. , 2020 ) . Therefore , unsupervised learning of graph representation is crucial . Recent state-of-the-art unsupervised graph representation learning methods ( Sun et al. , 2020 ; Hassani & Khasahmadi , 2020 ) are based on Infomax principle by Linsker ( 1988 ) . These methods learn the graph representation by maximizing the mutual information between the representation of the entire graph and the representations of individual patches of the graph . Here we follow ( Velickovic et al. , 2019 ; Sun et al. , 2020 ) and define patches as local subgraphs centered around a node . This approach allows the graph level representation to capture the globally relevant information from the patch representations ( Sun et al. , 2020 ) . Global-local disentanglement . We propose a novel approach for graph-level representation learning . Our observation is that many graphs are generated using multiple heterogeneous factors , with different factors providing different information . Specifically , the formation of many real-world graphs is driven by graph-level factors and node/patch-level factors . For example , an online discussion thread can be represented as a graph where nodes represent users who have participated in the discussion thread , and edges represent interaction between the users in the thread ( Yanardag & Vishwanathan , 2015 ) . Graph-level representation of such communication graphs can be used to classify the sub-community ( e.g . subreddits on Reddit ) of that discussion thread . However , the formation of these communication graphs is driven by global graph-level factors ( e.g. , the topic of the discussion-thread ) and local node-level factors ( e.g. , characteristics of individual users engaging in on-line discussion ) . The graph is formed with a complex process involving complicated interactions between global graph-level factors and local node/patch-level factors . It has been discussed in the literature that disentangling these generative factors can benefit many tasks in different domains ( Bengio et al. , 2013 ; Ridgeway , 2016 ) . This is because disentanglement enables to separate out explanatory generative factors which cause variations in data and facilitates selection of only those factors which are well suited for the downstream task . Importantly , removing the irrelevant factors from the prediction process increases the robustness of models ( Ma et al. , 2019 ) . Based on the above discussion , we hypothesize that graph representation learning that disentangles the graph-level and node/patch-level factors can be useful for many graph analysis tasks . In particular , the disentangled graph-level representation can be powerful for graph-level inference . Therefore , in this work , we propose GL-Disen : a global graph level - local node/patch level disentanglement method for graph level representation learning . Disentanglement learning is a novel direction for GNNs , and it has not been studied for graph level representations learning . Existing work have only focused on disentangling the factors which forms each neighbourhood , based on supervision from downstream tasks ( Ma et al. , 2019 ; Liu et al. , 2020 ; Yang et al. , 2020 ) and disentangling node and edge features in attributed graphs ( Guo et al. , 2020 ) . To summarize , our contributions are : • We propose GL-Disen : a novel global graph-level and local node/patch-level disentangling model . To the best of our knowledge , this is the first work of applying unsupervised disentangled learning for graph level representation learning . • We conduct extensive experiments to verify that our model learns meaningful disentangled global and local representations for graphs . The disentangled global representation achieves outstanding performance in graph classification . 2 RELATED WORK . Here we review the most relevant work on unsupervised graph level representation learning to ours . Reviews of disentangle learning and other unsupervised graph learning methods are in Appendix A . The most recent family of graph embedding methods are based on contrastive learning . Main idea is to train an encoder model to make it learn the contrast in between a representation which captures the structural and statistic information provided by original data and a negative sample . InfoGraph by Sun et al . ( 2020 ) was the first graph level embedding model which utilized contrastive learning and this method was inspired by Infomax principle based Deep Graph Infomax ( DGI ) ( Velickovic et al. , 2019 ) . It draws negative samples from other graphs and sum pooling is used as the readout function . Multi-view contrastive ( CMV ) learning method by Hassani & Khasahmadi ( 2020 ) enhances InfoGraph by introducing multi-view based data augmentation mechanism which uses contrastive learning to maximize mutual information among multiple structural views of the input graph . On the other hand , Graph Contrastive Coding ( GCC ) ( Qiu et al. , 2020 ) utilizes contrastive learning for learning universal graph embeddings which can be transferred to multiple downstream tasks . Infomax principle based and contrastive learning based methods have produced the best performance for graph embedding models so far . 3 GL-DISEN METHODOLOGY . 3.1 GRAPH GENERATION PROCESS . We follow the general framework of ( Higgins et al. , 2017 ) . However , our specific disentanglement method is quite different , as will be discussed . Let D = { G , Gf , Lf } be the set that consists of graphs and their ground truth generative factors for global and local level . Each graph G = ( V , A ) , contains a set of nodes V and A is the adjacency matrix . Gf and Lf represent two sets of generative factors : Gf contains global factors gf ⊂ Gf common for the entire graph and lf ⊂ Lf represents local factors which can differ from local patch to patch . In our model , gf and lf are conditionally independent given G , where p ( gf , lf |G ) = p ( gf |G ) · p ( lf |G ) . We assume that the graph G is generated using a true world generator which uses the ground truth generative factors : p ( G|gf , lf ) = Gen ( gf , lf ) . 3.2 GLOBAL GRAPH LEVEL AND LOCAL PATCH LEVEL DISENTANGLEMENT . Our goal is to develop an unsupervised deep graph generative model which can learn the joint distribution of graph G , the set of generative factors Z , using only the samples from G. This should be learnt in a way that the set of latent generative factors can generate the observed graphG , such that p ( G|Z ) ≈ p ( G|gf , lf ) = Gen ( gf , lf ) . A suitable approach to fulfill this objective is to maximize the marginal log-likelihood for the observed graph G over the whole distribution of latent factors Z. max θ Epθ ( Z ) [ log pθ ( G|Z ) ] ( 1 ) For an observed graphG , the inferred posterior probability distribution of the latent factors Z can be described as qφ ( Z|G ) . However , the graph generation process we described in Section 3.1 assumes two independent sets of generative factors representing global and local level information relevant for a graph . Therefore we consider a model where the latent factor set Z can be divided into two independent latent factor sets as Z = ( Zg , Zl ) . Zg represents the latent factors which capture the global generative factors of G and Zl captures the local counterpart . Therefore we can rewrite our inferred posterior distribution as follows : qφ ( Z|G ) = qφ ( Zg , Zl|G ) = qφ ( Zg|G ) qφ ( Zl|G ) ( 2 ) We discuss in detail the two posteriors : qφ ( Zg|G ) and qφ ( Zl|G ) . The graph G consists of |V | number of nodes . In a graph data structure , each node is not isolated . They are connected with its neighbours and propagates information . Therefore , we use the term patch to indicate the local neighbourhood centered at each node where the node interacts with . Therefore , qφ ( Zg|G ) and qφ ( Zl|G ) are the posterior distributions of all these |V | patches . However , if we consider the global latent posterior , it is common for all |V | patches , as the graph G was originally generated with gf common for all V . Hence , we propose to use a single latent zg to capture the global generative factors common for all patches . In particular , we use qφ ( zg|G ) to model this single posterior . On the other hand , the factors which contribute to generate each patch can vary significantly . Therefore in this model we assume the local latent factors are independent 1 . Therefore , we update Eq . 2 as : qφ ( Z|G ) = qφ ( zg , Zl|G ) = qφ ( zg|G ) |V |∏ i=1 qφ ( zl ( i ) |G ) ( 3 ) Here zl ( i ) is the latent factor that captures the local generative factors for a patch centered at node i . Now , our objective is to make sure the latent factors sampled from global and local latent posterior distributions can capture the global and local generative factors gf and lf respectively in a disentangled manner . Note that , we aim to only disentangle global latent factors from local latent factors in this work . This is because , since the intention of the global latent zg is to capture all the global factors for graph level representation , entanglement among individual factors in either global latent or local latent is not being focused . Thus , this is different from ( Higgins et al. , 2017 ) . To 1In the evaluation we assess the validity of this assumption . enforce the disentangling nature between these two latent factors , first , we try to match each of them to their respective priors p ( zg ) and p ( zl ) separately . We select unit Gaussians ( N ( 0 , 1 ) ) as priors . This leads to following constrained optimization problem ( Higgins et al. , 2017 ) . max θ , φ EG∼G [ Eqφ ( zg , Zl|G ) [ log pθ ( G|zg , Zl ) ] ] s.t . KL ( qφ ( zg|G ) ‖ p ( zg ) ) < KL ( qφ ( Zl|G ) ‖ p ( Zl ) ) < η ( 4 ) where and η are strengths of each constraint . Following ( Higgins et al. , 2017 ) , Eq . 4 can be written to obtain the variational evidence lower bound ( ELBO ) of a Graph Variational Autoencoder ( GVAE ) ( Kipf & Welling , 2016 ) ( Here we call this as GVAE because our input is a graph ) with two separate latent representations with additional coefficients as follows : F ( θ , φ , α , γ ; G , zg , Zl ) ≥ L ( θ , φ ; G , zg , Zl , α , γ ) = Eqφ ( zg , Zl|G ) [ log pθ ( G|zg , Zl ) ] − α KL ( qφ ( zg|G ) ‖ p ( zg ) ) − γ KL ( qφ ( Zl|G ) ‖ p ( Zl ) ) ( 5 ) Based on Eq.3 we can expand the KL divergence term KL ( qφ ( Zl|G ) ‖ p ( Zl ) ) and rewrite our objective function for a single graph G as : L ( θ , φ ; G , zg , Zl , α , γ ) = Eqφ ( zg , Zl|G ) [ log pθ ( G|zg , Zl ) ] − α KL ( qφ ( zg|G ) |p ( zg ) ) − γ |V |∑ i=1 KL ( qφ ( zl ( i ) |G ) ‖ p ( zl ( i ) ) ) ( 6 ) The training process maximizes this lower bound for all the graphs in a minibatch Gb from the full dataset G : Lθ , φ ( Gb ) = 1 |Gb| |Gb|∑ r=1 L ( θ , φ ; G , zg , Zl , α , γ ) ( 7 )

Summary: This paper proposes an unsupervised graph-level representation learning method considering global-local disentanglement. Specifically, the authors propose a GL-Disen model based on graph VAE architecture to jointly learn global and local representations for a graph. The global information is shared across the whole graph while the local information varies from patch to patch, corresponding to common and local factors, respectively. Empirical experimental results show that the learned representation achieves superior performance in the downstream graph classification task, and analyses demonstrate the learned representations exhibit some disentangle property.