Unsupervised Hierarchical Graph Representation Learning with Variational Bayes

1 INTRODUCTION . Graph representation learning has attracted a surge of interest recently , inspired by the widespread success of representation learning in the image and language domains through the use of deep neural networks for parameterization . A substantial number of graph neural network ( GNN ) architectures ( Bruna et al. , 2014 ; Henaff et al. , 2015 ; Duvenaud et al. , 2015 ; Defferrard et al. , 2016 ; Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Chen et al. , 2018 ; Velic̆ković et al. , 2018 ; Ying et al. , 2018a ; Liao et al. , 2019 ; Xu et al. , 2019 ) extend the convolution filters for a regular grid of data ( e.g. , image pixels , time series , and sequences ) to irregularly connected graph neighborhoods . This extension naturally stimulates the quest of also extending the pooling operation in convolutional neural networks ( CNN ) to graphs . The challenge lies in the irregular connections as opposed to a regular grid structure , whereby partitioning is straightforward . Graph pooling is used in at least two scenarios . One is global pooling : it pools the vector representations of all nodes to a single vector as the graph representation . Simple operators such as max or mean are applied . A slightly more complex operator is the weighted sum , wherein the weights are computed through attention ( Velic̆ković et al. , 2018 ; Lee et al. , 2019 ) . A recently proposed operator is top-k pooling ( Zhang et al. , 2018 ) , whereby a fixed number of node representations at the top of a sorted list is retained so that convolutions or feed-forward transformations are applied . The second use of pooling is the creation of a graph hierarchy . In this scenario , pooling is local , more similar to that in CNNs . It is interfaced with graph coarsening ( also called graph reduction or graph compression ) , a generic form of which is to cluster the nodes of the original graph into a node of the coarse graph . Then , the representations of the nodes in the cluster are pooled . The clustering may be obtained by using existing graph coarsening or graph clustering approaches , as in Bruna et al . ( 2014 ) ; Defferrard et al . ( 2016 ) ; Simonovsky & Komodakis ( 2017 ) ; or learned through parameterization as in Ying et al . ( 2018b ) ; Gao & Ji ( 2019 ) ; Lee et al . ( 2019 ) . In either approach , the result include both a hierarchy of graphs and the accompanying node representations . Representation learning in these local pooling approaches is supervised , with the training signal coming from labels of the downstream task . In this work , we propose an unsupervised learning approach , named BAYESPOOL , through the use of variational Bayes . We use an existing coarsening method to obtain the graph hierarchy , an example of which is shown in Figure 1 . Then , our contribution is the learning of node representations for all graphs in this sequence . The high-level idea is to employ an encoder-decoder architecture : the encoder takes the graph and its node features as input and produces node features for the next graph in the hierarchy , whereas the decoder uses these produced node features to construct the graph . The objective is to obtain a decoding result as close to the given next graph as possible . The tool we use is variational Bayes . It is , however , slightly different from variational autoencoders ( Kingma & Welling , 2014 ) , because our decoder does not intend to reconstruct the input graph . A clear benefit of unsupervised learning is that the learned representation is not tailored to a specific downstream task and hence may be more generalizable . Moreover , the coarsening method we adopt is a recent development that holds spectral guarantee ( Loukas & Vandergheynst , 2018 ; Loukas , 2019 ) on the quality of the coarse graphs . We demonstrate the effectiveness of such a combination of hierarchical production and node representation learning in the context of graph classification . In particular , the classification performance is rather competitive with state-of-the-art supervised GNN approaches . 2 RELATED WORK . This work is in part based on graph coarsening that produces a hierarchy for a given graph . Denote byG = ( V , E ) a graph with the vertex set V and the edge setE . Graph coarsening is concerned with computing a smaller ( coarse ) graphGc = ( Vc , Ec ) with |Vc| < |V | that retains the structure ofG . A multilevel coarsening technique recursively coarsens the graph , yielding a hierarchy . Graph coarsening has been studied in the context of graph partitioning ( Karypis & Kumar , 1998 ; Dhillon et al. , 2007 ) , graph visualization ( Harel & Koren , 2000 ) , machine learning ( Lafon & Lee , 2006 ; Gavish et al. , 2010 ; Ubaru & Saad , 2019 ) , and pooling in graph neural networks ( Bruna et al. , 2014 ; Defferrard et al. , 2016 ; Simonovsky & Komodakis , 2017 ) . A variety of heuristic coarsening techniques have been proposed in different disciplines , including matching ( Hendrickson & Leland , 1995 ; Ubaru & Saad , 2019 ) , first choice ( Cong & Shinnerl , 2013 ) , contraction-based schemes ( Dhillon et al. , 2005 ; Sanders & Schulz , 2011 ) , and algebraic multigrid ( AMG ) -inspired schemes ( Sharon et al. , 2000 ; Hu & Scott , 2001 ; Ron et al. , 2011 ; Chen & Safro , 2011 ) . Many well-known software packages exist for graph coarsening , e.g. , Jostle ( Walshaw & Cross , 2007 ) , Metis ( Karypis & Kumar , 1998 ) , and DiBaP ( Meyerhenke et al. , 2008 ) . Recently , a few graph coarsening techniques achieving certain theoretical guarantees were presented ( Moitra , 2011 ; Dorfler & Bullo , 2012 ; Loukas & Vandergheynst , 2018 ) . Loukas ( 2019 ) presented variational approaches for graph coarsening with spectral guarantees . In particular , it was shown that the coarse graphs preserve the top eigenspace ( whose dimension is an input to the method ) within a predefined error tolerance . Here , we use this variational approach to obtain the graph hierarchy . This work is concerned with unsupervised graph representation learning . Recent literature has focused on generative models to achieve the same ; see . e.g. , Kipf & Welling ( 2016 ) ; Li et al . ( 2018 ) ; Ma et al . ( 2018 ) ; Simonovsky & Komodakis ( 2018 ) ; Zhang et al . ( 2019 ) . For learning hierarchical representations of graphs , most of the works that we are aware of are based on supervised learning , including Bruna et al . ( 2014 ) ; Defferrard et al . ( 2016 ) ; Simonovsky & Komodakis ( 2017 ) ; Ying et al . ( 2018b ) ; Gao & Ji ( 2019 ) ; Lee et al . ( 2019 ) . Methods most relevant to our work include : DIFFPOOL ( Ying et al. , 2018b ) , where the coarsening matrices are learned in an end-to-end fashion ; GRAPH U-NET ( Gao & Ji , 2019 ) , where graph pooling is achieved using a learnable vector and node ranking ; and SAGPOOL ( Lee et al. , 2019 ) , which is similar to GRAPH U-NET but uses graph self-attention to compute the ranking . 3 METHOD . The proposed method BAYESPOOL is an extension of variational autoencoders . As the name suggests , the goal of an autoencoder is to reconstruct the original input object after encoding it in the latent space . Our approach does not reconstruct the original input , but rather , aims as decoding an output faithful to another prescribed object . To this end , we first revisit variational Bayes and justify the use of variational lower bound for learning . Then , the machinery is applied to the graph context . 3.1 VARIATIONAL BAYES . Let x be the observed ( data ) variable and z be the unobserved ( latent ) variable . A core subject of Bayesian inference is concerned with estimating the posterior distribution p ( z|x ) . It is related to the prior p ( z ) and the likelihood p ( x|z ) through the Bayes theorem p ( z|x ) = p ( x|z ) p ( z ) ∫ p ( x , z ) dz . The challenge lies in the marginalization over z in the denominator , which is generally computationally intractable . Hence , various approximations were developed . Typically one adopts a surrogate model q ( z ) independent of data ; and recently in the context of VAEs , the data dependent distribution q ( z|x ) is often used . In our setting , we introduce a new variable x̃ and consider q ( z|x̃ ) . The difference , in terms of the Kullback–Leibler divergence , between the surrogate ( variational ) posterior q ( z|x̃ ) and the true posterior p ( z|x ) may be decomposed as DKL ( q ( z|x̃ ) || p ( z|x ) ) = ∫ q ( z|x̃ ) log q ( z|x̃ ) p ( z|x ) dz = ∫ q ( z|x̃ ) log q ( z|x̃ ) p ( z ) dz︸ ︷︷ ︸ DKL ( q ( z|x̃ ) || p ( z ) ) + ∫ q ( z|x̃ ) log p ( x ) dz︸ ︷︷ ︸ log p ( x ) − ∫ q ( z|x̃ ) log p ( x|z ) dz︸ ︷︷ ︸ Eq ( z|x̃ ) [ log p ( x|z ) ] . It consists of three terms : the KL divergence between the variational posterior and the prior p ( z ) , the log-evidence log p ( x ) , and the marginal log-likelihood log p ( x|z ) under the surrogate distribution . Because the KL divergence is nonnegative , the log-evidence is lower bounded by the combination of the other two terms : log p ( x ) ≥ Eq ( z|x̃ ) [ log p ( x|z ) ] −DKL ( q ( z|x̃ ) || p ( z ) ) . ( 1 ) The better the surrogate , the tigher the lower bound . One sees that the right-hand side of ( 1 ) is almost the same as the usual log-evidence lower bound ( ELBO ) , except that the surrogate q ( z|x̃ ) appears in place of q ( z|x ) . This observation is not surprising , because the marginalization is over the latent variable z and has nothing to do with x and x̃ . We thus conclude that the usual machinery of VAE applies , with only a notational change of the variational posterior . In the usual VAE setting , the first term of the right-hand side of ( 1 ) is considered the decoding accuracy , whereas the second term is a regularization in the latent space . Our setting follows this interpretation . 3.2 GRAPH REPRESENTATION LEARNING WITH VARIATIONAL BAYES . In our setting , a pair of graphs—the original one and the coarse one—is given . Let A ∈ Rn×n and Ac ∈ Rm×m be the corresponding graph adjacency matrices , respectively . Similarly , denote by X ∈ Rn×d and Xc ∈ Rm×d ′ the corresponding node feature matrices . We apply the encoderdecoder formalism , whereby the encoder encodes A and X into the coarse graph features Xc that we seek , such that the decoder can use Xc to decode a coarse graph as similar to Ac as possible . See Figure 2 for an illustration . Specifically , in the language of generative modeling , the encoder is the parameterized inference model that produces the parameters of q ( Xc|A , X ) and the decoder is the parameterized generative model that produces the parameters of p ( Ac|Xc ) . Following ( 1 ) , Ac plays the role of x , Xc plays the role of z , and ( A , X ) plays the role of x̃ . The variational lower bound for model learning is thus ELBO = Eq ( Xc|A , X ) [ log p ( Ac|Xc ) ] −DKL ( q ( Xc|A , X ) || p ( Xc ) ) . ( 2 ) Maxmizing the ELBO amounts to maximizing the likelihood of decoding the coarse Ac ( given coarse node features Xc resulting from the encoder ) , while minimizing a regularization of the variational posterior q ( Xc|A , X ) that departs from the latent distribution p ( Xc ) .

Summary: This work proposes an unsupervised hierarchical graph representation learning method, named BayesPool. The method learns a coarsening sequence of graphs together with the corresponding node representations. The coarsening sequence is learned using the method in Loukas (2019). The node representations are learned using an encoder-decoder structure, where the encoder encodes a graph to coarsened node representations, and the decoder decodes the node representations to a coarsened graph. The adopted objective function is analogous to VAE, except that the decoder does not aims to reconstruct an identical graph. Experiments on graph classification is performed on 5 different datasets, and competitive accuracy is achieved.