Unsupervised Hierarchical Graph Representation Learning with Variational Bayes

1 INTRODUCTION . Graph representation learning has attracted a surge of interest recently , inspired by the widespread success of representation learning in the image and language domains through the use of deep neural networks for parameterization . A substantial number of graph neural network ( GNN ) architectures ( Bruna et al. , 2014 ; Henaff et al. , 2015 ; Duvenaud et al. , 2015 ; Defferrard et al. , 2016 ; Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Chen et al. , 2018 ; Velic̆ković et al. , 2018 ; Ying et al. , 2018a ; Liao et al. , 2019 ; Xu et al. , 2019 ) extend the convolution filters for a regular grid of data ( e.g. , image pixels , time series , and sequences ) to irregularly connected graph neighborhoods . This extension naturally stimulates the quest of also extending the pooling operation in convolutional neural networks ( CNN ) to graphs . The challenge lies in the irregular connections as opposed to a regular grid structure , whereby partitioning is straightforward . Graph pooling is used in at least two scenarios . One is global pooling : it pools the vector representations of all nodes to a single vector as the graph representation . Simple operators such as max or mean are applied . A slightly more complex operator is the weighted sum , wherein the weights are computed through attention ( Velic̆ković et al. , 2018 ; Lee et al. , 2019 ) . A recently proposed operator is top-k pooling ( Zhang et al. , 2018 ) , whereby a fixed number of node representations at the top of a sorted list is retained so that convolutions or feed-forward transformations are applied . The second use of pooling is the creation of a graph hierarchy . In this scenario , pooling is local , more similar to that in CNNs . It is interfaced with graph coarsening ( also called graph reduction or graph compression ) , a generic form of which is to cluster the nodes of the original graph into a node of the coarse graph . Then , the representations of the nodes in the cluster are pooled . The clustering may be obtained by using existing graph coarsening or graph clustering approaches , as in Bruna et al . ( 2014 ) ; Defferrard et al . ( 2016 ) ; Simonovsky & Komodakis ( 2017 ) ; or learned through parameterization as in Ying et al . ( 2018b ) ; Gao & Ji ( 2019 ) ; Lee et al . ( 2019 ) . In either approach , the result include both a hierarchy of graphs and the accompanying node representations . Representation learning in these local pooling approaches is supervised , with the training signal coming from labels of the downstream task . In this work , we propose an unsupervised learning approach , named BAYESPOOL , through the use of variational Bayes . We use an existing coarsening method to obtain the graph hierarchy , an example of which is shown in Figure 1 . Then , our contribution is the learning of node representations for all graphs in this sequence . The high-level idea is to employ an encoder-decoder architecture : the encoder takes the graph and its node features as input and produces node features for the next graph in the hierarchy , whereas the decoder uses these produced node features to construct the graph . The objective is to obtain a decoding result as close to the given next graph as possible . The tool we use is variational Bayes . It is , however , slightly different from variational autoencoders ( Kingma & Welling , 2014 ) , because our decoder does not intend to reconstruct the input graph . A clear benefit of unsupervised learning is that the learned representation is not tailored to a specific downstream task and hence may be more generalizable . Moreover , the coarsening method we adopt is a recent development that holds spectral guarantee ( Loukas & Vandergheynst , 2018 ; Loukas , 2019 ) on the quality of the coarse graphs . We demonstrate the effectiveness of such a combination of hierarchical production and node representation learning in the context of graph classification . In particular , the classification performance is rather competitive with state-of-the-art supervised GNN approaches . 2 RELATED WORK . This work is in part based on graph coarsening that produces a hierarchy for a given graph . Denote byG = ( V , E ) a graph with the vertex set V and the edge setE . Graph coarsening is concerned with computing a smaller ( coarse ) graphGc = ( Vc , Ec ) with |Vc| < |V | that retains the structure ofG . A multilevel coarsening technique recursively coarsens the graph , yielding a hierarchy . Graph coarsening has been studied in the context of graph partitioning ( Karypis & Kumar , 1998 ; Dhillon et al. , 2007 ) , graph visualization ( Harel & Koren , 2000 ) , machine learning ( Lafon & Lee , 2006 ; Gavish et al. , 2010 ; Ubaru & Saad , 2019 ) , and pooling in graph neural networks ( Bruna et al. , 2014 ; Defferrard et al. , 2016 ; Simonovsky & Komodakis , 2017 ) . A variety of heuristic coarsening techniques have been proposed in different disciplines , including matching ( Hendrickson & Leland , 1995 ; Ubaru & Saad , 2019 ) , first choice ( Cong & Shinnerl , 2013 ) , contraction-based schemes ( Dhillon et al. , 2005 ; Sanders & Schulz , 2011 ) , and algebraic multigrid ( AMG ) -inspired schemes ( Sharon et al. , 2000 ; Hu & Scott , 2001 ; Ron et al. , 2011 ; Chen & Safro , 2011 ) . Many well-known software packages exist for graph coarsening , e.g. , Jostle ( Walshaw & Cross , 2007 ) , Metis ( Karypis & Kumar , 1998 ) , and DiBaP ( Meyerhenke et al. , 2008 ) . Recently , a few graph coarsening techniques achieving certain theoretical guarantees were presented ( Moitra , 2011 ; Dorfler & Bullo , 2012 ; Loukas & Vandergheynst , 2018 ) . Loukas ( 2019 ) presented variational approaches for graph coarsening with spectral guarantees . In particular , it was shown that the coarse graphs preserve the top eigenspace ( whose dimension is an input to the method ) within a predefined error tolerance . Here , we use this variational approach to obtain the graph hierarchy . This work is concerned with unsupervised graph representation learning . Recent literature has focused on generative models to achieve the same ; see . e.g. , Kipf & Welling ( 2016 ) ; Li et al . ( 2018 ) ; Ma et al . ( 2018 ) ; Simonovsky & Komodakis ( 2018 ) ; Zhang et al . ( 2019 ) . For learning hierarchical representations of graphs , most of the works that we are aware of are based on supervised learning , including Bruna et al . ( 2014 ) ; Defferrard et al . ( 2016 ) ; Simonovsky & Komodakis ( 2017 ) ; Ying et al . ( 2018b ) ; Gao & Ji ( 2019 ) ; Lee et al . ( 2019 ) . Methods most relevant to our work include : DIFFPOOL ( Ying et al. , 2018b ) , where the coarsening matrices are learned in an end-to-end fashion ; GRAPH U-NET ( Gao & Ji , 2019 ) , where graph pooling is achieved using a learnable vector and node ranking ; and SAGPOOL ( Lee et al. , 2019 ) , which is similar to GRAPH U-NET but uses graph self-attention to compute the ranking . 3 METHOD . The proposed method BAYESPOOL is an extension of variational autoencoders . As the name suggests , the goal of an autoencoder is to reconstruct the original input object after encoding it in the latent space . Our approach does not reconstruct the original input , but rather , aims as decoding an output faithful to another prescribed object . To this end , we first revisit variational Bayes and justify the use of variational lower bound for learning . Then , the machinery is applied to the graph context . 3.1 VARIATIONAL BAYES . Let x be the observed ( data ) variable and z be the unobserved ( latent ) variable . A core subject of Bayesian inference is concerned with estimating the posterior distribution p ( z|x ) . It is related to the prior p ( z ) and the likelihood p ( x|z ) through the Bayes theorem p ( z|x ) = p ( x|z ) p ( z ) ∫ p ( x , z ) dz . The challenge lies in the marginalization over z in the denominator , which is generally computationally intractable . Hence , various approximations were developed . Typically one adopts a surrogate model q ( z ) independent of data ; and recently in the context of VAEs , the data dependent distribution q ( z|x ) is often used . In our setting , we introduce a new variable x̃ and consider q ( z|x̃ ) . The difference , in terms of the Kullback–Leibler divergence , between the surrogate ( variational ) posterior q ( z|x̃ ) and the true posterior p ( z|x ) may be decomposed as DKL ( q ( z|x̃ ) || p ( z|x ) ) = ∫ q ( z|x̃ ) log q ( z|x̃ ) p ( z|x ) dz = ∫ q ( z|x̃ ) log q ( z|x̃ ) p ( z ) dz︸ ︷︷ ︸ DKL ( q ( z|x̃ ) || p ( z ) ) + ∫ q ( z|x̃ ) log p ( x ) dz︸ ︷︷ ︸ log p ( x ) − ∫ q ( z|x̃ ) log p ( x|z ) dz︸ ︷︷ ︸ Eq ( z|x̃ ) [ log p ( x|z ) ] . It consists of three terms : the KL divergence between the variational posterior and the prior p ( z ) , the log-evidence log p ( x ) , and the marginal log-likelihood log p ( x|z ) under the surrogate distribution . Because the KL divergence is nonnegative , the log-evidence is lower bounded by the combination of the other two terms : log p ( x ) ≥ Eq ( z|x̃ ) [ log p ( x|z ) ] −DKL ( q ( z|x̃ ) || p ( z ) ) . ( 1 ) The better the surrogate , the tigher the lower bound . One sees that the right-hand side of ( 1 ) is almost the same as the usual log-evidence lower bound ( ELBO ) , except that the surrogate q ( z|x̃ ) appears in place of q ( z|x ) . This observation is not surprising , because the marginalization is over the latent variable z and has nothing to do with x and x̃ . We thus conclude that the usual machinery of VAE applies , with only a notational change of the variational posterior . In the usual VAE setting , the first term of the right-hand side of ( 1 ) is considered the decoding accuracy , whereas the second term is a regularization in the latent space . Our setting follows this interpretation . 3.2 GRAPH REPRESENTATION LEARNING WITH VARIATIONAL BAYES . In our setting , a pair of graphs—the original one and the coarse one—is given . Let A ∈ Rn×n and Ac ∈ Rm×m be the corresponding graph adjacency matrices , respectively . Similarly , denote by X ∈ Rn×d and Xc ∈ Rm×d ′ the corresponding node feature matrices . We apply the encoderdecoder formalism , whereby the encoder encodes A and X into the coarse graph features Xc that we seek , such that the decoder can use Xc to decode a coarse graph as similar to Ac as possible . See Figure 2 for an illustration . Specifically , in the language of generative modeling , the encoder is the parameterized inference model that produces the parameters of q ( Xc|A , X ) and the decoder is the parameterized generative model that produces the parameters of p ( Ac|Xc ) . Following ( 1 ) , Ac plays the role of x , Xc plays the role of z , and ( A , X ) plays the role of x̃ . The variational lower bound for model learning is thus ELBO = Eq ( Xc|A , X ) [ log p ( Ac|Xc ) ] −DKL ( q ( Xc|A , X ) || p ( Xc ) ) . ( 2 ) Maxmizing the ELBO amounts to maximizing the likelihood of decoding the coarse Ac ( given coarse node features Xc resulting from the encoder ) , while minimizing a regularization of the variational posterior q ( Xc|A , X ) that departs from the latent distribution p ( Xc ) .

Summary: The authors propose in this paper a new unsupervised graph representation learning method. The method leverages recent advances in graph coarsening, mainly Loukas' method. The key idea of the method consists in using a reconstruction target that is not the classical one in an auto-encoder setting. More precisely, the encoder takes as an input the original adjacency matrix and node features but the decode only aims at reconstructing the coarse adjacency matrix (obtained via Loukas' method).