A Deep Latent Space Model for Directed Graph Representation Learning

1 INTRODUCTION . Learning representations on graphs is a fundamental problem for graph analysis and benefits a number of downstream applications , typically including link prediction , community detection and influential node identification . Traditionally , a plethora of Bayesian-based random graph models have been proposed for learning graph representations ( Holland et al. , 1983 ; Hoff et al. , 2002 ; Karrer & Newman , 2011 ; Sewell & Chen , 2015 ) . Despite the ideal interpretability of these methods , they are unscalable to model large-scale graphs due to the expensive iterative inference procedures . Boosted by the powerful representation learning ability , graph neural networks ( GNNs ) have been proposed to learn the topology of graph-structured data ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Veličković et al. , 2018 ; Zeng et al. , 2020 ) . However , these GNN-based methods usually bring about the interpretability issues , which restrict the practical applications of these methods . As an effective deep learning method based on the Bayesian theory , the variational auto-encoders ( VAEs ) ( Kingma & Welling , 2014b ) have been proposed for representation learning on graphstructured data ( Kipf & Welling , 2016 ; Grover et al. , 2019 ; Salha et al. , 2019 ) . To capture the mutual facilitation of random graph models and deep learning based methods , some recent research attempts to combine them using VAEs , and build the deep generative random graph models ( Mehta et al. , 2019 ; Sarkar et al. , 2020 ) . However , most of these VAE-based methods are designed only for undirected graphs . In contrast , a large portion of real-world graphs , such as social and citation networks , are directed graphs , of which undirected graphs are special cases ( i.e . an undirected edge can be regarded as a bidirectional edge ) . The representation learning on directed graphs is particularly challenging since not only the existences but also the directions of edges need to be learned . Although directed graphs have been extensively studied by traditional research on random graph models ( Hoff et al. , 2002 ; Krivitsky et al. , 2009 ; Sewell & Chen , 2015 ) , there are still few deep learning based methods and no combined models existed to address them . In this paper , we present the first VAE-based generative model for directed graphs and propose the Deep Latent Space Model ( DLSM ) , which combines graph convolutional networks ( GCNs ) with the Bayesian random graph model using a hierarchical VAE architecture . Within our model , a deep GCN is leveraged to encode the asymmetric adjacency matrix and node features as hidden states , which are layer-wise transformed to variational parameters to generate latent random variables as interpretable node representations . Then , the hierarchical decoder network reconstructs the adjacency matrix using the learned representations . To better capture the characteristics of graphs for deep random graph models , our method generates three types of interpretable representations , including the latent positions , node random factors and community memberships . The latent positions measure the distances between nodes and enable our method to model the link reciprocity of directed graphs , which means that both bidirectional and unidirectional edges exist . To characterize the degree heterogeneity , we introduce a pair of node random factors , namely the social activity and popularity , thus the asymmetric structure of directed edges can be modeled . Besides , we use a binary latent variable , named the community membership , to accommodate an overlapping community structure . We further prove that these node-specific variables , including the latent positions and random factors , are interpretable for representing the existence and strength of node influences . For fast inference , we leveraged the stochastic gradient variational Bayes ( SGVB ) algorithm ( Kingma & Welling , 2014a ) , which is far more efficient and scalable compared to the traditional iterative inference methods . We conduct experiments on link prediction task to verify the effectiveness of directed graph modeling with several real-world graphs , and further demonstrate the interpretability of latent variables on community detection task . The main contributions of this work are summarized as follows : 1 . We propose the DLSM model for directed graphs to incorporate the latent space model into deep learning frameworks using a hierarchical VAE architecture , which can generate highly interpretable representations of graph nodes for multiple downstream tasks . 2 . To adapt our method to directed graphs , we introduce the social activity and popularity factors of each node , with theoretical proof of their capabilities in representing node influences . 3 . We conduct experiments on five real-world graph datasets , and the results show that our proposed method can well support downstream tasks and achieve the state-of-the-art performances in link prediction and community detection . 2 RELATED WORK . In this section , we briefly review the traditional Bayesian-based random graph models and deep learning based generative graph representation methods . 2.1 BAYESIAN-BASED RANDOM GRAPH MODELS . Classic Bayesian-based random graph models have developed for decades and are still valued in modeling and generating relational data . This methods regard graph nodes as random variables following a prior distribution , and learns low-dimensional embeddings via the posterior distributions of nodes . One of the most well-known methods is the stochastic blockmodel ( SBM ) ( Holland et al. , 1983 ) , which generates a latent variable indicating the community membership of each node . Following SBM , a large number of variants have been proposed , such as allowing an overlapping community structure ( Miller et al. , 2009 ) and involving degree heterogeneity ( Karrer & Newman , 2011 ) . Another important random graph model is the latent space model ( LSM ) ( Hoff et al. , 2002 ) , which assumes each node as a position in an unobservable latent space and use distances to measure the relationships between nodes . Such method has soon been extended to model directed graphs by involving the degree heterogeneity of nodes ( Krivitsky et al. , 2009 ; Sewell & Chen , 2015 ) . Although these above methods provide good theoretical properties and learn interpretable node embeddings , they rely on either the MCMC posterior sampling or mean-field variational inference with dramatically high computational complexity , and thus are usually unfeasible when modeling large-scale graphs . 2.2 DEEP LEARNING BASED MODELS FOR UNDIRECTED GRAPHS . The intriguing achievements of deep learning models for learning representations of Euclidean data have encouraged efforts to employ them on graph-structured data . The earliest attempts include DeepWalk ( Perozzi et al. , 2014 ) and node2vec ( Grover & Leskovec , 2016 ) , both of which encode local relations as node representations by conducting random walks on a graph . GNN-based graph models More recently , many GNN-based models have been proposed for graph representation learning . Kipf & Welling ( 2017 ) first leverages the spectral GCN to learn node embeddings using the global topology of a graph . GraphSAGE ( Hamilton et al. , 2017 ) samples a fixed-size neighborhood of each node for the inductive representation learning , which allows unseen nodes to be excluded during training . The graph attention networks ( Veličković et al. , 2018 ) introduce the attention mechanism to aggregate the information from neighbors . GraphSAINT ( Zeng et al. , 2020 ) constructs minibatches by sampling subgraphs across GCN layers . ( Zhang et al. , 2021a ) proposes a multi-node representation learning method for link prediction using the node labeling trick . These methods usually learn representations for a single or a set of nodes and only focus on some specific task of graph modeling , such as node classification or link prediction . Deep generative graph models Another stream of research is the deep generative models , which attempt to generate the full structures of graphs and are typically able to be applied for multiple downstream tasks . These methods usually consist of a inferential model to capture the inner probabilistic distribution of graphs and an unsupervised generative model to generate the similar data . The variational graph auto-encoders ( VGAE ) ( Kipf & Welling , 2016 ) , for example , sample node embeddings from variational Normal distributions to reconstruct the adjacency matrix . GraphGAN Wang et al . ( 2018 ) unifies a generator and a discriminator to learn node embeddings by playing a minimax game . GraphRNN You et al . ( 2018 ) generates small graphs through a sequence of node and edge formations . Recently , some work tries to incorporate traditional random graph models into deep learning frameworks . Mehta et al . ( 2019 ) first combines the classic SBM with GCN to learn sparse node embeddings . Furthermore , Sarkar et al . ( 2020 ) builds a Gamma ladder VAE ( Sønderby et al. , 2016 ) architecture to discover the communities at multiple levels of granularities . 2.3 DEEP LEARNING BASED MODELS FOR DIRECTED GRAPHS . Despite a plethora of literature for undirected graph modeling , the deep learning based methods for directed graphs are still relatively rare . Existing methods are mostly designed for some special cases and are not generalizable enough . For instance , Salha et al . ( 2019 ) and Zhang et al . ( 2021b ) focus on the unidirectional graphs and neglect the link reciprocity of generalized directed graphs . Funke et al . ( 2020 ) learns the low-dimensional statistical manifold embedding for unsupervised directed graph modeling . Zhu et al . ( 2021 ) learns a pair of source and target embeddings for each node using the generative adversarial network . In this paper , we propose a novel deep generative model and learn interpretable node representations for generalized directed graphs where both uni- and bi-directional edges exist . 3 DEEP LATENT SPACE MODEL . Consider a directed graph containing n nodes . The input data include an asymmetric adjacency matrix A = ( Aij ) ∈ { 0 , 1 } n×n , where each binary element Aij denotes the presence ( 1 ) or not ( 0 ) of the directed edge from node i to j , and a node attribute matrix X ∈ Rn×p . Throughout this paper we assume all edges to be conditionally independent and satisfy Aij |Θ ∼ Bernoulli ( pij ) , where Θ is the collection of latent variables ( representations ) and pij ∈ ( 0 , 1 ) is the posterior probability to form an edge . The objective is to learn the representations which can best reconstruct the adjacency matrix . We propose a deep VAE architecture composed of a GCN encoder and a hierarchical latent space model ( HLSM ) decoder , as shown in Fig . 1 . The encoder block takes the adjacency A and attribute matrices X , if available , as inputs and learns a hidden state for each node using a directed GCN . Then , the decoder block recursively generates interpretable latent variables from variational posteriors via Monte Carlo sampling and reconstructs the adjacency matrix as outputs . For inference , the hierarchical architecture enables the model to learn variables using not only the priors passed from previous layers of the decoder , but also the approximate likelihood learned by the encoder . 3.1 HLSM DECODER . The HLSM decoder learns interpretable node representations based on the classic LSM approach and generates an adjacency matrix . Assume each node to correspond to a latent position in an unobservable D-dimensional space , denoted as zi ∈ RD . The distances between latent positions indicate the relationships of nodes . The closer two latent positions are , the more possible an edge will exist . Formally , an directed edge is generated by following P ( Aij = 1|X , Θ ) = σ ( β0 − βout‖γi ( zi − zj ) ‖ − βin‖δj ( zi − zj ) ‖ ) , ( 1 ) where σ ( · ) is the sigmoid function . The latent positions zi are involved as Euclidean distances , indicating the relationships between nodes . γi and δi ∈ RD are the activity and popularity factors , the reverse of which represent the tendencies for a node to send and receive an edge , respectively . The global weights βout and βin are to measure the importance of activity and popularity , respectively , and β0 is a bias . To be mentioned , though Eq . ( 1 ) is designed for directed graphs , it can be easily degraded to an undirected variant by setting γi = δi and βout = βin . We build an HLSM decoder to generate three types of latent variables , i.e . the latent positions z ( l ) i ∈ RGl , the community membership s ( l ) i ∈ RGl and the node random factors γi , δ ( l ) i ∈ RGl , where Gl is the size of the l−th decoder layer . All of these variables are randomly sampled from the variational distributions . At the first layer ( l = 1 ) , the parameters of variational distributions are defined by priors solely , while for other layers ( l = 2 , . . . , L− 1 ) , the parameters are obtained by the feedforward representations generated from previous layers as well as priors . Latent position The latent positions z ( l ) i represent the location of each node in a latent space , generated as z ( l ) i ∼ Normal ( s ( l ) i f ( W ( l−1 ) z z ( l−1 ) i ) , diag ( σ ( l ) i 2 ) ) , ( 2 ) where f ( · ) is a nonlinear activation function ( e.g . leaky ReLU ) , denotes the element-wise multiplication , diag ( · ) denotes a diagonal matrix , σ ( l ) i 2 ∈ RGl is the prior variance to be specified , and W ( l−1 ) z ∈ RGl×Gl−1 is a weight matrix to transform the variable from dimension Gl−1 to Gl . Community membership The latent positions in Eq . ( 2 ) are separated into different communities by the sparse binary variables s ( l ) i = ( si1 , . . . , siGl ) ′ , thus an overlapping community structure can be adapted . Referring to Miller et al . ( 2009 ) , we employ the Indian buffet process ( IBP ) prior on s ( l ) i to learn the effective number of communities given a sufficiently large truncation parameter Gl , i.e . π ( l ) ig = logit ( g∏ h=1 v ( l ) ih ) , s ( l ) ig ∼ Bernoulli ( σ ( π ( l ) ig ) ) . ( 3 ) Typically , the log odds π ( l ) ig is generated using a stick-breaking construction , where v ( l ) ih is drawn from a Beta distribution ( Teh et al. , 2007 ) . In our model we simplify such hierarchical prior structure by specifying a global v for all nodes . At each layer , the community membership s ( l ) ig denotes whether node i belongs to community g , hence the size of each decoder layer Gl can be explained as the number of communities . Additionally , the proposed HVAE architecture enables our model to detect community structures at multiple levels of granularities . Letting the layer sizes Gl be downward increasing , the top layers indicate the coarse-grained communities and the bottom layers indicate the fine-grained communities . Node random factors To model the prevalent power-law of node degrees , we propose the Dirichlet random factors γ ( l ) i , δ ( l ) i ∈ RGl , i.e . γ ( l ) i ∼ Dirichlet ( ξ ( l ) i + s ( l ) i f ( W ( l−1 ) γ γ ( l−1 ) i ) ) , ( 4 ) δ ( l ) i ∼ Dirichlet ( ψ ( l ) i + s ( l ) i f ( W ( l−1 ) δ δ ( l−1 ) i ) ) , ( 5 ) where ξ ( l ) i and ψ ( l ) i are prior parameters to be specified , and W ( l−1 ) γ , W ( l−1 ) δ ∈ RGl×Gl−1 are weight matrices . Note that the node random factors are dependent on the community membership s ( l ) i as well , which can be explained as only the random effects within the community are involved in the representations . At the output layer ( l = L ) , the latent positions zi and node random factors γi , δi are obtained by linear transformation from dimension GL−1 to D. Each column of the weight matrices belongs to a ( D − 1 ) -simplex . Such transformation also changes the interpretation of the layer size , from the number of communities to the dimension of the latent space . Last , the adjacency matrix is reconstructed with the latent variables by Eq . ( 1 ) .

Summary: This paper proposes a VAE-based graph representation learning model for directed graphs. The model is composed of a GCN encoder and a hierarchical latent space model decoder. The decoder explicitly models three kinds of node representations: the node latent position vectors $z$, the community membership vectors $s$ and node random factors $\gamma,\delta$. Specifically, the community membership vectors $s$ are used to gate control which of $z$ and $\gamma,\delta$ pass to the next layer. And finally the adjacency matrix is reconstructed by the distance between $z$'s filtered by node influences $\gamma,\delta$. Experiments show better link prediction and community detection performance than plain VGAE and other VGAE methods.