A Deep Latent Space Model for Directed Graph Representation Learning

1 INTRODUCTION . Learning representations on graphs is a fundamental problem for graph analysis and benefits a number of downstream applications , typically including link prediction , community detection and influential node identification . Traditionally , a plethora of Bayesian-based random graph models have been proposed for learning graph representations ( Holland et al. , 1983 ; Hoff et al. , 2002 ; Karrer & Newman , 2011 ; Sewell & Chen , 2015 ) . Despite the ideal interpretability of these methods , they are unscalable to model large-scale graphs due to the expensive iterative inference procedures . Boosted by the powerful representation learning ability , graph neural networks ( GNNs ) have been proposed to learn the topology of graph-structured data ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Veličković et al. , 2018 ; Zeng et al. , 2020 ) . However , these GNN-based methods usually bring about the interpretability issues , which restrict the practical applications of these methods . As an effective deep learning method based on the Bayesian theory , the variational auto-encoders ( VAEs ) ( Kingma & Welling , 2014b ) have been proposed for representation learning on graphstructured data ( Kipf & Welling , 2016 ; Grover et al. , 2019 ; Salha et al. , 2019 ) . To capture the mutual facilitation of random graph models and deep learning based methods , some recent research attempts to combine them using VAEs , and build the deep generative random graph models ( Mehta et al. , 2019 ; Sarkar et al. , 2020 ) . However , most of these VAE-based methods are designed only for undirected graphs . In contrast , a large portion of real-world graphs , such as social and citation networks , are directed graphs , of which undirected graphs are special cases ( i.e . an undirected edge can be regarded as a bidirectional edge ) . The representation learning on directed graphs is particularly challenging since not only the existences but also the directions of edges need to be learned . Although directed graphs have been extensively studied by traditional research on random graph models ( Hoff et al. , 2002 ; Krivitsky et al. , 2009 ; Sewell & Chen , 2015 ) , there are still few deep learning based methods and no combined models existed to address them . In this paper , we present the first VAE-based generative model for directed graphs and propose the Deep Latent Space Model ( DLSM ) , which combines graph convolutional networks ( GCNs ) with the Bayesian random graph model using a hierarchical VAE architecture . Within our model , a deep GCN is leveraged to encode the asymmetric adjacency matrix and node features as hidden states , which are layer-wise transformed to variational parameters to generate latent random variables as interpretable node representations . Then , the hierarchical decoder network reconstructs the adjacency matrix using the learned representations . To better capture the characteristics of graphs for deep random graph models , our method generates three types of interpretable representations , including the latent positions , node random factors and community memberships . The latent positions measure the distances between nodes and enable our method to model the link reciprocity of directed graphs , which means that both bidirectional and unidirectional edges exist . To characterize the degree heterogeneity , we introduce a pair of node random factors , namely the social activity and popularity , thus the asymmetric structure of directed edges can be modeled . Besides , we use a binary latent variable , named the community membership , to accommodate an overlapping community structure . We further prove that these node-specific variables , including the latent positions and random factors , are interpretable for representing the existence and strength of node influences . For fast inference , we leveraged the stochastic gradient variational Bayes ( SGVB ) algorithm ( Kingma & Welling , 2014a ) , which is far more efficient and scalable compared to the traditional iterative inference methods . We conduct experiments on link prediction task to verify the effectiveness of directed graph modeling with several real-world graphs , and further demonstrate the interpretability of latent variables on community detection task . The main contributions of this work are summarized as follows : 1 . We propose the DLSM model for directed graphs to incorporate the latent space model into deep learning frameworks using a hierarchical VAE architecture , which can generate highly interpretable representations of graph nodes for multiple downstream tasks . 2 . To adapt our method to directed graphs , we introduce the social activity and popularity factors of each node , with theoretical proof of their capabilities in representing node influences . 3 . We conduct experiments on five real-world graph datasets , and the results show that our proposed method can well support downstream tasks and achieve the state-of-the-art performances in link prediction and community detection . 2 RELATED WORK . In this section , we briefly review the traditional Bayesian-based random graph models and deep learning based generative graph representation methods . 2.1 BAYESIAN-BASED RANDOM GRAPH MODELS . Classic Bayesian-based random graph models have developed for decades and are still valued in modeling and generating relational data . This methods regard graph nodes as random variables following a prior distribution , and learns low-dimensional embeddings via the posterior distributions of nodes . One of the most well-known methods is the stochastic blockmodel ( SBM ) ( Holland et al. , 1983 ) , which generates a latent variable indicating the community membership of each node . Following SBM , a large number of variants have been proposed , such as allowing an overlapping community structure ( Miller et al. , 2009 ) and involving degree heterogeneity ( Karrer & Newman , 2011 ) . Another important random graph model is the latent space model ( LSM ) ( Hoff et al. , 2002 ) , which assumes each node as a position in an unobservable latent space and use distances to measure the relationships between nodes . Such method has soon been extended to model directed graphs by involving the degree heterogeneity of nodes ( Krivitsky et al. , 2009 ; Sewell & Chen , 2015 ) . Although these above methods provide good theoretical properties and learn interpretable node embeddings , they rely on either the MCMC posterior sampling or mean-field variational inference with dramatically high computational complexity , and thus are usually unfeasible when modeling large-scale graphs . 2.2 DEEP LEARNING BASED MODELS FOR UNDIRECTED GRAPHS . The intriguing achievements of deep learning models for learning representations of Euclidean data have encouraged efforts to employ them on graph-structured data . The earliest attempts include DeepWalk ( Perozzi et al. , 2014 ) and node2vec ( Grover & Leskovec , 2016 ) , both of which encode local relations as node representations by conducting random walks on a graph . GNN-based graph models More recently , many GNN-based models have been proposed for graph representation learning . Kipf & Welling ( 2017 ) first leverages the spectral GCN to learn node embeddings using the global topology of a graph . GraphSAGE ( Hamilton et al. , 2017 ) samples a fixed-size neighborhood of each node for the inductive representation learning , which allows unseen nodes to be excluded during training . The graph attention networks ( Veličković et al. , 2018 ) introduce the attention mechanism to aggregate the information from neighbors . GraphSAINT ( Zeng et al. , 2020 ) constructs minibatches by sampling subgraphs across GCN layers . ( Zhang et al. , 2021a ) proposes a multi-node representation learning method for link prediction using the node labeling trick . These methods usually learn representations for a single or a set of nodes and only focus on some specific task of graph modeling , such as node classification or link prediction . Deep generative graph models Another stream of research is the deep generative models , which attempt to generate the full structures of graphs and are typically able to be applied for multiple downstream tasks . These methods usually consist of a inferential model to capture the inner probabilistic distribution of graphs and an unsupervised generative model to generate the similar data . The variational graph auto-encoders ( VGAE ) ( Kipf & Welling , 2016 ) , for example , sample node embeddings from variational Normal distributions to reconstruct the adjacency matrix . GraphGAN Wang et al . ( 2018 ) unifies a generator and a discriminator to learn node embeddings by playing a minimax game . GraphRNN You et al . ( 2018 ) generates small graphs through a sequence of node and edge formations . Recently , some work tries to incorporate traditional random graph models into deep learning frameworks . Mehta et al . ( 2019 ) first combines the classic SBM with GCN to learn sparse node embeddings . Furthermore , Sarkar et al . ( 2020 ) builds a Gamma ladder VAE ( Sønderby et al. , 2016 ) architecture to discover the communities at multiple levels of granularities . 2.3 DEEP LEARNING BASED MODELS FOR DIRECTED GRAPHS . Despite a plethora of literature for undirected graph modeling , the deep learning based methods for directed graphs are still relatively rare . Existing methods are mostly designed for some special cases and are not generalizable enough . For instance , Salha et al . ( 2019 ) and Zhang et al . ( 2021b ) focus on the unidirectional graphs and neglect the link reciprocity of generalized directed graphs . Funke et al . ( 2020 ) learns the low-dimensional statistical manifold embedding for unsupervised directed graph modeling . Zhu et al . ( 2021 ) learns a pair of source and target embeddings for each node using the generative adversarial network . In this paper , we propose a novel deep generative model and learn interpretable node representations for generalized directed graphs where both uni- and bi-directional edges exist . 3 DEEP LATENT SPACE MODEL . Consider a directed graph containing n nodes . The input data include an asymmetric adjacency matrix A = ( Aij ) ∈ { 0 , 1 } n×n , where each binary element Aij denotes the presence ( 1 ) or not ( 0 ) of the directed edge from node i to j , and a node attribute matrix X ∈ Rn×p . Throughout this paper we assume all edges to be conditionally independent and satisfy Aij |Θ ∼ Bernoulli ( pij ) , where Θ is the collection of latent variables ( representations ) and pij ∈ ( 0 , 1 ) is the posterior probability to form an edge . The objective is to learn the representations which can best reconstruct the adjacency matrix . We propose a deep VAE architecture composed of a GCN encoder and a hierarchical latent space model ( HLSM ) decoder , as shown in Fig . 1 . The encoder block takes the adjacency A and attribute matrices X , if available , as inputs and learns a hidden state for each node using a directed GCN . Then , the decoder block recursively generates interpretable latent variables from variational posteriors via Monte Carlo sampling and reconstructs the adjacency matrix as outputs . For inference , the hierarchical architecture enables the model to learn variables using not only the priors passed from previous layers of the decoder , but also the approximate likelihood learned by the encoder . 3.1 HLSM DECODER . The HLSM decoder learns interpretable node representations based on the classic LSM approach and generates an adjacency matrix . Assume each node to correspond to a latent position in an unobservable D-dimensional space , denoted as zi ∈ RD . The distances between latent positions indicate the relationships of nodes . The closer two latent positions are , the more possible an edge will exist . Formally , an directed edge is generated by following P ( Aij = 1|X , Θ ) = σ ( β0 − βout‖γi ( zi − zj ) ‖ − βin‖δj ( zi − zj ) ‖ ) , ( 1 ) where σ ( · ) is the sigmoid function . The latent positions zi are involved as Euclidean distances , indicating the relationships between nodes . γi and δi ∈ RD are the activity and popularity factors , the reverse of which represent the tendencies for a node to send and receive an edge , respectively . The global weights βout and βin are to measure the importance of activity and popularity , respectively , and β0 is a bias . To be mentioned , though Eq . ( 1 ) is designed for directed graphs , it can be easily degraded to an undirected variant by setting γi = δi and βout = βin . We build an HLSM decoder to generate three types of latent variables , i.e . the latent positions z ( l ) i ∈ RGl , the community membership s ( l ) i ∈ RGl and the node random factors γi , δ ( l ) i ∈ RGl , where Gl is the size of the l−th decoder layer . All of these variables are randomly sampled from the variational distributions . At the first layer ( l = 1 ) , the parameters of variational distributions are defined by priors solely , while for other layers ( l = 2 , . . . , L− 1 ) , the parameters are obtained by the feedforward representations generated from previous layers as well as priors . Latent position The latent positions z ( l ) i represent the location of each node in a latent space , generated as z ( l ) i ∼ Normal ( s ( l ) i f ( W ( l−1 ) z z ( l−1 ) i ) , diag ( σ ( l ) i 2 ) ) , ( 2 ) where f ( · ) is a nonlinear activation function ( e.g . leaky ReLU ) , denotes the element-wise multiplication , diag ( · ) denotes a diagonal matrix , σ ( l ) i 2 ∈ RGl is the prior variance to be specified , and W ( l−1 ) z ∈ RGl×Gl−1 is a weight matrix to transform the variable from dimension Gl−1 to Gl . Community membership The latent positions in Eq . ( 2 ) are separated into different communities by the sparse binary variables s ( l ) i = ( si1 , . . . , siGl ) ′ , thus an overlapping community structure can be adapted . Referring to Miller et al . ( 2009 ) , we employ the Indian buffet process ( IBP ) prior on s ( l ) i to learn the effective number of communities given a sufficiently large truncation parameter Gl , i.e . π ( l ) ig = logit ( g∏ h=1 v ( l ) ih ) , s ( l ) ig ∼ Bernoulli ( σ ( π ( l ) ig ) ) . ( 3 ) Typically , the log odds π ( l ) ig is generated using a stick-breaking construction , where v ( l ) ih is drawn from a Beta distribution ( Teh et al. , 2007 ) . In our model we simplify such hierarchical prior structure by specifying a global v for all nodes . At each layer , the community membership s ( l ) ig denotes whether node i belongs to community g , hence the size of each decoder layer Gl can be explained as the number of communities . Additionally , the proposed HVAE architecture enables our model to detect community structures at multiple levels of granularities . Letting the layer sizes Gl be downward increasing , the top layers indicate the coarse-grained communities and the bottom layers indicate the fine-grained communities . Node random factors To model the prevalent power-law of node degrees , we propose the Dirichlet random factors γ ( l ) i , δ ( l ) i ∈ RGl , i.e . γ ( l ) i ∼ Dirichlet ( ξ ( l ) i + s ( l ) i f ( W ( l−1 ) γ γ ( l−1 ) i ) ) , ( 4 ) δ ( l ) i ∼ Dirichlet ( ψ ( l ) i + s ( l ) i f ( W ( l−1 ) δ δ ( l−1 ) i ) ) , ( 5 ) where ξ ( l ) i and ψ ( l ) i are prior parameters to be specified , and W ( l−1 ) γ , W ( l−1 ) δ ∈ RGl×Gl−1 are weight matrices . Note that the node random factors are dependent on the community membership s ( l ) i as well , which can be explained as only the random effects within the community are involved in the representations . At the output layer ( l = L ) , the latent positions zi and node random factors γi , δi are obtained by linear transformation from dimension GL−1 to D. Each column of the weight matrices belongs to a ( D − 1 ) -simplex . Such transformation also changes the interpretation of the layer size , from the number of communities to the dimension of the latent space . Last , the adjacency matrix is reconstructed with the latent variables by Eq . ( 1 ) .

Summary: This paper studies a deep latent space model for directed graph representation. The model is basically a hierarchical variational auto-encoder architecture and some latent variables like social activity and popularity factors of node account for the edge directions. Some latent variables are interpreted as positions in the latent space or community membership. The model, therefore, is claimed to be interpretable and superior in experiments.  ----Update after rebuttal  The authors basically answered my questions and I'd like to increase the recommendation score. I still think this submission has incremental technical contributions so my new recommendation score is 6 (marginally above the acceptance threshold).