Low-dimensional statistical manifold embedding of directed graphs

We propose a novel node embedding of directed graphs to statistical manifolds , which is based on a global minimization of pairwise relative entropy and graph geodesics in a non-linear way . Each node is encoded with a probability density function over a measurable space . Furthermore , we analyze the connection between the geometrical properties of such embedding and their efficient learning procedure . Extensive experiments show that our proposed embedding is better in preserving the global geodesic information of graphs , as well as outperforming existing embedding models on directed graphs in a variety of evaluation metrics , in an unsupervised setting . 1 INTRODUCTION . In this publication , we study the directed graph embedding problem in an unsupervised learning setting . A graph embedding problem is usually defined as a problem of finding a vector representation X ∈ RK for every node of a graph G = ( V , E ) through a mapping φ : V → X . On every graph G = ( V , E ) , defined with set of nodes V and set of edgesE , the distance dG : V ×V → R+ between two vertices is defined as the number of edges connecting them in the shortest path , also called a graph geodesic . In case that X is equipped with a distance metric function dX : X ×X → R+ , we can quantify the embedding distortion by measuring the ratio dX / dG between pairs of corresponding embedded points and nodes . Alternatively , one can measure the quality of the mapping by using a similarity function between points . In contrast to distance , the similarity is usually a bounded value [ 0 , 1 ] and is in some ad-hoc way connected to a notion of distance e.g . inverse of the distance . Usually , we are interested in a low-dimensional ( K n ) embedding of graphs with n nodes as it is always possible to find an embedding [ 32 ] with L∞ norm with no distortion in an n-dimensional Euclidean space . Bourgain theorem ( 1985 ) [ 6 ] proved that it is possible to construct an O ( log ( n ) ) -dimensional Euclidean embedding of an undirected graph with n nodes with finite distortion . For the last couple of decades , different communities such as machine learning [ 16 ; 33 ; 10 ; 40 ; 14 ] , physics [ 29 ; 38 ; 35 ] , and computer science [ 42 ; 43 ] independently from mathematics community [ 32 ; 25 ; 12 ; 6 ] developed novel graph representation techniques . In a nutshell , they can be characterized by having one of the following properties ( i-iv ) . ( i ) The Type of loss function that quantifies the distortion is optimizing local neighbourhood [ 44 ; 21 ] or global graph structure properties [ 53 ; 40 ; 14 ] . ( ii ) The target property to be preserved is either geodesic ( shortest paths ) [ 42 ; 43 ; 12 ] or diffusion distance ( heat or random walk process ) [ 46 ; 40 ; 14 ] or another similarity property [ 21 ; 55 ; 39 ] . ( iii ) The mapping φ is a linear [ 23 ; 39 ] or non-linear dimensionality reduction technique such as SNE [ 21 ] , t-SNE [ 55 ] , ISOMAP [ 53 ] , Laplacian eigenmaps [ 3 ] or deep learning work DeepWalk [ 40 ] , node2vec [ 14 ] , HOPE [ 39 ] , GraphSAGE [ 17 ] , NetMF [ 33 ] and others [ 16 ] . ( iv ) The geometry of 20 0 20 20 0 20 0 50 100 150 200 epochs 25 75 0 50 100 150 200 epochs 0Figure 1 : Visualization of synthetic example network together with our embedding . The graph is embedded into the 2-variate normal distributions , which are represented by a σ ellipse ( boundary of 1 standard deviation around mean µ ) . The σ ellipses of the green nodes are contained in one of the greys nodes , which represents that the distance ( measured with divergence in embedded space ) between green and grey is small , but in the opposite direction very large . X has zero curvature ( Euclidean ) [ 32 ; 12 ] , positive curvature [ 57 ] ( spherical ) or negative curvature [ 29 ; 35 ; 10 ] ( hyperbolic ) or mixed curvatures [ 15 ] . However , for directed graphs the asymmetry of graph shortest path distances dG ( vi , vj ) 6= dG ( vj , vi ) is violating the symmetry assumption ( not a metric function ) . This is the reason why only recently this problem was tackled by constructing two independent representations for each node ( source and target representations ) [ 39 ; 59 ; 27 ] . In this paper , we propose a single-node embedding for directed graphs to the space of probability distributions , described with the geometry of statistical manifold theory . Up to this point , different probability distributions were used for different embedding purposes such as : ( i ) applications to word embedding [ 45 ; 56 ] , knowledge graphs [ 20 ] , attributed graphs [ 4 ] or ( ii ) generalized point embeddings with elliptical distributions [ 36 ] , but not for low-dimensional embedding of directed graphs , characterized by the theory of statistical manifolds . The main contributions of this paper are : ( i ) We propose a node embedding for directed graphs to the elements of the statistical manifolds [ 51 ] , to jointly capture asymmetric and infinite distances in a low-dimensional space . We determine the connection between geometry and gradient-based optimization . ( ii ) Furthermore , we develop a sampling-based scalable learning algorithm for largescale networks . ( iii ) Extensive experiments demonstrate that the proposed embedding outperforms several directed network embedding baselines , i.e . random walks based [ 59 ] and matrix factorization based [ 39 ] methods , as well as the deep learning undirected representative [ 40 ] and deep Gaussian embeddings for attributed graphs [ 4 ] . 2 STATISTICAL MANIFOLD DIVERGENCE EMBEDDING FOR DIRECTED GRAPHS . In this section , we analyze the limits of metric embedding . Then , we continue by providing an intuition of our embedding with a synthetic example before presenting the learning procedure of the embedding method . Bourgain theorem ( 1985 ) [ 6 ] states that it is possible to construct an O ( log ( n ) ) dimensional embedding ( φ : v 7→ xv ) of an undirected graph with n nodes to Euclidean space with O ( log ( n ) ) distortion by using random subset projection . But , on directed graph G = ( V , E ) , the directed shortest paths are not necessarily symmetric dv , u 6= du , v . This does not imply that it is not possible to provide low dimensional metric embedding [ 6 ; 32 ; 25 ; 12 ] with a variant of Bourgain theorem . In the appendix A.1 , we show that the main problem is not asymmetry , but the existence of infinite directed distances , which is quite common in real directed networks , see Table 1 . 2.1 LEARNING STATISTICAL MANIFOLD EMBEDDING . Instead of a conventional network embedding in a metric space , we propose to use the KullbackLeibler divergence defined on probability distributions to capture asymmetric finite and infinite distances in directed graphs . Each node is mapped to a corresponding distribution described by some parameters , which we learn during training . In this paper , we choose the class of exponential power distributions due to analytical and computational merits . With minor modifications , our method can also be applied to other distribution classes . Intuition . In Fig . 1 , we use a toy example to qualitatively illustrate that our embedding can represent infinite distances without requiring a high dimensional embedding space . More precisely , the network consists of five groups ( color-coded ) , with five nodes each . The two top blocks have a connection only to the center group and , similarly , the two bottom groups have only a link from the central group to them . Our model learns this connectivity pattern by embedding members of these groups in a similar fashion . In this example , each node u is embedded as a 2-variate normal distribution with mean µu = ( µ1u , µ 2 u ) and variances Σu = diag ( σ 1 u , σ 2 u ) , which can be visualized by the σ-ellipse , the curve which represent one-σ distance from their respective means . The nodes of graphs are embedded by minimizing the difference between the pair-wise graph distances and corresponding Kullback-Leibler divergences between embeddings . For two distribution p ( x ) , q ( x ) , if p ( x ) q ( x ) on an open subset U , the Kullback-Leibler divergence KL ( p ( x ) , q ( x ) ) is then relatively high . In other words , if in Figure 1 the σ-ellipse of node u is contained in or very similar to the σ-ellipse of node v , then the embedding represents d ( u , v ) < ∞ . Using this , we see that the embedding retrieved by our optimization and visualized in Fig . 1 includes most of the observed infinite distances . Statistical manifold embedding . Our embedding space X is the space of k-variate exponential power distributions [ 37 ] ( also called generalized error distributions ) , described by the following probability density function for λ > 0 ψλ ( x|Σ , µ ) = λΓ ( k2 ) 21+ k λπ k 2 det ( Σ ) 1 2 Γ ( kλ ) exp ( − [ ( x− µ ) TΣ−1 ( x− µ ) ] λ2 2 ) ( 1 ) with mean vector µ ∈ Rk and covariance matrix Σ ∈ Rk×k . Γ ( . ) denotes the Gamma function and xT denotes the transposed vector of x . Note that λ = 2 results in the multivariate Gaussian distribution and λ = 1 yields the multivariate Laplacian distribution . As we are interested in non-degenerate distributions , we enforce positive definite co-variance matrices and further restrict ourselves to diagonal matrices , i.e . Σu = diag ( σ1u , . . . , σ k u ) with σ i u ∈ R+ . With the latter , we reduce the degrees of freedom for each node to 2k and simplify our optimization by replacing a positive definite constraint on Σu with the constraints σiu > 0 . A common asymmetric function operating on continuous random variables is the Kullback-Leibler divergence . The asymmetric distance between nodes u and v , denoted by KLu , v , is KLu , v = KL ( p λ u , p λ v ) = ∫ pλu log pλu pλv dx with pλu = ψλ ( x|Σu , µu ) and pλv = ψλ ( x|Σv , µv ) . We approximate it with importance sampling Monte Carlo estimation . In particular , KLu , v can be expressed as the expectation ( for λ∗ = 2 ≤ λ ) KLu , v = Ex∼pλ∗u [ pλu pλ∗u log pλu pλv ] , where pλ∗u = ψλ∗ ( x|Σu , µu ) is easy to sample and a proposal function for pλ ( x|Σu , µu ) . If a closed expression for KLu , v is known , its evaluation replaces the importance sampling Monte Carlo estimation , like in the special case of λ = 2 . See Appendix A.9 for more details . Now , in order to learn these statistical manifold embeddings , we can define the loss function over asymmetric distances D = ( du , v ) u , v∈V of a directed graph and { ( µu , Σu ) } u as : L ( { ( µu , Σu ) } u ) = ∑ u6=v || ( 1 + τ KLu , v ) −1 − d−βu , v ||22 , ( 2 ) where τ ∈ R+ is a free ( trainable ) parameter and β ∈ R+ a fixed value . This loss function given in the Eq . ( 2 ) is minimized during the learning process so that the KLu , v based on learned node distribution representations captures the distances du , v in the directed graph . Empirically , we transform the given distances into finite numbers with d−βu , v ∈ [ 0 , 1 ] , for all u 6= v and a β ∈ R+ , which can be used to increase the differentiation between large du , v values and between finite and infinite distances . We modify the unbounded Kullback-Leibler divergence in a similar fashion ( 1 + τ KLu , v ) −1 ∈ [ 0 , 1 ] , where τ ∈ R+ is a free ( trainable ) parameter and the additional 1 ensures a value in the same interval [ 0 , 1 ] . We start the optimization from random initial parameters { µu , Σu } u∈V , and iteratively minimize the loss function with stochastic gradient descent optimizers , such as Adam [ 28 ] . For small k , which we will consider in the next section , more enhanced initialization strategies , such as using graph plotting algorithms to determine the means and exploiting memberships of strongly connected components for the covariance initialization , probably reduce the number of training epochs . Scalable learning procedure . The full objective function Eq . ( 2 ) consists of |V | ( |V | − 1 ) terms and only graphs with up to the magnitude of around 104 nodes can be applied . To extend our method beyond this limit , we propose an approximated solution , where the size of the training data scales linearly in the number of nodes and thus can be applied to large graphs . This approximation solution is based on a decomposition of the loss function Eq . ( 2 ) into a neighborhood term and a singularity term as : L ( { ( µu , Σu ) } u ) = ∑ u 6=v , du , v < ∞ || ( 1 + τ KLu , v ) −1 − d−βu , v ||22︸ ︷︷ ︸ neighborhood term + ∑ u6=v , du , v=∞ || ( 1 + τ KLu , v ) −1 − d−βu , v ||22︸ ︷︷ ︸ singularity term . We efficiently sample from each of these sums for each node a small number of B samples and optimize our model based on the sampled information about closeness and infinite distances . One straightforward approach to approximate the neighborhood term is to use all direct neighbors . Yet , the number of samples available via the directed neighbors is limited by |E| , which usually corresponds to a small B in sparse real-world examples . Therefore , we apply for each node a breadth-first-search into both directions until we retrieve B new samples . In this way , we obtain smaller du , v first , and the approximation tends to the original term in the limit B → |V | . For the singularity term , we use the topological sorting [ 26 ] of the strongly connected components . In a topological sorting of a directed acyclic graph , all edges are from lower indices to higher indices ( with respect to the topological sorting ) . Two important restrictions are noteworthy . First , the reverse statement does not hold , i.e . not all infinite distances are given via a single topological sorting [ 26 ] . As a consequence , we won ’ t sample uniformly from all infinities , but from a subset of the infinities given by the topological sorting . Second , the construction of topological sorting is only possible for acyclic graphs , and most directed graphs have cycles . Since we are only interested in sampling singularities and the graph defined by the strongly connected components of a directed graph is acyclic , we use Tarjan ’ s algorithm [ 52 ] to retrieve the strongly connected components and topological sorting of them . After this preprocessing , we can efficiently generate samples of infinite distance for the nodes . With these two approximations we can construct two sets Uclose and U∞ and the loss function reduces to L̃ = ∑ ( u , v ) ∈Uclose || ( 1 + τ KLu , v ) −1 − d−βu , v ||22 + ∑ ( u , v ) ∈U∞ || ( 1 + τ KLu , v ) −1 − d−βu , v ||22 . Lemma 1 . Let G = ( V , E ) be a graph . Then our embedding has 2k|V | + 1 degrees of freedom and one hyperparameter ( β ) . Generating the training samples for the full method , is equivalent to the all pair shortest path problem , which can be solved within O ( |V |2 log |V |+ |V ||E| ) for sparse graphs and evaluating the loss function has time complexity O ( |V |2 ) . The scalable variant has , for B |V | , a time complexity of O ( ( B + 1 ) |V |+ |E| ) and the loss function has O ( B|V | ) terms . See Appendix A.2 for proof using Johnson ’ s algorithm [ 24 ] and the pseudo-code of our methods .

Summary: This paper proposes an unsupervised method for learning node embeddings of directed graphs into statistical manifolds. Each node in the graph is mapped to a distribution in the space of k-variate power distributions, endowed with the KL divergence as asymetric similarity. The authors propose an optimization method based on a regularized KL divergence objective. They also propose an approximation of this objective based on finite neighborhoods, with a separate treatment of infinite distances based on a topological sorting. They also introduce a natural gradient correction to the gradient descent algorithm in this setting. They validate the fitness of their approach by showing that asymmetric distances in the graph translate into correlated asymetric distances between the node embeddings for various datasets.