Low-dimensional statistical manifold embedding of directed graphs

We propose a novel node embedding of directed graphs to statistical manifolds , which is based on a global minimization of pairwise relative entropy and graph geodesics in a non-linear way . Each node is encoded with a probability density function over a measurable space . Furthermore , we analyze the connection between the geometrical properties of such embedding and their efficient learning procedure . Extensive experiments show that our proposed embedding is better in preserving the global geodesic information of graphs , as well as outperforming existing embedding models on directed graphs in a variety of evaluation metrics , in an unsupervised setting . 1 INTRODUCTION . In this publication , we study the directed graph embedding problem in an unsupervised learning setting . A graph embedding problem is usually defined as a problem of finding a vector representation X ∈ RK for every node of a graph G = ( V , E ) through a mapping φ : V → X . On every graph G = ( V , E ) , defined with set of nodes V and set of edgesE , the distance dG : V ×V → R+ between two vertices is defined as the number of edges connecting them in the shortest path , also called a graph geodesic . In case that X is equipped with a distance metric function dX : X ×X → R+ , we can quantify the embedding distortion by measuring the ratio dX / dG between pairs of corresponding embedded points and nodes . Alternatively , one can measure the quality of the mapping by using a similarity function between points . In contrast to distance , the similarity is usually a bounded value [ 0 , 1 ] and is in some ad-hoc way connected to a notion of distance e.g . inverse of the distance . Usually , we are interested in a low-dimensional ( K n ) embedding of graphs with n nodes as it is always possible to find an embedding [ 32 ] with L∞ norm with no distortion in an n-dimensional Euclidean space . Bourgain theorem ( 1985 ) [ 6 ] proved that it is possible to construct an O ( log ( n ) ) -dimensional Euclidean embedding of an undirected graph with n nodes with finite distortion . For the last couple of decades , different communities such as machine learning [ 16 ; 33 ; 10 ; 40 ; 14 ] , physics [ 29 ; 38 ; 35 ] , and computer science [ 42 ; 43 ] independently from mathematics community [ 32 ; 25 ; 12 ; 6 ] developed novel graph representation techniques . In a nutshell , they can be characterized by having one of the following properties ( i-iv ) . ( i ) The Type of loss function that quantifies the distortion is optimizing local neighbourhood [ 44 ; 21 ] or global graph structure properties [ 53 ; 40 ; 14 ] . ( ii ) The target property to be preserved is either geodesic ( shortest paths ) [ 42 ; 43 ; 12 ] or diffusion distance ( heat or random walk process ) [ 46 ; 40 ; 14 ] or another similarity property [ 21 ; 55 ; 39 ] . ( iii ) The mapping φ is a linear [ 23 ; 39 ] or non-linear dimensionality reduction technique such as SNE [ 21 ] , t-SNE [ 55 ] , ISOMAP [ 53 ] , Laplacian eigenmaps [ 3 ] or deep learning work DeepWalk [ 40 ] , node2vec [ 14 ] , HOPE [ 39 ] , GraphSAGE [ 17 ] , NetMF [ 33 ] and others [ 16 ] . ( iv ) The geometry of 20 0 20 20 0 20 0 50 100 150 200 epochs 25 75 0 50 100 150 200 epochs 0Figure 1 : Visualization of synthetic example network together with our embedding . The graph is embedded into the 2-variate normal distributions , which are represented by a σ ellipse ( boundary of 1 standard deviation around mean µ ) . The σ ellipses of the green nodes are contained in one of the greys nodes , which represents that the distance ( measured with divergence in embedded space ) between green and grey is small , but in the opposite direction very large . X has zero curvature ( Euclidean ) [ 32 ; 12 ] , positive curvature [ 57 ] ( spherical ) or negative curvature [ 29 ; 35 ; 10 ] ( hyperbolic ) or mixed curvatures [ 15 ] . However , for directed graphs the asymmetry of graph shortest path distances dG ( vi , vj ) 6= dG ( vj , vi ) is violating the symmetry assumption ( not a metric function ) . This is the reason why only recently this problem was tackled by constructing two independent representations for each node ( source and target representations ) [ 39 ; 59 ; 27 ] . In this paper , we propose a single-node embedding for directed graphs to the space of probability distributions , described with the geometry of statistical manifold theory . Up to this point , different probability distributions were used for different embedding purposes such as : ( i ) applications to word embedding [ 45 ; 56 ] , knowledge graphs [ 20 ] , attributed graphs [ 4 ] or ( ii ) generalized point embeddings with elliptical distributions [ 36 ] , but not for low-dimensional embedding of directed graphs , characterized by the theory of statistical manifolds . The main contributions of this paper are : ( i ) We propose a node embedding for directed graphs to the elements of the statistical manifolds [ 51 ] , to jointly capture asymmetric and infinite distances in a low-dimensional space . We determine the connection between geometry and gradient-based optimization . ( ii ) Furthermore , we develop a sampling-based scalable learning algorithm for largescale networks . ( iii ) Extensive experiments demonstrate that the proposed embedding outperforms several directed network embedding baselines , i.e . random walks based [ 59 ] and matrix factorization based [ 39 ] methods , as well as the deep learning undirected representative [ 40 ] and deep Gaussian embeddings for attributed graphs [ 4 ] . 2 STATISTICAL MANIFOLD DIVERGENCE EMBEDDING FOR DIRECTED GRAPHS . In this section , we analyze the limits of metric embedding . Then , we continue by providing an intuition of our embedding with a synthetic example before presenting the learning procedure of the embedding method . Bourgain theorem ( 1985 ) [ 6 ] states that it is possible to construct an O ( log ( n ) ) dimensional embedding ( φ : v 7→ xv ) of an undirected graph with n nodes to Euclidean space with O ( log ( n ) ) distortion by using random subset projection . But , on directed graph G = ( V , E ) , the directed shortest paths are not necessarily symmetric dv , u 6= du , v . This does not imply that it is not possible to provide low dimensional metric embedding [ 6 ; 32 ; 25 ; 12 ] with a variant of Bourgain theorem . In the appendix A.1 , we show that the main problem is not asymmetry , but the existence of infinite directed distances , which is quite common in real directed networks , see Table 1 . 2.1 LEARNING STATISTICAL MANIFOLD EMBEDDING . Instead of a conventional network embedding in a metric space , we propose to use the KullbackLeibler divergence defined on probability distributions to capture asymmetric finite and infinite distances in directed graphs . Each node is mapped to a corresponding distribution described by some parameters , which we learn during training . In this paper , we choose the class of exponential power distributions due to analytical and computational merits . With minor modifications , our method can also be applied to other distribution classes . Intuition . In Fig . 1 , we use a toy example to qualitatively illustrate that our embedding can represent infinite distances without requiring a high dimensional embedding space . More precisely , the network consists of five groups ( color-coded ) , with five nodes each . The two top blocks have a connection only to the center group and , similarly , the two bottom groups have only a link from the central group to them . Our model learns this connectivity pattern by embedding members of these groups in a similar fashion . In this example , each node u is embedded as a 2-variate normal distribution with mean µu = ( µ1u , µ 2 u ) and variances Σu = diag ( σ 1 u , σ 2 u ) , which can be visualized by the σ-ellipse , the curve which represent one-σ distance from their respective means . The nodes of graphs are embedded by minimizing the difference between the pair-wise graph distances and corresponding Kullback-Leibler divergences between embeddings . For two distribution p ( x ) , q ( x ) , if p ( x ) q ( x ) on an open subset U , the Kullback-Leibler divergence KL ( p ( x ) , q ( x ) ) is then relatively high . In other words , if in Figure 1 the σ-ellipse of node u is contained in or very similar to the σ-ellipse of node v , then the embedding represents d ( u , v ) < ∞ . Using this , we see that the embedding retrieved by our optimization and visualized in Fig . 1 includes most of the observed infinite distances . Statistical manifold embedding . Our embedding space X is the space of k-variate exponential power distributions [ 37 ] ( also called generalized error distributions ) , described by the following probability density function for λ > 0 ψλ ( x|Σ , µ ) = λΓ ( k2 ) 21+ k λπ k 2 det ( Σ ) 1 2 Γ ( kλ ) exp ( − [ ( x− µ ) TΣ−1 ( x− µ ) ] λ2 2 ) ( 1 ) with mean vector µ ∈ Rk and covariance matrix Σ ∈ Rk×k . Γ ( . ) denotes the Gamma function and xT denotes the transposed vector of x . Note that λ = 2 results in the multivariate Gaussian distribution and λ = 1 yields the multivariate Laplacian distribution . As we are interested in non-degenerate distributions , we enforce positive definite co-variance matrices and further restrict ourselves to diagonal matrices , i.e . Σu = diag ( σ1u , . . . , σ k u ) with σ i u ∈ R+ . With the latter , we reduce the degrees of freedom for each node to 2k and simplify our optimization by replacing a positive definite constraint on Σu with the constraints σiu > 0 . A common asymmetric function operating on continuous random variables is the Kullback-Leibler divergence . The asymmetric distance between nodes u and v , denoted by KLu , v , is KLu , v = KL ( p λ u , p λ v ) = ∫ pλu log pλu pλv dx with pλu = ψλ ( x|Σu , µu ) and pλv = ψλ ( x|Σv , µv ) . We approximate it with importance sampling Monte Carlo estimation . In particular , KLu , v can be expressed as the expectation ( for λ∗ = 2 ≤ λ ) KLu , v = Ex∼pλ∗u [ pλu pλ∗u log pλu pλv ] , where pλ∗u = ψλ∗ ( x|Σu , µu ) is easy to sample and a proposal function for pλ ( x|Σu , µu ) . If a closed expression for KLu , v is known , its evaluation replaces the importance sampling Monte Carlo estimation , like in the special case of λ = 2 . See Appendix A.9 for more details . Now , in order to learn these statistical manifold embeddings , we can define the loss function over asymmetric distances D = ( du , v ) u , v∈V of a directed graph and { ( µu , Σu ) } u as : L ( { ( µu , Σu ) } u ) = ∑ u6=v || ( 1 + τ KLu , v ) −1 − d−βu , v ||22 , ( 2 ) where τ ∈ R+ is a free ( trainable ) parameter and β ∈ R+ a fixed value . This loss function given in the Eq . ( 2 ) is minimized during the learning process so that the KLu , v based on learned node distribution representations captures the distances du , v in the directed graph . Empirically , we transform the given distances into finite numbers with d−βu , v ∈ [ 0 , 1 ] , for all u 6= v and a β ∈ R+ , which can be used to increase the differentiation between large du , v values and between finite and infinite distances . We modify the unbounded Kullback-Leibler divergence in a similar fashion ( 1 + τ KLu , v ) −1 ∈ [ 0 , 1 ] , where τ ∈ R+ is a free ( trainable ) parameter and the additional 1 ensures a value in the same interval [ 0 , 1 ] . We start the optimization from random initial parameters { µu , Σu } u∈V , and iteratively minimize the loss function with stochastic gradient descent optimizers , such as Adam [ 28 ] . For small k , which we will consider in the next section , more enhanced initialization strategies , such as using graph plotting algorithms to determine the means and exploiting memberships of strongly connected components for the covariance initialization , probably reduce the number of training epochs . Scalable learning procedure . The full objective function Eq . ( 2 ) consists of |V | ( |V | − 1 ) terms and only graphs with up to the magnitude of around 104 nodes can be applied . To extend our method beyond this limit , we propose an approximated solution , where the size of the training data scales linearly in the number of nodes and thus can be applied to large graphs . This approximation solution is based on a decomposition of the loss function Eq . ( 2 ) into a neighborhood term and a singularity term as : L ( { ( µu , Σu ) } u ) = ∑ u 6=v , du , v < ∞ || ( 1 + τ KLu , v ) −1 − d−βu , v ||22︸ ︷︷ ︸ neighborhood term + ∑ u6=v , du , v=∞ || ( 1 + τ KLu , v ) −1 − d−βu , v ||22︸ ︷︷ ︸ singularity term . We efficiently sample from each of these sums for each node a small number of B samples and optimize our model based on the sampled information about closeness and infinite distances . One straightforward approach to approximate the neighborhood term is to use all direct neighbors . Yet , the number of samples available via the directed neighbors is limited by |E| , which usually corresponds to a small B in sparse real-world examples . Therefore , we apply for each node a breadth-first-search into both directions until we retrieve B new samples . In this way , we obtain smaller du , v first , and the approximation tends to the original term in the limit B → |V | . For the singularity term , we use the topological sorting [ 26 ] of the strongly connected components . In a topological sorting of a directed acyclic graph , all edges are from lower indices to higher indices ( with respect to the topological sorting ) . Two important restrictions are noteworthy . First , the reverse statement does not hold , i.e . not all infinite distances are given via a single topological sorting [ 26 ] . As a consequence , we won ’ t sample uniformly from all infinities , but from a subset of the infinities given by the topological sorting . Second , the construction of topological sorting is only possible for acyclic graphs , and most directed graphs have cycles . Since we are only interested in sampling singularities and the graph defined by the strongly connected components of a directed graph is acyclic , we use Tarjan ’ s algorithm [ 52 ] to retrieve the strongly connected components and topological sorting of them . After this preprocessing , we can efficiently generate samples of infinite distance for the nodes . With these two approximations we can construct two sets Uclose and U∞ and the loss function reduces to L̃ = ∑ ( u , v ) ∈Uclose || ( 1 + τ KLu , v ) −1 − d−βu , v ||22 + ∑ ( u , v ) ∈U∞ || ( 1 + τ KLu , v ) −1 − d−βu , v ||22 . Lemma 1 . Let G = ( V , E ) be a graph . Then our embedding has 2k|V | + 1 degrees of freedom and one hyperparameter ( β ) . Generating the training samples for the full method , is equivalent to the all pair shortest path problem , which can be solved within O ( |V |2 log |V |+ |V ||E| ) for sparse graphs and evaluating the loss function has time complexity O ( |V |2 ) . The scalable variant has , for B |V | , a time complexity of O ( ( B + 1 ) |V |+ |E| ) and the loss function has O ( B|V | ) terms . See Appendix A.2 for proof using Johnson ’ s algorithm [ 24 ] and the pseudo-code of our methods .

Summary: This paper proposed another graph embedding method. It focuses on directed graphs, and it embedded the graph nodes into exponential power distributions, which include the Gaussian distribution as a special case. The method is implemented by optimizing with respect to the free distributions on a statistical manifold so as to achieve the minimum distortion between the input/output distances.  The method is tested on several directed graph datasets and showed superior performance based on several metrics.