The Forward-Backward Embedding of Directed Graphs

1 Introduction . Learning from data structured as a graph most often requires to embed the nodes of the graph in some Euclidian space ( Yan et al . ( 2007 ) ; Grover and Leskovec ( 2016 ) ; Bronstein et al . ( 2017 ) ) . The ability to learn then critically depends on this embedding , that must incorporate the “ geometry ” of the graph in some sense . A classical embedding of undirected graphs is based on the spectral decomposition of the Laplacian ( Belkin and Niyogi ( 2003 ) ) ; after proper normalization , the distances between vectors in the embedding space are proportional to the mean commute times of the corresponding nodes by a random walk in the graph , making this embedding meaningful and easy to interpret ( Qiu and Hancock ( 2007 ) ; Fouss et al . ( 2007 ) ) . This result is not applicable to directed graphs . In this paper , we show that , after proper normalization , the embedding derived from the singular value decomposition ( SVD ) of the normalized adjacency matrix of a directed graph , as considered in ( Dhillon ( 2001 ) ; Rohe et al . ( 2012 ) ) , can also be interpreted in terms of a random walk in the graph . This random walk is particular in the sense that edges are followed alternately in forward and backward directions . We call it the forward-backward random walk , and the corresponding embedding as the forward-backward embedding . As for undirected graphs , the distances between vectors in the embedding space are proportional to the mean commute times between the corresponding nodes by the random walk . We show in fact that the forward-backward embedding of a graph is equivalent to the spectral embedding of some co-citation graph . The interest of our approach is that it does not require to build this co-citation graph , which is typically much denser than the original graph . The rest of the paper is organized as follows . We first present the related work . We then introduce the Laplacian matrix of a directed graph and the notion of co-citation graph . The embedding , and its interpretation in terms of the forward-backward random walk , are presented in Sections 5 and 6 , respectively , under the assumption that the co-citation graph is connected ; this assumption is relaxed in Section 7 . The link between our embedding and the spectral embedding is explained in Section 8 . The practically interesting case of bipartite graphs , that may be viewed as particular instances of directed graphs , is considered in Section 9 . We conclude the paper by experiments on real datasets in Section 10 . We denote by 1 the vectors of ones of appropriate dimension . For any square matrix M , M+ is the Moore-Penrose inverse of M and d ( M ) is the diagonal matrix which has the same diagonal as M . 2 Related work . While the spectral embedding of undirected graphs , and its interpretation in terms of random walks , is very well understood ( see Lovász ( 1993 ) ; Snell and Doyle ( 2000 ) ; Qiu and Hancock ( 2007 ) ; Fouss et al . ( 2007 ) ; Luxburg ( 2007 ) ) , the results do not easily extend to directed graphs . The main reason is that the Laplacian matrix is no longer symmetric and thus has complex eigenvalues in general . In ( Li and Zhang ( 2010 ) ; Boley et al . ( 2011 ) ) , it is shown that the mean commute times of the random walk can be expressed in terms of the pseudo-inverse of some proper Laplacian matrix , extending the results known for undirected graphs ( Qiu and Hancock ( 2007 ) ) . The considered Laplacian relies on the stationary distribution of the random walk , which is not explicit in general , and on the assumption that the graph is strongly connected . Moreover , no embedding of the graph ( e.g. , using the SVD of the Laplacian , as suggested in Li and Zhang ( 2010 ) ) is known to be directly related to the mean commute times of the random walk , as for undirected graphs . The embedding of directed graphs proposed in ( Dhillon ( 2001 ) ; Rohe et al . ( 2012 ) ) relies on the SVD of the normalized adjacency matrix . Our main contribution is a proper normalization of the singular vectors so that the distances in the embedding space can be interpreted in terms of mean commute times of the forward-backward random walk . In particular , our embedding is not an extension of the usual spectral embedding of undirected graphs , a point that is addressed specifically in Section 8 . The idea of a random walk alternating between forward and backward steps is in the spirit of HITS ( Kleinberg et al . ( 1999 ) ) , an algorithm proposed to rank Web pages in terms of their relative importances as so-called hubs and authorities . Various other embedding techniques rely on random walks in the graph ( see Perrault-Joncas and Meila ( 2011 ) ; Cai et al . ( 2018 ) ; Goyal and Ferrara ( 2018 ) ; Qiu et al . ( 2018 ) and references therein ) . These include DeepWalk ( Perozzi et al . ( 2014 ) ) and node2vec ( Grover and Leskovec ( 2016 ) ) , inspired by Natural Language Processing techniques , where the representation of the graph is learned from ( simulated ) random walks in the graph . Unlike the embedding proposed in the present paper , there is no direct interpretation of the distances in the embedding space in terms of random walk in the graph . 3 Laplacian matrix . Let G be some directed graph and A its adjacency matrix . For convenience , we present the results for unweighted graphs ( binary matrices A ) , but the results readily apply to weighted graphs ( non-negative matrices A ) . We denote by n the number of nodes and by m = 1TA1 the number of edges ( the total weight of edges for a weighted graph ) . Unless otherwise specified , we assume that there are neither sources nor sinks in the graph , i.e. , each node has a positive indegree and a positive outdegree . The extension to general graphs is considered in Section 7 . We refer to the ( normalized ) Laplacian as : L = I −D− 1 2 outAD − 12 in , ( 1 ) where Dout = diag ( A1 ) and Din = diag ( AT 1 ) are the diagonal matrices of outdegrees and indegrees , respectively . We refer to the matrix D − 12 outAD − 12 in as the normalized adjacency matrix . Observe that the matrix LT is the Laplacian of the reverse graph G− ( same graph as G but with reverse edges ) , with adjacency matrix AT . In particular , the Laplacian L is symmetric if and only if A is symmetric , in which case we recover the usual normalized Laplacian of undirected graphs , namely L = I −D− 12AD− 12 with D = Dout = Din . 4 Co-citation graph . The co-citation graph associated with G is the graph with adjacency matrix AAT . This is a weighted , undirected graph where the weight between nodes i and j is the number of common successors of i and j . We refer to the normalized co-citation graph associated with G as the graph with adjacency matrix Ā = AD−1in AT . This undirected graph has the same edges as the co-citation graph but the weight associated to any common successor of i and j is now normalized by its indegree . For instance , two articles of Wikipedia , say i , j , pointing to the same other article , say k , will be all the more similar ( higher weight ) in the corresponding normalized co-citation graph as article k is referenced by fewer other articles ( lower in-degree ) in the original graph . The node weights in the normalized co-citation graph ( total weights of incident edges ) are equal to the out-degrees in the graph G : Ā1 = AD−1in AT 1 = A1 . ( 2 ) The normalized Laplacian of the normalized co-citation graph is : L̄ = I − D̄− 12 ĀD̄− 12 , where D̄ = diag ( Ā1 ) is the diagonal matrix of node weights in the normalized co-citation graph . Observe that D̄ = Dout in view of ( 2 ) . Proposition 1 We have I − L̄ = ( I − L ) ( I − L ) T . Proof . This follows from I − L̄ = D̄− 12 ĀD̄− 12 = D− 1 2 outAD −1 in A TD − 12 out = ( I − L ) ( I − L ) T . 5 Forward-backward embedding . Consider a singular value decomposition ( SVD ) of the normalized adjacency matrix1 , D − 12 outAD − 12 in = UΣV T , ( 3 ) where Σ = diag ( σ1 , . . . , σn ) with σ1 ≥ . . . ≥ σn ≥ 0 and UTU = V TV = I . We get L = I − UΣV T , and by Proposition 1 , L̄ = U ( I − Σ2 ) UT . ( 4 ) Thus the normalized Laplacian of the normalized co-citation graph has eigenvalues 1− σ21 ≤ . . . ≤ 1 − σ2n with corresponding unitary matrix of eigenvectors U . These eigenvalues are non-negative , the multiplicity of the eigenvalue 0 being the number of connected components of this graph ( Luxburg ( 2007 ) ) . We deduce that σ1 = . . . = σK = 1 > σK+1 , where K is the number of connected components of the co-citation graph . Assume that the co-citation graph is connected . The general case is considered in Section 7 . Then σ1 = 1 > σ2 . Let Γ = ( I − Σ2 ) 1 2 and define : X = Γ+UTD− 1 2 out . ( 5 ) The columns x1 , . . . , xn of X define a representation of the graph in Rn . Specifically , each node i is represented by the vector xi ∈ Rn . Observe that the first coordinate of each vector xi is null so that at most n− 1 coordinates are informative . The Euclidian distances between these vectors are entirely defined by the Gram matrix of X , which is related to the pseudo-inverse of the normalized Laplacian of the normalized co-citation graph , L̄+ = U ( I − Σ2 ) +UT . 1A naive approach to the SVD of a matrix M is based on the spectral decomposition of the matrix MMT ; in our case , this would be equivalent to build the co-citation graph , which is typically much denser than the original graph . This is why we use a proper implementation of the SVD that works directly on the matrix M , see Halko et al . ( 2011 ) . In view of ( 5 ) , XTX = D− 1 2 out L̄ +D − 12 out . We show in the following section that the square distance between xi and xj is proportional to the mean commute time between nodes i and j of the forward-backward random walk in the graph G. In practice , the graph is embedded in some vector space of dimension d , chosen much lower than n. In this case , only the d leading singular vectors are considered , i.e. , those associated with the singular values σ1 , . . . , σd ( or the d+ 1 leading singular vectors is the first vector , which is not informative , is skipped ) .

Summary: The paper studies embeddings of directed graphs based on SVD. It proposes an interpretation of the embedding obtained from the normalized adjacency matrix in terms of the forward-backward random walk on the graph. In such random walk odd steps are taken using the edges of the graph while even steps are taken using reverse edges. Such an interpretation seems to be a trivial extension of the work for undirected graphs.