MagNet: A Neural Network for Directed Graphs

1 Introduction . Endowing a collection of objects with a graph structure allows one to encode pairwise relationships among its elements . These relations often possess a natural notion of direction . For example , the WebKB dataset [ 36 ] contains a list of university websites with associated hyperlinks . In this context , one website might link to a second without a reciprocal link to the first . Such datasets are naturally modeled by directed graphs . In this paper , we introduce MagNet , a graph convolutional neural network for directed graphs based on the magnetic Laplacian . Most graph neural networks fall into one of two families , spectral networks or spatial networks . Spatial methods define graph convolution as a localized averaging operation with iteratively learned weights . Spectral networks , on the other hand , define convolution on graphs via the eigendecompositon of the ( normalized ) graph Laplacian . The eigenvectors of the graph Laplacian assume the role of Fourier modes , and convolution is defined as entrywise multiplication in the Fourier basis . For a comprehensive review of both spatial and spectral networks , we refer the reader to [ 46 ] and [ 44 ] . Many spatial graph CNNs have natural extensions to directed graphs . However , these extensions typically only consider the outgoing neighbors of each vertex and neglect the incoming neighbors . Therefore , they run the risk of discarding potentially important information . Consider , for example , a 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . directed social network such as Twitter , where the nodes are Twitter accounts and a directed edge ( u , v ) ∈ E means that account u mentions account v ( using the @ functionality ) . To infer something about account v , there is important information to be gathered both from other accounts that v mentions , and accounts that mention v. Therefore , it is common for spatial methods to preprocess the data by symmetrizing the adjacency matrix , effectively creating an undirected graph . For example , while [ 43 ] explicitly notes that their network is well-defined on directed graphs , their experiments treat all citation networks as undirected for improved performance . Extending spectral methods to directed graphs is not straightforward since the adjacency matrix is asymmetric and , thus , there is no obvious way to define a symmetric , real-valued Laplacian with a full set of real eigenvalues that uniquely encodes any directed graph . We overcome this challenge by constructing a network based on the magnetic Laplacian L ( q ) defined in Section 2 . Unlike the directed graph Laplacians used in works such as [ 29 , 33 , 41 , 42 ] , the magnetic Laplacian is not a real-valued symmetric matrix . Instead , it is a complex-valued Hermitian matrix that encodes the fundamentally asymmetric nature of a directed graph via the complex phase of its entries . Since L ( q ) is Hermitian , the spectral theorem implies it has an orthonormal basis of complex eigenvectors corresponding to real eigenvalues . Moreover , Theorem 1 , stated in Section 5 of the supplement , shows that L ( q ) is positive semidefinite , similar to the traditional Laplacian . Setting q = 0 is equivalent to symmetrizing the adjacency matrix and no importance is given to directional information . When q = .25 , on the other hand , we have that L ( .25 ) ( u , v ) = −L ( .25 ) ( v , u ) whenever there is an edge from u to v but not from v to u . Different values of q highlight different graph motifs [ 16 , 17 , 20 , 32 ] , and therefore the optimal choice of q varies . Learning the appropriate value of q from data allows MagNet to adaptively incorporate directed information . We also note that L ( q ) has been applied to graph signal processing [ 19 ] , community detection [ 17 ] , and clustering [ 10 , 16 , 15 ] . In Section 3 , we show how the networks constructed in [ 6 , 13 , 24 ] can be adapted to directed graphs by incorporating complex Hermitian matrices , such as the magnetic Laplacian . When q = 0 , we effectively recover the networks constructed in those previous works . Therefore , our work generalizes these networks in a way that is suitable for directed graphs . Our method is very general and is not tied to any particular choice of network architecture . Indeed , the main ideas of this work could be adapted to nearly any spectral graph neural network , and some spatial ones . In Section 4 , we summarize related work on directed graph neural networks as well as other papers studying the magnetic Laplacian and its applications in data science . In Section 5 , we apply our network to node classification and link prediction tasks . We compare against several spectral and spatial methods as well as networks designed for directed graphs . We find that MagNet obtains the best or second-best performance on five out of six node-classification tasks and has the best performance on seven out of eight link-prediction tasks tested on real-world data , in addition to providing excellent node-classification performance on difficult synthetic data . We also provide a supplementary document with full implementation details , theoretical results concerning the magnetic Laplacian , extended examples , and further numerical details . 2 The magnetic Laplacian . Spectral graph theory has been remarkably successful in relating geometric characteristics of undirected graphs to properties of eigenvectors and eigenvalues of graph Laplacians and related matrices . For example , the tasks of optimal graph partitioning , sparsification , clustering , and embedding may be approximated by eigenvectors corresponding to small eigenvalues of various Laplacians ( see , e.g. , [ 9 , 38 , 2 , 40 , 11 ] ) . Similarly , the graph signal processing research community leverages the full set of eigenvectors to extend the Fourier transform to these structures [ 34 ] . Furthermore , numerous papers [ 6 , 13 , 24 ] have shown that this eigendecomposition can be used to define neural networks on graphs . In this section , we provide the background needed to extend these constructions to directed graphs via complex Hermitian matrices such as the magnetic Laplacian . We let G = ( V , E ) be a directed graph where V is a set of N vertices and E ⊆ V × V is a set of directed edges . If ( u , v ) ∈ E , then we say there is an edge from u to v. For the sake of simplicity , we will focus on the case where the graph is unweighted and has no self-loops , i.e. , ( v , v ) /∈ E , but our methods have natural extensions to graphs with self-loops and/or weighted edges . If both ( u , v ) ∈ E and ( v , u ) ∈ E , then one may consider this pair of directed edges as a single undirected edge . A directed graph can be described by an adjacency matrix ( A ( u , v ) ) u , v∈V where A ( u , v ) = 1 if ( u , v ) ∈ E and A ( u , v ) = 0 otherwise . Unless G is undirected , A is not symmetric , and , indeed , this is the key technical challenge in extending spectral graph neural networks to directed graphs . In the undirected case , where the adjacency matrix A is symmetric , the ( unnormalized ) graph Laplacian can be defined by L = D −A , where D is a diagonal degree matrix . It is well-known that L is a symmetric , positive-semidefinite matrix and therefore has an orthonormal basis of eigenvectors associated with non-negative eigenvalues . However , when A is asymmetric , direct attempts to define the Laplacian this way typically yield complex eigenvalues . This impedes the straightforward extension of classical methods of spectral graph theory and graph signal processing to directed graphs . A key point of this paper is to represent the directed graph through a complex Hermitian matrix L such that : ( 1 ) the magnitude of L ( u , v ) indicates the presence of an edge , but not its direction ; and ( 2 ) the phase of L ( u , v ) indicates the direction of the edge , or if the edge is undirected . Such matrices have been explored in the directed graph literature ( see Section 4 ) , but not in the context of graph neural networks . They have several advantages over their real-valued matrix counterparts . In particular , a single symmetric real-valued matrix will not uniquely represent a directed graph . Instead , one must use several matrices , as in [ 42 ] , but this increases the complexity of the resulting network . Alternatively , one can work with an asymmetric , real-valued matrix , such as the adjacency matrix or the random walk matrix . However , the spatial graph filters that result from such matrices are typically limited by the fact that they can only aggregate information from the vertices that can be reached in one hop from a central vertex , but ignore the equally important subset of vertices that can reach the central vertex in one hop . Complex Hermitian matrices , however , lead to filters that aggregate information from both sets of vertices . Finally , one could use a real-valued skew-symmetric matrix but such matrices do not generalize well to graphs with both directed and undirected edges . The optimal choice of complex Hermitian matrix is an open question . Here , we utilize a parameterized family of magnetic Laplacians , which have proven to be useful in other data-driven contexts [ 17 , 10 , 16 , 15 ] . We first define the symmetrized adjacency matrix and corresponding degree matrix by , As ( u , v ) : = 1 2 ( A ( u , v ) + A ( v , u ) ) , 1 ≤ u , v ≤ N , Ds ( u , u ) : = ∑ v∈V As ( u , v ) , 1 ≤ u ≤ N , with Ds ( u , v ) = 0 for u 6= v. We capture directional information via a phase matrix,1 Θ ( q ) , Θ ( q ) ( u , v ) : = 2πq ( A ( u , v ) −A ( v , u ) ) , q ≥ 0 , where exp ( iΘ ( q ) ) is defined component-wise by exp ( iΘ ( q ) ) ( u , v ) : = exp ( iΘ ( q ) ( u , v ) ) . Letting denote component-wise multiplication , we define the complex Hermitian adjacency matrix H ( q ) by H ( q ) : = As exp ( iΘ ( q ) ) . Since Θ ( q ) is skew-symmetric , H ( q ) is Hermitian . When q = 0 , we have Θ ( 0 ) = 0 and so H ( 0 ) = As . This effectively corresponds to treating the graph as undirected . For q 6= 0 , the phase of H ( q ) ( u , v ) encodes edge direction and the value H ( q ) ( u , v ) separates four possible cases : no edge , edge from u to v , edge from v to u , and undirected edge . If there is no edge , we will have Hq ( u , v ) = 0 . In the case of a directed edge , the Hermitian adjacency will be complex valued , and changing the direction of an edge will correspond to complex conjugation . For example , in the case where q = .25 , if there is an edge from u to v but not from v to u we have H ( .25 ) ( u , v ) = i 2 = −H ( .25 ) ( v , u ) . Thus , in this setting , an edge from u to v is treated as the opposite of an edge from v to u . On the other hand , if ( u , v ) , ( v , u ) ∈ E ( which can be interpreted as a single undirected edge ) , then H ( q ) ( u , v ) = H ( q ) ( v , u ) = 1 , and we see the phase , Θ ( q ) ( u , v ) = 0 , encodes the lack of direction in the edge . For the rest of this paper , we will assume that q lies in between these two extreme values , i.e. , 0 ≤ q ≤ .25 . We define the normalized and unnormalized magnetic Laplacians by L ( q ) U : = Ds−H ( q ) = Ds−As exp ( iΘ ( q ) ) , L ( q ) N : = I− ( D−1/2s AsD −1/2 s ) exp ( iΘ ( q ) ) . ( 1 ) 1Our definition of Θ ( q ) coincides with that used in [ 19 ] . However , another definition ( differing by a minus sign ) also appears in the literature . These resulting magnetic Laplacians have the same eigenvalues and the corresponding eigenvectors are complex conjugates of one another . Therefore , this difference does not affect the performance of our network since our final layer separates the real and imaginary parts before multiplying by a trainable weight matrix ( see Section 3 for details on the network structure ) . Note that when G is undirected , L ( q ) U and L ( q ) N reduce to the standard undirected Laplacians . L ( q ) U and L ( q ) N are Hermitian . Theorem 1 ( Section 5 of the supplement ) shows they are positivesemidefinite and thus are diagonalized by an orthonormal basis of complex eigenvectors u1 , . . . , uN associated to real , nonnegative eigenvalues λ1 , . . . , λN . Similar to the traditional normalized Laplacian , Theorem 2 ( Section 5 of the supplement ) shows the eigenvalues of LqN lie in [ 0 , 2 ] , and we may factor L ( q ) N = UΛU † , where U is the N ×N matrix whose k-th column is uk , Λ is the diagonal matrix with Λ ( k , k ) = λk , and U† is the conjugate transpose of U ( a similar formula holds for L ( q ) U ) . Furthermore , recall L = BB > , where B is the signed incidence matrix . Similarly , Theorem 3 ( Section 5 of the supplement ) shows that L ( q ) U = B ( q ) ( B ( q ) ) † , where B ( q ) is a modified incidence matrix . The magnetic Laplacian encodes geometric information in its eigenvectors and eigenvalues . In the directed star graph ( Section 6 of the supplement ) , for example , directional information is contained in the eigenvectors only , whereas the eigenvalues are invariant to the direction of the edges . On the other hand , for the directed cycle graph the magnetic Laplacian encodes the directed nature of the graph solely in its spectrum . In general , both the eigenvectors and eigenvalues may contain important information , which we leverage in MagNet .

Summary: This paper considers the design of graph neural networks for directed graphs. In particular, it uses the magnetic Laplacian, a complex-valued Hermitian matrix that encodes edge directions via a phase parameter, as a matrix from which convolutional filters are formed. By defining this matrix, the authors demonstrate how it can be used in place of more traditional Laplacians in a few existing graph neural network architectures. This approach is then empirically verified on synthetic and real datasets, where the utility of the the magnetic Laplacian for encoding edge directionality is demonstrated.