Tensor Graph Convolutional Networks for Prediction on Dynamic Graphs

1 INTRODUCTION . Graphs are popular data structures used to effectively represent interactions and structural relationships between entities in structured data domains . Inspired by the success of deep neural networks for learning representations in the image and language domains , recently , application of neural networks for graph representation learning has attracted much interest . A number of graph neural network ( GNN ) architectures have been explored in the contemporary literature for a variety of graph related tasks and applications ( Hamilton et al. , 2017 ; Seo et al. , 2018 ; Chen et al. , 2018 ; Zhou et al. , 2018 ; Wu et al. , 2019 ) . Methods based on graph convolution filters which extend convolutional neural networks ( CNNs ) to irregular graph domains are popular ( Bruna et al. , 2013 ; Defferrard et al. , 2016 ; Kipf and Welling , 2016 ) . Most of these GNN models operate on a given , static graph . In many real-world applications , the underlining graph changes over time , and learning representations of such dynamic graphs is essential . Examples include analyzing social networks ( Berger-Wolf and Saia , 2006 ) , predicting collaboration in citation networks ( Leskovec et al. , 2005 ) , detecting fraud and crime in financial networks ( Weber et al. , 2018 ; Pareja et al. , 2019 ) , traffic control ( Zhao et al. , 2019 ) , and understanding neuronal activities in the brain ( De Vico Fallani et al. , 2014 ) . In such dynamic settings , the temporal interdependence in the graph connections and features also play a substantial role . However , efficient GNN methods that handle time varying graphs and that capture the temporal correlations are lacking . By dynamic graph , we mean a sequence of graphs ( V , A ( t ) , X ( t ) ) , t ∈ { 1 , 2 , . . . , T } , with a fixed set V of N nodes , adjacency matrices A ( t ) ∈ RN×N , and graph feature matrices X ( t ) ∈ RN×F where X ( t ) n : ∈ RF is the feature vector consisting of F features associated with node n at time t. The graphs can be weighted , and directed or undirected . They can also have additional properties like ( time varying ) node and edge classes , which would be stored in a separate structure . Suppose we only observe the first T ′ < T graphs in the sequence . The goal of our method is to use these observations to predict some property of the remaining T − T ′ graphs . In this paper , we use it for edge classification . Other potential applications are node classification and edge/link prediction . In recent years , tensor constructs have been explored to effectively process high-dimensional data , in order to better leverage the multidimensional structure of such data ( Kolda and Bader , 2009 ) . Tensor based approaches have been shown to perform well in many image and video processing ap- plications ( Hao et al. , 2013 ; Kilmer et al. , 2013 ; Martin et al. , 2013 ; Zhang et al. , 2014 ; Zhang and Aeron , 2016 ; Lu et al. , 2016 ; Newman et al. , 2018 ) . A number of tensor based neural networks have also been investigated to extract and learn multi-dimensional representations , e.g . methods based on tensor decomposition ( Phan and Cichocki , 2010 ) , tensor-trains ( Novikov et al. , 2015 ; Stoudenmire and Schwab , 2016 ) , and tensor factorized neural network ( Chien and Bao , 2017 ) . Recently , a new tensor framework called the tensor M-product framework ( Braman , 2010 ; Kilmer and Martin , 2011 ; Kernfeld et al. , 2015 ) was proposed that extends matrix based theory to high-dimensional architectures . In this paper , we propose a novel tensor variant of the popular graph convolutional network ( GCN ) architecture ( Kipf and Welling , 2016 ) , which we call TensorGCN . It captures correlation over time by leveraging the tensor M-product framework . The flexibility and matrix mimeticability of the framework , help us adapt the GCN architecture to tensor space . Figure 1 illustrates our method at a high level : First , the time varying adjacency matrices A ( t ) and feature matrices X ( t ) of the dynamic graph are aggregated into an adjacency tensor and a feature tensor , respectively . These tensors are then fed into our TensorGCN , which computes an embedding that can be used for a variety of tasks , such as link prediction , and edge and node classification . GCN architectures are motivated by graph convolution filtering , i.e. , applying filters/functions to the graph Laplacian ( in turn its eigenvalues ) ( Bruna et al. , 2013 ) , and we establish a similar connection between TensorGCN and spectral filtering of tensors . Experimental results on real datasets illustrate the performance of our method for the edge classification task on dynamic graphs . Elements of our method can also be used as a preprocessing step for other dynamic graph methods . 2 RELATED WORK . The idea of using graph convolution based on the spectral graph theory for GNNs was first introduced by Bruna et al . ( 2013 ) . Defferrard et al . ( 2016 ) then proposed Chebnet , where the spectral filter was approximated by Chebyshev polynomials in order to make it faster and localized . Kipf and Welling ( 2016 ) presented the simplified GCN , a degree-one polynomial approximation of Chebnet , in order to speed up computation further and improve the performance . There are many other works that deal with GNNs when the graph and features are fixed/static ; see the review papers by Zhou et al . ( 2018 ) and Wu et al . ( 2019 ) and references therein . These methods can not be directly applied to the dynamic setting we consider . Seo et al . ( 2018 ) devised the Graph Convolutional Recurrent Network for graphs with time varying features . However , this method assumes that the edges are fixed over time , and is not applicable in our setting . Wang et al . ( 2018 ) proposed a method called EdgeConv , which is a neural network ( NN ) approach that applies convolution operations on static graphs in a dynamic fashion . Their approach is not applicable when the graph itself is dynamic . Zhao et al . ( 2019 ) develop a temporal GCN method called T-GCN , which they apply for traffic prediction . Their method assumes the graph remains fixed over time , and only the features vary . The set of methods most relevant to our setting of learning embeddings of dynamic graphs use combinations of GNNs and recurrent architectures ( RNN ) , to capture the graph structure and handle time dynamics , respectively . The approach in Manessi et al . ( 2019 ) uses Long Short-Term Memory ( LSTM ) , a recurrent network , in order to handle time variations along with GNNs . They design architectures for semi-supervised node classification and for supervised graph classification . Pareja et al . ( 2019 ) presented a variant of GCN called EvolveGCN , where Gated Recurrent Units ( GRU ) and LSTMs are coupled with a GCN to handle dynamic graphs . This paper is currently the stateof-the-art . However , their approach is based on a heuristic RNN/GRU mechanism , which is not theoretically viable , and does not harness a tensor algebraic framework to incorporate time varying information . Newman et al . ( 2018 ) present a tensor NN which utilizes the M-product tensor framework . Their approach can be applied to image and other high-dimensional data that lie on regular grids , and differs from ours since we consider data on dynamic graphs . 3 TENSOR M-PRODUCT FRAMEWORK . Here , we cover the necessary preliminaries on tensors and the M-product framework . For a more general introduction to tensors , we refer the reader to the review paper by Kolda and Bader ( 2009 ) . In this paper , a tensor is a three-dimensional array of real numbers denoted by boldface Euler script letters , e.g . X ∈ RI×J×T . Matrices are denoted by bold uppercase letters , e.g . X ; vectors are denoted by bold lowercase letter , e.g . x ; and scalars are denoted by lowercase letters , e.g . x . An element at position ( i , j , t ) in a tensor is denotes by subscripts , e.g . Xijt , with similar notation for elements of matrices and vectors . A colon will denote all elements along that dimension ; Xi : denotes the ith row of the matrix X , and X : :k denotes the kth frontal slice of X . The vectors Xij : are called the tubes of X . The framework we consider relies on a new definition of the product of two tensors , called the M-product ( Braman , 2010 ; Kilmer and Martin , 2011 ; Kilmer et al. , 2013 ; Kernfeld et al. , 2015 ) . A distinguishing feature of this framework is that the M-product of two three-dimensional tensors is also three-dimensional , which is not the case for e.g . tensor contractions ( Bishop and Goldberg , 2012 ) . It allows one to elegantly generalize many classical numerical methods from linear algebra , and has been applied e.g . in neural networks ( Newman et al. , 2018 ) , imaging ( Kilmer et al. , 2013 ; Martin et al. , 2013 ; Semerci et al. , 2014 ) , facial recognition ( Hao et al. , 2013 ) , and tensor completion and denoising ( Zhang et al. , 2014 ; Zhang and Aeron , 2016 ; Lu et al. , 2016 ) . Although the framework was originally developed for three-dimensional tensors , which is sufficient for our purposes , it has been extended to handle tensors of dimension greater than three ( Martin et al. , 2013 ) . The following definitions 3.1–3.3 describe the M-product . Definition 3.1 ( M-transform ) . Let M ∈ RT×T be a mixing matrix . The M-transform of a tensor X ∈ RI×J×T is denoted by X×3 M ∈ RI×J×T and defined elementwise as ( X×3 M ) ijt def = T∑ k=1 MtkXijk . ( 1 ) We say that X ×3 M is in the transformed space . Note that if M is invertible , then ( X ×3 M ) ×3 M−1 = X. Consequently , X ×3 M−1 is the inverse M-transform of X . The definition in ( 1 ) may also be written in matrix form as X×3 M def = fold ( Munfold ( X ) ) , where the unfold operation takes the tubes of X and stack them as columns into a T × IJ matrix , and fold ( unfold ( X ) ) = X . Appendix A provides illustrations of how the M-transform works . Definition 3.2 ( Facewise product ) . Let X ∈ RI×J×T and Y ∈ RJ×K×T be two tensors . The facewise product , denote by X4Y ∈ RI×K×T , is defined facewise as ( X4Y ) : :t def = X : :tY : :t. Definition 3.3 ( M-product ) . Let X ∈ RI×J×T and Y ∈ RJ×K×T be two tensors , and let M ∈ RT×T be an invertible matrix . The M-product , denoted by X ? Y ∈ RI×K×T , is defined as X ? Y def = ( ( X×3 M ) 4 ( Y×3 M ) ) ×3 M−1 . In the original formulation of the M-product , M was chosen to be the Discrete Fourier Transform ( DFT ) matrix , which allows efficient computation using the Fast Fourier Transform ( FFT ) ( Braman , 2010 ; Kilmer and Martin , 2011 ; Kilmer et al. , 2013 ) . The framework was later extended for arbitrary invertible M ( e.g . discrete cosine and wavelet transforms ) ( Kernfeld et al. , 2015 ) . A benefit of the tensor M-product framework is that many standard matrix concepts can be generalized in a straightforward manner . Definitions 3.4–3.7 extend the matrix concepts of diagonality , identity , transpose and orthogonality to tensors ( Braman , 2010 ; Kilmer et al. , 2013 ) . Definition 3.4 ( f-diagonal ) . A tensor X ∈ RN×N×T is said to be f-diagonal if each frontal slice X : :t is diagonal . Definition 3.5 ( Identity tensor ) . Let Î ∈ RN×N×T be defined facewise as Î : :t = I , where I is the matrix identity . The M-product identity tensor I ∈ RN×N×T is then defined as I def= Î×3 M−1 . Definition 3.6 ( Tensor transpose ) . The transpose of a tensor X is defined as X > def= Y ×3 M−1 , where Y : :t = ( X×3 M ) > : :t for each t ∈ { 1 , . . . , T } . Definition 3.7 ( Orthogonal tensor ) . A tensor X ∈ RN×N×T is said to be orthogonal if X ? X > = X > ? X = I. Leveraging these concepts , a tensor eigendecomposition can now be defined ( Braman , 2010 ; Kilmer et al. , 2013 ) : Definition 3.8 ( Tensor eigendecomposition ) . Let X ∈ RN×N×T be a tensor and assume that each frontal slice ( X ×3 M ) : :t is symmetric . We can then eigendecompose these as ( X ×3 M ) : :t = Q̂ : :tD̂ : :tQ̂ > : :t , where Q̂ : :t ∈ RN×N is orthogonal and D̂ : :t ∈ RN×N is diagonal ( see e.g . Theorem 8.1.1 in Golub and Van Loan ( 2013 ) ) . The tensor eigendecomposition of X is then defined as X def = Q ? D ? Q > , where Q def= Q̂×3 M−1 is orthogonal , and D def = D̂×3 M−1 if f-diagonal .

Summary: this work uses tensor methods to improve graph convolution for dynamic graph, where the nodes are fixed and the edges are changing. Specifically, it uses the M-product technique to develop the operations of sequence of matrices that analog to these operations of matrices. In the M-product notations, everything seems to be as neat as matrix operations. The works also shows decent supremacy on edge classification tasks.