Space-Time Graph Neural Networks

We introduce space-time graph neural network ( ST-GNN ) , a novel GNN architecture , tailored to jointly process the underlying space-time topology of time-varying network data . The cornerstone of our proposed architecture is the composition of time and graph convolutional filters followed by pointwise nonlinear activation functions . We introduce a generic definition of convolution operators that mimic the diffusion process of signals over its underlying support . On top of this definition , we propose space-time graph convolutions that are built upon a composition of time and graph shift operators . We prove that ST-GNNs with multivariate integral Lipschitz filters are stable to small perturbations in the underlying graphs as well as small perturbations in the time domain caused by time warping . Our analysis shows that small variations in the network topology and time evolution of a system does not significantly affect the performance of ST-GNNs . Numerical experiments with decentralized control systems showcase the effectiveness and stability of the proposed ST-GNNs . 1 INTRODUCTION . Graph Neural Networks ( GNNs ) are powerful convolutional architectures designed for network data . GNNs inherit all the favorable properties convolutional neural networks ( CNNs ) admit , while they also exploit the graph structure . An important feature of GNNs , germane to their success , is that the number of learnable parameters is independent of the size of the underlying networks . GNNs have manifested remarkable performance in a plethora of applications , e.g. , recommendation systems ( Ying et al. , 2018 ; Wu et al. , 2020 ) , drug discovery and biology ( Gainza et al. , 2020 ; Strokach et al. , 2020 ; Wu et al. , 2021 ; Jiang et al. , 2021 ) , resource allocation in autonomous systems ( Lima et al. , 2020 ; Cranmer et al. , 2021 ) , to name a few . Recently , there has been an increased interest in time-varying network data , as they appear in various systems and carry valuable dynamical information . This interest is mostly prominent in applications as decentralized controllers ( Tolstaya et al. , 2020 ; Gama et al. , 2020b ; Yang and Matni , 2021 ; Gama and Sojoudi , 2021 ) , traffic-flow forecasting ( Yu et al. , 2018 ; Li et al. , 2018 ; Fang et al. , 2021 ) , and skeleton-based action detection ( Yan et al. , 2018 ; Cheng et al. , 2020 ; Pan et al. , 2021 ) . The state-ofthe-art ( SOTA ) usually deploys an additional architecture side by side with a GNN so the latter learns patterns from the graph domain while the former works on the time sequences . One choice could be a CNN as in ( Li et al. , 2020 ; Isufi and Mazzola , 2021 ; Wang et al. , 2021 ) or a recurrent neural network ( RNN ) as in ( Seo et al. , 2018 ; Nicolicioiu et al. , 2019 ; Ruiz et al. , 2020 ) . However , these joint architectures are performed in a centralized manner in the sense that the up-to-date data of all nodes are given at any given time . While this is well suited for , but not limited to , the case of social networks and recommendation systems , many physical-network applications are decentralized in nature and suffer from time delays in delivering messages . In this paper , we close the gap by developing a causal space-time convolutional architecture that jointly processes the graph-time underlying structure . That is , the convolutional layers preserve time delays in message passing . Our work is motivated by the following question . Is it possible to transfer learning between signals and datasets defined over different space-time underlying structures ? This is a well motivated question since in practice we execute these architectures on graphs that are different from the graphs used in training and signals are sampled at different sampling rates between training and execution . The answer to the above question was provided for the case of static graph signals in ( Gama et al. , 2020a ) , where the stability of traditional GNNs to graph perturbations was proved . In this work we give an affirmative answer to the above question in the case when time-varying graph signals are considered and space-time convolutional architectures are employed . The contribution of this paper is twofold . First we introduce a novel convolutional architecture for time-varying graph signals , and second we prove its stability . Specifically , we provide a general definition of convolutions for any arbitrary shift operator and define a space-time shift operator ( STSO ) as the linear composition of the graph shift operator ( GSO ) and time-shift operator ( TSO ) . We then introduce space-time graph neural networks ( ST-GNNs ) , a cascade of layers that consist of space-time graph filters followed by point-wise nonlinear activation functions . The proposed ST-GNN allows processing continuous-time graph signals , which is pivotal in the stability analysis . Furthermore , we study the effect of relative perturbations on ST-GNNs and prove that small variations in the graph and/or irregularities in the sampling process of time-varying graph signals do not essentially affect the performance of the proposed ST-GNN architecture . Our theoretical findings are also supported by thorough experimental analysis based on decentralized control applications . The rest of this paper is structured as follows . The related work is summarized in Section 2 . Sections 3 and 4 present our contributions listed above . Numerical experiments and conclusions are presented in Sections 5 and 6 , respectively . The proofs and extended experiments are provided in the appendices . Notation : Bold small and large symbols , i.e . x and X , denote vectors and matrices , respectively . Calligraphic letters mainly represent time-varying graph signals unless otherwise is stated . 2 RELATED WORK . GNNs for Time-varying Graph Signals . One of the early architectures was ( Yan et al. , 2018 ) , which introduced a convolutional filter that aggregates information only from 1-hop neighbors . Relying on product graphs , ( Isufi and Mazzola , 2021 ) introduced a convolutional architecture , where each node has access to the present and past data of the other nodes in the graph . A similar convolutional layer was studied in ( Pan et al. , 2021 ; Loukas and Foucard , 2016 ) , and while restricted to a domain that preserves time delays , ( Pan et al. , 2021 ) shows that it is not necessarily less expressive . Meanwhile , ( Wang et al. , 2021 ) performs GNNs over graph signals at each time instance separately before a temporal convolution is performed at the GNN outputs to capture the evolution of graph embeddings . Graph ( RNNs ) are another architecture developed to deal with time-varying graph signals . For example , ( Pareja et al. , 2020 ) uses an RNN to evolve the GNN parameters over time . Moreover , ( Hajiramezanali et al. , 2019 ) combined the GRNN with a variational autoencoder ( VGAE ) to improve the former ’ s expressive power . However , all these architectures did not take into account the physical restrictions in the form of time delays , associated with decentralized applications . Similar to our work , other architectures considered the diffusion equation to form message-passing layers , e.g. , ( Xhonneux et al. , 2020 ; Poli et al. , 2021 ; Fang et al. , 2021 ) . These architectures parameterize the dynamics of graph signals with GNNs . In simple words , they learn a parameterization that helps find the current state variables from the previous ones . The architecture in ( Chamberlain et al. , 2021 ) is another example of these architectures , but learns the graph weights instead of a limited number of filter coefficients ( i.e. , it resembles the parameterization of graph attention networks in ( Veličković et al. , 2018 ) ) . Stability . Deformation stability of CNNs was studied in ( Bruna and Mallat , 2013 ; Bietti and Mairal , 2019 ) . The notion of stability was then introduced to graph scattering transforms in ( Gama et al. , 2019 ; Zou and Lerman , 2020 ) . In a following work , Gama et al . ( 2020a ) presented a study of GNN stability to graph absolute and relative perturbations . Graphon neural networks was also analyzed in terms of its stability in ( Ruiz et al. , 2021 ) . Moreover , ( Pan et al. , 2021 ) proves the stability to absolute perturbations of space-time graph scattering transforms . 3 SPACE-TIME GRAPH NEURAL NETWORKS . In this section , we present the proposed ST-GNN architecture for time-varying graph signals . First , we provide a general definition of convolutions and then we develop the space-time graph filters , which are the cornerstone of the ST-GNN architecture . Our analysis starts with the homogeneous diffusion equation , which is defined with respect to the Laplacian differential operator L ∂X ( t ) ∂t = −LX ( t ) . ( 1 ) Equation ( 1 ) describes processes of signals that evolve across the diffusion dimension , t. The diffused signal , X ( t ) , is modeled as an abstract vector in a vector space V. The vector space is associated with an inner product ⟨. , .⟩V , and is closed under addition and scalar multiplication , e.g. , n-dimensional vector spaces and function spaces . The solution of ( 1 ) is X ( t ) = e−tLX ( 0 ) = e−tLX0 = e−tLX ( the subscript is omitted in what follows ) , and describes the signal after time t from the initial state . 3.1 CONVOLUTION . With the diffusion equation in our hands , we can now define the convolution operation as the linear combination of the diffusion sequence with a filter h ( t ) . Definition 1 ( Convolution Operator ) . For a linear shift-invariant filter with coefficients h ( t ) , t ≥ 0 , the convolution between the filter and an input signal X ∈ V , with respect to a linear differential operator L : V → V , is defined as h ∗D X = ∫ ∞ 0 h ( t ) e−tLXdt , ( 2 ) where ∗D denotes the convolution operator applied on signals with underlying structure D. Definition 1 is a valid convolution operator . With abuse of notation , we refer to L as the shift operator but it should be understood that the shift operation is executed using e−L . Definition 1 establishes a generalized form of convolutions for a wide variety of shift operators and signals . Next we discuss two convolution types of practical interest that follow directly from ( 2 ) , i.e. , time convolutions and graph convolutions . Time Convolutions : They involve continuous-time signals , denoted by x ( τ ) , that are elements of square-integrable function spaces , i.e , x ( τ ) ∈ L2 ( R ) . The underlying structure of x ( τ ) is the real line , since the time variable τ ∈ R. In order to generate the traditional definition of time convolution , the time shift operator ( TSO ) is chosen as the differential operator , Lτ = ∂/∂τ . One can show that e−t∂/∂τx ( τ ) = x ( τ − t ) , using the Taylor series , e−t ∂/∂τx ( τ ) = ∞∑ n=0 ( −t ) n n ! · ∂ n ∂τn x ( τ ) = ∞∑ n=0 ( u− τ ) n n ! · ∂ n ∂τn x ( τ ) = x ( u ) , with u = τ − t. In other words , the TSO performs a translation operation . Then ( 2 ) reduces to x ( τ ) ∗R h ( τ ) = ∫ ∞ 0 h ( t ) e−t ∂/∂τx ( τ ) dt = ∫ ∞ 0 h ( t ) x ( τ − t ) dt , ( 3 ) which is indeed the traditional definition of convolutions between time signals . It is also worth noting that ejωτ , ∀ω ∈ [ 0 , ∞ ) are eigenfunctions of the TSO Lτ = ∂/∂τ with associated eigenvalues jω . This observation is pivotal in the stability analysis of section 4 . Graph Convolutions : They involve graph signals , x ∈ L2 ( RN ) , that are N -dimensional squaresummable vectors containing information associated with N different nodes . The underlying structure of x is represented by a graph G = ( V , E , W ) , where V is the set of nodes with cardinality |V| = N , E ⊆ V × V is the set of edges , and W : E → R is a map assigning weights to the edges ( Shuman et al. , 2012 ) . The graph shift operator ( GSO ) , S ∈ RN×N , is a matrix representation of the graph sparsity , e.g. , graph adjacency or Laplacian ( Ortega et al. , 2018 ) . Applying S to a signal x results in a signal Sx , where [ Sx ] i represents aggregated data from all nodes j that satisfy that ( i , j ) ∈ E or ( j , i ) ∈ E . We focus on undirected graphs , and therefore , S is a real symmetric and diagonalizable matrix with a set of N eigenvectors { vi } Ni=1 , each associated with a real eigenvalue λi . The GSO is a linear operator S : L2 ( RN ) → L2 ( RN ) that deals with discrete signals . Therefore , it is more common to discretize the diffusion process over graphs at times kTs , k ∈ Z+ , where Ts is the sampling period . The convolution is then defined as x ∗G h = K−1∑ k=0 hke −kSx , ( 4 ) where { hk } k is the filter coefficients and K is the number of filter taps . The finite number of taps is a direct consequence of Cayley-Hamilton theorem , and h is a finite-impulse response ( FIR ) filter . Equation ( 4 ) generalizes graph convolutions in the literature , where convolutions are defined as polynomials in the shift operator , i.e. , x ∗G h = ∑K−1 k=0 hkS kx , ( Sandryhaila and Moura , 2013 ) . This definition executes the shift operation using the matrix S directly compared to e−S in ( 4 ) .

Summary: This paper proposes a new deep learning architecture called ST-GNN that learns representations on graphs that evolve over time. Their work focuses on developing an interesting time-varying convolutional architecture, which exploits the graph-time underlying structure of the signals, processing across both the graph and time domains. The authors introduce new graph shift operators (GSO), time shift operators (TSO), followed by a linear combination of those which results in a space-time shift operator termed STSO. The advantage of this paper is that it can handle continuous-time graph signals and the stability of this proposed architecture is studied theoretically (Section 4) with a significant result outlined in Proposition 1 of the paper where the difference between the space-time graph filters of the original graph and the perturbed graph are upper bounded by the order of error introduced.