Spatio-Temporal Graph Scattering Transform

1 INTRODUCTION . Processing and learning from spatio-temporal data have received increasing attention recently . Examples include : i ) skeleton-based human action recognition based on a sequence of human poses ( Liu et al . ( 2019 ) ) , which is critical to human behavior understanding ( Borges et al . ( 2013 ) ) , and ii ) multi-agent trajectory prediction ( Hu et al . ( 2020 ) ) , which is critical to robotics and autonomous driving ( Shalev-Shwartz et al . ( 2016 ) ) . A common pattern across these applications is that data evolves in both spatial and temporal domains . This paper aims to analyze this type of data by developing novel spatio-temporal graph-based data modeling and operations . Spatio-temporal graph-based data modeling . Graphs are often used to model data where irregularly spaced samples are observed over time . Good spatio-temporal graphs can provide informative priors that reflect the internal relationships within data . For example , in skeleton-based human action recognition , we can model a sequence of 3D joint locations as data supported on skeleton graphs across time , which reflects both the human physical constraints and temporal consistency ( Yan et al . ( 2018 ) ) . Recent studies on modeling spatio-temporal graphs have followed either joint or separable processing frameworks . Joint processing is based on constructing a single spatio-temporal graph and processing ( e.g. , filtering ) via operations on this graph ( Kao et al . ( 2019 ) ; Liu et al . ( 2020 ) ) . In contrast , a separable processing approach works separately , and possibly with different operators , across the space and time dimension . In this case , independent graphs are used for space and ∗This work was mainly done while Chao Pan and Siheng Chen were working at Mitsubishi Electric Research Laboratories ( MERL ) . time ( Yan et al . ( 2018 ) ; Cheng et al . ( 2020 ) ) . However , no previous work thoroughly analyzes and compares these two constructions . In this work , we mathematically study these two types of graphs and justify the benefits of separable processing from both theoretical and empirical aspects . Spatio-temporal graph-based operations . Graph operations can be performed once the graph structure is given . Some commonly used graph operations include the graph Fourier transform ( Shuman et al . ( 2013 ) ) , and graph wavelets ( Hammond et al . ( 2011 ) ) . It is possible to extend those operations to the spatio-temporal graph domain . For example , Grassi et al . ( 2017 ) developed the short time-vertex Fourier transform and spectrum-based time-vertex wavelet transform . However , those mathematically designed , linear operations show some limitations in terms of empirical performances . In comparison , many recent deep neural networks adopt trainable graph convolution operations to analyze spatio-temporal data ( Yan et al . ( 2018 ) ; Liu et al . ( 2020 ) ) . However , most networks are designed through trial and error . It is thus hard to explain the rationale behind empirical success and further improve the designs ( Monga et al . ( 2019 ) ) . In this work , to fill in the gap between mathematically designed linear transforms and trainable spatio temporal graph neural networks , we propose a novel spatio-temporal graph scattering transform ( ST-GST ) , which is a mathematically designed , nonlinear operation . Specifically , to characterize the spatial and temporal dependencies , we present two types of graphs corresponding to joint and separable designs . We then construct spatio-temporal graph wavelets based on each of these types of graphs . We next propose the framework of ST-GST , which adopts spatio-temporal graph wavelets followed by a nonlinear activation function as a single scattering layer . All the filter coefficients in ST-GST are mathematically designed beforehand and no training is required . We further show that i ) a design based on separable spatio-temporal graph is more flexible and computationally efficient than a joint design ; and ii ) ST-GST is stable to small perturbations on both input spatio-temporal graph signals and structures . Finally , our experiments on skeletonbased human action recognition show that the proposed ST-GST outperforms spatio-temporal graph convolutional networks by 35 % accuracy in MSR Action3D dataset . We summarize the main contributions of this work as follows : •We propose wavelets for both separable and joint spatio-temporal graphs . We show that it is more flexible and computationally efficient to design wavelets based on separable spatio-temporal graphs ; •We propose a novel spatio-temporal graph scattering transform ( ST-GST ) , which is a non-trainable counterpart of spatio-temporal graph convolutional networks and a nonlinear version of spatiotemporal graph wavelets . We also theoretically show that ST-GST is robust and stable in the presence of small perturbations on both input spatio-temporal graph signals and structures ; • For skeleton-based human action recognition , our experiments show that : i ) ST-GST can achieve similar or better performances than spatio-temporal graph convolutional networks and other nondeep-learning approaches in small-scale datasets ; ii ) separable spatio-temporal scattering works significantly better than joint spatio-temporal scattering ; and iii ) ST-GST significantly outperforms spatio-temporal graph wavelets because of the nonlinear activation function . 2 RELATED WORK . Scattering transforms . Convolutional neural networks ( CNNs ) use nonlinearities coupled with trained filter coefficients and are well known to be hard to analyze theoretically ( Anthony & Bartlett ( 2009 ) ) . As an alternative , Mallat ( 2012 ) ; Bruna & Mallat ( 2013 ) propose scattering transforms , which are non-trainable versions of CNNs . Under admissible conditions , the resulting transform enjoys both great performance in image classification and appealing theoretical properties . These ideas have been extended to the graph domain ( Gama et al . ( 2019a ) ; Zou & Lerman ( 2020 ) ; Gao et al . ( 2019 ) ; Ioannidis et al . ( 2020 ) ) . Specifically , the graph scattering transform ( GST ) proposed in ( Gama et al . ( 2019a ) ) iteratively applies predefined graph filter banks and element-wise nonlinear activation function . In this work , we extend classical scattering transform to the spatio-temporal domain and provide a new mathematically designed transform to handle spatio-temporal data . The key difference between GST and our proposed ST-GST lies in the graph filter bank design , where ST-GST needs to handle both spatial and temporal domains . Spatio-temporal neural networks . Deep neural networks have been adapted to operate on spatiotemporal domain . For example , Liu et al . ( 2019 ) uses LSTM to process time series information , while ST-GCN ( Yan et al . ( 2018 ) ) combines a graph convolution layer and a temporal convolution layer as a unit computational block in the network architecture . However , those networks all require a huge amount of high-quality labeled data , and training them is computationally expensive , which may make them impractical for many real-world scenarios . Furthermore , many architectures are designed through trial and error , making it hard to justify the design choices and further improve them . In this work , the proposed ST-GST is a nonlinear transform with a forward procedure similar to that of ST-GCN . However , ST-GST does not require any training , which is useful in many applications where only limited training data is available . Furthermore , since all filter coefficients in ST-GST are predefined , it allows us to perform theoretical analysis and the related conclusions potentially shed some light on the design of spatio-temporal networks . Skeleton-based human action recognition . Conventional skeleton-based action recognition models learn semantics based on hand-crafted features ( Wang et al . ( 2012 ) ) . To handle time series information , some recurrent-neural-network-based models are proposed to capture the temporal dependencies between consecutive frames ( Kim & Reiter ( 2017 ) ) . Recently , graph-based approaches have gained in popularity while achieving excellent performance ( Yan et al. , 2018 ; Li et al. , 2019 ) . In this work , our experiments focus on this task and show that ST-GST outperforms the state-of-the-art spatio-temporal graph neural networks , like MS-G3D ( Liu et al. , 2020 ) , on small-scale datasets . 3 SPATIO-TEMPORAL GRAPH SCATTERING TRANSFORM . In this section , we first define spatio-temporal graph structures and signals . We next design our spatio-temporal graph wavelets . Finally , we present ST-GST . 3.1 SPATIO-TEMPORAL GRAPH STRUCTURES AND SIGNALS . Spatio-temporal data can be represented as a matrix X ∈ RN×T , where N is the number of the spatial positions and T is the number of time stamps . In this matrix , each row is a time-series for a spatial node , and each column is a spatial signal at a certain time stamp . Note that the index of spatial information can be arbitrary : we will associate each spatial location to a vertex on the spatial graph , and the edges will provide information about the relative position of the nodes . We can reshape the matrix to form a vector x of length NT , where the element x ( s , t ) : = x ( s−1 ) T+t is the feature value corresponding to the s-th vertex at time t. To construct a spatio-temporal graph , we create connections based on physical constraints . For example , for skeleton-based action recognition , the spatial graph is the human skeleton graph , reflecting bone connections ; see Fig . 1 ( a ) ; and the temporal graph is a line graph connecting consecutive time stamps ; see Fig . 1 ( b ) . As a starting point , we choose a spatial graph Gs = ( Vs , Es , As ) with |Vs| = N , reflecting the graph structure of each column in X and a temporal graph Gt = ( Vt , Et , At ) with |Vt| = T , reflecting the graph structure of each row in X . The separable spatio-temporal design is achieved by processing the columns and rows of X separately based on their respective graphs . As an alternative , a product graph , denoted as G = Gs Gt = ( V , E , A ) can be constructed to unify the relations in both the spatial and temporal domains , allowing us to process data jointly across space and time . The product graph Gs Gt has |V| = NT nodes and an appropriately defined NT ×NT adjacency matrix A . The operation interweaves two graphs to form a unifying graph structure . The edge weight A ( s1 , t1 ) , ( s2 , t2 ) : = A ( s1−1 ) T+t1 , ( s2−1 ) T+t2 characterizes the relation , such as similarity or dependency , between the s1-th spatial node at the t1-th time stamp and the s2-th spatial node at the t2-th time stamp . There are three commonly used product graphs ( Sandryhaila & Moura , 2014 ) : i ) Kronecker product : G = Gs⊗Gt with graph adjacency matrix as A = As⊗At and ⊗ represents the Kronecker product of matrices ; see Fig 1 ( c ) ; ii ) Cartesian product : G = Gs × Gt with A = As ⊗ IT + IN ⊗ At ; see Fig 1 ( d ) ; and iii ) strong product : G = Gs Gt with A = As⊗At + As⊗ IT + IN ⊗At , which can be viewed as a combination of Kronecker and Cartesian products ; see Fig 1 ( e ) . The joint spatio-temporal design is achieved based on a product graph . In this paper , we consider designs based on both separable graphs and product graphs .

Summary: The authors propose wavelets for both separable and joint spatio-temporal graphs. And then the authors design a spatio-temporal graph scattering transform (ST-GST), which is a non-trainable counterpart of spatio-temporal graph convolutional networks and a nonlinear version of spatiotemporal graph wavelets. Finally, the proposed SF-GST is conducted by experiments, and the results show that it appears to be effective. However, The authors did not give the explanation of the motivation about why did the STG should be scattered by wavelets. Besides, from the results in Table 1,  the joint versions based on the proposed method. i.e., Joint Kronecker, Joint Cartesian and Joint Strong, have not achieved the satisfied performance, though only separable versions performs best.