Adaptive Spatial-Temporal Inception Graph Convolutional Networks for Multi-step Spatial-Temporal Network Data Forecasting

1 INTRODUCTION . Spatial-temporal data forecasting has attracted attention from researchers due to its wide range of applications and the same specific characteristics of spatial-temporal data . Typical applications include mobile traffic forecast ( He et al. , 2019 ) , traffic road condition forecast ( Song et al. , 2020 ; Yu et al. , 2017 ; Guo et al. , 2019 ; Zheng et al. , 2020 ; Li et al. , 2017 ) , on-demand vehicle sharing services passenger demand forecast ( Bai et al. , 2019 ) and geo-sensory time series prediction ( Liang et al. , 2018 ) etc . The accurate forecast is the foundation of many real-world applications , such as Intelligent Telecom Network Operation and Intelligent Transportation Systems ( ITS ) . Specifically , accurate traffic forecast can help transportation agencies better control traffic scheduling and reduce traffic congestion ; The traffic volumes prediction of the wireless telecommunication network plays an important role for the network operation and optimization , for example , it can help to infer the accurate sleep periods ( low traffic periods ) of the base stations to achieve energy saving without sacrificing customer experience . However , as we all know , accurate spatialtemporal data forecasting faces multiple challenges . First , it is inherent with complex spatial-temporal correlations . In the spatialtemporal graph , different neighbors may have different impacts on the central location at the same time step , as the bold lines shown in Figure1 , which called spatial correlations.Different historical observations of the same location influence the future moments of itself variously due to temporal correlations . The observations of different neighbors at historical moments can directly affect the central node at future time steps due to the spatial-temporal joint correlations . As shown in Figure1 , the information of the spatialtemporal network can propagate along the spatial and temporal dimensions simultaneously , and the transmission process can be discontinuous due to complex external factors , which result in spatialtemporal joint correlations of the spatial-temporal data in a short period . Spatial-temporal data is heterogenous in both spatial and temporal dimensions ( Song et al. , 2020 ) . Nodes in different regions of the graph have various properties and local spatial structures , so the corresponding data distribution can be different . For example , the traffic flow distribution of urban and suburban areas are quite different , while the traffic of urban area is denser and that of suburban area is relatively sparse . Besides , the traffic flow in the same region also exhibit heterogeneity in different time periods . For example , the mobile traffic in business district would decrease at night compared to the daytime , while it ’ s opposite in the residential district . In addition , multi-step time series forecasting is often accompanied by error accumulation problem . Typical methods like RNNs often cause error accumulation due to iterative forecasting , leading to rapid deterioration of the long-term prediction accuracy . ( Yu et al. , 2017 ; Zheng et al. , 2020 ) . Most of the previous work is mainly to solve the above challenges . To model the spatial-temporal dependency , STGCN ( Yu et al. , 2017 ) and DCRNN ( Li et al. , 2017 ) extract spatial and temporal correlations separately . ASTGCN ( Guo et al. , 2019 ) introduced spatial and temporal attention to model the dynamic spatial and temporal correlations . STG2Seq ( Bai et al. , 2019 ) aimed at using GCN to capture spatial and temporal correlations simultaneously . But they all didn ’ t consider the spatialtemporal joint correlations and heterogeneity . Different from the above methods , STSGCN ( Song et al. , 2020 ) used multiple local spatial-temporal graphs to model the spatial-temporal synchronous correlations and spatial-temporal heterogeneity of the local adjacent time steps . But STSGCN can only model the spatial-temporal synchronous correlations of its defined local spatial-temporal graphs and it is equipped with complex structure . In this paper , we propose a novel model called ASTI-GCN , Adaptive spatial-temporal Inception Graph Convolutional Networks , to address the above issues with multi-step spatial-temporal data forecasting . We propose the spatial-temporal joint convolution to directly model the spatial-temporal joint correlations without introducing elaborately constructed mechanisms . And we introduce the inception mechanism to build multi-scale spatial-temporal features to adapt to graph nodes with different properties . Then , to achieve the heterogeneity modeling , we construct the spatial-temporal Inception Graph Convolution Module , which combined the spatial-temporal inception mechanism with the graph attention to build the adaptive ability of graph nodes with different properties . After multiple spatial-temporal inception-GCMs , two decoder modules named sequence decoder and short-term decoder are designed to directly establish the relationships between the historical and future time steps to alleviate error accumulation . Overall , our main contributions are summarized as follows : • We propose a novel spatial-temporal joint graph convolution network to directly capture spatial-temporal correlations . Moreover , we introduce inception with graph attention to adaptively model the graph heterogeneity . • We propose to combine the sequence decoder and short-term decoder together for multistep forecasting to model direct relationships between historical and future time steps to alleviate the error propagation . • We evaluate our model on three real-world datasets from two fields , and the experimental results show that our model achieves the best performances among all the eight baselines with good generalization ability . 2 RELATED WORK . Spatial-temporal data information can be extracted using the deep learning method from European space , such as ConvLSTM ( Xingjian et al. , 2015 ) , PredRNN ( Gowrishankar & Satyanarayana , 2009 ) and so on . However , most of the spatial-temporal data in real scenes are graph data with complex and changeable relationships.Common timing prediction models , such as HA and ARIMA ( Williams & Hoel , 2003 ) , can not be simply migrated to such scenarios . Graph based methods like DCRNN ( Li et al. , 2017 ) modeled traffic flow as a diffusion process on a directed graph . Spatial dependencies and temporal dependencies are captured by bidirectional random walk and DCGRU based encoder-decoder sequence to sequence learning framework respectively . STGCN ( Yu et al. , 2017 ) constructed an undirected graph of traffic network , which is combined with GCN and CNN to model spatial and temporal correlation respectively . ASTGCN ( Guo et al. , 2019 ) innovatively introduced attention mechanisms to capture dynamic spatial and temporal dependencies . Similarly , GMAN ( Zheng et al. , 2020 ) used temporal and spatial attention to extract dynamic spatial-temporal correlations with spatial-temporal coding . The above models extract spatial-temporal correlation with two separate modules , which can not learn the influence of neighbor node at the same time and the influence of center node at the historical moment simultaneously . To address this problem , Bai et al . ( 2019 ) proposed STG2seq to learn the influence of spatial and temporal dimensions at the same time , which is a purely relies on graph convolution structure . However , all the above methods fail to take the heterogeneity of spatial-temporal data into account , that is , the scope of each node influencing its neighbor nodes at future time steps is different . To solve this problem , Song et al . ( 2020 ) proposed STSGCN with multiple modules for different time periods to effectively extract the heterogeneity in local spatial-temporal maps . However , this method pays more attention to local information and lacks of global information extraction . Besides , STSGCN runs slowly due to too many parameters . Therefore , we propose an Adaptive spatial-temporal Inception Graph Convolutional Networks The Temporal and spatial correlations are extracted simultaneously by spatial-temporal convolution , and the node heterogeneity is modeled by Inception mechanism . At the same time , considering the different influences of each node and time step , the attention mechanism is introduced to adjust the influence weight adaptively . 3 METHODOLOGY . 3.1 PRELIMINARIES . In this paper , we define G = ( V , E , A ) as a static undirected spatial graph network . V represents the set of vertices , |V | = N ( N indicates the number of vertices ) . E is the set of edges representing the connectivity between vertices . A ∈ RN×N is the adjacency matrix of the network graph G where Avi , vj represents the connection between nodes vi and vj . The graph signal matrix is expressed as Xt ∈ RN×C , where t denotes the timestep and C indicates the number of features of vertices . The graph signal matrix represents the observations of graph network G at time step t. Problem Studied Given the graph signal matrix of historical T time steps χ = ( Xt1 , Xt2 , . . . , XtT ) ∈ RT×N×C , our goal is to predict the graph signal matrix of the next M time steps Ŷ = ( X̂tT+1 , X̂tT+2 , ... , X̂tT+M ) ∈ RM×N×C . In other words , we need to learn a mapping function F to map the graph signal matrix of historical time steps to the future time steps : ( X̂tT+1 , X̂tT+2 , · · · , X̂tT+M ) = Fθ ( Xt1 , Xt2 , · · · , XtT ) ( 1 ) where θ represents learnable parameters of our model . 3.2 ARCHITECTURE . The architecture of the ASTI-GCN proposed in this paper is shown in Figure 2 ( a ) . The main ideas of ASTI-GCN can be summarized as follows : ( 1 ) We propose spatial-temporal joint graph convolution to directly extract the spatial-temporal correlations ; ( 2 ) We build the Spatio-Temporal Inception Graph Convolutional Module ( STI-GCM ) to adaptively model the graph heterogeneity ; ( 3 ) We use short-term decoder combined with sequence decoder to achieve accurate multi-step forecast . 3.3 SPATIAL-TEMPORAL INCEPTION-GCM . Spatial-temporal joint graph convolution In order to extract spatial-temporal correlations simultaneously , we propose spatial-temporal joint graph convolution . In this paper , we construct spatial-temporal joint graph convolution based on graph convolution in the spectral domain . The spectral graph convolution implemented by using the graph Fourier transform basis which is from eigenvalue decomposition of the Laplacian matrix ( L ) to transform the graph signals from spatial into the spectral domain . But the computation cost of the eigenvalue decomposition of L is expensive when the graph is large . To reduce the number of parameters and the computation complexity , Chebyshev polynomial Tk ( x ) is used for approximation . The spectral graph convolution can be written as ( Yu et al. , 2017 ; Guo et al. , 2019 ; Kipf & Welling , 2016 ) : Θ∗Gx = Θ ( L ) x ≈ K−1∑ k=0 θkTk ( L̃ ) x ( 2 ) where ∗G is graph convolution operator , Θ is graph convolution kernel , x ∈ RN is the graph signal , Tk ( L̃ ) ∈ RN×N is the Chebyshev polynomial of order k with the scaled Laplacian L̃ = 2 λmax L − IN ( L is the graph Laplacian matrix , λmax is the largest eigenvalue of L , IN is identity matrix ) ( Yu et al. , 2017 ) . θk is the coefficient of the k-th order polynomial . Based on spectral domain graph convolution , we propose spatial-temporal joint graph convolution . First , these K-hop Tk ( L̃ ) are concatenated as the furthest receptive field in the spatial dimension . Then we construct the spatial-temporal joint graph convolution kernel Θs , t , Θs , t ∈ RKt×Ks×C×F , where Kt represents the kernel size in the temporal dimension and Ks represents the kernel size in the spatial dimension , C is the input feature dimensions , F is the number of filters . So the kernel Θs , t has the local spatial-temporal receptive field of Kt × Ks , and Ks should be lower than K ( which can be written as Ks < K ) , because of the largest graph convolution perceived field of Khop . And the spatial-temporal joint graph convolution can be formulated as : TK ( L̃ ) = Concat ( T0 ( L̃ ) , T1 ( L̃ ) , . . . , TK−1 ( L̃ ) ) ( 3 ) Xout = Θs , t ∗X = Θs , tTK ( L̃ ) X ( 4 ) where TK ( L̃ ) ∈ RK×N×N is the concatenation of all Chebyshev polynomials in ( K-1 ) hop . ∗ is the convolution operation between Θs , t and X , X ∈ RN×T×C is the spatial-temporal signal of the input graph , T is the input time steps . After the spatial-temporal joint graph convolution , the output can be written as Xout ∈ RN× ( T−Kt+1 ) × ( K−Ks+1 ) ×F . Besides , the neighbors have various influences on the central node , so we implement a learnable spatial mask matrix Wmask ∈ RN×N ( Song et al. , 2020 ) to adjust the graph adjacency relationship for assigning weights to different neighbors . Spatial-temporal inception-attention Different from the images , each node of the spatial-temporal graph usually represents a road or eNodeB etc . Then , affected by external factors like geographic location and surrounding environment , the spatial-temporal data properties of different nodes are various , namely the heterogeneity . To solve this problem , an intuitive method is to learn different models for each node , but this method could cause extensive parameters and maintain low generalization ability . So we take another way in this paper . We understand that heterogeneity is manifested in the differences of local spatialtemporal receptive fields of each node of the graph which result from nodes ’ various properties and local spatial structures . Inspired by ( Song et al. , 2020 ; Zheng et al. , 2020 ; Zhou et al. , 2018 ; Vaswani et al. , 2017 ) , we apply a learnable graph node embedding Se ∈ RN×E to represent the properties of each node . Meanwhile , we introduce inception ( Szegedy et al. , 2015 ) to extract multi-scale spatialtemporal correlations through spatial-temporal joint graph convolution . Then , we combine the graph attention with inception to achieve node-level attention for modeling the heterogeneity . Firstly , we implement inception , as shown in Figure 2 ( b ) . For example , the 3×2 block represents that it involves the kernel θs , t ∈ R3×2×C×F , which means it can extract the spatial-temporal correlations of the node itself and its neighbors in the three adjacent time steps by one layer , which is needed by two layers STSGCM in STSGCN ( Song et al. , 2020 ) . We use the padding method when implement inception , so after B branches , the output of the inception module can be Cout ∈ RN×T×K× ( F×B ) , where we set the number of output filters of each branch to be the same . Then we combined with the graph node attention , which has being widely used ( Vaswani et al. , 2017 ) . We use the Q = SeWq , Wq ∈ RE×F to get the queries of graph nodes . For each branch in inception , we apply the idea of SKNET ( Li et al. , 2019 ) to do global pooling Cgl = T∑ i=1 K∑ j=1 Cout , Cg ∈ RN× ( F×B ) as the corresponding keys of the branches ( one can also use Wk ∈ RF×F to do transform as ( Vaswani et al. , 2017 ) , here we omit it for simplicity ) . Then we compute the attention by S = QKT ( Vaswani et al. , 2017 ) . Take a graph node vi as an example , Svi , b = qvi•cg , vi , b√ F , qvi ∈ R1×F , cg , vi , b ∈ R1×F denotes the attention of vi and each branchCg , vi , b which represents the corresponding spatialtemporal receptive field perception . Then we concatenate the inception branch results adjusted by attention score to obtain the output . The calculation can be formulated as follows : αvi , b = exp ( Svi , b ) B∑ bc=1 exp ( Svi , bc ) ( 5 ) Attvi = ||Bb=1 { αvi , b · Cout , vi , b } ( 6 ) where αvi , b ∈ R , Cout , vi , b ∈ RT×K×F , Attvi ∈ RT×K×F×B represent the output of node vi from inception-attention block . Therefore , the final output of spatial-temporal inception-attention block is Catt ∈ RN×T×K×F×B . STI-GCM output Layer Then , we use spatial convolution to generate the output of STI-GCM . We first reshape Catt into Catt ∈ RN×T×K× ( F ·B ) . Next , the learnable weight matrix Ws ∈ RK× ( F ·B ) × ( F ·B ) is used to convert Catt to Csatt ∈ RN×T× ( F ·B ) . We also implement SE-net ( Hu et al. , 2020 ) to model the channel attention . Finally , the output is converted to Cstatt ∈ RN×T×F using the full connection layer with Wo ∈ R ( F ·B ) ×F . And the process can be formulated as Cstatt = CattWsWo .

Summary: This paper proposes a spatial-temporal graph neural network, which is designed to adaptively capture the complex spatial-temporal dependency. Further, the authors design a spatial-temporal attention module, which aims to capture multi-scale correlations. For multi-step prediction instead of one-step prediction, they further propose the sequence transform block to solve the problem of error accumulations. The authors conducted experiments on three real-world datasets (traffic on highways and mobile traffic), which shows their method achieves the best performance.