Adaptive Spatial-Temporal Inception Graph Convolutional Networks for Multi-step Spatial-Temporal Network Data Forecasting
Authors: Xing Wang
Lin Zhu
Juan Zhao
Zhou Xu
Zhao Li
Junlan Feng
Chao Deng
Venue:  (2021)

Abstract: Spatial-temporal data forecasting is of great importance for industries such as telecom network operation and transportation management. However, spatial-temporal data is inherent with complex spatial-temporal correlations and behaves heterogeneities among the spatial and temporal aspects, which makes the forecasting remain as a very challenging task though recently great work has been done. In this paper, we propose a novel model, Adaptive Spatial-Temporal Inception Graph Convolution Networks (ASTI-GCN), to solve the multi-step spatial-temporal data forecasting problem. The model proposes multi-scale spatial-temporal joint graph convolution block to directly model the spatial-temporal joint correlations without introducing elaborately constructed mechanisms. Moreover inception mechanism combined with the graph node-level attention is introduced to make the model capture the heterogeneous nature of the graph adaptively. Our experiments on three real-world datasets from two different fields consistently show ASTI-GCN outperforms the state-of-the-art performance. In addition, ASTI-GCN is proved to generalize well. 

joint graph Under review as a conference paper at ICLR 2021
ADAPTIVE SPATIAL-TEMPORAL INCEPTION GRAPH CONVOLUTION NETWORKS FOR MULTI-STEP SPATIAL-TEMPORAL DATA FORECASTING
Anonymous authors Paper under double-bling review
ABSTRACT
Spatial-temporal data forecasting is of great importance for industries such telecom network operation and transportation management. However, spatial- temporal data are inherent with complex spatial-temporal correlations and behaves heterogeneities among the spatial and temporal aspects, which makes the forecast- ing remain very challenging task though recently great work has been done. In this paper, we propose a novel model, Adaptive apatial-Tempone Inception Graph Convolution Networks (ASTI-GCN), to solve the multi-step spatial-temporal data forecasting problem. The model proposes multi-scale convolution block to directly model the spatial-temporall joint correlations without introducing elaborately constructed mechanisms. Moreover inception mechanism combined with the graph node-level attention introduced to make the model capture the heterogeneous nature of the graph adaptively. Our experiments on three real-world datasets from two different fields consistently show ASTI-GCN outperforms the state-of-the-art performance. In addition, ASTI-GCN is proved to generalize well.
INTRODUCTION
Spatial-temporal data forecasting has attracted attention from researchers due to its wide range of applications and the same specific characteristics of spatial-temporal data. Typical applications include mobile traffic forecast (He et al. 2019). traffic road condition forecast (Song et al.) 2020) Yu et al., 2017; Guo al. 2019: Zheng
2020 Li et al., 2017), on-demand vehicle sharing services passenger demand forecast (Bai et al. 2019) and geo-sensory time series prediction (Liang al. 2018) etc. The accurate forecast the foundation of many real-world applications, such as Intelligent Telecom Network Operation and Intelligent Transportation Systems (ITS). Specifically, accurate traffic forecast can help transportation agencies better control traffic scheduling and reduce traffic congestion; The traffic volumes prediction of the wireless telecommunication network plays an important role for the network operation and optimization, for example, it can help to infer the accurate sleep periods (low traffic periods) of the base stations to achieve energy saving without sacrificing customer experience.
However, as we all know, accurate spatial- temporal data forecasting faces multiple chal- lenges. First, is inherent with complex spatial-temporal correlations. In the spatial- temporal graph, different neighbors may have different impacts on the central location at the same time step, as the bold lines shown in Fig- u which called spatial correlations.J correlations.Different Different historical observations of the same location in- fluence the future moments of itself variously due to temporal correlations. The observations of different neighbors historical moments can directly affect the central node at future time steps due to the temporal network propagate along the spatial and temporal dimensions simultaneously, and the
Spaial convbica
Figure Spatial-tempore correlations.
joint correlations. As shown in Figurq] the information of the spatial-

---

Under review as a conference paper at ICLR 2021
transmission process be discontinuous due to complex external factors, which result in spatial- temporal joint correlations of the spatial- data in a short period.
Spatial-temporal data is heterogenous in both spatial and temporal dimensions (Song et al. 2020 Nodes in different regions of the graph have various properties and local spatial structures, so the corresponding data distribution can be different. For example, the traffic flow distribution of urban and suburban areas are quite different, while the traffic of urban area is denser and that of suburban area is relatively sparse. Besides, the traffic flow in the same region also exhibit heterogeneity in different time periods. For example, the mobile traffic in business district would decrease at night compared the daytime, while it's opposite in the residential district. In addition, multi-step time series forecasting is often accompanied by error accumulation problem. Typical methods like RNNs often cause error accumulation due to iterative forecasting, leading to rapid deterioration of the long-term prediction accuracy. (Yu et al. 2017 Zheng et al. 2020).
Most of the previous work is mainly to solve the above challenges. To model the spatial-tempont dependency, STGCN (Yu et al. (017) and DCRNN (Li al.) 2017) extract spatial and temporal cor- relations separately. ASTGCN (Guo et al., 2019) introduced spatial and temporal attention to model the dynamic spatial and temporal correlations. STG2Seq (Bai et al., (019) aimed at using GCN to capture spatial and temporal correlations simultancously. But they all didn't consider the spatial- temporal joint correlations and heterogeneity. Different from the above methods, STSGCN (Song et al. 2020) used multiple local spatial-temporal graphs to model the spatial-temport synchronous correlations and spatial-temporal heterogeneity of the local adjacent time steps. But STSGCN can only model the temppolll synchronous correlations of its defined local spatial-tempont graphs and it equipped with complex structure.
In this paper, we propose a novel model called ASTI-GCN, Adaptive spatial-temporal Inception Graph Convolutional Networks, address the above issues with multi-step spatial-temporat data forecasting, We propose the spatial-temporal joint convolution to directly model the spatial-temper joint correlations without introducing elaborately constructed mechanisms. And we introduce the inception mechanism to build multi-scale spatial-tempora features to adapt to graph nodes with dif- ferent properties. Then, to achieve the heterogeneity modeling, we construct the spatial-temporal Inception Graph Convolution Module, which combined the spatial-temporal inception mechanism with the graph attention to build the adaptive ability of graph nodes with different properties. Af- ter multiple spatial-temporey short-term decoder are designed to directly establish the relationships between the historical and future time steps to alleviate error accumulation.
CrreptionGGMMs two decoder modules named sequence decoder and
Overall, our main contributions are summarized as follows:
We propose novel spatial-temporal joint graph convolution network to directly capture seatial-temporal correlations. Moreover, we introduce inception with graph attention to adaptively model the graph heterogeneity.
We propose combine the sequence decoder and short-term decoder together for multi- step forecasting to model direct relationships between historical and future time steps to alleviate the error propagation.
We evaluate our model on three real-world datasets from two fields, and the experimental results show that our model achieves the best performances among all the eight baselines with good generalization ability.
2 RELATED WORK
Spatial-temporal data information can be extracted using the deep learning method from European space, such as ConvLSTM (Xingjian et al. 20155), PredRNN (Gowrishankar & Satyanarayana 2009) and so on. However, most of the spatial-temporal data in real scenes are graph data with complex and changeable relationships. Com.nssmssssooooonnnoooo Common timing prediction models, such as HA and ARIMA (Williams & Hoel 2003), cannot be simply migrated to such scenarios. Graph based methods like DCRNN (Li et al. (017)) modeled traffic flow as a diffusion process on a directed graph. Spatial depen- dencies and temporal dependencies are captured by bidirectional random walk and DCGRU based
2

---

Under review as a conference paper at ICLR 2021
mecder-deccodr sequence to sequence learning framework respectively. STGCN YYu et al. 2017) constructed an undirected graph of traffic network, which is combined with GCN and CNN to model spatial and temporal correlation respectively. ASTGCN (Guo al. 2019) innovatively introduced attention mechanisms to capture dynamic spatial and temporal dependencies. Similarly, GMAN (Zheng et al., 2020) used temporal and spatial attention to extract dynamic spatial-temporal corre- lations with spatial-temporal coding. The above models extract spatial-tempors two separate modules, which cannot learn the influence of neighbor node at the same time and the influence of center node at the historical moment simultaneously. To address this problem, Bai et al. (2019) proposed STG2seq to learn the influence spatial and temporal dimensions at the same time, which is purely relies on graph convolution structure. However, all the above methods fail to take heterogeneity of spatial-temporal data into account, that is, the scope of each node influencing its neighbor nodes at future time steps is different. To solve this problem, [Song et al. (2020) proposed STSGCN with multiple modules for different time periods to effectively extract the heterogeneity in local spatial-tempor maps. However, this method pays more attention to local information and lacks of global information extraction. Besides, STSGCN runs slowly due to too many parameters.
correlation with
Therefore, propose Adaptive spatial-temporal Inception Graph Convolutional Networks The Temporal and spatial correlations are extracted simultaneously by spatial-temporal convolution, and the node heterogeneity is modeled by Inception mechanism. At the same time, considering the different influences of each node and time step, the attention mechanism is introduced to adjust the influence weight adaptively.
METHODOLOGY
3.1 PRELIMINARIES
In this paper, we define 9 = (V. F, A) as a static undirected spatial graph network. V represents the of vertices, |V| N(N indicates the number of vertices). E' is the connectivity between vertices. A E N is the adjacency matrix of the network graph Ç where Ag,,, represents the connection between nodes 0; and l/ The graph signal matrix is expressed as X, NR where denotes the timestep and indicates the number of features vertices. The graph signal matrix represents the observations of graph network G at time step t.
edges representing the
Problem Studied Given the graph signal matrix of historical T time steps X (Xt,, Xty...., Xtr) E RT'NNxC' our goal is to predict the graph signal matrix of the next M time steps Ý = (Rern: Xtrezi..,, { E MxXxC In other words, we need to learn mapping function to map the graph signal matrix of historical time steps to the future time steps:
(Xtri1s &
Seri Foo (Xa,, ,
(1)
where 0 represents learnable parameters of our model.
3.2 ARCHITECTURE
The architecture of the ASTI-GCN proposed in this paper shown in Figure 2(a) The main ideas of ASTI-GCN can be summarized as follows: (1) We propose spatial-temporal joint graph convolution to directly extract the spatial- -patial-temporal correlations; (2) We build the Spatio- Temporal Inception Graph Convolutional Module (STI-GCM) to adaptively model the graph heterogeneity; (3) We use short-term decoder combined with sequence decoder to achieve accurate multi-step forecast.
3.3 SPATIAL-TEMPORAL INCEPTION-GCM
Spatial-temporal joint graph convolution
In order to extract
correlations simultaneously. we propose spatial-temporal joint graph convolution. In this paper, construct peatial-temport joint graph convolution based on graph convolution in the spectral domain. The spectral graph convolution implemented by using the graph Fourier transform basis which is from eigenvalue decomposition of the Laplacian matrix (I.)
3

---

Under review as a conference paper at ICLR 2021
C.. d R Mrru
Sponill Cono- PCS
Sprisl-Tomperrl hecpioentacaiiin
3Gs
laceptin
STI-GM
STD-EM
Teupeal
Spesil
XeRM
(a) ASTI-GCN
(b) STI-GCM
(c) Spatial-temporal graph con- volution
Figure 2: (a) The entire architecture of ASTI-GCN. (b) spatial-temports inception graph convo- lutional module (the convolutional kernel size in the inception block can be changed according to the data). (c) A schematic diagram of spatial-temporal joint graph convolution.
to transform the graph signals from spatial into the spectral domain. But the computation cost of the
eigenvalue decomposition of I. is expensive when the graph is large. To reduce the number of pa- rameters and the computation complexity, Chebyshev polynomial Tg (x) is used for approximation.
The spectral graph convolution can be written as (Yu et al. 2017 2016
Guo et al. 2019 Kipf & Welling
Oxgr = 0
Σ TT (
where #g is graph convolution operator, θ is graph convolution kernel, x E R- is the graph signal, Tk [ E pNNN is the Chebyshev polynomial of order k with the scaled Laplacian L =
IN (L is the graph Laplacian matrix, Amar is the largest eigenvalue L, IN identity Yu et al. 2017). 6% is the coefficient of the k-th order polynomial.
Amag
matrix)
Based on spectral domain graph convolution, propose spatial-temporal joint graph convolution.
First, these K-hop Ti L are concatenated as the furthest receptive field in the spatial dimension.
Then we construct the spatial-temporal joint graph convolution kernel O.,t, Os,t E RR xCxF where K, represents the kernel size the temporal dimension and K2 represents the kernel size in the spatial dimension, C' is the input feature dimensions, F is the number of filters, So the kernel O,,: has the local spatial-temporal receptive field of K, x Ks, and K, should be lower than K which be written as Kg < K), because of the largest graph convolution perceived field of K- hop. And the mattialteemorall joint graph convolution can formulated as:
Tk (t) Coo (i). Ti (i) ,..., ,.,,,KTT (i)
Xout O.tt X TK, (i) x
(4)
where Tk i E RKXNNN is the concatenation of all Chebyshev polynomials in (K-1) hop.
the convolution operation between O,,t and X, X E RNTTXC is the spatial-temporal signal of the input graph, T is the input time steps. After the spatial-temporal joint graph convolution, the output can be written X R (x) (K-K,+1)xP Besides, the neighbors have various influences on the central node, so we implement a learnable spatial mask matrix W Waskk E Nx (Song al. (200) to adjust the graph adjacency relationship for assigning weights to different neighbors.
Spatial-temporal inception-attention
4

---

Under review as a conference paper at ICLR 2021
Different from the images, each node of the neatial-temporal graph usually represents a road eN- odeB etc. Then, affected by external factors like geographic location and surrounding environment, the spatial-temporal data properties of different nodes various, namely the heterogeneity. To solve this problem, an intuitive method to learn different models for each node, but this method could cause extensive parameters and maintain low generalization ability. So take another way in this paper. We understand that heterogeneity is manifested in the differences of local spatial- temporal receptive fields of each node of the graph which result from nodes' various properties and local spatial structures. Inspired by (Song et al., 2020) Zheng et al. 2020 et al. [0188] Vaswani al. 2017). we apply learnable graph node embedding S. E R to represent the properties of each node. Meanwhile, we introduce inception (Szegedy et al. temporal correlations through spatial-temporal joint graph convolution. Then, we combine the graph attention with inception to achieve node-level attention for modeling the heterogeneity.
2015) to extract multi-scale spatial-
Firstly, we implement inception, as shown in Figure(((b) For example, the 3x2 block represents that it involves the kernel 0,,, E [X*x2XCXX' which means it can extract the spatial-temporal correlations of the node itself and its neighbors in the three adjacent time steps by one layer, which is needed by two layers STSGCM in STSGCN (Song al.. 2020). We use the padding method when implement inception, so after B branches, the output of the inception module can be Cout R X (FFx BB) where we set the number of output filters of each branch to be the same. Then we combined with the graph node attention, which has being widely used (Vaswani et al.) 2017). We use the Q S, Wa Wa E REXF to get the queries of graph nodes. For each branch in inception, we apply the idea K
Cout.Cy RNx((FB) as the of SKNET (Li al. 2019) to do global pooling Cgt Σ
corresponding keys of the branches (one can also use W E REXF to do transform as Vaswani 2017). here omit for simplicity). Then we compute the attention by S = QK" Vaswani 2017 Take a graph node U; as an example.S,.b
al.
al.
gu, Scparish VF qu E RIxF Cg.yg
R denotes the attention of and each branch Cg,,, which represents the corresponding spatial- temporal receptive field perception. Then we concatenate the inception branch results adjusted by attention score to obtain the output. The calculation can be formulated as follows:
exp( (p,,.))
av,
(5)
exp(Smr.b)) b==1
Σ
Att. = -lkm {ameb Cout,or.b
(6)
where on,,b b R, Cout,e,b E RTxKxF, Att., E RTxKxFxB represent the output of node os from inception-attention block. Therefore, the final output of spatial-temporal inception-attentin block Cau E XNxTxXxFxB
STI-GCM output Layer
Then, we use spatial convolution generate the output of STI-GCM. We first reshape Catt into Can E XXXTXXX(F-B)) Next, the learnable weight matrix We E R x(F (F-B))(FFBB) is used to convert Can to Czau E (xFFB) We also implement SE-net (Hu et al. 2020) to model the channel attention. Finally, the output is converted to Cstatt E R xIxF using the full connection layer with W. E R(F-B)xF And the process can be formulated as Cstalt = Ca,W
3.4 FUSION OUTPUT MODULE
The iterative forecasting can achieve high accuracy in short-term forecasting, but in long term pre- diction, its accuracy would decrease rapidly due to the accumulation of errors. So, we propose the se- quence transform block named sequence decoder, to build direct relationships between historical and future multi time steps. Since different historical time steps have different effects different future time steps. we introduce temporal attention by Y, out Cstatt WFWW(WW to adjust the corresponding weights of historical temporal features. At last, in order to benefit from both, we adopt the fusion of iterative prediction and sequence transform prediction results:
Fx1 W. RtxM)
5

---

Under review as a conference paper at ICLR 2021
Y == y=g(Ymm mommsu Yooot)
where om Vs oout represent the prediction of the short-term decoder and sequence decoder pre- diction result respectively, g represents the weighted fusion function, Y the final multi-step forecasting result.
In this paper, we use the mean square error (MSE) between the predicted value and the true value as the loss function and minimize it through backpropagation.
imt+
L(O))
Σ (V,-);)² 1=t+1
(8)
where 8 represents all learnable parameters of our model, Y, is the ground truth. Y, denotes the model's prediction of all nodes at time step i.
EXPERIMENT
We evaluate ASTI-GCN on two highway datasets (PEMSD4, PEMSD8) and one telecommunica- tions dataset. The traffic datasets come from the Caltrans Performance Measurement System (PeMS) Chen et al. 200T). The network traffic dataset comes from the mobile traffic volume records Mi- lan provided by Telecom Italia (G et al. 2015).
PEMSD4: PEMSD4 comes from the Caltrans Performance Measurement System (PeMS) (Chen 2001 It refers to the traffic data in San Francisco Bay Area from January February 2018.
al.
PEMSD8: PEMSD8 comes from the Caltrans Performance Measurement System (PeMS) (Chen al. 2001 It is the traffic data in San Bernardino from July August 2016.
Mobile traffic: It contains records of mobile traffic volume over 10-minute intervals in Milan where divided into 100* 100 regions. We use the data in November 2013.
We divide the road traffic datasets at a ratio of 6:2:2 for train set, validation set and test set. For the mobile traffic dataset,
the first 20 days of data train models and the next 10 days for testing. All datasets are normalized by standard normalization method before training and renormalized be- fore evaluation. We implemented our model in Python with TensorFlow 1.14. In the experiment, use one-hour historical data to predict the data of the next hour. The hyperparameters of ASTI-GCN are determined by the performance on the validation set. The best size of temporll kernels is inception layers respectively.
set to 3 1, 1 33, 5 x,2, 3x2, 2x3, which are used in 4 We evaluate the performance of our network with three evaluation metrics: Mean Absolute Error (MAE), Root Mean Square Error (RMSE) and Mean Absolute Percentage Error (MAPE). For the road traffic datasets, if two nodes are connected, the corresponding value in the adjacency matrix is set to 1, otherwise O. For the Milan dataset, Spearman correlation coefficient was used to define the adjacency matrix. We compare ASTI-GCN with other widely used forecasting models, includ- ing HA, ARIMA (Williams & Hoel 2003), LSTM (Hochreiter & Schmidhuberi 1997). DCRNN, STGCN, ASTGN, STSGCN, AGCRN. See Appendix A for more detail.
4.1 BASELINES
HA: Historical Average method. We use the average of the traffic over last hour to predict the next time slice.
ARIMA (Wiliaas & Hoel 2003): ARIMA is well-known model for time series prediction.
LSTM (Hochreiter & Schmidhuber 1997): : LSTM can extract long-term and short-term temporal dependency and is widely used in time semis forecasting.
DCRNN (Li et al., 2017): A deep learning framework for traffic forecasting that incorporates both spatial and temporal dependency in the traffic flow.

---

Under review as a conference paper at ICLR 2021
STGCN (Yu et al. 2017): STGCN uses spectral GCN and CNN to capture spatial and temporal dependencies respectively.
ASTGCN introduces a temporal and spatial attention mechanism to capture dynamic temporal and spatial correlations.
ASTGN (Guo et al., 2019):
STSGCN (Song al.
2020
STSGCN purely uses GCN to extract spatial-tempon information
simultaneously.
AGCRN (Bai et al. 2020 AGCRN uses adaptive graph convolutional recurrent network to capture node information.
4.2 COMPARISON AND RESULT ANALYSIS
Table shows the overall prediction results including the average MAE, RMSE and MAPE of our proposed method and baselines. Due the huge computation cost, it is hard measure the performance of STSGCN on the Milan dataset. Compared with the models that can model spa- tial correlation, other models that only model temporal correlation (Historical Average method, ARIMA, LSTM) have poor performance in the three datasets. This is because such models ig- nore the spatial influence between nodes and only use the historical information of a single node. Among spatial-tempont cept MAPE of PEMSD4 and MILAN dataset. Because STGCN, ASTGCN and DCRNN did not consider the heterogeneity in the spatial-temporal data, while STSGCN only considered the local spatial-temporal heterogeneity information extraction, model performed better. Besides, find AGCRN performs well on road datasets, which is close to our results, but has poor performance on the mobile traffic dataset. We conjecture that the reason is large number of nodes and the large difference in node distribution in MILAN dataset. Mcanwhile, it further indicates that ASTI-GCN has a stronger generalization ability than AGCRN.
models, ASTI-GCN achieves the best performance in all indicators ex-
three adjacent time steps. which was insufficient for the global
Table 1: Performance comparison of different approaches(The The best results are bold, the second best results are underlined).
Dataset
PeMSD4
PeMSD8
Milan
Model
MAE
RMSE
MAPE
MAE
RMSE
MAPE
MAE
RMSE
MAPE
HA
61.8
87.27
51.16
31.94
46.34
20.51
61.28
120.73
49.8
ARIMA
32.11
68.13
28.98
24.04
43.3
26.32
54.61
78.94
49.86
LSTM
29.04
45.63
40.71
25.93
38.64
20.77
42.93
80.29
56.33
STGCN
22.15
34.68
14.54
18.49
28.18
11.63
34.77
63.28
75.99
ASTGCN
22.99
35.42
15.75
18.89
28.64
12.64
39.42
72.04
38.17
DCRNN
22.23
35.25
14.78
16.45
25.41
10.73
33.9
65.52
17.01
STSGCN
21.29
33.81
13.87
17.18
26.85
10.91
AGCRN
19.68
32.42
12.89
15.95
25.22
10.09
42.27
135,18
100.37
Ours 19.45
31.57
13.09
15.53
24.6
9.71
29.84
57.73
20.23
Figure 3 shows the changes of different metrics on the three datasets with the increase of predicted time steps. As we can see from the Figure 3,, the prediction error increases over time, indicating that the prediction becomes more difficult. Those methods that only consider temporal correlation (ARIMA, HA, LSTM) perform well in short-term forecasting, but as the time interval increases their performance deteriorates sharply. The performance of GCN-based methods is relatively stable, which shows the effectiveness of capturing spatial information. Although our model has no out- standing performance in short-term forecasting tasks, it shows the best performance in medium and long-term forecasting tasks. This benefits from our spatial-tempont inception-GCN module which reduces the accumulation of emors significantly.

---

Under review as a conference paper at ICLR 2021
CCho
N42 on PEN1004
on
on
Timestpp
Timestep
Timrstep
D750
2529
5002E on PEPSOC
RMSSE on Milan
130
1004
SIACE on
Figure 3: Prediction performance comparison each time step.
4.3 COMPONENT ANALYSIS
In order to prove the effectiveness of each key component, we carried out component analysis ex- periments PEMSD8 dataset. The basic information of each model is follows:
Basic model: the model consists of two blocks. In each block, spatial and temporal convolution performed respectively without the multi-scale spatial-temporal joint graph convolutions and graph attention. does not use the sequence decoder with temporal attention as well. The output prediction result of the next time step simply layers to generate.
convolutional layer and the fully connected
STI-GCM: Based on the basic model, multi-scale spatial temporal joint graph convolutions used to replace the separate spatial and temporal convolutions to extract spatial-temporal correla- tions. In addition, the graph attention is introduced to capture the heterogeneity.
+ Mask: We equip the basic model with the Mask matrix to learn the influence weights of different neighbors on the central node adaptively.
+ Sequence decoder with attention: We add to the basic model with the proposed sequence decoder and the temporal attention for multi-step forecasting.
+ alternant sequence decoder and short-term decoder: The output layer is modified into the fusion of the sequence forecasting result and short-term decoder forecasting result to benefit from both of them.
Except for the different control variables, the settings of each experiment were the same, and each experiment was repeated 10 times. The results which are shown in Table2 that the model us- ing STI-GCM has better performance, because the network can adaptively capture the heterogeneity of spatial-temporal network data. The sequence decoder with temporal attention shows good per- formance. The Mask mechanism also contributes some improvements. In the process of fusion, we train the short-temn decoder and sequence decoder model circularly, and fuse the results of the two models as the final prediction output. It can not only ensure the short-term forecasting accuracy, but also avoid the influence of error accumulation the long-term forecasting accuracy.
8

---

Under review as a conference paper at ICLR 2021
Table 2: Component Analysis
Methods
MAPE
MAE
RMSE
basic
11.25
18.69
28.32
+STI-GCM
10.64
17.36
26.96
+ Mask
10.46
17.51
28.12
+ Sequence decoder with attention
10.53
16.76
26.27
alternant sequence decoderand short-term decoder
10.59
17.00
26.42
ASTI-GCN
9.71
15.53
24.60
CONCLUSION
This paper introduces new deep learning framework for multi-step We propose spatial-temporal joint convolution to directly capture spatial-tempon correlation. At the same time, employ the inception mechanisms extract multi-scale spatial-temporal correla- tion, and introduce graph attention to model graph heterogeneity. Moreover, we combine short-term decoding and sequence decoding to map historical data future time steps directly, avoiding the accumulation of errors. We evaluate our method on real-world datasets from 2 different fields, and the results show that our method is superior other baselines with good generalization ability.
forecasting.
REFERENCES
Lei Bai, Lina Yao, Salil Kanhere, Xianzhi Wang, Quan Sheng, et al. Stg2seq: Spatial- temporal graph to sequence model for multi-step passenger demand forecasting. arXiv preprint 1000 2019.
Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. 2020.
Chao Chen, Karl Petty, Alexander Skabardonis, Pravin Varaiya, and Zhanfeng Jia. Freeway perfor- mance measurement system: Mining loop detector data. Transportation Research Record Journal of the Transportation Research Board, 1144:96--02, 2001.
Barlacchi G, Nadai M, D, Larcher R, Casella A, Chitic C, Torrisi G, Antonelli F, Vespignani A, Pentland A, and Lepri B. multi-source dataset of urban life the city of milan and the province of trentino. Scientific Data, 2, 2015.
S Gowrishankar and S. Satyanarayana. A time series modeling and prediction of wireless network traffic. International Journal of Interactive Mobile Technologies (iJMM), 3(1), 2009.
Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based spatial- temporal graph convolutional networks for traffic flow forecasting. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 922-929, 2019.
Kaiwen He, Yufen Huang, Xu Chen, Zhi Zhou, and Shuai Yu. Graph attention network for deep learning based mobile traffic prediction. In 2019 IEEE Global Communications Conference (GLOBECOM), pp. 1-6. IEEE, 2019.
Hochreiter and Schmidhuber. Long short-term memory. Neural Computation, 18) 1997.
Jie Hu, Shen, Samuel Albanie, Gang Sun, and Enhua Wu. Squeeze-and-exiinn networks. IEEE Trans. Pattern Anal. Mach. Intell. 42(8):2011 1-2333, 2020. doi: 10.11099/AMM.0119 2913372. URLptttt. //oi org 1109 TPAMT 2019. 2913372
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: 1609.0290, 2016.

---

Under review as a conference paper at ICLR 2021
Xiang Li. Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. 2019.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural net-
work: Data-driven traffic forecasting. arXiv preprint urXiv:/707.0192, 2017.
Yuxuan Liang, Songyu Ke, Junbo Zhang, Xiuwen Yi, and Yu Zheng. Geoman: Multi-level attention networks for geo-sensory time series prediction. In IJCAI, pp. 3428-3434, 2018.
Chao Song, Youfang Lin, Shengnan Guo, and Huaiyu Wan. Spatial-tempord synchronous graph convolutional networks: A new framework for spatial-temporal network data forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 914-921, 2020.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 1-9. IEEE Computer Society, 2015. doi: 10. (0.1109/CVP22:211572999999 URL https: www.dooo.oooooo 20t/:/...000..111111///0....121. 729579959444
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv, 2017.
Billy M Williams and Lester A Hoel. Modeling and forecasting vehicular traffic flow as a seasonal arima process: Theoretical basis and empirical results. Journal of transportation engineering, 129(6):664-672, 2003.
SHI Xingjian, Zhourong Chen, Hao Wang, Dit- Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. In Ad- vances in neural information processing systems, pp. 802-810, 2015.
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. arXiv preprint arXiv:1709.04875, 2017.
Chuanpan Zheng, Xiaoliang Fan, Cheng Wang, and Jianzhong Qi. Gman: A graph multi multi-attention network for traffic prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 2234-1241, 2020.
Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. arXiv preprint arXiv:1812.084334, 2018.
A APPENDIX
A.I DATASETS
We evaluate ASTI-GCN on two highway datasets (PEMSD4, PEMSD8) and one telecommunica- tions dataset. PEMSD4 refers to the traffic data San Francisco Bay Area from January to February 2018. PEMSD8 is the traffic data in San Bernardino from July August 2016. The telecommu- nications dataset contains records of mobile traffic volume over 10-minute intervals in Milan. We summarize the statistics of the datasets in Table 3
Table 3: Dataset description
Datasets
Samples
Nodes
Timeslot
Input Length
Output Length
PEMSD4
16992
307
5min
12
12
PEMSD8
17856
170
5min
12
12
Mobile traffic
4320
900
10min
6
10

---

Under review as a conference paper at ICLR 2021
A.2 ADJACENCY MATRIX DEFINITION
For the road traffic datasets, if two nodes are connected, the corresponding value in the adjacency matrix is to otherwise 0. The spatial adjacency matrix be expressed
A15 = {
1, if connects to eyy
(9)
0, otherwise
where A is the edge weight of node and node j, v is node i.
For the Milan dataset, Spearman correlation coefficient was used to define the adjacency matrix:
1, i and Spearman (for, kny,
Ac. =
(10)
otherwise
where be and byy are time series training data of node and node j. respectively. Spearman() non-parametric indicator that measure the correlation between the two time series, å is the threshold that controls the distribution and sparsity of the matrix, which is set to 0.92 in this article.
A.3 METRICS
We evaluate the performance of our network with three evaluation metrics: Mean Absolute Error (MAE), Root Mean Square Error (RMSE) and Mean Absolute Percentage Error (MAPE).
MAE
10
(11)
100%
2/1
MAPF
(12)
m
V1
1
RMSE
(U= y/ )²
(13)
where ú; and ys represent the predicted value and the ground truth respectively. m is the total number of predicted values.


Reviews:
[{"review_id": "RkZe6guezlD", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a GCN-based solution for multi-step spatio-temporal data forecasting. The main contributions are 1) a spatio-temporal joint graph convolution network to simultaneously capture spatial and temporal correlations; 2) a combination of the sequence decoder and short-term decoder to alleviate the error accumulation when modeling the temporal dependencies. I agree with the incremental contribution of this paper. To verify the effectiveness of their approach, the authors conduct experiments on three real-world datasets.\n\nHowever, I have the following concerns:\n\nPresentation: \nThe major concern is the writing of this paper. There are many parts (e.g., notations, statements) that are unreadable or hard to understand. For example, \n1) What is “ST in GCM” in Figure 2(a)?\n2) In page 4, x \\in \\mathbb{R}^n. What is n here? It seems to be inconsistent with the notation N in Section 3.1.\n3) This paper introduces a learnable spatial mask matrix W_{mask}, but how to use it is not clear.\n4) Does Figure2(c) in page 5 mean Figure 2(b)?\n5) In the last equation of page 5 (for computing Y_{s_out}), what is t? Moreover, since W_F is of shape F by 1 and W_t is t by M, how can we multiply W_F by W_t?\n6) How to obtain Y_{m_out} in equation 7?\n7) Many typos and grammatical errors. I only point out several of them. Spatial-temporal data is -> are, correlations STG2Seq -> correlations. STG2Seq, Laplqaacian->Laplacian, which needed -> which is needed, we conjecture the reason is-> we conjecture that….\n\nTechnical:\n1) Novelty of the proposed model is limited. The proposed spatio-temporal joint graph convolution is very similar to the concept of 3D GCN [1]. However, in this paper, I could not see any discussion about the difference and comparison between them.\n2) In section 3.1, this paper assumes that the graph is undirected. However, the propagation of traffic in spatio-temporal domains are certainly directed (e.g., downstream, upstream). \n3) What is the difference between the temporal attention in Section 3.4 and an FFN (two fully-connected (FC) layers)? In my view, this paper only uses two matrices for the dimension transformation, which is the same as two FC layers.\n\nExperiments:\n1) It is more reasonable to compare the proposed model with CNN-based solutions in the Mobile traffic dataset, as there is no explicit graph structure (only Euclidean structure exists). Baselines like ST-ResNet, ConvLSTM, DeepSTN+ or DeepLGR should be included for comparison.\n2) According to the paper, I cannot see any detail of the multi-run experiments (e.g., in Table 1 and Figure 3). How about the stability (i.e., variance) of the proposed method?\n3) The experiment shows little improvement to the existing approaches in the first two datasets. There should be significant test (e.g., student t-test) conducted to make sure the experimental result is reliable. Considering the limited improvement, it is also questionable whether adding such complexity to the model is worthy. It may not make sense to improve the MAE by 0.02 in the first dataset by increasing the runtime a lot.\n4) No experiments to show the effects of the spatio-temporal kernels. Why should we set it 3x1, 1x3 and so on?\n5) It seems that the MAPE in Table 1 largely exceeds 100%, which is unusual to see. What is the unit of MAPE here?\n\nMinor issues:\n1) All acronyms should be expanded for the first time in text (e.g., HA is not expanded. What is HA? Historical average?).\n2) There should be one space before each citation. For example, DCRNN(Li et al., 2017) -> DCRNN (Li et al., 2017).\n3) In the first paragraph of Section 1, the paper claims that “Typical applications include …, traffic road condition forecast (…, Liang et al., 2018), … and geo-sensory time series prediction (Li et al., 2017).” However, Liang et al., 2018 is for geo-sensory time series modeling while Li et al., 2017 is for traffic prediction. Please carefully read these papers before citing them.\n4) It is difficult for those who are not familiar with the Inception Network to understand how your model draws insights from it. In addition, the paper proposing the Inception Network should be cited.\n5) No future work discussed in this paper.\n\nIn summary, I recommend not to accept this paper in its present form.\n\nReference:\n[1] Yu et al., 3D Graph Convolutional Networks with Temporal Graphs: A Spatial Information\nFree Framework For Traffic Forecasting, Arxiv 2019.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "R39PO9M5uLN", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors propose ASTI-GCN to solve the multi-step spatial-temporal data forecasting problem. The model uses a convolutional block to model the spatial-temporal correlations and an inception attention based module to capture the graph heterogeneity. They evaluate the proposed method on three different traffic prediction datasets.\n\n\nPros:\n1. The problem of traffic prediction is important.\n\nCons:\n1. The contributions are limited in this paper. The ideas of jointly model spatial-temporal information via convolutional layer (see 3D GNN to model irregular regions [1] and 3D CNN to model regular regions [2]), and multi-scale spatial-temporal modeling (see [3, 4]) are not new. These papers have been released for more than one year. Especially, the multi-scale motivations in [3] [4] are almost the same in the paper. The only difference is that this paper involves attention to weight different scales, which is also a very common practice. Thus, I think the contributions are not enough to be accepted by top machine learning conference.\n\n2. The experiments could be improved.\n\n- Besides the ablation studies in this paper (add modules to the base model), it would be more convincing to add the ablation studies by removing some components. Since only combining one module (results in Table 2) performs worse than AGCRN, removing some components and keep the rest could provide deeper analysis. In addition, it would be more convincing to conduct all the ablation studies on all three datasets.\n\n- To support the claim of region heterogeneity, it would be more interesting to show some case studies to verify the motivation and see the reasons for the improvement. Otherwise, the improvement may come from the increasing of the number of parameters.\n\n- It would be better to show the error bar for each result since the improvement in some datasets is limited (e.g., PeMS04). \n\n3. Some figures could be improved. For example, some arrows in Figure 2(b) are broken.\n\n[1] Yu, Bing, et al. \"3d graph convolutional networks with temporal graphs: A spatial information free framework for traffic forecasting.\" arXiv preprint arXiv:1903.00919 (2019).\n\n[2] Chen, Cen, et al. \"Exploiting spatio-temporal correlations with multiple 3d convolutional neural networks for citywide vehicle flow prediction.\" 2018 IEEE international conference on data mining (ICDM). IEEE, 2018.\n\n[3] Geng, Xu, et al. \"Spatiotemporal multi-graph convolution network for ride-hailing demand forecasting.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019.\n\n[4] Cui, Zhiyong, et al. \"Traffic graph convolutional recurrent neural network: A deep learning framework for network-scale traffic learning and forecasting.\" IEEE Transactions on Intelligent Transportation Systems (2019).", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q1yRKAcIE4R", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the important problem of spatial-temporal forecasting, and proposes a multi-scale spatial-temporal joint graph convolution that jointly model the heterogeneous spatial-temporal correlations. Empirical results on multiple real-world datasets shows promising results. \n\nThe draft seems to be written in a rush, with mistake/typos in paragraphs, figures and tables. Besides, the some of the major claims are not well supported by the empirical results. Here are details about the main concerns:\n\nD1: Some major claims are not well supported by the experimental results.\n- The main novelty of this paper the is  the spatial temporal joint convolution. Yet, its design needs more theoretical and empirically justifications.  The idea of the concatenating K order of Laplacian matrix is a bit ad-hoc lacking of theoretical justification.  Besides, the spatial temporal joint graph convolution is not necessary \"more joint\" than baseline algorithm like DCRNN which incorporates the graph convolution operation into the each sub-step of the RNN operation.  \n- Besides, the claimed convolution in the spectral domain is actually operations in the spatial domain since the final form of the convolution is essentially a polynomial of the Laplacian matrix without involved the transformation into the spectral domain. \n- Limited ablation studies are conducted to show its effectiveness. There are some ablation study in Table 2, but not enough to well support the claims.  For example, what is performance gain by simplify replacing the graph convolution with the proposed spatial-temporal joint graph convolution? Is inception style convolution is actually needed? What is the different with and without it? \n- Besides, as shown in Table 2, even after the adding the spatiotemporal convolution and the inception attention mechanism, the performance is still worse than some baselines, including DCRNN and AGCRN. Potentially it is also possible that the mask (which can be applied to other baselines) actually contributes to the major improvement.\n\nD2: Presentation. This paper seems to be written in a rush, and the presentation need more polish. \n\n- The technical part is a bit hard to follow. For example, when explaining the spatial-temporal joint convolution, it might be easier to understand if the author provides an additional figure about it, e.g., a 2-D matrix with size T x K, with filters of size K_t and K_s applied on it, where graph convolution is used for the feature extraction.\n- Inconsistent name of the proposed model. The name of the proposed model is not consistent, for example it is called ASTI-GCN in abstract, while in Figure 2a it is called ATI-GCN. Yet, in Section 4 is is call ASTIGCN, and in Table 2 it is also called ours.\n- Errors/typos in table and paragraphs. For example, in Table 1 the RMAE should RMSE. The Laplqaacian in Section 3.2 should be Laplacian. - In Equation (8), we may rewrite \\hat{Y}_i as \\hat{Y}(\\Theta)_i so the loss is a function of the parameter \\Theta.  In Section 4.2, we may Capitalize the first letter of alternative to make it consistent with the other three. Also remove the duplicate periods at the end of the sentence. \n- In Equation (2), the T_k(L) probably represents the order k instead of order k-1 otherwise T_0(L) will become order -1.\n- The datasets use, i.e., PeMS04, PeMS08, seems to be PeMSD4 and PeMSD8, we may keep it consistent with previous methods to facilitate comparison of metrics in baseline papers.\n-  In the Table 2, it is mentioned that each setting has been run for 10 time, and it is helpful to also include the standard deviation to show the statistical significance of the improvement\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AUXeLRlFtsz", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a spatial-temporal graph neural network, which is designed to adaptively capture the complex spatial-temporal dependency. Further, the authors design a spatial-temporal attention module, which aims to capture multi-scale correlations. For multi-step prediction instead of one-step prediction, they further propose the sequence transform block to solve the problem of error accumulations. The authors conducted experiments on three real-world datasets (traffic on highways and mobile traffic), which shows their method achieves the best performance.\n\nOverall, I think the problem of capturing spatial-temporal dependency shown in Figure 1 is interesting. But the description of the method is unclear and the writing of this paper still needs improvements. Furthermore, I didn't find enough novelty that meets the standard of ICLR. \n\nConcerns: \n\nMethods: \n\nI don't understand why the authors introduce spectral graph convolution networks while it seems that the authors only use GCN in the spatial domain. \n\nThe equation (4) is confusing: $X$ in the left changes to be $T_K(L)X$ without more explanations. \n\nIt's unclear about how to use the mask matrix $W_{mask}$ in the proposed model.\n\nThe equation $C_{gl} = \\sum_{i=1}^T \\sum_{j=1}^K C_{out}$ is confusing. \n\nThere are many method descriptions like the above examples, which the authors should describe more clearly.  \n\nTypos: \n\nPage 3 Last Line: Laplqaacian\n\nPage 5 Paragraph 2 Line 12: node $v_i$as\n\nPage 8 Line 4: performed respectivelywithout\n\nPage 8 Line 17: both of them.. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a multi-scale spatial-temporal joint graph convolution for spatiotemporal forecastings. Many reviewers have concerns regarding novelty, baseline comparisons, and writing clarity of the draft.
Decision: Reject