AutoST: Towards the Universal Modeling of Spatio-temporal Sequences
Authors: Jianxin Li
Shuai Zhang
Hui Xiong
Haoyi Zhou
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neural Network
Spatio-temporal Forecasting
Architecture Search

Abstract: The analysis of spatio-temporal sequences plays an important role in many real-world applications, demanding a high model capacity to capture the interdependence among spatial and temporal dimensions. Previous studies provided separated network design in three categories: spatial first, temporal first, and spatio-temporal synchronous. However, the manually-designed heterogeneous models can hardly meet the spatio-temporal dependency capturing priority for various tasks. To address this, we proposed a universal modeling framework with three distinctive characteristics: (i) Attention-based network backbone, including S2T Layer (spatial first), T2S Layer (temporal first), and STS Layer (spatio-temporal synchronous). (ii) The universal modeling framework, named UniST, with a unified architecture that enables flexible modeling priorities with the proposed three different modules. (iii) An automatic search strategy, named AutoST, automatically searches the optimal spatio-temporal modeling priority by network architecture search. Extensive experiments on five real-world datasets demonstrate that UniST with any single type of our three proposed modules can achieve state-of-the-art performance. Furthermore, AutoST can achieve overwhelming performance with UniST.

AutoST: Towards the Universal Modeling of Spatio-temporal Sequences
Jianxin Li* BDBC Beihang University Beijing, China 100191 1 jxxbuaa. edu. cn
Shuai Zhang BDBC Beihang University Beijing, China 100191 zhangs@act, buaa. edu. cn
Hui Xiong HKUST(GZZ)
Haoyi Zhou BDBC Beihang University Beijing, China 100191 hacyi@buaa (euaaa edu. cn
HKUST FYTRIS Guangzhou, China 511455 xionghu @ust.uuuuuu.........
Abstract
The analysis of spatio-temporal sequences plays an important role in many real- world applications, demanding a high model capacity to capture the interdepen- dence among spatial and temporal dimensions. Previous studies provided separated network design in three categories: spatial first, temporal first, and spatio-temporal synchronous. However, the manually-designee heterogeneous models can hardly meet the
dependency capturing priority for various tasks. To address this, we proposed a universal modeling framework with three distinctive characteristics: (i) Attention-baser network backbone, including S2T Layer (spatial first), T2S Layer (temporal first), and STS Layer (spatio-temporal synchronous). (ii) The universal modeling framework, named UniST, with unified architecture that enables flexible modeling priorities with the proposed three different mod- ules. (iii) An automatic search strategy, named AutoST, automatically searches the optimal apatio-tempont modeling priority by network architecture search. Ex- tensive experiments five real-world datasets demonstrate that UniST with any single type of our three proposed modules can achieve ofate-of-the-ar performance. Furthermore, AutoST can achieve overwhelming performance with UniST.
Introduction
Modeling and predicting the future of spatio-temporal (ST) sequences based on past observations has been extensively studied and has been successfully applied in many fields, such as road traffic [3]
medical diagnosis 2 and meteorological research 24. Traditional statistical methods typically require input sequence satisfying certain assumptions, which limits its ability in capturing the complex spatial-temporal dependency. Then, recurrent neural network (RNN) methods [15] leverage the universal approximation property to build separated network branches to model dependency and make predictions with fusion gate blocks from the stacking branches. The intrinsic gradient flow in the nack-propagation training process [ may bring the ST dependency into incorrespondence with the
"Jainxin Li is the corresponding author.
*BDBC: Beijing Advanced Innovation Center for Big Data and Brain Computing. HHKUST(ZZ): Hong Kong University of Science and Technology (Guangzhou). $ HKKSS FYTRI: Guangzhou HKUST Fok Ying Tung Research Institute.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Idiaal
Sputial-Frst
Temporal fins
Spatio tempurai synchronous
Ours
tonk
Stuah bak
task
task
task
task
task
task
Syunal
Sperial
Spassil
pauds
tok
Tempreal
Tempoeal
Temporil
Temporai
Tempani
(a)
(b)
(c)
(d)
(c)
Figure Modeling orders ST data. The stars are tasks with different spatial-temporal dependencies. The red lines going darknesss/ightness refers to the modeling ability of the model along with the current modeling tendency.
modeling procedure with anisotropic tendency. The red lines' color
network branches' configuration, especially for the deeper network [[]. Recently, the Transformer-based models show larger model- ing capacity in both spatial and temporal mod- eling [2] 32 31, which motivates us find
takk
tisk
Spatil
task
Time
Tanponi
universal way to capture the ST dependency in universal framework simultaneously.
(a)
(b)
Figure 1: (a) Dependencies in ST sequence data: Green arrows are spatial dependencies; Red arrows are temporal dependencies; Blue arrows are depen- dencies across both spatial and temporal. (b) Task 1 has high spatial dependency but low temporal dependency; Task has high temporal dependency but low spatial dependency; Task 2 has equal spa- tial and temporal dependencies.
As shown in Fig [ the ST dependency indi- vidually exists in sequences: spatial correspon- dences, temporal correspondences, and spatio- temporal correspondences. Take the road traffic forecasting as an example, previous research fall into three typical paradigms. (a) Spatial-first modeling [1] 27 33: Predicting the traffic for next road junction, which has strong connec-
tions to previous intersections, stops, and surrounding traffic. (b) Temporal- cimporal--frstt first modeling 23 5
Predicting the traffic of high school on Friday afternoon, which shows strong periodic relationships with the school's times schedule. (c) Spatio-temporal synchronous modeling [19
12
28
Analyzing the city traffic, which tightly entangled both spatial and temporal. During the sequences modeling, the main problem is to align the network design with the natural apatio-tempora distribution.
However, the distribution of ST dependencies varies and depends on the forecasting task and corre- sponding datasets. They are mixed in a compound way when modeling ST sequences, and the three tasks in Fig (bb) are the representative ones. What makes it worse is that, the prevalent modeling methods show anisotropic tendency to capture the ST dependency. If use the spatial-first models on the three tasks in Fig (b), the task I's states are highly influenced by the surrounding information, and the periodic pattern the underlying factors, which makes the spatial-fis -aattt first model fits it properly. We can compare the model ability (red lines) with the ideal one in Fig Z(a, this kind of model will be insufficient for task 2 and task 3. Similarly, suppose we use the temporal- first models on the three tasks in Fig c). In that case, the model ability only matches task 3, where the periodic pattern decides the states other than the spatial information. The previous analysis also applies to the spatio-temporal synchronous situation, where the states are mainly influenced by the complex associations across the spatial and temporal, like semantic relationships. this paper, we aim to propose universal model that alleviates the the modeling gap on different tasks.
The contributions are: The first to raise and address the modeling order proplem in spatio-temperal forecasting tasks by proposing a universal modeling framework UniST and an automatic structure search strategy AutoST. 2) Proposing replacable and unified aitention-based modeling units named S2T, T2S and STS, which model opatio-timo sequence with three different priorities: spatial first, temperal first and spatio-temperal synchronous. 3) Extensive experiments on 5 datasets and sequence forcasting tasks demonstrate that only using our three modeling units (S2T, T2S, and STS) outperforms the baseline methods, and our framework together with AutoST achieves the new Mlate-ofthhe-s performance.

---

2
Related Work
Existing spatio-temport forecasting methods can be roughly grouped into three categories: spatial- first, temporal-first, and spatio-temporal synchronous methods. Spatial-first: STG2Seq [ uses stacking GCN layers to capture the entire inputs sequence, where each GCN layer operates on limited historical time window, and the final results are concatenated together to make forecasting. In the view of this paper, it belongs to spatial-first modeling. STGCN [27] propose the blocks that contains two temporal gated convolution layers with one spatial graph convolution layer in the middle, which starts from a convolution based temporal layer. DCRNN 133 is proposed to forecast traffic flow using diffusion convolution and recurrent units to capture spatial and temporal information successively. Temporal-first Graph WaveNet (33 built the basic modeling layers with two gated temporal convolution modules the beginning and followed by a graph convolution module, which models from temporal to spatial. GSTNet [ builds several layers of spatial-tempon blocks to produce the forecasting, which is consists of multi-resolution temporal module followed by global correlated spatial module. Spatio-temporal synchronous: STSGCN 19 construct a spatio- temporal synchronous extraction module composed of graph convolutional networks. STFGNN 12] modeling apatio-tempone correlations simultaneously by fusing a dilated convolutional neural network with a gating mechanism and spatio-temporal fusion graph module. ST-ResNet [2X] using convolution on sequence of image-like 2D matrices to model spatio- temporal at the same time. ASTGCN G proposed spatial-temporal convolution that simultaneously captures the spatial patterns and temporal features.
3 Preliminary
3.1 Spatio-temporal Sequence Forecasting
Spatio-temporal sequence forecasting (STSF) to predict the future sequence of spatio-temporal inputs based on the historical observations. Specifically, given a graph G = (V, E, A), where V and E are the node set and edge set, and N the number of nodes, E R N N is the adjacency matrix of G. If es 12 E V and (v,, vU)) E, Aij sequence of 7 time steps, where X E TTNNCC The snapshot at time step f is denoted as x, E NxC where is the feature dimension of a node. Then the ST sequence forecasting problem can be defined as: given S time steps historical observations of input graph G, the goal is to predict the future sequence of the features on each node with a learning function f: [X(t-):tt ((t-S)):+ G] 4 ((++))((++))
otherwise A.j O. X (X,, X Xr is ST
where (t-S)) S):t and ((++))((++)) are the ST sequence with length S and P respectively.
3.2 Network Architecture Search
Network (neural) architecture search (NAS) are automated methods for generating and optimizing neural networks. A representative gradient-basee approach is DARTS [[]], which is the foundation of our proposed training framework. DARTS aims to search optimal directed edge connections directed acyclic graph with predefined computing cells as nodes. The result connections of node j is denoted as 2 Eies o(h-) (zx)),, where (f-3) is an operator, e.g. layers model, represented by a directed edge from node to node j. DARTS proposes a method to relax the discrete searching space to be continuous, and uses bi-level optimization to learn a differentiable objective on the joint optimization problem of both network architecture and model weights. The objective function is: min, Lyat (w" (a:), a)) s.t. w=
argmin,, Ltrain (w, a.), where is the architecture, and w is the model weights. In Section 4.4 we improve the design of the directed acyclic graph of the search architecture, and the two-stage optimization of the architecture parameters.
4 Methods
In this section, we firstly introduce two basic modeling units: the time series linear celf-attention. and the high order mix graph convolution. Then we proposed three layers as different network backbones, and we build universal modeling framework based on the tree "atomic" layers. Next, we propose an automatic searching strategy for spatio-temporal information fusion, which aimed for the optimal order of spatio-temporal modeling on various downstream tasks.
3

---

S2T Layer
T2S Layer
STS Layer
XL
X
X
& Noem
Food Furvarni
Add
Sele-Amention
Highlnder MaGG
Lisear Self-Amettion
Highhnndrr Macic
Lieaar Self Attention
MALGC
K
XLa
XLA
XL1
Figure 3: Three ST modeling modules different modeling orders. (a) S2T Layer: firstly model spatial information, then combine with temporal information within self-attention mechanism; (b) T2S Layer: firstly model temporal information, then use the high-order graph convolution to capture the complex dependency; (c) STS Layer: model the spatial and temporal information simultaneously, and concatenate the two feature map as the final representation.
4.1 Spatial Temporal Modeling Unit
4.1.1 Time Series Linear Self-Attention
Self-attention mechanism 21 has been widely used nature language processing, computer vision, and time series forecasting, which is defined as: Attention( K, V) V
Softmax QK /V/V,, where Q XWQ.K- K XWx, V =XWv, and the projection ma- trix We E RCOD W E RCDD Wy ERCD,
However, the original self-attention suffers from high computational and memory cost. Because the dot product computation of Q and K leads to O time and space complexity. (9] proposed linear self-attention, which represents the similarity function of Q and K in the self-attention by a kernel function: V = Q.) Σ (K,) V (Q,) Σ ø (K)). Such that for each query Qi,
the two terms Σソ V and ΣX- (K,) are the same and reused for efficient computing. Following B1I we use the technique of linear self-attention in representing time series features.
4.1.2 High-order Mix Graph Convolution
To acquire better spatial information representation, we propose a high-order mix graph convolutional operation for spatial information mixing and feature extraction of the original inputs, it is defined as:
HighOrder(X, A, order)
X
if order =
def
Horder
Mix A) MixGC(H () (o
if order
if order 1,
where order denotes the total order of the graph convolution operations, i.e., to consider order- hop neighbor relationship of each node. In this paper, we define the 1 mix convolutional operation by combining the 1 order ChebNet [0] and the Adaptive Diffusion Convolution [23 Mix A) ChebNet X, A) + AdapDC(X, A) Áxw, g P P,XW, + P,XW, À Aadp W adp, where À = D 1/2 AD-1// is normalized adjacency matrix with self-loop. ChebNet focuses on Ist-order neighbor information, while AdapDC focuses on multi-hop information. Á A defined as À A I, where D Aij. an identity matrix. P, rowaum(A)
Pb rowsum AT) refers to a forward and backward state transition matrix, respectively. Audp is an adaptive matrix for complementary spatial state information, which is calculated by two learnable node embedding matrices E,, Eg NxC 20 Audp Softmax (ReiiiaaaReeLLeEBLL1eee

---

4.2 Unified Spatio-temporal Modeling Backbone
In order to solve the problem of spatio-temporal dependency distribution differences in the modeling procedure, we first propose three novel modules: S2T Layer, T2S Layer, STS Layer, that suitable for three typical spatio-temporal dependencies: spatial-first, temporal-fing -irss,, spatio-temporal synchronous, respectively. We design all these three modeling module to have the same dimension of inputs and outputs. This provides a solid foundation for our later flexible and universal modeling.
4.2.1 Spatial-first Modeling Layer
The spatial-first sequence modeling method, S2T Layer, models from spatial to temporal. The spatial information between the nodes on the graph first characterized on single slice. After that, node information at different times is exchanged along the time dimension, whose spatial information has been shared with its neighbors.
As shown in Fig D a), S2T Layer first uses two high-order mix graph convolution defined in Eq. I to process the input spatio- temporal sequences X to obtain two sequence representations with mixed spatial information. Then the key K and value V of the input of the subsequent self-attention are obtained by transformation using the parameter matrix W K, Wv, respectively, while the query Q is obtained by transforming the original input ST sequence using the parameter matrix as follows: Q =
X W Q, K HighOrder, (X1-1, A, order) W K, V HighOrder, (X1-1, A, order) Wv.
Then the original ST sequence and the new sequence with mixed spatial information are processed using a multi-head linear self-attention, from which it learns temporal dependencies and exchanges information at different time slices to obtain further representations of the ST sequence:
Z Attention(4)) Q K, V)
(2)
The output is concatenated with the initial input once for residuals and processed a layer normalization, followed by two-layer fully connected network for further ST representation learning. This network is applied separately and identically to each point-in-time position in the ST sequence, thus maintaining the continuous transfer of position-encodee information. Finally, the resulting ST sequence representation is again connected to the initial input with one residual and layer normalization obtain the output X = Norm(max Norm (Z + X + by) + b2).
4.2.2 Temporal-firs: Modeling Layer
This module is designed model the ST sequence from temporal spatial, named 72S Layer. Different from spatial-first modeling, at the beginning. the original inputs are projected into Q K, V by three weight matrices as Q X WQ. K X W K. and V = X Wy. The projection results are used to calculate temporal representations at first, using the time series linear self-attention in Eq.Q) Then the temporal representation on each node are send the high-order mix graph convolution, together with the adjacency matrix, to fusion the temporal information from every neighbors. Z' HighOrder(Z,, A, drder). .. Finally, the output representations are executed with the feed forward and layer normalization operations the same way as the S2T Layer.
4.2.3 Spatial-temporal Synchronous Layer
This module named STS Layer, which aims to model the spatial and temporal information si- multaneously. Different from the former two modules, the inputs are directly used to caleu- late spatial and temporal representations at the same time. For temporal modeling part, it still project the original inputs into Q, K, V. and execute linear self-attention operation for a tem- poral representation. For the spatial part, it accepts the original inputs and the spatial infor- mation and uses high-order mix graph convolution operation to construct spatial representation. Temporal + Z Attention( Q, K, Spatial <- Z₂ HighOrder(X A, order). The outputs are concatenated together as: Z' concat Z, Zz]. Then is executed with the following operations and output as X similar with the former two modules.
4.3 Universal Modeling Framework
5

---

Targeting to the ST sequence forecasting task, we propose unified ST sequence modeling frame- work (UniST) with the proposed unified modeling backbones Fig E which follows the encoder- decoder architecture. It uses unified architecture with interchangeable and replaceable mode units.
Outputs
Foed Forward
Bel
Decoder
Feed Forward
4.3.1
Spatio-temporal Embedding Layer
Since the Transformer model solely relies the self-attention for global alignments, the positional embedding [21] and extra embeddings needed to capture spatio-tempon dependency. Then introduce four types of embeddings Ep, Ev, Es, Ey in Appendix
Encondr
Lager
2 are
Positioull Embeelling
S1S Layer
Eimbedding Layer
Token Embediing
Spatial Euibealing
Inputs
Temponal Embeediing
Fusion embedding. The four embeddings are summed together as the final embedding added the inputs: E, Ep Ey Eg + E7, note that the shape token embedding Ey E R R while other embeddings" shape are Ep E R Rixixd Eş E RlxNxx Ey E 22'xxxd When calculat- ing the summation, they will be replicated and expanded with broadcast on the respective missing dimensions.
Figure 4: Overview of the UniST framework, starting with an embedding layer and followed by an encoder and decoder. The embedding layer generates the sum of four embeddings. The encoder consists of multiple spatio-temporal ex- tractors, including S2T, T2S STS Layer.
4.3.2 Encoder
The encoder of UniST consists of multiple Spatio-Temporal Extractors (STE(-)), which can be arbitrarily chosen from (725 Layer, S2'' Layer, STS Layer). All extractors are connected end to end, i.e., the output of the previous representation, the outputs of each extractor are added to form the final output of the encoder. Let the outputs of the embedding layer be Xo. the encoder is computed as: Xen = E/ STE STE((Xo), where refers to the number of spatio-temporal extractors.
is the input of the next one. To acquire more diversity
4.3.3 Decoder
The decoder accepts the output of encoder, i.e., T. outputs from 7. spatio-temporal extractors. They are firstly added as unified spatio-temporal representation. Then the results are through two times of ReLU activation and Linear projection, and produce the final sequence forecasting result. Denote X, as the output of extractor É, we have the calculation of decoder as: Y X = X,)))).
Linear Linear(ReLUUg
4.4 Automated Search for UniST
With the proposed unified ST sequence modeling framework UniST, still suffers from the poten- tial wrong network configuration problem, where we build an arbitrary modeling order with the replaceable model units (72S Layer, S2T Layer, STS Layer). Considering the various downstream tasks, how we build universal model with optimal configuration? Here we propose the Automated Temmorll modeling approach (AutoST), which learns the optimal combinatorial order that suits the spatio-temporal dependency of the current task. We designed two schemes for layer combination. In this section, we first define the basic searching unit of AutoST, then we introduce two designs of AutoST with different searching schemes.
4.4.1 AutoST Cell
The basic searching unit in AutoST is the network cell. Here we define its structure and computing process.
Definition 1. AutoST Cell. Let G (V,&) be direct acyclic graph (DAG), V denotes the node set, each node refers to representation comes from the outputs of computation layer. The representation on node is defined as H. [Vxxx, where T is the length of spatio-temporal seqence, d is the

---

H
S1T Tager
HP
F25 Low
Layer
Encoder
Eneosder
TE
AUBEST Call
Hle
(a) 1 the sequential stacking scheme.
(b) AutoST2: the hybrid assembling scheme.
Figure 5: Two AutoST searching schemes. (a) AutoST, is the sequential stacking scheme. Only search for of the three modeling layers build the AutoST Cell. The cells are stacking sequentially to form the encoder, and each cell is skip-connected. (b) AutoST2 is the hybrid assembling scheme, with only AutoST Cell in the encoder. The cell consists of multiple nodes for intermediate representations. The connections between nodes are candidate layers.
feature dimension. The input of each AutoST Cell is denoted as H and the output of each cell is the summation of all nodes, i.e., all interval representations: Hout = = H' On graph 9, the directed edge (i,j) j) from node i to node j stands for mixture of all candidate modeling modules O = (T2S Layer, S2T Layer, STS Layer], and it is represented as o(ا-)), So that the representation between node j and other nodes can be written as: Hj Lies H'. On each directed edge,
there exist set of weight parameters (13) {ao (ab--0)0 E O), which indicates the probability of the corresponding modeling module should be retained. Every weight parameters of the candidate exp(a!)a)
modeling module is calculated Hi Eies EoesEoE0 Edeo exp (a) (H)).
4.4.2 Sequential Stacking Search
H
IF
Based on the proposed UniTS, we propose two searching schemes to find better combination of the modeling layers. The proposed searching schemes are lossless replacements for the encoder of UniTS. We simply replace the encoder' spatio- temporal extractor with the AutoST Cell.
SZU Layer
125 Layer
STS Layer
ST'S Layer
H
W
The first one is multi-layer sequential stacking searching scheme, with which the whole model is named AutoST As illustrated in Fig 5(a) has simple structure within each AutoST Cell, while has more complicated stacking structure between cells. Each cell holds DAG with three nodes, one input node H one output node Hout and an intermediate node H' And two directed edges are pre-defined between the former two nodes' output and the output node: (H" Hout) and (H² Hour).
Hay
Hus
Figure 6: Searching process of the AutoST Cell. The searching goal is choose the optimal one from the candidate modules between the given computation node Hè and H
The searching space of this scheme is shown the directed red dotted line. The red dashed box is the search candidate set, including (725 Layer, S2T Layer, STS Layer). We conduct two gradient based network architecture search methods in the experiments. i.c.. DARTS [16], PAS 22 After searching. the cell essentially becomes one of the three model units. This scheme allows multiple stacking of cells, therefore, the new encoder of the whole model will become the sequential stacking between different modeling layers.
4.4.3 Hybrid Assembling Search
The another searching scheme is called hybrid assembling searching. The structure is similar with the sequential stacking searching. However, in this scheme, the encoder only consists of single AutoST Cell. The cells are not stacked layer-by-layer, will conduct searching on a more complicated DAG on single AutoST Cell.
In this searching scheme, the DAG of AutoST Cell is shown as Fig 5(b) There are multiple nodes in the cell, generally, it will be set as 4-7 nodes in the experiments. The pre-defined connections
7

---

each node's output to the output node. The multiple directed red dotted lines show the searching space of this scheme. The candidate set is expanded with two operations: Identity and Zero. Identity means no modeling module in this edge, i.e., build directly connection with no operation. Zero means to set all the data passing through it to zero, i.e., no connection is made. This scheme makes the entire DAG form a complex and deep network structure. Through the complex structure design inside the cell, multiple spatio-temporal modeling modules are combined to fit the spatio-temporal dependency distribution in a target task.
5
Experiments
This section empirically evaluates the effectiveness of UniST and AutoST models with short-term, medium-term, and long-term ST sequence forecasting tasks on five real-world datasets. Platform: Intel(R) Xeon(R) CPU 2.40GHz x 2 + NVIDIA Tesla V100 GPU (32 GB) x 4. The code is available at https: com/ www..aaaaaaaatssttteteee
5.1 Datasets
In order to study the effect of various ST sequence forecasting methods under complex spatio- temporal distribution, five weal-world datasets with different tasks and data states are selected. The statistic information of the five datasets are listed in Table E
METR-LA H The traffic speed dataset contains months of data from March 2012 to June 30, 2012, recorded by sensors at 207 different locations on highways in Los Angeles County, USA. The data granularity is 5 minutes per point, and the spatial information provided by the dataset includes the coordinates of each sensor and the distance between the sensors.
The traffic speed dataset comes from the California Transportation Agencies (CalTrans) Performance Evaluation System (PeMS). The dataset contains data recorded by 325 sensors in the Bay area for total of 6 months from January 1, 2017 to May 31, 2017, and the data granularity is 5 minutes per point. The spatial information provided by the dataset includes the coordinates of each sensor and the distance between the sensors.
PEMS-BAY 14:
PEMSS-03/04/08 The three traffic datasets are also from the PeMS system of the California Transportation Agency, and each dataset is data recorded by sensors in a certain area of California. The PEMS-03 dataset contains data recorded by 358 sensors for 3 months from September 1, 2018 to November 30, 2018. PEMS-04 dataset contains 2 months of data recorded by 307 sensors from January I, 2018 to February 28, 2018. PEMS-08 dataset contains 2 months of data recorded by 170 sensors from July I, 2016 to August 31, 2016. The data granularity of these three datasets is 5 minutes per point, and the spatial information provided by the datasets only includes the connectivity between sensors.
5.2 Main Results
Table I summarizes the ST sequence forecasting results. UniST UniSTr, UniST stands for our UniST framework with three same stacking layers of S2T Layer, T2S Layer, STS Layer, respectively.
The results in bold font in Table Table show that our proposed UniST outperforms all baseline methods and achieves State-of-the Aft on all 9 tasks of 5 datasets, with all three proposed layers. This demonstrates our proposed unified spatio-temporal modeling layers and the unified forecasting framework are more expressive than traditional methods. Specifically, compared with GMAN, which is also based on self-attention mechanism, our methods achieve at most 18, 18.41%, 15.31 1%, 13.82% MAE decreases on the short-term, medium-tent and long-term forecasting task, respectively. Compared with the most advanced method STFGNN, our methods gain 12.06%, %, 10.61% %, 9.43% MAE decreases on the short, medium, long-term forecasting tasks.
From the last two columns of Table [ we can see that both of AutoST, and AutoST, beat all other methods on every metric of every task, including our proposed UniST. Recall that the difference between AutoST and UniST is the encoder part, UniST uses all same layers from (S2T Layer, T2S Layer, STS Layer), while AutoST aims to search for better combination and connection using the three types of layers. This demonstrates that combining and integrating modules with different spatio-temporal modeling abilities can better deal with uncertainty apatio-tepport dependency and
8

---

Table 1: Spatio-temporal sequence forecasting performance. The bold and shaded numbers are the best results of all methods. The bold numbers are the best results of manually designed models.
Dataset
Merric]
VAR
SVR
ARI.
WAV.
DCR.
STG.
G.WVV
STF.
GMA]
UniST:s
UniSTy UniSTs AutoST;
AutoST
1.5 mio
RMSE
7.89
8.45
8.21
5.89
5.38
5.74
5.15
4.73
5.48
4.79
4.43
4.47
4.38
4.33
MAE
4.42
3,99
3,99
2.99
2.77
2,88
2.69
2.57
2.77
2.25
2.29
2.33
2:23
2.19
METR-LA
MAPE
10:20
9.30
9.60
8.04
7.30
7.62
6.90
6.51
7.25
5.63
5.75
5.89
5.58
5.51
30 min
RMSE
9.13
10.87
10:45
7.28
6.45
7.24
6.22
5.46
6.34
5.20
5.12
5.22
5.09
5.02
MAE
5.41
5.05
5.15
3.59
3.15
3.47
3.07
2,83
3.07
2.57
2.53
2.61
2.50
2.44
MAPE
12:70
12:10
12.70
10.25
8.80
9.57
8.37
7.46
8.35
6.88
6.76
6.97
6.64
6.54
60 min
RMSE
10.11
13.76
13.23
8,93
7.60
9.40
7.37
6.40
7.21
6.09
6.07
6.23
6,03
6.00
MAE
6.52
6,72
6.90
4:45
3.60
4,59
3,53
3.18
3.40
2.91
2.89
3.02
2.85
2.78
MAPI:
15.80
16.70
17,40
13.62
10.50
12.70
10.01
8.81
9.72
8.17
8.12
8.47
7.88
7.79
15 min
RMSE
3.16
3.59
3.30
3.01
2.95
2.96
2.74
2.33
2.82
2.26
2.27
2.32
2.21
2.16
MAE
1.74
1,85
1.36
1.30
1.11
1.14
PEMS BAY
MAPT:
3.60
3,80
3.50
2.90
2.73
2.41
2.28
30 min
RMSE
4.25
5.18
4.76
4.21
3.97
4.27
3.70
3.02
3.72
2.94
2.96
2.92
2.89
2.85
MAE
2,48
2.33
1.63
1.39
1.33
1.32
MAPE
5.50
3.90
3.67
3.02
3.63
2.83
2.79
60 mio
RMSG
5.44
7,08
6.50
5.43
4.74
5.69
4.52
3:74
4.32
3.66
3.67
3.64
3.60
3.58
MAE
3.28
3.38
2.49
1,95
1.62
1.61
MAPE
6.50
8.00
8.30
5.87
4.90
5.79
4.63
3.77
4.31
3.59
3.60
3.58
PEMS3
60 min
RMSG
33.65
30.31
30.12
33.21
26.75
26.61
27.09
26.26
MAE
23.65
21.977
21.42
20.43
18.15
17.49
19.85
16.77
17:21
15.60
15.67
15.76
15.45
15.29
MAPF
24:51
21.51
21.12
20.19
18.91
17:15
19.31
16.30
18.27
16.08
16.14
16.24
15.98
15.77
PIMS4
60 min
RMSE
36.66
44.56
43.92
41,27
38.12
35.55
39.70
31.88
33.34
31.35
31.21
31,31
30.68
30.59
MAE
23:75
28.
28:45
26.88
24:70
22:70
25.45
19.83
20.93
19:60
19:45
19.67
19.05
19.01
MAPE
18.09
19.20
18,90
17.95
17.12
14.59
17.29
13.02
14.06
12,90
12.89
12,95
12,76
12.68
PEMS8
60 min
RMSE
36.33
36.16
35.34
33.62
27.83
27.83
31.05
26.22
26.
24.53
24.39
24.67
23,99
23.63
MAE
23
23.25
22.76
21.59
17.86
18.00
19.13
16.64
16.97
15,60
15.44
15.63
14,90
14.72
MAPE
15.42
14.64
14.51
14.10
11.45
11.40
12.68
10.60
11.32
10.55
10.42
10.63
10.19
10.01
Count
0
0
0
0
0
8
13
6
27
more comprehensively model spatio-temporal sequences. At the same time, AutoST2 beats AutoST, on every task. It shows that using a single-layer but complex internal connection searching method for these tasks is more effective.
Moreover, we can see that the best methods among all baselines are STFGNN, the only method in spatio-temporal synchronous modeling. Comparing GMAN and Graph WaveNet, the two represen- tative methods in spatial-firy first modeling and temporal- first modeling, respectively, we can see that although they beat other baselines, while our proposed Uni- series model shows better performance. From this perspective, the proposed method matches the design.
5.3 Ablation Study
In AutoST, we design to use gradient-based network architecture search methods to optimize the connections of the modeling modules, and we choose DARTS and PAS as the NAS methods. At the same time, we also conduct experiments to compare the method of random search and AutoSTG [17], which also uses the network architecture search technology for ST sequence prediction. The results are shown in Table We can conclude that our proposed AutoST can outperform AutoSTG with all searching methods. That is because AutoSTG uses more fine-grained temporal convolution and graph convolution structure as the candidates, while AutoST's candidate layers are designed to model the three different spatio-temporal dependencies, which can make better use of the modeling ability of the three layers on different temporal and spatial relations. It reduces the searching space and the search time simultaneously so that AutoST can make better efficiency and accuracy.
5.4 Result Visualization
We randomly selected one day from the PEMS-BAY BAY dataset and com- pared our methods' forecasting results and baseline methods in a visualization way. Typical results are shown in Fig the time span is selected from 13:00 to 19:00, which can represent the most typical scene of the rush hour from afternoon to evening. From the forecasting lines compared to the ground truth, we can see that our proposed methods outperform the two typical baseline methods mostly when forecasting smooth traffic and traffic jam. The bar lies on the bottom of the figures is the forecasting error of each method. We can find that all methods encountered accuracy drop
Forecasting Visualization:
9

---

Table 2: The results of network searching methods.
The search results of (h) The search results of AutoST2.
AutoST
Method
METR-LA (60 min)
PEMS-BAY (60 min)
RMSE
MAE
MAPE
RMSE
AutSTTG
7:27
3,47
AmtSTT
3.65
AutoST R
6.08
2.92
8.26
3.68
1.62
3.66
AutoST,
6.09
2.85
7.96
3,74
1,68
3.72
Lavia
AutoST2-D
6.16
3.25
8.48
3.64
1.60
3.61
AutoST -P
6:03
2.85
7,88
3.60
1.57
3,52
AutoST2-P
6,00
2.78
7.79
3,58
1,52
3,48
Figure 7: The learned architectures of AutoST.
mac
18-
12
Figure 8: The forecasting visualization in the rush hour of PEMS-BAY dataset. The red and yellow bars on the bottom show the difference between the result and the ground truth at each time step.
around 14:00 and 17:30. That is because the status of the road is changed dramatically. However, our methods are more stable with the changes and can quickly be adapted to the new road state. For the complete data visualization of the day, please refer to Fig the appendix.
Learned Architecture Visualization: The learned architectures of the AutoST, and AutoST, on PEMS-08 dataset are shown in Fig We can find that although UniSTr has a better effect when modeling with single type modeling module, the search result of AutoST, show that the STS Layer occupies larger number, and the model achieves better performance than UniSTr. This demonstrates that combining and stacking multiple apatio-temporal dependency modeling methods reasonably can better fit the real spatio-tempora dependencies.
In addition, we can find that AutoST2 obtains a complex connections between the four computation nodes in a single cell. And this learned architecture helps AutoST, achieve state-of-the-art this task. Although we cannot yet explain why stacking the modules leads to better results, we can see a potentially broad range of applications 25 for unified architecture searching in this way.
6 Conclusion
In this work, we illustrated the existence of the modeling gap problem, especially the modeling order, in the spatio-temporal analysis. Moreover, build three different layers, namely S2T, T2S, and STS, as new network modeling backbones. Then, an automatic searching strategy is proposed to search the optimal modeling priority automatically. Extensive experiments on five real-world datasets show the overwhelming performance over SOTA baselines.
Acknowledgments
This work was supported by grants from the Natural Science Foundation of China (U20B2053, 62202029) and Foshan HKUST Projects FFSUST2I-FYTRIOLA FSUST2I-FYTRIO2A, Thanks for computing infrastructure provided by Beijing Advanced Innovation Center Big Data and Brain Computing. This work was also sponsored by CAAI-Huawei MindSpore Open Fund. We gratefully acknowledge the support of MindSpore, CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.
10

---

References
[1] Lei Bai, Lina Yao, Salil Kanhere, Xianzhi Wang, Quan Sheng, et al. Stg2seq: Spatial- temporal graph to sequence model for multi-step passenger demand forecasting. arXiv preprint arXiv: 10069, 2019.
[2] George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time series analysis: forecasting and control. John Wiley Sons, 2015.
[3] Chao Chen, Karl Petty, Alexander Skabardonis, Pravin Varaiya, and Zhanfeng Jia. Freeway performance measurement system: mining loop detector data. Transportation Research Record, 1748(1):96-102, 2001.
[4] Rong Chen, Chang- Yong Liang, Wei-Chiang Hong, and Dong-Xin Xiao Gu, Forecasting holiday daily tourist flow based on seasonal support vector regression with adaptive genetic algorithm. Applied Soft Computing, 26:435 443, 2015.
Chang-Yong
[5] Shen Fang, Qi Zhang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Gstnet: Global spatial-temporal network for traffic flow prediction. In IJCAI, pages 2286-2293, 2019.
[6] Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based spatial- temporal graph convolutional networks for traffic flow forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 922-929, 2019.
[7] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 173: 1997.
[8] H. V. Jagadish, Johannes Gehrke, Alexandros Labrinidis, Yannis Papakonstantinou, Jignesh M. Patel, Raghu Ramakrishnan, and Cyrus Shahabi. Big data and its technical challenges. Commun. ACM, 57(7):86-99) 2014.
[9] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mms:: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156-5165. PMLR, 2020.
[10] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: 1609.020770 2016.
[11] Guokun Lai, Wei-Cheng Chang, Yiming Yang. and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 95-104, 2018.
[12] Mengzhang Li and Zhanxing Zhu. Spatial-temporal fusion graph neural networks for traffic flow forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 4189-4196, 2021.
[13] Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. arXiv preprint arXiv: /707v0/707 2017.
[14] Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data- Data-driven traffic forecasting. In 6th International Conference on Learning Rep- resentations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
[15] Zachary C Lipton, John Berkowitz, and Charles Elkan. A critical review of recurrent neural networks for sequence learning. arXiv preprint 10Xiv:/506.00019 2015.
[16] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint wet:.090000...3 2018
[17] Zheyi Pan, Songyu Ke, Xiaodu Yang, Yuxuan Liang, Yong Yu, Junbo Zhang, and Yu Zheng Autostg: Neural architecture search for predictions of spatio-temporal graph. In Proceedings of the Web Conference 2021, pages 1846-1855, 2021.

---

[18] Christopher A Sims et al. Are forecasting models usable for policy analysis? Quarterly Review, 10(Win):2-1,, 1986.
[19] Chao Song, Youfang Lin, Shengnan Guo, and Huaiyu Wan. temporall synchronous graph convolutional networks: A new framework for
network data forecasting. In Pruceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 914-921,
2020.
[20] Aäron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: generative model for raw audio. SSW, 125:2, 2016.
[21] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[22] Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsich. Re- thinking architecture selection in differentiable nas. arXiv preprint vrXiv:2108.04333 2021.
[23] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling. arXiv preprint (oXiv:1906.00122 2019.
[24] Meiling Xu, Yuanzhe Yang. Min Han, Tie Qiu, and Hongfei Lin. Spatio-temporal interpolated echo state network for meteorological series prediction. IEEE transactions on newral networks and learning systems, 30(6): 1621-1634 2018.
(25) Yang Yang, De-Chuan Zhan, Yi-Feng Wu, Zhi-Bin Liu, Hui Xiong, and Yuan Jiang. Semi- supervised multi-modal clustering and classification with incomplete modalities. IEEE Transac- tions on Knowledge and Data Engineering, 33(2):682-695, 2019.
[26] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Yan Liu. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 34, 2021.
[27] Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: deep learning framework for traffic forecasting, arXiv preprint arXiv: 747777 2017.
[28] Junbo Zhang, Yu Zheng, and Dekang Qi. Deep spatio-temporal residual networks for citywide crowd flows prediction. In Thirty-first AAAI conference on artificial intelligence, 2017.
[29] Aite Zhao, Lin Qi, Jie Li, Junyu Dong, and Hui Yu. hybrid spatio-temporal model for detection and severity rating of parkinson's 's disease from gait data. Neurocomputing, 315:1-8, 2018.
[30] Chuanpan Zheng, Xiaoliang Fan, Cheng Wang, and Jianzhong Qi. Gman: A graph multi- attention network for traffic prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 1234-1241, 2020.
[31] Haoyi Zhou, Hao Peng, Jieqi Peng, Shuai Zhang, and Jianxin Li. Polla: Enhancing the local structure awareness in long sequence ppatial-tempore modeling. ACM Transactions Intelligent Systems and Technology (TIST), 12(6))1-24, 2021.
[32] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of AAAI, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Due to the space limits, the limitations are in the supplemental material.
12

---

(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] The code, data and instructions to reproduce the experimental results are provided in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] These details are in the Appendix Section [
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either the supplemental material or as URL? [Yes]
Our code are included in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We only used public datasets in this paper.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] To the best of our knowledge, the traffic datasets used our paper do not have any personally identifiable information or offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] No human subjects are involved.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A] No human subjects are involved.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] No human subjects are involved.
13


Reviews:
[{"review_id": "s6dsWaU6OZd", "reviewer": "Reviewer_dPZy", "summary": "Two Network architecture search algorithms are proposed, called Automated Spatio-Temporal\nmodeling approach, AutoST1 and AutoST2. They are focused on automatically crafting spatio temporal forecasting systems. Both the search algorithms operate by selecting from a layer candidate set,  formed by three types of layer: T2S Layer (Temporal-first Modeling Layer), S2T Layer (Spatial-first Modeling Layer), STS Layer(Spatial-temporal Synchronous Layer). All of these three layers have exactly same inputs and outputs,  and operate by assembling in different fashion a Time Series Linear Self-Attention Unit and a High-order Mix Graph Convolution Unit. AutoST2 is the most performing one, which creates a serial architecture of hidden layers, each one of them can be a T2S, S2T, STS layer. Results on traffic datasets support the search algorithm on five traffic forecasting datasets. \n", "questions": "- See my question on the time complexity.\n- Having five datasets of traffic forecasting, most of them having very similar structure and content, is definitely a cons. It would have been definitely better to take into account a dramatically different scenario/task (eg pose forecasting)\n", "limitations": "No limitations or potential negative impacts have been taken into account by the authors. I think it should be, since managing traffic data could be very impactful on the privacy management. ..", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The idea of learning an architecture which understands whether spatial or temporal relations are more important to forecast a given signal is intriguing. Figure 2 has a very effective visualization, which makes the intuition of the authors crystal clear.\n\n\n\nWeaknesses:\n-- The experiments focus on very specific datasets, all of them representing the same problem (traffic forecasting). It would have been definitely better to have more heterogenous tasks, as for example pose forecasting, where the spatial part is the body skeleton (expressed as a graph) and the temporal part let the body joints move accordingly to anatomical costraints. An interesting comparison would have been \nTheodoros Sofianos, Alessio Sampieri, Luca Franco and Fabio Galasso  Space-Time-Separable Graph Convolutional Network for Pose Forecasting In Proc. International Conference on Computer Vision (ICCV) Virtual, October 2021 \nThey use a Space-Time Separable GCN (STS-GCN. Which is *not* the same as the STSGCN cited in the proposed paper as [19]). \n\n-- The Complexity of the AutoST1, and especially AutoST2 is not detailed. If complexity cannot be given, timings become very important.  In particular, since AutoST2 has to look for multiple optimal layers at each cell, what is the time spent overall, and for completing each layer of Fig.5.b?\n\n-- The definition of Adaptive Diffusion Convolution is clear (ChebNet + ADC), but its motivation is not. A sentence on the rationale for this choice would be beneficial.\n\n-- In section 3.2, it is not clear in the result connection formula what is o^{(i,j)}, and why i<j, w.r.t. which ordering?\n\n-- The rationale why T2S, S2T, STS have been built as described in 4.2.1, 4.2.2., 4.2.3 ,with that precise architecture, is not given.\n\nMINOR:\n \n--Fig:3 S2T Layer: firstly model temporal... -->T2S Layer: firstly model temporal...\n--pag.3: is automated methods --> are automated methods\n--pag.3 our proposed trainning framework --> our proposed training framework\n--In the objective function of section 3.2, w and alpha are undefined. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Privacy and Security (e.g., consent)", "Legal Compliance (e.g., GDPR, copyright, terms of use)"]}, "extra_text": {}}, {"review_id": "I9sV6noP0cH", "reviewer": "Reviewer_2S7t", "summary": "Paper proposes a novel universal framework for spatio-temporal forecasting. The framework combines spatial first, temporal first and spatio-temporal synchronous approaches in one network. These are implemented as S2T Layer, T2S Layer and STS Layer respectively. This is termed UniST.\n\nPaper further proposes a novel automatic search strategy by network architecture search. \n\nExperimental results on 5 datasets across 9 tasks outperforms existing methods significantly.", "questions": "No question.", "limitations": "Not applicable.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. Paper is well-written and easy to understand.\n2. Method is presented clearly with detailed explanations and good use of diagrams.\n3. Experiments are extensive and strongly supports the main claims of the paper\n\nWeakness\n1. Minor. The labeling of Table 1 is too brief. It should contains more information to explain the Table content.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-QTlvud19hY", "reviewer": "Reviewer_MfGi", "summary": "The paper focuses on architecture design for spatio-temporal modeling for forecasting tasks. The paper first proposes three types of layers (i.e., S2T, T2S, STS) which have emphasis on modeling spatial, temporal and spatio-temporal information, respectively. The three layers are then applied in a unified encoder-decoder framework for spatio-temporal forecasting tasks. An automated architecture search algorithm is further adopted to better decide the connection configuration of different modeling layers. The final model, AutoST, achieves better results than baseline methods as well as the proposed UniST, which only involves one of the three modeling layers (S2T / T2S / STS).", "questions": "Minor:\n1. A comma is missing in the last equation of Eqn. 1.\n2. The equation in L204 is inconsistent with the text \"the outputs of each extractor are added to form the final output\".\n", "limitations": "Not discussed in the paper.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1. The idea of analyzing and unifying different types of spatio-temporal modeling (i.e., spatial first, temporal first, spatio-temporal synchronous) is well-motivated and potentially valuable to the research community.\n2. The paper provides clear ablation studies (in Table1) that show (1) the performance difference of the three modeling types on different datasets / temporal setups; (2) the effectiveness of introducing automated architecture search to fuse the three types of modeling layers.\n\nWeaknesses\n1. Although I like the idea of unifying different types of spatio-temporal modeling, the technical novelty and contribution of this work is not significant enough. Most of the techniques in the paper are adopted from prior works with minor modifications, for example, linear self-attention, mix graph convolution (with combination of ChebNet and AdapDC), and DARTS / PAS for automated architecture search. Even from the framework design perspective, the ideas of spatial-first / temporal-first and joint spatio-temporal modeling are widely explored in previous work mentioned as baselines in Table 1, as well as the decoupled spatio-temporal modeling design [1,2] for video action recognition. The encoder-decoder framework adopted in UniST is also widely used for both NLP and CV tasks in recent years.\n\nIn addition, although the paper provides comparison between UniST and other baseline methods in Table 1, the comparison is not totally fair due to different model complexity, computation cost and even optimization details. This also makes the contribution of the proposed UniST insufficient.\n\n2. The major contribution of this work to me is using architecture search algorithms to unify three types of modeling layers, i.e. the final AutoST model. While the paper compares the results of using different NAS algorithms, an important baseline is missing. Instead of only using one the the three types of modeling layers (UniST), it is intuitive to have a model that includes all of the three layers at each STE and fuses all three outputs, optionally augmented with a gating module to introduce a weighted fusion mechanism. With this baseline result provided, the necessarity of using NAS to select modeling layers would be more convincing.\n\n\n\n[1] Tran, Du, et al. \"A closer look at spatiotemporal convolutions for action recognition.\" Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2018.\n[2] Qiu, Zhaofan, Ting Yao, and Tao Mei. \"Learning spatio-temporal representation with pseudo-3d residual networks.\" proceedings of the IEEE International Conference on Computer Vision. 2017.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept