TESTAM: A Time-Enhanced Spatio-Temporal Attention Model with Mixture of Experts
Authors: Hyunwook Lee
Sungahn Ko
Venue: ICLR 2024 poster (2024)
Keywords: Traffic Prediction
Deep Learning
Spatio-Temporal data modeling
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a novel mixture-of-experts model named TESTAM that enables in-situ modeling of the traffic data

Abstract: Accurate traffic forecasting is challenging due to the complex dependency on road networks, various types of roads, and the abrupt speed change due to the events. Recent works mainly focus on dynamic spatial modeling with adaptive graph embedding or graph attention having less consideration for temporal characteristics and in-situ modeling. In this paper, we propose a novel deep learning model named TESTAM, which individually models recurring and non-recurring traffic patterns by a mixture-of-experts model with three experts on temporal modeling, spatio-temporal modeling with static graph, and dynamic spatio-temporal dependency modeling with dynamic graph. By introducing different experts and properly routing them, TESTAM could better model various circumstances, including spatially isolated nodes, highly related nodes, and recurring and non-recurring events. For the proper routing, we reformulate a gating problem into a classification problem with pseudo labels. Experimental results on three public traffic network datasets, METR-LA, PEMS-BAY, and EXPY-TKY, demonstrate that TESTAM achieves a better indication and modeling of recurring and non-recurring traffic.

Published as a conference paper at ICLR 2024
TESTAM: A TIME-ENHANCED SPATIO-TEMPORAL ATTENTION MODEL WITH MIXTURE OF EXPERTS
Hyunwook Lee & Sungahn Ko Ulsan National Institute of Science and Technology (gusdnr0916, sa sako)@un @uni st. ac. kr
ABSTRACT
Accurate traffic forecasting is challenging due to the complex interdependencies of large road networks and abrupt speed changes caused by unexpected events. Recent work has focused on spatial modeling with adaptive graph emhedding graph attention but has paid less attention to the temporal characteristics and effectiveness of in-situ modeling. In this paper, we propose the time-enhanced spatio-temporal attention model (TESTAM) better capture recurring and non- recurring traffic patterns with mixture-of -ffeppersss model with three experts for tem- poral modeling. spatio-temporal modeling with static graph, and spatio-temporal dependency modeling with a dynamic graph. By introducing different experts and properly routing them, TESTAM better captures traffic patterns under various circumstances, including cases of spatially isolated roads, highly interconnected roads, and recurring and non-recurring events. For proper routing, we reformulate gating problem as classification task with pseudo labels. Experimental results on three public traffic network datasets, METR-LA, PEMS-BAY, and EXPY-TKY, demonstrate that TESTAM outperforms 13 existing methods in terms of accuracy due to its better modeling of recurring and non recurrinnn traffic patterns. You can find the official code from https: btttps://github.. com htt.://...coo//yyooooo/////AAAa
INTRODUCTION
Spatio-temporal modeling in oon-Eulideean space has received considerable attention since it can be widely applied to many real-world problems, such social networks and human pose estimation. Traffic forecasting is a representative real-world problem, which is particularly challenging due to the difficulty of identifying innate spatio-temporal dependencies between roads. Moreover, such de- pendencies are often influenced by numerous factors, such as weather, accidents, and holidays (Park
al. 2020
Lee al. 2020 Lee et 2022).
modeling, many deep learning models have been proposed, including graph convolutional networks (GCNs), recurrent neural networks (RNNs), and Transformer. Li et al. (018)) have introduced DCRNN, which injects graph convolution into recurrent units, while Yu et al. (2018) have combined graph convolution and convolutional neural networks (CNNs) model spatial and temporal features, outperforming traditional methods, such as ARIMA. Although effective, GCN-based methods require prior knowledge of the topological characteristics of spatial dependencies. In addition, as the pre-defined graph relies heavily the Euclidean distance and empirical laws (Tobler's s first law of geography). ignoring dynamic changes in traffic (e.g., rush hour and accidents), it is hardly WaveNet, proposed by Wu et al. (2019). is the first model to address this limitation by using node embedding, building learnable adjacency matrix for spatial modeling. Motivated by the success of Graph-WaveNet and DCRNN, line of research has focused on learnable graph structures, such as AGCRN (Bai et al. 2020) and MTGNN (Wu al., 2020).
To overcome the challenges related to
optimal solution (Jiang al. 2023). Graph-
Although spatial modeling with learnable static graphs has drastically improved traffic forecast- ing, researchers have found that it can be further improved by learning networks dynamics among time, named time-varying graph structure. SLCNN (Zhang et al. (2020) and StemGNN (Cao et al.
"Corresponding author

---

Published as a conference paper at ICLR 2024
attempt to learn time-varying graph structures by projecting observational data. Zheng et al. 2020 have adopted multi-head attention for improved dynamic spatial modeling with no spatial restrictions, while Park et al. (2020) have developed ST-GRAT, modified Transformer for traffic forecasting that utilizes graph attention networks (GAT). However, time-varying graph modeling is noise sensitive. Attention-based models be relatively less noise sensitive, but recent study reports that they often fail to generate an informative attention map by spreading attention weights over all roads (Jin et al. 2023). MegaCRN (Jiang al. 2023) utilizes memory networks for graph learning, reducing sensitivity and injecting temporal information, simultaneously. Although effec- tive, aforementioned methods focus on spatial modeling using specific spatial modeling methods, paying less attention to the use of multiple spatial modeling methods for in-situ forecasting.
2020)
Different spatial modeling methods have certain advantages for different cireumstances. For in- stance, learnable static graph modeling outperforms dynamic graphs in recurring traffic situa- tions (Wu et al.] 2020 Jiang al., (023). On the other hand, dynamic spatial modeling is ad- vantageous for non-recurring traffic, such as incidents
abrupt speed changes (Park et al. 2020
Zheng et al. 2020). Park et al. (2020) have revealed that preserving the the road information itself improves forecasting performance, implying the need of temporal-only modeling, Jin et al. (2023) have shown that a static graph built on temporal similarity could lead to performance improvements when combined with a dynamic graph modeling method. Although many studies have discussed the importance of effective spatial modeling for traffic forecasting, few studies have focused on the dynamic use of spatial modeling methods in traffic forecasting (i.c., in-situ traffic forecasting).
In this paper, we propose time-enhanced spatio-temporal attention model (TESTAM), a novel Mixture-of-Expert (MoE) model that enables in-situ traffic forecasting. TESTAM consists of three experts, each of them has different spatial modeling: I) without spatial modeling, 2) with learnable static graph, 3) with with dynamic graph modeling, and one gating network. Each expert consists of transformer-based blocks with their own spatial modeling methods. Gating networks take each expert's 's last hidden state and input traffic conditions, generating candidate routes for in-situ traffic forecasting. To achieve effective training of gating network, we solve the routing problem as a clas- sification problem with two loss functions that are designed to avoid the worst route and lead to the best route. The contributions of this work can be summarized as follows:
We propose novel Mixture-of-Expers model called TESTAM for traffic forecasting with diverse graph architectures for improving accuracy in different traffic conditions, including recurring and non-recurring situations.
We reformulate the gating problem as classification problem have the model better contextualize traffic situations and choose spatial modeling methods (i.c., experts) during training.
The experimental results over the state-of-the models using three real-world datasets indicate that TESTAM outperforms existing methods quantitatively and qualitatively.
2 RELATED WORK
2.1 TRAFFIC FORECASTING
Deep learning models achieve huge success by effectively capturing spatio-temporal features in traf- fic forecasting tasks. Previous studies have shown that RNN-based models outperform conventional temporal modeling approaches, such as ARIMA and support vector regression (Vlahogianni et al., 2014 Li Shahabi 2018). More recently. substantial research has demonstrated that attention- based models (Zengg et al. 2020 Park et al.] 2020) and CNNs (Yu et al. 2018 Wu et al.) 2019 2020) perform better than R RNN-based model in long-term prediction tasks. For spatial modeling, Zhang et al. (2016) have proposed a CNN-based spatial modeling method for Euclidean space. An- other line of modeling methods using graph structures for managing complex road networks (c.g., GCNs) have also become popular. However, using GCNs requires building an adjacency matrix, and GCNs depend heavily on pre-defined graph structure.
To overcome these difficulties, several approaches, such graph attention models, have been pro- posed for dynamic edge importance weighting (Park et al. 2020). Graph-Was WaveNet (Wu et al. 2019) uses learnable static adjacency matrix capture hidden spatial dependencies in training.
2

---

Published as a conference paper at ICLR 2024
SLCNN (Zhang al. 2020) and StemGNN (Cao et al. 2020) try to learn time-igrying graph by projecting current traffic conditions. MegaCRN (Jiang et al. 2023) uses memory- graph learning to construct noise-robust graph. Despite their effectiveness, forecasting models still suffer from inaccurate predictions due to abruptly changing speeds, instability, and changes in spatial de- pendency. To address these challenges, we design TESTAM change its spatial modeling methods based on the traffic context using the Mixture-of-Expert technique.
2.2 MIXTURE OF EXPERTS
The Mixture-of of of-eexstts (MoEs) is machine learning technique devised by [Shazeer al. (2017) that has been actively researched as a powerful method for increasing model capacities without ad- ditional computational costs. MoEs have been used in various machine learning tasks, such com-
puter vision (Dryden & Hoefler 2022) and natural language processing (Zhou et al.) 2022 Fedus
al. 2022). Recently, MoEs have gone beyond being the purpose of increasing model capacities
and are used to "specialize" each expert in subtasks at specific levels, such as the sample (Eigen
2014 McGill & Perona, 2017, Rosenbaum et al., 2018), token (Shazeer et al., 2017, Fedus et al.
al.
2022). and patch levels (Riquelme et al.) 2021). These coarse-grained routing of the MoEs are frequently trained with multiple auxiliary losses, focusing on load balancing (Fedus et al. 2022 Dryden & Hoefler 2022), but it often causes the experts to lose their opportunity specialize. Fur- thermore, MoEs assign identical structures to every expert, eventually leading to limitations caused by the architecture, such as sharing the same inductive bias, which hardly changes. Dryden & Hoe- (SMoEs) that introduces fine-grained routing
fler (2022) have proposed Spatial Mixture-of-Expert to solve the regression problem. SMOES induce inductive bias via fine-grained, location-dependent routing for regression problems. They utilize one routing classification loss based on the final output losses, penalize gating networks with output error signals, and reduce the change caused by inac- curate routing for better routing and expert specialization. However, SMoEs only attempt to avoid incorrect routing and pay less attention to the best routing. TESTAM differs from existing MoEs in two main ways: it utilizes experts with different spatial modeling methods for better generalization, and it can be optimized with two loss functions-one for avoiding the worst route and another for choosing the best route for better specialization.
METHODS
3.1 PRELIMINARIES
Problem Definition Let us define a road network as 9 = E,E,A)) where V is a set of all roads in road networks with [V] = N, E is set of edges representing the connectivity between roads, and A E IR N is a matrix representing the topology of G. Given road nerworks, we formulate our problem special version of multivariate time series forecasting that predicts future T graph signals based on 7" historical input graph signals:
X (1-
(1)
10
[ ((+1)
X
(((+T)
where _ E N C is the number of input features. We aim to train the mapping function f()) : RRTXXXX XXXC + xxC which predicts the next T steps based on the given T'' observations. For the sake of simplicity, omit G from Xg hereinafter.
Spatial Modeling Methods in Traffic Forecasting To effectively forecast the traffic signals, we first discuss spatial modeling. which is one of the necessities for traffic data modeling. In traffic forecasting, we can classify spatial modeling methods into four categories: 1) with identity matrix (i.e., multivariate time-series forecasting). 2) with a pre-defined adjacency matrix, 3) with train- able adjacency matrix, and 4) with attention (i.e., dynamic spatial modeling without prior knowl- edge). Conventionally, graph topology A is constructed via an empirical law, including inverse distance (Li et al. 2018 Yu et al., 2018) and cosine similarity (Geng et al.) 2019). However, these empirically built graph structures modeling quality. To address this challenge, a line of research (Wu et al.] 2019 Bai et al. 20200] Jiang al. 2023) is proposed to capture the hidden spatial information. Specifically, trainable
not necessarily optimal, thus often resulting in poor spatial
3

---

Published as a conference paper at ICLR 2024
troor loodock
Identity
Feed Forward
7
Pnap
Gatling
Fully
Time-frhanced Attertion
Palnwise Distance
Adapthe
gill - (Pi-1
Iderdity Experts
Adaytive
RM
Fully Connected Networts
Temporal
Atsention
X = (x(t-7"+t)...,x"
H Attentionial--ty
/0
TIM
FC
Xie (elt-
Figure 1: Overview of TESTAM. Left: The architecture of each expert. Middle: The workflow and routing mechanism of TESTAM. Solid lines indicate forward paths, and the dashed lines represent backward paths. Right: The three spatial modeling methods of TESTAM. The black lines indicate spatial connectivity, and red lines represent information flow corresponding to spatial connectivity. Identity, adaptive, and attention experts are responsible for temporal modeling, spatial modeling with learnable static graph, and with dynamic graph (i.c., attention), respectively.
function 0): is used to derive the optimal topological representation Á as:
À so tootttrrrrlll(("" (x)( ,0),
where ((X(1), 0) E RRNxe, and is the embedding size. Spatial modeling based on Eq. 1 can be classified into two subcategories according to whether g(; 0) depends on (+) Wu et al. (011 define g(., 8)
Nxe which is time-independent and less noise-sensitive but less in-situ modeling. Cao et al. (2020): Zhang et al. (2020) propose time-varying graph structure modeling with g 0) HWW, where W E Rd projecting hidden states to another embedding space. Ideally, this method models dynamic changes in graph topology, but it is noise-sensitive.
To reduce noise-sensitivity and obtain time-varying graph structure, Zheng et al. (2020) adopt spatial attention mechanism for traffic forecasting. Given input H, of node and spatial neighbor N₁, they compute spatial attention using multi-head attention as follows:
H = Concat(o (1) .
(K) Wo;
Σ SEN
ais 1(0)
(2)
exp(ei.))
(K)
(H.)) (H
Oig ESEN; exp(er.))
Ci.j
ddz
where W a projection layer, de is a dimension key vector, and f f(()) ()), f ()), and (!)))) are query, key, and value projections of the k-th head, respectively. Although effective, these attention- based approaches still suffer from irregular spatial modeling, such as less accurate self-attention (i.e., from node i) (Park al. 2020)) and uniformly distributed uninformative attention, regardless of its spatial relationships (Jin et al. 2023)
3.2 MODEL ARCHITECTURE
Although transformers are well-established structures for time-series forecasting, it has a couple of problems when used for spatio-temporal modeling: they do not consider spatial modeling, consume considerable memory resources, and have bottleneck problems caused by the autoregressive decod- ing process. Park al. (2020) have introduced improved transformer model with graph attention (GAT), but the model still has auto-regressive properties. To eliminate the autoregressive character- istics while preserving the advantage of the encoder-decoder architecture, TESTAM transfers the
4

---

Published as a conference paper at ICLR 2024
attention domain through time-enhame attention and temporal information embedding. As shown in Fig. (left), in addition to temporal information embedding, each expert layer consists of four sub- layers: temporal attention, spatial modeling, time-enhanced attention, and point-wis, ferd-forward neural networks. Each sublayer connected to a bypass through skip connections. To improve gen- eralization, we apply layer normalization after each sublayer. All experts have the same hidden size and number of layers and differ only in terms of spatial modeling methods.
Temporal Information Embedding Since temporal features (e.g., time of day) work as a global position with specific periodicity, omit position embedding in the original transformer chitecture. Furthermore, instead of normalized temporal features, we utilize Time2Vec embed- ding (Kazemi et al., 2019) for periodicity and linearity modeling. Specifically, for the temporal feature 7 E 14, we represent r with /t-dimensional embedding vector u(T) and the learnable param- eters Wj, Φ; for each embedding dimension as below:
(.)(T) + oi. [( + +.)
if iflsigh-1
TIM(
(4)
where F is periodic activation function. Using Vee embedding, enable the model to uti- lize the temporal information of labels. Here, temporal information embedding of an input sequence concatenated with other input features and then projected onto the hidden size h.
Temporal Attention As temporal attention in TESTAM is the same that of transformers, we de- scribe the benefits of temporal attention. Recent studies (Li et al. 2018 Bai et al., 2020) have shown that attention is appealing solution for temporal modeling because, unlike recurrent unit-based or convolution-baal temporal modeling, it can be used to directly attend to features across time steps with restrictions. Temporal attention allows parallel computation and is beneficial for long-term sequence modeling. Moreover, has less inductive bias in terms locality and sequentiality. Al- though strong inductive bias can help the training, less inductive bias enables better generalization. Furthermore, for the traffic forecasting problem, causality among roads is an unavoidable factor (Jin al. 2023) that cannot be easily modeled in the presence of strong inductive bias, such as sequen- tiality or locality.
Spatial Modeling Layer In this work, we leverage three spatial modeling layers for each expert, as shown in the middle of Fig. T spatial modeling with an identity matrix (i.e., no spatial model- ing), spatial modeling with learnable adjacency matrix (Eq. Th. and spatial modeling with attention (Eq. 2 and Eq. B.. We calculate spatial attention using Eqs. 2 and [3] Specifically, we compute tention with Vie,,N; = V. which means attention with no spatial restrictions. This setting enables similarity-based attention, resulting in better generalization.
Inspired by the success of memory-augmenter graph structure learning (Jiang et al., 2023 Lee et al.) 2022). we propose a modified meta-graph learner that learns prototypes from both spatial graph modeling and gating networks. Our meta-graph learner consists of two individual neural networks with a meta-node bank M E Rm where m and e denote total memory items and a dimension of each memory, respectively, hyper-network (Ha et al.] (017) for generating node embedding conditioned on M, and gating networks to calculate the similarities between experts' hidden states and queried memory items. In this section, we mainly focus on the hyper-network. We construct graph structure with a meta-node bank M and a projection We E Reed as follows:
E MW e: A softmaz(relu(EE
By constructing memory--aggmentee graph, the model achieves better context-await spatial mod- eling than that achieved using other learnable static graphs (e.g., graph modeling with E E N Detailed explanations for end-to-end training and meta-node bank queries provided in Sec.
Time-Enhanced Attention To eliminate the error propagation effects caused by auto-regressive characteristics, we propose a time-enhanced attention layer that helps the model transfer its domain from historical 7" time steps (i.e., source domain) next T time steps (i.e., target domain). Let
5

---

Published as a conference paper at ICLR 2024
Tabel [-((+1), 1 p(++) be temporal feature vector of the label. We calculate the attention
(1)
score from the source time step i to the target time step j as:
exp(es.))
0i Lkel1 exp(ei,A)
WH (TM
(5)
vdk
where d; = d/ , K is the number of heads, and (A) W We can calculate the attention output using the same process as in Eq. [ except that time-enhanced attention attends to the time steps of each node, whereas Eq. attends to the important nodes at each time step.
are linear transformation matrices.
3.3 GATING NETWORKS
In this section, describe the gating networks used for in-situ routing. Conventional MoE models have multiple experts with the same architecture and conduct coarse-grained routing, focusing on increasing model capacity without additional computational costs (Shazeer et al. 2017). However, coarse-grained routing provides experts with limited opportunities for specialization. Furthermore, in the case of the regression problem, existing MoEs hardly change their routing decisions after ini- tialization because the gate is not guided by the gradients of regression tasks, as Dryden & Hoefler (2022) have revealed. Consequently, gating networks cause "mismatches," resulting uninforma- tive and unchanging routing. Moreover, using the same architecture for all experts is less beneficial terms of generalization since they also share the same inductive bias.
To resolve this issue, we propose novel memory-based gating networks and two classification losses with regression error-based pseudo labels. Existing memory-based traffic forecasting ap- proaches (Lee et al. 2022. Jiang et al.) 2023) reconstruct the encoder's hidden state with memory items, allowing memory to store typical features from seen samples for pattern matching. In con- trast, aim learn the direct relationship between input signals and output representations. For node i at time step I, we define the memory-queryinn process as follows:
(0) = xx W₂ + ba e MD
aj EE exp(Q MD
0 m -, a M[1]
where M[j] is the i-th memory item, and W,, and by are learnable parameters for input projection. Let ze be an output representation of expert e. Given the queried memory 0, R", we calculate the routing probability as shown below:
Te = g(a, (()): P. Eeelet...ees)"
Te
where E is the number experts. Since we use the similarity between output states and queried memory the routing probability, solving the routing problem induces memory learning of typical output representation and input-output relationship. We select the top-1 expert output as final output.
Routing Classification Losses To enable fine-grained routing that fits the regression problem, we adopt two classification losses: a classification loss to avoid the worst routing and another loss function find the best routing. Inspired by SMoEs, we define the worst routing avoidance loss as the cross entropy loss with pseudo label l, as shown below:
Loworst (p)
(dlo((pe)
(6)
if T. ú) is smaller than q-th quantile and p. = argmar(p))
4 = 1/(E-1) if L(y. @) is greater than q-th quantile and P. + argmar(p) otherwise

---

Published as a conference paper at ICLR 2024
Table 1: Experimental results on three real-world datasets with 13 baseline models and TESTAM.
The values in bold indicate the best, and underlined values indicate the second-best performance.
METR-LA
min
man
60 min
МАРР
MAF
MAPK
RMSK
Ligga POLSi
STOCK PARE
2,88
7.62%
3.47
7.24
9.57%
4.59
9.40
12.70%
ICKNN SEpeole MiTEE
2.77
1.38
7 1.00%
3.15
6.45
8.8015
3.64
7-59
10.50 50%
Graph Wa USSE-- Bile 2019
2.69
5.15
6.90%
3.07
6.22
8.37%
3.53
7.37
10.01
ST Certi Part
2.79
3.48
7.19%
1.16
0.50
8.5.5
3.61
7.60
10. 16%
GMAN CeNIeEEE AUAD)
2,80
7.41%
3.12
6.49
8.73%
7.35
10.07
MIGNN
6.17
NN WORER
3.33%
AGCRN
2.86
7.55%
3.25
6.57
8.99%
3.68
7.56
10.46%
OCRNN
2.85
3.54
7 1.50%
1.24
6.54
5.90%
3.73
765
59%
GTS (Shipe: KIND
2.65
5.20
6.80%
3.05
6.22
8.28%
3.47
7.29
9.83%
PM-Meirmee Lnvauai Purl
2.63
3.29
701%
1.0
6.29
N.42%
7.29
9.97%
MegaCRN
Jonne Call 20533
2.52
4.94
0.44%
2.93
5.06
7.96'
7.23
9.72%
2.54
6.42%
6.04
60 min
PEMS-BAY BAY
RMSE
МАРE
RMSE
RMSE
HA lL 2013
2,88
5.59
6.80%
2.88
5.59
6.80%
2.88
5.59
6,80%
STGOK Ci
1.36
1.96
2 2.90%
1.81
4.27
4.17%
2.49
5.69
5.79%
DCRNN LEPERI PiTES
1.38
2.95
2.905
1.74
3.97
3.90%
2.07
4.74
4,90%
Graph
WaveNNE ASSEE-O 2099
1.30
1.7%%
1.63
3.70
3.67%
4.52
4.63%
ST
89%
1.67
3.79
7889
4.50
4.5%%
MIGN
2 2.77%
1.44
4.30
$355
StemGNN
1.23
1.18
1.3%
N/A from
al.
2020
N/A From
(Cao al.
2020
AGORN
1.36
1888
293%
1.09
128099
1.98
4:19
2.60%
CCRNN Bi POSER
1.38
2.00
2.90%
1.74
3.87
1.90%
2.07
4.65
4.87%
GTS (Shome-- DE POME
1.34
2.83%
1.67
3.83
7996
1.98
4.56
4.59%
PM-Merti LeSTre RO2R
1282
2.81%
1.65
3.76
1.71%
4.49
4.54%
MegaCRN
3.56%
4.31%
10 min
30
60 min
MAE
RMSE
МАРE
MAE
RMSE
MAPE
MAE
RMSE
MAPE
HA FL 201K
7.63
11:96
11. 26%
7.63
11.96
31 15%2
7.63
11.96
$1.24 1..2%%%
STGCK FOEWER 201
6.09
9.60
31.8%%
6.91
10.99
30.24%
8.41
12.70
32.90%
Graph Graph-Wave
DCRNN
Wave
6.04
5.5%%
6.85
10.87
31.02%
7-15
11.66
11.07
31.61
22.52%
GMAN
6.09
9.49
26. 52%
6.64
10.55
30. 30%19%
7.05
11.28
$2.91%
MIGNN
5.86
9.26
24 80F6
6.49
10.44
23.239%
6:81
11.01
31 19%
StemCCNN
6.08
9.46
15.87%
6.85
10.80
31.25%
7.46
1 1188
35.11%
AGORN INTERMETT PORM
5.99
9.38
75.71%
6.64
10.63
29.81% 1%
6.99
11.29
37. 1.1
CCRNN SREEORR AUAGE
5,90
9.29
24.53%
6.68
10.77
29.93%
7.11
11.56
32.56%
GTS (ShIISEREI PiIPER
5.94
31 31.22%
MegoCRN
3,81
9.20
24.49%
6.43
10.33
28.92%
11.04
31,0255
TESTAM
5,84
9.23
25 35.36%
6.42
10.24
28.90% %
6.75
11.01
31.01%
where ý is the output of the selected expert, and is an enror quantile. If an expert is incorrectly selected, its label becomes zero and the unselected experts have the pseudo label 1/(E 1), which means that there are equal chances of choosing unselected experts.
We also propose the best-route selection loss for more precise routing. However, as traffic data are noisy and contain many nonstationary characteristics, the best-route selection is not an easy task. Therefore, instead of choosing the best routing for every time step and every node, we calculate node-wise routing. Our best-route selection loss is similar to that in Eq. 6 except that it calculates node-wise pscudo labels and the routing probability. and the condition for pseudo labels is changed from "L(y, 9) is greater/smaller than q-th quantile" to "L(y, f) is greater/smaller than (1 q)-th quantile." Detailed explanations are provided Appendix A
4 EXPERIMENTS
In this section, we describe experiments and compare the accuracy of TESTAM with that of existing models. We use three benchmark datasets for the experiments: METR-LA, PEMS-BAY, and EXPY- TKY. METR-LA and PEMS-BAY contain four-month speed data recorded by 207 sensors on Los Angeles highways and 325 sensors on Bay Arca, respectively (Li et al.) 2018). EXPY-TKY con- sists of three-month speed data collected from 1843 links in Tokyo, Japan. EXPY-TKY covers a larger number of roads in a smaller area, its spatial dependencies with many abruptly changing speed patterns are more difficult to model than those in METR-LA or TEMS-BAY METR-LA and PEMS- BAY datasets have 5-minute interval speeds and timestamps, whereas EXPY-TKY has 10-minute interval speeds and timestamps. Before training TESTAM, we have performed z-score normaliza- tion. In the cases of METR-LA and PEMS-BAS -AY. validation, and 20% for evaluation. For the EXPY-TKY. we utilize the first two months for training and validation and the last month for testing, as in the MegaCRN paper (Jiang et al. 2023)
70% of the data for training. 10% for

---

Published as a conference paper at ICLR 2024
4.1 EXPERIMENTAL SETTINGS
For all three datasets, we initialize the parameters and embedding using Xavier initialization. After performing a greedy search for hyperparameters, we set the hidden size d = e 32, the memory size m = 20, the number of layers | = 3, the number of heads K 4, the hidden size for the foed-forward networks his = 128, and the error quantile g = 0.7. We use the Adam optimizer with 31 = 0.9, B2 = 0.98, and c = 10-9, 9 as in Vaswani al. (2017). We vary the learning rate during training using the cosine annealing warmup restart scheduler (Loshchilov & Hutter 2017 according to the formula below:
Toue [Imin + (Irmar- (rmmn)) Tuarm [Immin + 2 (Irmar Iimm (1 cosl
For the first Twarm steps otherwise
Irate-
Tour r))
Tyree"
where Tour is the number of steps since the last restart. We use Twarm = -Tyreg = 4000, Iymin 10-7 for all datasets and set Immar = 3 * 10-3 for METR-LA and PEMS-BAY and Irmaz = 3 10-1 for EXPY-TKY. We follow the traditional 12-sequence (1 hour) input and 12-sequence output forecasting setting for METR-LA and PEMS-BAN BAY and the 6-sequence (1 hour) input and 6-sequence output setting for EXPY-TKY, in Jiang et (2023). We utilize mean absolute error (MAE) as a loss function and root mean squared error (RMSE) and mean absolute percentage error (MAPE) as evaluation metrics. All experiments are conducted using an RTX 3090 GPU.
We compare TESTAM with 13 baseline models: (I) historical average; (2) STGCN YYu et al. 2018
a model with GCNs and CNNs: (3) DCRNN (Li et al. 2018), model with graph convolutional recurrent units; (4)
WaveNet (Wu et al., 2019) with parameterized adjacency matrix; (5) STTN (Xu et al. 2020) and (6) GMAN (Zheng al. 2020). state-of-the-are attention-based models; (7) MTGNN (Wu et al. 2020), (8) StemGNN (Cao et al. 2020). and (9) AGCRN (Bai et al. 2020),
advanced models with an adaptive matrix; (10) CCRNN (Ye et al. (021), a model with multiple adaptive matrices; (11) GTS (Shang et al., 2021), model with graph constructed with long term historical data; and (12) PM-MemNet (Lee et al. 2022) and (13) MegaCRN (Jiang et al. 2023). state-of the-art models with memory units.
4.2 EXPERIMENTAL. RESULTS
The experimental results are shown in Table TESTAM outperforms all other models, especially in long-term predictions, which are usually more difficult. Note that we use the results reported in the respecive papers after comparing them with reproduced results from official codes provided by the authors. The models with learnable static graphs (Graph- Waapee,,,, MTGNN, and CCRNN) and dy- namic graphs (STTN and GMAN) show competitive performance, indicating that they have certain advantages. In terms of temporal modeling, RNN-based temporal models (DCRNN and AGCRN) show worse performance than the other methods in long-tern forecasting due to error- ancumuaccumllation of RNNs. Conversely, MegaCRN and MeMmeet maintained their advantages even in long-term forecasting by injecting memory augmented representation vector into the decoder. GMAN and StemGNN have performed worse with EXPY-TKY. indicating a disadvantage of the attention meth- ods, such long-tail problems and uniformly distributed attention (Jin et al. 2023).
As EXPY-TKY has a 6-9 times larger number of roads than the other two datasets, experimental results with EXPY-TKY highlight the importance of spatial modeling. For example, attention-based spatial modeling methods show disadvantages and the results of modeling with time-varying net- works (e.g., StemGNN) suggest that it could not properly capture spatial dependencies. In contrast, our model, TESTAM, shows its superiority to all other models, including those with learnable ma- trices. The results demonstrate that in-situ spatial modeling crucial for traffic forecasting.
4.3 ABLATION STUDY
The ablation study has two goals: to evaluate actual improvements achieved by each method, and to test two hypotheses: (1) in-situ modeling with diverse graph structures is advantageous for traffic forecasting and (2) having two loss functions for avoiding the worst route and leading to the best route is effective. To achieve these aims, we have designed a set of TESTAM variants, which are described below:
8

---

Published as a conference paper at ICLR 2024
Table Ablation study results across all prediction windows (i.e., average performance)
METR-LA
PEMS-BAY
EXPY-TKY
Ablation
MAE
RMSE
MAPE
MAE
RMSE
MAPE
MAE
RMSE
MAPE
w/o gating
3.00
6.12
8.29%
1.58
3.57
3,53%
6.74
10.97
29.48%
Ensemble
2.98
6.08
8.12%
1.56
3.53
3.50%
6.66
10.68
29.43%
worst-route avoidance only
2.96
6.06
8.11%
1.55
3.52
3.48%
6.45
10.50
28.70%
Replaced
2.97
6.04
8.05%
3.54
3.47%
10.62
29.20%
w/o TIM
2.96
5.98
8.07%
1.54
3.45
3.46%
6.44
10.40
28.94%
w/o time-enhanced attention
2.99
6.03
8.15%
1.58
3.59
3.52%
6.64
10.75
29.85%
TESTAM
2.93
5,95
7.99%
1.53
3,47
3,41%
6,40
10.40
28.67% %
w/o gating It uses only the output of the attention experts without ensembles or any other gating mechanism. Memory items are not trained because there are no gradient flows for the adaptive expert or gating networks. This setting results in an architecture similar to that of GMAN.
Ensemble Instead of using MoEs, the final output is calculated with the weighted summation of the gating networks and each expert's output. This setting allows the use of all spatial modeling methods but no in-situ modeling.
worst-route avoidance only excludes the loss for guiding best route selection. The exclusion of this relatively pharse-grained loss function is based on the fact that coarse-grained routing tends not to change its decisions after initialization (Dryden & Hoefler, 2022).
Replaced It does not exclude any components. Instead, it replaces identity expert with GCN- based adaptive expert, reducing spatial modeling diversity. The purpose this setting is to test the hypothesis that in-situ modeling with diverse graph structures is helpful for traffic forecasting.
w/o TIM It replaces temporal information embedding (TIM) with simple embedding vectors with- out periodic activation functions.
w/o time-enhanced attention It replaces time-enhanced attention with basic temporal attention as we described in Sec. 3.2
The experimental results shown in Table 2 connote that our hypotheses are supported and that TES- TAM complete and indivisible set. The results "w/o gating" and "ensemble" suggest that in-situ modeling greatly improves the traffic forecasting quality. The "w/o gating" results indicate that the performance improvement not due to our model but due to in-situ modeling itself since this setting lead to performance comparable to that of GMAN (Zheng et al. 2020). "worst-route avoidance only" results indicate that our hypothesis that both of our routing classification losses crucial for proper routing is valid. Finally, the results of "replaced," which indicate significantly worse performance even than "worst route avoidance only," confirm the hypothesis that diverse graph structures is helpful for in-situ modeling. Additional qualitative results with examples provided in Appendix
5 CONCLUSION
In this paper, we propose the time-enhanced spatio-temporal attention model (TESTAM), a novel Mixture-of-Ein of-Experts model with attention that enables effective in-situ spatial modeling in both re- curring and non-recurring situations. By transforming routing problem into a classification task, TESTAM can contextualize various traffic conditions and choose the most appropriate spatial mod- eling method. TESTAM achieves superior performance to that of existing traffic forecasting models in three real-world datasets: METR-LA, PEMS-BAY, and EXPY-TKY. The results obtained using the EXPY-TKY dataset indicate that TESTAM is highly advantageous for large-scale graph struc- tures, which are more applicable to real-world problems. We have also obtained qualitative results to visualize when and where TESTAM chooses specific graph structures. In future work, we plan to further improve and generalize TESTAM for the other spatio-temporal and multivariate time series forecasting tasks.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) N0.RS-2023-00211811 No. 202/R1A2C1004542) by the Institute of Information & Communications Technology Planning Evaluation (IITP) grants (No. 2020-0- Artifcclil Intelligence Graduate School Program, UNIST), and by the Green Venture R&D Program (No. S3236472), funded by the Ministry SMEs and Startups (MSS, Korea)
REFERENCES
Lei Bai, Lina Yao, Can Li, Xinazhi Wang, and Can Wang. Adaptive graph convolutional recur- rent network for traffic forecasting. In Advances in Neural Information Processing Systems, vol- ume 33, 2020.
Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, and Qi Zhang. Spectral temporal graph neural network for multivariate time-series forecasting. In Advances in Neural Information Processing Systems, volume 33, 2020.
Nikoli Dryden and Torsten Hoefler. Spatial mixture-of-experts In Advances in Neural Information Processing Systems, volume 35, 2022.
David Eigen, Mare Aurelio Ranzato, and Ilya Sutskever, Learning factored representations in a deep mixture of experts. In International Conference on Learning Representations, 2014.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23:120:1- 120:39, 2022.
Xu Geng, Yaguang Li, Leye Wang. Lingyu Zhang, Qiang Yang, Jieping Ye, and Yan Liu. Spatiotem- poral multi-graph convolution network for hide-hailing demand forecasting. Proceedings of the AAAI Conference Artificial Intelligence, 33(01):3656-366 2019.
David Ha, Andrew M. Dai, and Quoe V. Le. Hypereetorrks. In International Conference Learning Representations, 2017.
Renhe Jiang, Zhaonan Wang, Jiawei Yong, Puneet Jeph, Quanjun Chen, Yasumasa Kobayashi, Xuan Song, Shintaro Fukushima, and Toyotaro Suzumura. Spatio-temporal meta-graph learning for traffic forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 8078-8086, 2023.
Seungmin Jin, Hyunwook Lee, Cheonbok Park, Hyeshin Chu, Yunwon Tae, Jaegul Choo, and Sun- gahn Ko. A visual analytics system for improving attention-based traffic forecasting models. IEE Transaction on Visualization and Computer Graphics, 29(1): 102-1112, 2023.
Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus A. Brubaker. Time2vec: Learning a vector representation of time. CoRR, abs/1907.05322 2019. URL httt:://axxi...rrrr/////
abs abss/197..m 05321
C. Lee, Y. Kim, S. Jin, D. Kim, R. Maciejewski, D. Ebert, and S. Ko. A visual analyties system for exploring, monitoring, and forecasting road traffic congestion. IEEE Transactions on Visualiza- tion and Computer Graphics, 26(1 (6(11):3133-3146, 2020.
Hyunwook Lee, Seungmin Jin, Hyeshin Chu, Hongkyu Lim, and Sungahn Ko. Learning to re- member patterns: Pattern matching memory networks for traffic forecasting. In International Conference on Learning Representations, 2022.
Yaguang Li and Cyrus Shahabi. A brief overview of machine learning methods for short-term traffic forecasting and future directions. SIGSPATIAL Special, 10(1):3-9, 2018.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural net- work: Data-driven traffic forecasting. International Conference on Learning Representations, 2018.
10

---

Published as a conference paper at ICLR 2024
Iyy Loshchilov and Frank Hutter. Sgdr: tional Conference on Learning Representations, 2017.
Stochastic gradient descent with warm restarts. In Interna-
Mason McGill and Pietro Perona. Deciding how to decide: Dynamic routing in artificial neural networks. In Proceedings of the International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2363-2372, 2017.
Cheonbok Park, Chunggi Lee, Hyojin Bahng, Yunwon Tae, Seungmin Jin, Kihwan Kim, Sungahn Ko, and Jaegul Choo. ST-GRAT: A novel spatio-temporal graph attention networks for accurately forecasting dynamically changing road speed. In CIKM '20: The 29th ACM International Confer- ence on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020,
pp. 1215-1224. ACM, 2020.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Su- sano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. In Advances in Neural Information Processing Systems, pp. 8583-8595, 2021.
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. In International Conference on Learning Represen- tations, 2018
G. Ryan, A. Mosca, R. Chang, and E. Wu. At a glance: Pixel approximate entropy as a measure of line chart complexity. IEEE Transactions on Visualization and Computer Graphics, 25(01): 872 18,, 2019. ISSN 1941-0506. doi: 10. 10.11099TVCG
Chao Shang, Jie Chen, and Jinbo Bi. Discrete graph structure learning for forecasting multiple time series. In International Conference on Learning Representations, 2021.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, volume 30, 2017.
Eleni Vlahogianni, Matthew G. Karlaftis, and John Golias. Short-term traffic forecasting: Where we are and where we'ne 'e going. Transportation Research Part C: Emerging Technologies, 43:3-19, 2014. Special Issue on Short-term Traffic Flow Forecasting.
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling. In Proceedings of the International Joint Conference on Artifi- cial Intelligence, pp. 1907-1913, 2019.
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Con- necting the dots; Multivariate time series forecasting with graph neural networks. In P Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, 2020.
Mingxing Xu, Wenrui Dai, Chunmiao Liu, Xing Gao, Weiyao Lin, Guo-Jun Qi, and Hongkai Xiong. Spatial-temporal transformer networks for traffic flow forecasting. arXiv preprint arX1v:2001.02908, 2020.
Junchen Ye. Leilei Sun, Bowen Du, Yanjie Fu, and Hui Xiong. Coupled layer-wise graph convo- lution for transportation demand prediction. Proceedings of the AAAI Conference on Artificial Intelligence, 35(5):4617-462 2021.
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. 3634-3640. 2018.
Junbo Zhang, Yu Zheng, Dekang Qi, Ruiyuan Li, and Xiuwen Yi. Dan-based prediction model for spatio-tempora data. In Proceedings of the 24th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, SIGSPACIAL. '16, 2016.

---

Published as a conference paper at ICLR 2024
Qi Zhang, Jianlong Chang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. graph structure learning for traffic forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, 34(01):1177-1188) 2020.
Chuanpan Zheng, Xiaoliang Fan, Cheng Wang, and Jianzhong Qi. GMAN: A graph multi-attention network for traffic prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 1234-1241 2020.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M. Dai, Zhifeng Chen, Quoc V. Le, and James Laudon. Mixture-of-expere with expert choice routing, In Advances in Neural Information Processing Systems, volume 35, 2022.
12

---

Published as a conference paper at ICLR 2024
A ROUTING CLASSIFICATION LOss FUNCTION
In this section, provide detailed information the routing classification loss function. Both functions for worst-route avoidance and best-route selection are cross-entropy loss functions with different pseudo labels and routing levels. For the worst-route avoidance, compute fine-grained routing for each point of each road, as Dryden & Hoefler (2022) do. However, utilizing worst- route avoidance is suboptimal because experts have less opportunities be specialized for the best routing. Therefore, we adopt the best-route selection loss function for the routing problem. While designing the best-route selection loss functiin, we have two main concerns: 1) traffic data often shows severe fluctuation, which prevents model to consistently choose best-fit experts, and 2) the best-route selection itself more complex task than worst-route avoidance, resulting the model being hardly trained with fine-grained routing. To overcome those challenges, we have decided to construct node-wise best-route selection loss.
A.1 WORST-ROUTE AVOIDANCE Loss
For the worst-route avoidance loss function, have built our pseudo label Le as Eq. 6 In this section, describe how those labels chosen. Given prediction Y Nx and ground truth Y RRXI have point-wise distance L(Y, Y) RNXT between prediction and ground truth. Given point-wise distances and error quantile q. we say that routing of road ni at time is incorrect (or the worst routing) if (Vvo,t fv,t) greater than q-th quantile. Therefore, if Lyyn.e 9n.e) greater than q-th quantile, the pscudo label of the selected expert will be zero, and the labels for the other unselected experts will be 1/(E 1), where E is total number of experts. However, is smaller than g-th quantile, which means it is correctly routed and the worst route is avoided, the selected experts will have a pseudo label of one and the other experts will have pseudo label of zero. Formally, we can define pseudo label of expert as follows:
Lynn.t (n.e)
Lyy, 9) smaller than q-th quantile and Pe = argmar(p) Le 1/(E-1) 1) if Lyv, 9) is greater than q-th quantile and * argmar otherwise
A.2 BEST-ROUTE SELECTION LOSS
For the best-route selection loss function, we define node-wise pseudo labels by converting each condition of pseudo labeling for worst-route avoidance. For worst-route avoidance, we assume that the routing incorrect (i.e., the worst) if (y,,,tūn.e) is greater than g-th quantile. In the best-route selection, we define that the routing is correct (i.e., best) (ynnm) is smaller than g-th quantile; otherwise, its incorrectly routed, shown below:
L(y () is smaller than 1 - q-th quantile and pe = argmar(p) 1/(E 1) if L(y, ú) is greater than q-th quantile and p. + argmarr otherwise
4,-
Table 3: Computation time of the models with the METR- -AA dataset
Training time/epoch
Inference time
# of params
STGCN DCRNN
14.8 secs
16.70 secs
320k
122.22 secs
13.44 secs
372k
Graph- WaveNet
48.07 secs
3.69 secs
309k
GMAN
312.1 secs
33.7 secs
901k
MegaCRN
84.7 secs
11.76 secs
339k
TESTAM
150 secs
7.96 secs
224k
13

---

Published as a conference paper at ICLR 2024
B COMPUTATIONAL COST ANALYSIS
For the computational cost analysis, we use five models as baselines: 1) STGCN (Yu et al. 2018). the lightest model that utilizes GCNs and CNNs forecast 1-step future traffic condition; 2) DCRNN (Li al., 2018), well-known traffic forecasting model with graph-convolutional recurrent units; 3) Graph- WaveNet (Wu et al., 2019), a model that forecasts values by parallel computation with GCNs and CNNs; 4) GMAN (Zheng et al. 2020). a ppatio-temporal attention model for traffic forecasting, and 5) MegaCRN (Jiang et al. 2023 one of state-of-the models using GCRNN and memory network concepts.
We have investigated other models for comparison but decided to exclude them after careful con- siderations. For example, we have excluded MTGNN and StemGNN since they are an improved version of Graph-Waye WaveNet and have similar computational costs compared to Graph- WeveNet. Similarly, AGCRN, CCRNN, and GTS are excluded from baselines because they are variants of DCRNN, with few changes in computational costs. PM-MemNet and MegaCRN utilize sequence- to-sequence modeling with shared memory units; however, PM-MemNe experiences computational bottleneck with its stacked memory units, which requires times larger computational costs than those of MegaCRN.
Even though TESTAM utilizes three individual experts for the prediction, we emphasize that it has smaller number of parameters compared the other models due its small number of layers per each expert, which highly affects the computational costs, Furthermore, TESEAM only uses the encoder architecture of the transformer with a time-enhanced attention module that enables parallel computation, eliminating a computational bottleneck caused by the decoding process. As result, in terms of computational costs, TESTAM two times cheaper than the attention- GMAN), illustrating similar training time with DCRNN. Furthermore, in the inference phase, TESTAM shows the second fastest computation with the smallest number of the parameters.
model (i.e.,
C DETAILED EXPERIMENTAL RESULTS
Table presents experimental results under various environment settings. In the table, we pose three scenarios: (I), (H), and (E). Each scenario represents difficult conditions in making accurate forecasting. (I) is a set of isolated roads, chosen by considering spatial locations and quantitative analysis results on the adjacency matrix. (H) is the hard-to-predice roads, including intersections and the roads with high traffic fluctuations. We have determined roads for (H) by visually exploring the roads (for intersections) and by selecting the roads with top 10% entropy-based time-series complexity. (E) contains roads and time with sudden events, including accidents, traffic controls, or holidays (e.g., Christmas). In the experiments, TESTAM, which are selected as representative model for each spatial modeling method.
compare the performances of three baselines with
As shown in Table TESTAM outperforms other models in non-recurring situations (i.e., (E)) and the roads that have unique spatial and topological features (() and (E)). Especially, TESTAM con- sistently proves its superiority for the roads with spatially unique features, outperforming existing models from 4% to 7% in general. Among all of three baselines, we observe that the attention- based modeling method has better encoded spatial information for most of the hard-to-predict sce- narios. However, there are cases where the attention-basee based model fails, such as PEMS-BAY (I) or PEMS-BAY (H). From the perspective temporal modeling, the attention -based modeling method indicates better long-term forecasting performance, while CNN- and RNN-based modeling methods are advantageous in short-term forecasting. TESTAM, in contrast, despite of its temporal model- ing methods, it outperforms all the baselines in terms of both short-term and long-term forecasting cases. The results indicate that temporal information embedding and time-enhanced attention help the model to effectively transfer information from the input domain to the output domain.
14

---

Published as a conference paper at ICLR 2024
Table 4: Case-specific experimental results on three real-world datasets. The numbers in bold mean the best performance, and those underlined mean the second-bes best performance. (I) means isolated roads. (H) means hard-to-preent --e-diit roads, including intersections and the roads with high traffic fluc- tuations. (E) means the non-recurring circumstances, such as holidays or accidents.
15 min
30 min
60 min
METR-LA (I)
MAE
RMSE
MAPE
MAE
RMSE
MAPE
MAE
RMSE
MAPE
Graph- Graph-WaveNet [Wu et 2019
3.58
5.93
8.43%
3.90
6.74
9.59%
4.29
7.50
10.98%
GMAN (Zheng 2020
3.81
6.99
9.15%
4.03
7.48
9.97%
4.32
8.13
11.13%
MegaCRN Drang et 2023
3.54
5,88
8.46%
3.88
6.69
9.74%
4.35
7.67
11.39%
TESTAM
3.52
5.89
8.37%
3.80
6.59
9.43%
4.13
7.31
10.72%
min
METR-LA (H)
MAE
RMSE
MAPE
MAE
RMSE
MАРE
MAE
RMSE
MAPE
Graph-WaveNet (w al. 2019
4.07
6.74
11.75%
4.73
8.03
14.38%
5.48
9.34
17.20%
GMAN Zheng et al 2020
4.37
7.79
12.83%
4.86
8.75
14.77%
5.37
9.64
16.77%
MegaCRN (Jang et
4.02
6.68
11.61%
4.73
14.46%
9.72
17.67%
TESTAM
3,96
6.62
11.42%
4.51
7.75
13.57%
5.19
9.03
16.04% %
15 min
30 min
60 min
METR-LA (8)
MAR
RMSE
MAPE
MAF
RMSE
MAPE
MAR
RMSE
MAPE
Graph-Wave WaveNet (Wu et al 2019
4.22
7.14
13.08%
5.12
8.84
16.81%
6.17
10.62
21.19%
(MMAN (heng
4.45
7.67
14.49%
MegaCRN Drang 2023
4.03
6.91
12.37% %
4.96
8.75
16.10%
6.01
10.69
20.58%
TESTAM
4.11
7.09
12.54%
4.92
8.71
15.71%
5.89
10.46
19.69 %
15 min
min
60 min
PEMS-BAY
MAE
RMSE
MAPE
Graph- WaveNet GMAN Zhene et al. 2020
2.28
4.06
4.89%
6.65%
3.48
6.47
8.86%
2.51
5.12
5.65%
3.06
6.20
7.34%
3.55
7.10
8.90%
MegaCRN Uiang ot 2023
2.28
4.10
5.06%
2.92
5.58
7.27%
3.49
6.76
9.22%
TES
2.26
4.03
4.67%
2.86
5.24
6.45%
3.36
6.45
8.55
15
min RMSE
60 min
PEMS-BAY (H)
MAE
RMSE
MAPE
MAE
МАРE
MAE
RMSE
MAPE
Graph- Ware-Wa (Wn et al. 2019
2,46
4.42
5.44%
3.14
5.89
7.75%
3.81
7.23
10.50%
GMAN Zhenk et 2020
2.72
5.50
6.28%
3.34
6.76
8.42%
3.88
7.76
10.37%
MegaCRN Diang
2.47
4.44
5.62%
3.19
8.34%
3.82
7.48
10.76%
TESTAM
2,46
4.52
5.48%
3.10
5.75
7.62% %
3.69
7.16
9.96%
15 mün
30 min
60 min
PEMS BAY (F)
MAE
RMSE
MAPE
MAE
RMSE
MAPE
MAE
RMSE
MAPE
Graph- WaveNet (wu et
2019
2.64
4.73
6.22%
3,39
8.91%
4.20
7.78
12.69%
GMAN Zheng 2020
2.82
5.11
6.99%
3.55
6.68
9.72%
4.19
7.83
12.23%
MegaCRN Drang et a. 2026
2.61
4.65
6.25%
3.51
6.70
10.01%
4.24
8.14
13.11%
TESTAM
2.59
4.58
3.98%
6.15
8.82%
7.57
11.45 %
min RMSE
EXPY-TKY
MAE
RMSE
MAPE
MAE
RMSE
MAPE
MAE
MAPE
Graph- Traph-WaveNer WaveNet (W al. 2019
8.28
12.56
28.65%
9.40
14.23
33.69%
10.24
15.35
36.99%
GMAN Zheng et 2020
8.14
12.45
28.81%
8.75
13.43
31.11%%
9.26
14.20
32.62%
MegaCRN Dhang et
8.06
12.30
27.94%
13.71
32.22%
14.63
35.07%
TESTAM
7.87
12.26
26.95 %
8.60
13.44
29.69%
9.03
14.06
31.83%
10 min
30 min
60 min
EXPY-TKY (H)
MAE
RMSE
MAPE
MAE
RMSE
MAPE
MAE
RMSE
MAPE
Giraph-WaveNer (Wner al. 2019
8.45
12.79
30.97%
9.63
14.50)
36.07%
10.52
15.64
39.75%
GMAN Zhenk
8.30
12.63
31.08%
8.90
13.57
33.58%
9.44
14.33
35.19%
MegaCRN Diang et 2013
8.24
12.49
30.47%
9.21
13.92
34.60%
9.90
14.87
38.00%
TESTAM
8.06
12.48
29.08 %
8.81
13.67
31.88%
9.26
14.31
34.23%
10 min
min RMSE
60 min
EXPY-TKY
MAE
RMSE
MAPE
MAE
MAPE
MAB
RMSE
MAPE
Graph- WaveNet (W 2019
11.18
16.23
65.97%
11.93
17.21
71.71%
12.32
17.69
74.75%
GMAN Zheng et m220
11.02
16.06
64.93%
11.30
16.47
66.84 36.844%
11.49
16.74
67.10%
MegaCRR Unang et 2023
11.04
16.32
62.13%
11.67
17.07
67. 93%
11.94
17.37
71.43%
TESTAM
10.82
15.94
63.30%
11.28
65.51%
11.49
66.94%
15

---

Published as a conference paper at ICLR 2024
C.I QUALITATIVE EVALUATION
We perform a qualitative evaluation on TESTAM by visualizing the impact of our context-aware spatial modeling four different types of cases: 1) hard-to-predic roads with recurring patterns: 2) isolated roads (I) 3) roads with unique traffic patterns; and 4) roads with non-recurring patterns for evaluating the event awareness of TESTAM. We use the EXPY-TKY dataset, which contains complex urban road networks with various traffic patterns.
prediction 2021-12-14
berm prediction for read 1369 2021-12-15
40-
GPAAN
USAA
MagaCTI
MayoCHHN
06:00
09:02
10:00
21:00
02:00
00:22
12:00
35:00
Long-term prediction for road 1349
on 2021-12-16
Long term orediction for road 1369 on 2021-12-17
TES1AM
23
1151AM
Figure 2: Visualization of recurring pattern in haad-to-preddee road, Road 1349, from Dec 14th to Dec 17th. Road 1349 is a highway entrance located near Tokyo station. The locations of the roads are indicated in Fig.
Recurring Patterns on Hard-to-Predic Roads With the hard-to to-toodppeectt roads, we observe that previous models often fail to effectively encode the spatial and temporal correlation of the roads, as Fig. shows. For example, Road 1349 is a highway entrance located near the Tokyo station that has one of the largest traffic volumes in Tokyo and accordingly has severe fluctuation data. Because of the fluctuations and complex spatio-temporal dependencies of the roads, prior models have shown their limitations in spatial modeling. particular, Graph-WaveNet WaveNet (the green line in Fig. 2 which relies on the learnable static graph, fails to timely catch both the speed drop and rise in the red box of Fig. 2 However, GMAN and MegaCRN (the red and violet lines) properly model the ends of rush hour, but they also fail to predict the start of rush hour. Furthermore, McgaCRN exhibits noise-sensitive behavior on Dec. 1 14th (top-left) and 17th (bottom-right) in Fig.
Long term prediction for road 1165 2021-12-3
term prediction for read 1166 2021-12-14
30
TESTIM
Figure 3: Qualitative forecasting result analysis for spatially isolated roads (I). The locations of the roads are indicated in Fig.
Spatially Isolated Roads (I) When forecasting spatially isolated roads, the model should focus on the road itself, instead of referring to the other roads, which is less informative for prediction.
16

---

Published as a conference paper at ICLR 2024
However, since existing models have less consideration for the importance of self-referenciin, they fail to properly model rapid speed changes (c.g., noon in Fig. 3). or they become confused by in- formation the other roads (MegaCRN at 15:00 on Dec 3rd in Fig. 3), resulting in poor forecasting. In contrast, TESTAM accurately forecasts the rapid speed changes that occurred at 3:00 and 12:00, it enhances temporal modeling with identity experts, temporal information embedding, and time-enhanced attention layer.
Long-Lerm prediction for road 1111 on 2021-12-9
Long-lerm prediction for road 1111 on 2021-12-10
80-
TESTMA
20-
g
TESTAM
SMAAN
CHA
60-
MeqacIIN
MequCIN
40-
16-
00:00
03:00
05:00
09.00
12:00
15.00
00.00
02:00
03:00
12:00
15200
18:00
Figure 4: Qualitative forecasting result analysis for Road 1111, a highway ramp located in Shibuya, with unique traffic patterns. The locations of the roads are indicated in Fig.
Roads with Unique Traffic Patterns: The case of Highway Ramp In the EXPY-TKY, there are many roads showing unique patterns due to complex urban road networks and various traffic behaviors (e.g.. commuting and traveling). Road 1111 is highway ramp located in Shibuya and has unique patterns. One unique pattern of the road is that tends to have lower speed than 30km/h for all day, except 3:00 (the red box in Fig. 4. GMAN and Graph-WaveNet WaveNet cannot handle such unique patterns properly, failing to model the increasing speed at 3:00. On the other hand, MegaCRN predicts high-speed situation by its pattern- awareness of memory units, but it still fails to timely forecast it. In contract, TESTAM timely forecasts the traffic changes, revealing its superiority for modeling the unique behaviors of roads, shown in Fig.
crecesion and LISE 2323-1-44
aredieteess 2021-1244
preckciion reRd LIK 3328-12-0
Figure 5: Qualitative analysis results for Road 1196 (a metropolitan expressway) on Dec. 14th, where traffic control may occur because of heavy snow (red boxes)
Event-Aware Forecasting Case We qualitatively evaluate and show the importance context- aware spatial modeling in improving forecasting performance in various traffic conditions, such as sudden traffic control. Fig. visualizss the recurring and non-recurring traffic conditions caused by traffic control for heavy snow. Because of unexpected traffic controls, there exist sudden speed drops from morning to noon. In such non-recurring traffic condition, TESTAM shows better forecasting results due to its context-awareness. GMAN and MegaCRN partially capture the sudden changes but cannot make timely predictions.
D DETAILED SELECTION PROCEDURES AND LOCATIONS OF THE ROADS FOR CASE STUDY
In this section, we describe how we selected roads and time for each scenario, (I), (H), and (E). In the cases of (1) and (H), extracted the roads regardless of time.
17

---

Published as a conference paper at ICLR 2024
Figure 6: Location visualization for spatially isolated roads. The read circles are selected roads and the blue circles unselected ones. (Left) the 262 roads before filtering, (middle) newly selected roads from visual investigation, and (right) finalized list of 162 roads. The pink circle location of Road 1165 and 1166.
Spatially Isolated Roads (I) We have selected spatially isolated roads with two procedures: 1) investigating network topology: and 2) filtering out them by visually investigating their locations. From the investigation of network topology, connection. However, the random sampling process for building traffic dataset makes the network topology be sparse, which could not fully represent the connectivity in the real-world. Therefore, instead of directly utilize total 262 roads, we filter them out by visual investigation. After visual investigation, have filtered out 150 roads from the list and added 50 roads the list. Finally, have total 162 roads for (I), as shown in Fig. 6
found that there total 262 roads without any
Figure 7: Location visualization for hard-to-predict roads (left) and specific location of Road 1111 (middle) and 1349 (right). The green arrows indicate main traffic flows and directions for each road.
Hard-to-Predict Roads (H) For the hard-to-predie roads (H), conduct two selection pro- cesses, visual exploration and entropy-based time-series complexity. Inspired by Ryan et al.) (2019). entropy-baat -bsodd time-series complexity, which could measure noise level and unpredictability of changes in a series of points by estimating the probability of whether similar patterns will be repeated. From whole of 1843 roads, we have extracted the 184 roads with the top 10% largest entropy, which are the most unpredictable roads. Furthermore, we additionally insert 34 hard-to- predict roads, such intersections, ramp, and highway entrances by visually investigating roads. We finalized our list as shown in Fig. left.
Roads and Time with Sudden Events (E) The circumstances in (E) are especially important, since they are location- and time-specific events, and finding those samples requires tremendous ef- forts. In this paper, we have found the events with two strategies: 1) find specific time intervals (c.g., Christmas) of hard-to-predic roads (Fig. left); and find traffic controls and constructions that
18

---

Published as a conference paper at ICLR 2024
relatively easy to find than accidents and have credible source, including official announcements from the Metropolitan Expressway Co., Ltd" As a result, we have chosen the data at holiday inter- vals in hard-to-predict roads and the roads and intervals in construction sudden traffic controls for (E).
htttss///ww.shutooooooo.p jp/
19


Reviews:
[{"review_id": "v7BprPt9TZ", "reviewer": "Reviewer_orKb", "summary": "This paper proposes a novel deep learning model named TESTAM, which individually models recurring and non-recurring traffic patterns by a mixture-of-experts model with three experts on temporal modeling, spatiotemporal modeling with static graph, and dynamic spatio-temporal dependency modeling with dynamic graph.\nBy introducing different experts and properly routing them, TESTAM could better model various circumstances, including spatially isolated nodes, highly related nodes, and recurring and non-recurring events.\nThe evaluation is conducted with three open datasets and compares the proposed method and existing methods.", "questions": "Once the information pointed out in Weakness is clarified, the validity of the proposed method will become clearer.\nCould the authoer please clarify these points?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of the proposed method, which utilizes multiple experts models and switches between them adaptively to deal with various traffic conditions, is reasonable and interesting for forecasting urban traffic affected by various factors.\n\nThe proposed method's level of prediction compared to existing techniques is demonstrated by utilizing three datasets and comparing it to a number of existing methods.\n\nThe structure of the paper is easy to understand and the individual contents are clearly described.", "weaknesses": "Table 1 shows that the error is small in many conditions compared to existing methods, but the difference is not large, and it is not clear what the significance of this error is.\n\nIt is also not clear where the computational cost of the proposed method in model building and forecasting stands in comparison to other methods. In my opinion, this is important information, especially how the proposed method relates in terms of computational cost to methods that do not have very large differences in accuracy.\n\nIt is not clear how the proposed method is able to deal with the \"various circumstances\" described in the intro of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GyztradZTq", "reviewer": "Reviewer_gqdu", "summary": "This paper presents a mixture of experts model for traffic forecasting where each expert model uses a different spatial correlation learning model. In addition, a time-enhanced attention is added to help model the temporal correlation on top of the usual attention module. Experiments haven been conducted on three datasets, two of which of commonly used benchmark datasets. There are 13 baseline models, including one from 2022 and one from 2023.", "questions": "See the weak points.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. While the core of the proposed model is based on mixture of experts, there are new designs to the components including an extra time-enhanced attention module and an enhanced routing classification loss considering the best-routing selection. \n\n2. The paper is very well written overall and is easy to follow. \n\n3. There are quite a few baseline models including some of the latest.", "weaknesses": "The main issue of the paper is perhaps the relatively weak experimental results. As shown in Table 1, the proposed model TESTAM has very similar performance to MegaCRN (Jiang et al., 2023) and sometimes PM-MemNet (Lee et al., 2022). \n\nThere are no model training time results. I wonder if TESTAM is even slower than MegaCRN and/or PM-MemNet to train. \n\nThe discussion of the results have focused on comparisons with GMAN and StemGNN which are from 2020 and are relatively \"old\" methods already. \n\nOverall, the importance of the proposed techniques has not been shown with strong evidence. \n\nMinor presentation issues:\nTypo: \"is noisy sensitive\", \"ahve\", \"Let define\";\nCheck \"where f(X(t), θ)\" below Equation 1;\nGrammar: \"traffic data is noisy and contains many non-stationary, make the best route selection hard.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "piurepO2PF", "reviewer": "Reviewer_FrvU", "summary": "In this paper, authors introduce TESTAM, a novel Time-Enhanced Spatial-Temporal Attention Model to realize accurate traffic forecasting. TESTAM is a Mixture-of-Experts model that incorporates attention mechanisms, allowing real-time spatial modeling under both normal and abnormal circumstances. By reformulating the routing problem as a classification task, TESTAM can adapt to different traffic conditions, enabling the selection of appropriate spatial modeling methods.", "questions": "Q1- Q2: Please refer  to W3 and W4.\nQ3. In memory-based gating networks, why the similarity between the model's input and output is used as the routing probability? This lacks a certain theoretical foundation (or some analysis) and explanation.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- S1. Considering the complex dependency on road networks, the use of an expert mixture model in spatial-temporal forecasting is an innovative design.\n\n- S2. The paper provides a clear categorization of spatial modeling types for spatial-temporal forecasting, and the selection of different spatial models (identity experts, adaptive experts, attention experts) is highly representative.", "weaknesses": "W1. There are many typos, and some incorrect writings can seriously mislead the readers:\n   - '...choose spatial modeling methods (i.e., expert) properly; and' on page one.\n   - $f(X^{(t)},\\theta)$ should be replaced by 'g(X^{(t)},\\theta)' on page three.\n   - In the formulas at the bottom of page five, the superscript (k) in $W_{q}^{(k)}$ and $W_{k}^{(k)}$ appears to be meaningless and should be replaced by $W_q$ and $W_k$.\n   - On the fifth page, $\\[\\tau^{(t+1),...,\\tau^{(t+T)}\\]$ should be changed to $\\[\\tau^{(t+1),...,\\tau^{(t+T')}\\]$.\n   - In formula 5 on the sixth page, 'larger than q-th quantile' should be corrected to 'smaller than q-th quantile.'\n   - In the seventh page, the sentence 'excludes the relatively coarse-grained loss function only' should have 'only' removed.\n   - The functions such as 'Concat' and the activation functions 'relu' should be written as upright letters.\n   - Some formulas have equation numbers, while others do not.\n\nW2. The effectiveness of Time-enhanced Attention has not been verified through ablation experiments, and readers cannot determine whether it truly enhances the model's predictive performance.\n\nW3. The distinction between best-route selection and worst routing avoidance is not clear. When introducing best-route selection, the authors mention node-wise routing and node-wise pseudo-label but haven't explained them. Additionally, changing 'smaller than q-th quantile' to 'smaller than (1-q)-th quantile' alone is considered best-route selection, but the reason for this change has not been provided.\n\nW4. Experimental results:\n   - The experimental results are inconsistent. In Table 2, the results for TESTAM in the ablation experiment are for a 30-minute forecast, but they do not match the results in Table 1. Additionally, I noticed that in Table 2, the PEMS-BAY dataset performs better in the 'w/o gating' condition than the complete TESTAM model in Table 1.\n\n   - The magnitude of the performance improvement is minimal. Table 1 shows that many metrics only slightly improve in terms of percentiles, at the cost of three times the computational time and memory for building three expert models.\n\nW5. The title doesn’t well match with this study itself. Actually, this work mostly focuses on traffic forecasting, but the title   suggesting a new general Spatio-Temporal Attention Model with MoE seems much more general and broader.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9fPC8HjMmW", "reviewer": "Reviewer_SSH9", "summary": "The authors point out that accurate traffic forecasting requires capturing two distinct traffic patterns, recurring and non-recurring. To accomplish this goal, they propose TESTAM, a mixture-of-expert model with three heterogeneous experts responsible for different aspects of spatio-temporal modeling. By properly routing the experts, the model learns to handle various circumstances adaptively. Experimental results on three benchmark datasets verify the effectiveness of TESAM.", "questions": "1. Why the proposed method is abbreviated as TESTAM?\n2. What does \"in-situ\" mean in the context of this manuscript?\n3. The temporal information embedding and time-enhanced attention seem novel for traffic forecasting. Do they affect the performance? Some discussion or ablation study is needed.\n4. In the abstract, the authors state that TESTAM can model isolated nodes, highly related nodes, and recurring and non-recurring events. Can you provide some experimental results and analysis for this claim, e.g. case study?\n\nMinor:\n1. Please add comments to the meaning of results in bold and underlined in Table 1.\n2. Descriptions and analysis of the qualitative examples in the supplementary material are missing.\n3. Why are some equations in section 3 not numbered? I believe some of them are proposed by the authors, e.g. time-enhanced attention, and thus should be numbered.\n4. Please provide an explicit formula for the best-route loss, at least in the supplementary material.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The manuscript is well-organized and easy to follow.\n2. The introduction of mixture-of-expert models increases the capacity of spatial modeling for traffic forecasting, and to some extent, increases the interpretablility.", "weaknesses": "1. In terms of spatial and temporal modeling, this manuscript does not seem to bring new techniques.\n2. There are multiple sets of terminologies describing the same thing without an explicit connection. For example, when categorizing traffic patterns, both \"recurring vs non-recurring\" and \"normal vs abnormal\" (in Conclusion) are used; when describing the three experts, \"temporal modeling, spatio-temporal modeling with static graph, spatio-temporal modeling with dynamic graph\" (in Abstract) and \"identity experts, adaptive experts, attention experts\" (in Figure 1). Clarifying these key concepts will make the presentation clearer.\n3. Only the overall metrics for predicting accuracy are reported. It is unclear what recurring and non-recurring patterns that TESTAM captured while other methods miss. Some in-depth experimental analysis (see Q4 for details) will better support the claim of the first contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)