Learning to Remember Patterns: Pattern Matching Memory Networks for Traffic Forecasting
Authors: Hyunwook Lee
Seungmin Jin
Hyeshin Chu
Hongkyu Lim
Sungahn Ko
Venue: ICLR 2022 Poster (2022)
Keywords: Traffic Forecasting
Deep Learning

Abstract: Traffic forecasting is a challenging problem due to complex road networks and sudden speed changes caused by various events on roads. Several models have been proposed to solve this challenging problem, with a focus on learning the spatio-temporal dependencies of roads. In this work, we propose a new perspective for converting the forecasting problem into a pattern-matching task, assuming that large traffic data can be represented by a set of patterns. To evaluate the validity of this new perspective, we design a novel traffic forecasting model called Pattern-Matching Memory Networks (PM-MemNet), which learns to match input data to representative patterns with a key-value memory structure. We first extract and cluster representative traffic patterns that serve as keys in the memory. Then, by matching the extracted keys and inputs, PM-MemNet acquires the necessary information on existing traffic patterns from the memory and uses it for forecasting. To model the spatio-temporal correlation of traffic, we proposed a novel memory architecture, GCMem, which integrates attention and graph convolution. The experimental results indicate that PM-MemNet is more accurate than state-of-the-art models, such as Graph WaveNet, with higher responsiveness. We also present a qualitative analysis describing how PM-MemNet works and achieves higher accuracy when road speed changes rapidly.

Published as a conference paper at ICLR 2022
LEARNING TO REMEMBER PATTERNS: PATTERN MATCHING MEMORY NETWORKS FOR TRAFFIC FORE-
CASTING
Hyunwook Lee, Seungmin Jin, Hyeshin Chu, Hongkyu Lim, and Sungahn Ko
Ulsan National Institute of Science and Technology {gusdnr 0ocenr09116, 0916, skyjin, hyeshinchu, Limhongkyul219, sako)
ABSTRACT
Traffic forecasting is a challenging problem due to complex road networks and sudden speed changes caused by various events on roads. Several models have been proposed to solve this challenging problem, with focus on learning the spatio-temporal dependencies of roads. In this work, we propose a new perspec- tive for converting the forecasting problem into pattern-matching task, assuming that large traffic data can be represented by a set of patterns. To evaluate the va- lidity of this new perspective, Pattern- Matthenn Memory Networks (PM-MemNet), which learns to match input data representative patterns with key-value memory structure. We first extract and cluster representative traffic patterns that serve as keys in the memory. Then, by matching the extracted keys and inputs, PM-MemNet acquires the necessary in- formation on existing traffic patterns from the memory and uses for forecasting. To model the spatio-temporal correlation of traffic, we proposed novel memory architecture, GCMem, which integrates attention and graph convolution. The ex- perimental results indicate that PM-MemNet is more accurate than state-of-the-art models, such Graph WaveNet, with higher responsiveness. We also present a qualitative analysis describing how PM-MemNet works and achieves higher ac- curacy when road speed changes rapidly.
design novel traffic forecasting model called
INTRODUCTION
Traffic forecasting is a challenging problem due to complex road networks, varying patterns in the data, and intertwined dependencies among models. This implies that prediction methods should not only find intrinsic spatio-temporal dependencies among many roads, but also quickly respond to irregular congestion and various traffic patterns (Lee et al. 2020) caused by external factors, such accidents weather conditions (Vlahogianni et al., 2014 Li & Shahabi 2018 Xie et al.) 2020 Jiang Luo| 2021). To resolve these challenges and successfully predict traffic conditions, many deep learning models have been proposed. Examples include the models with graph convolutional neural networks (GCNs) (Bruna et al.) 2014) and recurrent neural networks (RNNs) (Siegelmann & Sontag 1991). which outperform conventional statistical methods such as autoregressive integrated moving average (ARIMA) VVlahogianni et al. 2014 Li et al. 2018). Attention- based models, such as GMAN (Zheng et al. 2020), have also been explored to better handle complex
dependency of traffic data. Graph WaveNet (Wu et al. 2019) adopts diffusion process with self-learning adjacency matrix and dilated convolutional neural networks (CNNs), achieving state- of-the-art performance. Although effective, existing models have weakness in that they do not accurately forecast when conditions are abruptly changed (e.g., rush hours and accidents).
In this work, we aim to design novel method for modeling the spatio-temporal dependencies of roads and to improve forecasting performance. To achieve this goal, we first extract representative traffic patterns from historical traffic data, as find that there similar traffic patterns among roads, and a set of traffic patterns can be generalized for roads with similar spatio-temporal features. Figure I shows the example speed patterns (left, 90-minute window) that we extract from many
"Corresponding Author

---

Published as a conference paper at ICLR 2022
Speed
Spood
Mal!! praali
mn/
mn
Figure 1: (left) Multiple traffic data with similar pattern, (right) extracted representative pattern
different roads and a representative traffic pattern (right time series). With the representative pat- terns, we transform the conventional forecasting problem into pattern-matching task to find out which pattern would be the best match for the given spatio-temporal features to predict future traffic conditions. With insights from the huge success of neural memory networks in natural language processing and machine translation (Weston et al. 2015 Sukhbaatar et al., 2013 Kaiser et al. 2017] Madotto et al. 2018). we design graph convolutional memory networks called GCMem to man- age representative patterns spatio-temporal perspective. Lastly, we design PM-MemNet, which utilizes representative patterns from GCMem for traffic forecasting. PM-MemNet consists of an coder and a decoder. The encoder consists of temporal embedding with stacked GCMem, which generates meaningful representations via memorization, and the decoder is composed of gated recurrent unit (GRU) with GCMem. We compare PM-MemNet to existing state-of-the-art models and find that PM-MemNet outperforms existing models. We also present a qualitative analysis in which we further investigate the strengths of PM-MemNet in managing a traffic pattern where high responsiveness of model to abrupt speed changes desired for accurate forecasting.
The experimental results indicate that PM- MemNett achieves state-of-ther ohethe--rtt art performance, especially long-term prediction, compared existing deep learning models. To further investigate the char- acteristics of PM--MeNNet, we conduct an ablation study with various decoder architectures and find that PM-MemNet demonstrates the best performance. We also investigate how the number of representative patterns affects model performance. Finally, we discuss the limitations of this work and future directions for neural memory networks the traffic forecasting domain.
The contributions of this work include: (1) computing representative traffic patterns of roads, (2) design of GCMem to manage the representative patterns, (3) presenting a novel traffic prediction model. PM-MemNer that matches and uses the most appropriate patterns from GCMem for traffic forecasting, (4) evaluation of PM-MemNet compared to state-of-the-arr models, (5) qualitative anal- ysis identify the strengths of PM-MemNet, and (6) discussion of limitations and future research directions.
RELATED WORK
2.1 TRAFFIC FORECASTING
Deep learning models achieve huge success by effectively capturing spatio-temporal features in traffic forecasting tasks. Past studies ahve shown that RNN-based models outperform conventional temporal modeling approaches, such ARIMA and support vector regression (SVR) (Vlahogianni ct al. 2014: Li et al. 2018). More recently, many studies have demonstrated that attention-
models (Zheng al.
2020 Park al. 2020) and CNNs (Yu al. 2018 Wu al. 2019) record
2

---

Published as a conference paper at ICLR 2022
better performance in long-term period prediction tasks, compared RNN-based models. In terms of spatial modeling, Zhang et al. (2016) propose a CNN-based spatial modeling method for Eu- clidean space. Another line of modeling methods, such as GCNs, using graph structures for manag- ing complex road networks also become popular. However, there are difficulties in using GCNs in the modeling process, such as the need to build adjacency matrix and the dependence of GCNs on invariant connectivity in the adjacency matrix. To overcome these difficulties, a set of approaches, such as graph attention models (GAT's, have been proposed to dynamically calculate edge impor- tance (Park et al., 2020). GWNet (Wu et al., 2019) adopts self-adaptive adjacency matrix to capture hidden spatial dependencies in training. Although effective, forecasting models still suffer from in- accurate predictions due to abruptly changing road speeds and instability, with lagging patterns in long-term periods. address these challenges, we build, save, and retrieve representative traffic patterns for predicting speed rather than directly forecasting with an input sequence.
2.2
NEURAL MEMORY NETWORKS
Neural memory networks are widely used for sequence-to-segeee to-sequnnce modeling in the natural lan- guage processing and machine translation domains. Memory networks are first proposed by Weston et al. (2015) to answer query more precisely even for large datasets with long-term memory. Mem- ory networks perform read and write operations for given input queries. Sukhbbatar et al. (2015) introduce end-to-end memory networks that can update memory in an end-to-end manner. Through the end-to-end memory learning, models can be easily applied to realistic settings. Furthermore, by using adjacent weight tying, they can achieve recurrent characteristics that can enhance generaliza- tion. Kaiser et al. (2017) propose novel memory networks that can be utilized in various domains where life-long one-shot learning is needed. Madotto et al. (2018) also introduce Mem2Seq, which integrates the multi-hop attention mechanism with memory networks. In our work, we utilize mem- ory networks for traffic pattern modeling due to the similarity of the tasks and develop novel graph convolutional memory networks called GCMem to better model the the given traffic patterns.
correlation of
3 PROPOSED APPROACH
In this section, we define the traffic forecasting problem, describe how we extract key patterns in the traffic data that serve as keys, and introduce our model, PMM-MeNNet.
3.1 PROBLEM SETTING
To handle the spatial relationships of roads, utilize road network graph. We define road network graph as G = (V, AA,, where V is set of all different nodes with |/ = N, E is a set of edges representing the connectivity between nodes, and A E NN is weighted adjacency matrix that contains the connectivity and edge weight information. An edge weight is calculated based on the distance and direction of the edge between two connected nodes. As used in the previous approaches (Li et al. 2018 Wu et al., 2019 Zheng et al. 2020 Park et al. (020)), we calculate
edge weights via the Gaussian kernel as follows: A.,
), where dist:y is the distance between node and node j and a is the standard deviation of the distances.
exp
dist
Prior research has formulated a traffic forecasting problem as a simple spatio-temporal data predic- tion problem (Li et al., 2018 Wu al., 2019 Zheng al., 2020) Park et al. (2220) aiming to predict values in the next 7 time steps using previous 7" historical traffic data and an adjacency matrix. Traffic data at time is represented by a graph signal matrix, X E IR- xd.., where dn is the num- ber of features, such as speed, flow, and time of the day. In summary, the goal of the previous work to learn mapping function f(-) to directly predict future graph signals from T" historical input graph signals:
[Xg (1T''T++
(0) f()) Xgg (L-1)
Xg (L+T)
The goal of this study is different from previous work in that we aim to predict future traffic speeds from patterned data, instead of utilizing input Xg directly. We denote P C RTT as a set of repre- sentative traffic patterns, p TP as traffic pattern in P., and d X P [0, oc) as distance
3

---

Published as a conference paper at ICLR 2022
Smilentty Deettturiton of Estracted Pattern
Simber strbbtioon Clustared Pattern Set
siniartyy
Figure 2: (a) Example daily patterns. Each part between the red dash lines is the speed patterns sliced by given time window, (b) cosine similarity distribution of the original pattern set with class imbalance, and (c) the clustered pattern set.
function for pattern matching. Detailed information about traffic pattern extraction will be diseussed in the next subsection. Our problem is to train the mapping function f(-) as follows:
Xg (1 T +1
Xg(0) (0)
d.).K-NN,
PM
ft-)
[xg ...... Xg ((T)),
where P. = {pi..... pk J is set of k-nearest neighboring traffic patterns of node in time t, with a distance function d. Note that pj the j-th nearest neighbor pattern.
3.2 KEY EXTRACTION FROM TRAFFIC PATTERNS
Analyzing the traffic data, we find that the data has repeating patterns. In traffic data, the leading and trailing patterns have a high correlation, even during short-term periods. To take advantage of these findings, in model, we build a representative pattern set, P. First, from historical data, compute an average daily pattern, which consists of 288 speed data points (total 24 hours with 5- minute intervals) for each vertex given window size 7", as shown in Figure (a). At this stage, [PP] = N x T J. After we collect the patterns, investigate similarity distributron of the extracted pattern set, IP, via cosine similarity (Figure (b)) and find that the pattern set has a biased distribution with too many similar patterns (i.e.. class imbalance). Since such class imbalance causes memory ineffectiveness in accurate mem- ory retrieval and gives biased training results, we use clustering custering-based undersampling (Lin et al.) 2017] with cosine similarity. For example, if pattern p and pattern p' have a cosine similarity larger than 8, they are in same cluster. We utilize the center of each cluster as representative pattern of that cluster. After undersampling by clustering, we have balanced and representative pattern set, P. shown in Figure (c), which we use keys for memory access. Table presents the effect of different 5 and P on forecasting performance.
V. We then extract pattern p by slicing the daily patterns with 288
3.3 NEURAL MEMORY ARCHITECTURE
Conventionally, memory networks have used the attention mechanism for memory units to enhance memory reference performance, but this attention-only approach cannot effectively capture spatial dependencies among roads. To address this issue, we design new memory architecture, GCMem (Figure 3 (b)), which integrates multi-layer memory with the attention mechanism (Madotto et al., 2018 and graph convolution (Bruna et al., 2014). By using GCMem, model can capture both pattern-level attention and graph-aware information sharing via GCNs.
To effectively handle representative patterns from a spatio-temporal perspective, we utilize sev- eral techniques. First, we use modified version of the adjacent weight tying technique in MemNN (Sukhbaatar et al. 2015 Madotto et al., 2018), which has been widely used for sentence memorization and connection searches between query and memorized sentences. SSkhbaatar et al. 20033;: Madotto et al. (2018) propose the technique to capture information from memorized sen- of sentence-level attention. However, their methodology only learns pattern
tences by making similarity and cannot handle spatial dependency. Using the same method in traffic forecasting is in- sufficient since handling graph structure is essential for building spatial dependencies of roads. In
4

---

Published as a conference paper at ICLR 2022
60Mee
PM
GEMee
Represrtaatvo mamory
Decoder
(4)
(b)
Figure 3: (a) Overall architecture of PM-MemNet where = 3. Dashed line means adjacent weight tying. (b) GCMem architecture with GCNs (gray blocks) and (c) representative memory selection among k-nearest patterns for X; with k=3, dj = d(pj, X²).
order to consider the graph structure while maintaining the original centence-level attention score, we use an adjacency matrix, a learnable adaptive matrix, and attention scores for the GCNs. By main- taining pattern-level attention, the model takes advantage of both pattern-level information sharing and adjacent weight tying (Madotto et al., 2018). As result, adjacent memory cells can effectively retain attention mechanisms while considering graph structure.
Figure 3 (a) and (b) shows PM-MemNet and our proposed graph convolution memory architecture. The memories for the GCMem are embedding matrices M {M', ........ "MM,...,..++1) M where M' gFixd,, For memory reference, utilize pattern set IP, which contains the extracted traffic patterns
(in Section 3.2 and k-Nearest Neighbor (k-NN) with distance function, d(-). For each input traffic
data X, E R xd. in node i, based on k-nearest patterns and the distance function, we build representative memory M.- as follows:
M Softmax(-dy)
(1)
where m (p;) is the memory context for py and layer dj g(X,, and Softmax e We summarize representative memory selection process in 3
After calculating representative memory, GCMem calculates hidden state h Nxdx with previ- ous hidden state 11'-1 and representative memory M' and M (Figure 3 (b)). For each represen-
tative memory M). PM-MemNet calculates pattern-level attention scores, of as follows:
1/ (M))
a = Softmax
(2)
where 1/-1 E Rd the previous hidden state of node i. We denote an attention matrix as C¹ E N x N where ale is the entry in the i-th row and j-th column of C.. Then, given memory unit M E Nxds for the next layer 1 + 1, we calculate output feature, with graph convolution as shown below:
h
M Ā W W M++C))) (W) M A + WL A, Σ
(3)
where A E N is an adjacency matrix and Ā So LU E, is learnable matrix, which captures hidden spatio-temports connections (Wu et al., 2019 Shi et al. 2019). E,, Eg e R Nxd, are learnable node embedding vectors and W 12 is learnable matrix. We use ReLU as an activation function. Then, we update hidden states by = h 1--1 + o'_ Before we update the hidden states, we apply batch normalization on o'.
3.4 ENCODER ARCHITECTURE
The left side of Figure 3 (a) shows the proposed encoder architecture. PM-MemNet handles traffic patterns and their corresponding memories. Although the patterns in the memory provide enough
5

---

Published as a conference paper at ICLR 2022
information for training, there other types of data that can also be used for prediction. For ample, different roads have their own patterns that may not be captured in advance (e.g., the unique periodicity of roads around an industry complex (Lee et al. 2020)). In addition, there would be some noise and anomalous patterns due to various events (e.g., accidents), which are not encoded when patterns grouped. As such, provide embedding for the time (i.e., emb) and noise (i.e., N₂) that the encoder uses to generate input query Ao for GCMem. Specifically, for the time series T T +l,.... f] and noise N; X; P1, we calculate its representation, h2 shown below:
h = emb(T) + W,,,,,
(4)
where emb and Wn represent learnable embedding for the time of day and learnable matrix, respectively. Note that emb(T) E Ihth and Wn E Rd4 XT PM-MemNet updates hy using L-layer GCMem. We use the output of the encoder 1.2 E R.N xda as an initial hidden state in the decoder.
3.5 DECODER ARCHITECTURE
As shown in Figure 3 (a), we build our decoder architecture using single layer GRU, followed by the L-stacked GCMem. For each step, the decoder predicts the value at time step using the previous prediction gr-1 and GRU hidden state h Initially, g is zero matrix and ho is encoder hidden state ht The hidden states from the GRU will be an input for the L-stacked GCMem. Similar to encoder architecture, GCMem updates hidden states with attention and GCNs. Instead of using updated hidden states for prediction directly, we utilize layer-level self-attention in the decoder. Specifically, for each GCMem layer / and node i, we calculate energy, ex,1 using the previous hidden state h and memory context M\ as shown below:
(h) (M/WTT
ddn
where da is the hidden size and W E mathbbld xd. is learnabee matrix. Then, with the output feature of of each layer and node i, we can predict 0 as:
1o W proj: 0 Σ
Wrroj:
where W Wproj& E Rd. Xdout is a projection layer and a; = Softmax(e,,) Note that is equivalent to hidden state of GRU cell, h2 Using layer-level attention, PM-MemNet utilizes information from each layer more effectively.
EVALUATION
In this section, we explain the experiments conducted to compare PM-MemNet to existing mod- els in terms of accuracy. We use two datasets for the experiment-METR--L. and NAVER-Seou B METR-LA contains 4-month speed data from 207 sensors of Los Angeles highways (Liet al. 2018) NAVER- Scoul has 3-month speed data collected from 774 links in Scoul, Korea. As NAVER- data covered the main arterial roads in Scoul, it can be considered a more difficult dataset with many abruptly changing speed patterns compared to METR-LA data. Both datasets have five-minute in- terval speeds and timestamps. Before training the PM-MemNet, we have filled out missing values using historical data and applied z-score normalization. We use 70% of the data for training, 10% for validation, and the rest for evaluation, as Li et al. (0118 have done their work.
Wu et al. (2019); Zheng et al. (2020)
4.1 EXPERIMENTAL SETUP
In our experiment, we use 18 sequence data points as model input (T" = 18, one and half hours) and predict the next 18 sequences. For the N--N, we utilize the cosine similarity function for the similarity measurement and set k = 3. To extract P. utilize the training dataset and initialize parameters and embedding using Xavier initialization. After performing a greedy search among d, [16, 32, 64, 128), T. [1, 2, 3, 4), and various [P]] values, we set d, as 128, T. as 3, and

---

Published as a conference paper at ICLR 2022
Table I: Experimental Results for HAVER-Seoul and METR-LA datasets
STGCN
GCRNN
DCRSN
ASTGCN
GWNet
PM-MemNel
MAE
6.34
5.28
4.63
5.09
4.57
15min
MAPE
18.24
16.86
14.49
15.23
15.35
16.14
14.86
16.98
14,43
RMSE
9.32
7.78
6.92
7.18
7.12
7.44
7.24
6.72
MAF.
7.16
6.13
5.50
5.73
5.67
5.71
5.26
5.35
5.04
30min
MAPE
20.15
20.05
17.37
18.17
18.38
18.78
16.16
17.47
16.34
NAVER-Scoul
RMSF:
10.18
9.51
8.83
9.03
8.80
8.73
8.13
8.67
7.86
MAE
8.22
7,08
6.77
6.38
6.22
5.24
60min
MAPE
23.37
23.44
20.42
20.95
21.09
20.37
16.97
17.89
16.94
RMSE
11.54
11.13
10.89
10.58
10.06
9.58
8.77
8.39
MAF
9.24
7.79
8.06
7.14
6.86
6.76
5.87
5.58
5,40
90min
MАPE
26.40
26.08
22.93
21.83
17.89
18.18
RMSE
12.77
12.17
12.86
11.43
10.69
10.32
9.33
9.09
8.68
MAE
4.23
2.93
2.61
2.39
3.25
2.72
2.60
15min
MAPE
9.76
7.76
6.59
6.73
6.67
9.27
1.14
7.67
7.06
5.28
MAF:
4,80
3,00
3.22
3,08
3,01
3.80
3.12
3.14
3.02
METR-LA
RMSE
8.34
7.29
6.63
6.32
6.29
7.59
6.34
6.54
6.28
60min
MAPE
14.04
13.68
11.13
11.50
10.73
13.69
10.30
10.10
9.88
MAF
6.65
5,58
5.41
4.23
4,06
4.97
3,85
3.71
3,64
10.74
RMSF
10.97
10.52
10.47
8.79
8.58
9.71
8.12
7.71
7.74
IP| R 100 with 5 = 0.7 and 0.9 for NAVER-Seoul and METR-LA, respectively. presents the experimental results with different memory and pattern sizes (i.e., [P]) We apply the Adam optimizer with learning rate of 0.001 and use the mean absolute error (MAE) as loss function.
We compare PM-MemNet to the following baseline models: (1) multilayer perceptron (MLP); (2) STGCN (Yu et al. 2018), which forecasts future step using graph convolution and CNNs; (3) Graph Convolution Recurrent Neural Network (GCRNN); (4) DCRNN et 2018). a sequence-
to-sequence model that combines diffusion convolution in the gated recurrent unit; (5) AST-
GCN (Guo et al. 2019), which integrates GCN, CNN, and spatial and temporal attention; (6) Graph- WaveNet (etNet) (Wu et al., 2019), which forecasts multiple steps once by integrating graph convolution and dilated convolution; and (7) Graph Multi-Attention Network (GMAN) (Zheng et al., 2020) which integrates spatial and temporal attention with gated fusion mechanism. GMAN also predicts multiple steps at once. To allow detailed comparisons, we train the baseline models to fore- cast the next 90 minutes of speeds at 5-minute intervals, given the past 18 5-minute interval speed data points. We train the baseline models in an equivalent environment with PyTorct] using the public source codes and settings provided by the authors. Detailed settings, including the hyperpa- rameters, are available in Appendix A. To ease the performance comparison with previous work, we provide additional experimental results with a commonly used setting, where = T = 12. in Table/olin Appendix. We also present qualitative analysis of our method for NAVER-Scoul dataset in Appendix A.4 Our source codes are available on GitHut?
4.2 EXPERIMENTAL RESULTS
Table displays displayss the experimental results for NAVER-Seoul and METR-LA for the next 15 minutes, 30 minutes, 60 minutes, and 90 minutes using mean absolute error (MAE), mean absolute percentage error (MAPE), and root mean square егтог (RMSE). Note that Table and Figure 5 in Appendix show more detailed experimental results and error information. As Table shows, PM-MemNet achieves state-of- of-the-art performance for both datasets. Specifically, PM-MemNet yields the best performance in all intervals with NAVER- -eouu data, while it outperforms other models in the long- term prediction (i.e., 60 and 90 minutes) with METR-LA data.
In both datasets, we find interesting observations. First, RNN-based models perform better than other models for short term periods (i.e., 15 minutes), but they show a weakness for long-term pe- riods. This problem occurs in sequence-to- sruuence-to-sequence RNNs due to error accumulation caused by its auto-regressive property. Compared the RNN-based models, PM-MemNet shows a lesser perfor-
https://pytorch.orr
https://github. com/ http:/ccoo/mooooooooooooo H
7

---

Published as a conference paper at ICLR 2022
Table 2: Ablation study result. Note that 'Ours' means PM-MemNet.
SimpleMem
Deadeer
(IF
25
MAE
4.57
5.72
4.56
4,67
4.72
4,66
4.59
15min
MАPE
18.18
14,84
14.77
RMSE
6.72
8.79
6.71
6.83
6.87
6.89
6.73
MAE
5.04
5.84
5.06
5.19
5:22
5.21
NAVER-Seuee
RMSE
7.86
9.24
7,90
8.02
8.03
8.18
7.97
MAE
5.24
6.38
3.32
5,47
5.52
5.51
3.36
RMSE
8,39
10.08
8.51
8.69
8.70
8.92
8.67
MAE
5,40
6.95
5.55
5.70
5.72
5.74
5.52
90min
MAPE
17.44
23.89
17.99
18.73
18.73
17.76
10.88
9.10
9.31
MAE
2.66
3.01
2.63
2,68
2.68
2,67
2.68
15min
MAPE
7.10
7.11
7,09
7.13
5.31
5.35
5.31
MAE
3.02
3:27
3.01
3.06
3.36
3.06
3.04
30min
MAPE
8.49
9.20
8.46
8.56
8.59
8.67
8.51
METR-LA
6.32
6.36
MAE
3,40
3.72
1:41
3.46
3:47
3.49
3.45
60min
MAPE
9.88
10.94
9.88
10.02
10:07
10.34
9.87
7.31
MAT
3.64
4.09
3.05
3.71
3:73
3.75
3.69
50min
MAPE
10.74
12:25
10.85
10.87
10.98
11.30
10.63
mance decrease, although it had decoder based RNN architecture. This is because of GCMem and pattern data, which make representations for traffic forecasting more robust than other methods. observe that all models suffer from decreased accuracy due more complicated urban traffic patterns and road networks than those in METR-LA. In spite of these dif- ficulties, PM-MemNet proves its efficiency with representative traffic patterns and the memorization technique. This result strengthens our hypothesis that the traffic data used for traffic forecasting can be generalized with small number of pattern even when the traffic data are complex and hard to forecast.
In the of NAVER-Seoul,
4.3 ABLATION STUDY
We further evaluate our approach by conducting abla- tion studies (Table 2 Table in Appendix). First, we check whether GCMem can effectively model spatio- temporal relationships among road networks. To this end, we compare the performance of PM-MemNet to that of SimpleMem, a simplified version of PM- MemNet in which the memory layer depends only on pattern-level attention and does not consider any graph- based relationship. By comparing PM-MemNet and SimpleMem, observe that SimpleMem shows nearly 10% decrease in performance, which shows the impor- tance of a graph structure in modeling traffic data. In the same context, according to Table and Table 2 Simple- Mem has lower accuracy than previous models, such as DCRNN, which considers graph structure.
Pffectel (P] NAuTD-Somal
Pffect METB--A
1353
1225
Next, combine GCMem with various decoder chitectures to investigate how the performance of GCMem changes with different decoder combinations. The results indicate that all decoder combinations achieve state-of-the-an performance for both NAVER- Seoul and METR-LA datasets. This result implies that GCMem effectively learns representations, even with a simple decoder architecture, such single layer RNN. Also, from the perspective of error accumu- lation, we notice that PM-MemNet outperforms the CNN decoder even for the 90-minute prediction. In contrast, the lay-red RNN decoder shows lower performance than the CNN decoder. How- ever, when comparing the RNN decoder to existing ANN-based models, RNN decoder can predict more precisely for the long-term prediction. These results also show that GCMem is robust solu-
120
2000
3000
51000
300
7000
2000
Figure Effect of P| in NAVER-Seoul (left) and METR-LA (right).
8

---

Published as a conference paper at ICLR 2022
tion for long-term dependency modeling. Modifying the GCMem layer depth, we discover that PM- MemNet generates sufficiently accurate predictions with a single memory layer (i.e., PM-MemNet w/L=1). Although PM-MemNee needs three-layered GCMem to achieve the highest accuracy, can still consider deploying the model with a lightweight version while ensuring its state-of-the-art performance.
Next we analyze how different memory sizes (i.e., different pattern numbers) affect the performance. In this experiment, we set [PP] 3, 100, 2000, and 3000. Note that we set [F] as 3 to simulate an extreme case, where PM-MemNet has very rare memory space. Figure I shows the experimental results. We can observe that PM-MemNet records the best performance with 100 traffic patterns; attaching large [PP to model does not always allow the best forecasting performance. We provide the time consumption of the models in Table ii Appendix.
5 LIMITATIONS, DISCUSSION, AND FUTURE DIRECTIONS
This work is the first attempt to design neural memory networks for traffic forecasting. Although effective, as shown in the evaluation results, there are limitations to this approach. First, we extract traffic patterns memorize them in advance; however, it is possible that the extracted patterns are redundant, even after strict filtering. As such, there is need to find important patterns and optimize the number of memory slots. For example, future study could investigate how to learn and extend the key space during the training phase (Kaiser et al. (2017). Second, the learning of embedding matrices in PM-MemNet proceeds only based on referred patterns. Because there are no further losses to optimize memory itself, patterns not referred to are not trained. This training imbalance among memories is of interest, the model cannot generate meaningful representations from rare patterns. A future study may research not only how such a representation imbalance affects performance, but also design a loss function to reduce the representation gap between rare and frequent events. Third, we use cosine similarity in this work, but it may not be an optimal solution, since it causes mismatching with noisy traffic data. Also, the optimal window size for pattern matching remains to be addressed. A future study may focus on approaches to effectively compute the similarity of traffic patterns. Designing a learnable function for the distance measurement is one possible direction. Finally, we show that the model can effectively forecast traffic data with small group of patterns. This implies new research direction for comparing results and learning methods that work with sparse data, such meta learning and few or zero shot learning (Kaiser al. 2017).
6 CONCLUSION
In this work, we propose PM-MemNet, novel traffic forecasting model with graph convolutional memory architecture, called GCMem. By integrating GCNs and neural memory architectures, PM- MemNet effectively captures both spatial and temporal dependency. By extracting and computing representative traffic patterns, we reduce the data space to small group of patterns. The experimen- tal results for METR-LA and NAVER- -coull indicate that PM-MemNet outperforms state-of-the- art models. It proves our hypothesis "accurate traffic forecasting can be achieved with a small set of representative patterns" is reasonable. We also demonstrate that PM-MemNet quickly responds to abruptly changing traffic patterns and achieves higher accuracy compared to the other models. Lastly, we discuss the limitations of this work and future research directions with the application of neural memory networks and a small set of patterns for traffic forecasting. In future work, we plan to conduct further experiments using PM-MemNet for different to investigate whether the insights and lessons from this work can be generalized to other domains.
domains and datasets
ACKNOWLEDGEMENTS
This work was supported by the Korean National Research Foundation (NRF) grant (No. 2021RIA2C100454)) and by the Institute of Information & Communications Technology Plan- ning & Evaluation (IITP) grants (No. 2020-0-01336-Artffeee
Intelligence Graduate School Pro- gram (UNIST), No. 2021-0-01 198-1ITT R&D Innovation Voucher Program), funded by the Korea government (MSIT). This work was also partly supported by NAVER Corp.

---

Published as a conference paper at ICLR 2022
REFERENCES
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. In International Conference on Learning Representations, 2014.
Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based spatial- temporal graph convolutional networks for traffic flow forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 922-929, 2019.
Weiwei Jiang and Jiayun Luo. Graph neural network for traffic forecasting: A survey, CoRR, abs/2101. 11174, 2021. https::///axx
abs / 22121111111111111....1
Lukasz Kaiser, Ofir Nachum, Aurko Roy, and Samy Bengio. Learning to remember rare events. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
C. Lee, Y. Kim, S. Jin, D. Kim, R. Maciejewski, D. Ebert, and S. Ko. A visual analytics system for exploring, monitoring, and forecasting road traffic congestion. IEEE Transactions on Visualiza- tion and Computer Graphics, 26(1 1):3133-3146, 2020.
Yaguang Li and Cyrus Shahabi. A brief overview of machine learning methods for short-term traffic forecasting and future directions. SIGSPATIAL Special, 10(1):3-9, 2018.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural net- work: Data-driven traffic forecasting. International Conference on Learning Representattons, 2018.
Wei- Wei-ChaoIee Lin, Chih-Fong Tsai, Ya-Han Hu, and Jing- Sian-Shang Jhang. Clustering- -ased undersampling in class-imbalanced data. Information Sciences, 409-410:17-266 2017.
Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. Mem2seq: Effectively incorporating knowl- edge bases into end-to-end task-oriented dialog systems. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pp. 1478-1478, 2018.
Cheonbok Park, Chunggi Lee, Hyojin Bahng, Yunwon Tae, Seungmin Jin, Kihwan Kim, Sungahn Ko, and Jaegul Choo. ST-GRAT: A novel spatio-temporal graph attention networks for accurately forecasting dynamically changing road speed. In CIKM '20: The 29th ACM International Confer- ence on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020, pp. 1215-1224. ACM, 2020.
Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Two-stream adaptive graph convolutional net- works for skeleton-based action recognition. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20. 2019, pp. 12026-12035, 2019.
Hava T. Siegelmann and Eduardo D. Sontag. Turing computability with neural nets. Applied Math- ematics Letters, 4(6):77 80, 1991.
Sainbayar Sukhbaatar, arthur szlam, Jason Weston, and Rob Fergus. End-to-end memory networks. In Advances Neural Information Processing Systems, volume 28, 2015.
Eleni I. Vlahogianni, Matthew G. Karlaftis, and John C. Golias. Short-term traffic forecasting: Where we are and where we' re going. Trunsportation Research Part C: Emerging Technologies, 43:3-19, 2014. Special Issue on Short-term Traffic Flow Forecasting.
Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. 3rd International Confer- on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling. In Proceedings of the International Joint Conference on Artifi- cial Intelligence, pp. 1907-1913, 2019.
10

---

and Published as a conference paper at ICLR 2022
Peng Xie, Tianrui Li, Jia Liu, Shengdong Du, Xin Yang, and Junbo Zhang. Urban flow prediction from spatiotemporal data using machine learning: A survey. Information Fusion, 59:1-12, 2020.
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep
learning framework for traffic forecasting. In Proceedings of the International Joint Conference on Artificial Intelligence, pp. 3634-3640, 2018.
Junbo Zhang, Yu Zheng, Dekang Qi, Ruiyuan Li, and Xiuwen Yi. Dan-based prediction model for spatio-temporal data. In Proceedings of the 24th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, SIGSPACIAL '16, 2016.
Chuanpan Zheng, Xiaoliang Fan, Cheng Wang, and Jianzhong Qi. GMAN: A graph multi-attention network for traffic prediction. In Proceedings of the AAAI Conference on Artificial Intellignce, pp. 1244-1241, 2020.
A APPENDIX
A.I DETAILED EXPERIMENTAL SETUP
MLP Multi-layer perceptron with two hidden layers; each layer contains 64 units and rectified linear unit (ReLU) activation.
STGCN Tempora Graph Convolutional Networks Yu et al. 2018 STGCN models spa- tial features using spectral-based graph convolution with Chebyshev polynomial approximation and temporal features using gated CNNs. In our experiment, like the original paper, we set STGCN with graph convolution kernel size K 3, temporal convolution kernel size of K₁ 3, and three layers with 64, 16, and 64 hidden units, respectively.
DCRNN Diffusion Convolutional Recurrent Neural Networks al. 2018). DCRNN is the model that integrates diffusion convolution with RNNs, especially the GRU. In our experiment, both en- coder and decoder contain two recurrent layers with 64 hidden units and maximum steps of random walks set as (i.e., K 3)-
GCRNN Graph Convolutional Recurrent Neural Networks. GCRNN is variant of DCRNN that integrates simple bidirectional graph convolution and seuuencc-to-sequeeee architecture. Both en- coder and decoder contain two recurrent layers with 64 hidden units and maximum steps of random walks set as 3 (i.e., K = 3).
ASTGCN Attention based Spatio-Temporal Graph Convolutional Networks (Guo al. 2019). ASTGCN models spatio-temporal features by attention, GCNs, and CNNs. For the GCNs, AST- GCN utilizes spectral-based graph convolution with Chebyshev polynomials K and kernel size of CNNs K, = 3. Furthermore, all layers have the same hidden unit size of 64.
GWNet Graph WaveNet (Wu et al. 2019). GWNet combines graph convolution and dilated causal convolution. In our experiment, to cover the input sequence length, we utilize 12 layers with dilation factors of 1, 2, 1, 2, ............... 1, Also, following original paper, we set the diffusion step F1, E2 E Nx10
GMAN Graph Multi-Attention Network (Zheng al. 2020). GMAN handles both spatial and tem- poral features using attention and pretrained node embedding. In our experiment, we utilize the same setting as the original paper, that is, three layers for both encoder and decoder with eight attention heads and 64 hidden units (8 hidden units per attention head).
PM-MemNet Pattern Matching Memory Networks. Both encoder and decoder have three GCMem layers with 128 hidden units and IP| s 100. Note that the decoder also has GRU cell before GCMem with 128 hidden units. For the k-NN, we utilize & = 3 and cosine similarity. Also, for the graph convolution, set the diffusion step K 2.
In the experiment, we utilize the MAE loss function and Adam optimizer with a learning rate of - for training. The learning rate reduces to for every 10 epochs, starting 30 epochs, until it reaches 6 For each model, we trained 100 epochs, with early termination by monitoring the validation error.
11

---

Published as a conference paper at ICLR 2022
A.2
DETAILED RESULTS WITH 18-STEP INPUT AND OUTPUT SEQUENCES
Table 3: Detailed experimental results with state sfate-of-the-art models with 18-step input and prediction
sequences.
Dataset
Metric
HA
MLP
STGCN
GCRNN
DCRNN
ASTGCN
GWNet
GMAN
PM-MemNet
MAE
6.34
5:28
4.63
4.87
4,36
3.09
4,91
5.20
4.57
15min
MAPE
18.24
16.86
14.49
15.23
15.35
16.14
14.86
16.98
14.43
RMSE
9.32
7.78
6.92
7.18
7.12
7.44
7.24
8.32
6.72
MAE
7.16
6.13
5.30
5.73
3.71
5,26
5.04
30min
MAPE
20.15
20.05
17.37
18.17
18.38
18.78
16.16
17.47
16.34
RMSE
10.18
9.51
8.83
9.03
8.80
8.73
8.13
8.67
7.86
MAE
7.70
6.68
6.16
6.24
6.12
6.01
5.43
5:43
5.18
45min
MAPE
21.81
22.01
19:15
19.85
20.06
19.73
16.70
17.72
16.81
NAVER-Scoul
RMSE
10.89
10.48
9.95
9.99
9.61
9.28
8.53
8.84
8.24
MAE
8.22
7.08
6.77
6.58
6.40
6.22
5.55
5.48
5.24
60min
MAPE
23.37
23.44
20.42
20.93 95
21.09
20.37
16.97
17.89
16. 94
RMSE
11.54
11.13
10.89
10.58
10.06
9.58
8.77
8.94
8.39
MAE
8.73
7.44
7.40
6.87
6.63
6:46
5.68
5.33
$.30
75min
MAPE
24.91
24.76
21.62
21.91
21.93
20.93
17.31
18.05
17.10
RMSE
12.17
11.67
11.85
11.02
10.39
9.91
9.01
9.03
8.50
MAE
9.24
7.79
8.06
7.14
6.86
6.76
5.87
5.58
5,40
90min
MAPE
26.40
26.08
22.93
22.86
22.74
21.83
17.89
18.18
17.44
RMSE
12.77
12.17
12.86
11.43
10.32
9.33
9.09
8.68
MAE
4.23
2.93
2.61
2.39
3:25
2.72
2.56
15min
MAPE
9.76
7.76
6.59
6.73
6.67
9.27
7.14
7.67
7.06
RMSE
7.46
5.81
5.19
5.12
5.10
6.28
5,20
5.77
5.28
MAE
4,80
3,66
3.22
3.08
3,01
3.50
3.12
3.14
3,02
30min
MAPE
11.30
10.00
8.39
8.72
8.42
11.28
8.66
8.79
8.49
RMSE
8.34
7.29
6.63
6.32
6.29
7.59
6.34
6.54
6.28
MAE
5.32
4.17
3.78
3.46
3.34
4.20
3.39
3:24
45min
MAPE
12.71
11.93
9.84
10.26
9.71
12.64
9.63
9.55
9.33
RMSE
9.18
8..37
7.76
7.12
7.08
8.40
7,06
7,00
6.87
METR LA
MAE
5,80
4,69
4.31
3.74
3.60
4.49
3.58
3.48
3,40
60min
MAPE
14.04
13.68
11.13
11.50
10.73
13.69
10.30
10.10
9.88
RMSE
9.86
9.24
8.71
7.71
7.65
8.94
7.53
7.30
7.24
MAE
6.25
3.17
4.35
4:01
4.73
3.73
3.60
3.53
75min
MAPE
15.26
15.47
12.45
12.59
11.68
14,60
10:90
10.56
10.31
RMSE
10.46
9.95
9.62
8.21
8.15
9.34
7.87
7.52
7.51
MAE
6.65
3.38
3.41
4.23
4,06
4.97
3.83
3.71
3.64
90min
MAPE
16.37
17.08
13.76
13.49
12.53
15.53
11.39
11.00
10.74
RMSE
10.97
10.52
10:47
8.79
8.58
9.71
8.12
7.71
7.74
Experiment
MAE,
Experiment MAE, on METR-LA
HA
HA
GCRNN
GCRNN
DCAN
6
DCNNN
ASTGON
ASTGON
GMAN
GMAN
Ours
Our
7
6-
5-
Figure 5: Performance result plots with error bars with NAVER Seoul (left) and METR-LA (right) datasets.
12

---

Published as a conference paper at ICLR 2022
Table 4:
The computation times for each model with METR-LA
Computation Time
DCRNN
GWN
GMAN
PM.MemNer
SempleMMen
CNN Decoder
RNN Decoder
PM-MERNer
Thining time per opoch (sec) Inference Time (sec)
$6.30
6,00
9.34
14.9
12.14
10.2
4.26
10.77
A.3 DETAILED ABLATION STUDY RESULTS
Table 5: Detailed ablation study result.
Dutaset
Metric
Ours
SimpleMem
CNN Decoder
RNN Decoder
Ours (L=1)
Ous (1F)
Ous [FF
>>> 1000)
MAE
4.57
5.72
4.56
-.67
4.72
4.56
4.59
15mia
MAPE
14.43
18.18
14.40
14.84
14.98
14.77
14.48
MAE
5.04
5.84
5.06
5.19
5.22
5.21
5.00
30mia
MAPE
16.34
18.86
16.36
16.87
16.97
16.91
16.41
8.02
MAE
5.18
6.10
5.38
5.43
5.43
45min
MAPE
16.81
19.98
16.98
17.61
17.66
17.72
16.97
MAE
5.32
3.37
5.32
5.33
5.36
60min
MAPE
16.94
21,42
17.19
17.91
17.97
18.05
17.19
MAE
5,30
5,39
5.36
5.62
5.42
75min
MAPE
17.10
22.52
17.38
18,20
18.23
18.32
17.40
RMSE
8.50
10.48
8.62
8.86
8.85
9.09
8.80
90min
MAPE
17.44
23.89
17.99
18:73
18.63
18.73
17.76
RMSE
8.68
10.88
8.82
9,10
9.05
9.31
8.94
2.67
15min
MAPE
7.06
8.03
6.98
7.10
7.11
7.09
7.13
RMSE
5.28
5.94
5.32
5.31
5.31
5.35
5.31
30min
MAPE
9.20
8.56
8.67
8.51
RMSE
6.28
6.68
6.36
6.32
6.27
6.36
6.32
45mia
MAPE
9.33
10.18
9.30
9.44
9.47
9.67
9.31
METR-LE BATTR-LA
RMSE
6.87
7.28
6.94
6.93
6.86
7,00
6.93
3.49
RMSE
7.24
7.70
7.28
7,31
7.25
7,39
7.32
MAE
3.53
3.91
3,52
3.59
3.60
3.62
3.58
RMSE
7.51
8.06
7,50
7.59
7.53
7.67
7.59
MAE
3.64
4,09
3.65
3.71
3.73
3.75
3.69
RMNF:
7.74
8.38
7.71
7.81
7:35
7.91
7.81
13

---

Published as a conference paper at ICLR 2022
minutes prediction NAVER-SeoL)
(8)
(A)
40-
Ground Truth
Our
20
GWheet
DCRNN
23 00:00
29 06 00
20 1299
20 15:00
21 02:00
60 minutes prediction NAVER-Seoul
100
80-
(D)
80-
(C)
G1 02:00
C1 06 dD
0C 12:00
DL. 18:00
00:00
Figure 6: NAVER-Scoul speed prediction visualization for (upper) 90-minute forecasting and (lower) 60-minute forecasting. Red circles show the interval with abrupt speed changing, which successfully detected by PM-MemNet.
A.4 QUALITATIVE EVALUATION
We present qualitative analysis using the NAVER-Seone dataset, which contains complex road network and the dynamic traffic conditions of urban areas. In this analysis, we evaluate whether approach effectively patterned traffic data for the traffic forecasting problem. If the approach is valid, we expect PM-MemNet to be more accurate in predicting difficult trailing patterns due to the high correlation between leading and trailing patterns. To verify our expectations, we visualize the long-term traffic prediction results, as shown in Figure o (top), where PM-MemNet predicts the end of congestion at 12:00 PM (B) more accurately. Also, in 7:00 AM (A), PM-MemNet quickly catches up on the unexpected peak speed (this continues for about 30 minutes, and PM- PM-MemNet efficiently catches up within minutes). From Figure (bottom), we can see that PM-MemNet predicts speed drop and its recovery more accurately compared to the other models. For example, GMAN and GWNet predict the speed drop carlier than the actual speed drop, and DCRNN predicts the speed drop later than the real speed drop, shown in (C). However, PM-MemNet predicts the occurrence of slowdowns on time, even for long-term traffic prediction (D). Overall, with represen- tative traffic patterns and memorization, PM-MemNet effectively handles abruptly changing traffic conditions, even for long-term predictions.
Next, analyze how PM-MemNet effectively matches input patterns with memorized patterns. Figure presents three examples, each of which has two time-series lines. The red line denotes the current time step /. Before the current time step The blue and orange line means input and corresponding matched pattern, respectively. After the current time step The blue and orange line means ground truth and prediction results, respectively. In each example, we find that the model effectively retrieves memorized pattern that matched the input sequence well for prediction. Note that all examples show how PM-MemNet operates for difficult roads and time with rapid speed drops.
14

---

Published as a conference paper at ICLR 2022
roxl groundd suuth mutehed peed
Irith meiicnd pood
muthted and
32
22-
Figure 7: Sample input (ground truth, blue) and matched representative pattern for prediction (or- ange). the red line means the current time step t.
A.5 EXPERIMENT WITH 12 SEQUENCE SETTING
Table 6: Experimental results with state-of-thee of-tth--rtttttttt-- models common 12 sequence prediction set-
ting. For SEMS--AY. d = 0.9 and P Re 100.
Dataset
Metric
HA
MLP
STGCN
GURNN
DCRNN
ASTGCN
GWNet
GMAN
PM-MemiNet
MAF
6.22
5.28
4.69
5.00
4.92
4.91
4,50
4.90
4.52
15min
RMSE
9.54
7.78
7.02
7.34
7.20
7.23
6.61
7.64
6.67
30min
MAPE
21.77
19.92
17.61
18.86
18.59
17.14
16.71
16.61
16.10
DAVER-Seoul
9.53
MAE
7.90
7.10
6.39
6.75
6.35
3.86
3.44
5.38
5.30
RMSE
12.28
11.16
11.01
10.75
10:29
9.16
8,57
8.71
8.51
5.82
5.39
Avg.
MAPE
21.77
18.84
16.67
17.76
17:49
16.79
15.44
16.48
14.97
8.96
MAE
3.75
2.92
2.38
2,30
2.73
3.07
2.69
2.81
2.65
7.71
RMSE
7.31
5.82
5.74
5.51
5.27
8.23
5.16
5.55
5.29
4.37
30min
MAPE
11.85
9.93
9.57
9.00
8.65
10.34
8.43
8.35
8.42
7.31
METR-LA
MAE
5.45
4.70
4.39
3.81
3.58
4,42
3.53
3.46
3.46
60min
MAPE
15.04
13.64
13.35
10.05
RMSE
10.39
9.26
9.40
8.16
7.60
8.73
7.31
7.37
7.29
2399
Avg.
MAPE
12.02
10.07
9.96
9.13
8.72
10.32
8.42
8.61
8.27
RMSE
8.66
7.23
7.18
6.26
MAE
2.26
1.49
1.42
1.34
1.33
1.55
1,30
1,36
1.27
15min
MAPE
5.03
3.15
2.79
3.44
2.75
RMSE
5.18
3.24
3.08
2,82
2.79
3.17
2.74
2.88
2.80
2.72
2.01
30min
MAPE
6.18
4.55
4.49
3.87
3.78
4.66
3.67
3.71
3.66
RMSE
6.26
4.70
3.65
PEMS-BAY
MAE
3.52
2.80
2.53
2.08
2.01
2.57
1.95
1,90
1.95
60min
MAPE
8.19
RMSE
7.94
6.34
5.92
4.72
4.59
5.27
4,52
4.40
4.51
2.76
2.02
1.97
1.55
Avg.
MAPE
6.29
4.61
4.28
3.75
3.64
4.54
3.67
3.59
3.62
RMSE
4.57
3.44
15


Reviews:
[{"review_id": "hHWOXFqvCoj", "reviewer": "Reviewer_xDZJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The idea of using both extracted traffic patterns and spatial dependencies is interesting.\n\nWeakness:\n\nOne of the major concerns is that the paper is not well presented, and critical information on the model and problem setup is missing. \n- In the problem setup, the definition of traffic patterns is not clear. Based on Figure 1, it seems to be a sequence of traffic speeds along the timeline. \n- Section 3.2 is difficult to follow. It says that traffic patterns are average speed on each road. Intuitively, the traffic situation of each road - could be very different. Sometimes, even on the same road, the traffic situation in different directions could be very different. \n- The author's claim that the pattern set is imbalanced is unconvincing since the only evidence provided is the one arbitrary pattern in Figure 2.\n- Cosine similarity is a continuous value. However, the authors treat them as discrete bins without mentioning the motivation.\n- The input, output of each component is unclear. In particular, the connection between each component is not clearly described in either the overview figure or the model section.  For example, it's unknown how the p_j memory context m_j is obtained. In equation 1, M_j^l's calculation is independent with layer l, which means M_j^l, m_j^{l + 1}, ... etc., all have the same value. It doesn't make sense. If C_{i,j}^l = \\alpha_{i,j}^l, then why use this new symbol C_{i,j}^l. The dimension of node embedding E_1 and E_2 is not clear. The encoder and decoder structure is unclear, either. \n\nAnother concern is that the paper's contribution seems incremental. Just as the authors stated, GCMeme is a combination of (Madotto et al., 2018) and (Bruna et al., 2014), which takes care of memory attention and graph convolution separately. \n\nThe last concern is that some important baselines, such as ASTGCN, STGCN, etc., are missing. Since table 1 includes the main result, it is necessary to include the error bars instead of simply providing the prediction results.", "summary_of_the_paper": "This work studied the problem of traffic speed forecasting. In particular, the authors proposed a framework that improves forecasting performance by leveraging both the spatio-temporal dependency and extracted traffic patterns. ", "summary_of_the_review": "The paper is not well presented, and critical information regarding the problem and model setup is missing, which makes the paper difficult to understand. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fRD3xMnCt4G", "reviewer": "Reviewer_Cncx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is exciting. It proposes a new way of looking at a very well-studied problem in traffic management. The new approach is novel, technically sound, and well motivated. As a first step, there is much opportunity for further research into more sophisticated models for intelligent pattern extraction, filtering, and inference.\n\nI wholeheartedly recommend acceptance not simply because the idea is novel, but because I strongly believe this paper will birth a new direction of research into traffic flow forecasting.\n\nIn order to further improve the paper, I suggest the following:\n1. Currently, the results in Table 1 do not look too impressive. This may call the significance of the approach into question. Do you have any opinion on how to further improve performance?\n\n2. Can you provide visual results for several examples where you show an input sequence, its extracted pattern, and finally the resulting prediction and comparison with ground truth.\n\n3. The paper exposition can be improved significantly. For example, in 3.2, what do you mean by \"zero-based signals\", \"without any duplication in range\". Such cryptic phrases are scattered throughout the paper. This concern has also been raised by other reviewers, but in my opinion, assuming that the authors improve the clarity, the paper could be accepted.", "summary_of_the_paper": "The paper proposes a new approach for macroscopic traffic flow forecasting in which the traffic forecasting problem is reformulated as a key-value pair matching problem as opposed to the conventional way of using a neural network to predict a sequence given past input. Results have been provided on two datasets along with ablation results.\n\n\n", "summary_of_the_review": "I recommend acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "DiZEtrQ1_46", "reviewer": "Reviewer_L94p", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\nS1 The idea of using a memory network to remember traffic patterns is very interesting. It has great research value. \nS2 The experiment results are promising. \nS3 The paper is well written and easy to follow. \n\nWeaknesses\nW1 It is interesting to know how the size of P affects the prediction accuracy. An ablation study on it (and other configurations) would potentially make this paper more interesting. \n\n", "summary_of_the_paper": "This paper explores a new direction of model design in traffic forecasting tasks. It proposes a neural memory module to model the spatio-temporal traffic data and designs a new traffic forecasting model based on the memory module. Experiments on a few public datasets demonstrate the effectiveness of the proposed scheme. ", "summary_of_the_review": "This is a good paper - it proposes a novel idea to solve the traffic prediction problem and conducts sufficient experiments to prove the effectiveness of the proposed idea.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6Sl_nKCbRqF", "reviewer": "Reviewer_mucH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies the traffic forecasting problem and proposes to conduct prediction by pattern matching. Authors first extract key patterns from the historical data in an offline manner and then fetch the patterns for each time series with a distance function (e.g., cosine similarity). Then, the patterns of different nodes are interacted with GCN to get node representation.\nWhile using pattern matching for traffic forecasting is reasonable and the design contains some new ideas, the paper should be carefully improved on the following points:\n1. The writing is not clear. Especially, the descriptions of the extracted patterns are not enough, are they just averaged speed? Are they representative? How to sample the patterns with a time window? Eq.1 should be a sum from j=0 to k? What is $p_1$ to get the noise in Eq.4? \n2. In section 3.3, there lacks justifications about why capturing interactions among fetched patterns is necessary and also why pattern level attention is necessary?\n3. The experimental results are wired. I doubt the necessity of using a different setting (e.g., 18 steps instead of 12) with existing works, which makes it hard to conduct a fair comparison. And the experimental results of some baseline methods are lower than the reported performance. I would strongly suggest authors use the same settings with existing works to demonstrate their performance and also take PeMS (e.g., PeMS-bay) dataset into consideration. \n4. Many traffic forecasting works use MLP for prediction directly, does the proposed decoder performs better than this baseline?\n5. There are many grammar mistakes in the paper, such as:\n\t* “The goal in this work is different the previous work”\n\t* “Similarity distribution of original pattern” in Figure 2\n\t* “we further use a adjacency matrix”\n", "summary_of_the_paper": "This paper studies the traffic forecasting problem and proposes to conduct prediction by pattern matching. Authors first extract key patterns from the historical data in an offline manner and then fetch the patterns for each time series with a distance function (e.g., cosine similarity). Then, the patterns of different nodes are interacted with GCN to get node representation.", "summary_of_the_review": "1. The writing of the paper should be carefully improved;\n2. Current experimental results can not demonstrate the effectiveness of the method;", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a neural architecture based on neural memory modules to model the spatiotemporal traffic data. The reviewers think this is an important application of deep learning and thus fits the topic of ICLR. The writing and the novelty of the proposed method need improvement.
Decision: Accept (Poster)