Learning to Remember Patterns: Pattern Matching Memory Networks for Traffic Forecasting

1 INTRODUCTION . Traffic forecasting is a challenging problem due to complex road networks , varying patterns in the data , and intertwined dependencies among models . This implies that prediction methods should not only find intrinsic spatio-temporal dependencies among many roads , but also quickly respond to irregular congestion and various traffic patterns ( Lee et al. , 2020 ) caused by external factors , such as accidents or weather conditions ( Vlahogianni et al. , 2014 ; Li & Shahabi , 2018 ; Xie et al. , 2020 ; Jiang & Luo , 2021 ) . To resolve these challenges and successfully predict traffic conditions , many deep learning models have been proposed . Examples include the models with graph convolutional neural networks ( GCNs ) ( Bruna et al. , 2014 ) and recurrent neural networks ( RNNs ) ( Siegelmann & Sontag , 1991 ) , which outperform conventional statistical methods such as autoregressive integrated moving average ( ARIMA ) ( Vlahogianni et al. , 2014 ; Li et al. , 2018 ) . Attention-based models , such as GMAN ( Zheng et al. , 2020 ) , have also been explored to better handle complex spatio-temporal dependency of traffic data . Graph WaveNet ( Wu et al. , 2019 ) adopts a diffusion process with a self-learning adjacency matrix and dilated convolutional neural networks ( CNNs ) , achieving stateof-the-art performance . Although effective , existing models have a weakness in that they do not accurately forecast when conditions are abruptly changed ( e.g. , rush hours and accidents ) . In this work , we aim to design a novel method for modeling the spatio-temporal dependencies of roads and to improve forecasting performance . To achieve this goal , we first extract representative traffic patterns from historical traffic data , as we find that there are similar traffic patterns among roads , and a set of traffic patterns can be generalized for roads with similar spatio-temporal features . Figure 1 shows the example speed patterns ( left , 90-minute window ) that we extract from many different roads and a representative traffic pattern ( right time series ) . With the representative patterns , we transform the conventional forecasting problem into a pattern-matching task to find out which pattern would be the best match for the given spatio-temporal features to predict future traffic conditions . With insights from the huge success of neural memory networks in natural language processing and machine translation ( Weston et al. , 2015 ; Sukhbaatar et al. , 2015 ; Kaiser et al. , 2017 ; Madotto et al. , 2018 ) , we design graph convolutional memory networks called GCMem to manage representative patterns in spatio-temporal perspective . Lastly , we design PM-MemNet , which utilizes representative patterns from GCMem for traffic forecasting . PM-MemNet consists of an encoder and a decoder . The encoder consists of temporal embedding with stacked GCMem , which generates meaningful representations via memorization , and the decoder is composed of a gated recurrent unit ( GRU ) with GCMem . We compare PM-MemNet to existing state-of-the-art models and find that PM-MemNet outperforms existing models . We also present a qualitative analysis in which we further investigate the strengths of PM-MemNet in managing a traffic pattern where high responsiveness of a model to abrupt speed changes is desired for accurate forecasting . The experimental results indicate that PM-MemNet achieves state-of-the-art performance , especially in long-term prediction , compared to existing deep learning models . To further investigate the characteristics of PM-MemNet , we conduct an ablation study with various decoder architectures and find that PM-MemNet demonstrates the best performance . We also investigate how the number of representative patterns affects model performance . Finally , we discuss the limitations of this work and future directions for neural memory networks in the traffic forecasting domain . The contributions of this work include : ( 1 ) computing representative traffic patterns of roads , ( 2 ) design of GCMem to manage the representative patterns , ( 3 ) design of PM-MemNet that matches and uses the most appropriate patterns from GCMem for traffic forecasting , ( 4 ) evaluation of PMMemNet compared to state-of-the-art models , ( 5 ) qualitative analysis to identify the strengths of PM-MemNet , and ( 6 ) discussion of limitations and future research directions . 2 RELATED WORK . 2.1 TRAFFIC FORECASTING . Deep learning models achieve huge success by effectively capturing spatio-temporal features in traffic forecasting tasks . Past studies ahve shown that RNN-based models outperform conventional temporal modeling approaches , such as ARIMA and support vector regression ( SVR ) ( Vlahogianni et al. , 2014 ; Li et al. , 2018 ) . More recently , many studies have demonstrated that attention-based models ( Zheng et al. , 2020 ; Park et al. , 2020 ) and CNNs ( Yu et al. , 2018 ; Wu et al. , 2019 ) record better performance in long-term period prediction tasks , compared to RNN-based models . In terms of spatial modeling , Zhang et al . ( 2016 ) propose a CNN-based spatial modeling method for Euclidean space . Another line of modeling methods , such as GCNs , using graph structures for managing complex road networks also become popular . However , there are difficulties in using GCNs in the modeling process , such as the need to build an adjacency matrix and the dependence of GCNs on invariant connectivity in the adjacency matrix . To overcome these difficulties , a set of approaches , such as graph attention models ( GATs ) , have been proposed to dynamically calculate edge importance ( Park et al. , 2020 ) . GWNet ( Wu et al. , 2019 ) adopts a self-adaptive adjacency matrix to capture hidden spatial dependencies in training . Although effective , forecasting models still suffer from inaccurate predictions due to abruptly changing road speeds and instability , with lagging patterns in long-term periods . To address these challenges , we build , save , and retrieve representative traffic patterns for predicting speed rather than directly forecasting with an input sequence . 2.2 NEURAL MEMORY NETWORKS . neural memory networks are widely used for sequence-to-sequence modeling in the natural language processing and machine translation domains . Memory networks are first proposed by Weston et al . ( 2015 ) to answer a query more precisely even for large datasets with long-term memory . Memory networks perform read and write operations for given input queries . Sukhbaatar et al . ( 2015 ) introduce end-to-end memory networks that can update memory in an end-to-end manner . Through the end-to-end memory learning , models can be easily applied to realistic settings . Furthermore , by using adjacent weight tying , they can achieve recurrent characteristics that can enhance generalization . Kaiser et al . ( 2017 ) propose novel memory networks that can be utilized in various domains where life-long one-shot learning is needed . Madotto et al . ( 2018 ) also introduce Mem2Seq , which integrates the multi-hop attention mechanism with memory networks . In our work , we utilize memory networks for traffic pattern modeling due to the similarity of the tasks and develop novel graph convolutional memory networks called GCMem to better model the spatio-temporal correlation of the given traffic patterns . 3 PROPOSED APPROACH . In this section , we define the traffic forecasting problem , describe how we extract key patterns in the traffic data that serve as keys , and introduce our model , PM-MemNet . 3.1 PROBLEM SETTING . To handle the spatial relationships of roads , we utilize a road network graph . We define a road network graph as G = ( V , E , A ) , where V is a set of all different nodes with |V| = N , E is a set of edges representing the connectivity between nodes , andA ∈ RN×N is a weighted adjacency matrix that contains the connectivity and edge weight information . An edge weight is calculated based on the distance and direction of the edge between two connected nodes . As used in the previous approaches ( Li et al. , 2018 ; Wu et al. , 2019 ; Zheng et al. , 2020 ; Park et al. , 2020 ) , we calculate edge weights via the Gaussian kernel as follows : Ai , j = exp ( − dist 2 ij σ2 ) , where distij is the distance between node i and node j and σ is the standard deviation of the distances . Prior research has formulated a traffic forecasting problem as a simple spatio-temporal data prediction problem ( Li et al. , 2018 ; Wu et al. , 2019 ; Zheng et al. , 2020 ; Park et al. , 2020 ) aiming to predict values in the next T time steps using previous T ′ historical traffic data and an adjacency matrix . Traffic data at time t is represented by a graph signal matrix , XtG ∈ RN×din , where din is the number of features , such as speed , flow , and time of the day . In summary , the goal of the previous work is to learn a mapping function f ( · ) to directly predict future T graph signals from T ′ historical input graph signals : [ XG ( t−T ′+1 ) , · · · , XG ( t ) ] f ( · ) −−→ [ XG ( t+1 ) , · · · , XG ( t+T ) ] The goal of this study is different from previous work in that we aim to predict future traffic speeds from patterned data , instead of utilizing input XG directly . We denote P ⊂ RT ′ as a set of representative traffic patterns , p ∈ P as one traffic pattern in P , and d : X × P → [ 0 , ∞ ) as a distance function for pattern matching . Detailed information about traffic pattern extraction will be discussed in the next subsection . Our problem is to train the mapping function f ( · ) as follows : [ XG ( t−T ′+1 ) , · · · , XG ( t ) ] d ( · ) , k−NN−−−−−−−→ [ P t1 , . . . , P tN ] f ( · ) −−→ [ XG ( t+1 ) , · · · , XG ( t+T ) ] , where P ti = { p1 , . . . , pk } is a set of k-nearest neighboring traffic patterns of node i in time t , with a distance function d. Note that pj is the j-th nearest neighbor pattern . 3.2 KEY EXTRACTION FROM TRAFFIC PATTERNS . Analyzing the traffic data , we find that the data has repeating patterns . In traffic data , the leading and trailing patterns have a high correlation , even during short-term periods . To take advantage of these findings , in our model , we build a representative pattern set , P. First , from historical data , we compute an average daily pattern , which consists of 288 speed data points ( total 24 hours with 5- minute intervals ) for each vertex v ∈ V . We then extract pattern p by slicing the daily patterns with a given window size T ′ , as shown in Figure 2 ( a ) . At this stage , |P| = N ×b 288T ′ c. After we collect the patterns , we investigate similarity distribution of the extracted pattern set , P , via cosine similarity ( Figure 2 ( b ) ) and find that the pattern set P has a biased distribution with too many similar patterns ( i.e. , class imbalance ) . Since such class imbalance causes memory ineffectiveness in accurate memory retrieval and gives biased training results , we use clustering-based undersampling ( Lin et al. , 2017 ) with cosine similarity . For example , if pattern p and pattern p′ have a cosine similarity larger than δ , they are in same cluster . We utilize the center of each cluster as a representative pattern of that cluster . After undersampling by clustering , we have a balanced and representative pattern set , P , as shown in Figure 2 ( c ) , which we use as keys for memory access . Table 2 presents the effect of different δ and |P| on forecasting performance .

Summary: This paper explores a new direction of model design in traffic forecasting tasks. It proposes a neural memory module to model the spatio-temporal traffic data and designs a new traffic forecasting model based on the memory module. Experiments on a few public datasets demonstrate the effectiveness of the proposed scheme.