Learning to Remember Patterns: Pattern Matching Memory Networks for Traffic Forecasting

1 INTRODUCTION . Traffic forecasting is a challenging problem due to complex road networks , varying patterns in the data , and intertwined dependencies among models . This implies that prediction methods should not only find intrinsic spatio-temporal dependencies among many roads , but also quickly respond to irregular congestion and various traffic patterns ( Lee et al. , 2020 ) caused by external factors , such as accidents or weather conditions ( Vlahogianni et al. , 2014 ; Li & Shahabi , 2018 ; Xie et al. , 2020 ; Jiang & Luo , 2021 ) . To resolve these challenges and successfully predict traffic conditions , many deep learning models have been proposed . Examples include the models with graph convolutional neural networks ( GCNs ) ( Bruna et al. , 2014 ) and recurrent neural networks ( RNNs ) ( Siegelmann & Sontag , 1991 ) , which outperform conventional statistical methods such as autoregressive integrated moving average ( ARIMA ) ( Vlahogianni et al. , 2014 ; Li et al. , 2018 ) . Attention-based models , such as GMAN ( Zheng et al. , 2020 ) , have also been explored to better handle complex spatio-temporal dependency of traffic data . Graph WaveNet ( Wu et al. , 2019 ) adopts a diffusion process with a self-learning adjacency matrix and dilated convolutional neural networks ( CNNs ) , achieving stateof-the-art performance . Although effective , existing models have a weakness in that they do not accurately forecast when conditions are abruptly changed ( e.g. , rush hours and accidents ) . In this work , we aim to design a novel method for modeling the spatio-temporal dependencies of roads and to improve forecasting performance . To achieve this goal , we first extract representative traffic patterns from historical traffic data , as we find that there are similar traffic patterns among roads , and a set of traffic patterns can be generalized for roads with similar spatio-temporal features . Figure 1 shows the example speed patterns ( left , 90-minute window ) that we extract from many different roads and a representative traffic pattern ( right time series ) . With the representative patterns , we transform the conventional forecasting problem into a pattern-matching task to find out which pattern would be the best match for the given spatio-temporal features to predict future traffic conditions . With insights from the huge success of neural memory networks in natural language processing and machine translation ( Weston et al. , 2015 ; Sukhbaatar et al. , 2015 ; Kaiser et al. , 2017 ; Madotto et al. , 2018 ) , we design graph convolutional memory networks called GCMem to manage representative patterns in spatio-temporal perspective . Lastly , we design PM-MemNet , which utilizes representative patterns from GCMem for traffic forecasting . PM-MemNet consists of an encoder and a decoder . The encoder consists of temporal embedding with stacked GCMem , which generates meaningful representations via memorization , and the decoder is composed of a gated recurrent unit ( GRU ) with GCMem . We compare PM-MemNet to existing state-of-the-art models and find that PM-MemNet outperforms existing models . We also present a qualitative analysis in which we further investigate the strengths of PM-MemNet in managing a traffic pattern where high responsiveness of a model to abrupt speed changes is desired for accurate forecasting . The experimental results indicate that PM-MemNet achieves state-of-the-art performance , especially in long-term prediction , compared to existing deep learning models . To further investigate the characteristics of PM-MemNet , we conduct an ablation study with various decoder architectures and find that PM-MemNet demonstrates the best performance . We also investigate how the number of representative patterns affects model performance . Finally , we discuss the limitations of this work and future directions for neural memory networks in the traffic forecasting domain . The contributions of this work include : ( 1 ) computing representative traffic patterns of roads , ( 2 ) design of GCMem to manage the representative patterns , ( 3 ) design of PM-MemNet that matches and uses the most appropriate patterns from GCMem for traffic forecasting , ( 4 ) evaluation of PMMemNet compared to state-of-the-art models , ( 5 ) qualitative analysis to identify the strengths of PM-MemNet , and ( 6 ) discussion of limitations and future research directions . 2 RELATED WORK . 2.1 TRAFFIC FORECASTING . Deep learning models achieve huge success by effectively capturing spatio-temporal features in traffic forecasting tasks . Past studies ahve shown that RNN-based models outperform conventional temporal modeling approaches , such as ARIMA and support vector regression ( SVR ) ( Vlahogianni et al. , 2014 ; Li et al. , 2018 ) . More recently , many studies have demonstrated that attention-based models ( Zheng et al. , 2020 ; Park et al. , 2020 ) and CNNs ( Yu et al. , 2018 ; Wu et al. , 2019 ) record better performance in long-term period prediction tasks , compared to RNN-based models . In terms of spatial modeling , Zhang et al . ( 2016 ) propose a CNN-based spatial modeling method for Euclidean space . Another line of modeling methods , such as GCNs , using graph structures for managing complex road networks also become popular . However , there are difficulties in using GCNs in the modeling process , such as the need to build an adjacency matrix and the dependence of GCNs on invariant connectivity in the adjacency matrix . To overcome these difficulties , a set of approaches , such as graph attention models ( GATs ) , have been proposed to dynamically calculate edge importance ( Park et al. , 2020 ) . GWNet ( Wu et al. , 2019 ) adopts a self-adaptive adjacency matrix to capture hidden spatial dependencies in training . Although effective , forecasting models still suffer from inaccurate predictions due to abruptly changing road speeds and instability , with lagging patterns in long-term periods . To address these challenges , we build , save , and retrieve representative traffic patterns for predicting speed rather than directly forecasting with an input sequence . 2.2 NEURAL MEMORY NETWORKS . neural memory networks are widely used for sequence-to-sequence modeling in the natural language processing and machine translation domains . Memory networks are first proposed by Weston et al . ( 2015 ) to answer a query more precisely even for large datasets with long-term memory . Memory networks perform read and write operations for given input queries . Sukhbaatar et al . ( 2015 ) introduce end-to-end memory networks that can update memory in an end-to-end manner . Through the end-to-end memory learning , models can be easily applied to realistic settings . Furthermore , by using adjacent weight tying , they can achieve recurrent characteristics that can enhance generalization . Kaiser et al . ( 2017 ) propose novel memory networks that can be utilized in various domains where life-long one-shot learning is needed . Madotto et al . ( 2018 ) also introduce Mem2Seq , which integrates the multi-hop attention mechanism with memory networks . In our work , we utilize memory networks for traffic pattern modeling due to the similarity of the tasks and develop novel graph convolutional memory networks called GCMem to better model the spatio-temporal correlation of the given traffic patterns . 3 PROPOSED APPROACH . In this section , we define the traffic forecasting problem , describe how we extract key patterns in the traffic data that serve as keys , and introduce our model , PM-MemNet . 3.1 PROBLEM SETTING . To handle the spatial relationships of roads , we utilize a road network graph . We define a road network graph as G = ( V , E , A ) , where V is a set of all different nodes with |V| = N , E is a set of edges representing the connectivity between nodes , andA ∈ RN×N is a weighted adjacency matrix that contains the connectivity and edge weight information . An edge weight is calculated based on the distance and direction of the edge between two connected nodes . As used in the previous approaches ( Li et al. , 2018 ; Wu et al. , 2019 ; Zheng et al. , 2020 ; Park et al. , 2020 ) , we calculate edge weights via the Gaussian kernel as follows : Ai , j = exp ( − dist 2 ij σ2 ) , where distij is the distance between node i and node j and σ is the standard deviation of the distances . Prior research has formulated a traffic forecasting problem as a simple spatio-temporal data prediction problem ( Li et al. , 2018 ; Wu et al. , 2019 ; Zheng et al. , 2020 ; Park et al. , 2020 ) aiming to predict values in the next T time steps using previous T ′ historical traffic data and an adjacency matrix . Traffic data at time t is represented by a graph signal matrix , XtG ∈ RN×din , where din is the number of features , such as speed , flow , and time of the day . In summary , the goal of the previous work is to learn a mapping function f ( · ) to directly predict future T graph signals from T ′ historical input graph signals : [ XG ( t−T ′+1 ) , · · · , XG ( t ) ] f ( · ) −−→ [ XG ( t+1 ) , · · · , XG ( t+T ) ] The goal of this study is different from previous work in that we aim to predict future traffic speeds from patterned data , instead of utilizing input XG directly . We denote P ⊂ RT ′ as a set of representative traffic patterns , p ∈ P as one traffic pattern in P , and d : X × P → [ 0 , ∞ ) as a distance function for pattern matching . Detailed information about traffic pattern extraction will be discussed in the next subsection . Our problem is to train the mapping function f ( · ) as follows : [ XG ( t−T ′+1 ) , · · · , XG ( t ) ] d ( · ) , k−NN−−−−−−−→ [ P t1 , . . . , P tN ] f ( · ) −−→ [ XG ( t+1 ) , · · · , XG ( t+T ) ] , where P ti = { p1 , . . . , pk } is a set of k-nearest neighboring traffic patterns of node i in time t , with a distance function d. Note that pj is the j-th nearest neighbor pattern . 3.2 KEY EXTRACTION FROM TRAFFIC PATTERNS . Analyzing the traffic data , we find that the data has repeating patterns . In traffic data , the leading and trailing patterns have a high correlation , even during short-term periods . To take advantage of these findings , in our model , we build a representative pattern set , P. First , from historical data , we compute an average daily pattern , which consists of 288 speed data points ( total 24 hours with 5- minute intervals ) for each vertex v ∈ V . We then extract pattern p by slicing the daily patterns with a given window size T ′ , as shown in Figure 2 ( a ) . At this stage , |P| = N ×b 288T ′ c. After we collect the patterns , we investigate similarity distribution of the extracted pattern set , P , via cosine similarity ( Figure 2 ( b ) ) and find that the pattern set P has a biased distribution with too many similar patterns ( i.e. , class imbalance ) . Since such class imbalance causes memory ineffectiveness in accurate memory retrieval and gives biased training results , we use clustering-based undersampling ( Lin et al. , 2017 ) with cosine similarity . For example , if pattern p and pattern p′ have a cosine similarity larger than δ , they are in same cluster . We utilize the center of each cluster as a representative pattern of that cluster . After undersampling by clustering , we have a balanced and representative pattern set , P , as shown in Figure 2 ( c ) , which we use as keys for memory access . Table 2 presents the effect of different δ and |P| on forecasting performance .

Summary: This paper studies the traffic forecasting problem and proposes to conduct prediction by pattern matching. Authors first extract key patterns from the historical data in an offline manner and then fetch the patterns for each time series with a distance function (e.g., cosine similarity). Then, the patterns of different nodes are interacted with GCN to get node representation.