CGT: Clustered Graph Transformer for Urban Spatio-temporal Prediction

1 INTRODUCTION . The aim of this paper is to use urban data to study spatio-temporal prediction problems , whose goal is to forecast region-based spatial distribution in the future ( Shi & Yeung , 2018 ; Wang et al. , 2019 ) . Recently , region-based spatio-temporal forecasting has been extensively studied with various applications in traffic ( Li et al. , 2018b ) , ride-hailing services ( Zhang et al. , 2018 ; Li & Zheng , 2019 ) , environment ( Liang et al. , 2018 ) , resources ( Li & Zheng , 2019 ) , and human flows ( Wang et al. , 2019 ; Shi et al. , 2020 ) , among others . Such spatio-temporal forecasting is crucial for various tasks , such as dispatching and pricing , in urban computing ( Zheng et al. , 2014 ) . Accurate spatiotemporal forecasting methods may not only lower the barrier for decision making , but also improve the quality of various services including traffic management , transportation , and air quality management . However , achieving accurate prediction represents major challenges for many existing methods due to some difficulties in urban data , such as unsmoothness . As an illustration , we will show throughout the paper that urban spatio-temporal prediction task suffers from not appropriately handing spatial and temporal unsmoothness . Unsmoothness is a common phenomenon in many spatial and/or temporal data sets . The observations in some locations/time-steps differ substantially from the observations in their neighboring locations/time-steps . For example , rush-hour traffic surge is temporally unsmoothed , while it occurs in many regions in a city . Figure 1 shows potential spatial unsmoothness due to different points of interest . Specifically , S1 , S2 and S3 denote business , residential , and recreational areas , respectively , whereas U1 , U2 , and U3 are their boundaries . Traffic observations in the interior areas of S1 , S2 and S3 may be quite close to each other , whereas observations along U1 , U2 , and U3 may have much larger variations . Figure 2 shows 24-hour temporal patterns in the spatial neighborhood of a selected smooth region ( 2a ) and a selected unsmooth region ( 2b ) . The temporal patterns for the smooth region are close to those of most of their closest neighbors except the region with yellow color 1 in terms of shape and magnitude . The temporal patterns of the unsmooth region differ more significantly from those in all his closest neighbors . Moreover , we also discover some sharp surge and drop patterns in the unsmooth region . For more quantitative metrics for measuring unsmoothness , please refer to Appendix A for details . However , the attempt to handling unsmoothness is quite limited both at model and feature levels . Many existing prediction methods share their spatial and temporal feature extractors universally across all time-steps and/or all regions . Thus , it greatly limits the expressiveness of those prediction methods , since they are greatly vulnerable to the presence of unsmoothness in urban data . In this paper , we develop a novel Clustered Graph Transformer ( CGT ) framework by integrating Graph Attention Network ( GAT ) ( Veličković et al. , 2018 ) with Transformer ( Vaswani et al. , 2017 ) based on a unified attention encoder-decoder architecture ( Bahdanau et al. , 2015 ) . The GAT and Transformer are effective methods for spatial and temporal feature extraction based on attention mechanism . To handle spatial unsmoothness , we develop a novel Clustered Graph Attention Network ( CGAT ) by leveraging different graph attention kernels on different regions . The CGAT learns clustering assignments for each region according to their temporal patterns so that regions with different temporal patterns are assigned with different spatial feature extractors . To handle temporal unsmoothness , we propose additivity-preserved multi-view position encoding ( MVPE ) by characterizing different kinds of temporal relationship including weekly or daily periodicity and temporal closeness ( Zhang et al. , 2017 ) . In summary , our major contributions are summarized as follows : • To the best of our knowledge , CGT is the first of its kind in using graph attention network with a transformer under an encoder-decoder architecture for long sequence spatiotemporal forecasting . • We propose to improve the expressiveness of spatial feature extraction using the clustering technique . We also handle the temporal unsmoothness by using MVPE , which is a tailored position encoding for urban computing . • Compared with most existing baseline models , CGT achieves improvement ranging from 10 % to 25 % in spatiotemporal prediction on various tasks , datasets and cities . Error reduction on unsmooth areas dominates this improvement . 1Except region 1 . Details are in section 4.3 2 METHODOLOGY . We formalize the learning problem of urban spatiotemporal forecasting and introduce the CGT framework to capture the spatial and temporal dependencies and unsmoothness of urban data . 2.1 URBAN SPATIOTEMPORAL FORECASTING . We consider urban spatiotemporal forecasting problems in a city . Suppose that the city can be divided into V disjoint regions and we observe temporal information across T time intervals within each region . Our spatio-temporal data is represented as a |V | × T matrix . As an example , we consider taxi demand forecasting problem and each entry in the matrix represents the total number of taxi orders inside a region within a specific time interval . Our prediction problem is formulated as f : R|V |×Tx → R|V |×Ty such that we use the historical data within time period Tx to predict the future trend in time period Ty , that is , Xt−Tx+1 : t = [ X t−Tx+1 , . . . , Xt ] f ( · ) −−→ Xt+1 : t+Ty = [ Xt+1 , . . . , Xt+Ty ] . ( 1 ) Problem ( 1 ) is a one-step prediction problem as Ty = 1 and a multi-step prediction problem as Ty > 1 . We regard the whole city as a graph with V regions ( or vertices ) , each of which has both demand and supply numbers across T time intervals . 2.2 FRAMEWORK OVERVIEW . Figure 3 shows the overview of the CGT framework . The CGT is an encoder-decoder based architecture . The clustered graph attention layers ( CGAT ) and temporal attention layers are stacked alternately in the encoder and decoder . For prediction problem ( 1 ) , the input tensor to encoder is Xt−Tx+1 : t. In the training phase , the whole target sequence Xt : t+Ty is input to the decoder and the masked attention controls the visibility of target sequence in each step of decoding . The Xt is used to initialize the decoder state at the initial decoding step . In the testing phase , at step Ts ( Ts ≤ Ty ) , the decoder input tensor is [ Xt , X̂t+1 : t+Ts ] , where X̂ represents predicted values from previous steps . The multi-view position encoding ( MVPE ) is added to input tensors in both encoder and decoder . Our key novel contribution in CGT lies in the design and use of CGAT and MVPE . On spatial mode , CGAT integrates self-clustering and a set of graph attention kernels ( GAT ) , which are used for region clustering and spatial feature extraction , respectively . CGATs firstly generate soft clustering assignment scores for regions according to their temporal pattern in the sampled period . The cluster scores are used as the weights to apply different GATs to each region . On temporal mode , we use multi-view position encoding ( MVPE ) to manually inject ordering information , periodicity information , and closeness information to the model . 2.3 MULTI-VIEW POSITION ENCODING . We design a multi-view position encoding mechanism ( MVPE ) for urban computing . The position encoding consists of five channels as follows . There are two channels for weekly periodicity given by PE1 ( x ) = ( 1 + sin ( 2πx/7 ) ) /2 and PE2 ( x ) = ( 1 + cos ( 2πx/7 ) ) /2 . There are two channels for daily periodicity given by PE3 ( x ) = ( 1 + sin ( 2πx ) ) /2 and PE4 ( x ) = ( 1 + cos ( 2πx ) ) /2 . There is one channel for closeness given by PE5 ( x ) = exp ( −ax2 ) , where x represents the temporal difference between a fixed timestep ( e.g . the time t+1 in formulation ( 1 ) ) and a specific timestep in inputs or outputs . The MVPE mechanism injects ordering information and relevance information to the temporal modeling . First , MVPE provides unique identifiers for all time-steps in temporal sequences , which is critical for non-recurrent to model sequences . Second , according to Zhang et al . ( 2016 ) , the weekly and daily periodicity , as well as closeness are considered to be highly important in modeling urban time series . The periodicity channels provide constant crest/trough values for timesteps spanned by an integer number of periods . The closeness channel provides a surging value for the nearby timesteps . The model will be trained to make predictions according to different temporal relevance to avoid making over-smoothing ones . 2.4 CLUSTERED GRAPH ATTENTION . We propose the clustered graph attention layer ( CGAT ) for spatial feature extraction . It learns different graph attention kernels for different regions based on a gradient-based self-clustering assignment such that different regions are treated differently in spatial dependency modeling . First , a vertex-level soft-assignment to K clusters is learnt from the temporal pattern of each vertex : C = σs ( σr ( XfWf ) tWt ) , ( 2 ) where C is the cluster assignment score for each vertex to K clusters . Wf and Wt are parameters for linear layers on the feature mode and temporal mode , respectively , and σr and σs represent the relu and softmax activation functions . The feature dimension of input tensor Xf is first squeezed to 1 using Wf , in order to provide summarized temporal pattern at each vertex . The Wt is further applied to the temporal pattern to calculate a K−dimensional cluster assignment score . Second , we use cxi , k to denote the assignment score of C for assigning vertex xi to cluster k. Then , C is used to re-weight the latent output by using K GATs as follows : hxi = K∑ k=0 G ( xi ) cxi , k , ( 3 ) where hxi is the summed output for a CGAT layer at vertex xi and Gθk ( xi ) is the output for each GAT at vertex xi . We use the GAT defined by Veličković et al . ( 2018 ) . The attention between two vertex xi and xj is calculated as : αi , j = exp ( σ ( aT [ Wxi||Wxj ] ) ) ∑ xm∈Nxi exp ( σ ( aT [ Wxi||Wxm ] ) ) , ( 4 ) where W and aT are network parameters , Nxi represents the neighborhood of region xi , [ ·||· ] is the concatenation operation , and σ ( · ) is the Leakyrelu activation function . The aggregation function for GAT is given by G ( xi ) = σ ( ∑ xj∈Ni αi , jWxj ) . 2.5 TEMPORAL ATTENTION AND TRANSFORMER . The temporal features are extracted by adapting the multi-head attention model to the temporal mode of spatio-temporal tensor ( Vaswani et al. , 2017 ) . Detailed information is included in Appendix C . 2.6 OPTIMIZATION OBJECTIVE . The optimization objective of CGT consists of two major components . The first component is the prediction error given by Lpred = ||y − ŷ||2 , ( 5 ) where || · ||2 the standard L2 norm and ŷ and y are , respectively , the tensor of predicted values and that of true values . The second component consists of three constraints for obtaining a good vertex cluster schema C. Let the clustering result C = ( Cb , l , v , k ) be a tensor of order ( B , L , V , K ) , where B is the batch size , L is the number of all CGAT layers , V is the number of vertices , and K is the cluster number . Thus , Cb1 , l1 , · , · denotes a V × K matrix and Cb , l , v , · is a K × 1 vector . The three constraints including consistency , purity , and diversity are summarized below . • Consistency : The clustering assignment is generated through the temporal pattern of each vertex such that C should be temporally ( batch ) invariant and layer invariant . The loss function for ensuring consistency is defined as Lc = 4 BL ( B − 1 ) ( L− 1 ) V 2 B−1∑ b1=1 B∑ b2=b1+1 L−1∑ l1=1 L∑ l2=l1+1 ||Cb1 , l1 , · , · − Cb2 , l2 , · , ·||22 . ( 6 ) • Purity : Ideally , each vertex should be only assigned to one cluster . The loss function for encouraging purity is given by Lp = 1 BLV B∑ b=1 L∑ l=1 V∑ v=1 ( 1− ||Cb , l , v , ·||22 ) . ( 7 ) • Diversity : To balance the cluster size , we define the loss function for ensuring diversity as follows : Ld = 1 BLV 2 B∑ b=1 L∑ l=1 || V∑ v=1 Cb , l , v , ·||22 . ( 8 ) The overall minimization goal is to minimize the following object function L = Lpred + α ( Lc + Lp + Ld ) , ( 9 ) where α is a tuning parameter . It is worth noting that each batch consists of training snippets within a continuous short period of time ( e.g. , several hours ) . Keeping the clusters stable by using above regularizations allows to capture the short-term stability of urban time series data , while it does not violate the daily or long-term variability . We verify this short-term stability feature in Appendix B .

Summary: This paper proposes a clustering attention-based approach to handle the problem of unsmoothness while modeling spatio-temporal data, which may be divided into several regions with unsmooth boundaries. With the help of a graph attention mechanism between vertices (which correspond to different regions), the CGT model is able to model the (originally unsmooth) cross-region interactions just like how Transformers are applied in NLP tasks (where words are discrete). Experiments seem to suggest a big improvement when compared to baselines.