CGT: Clustered Graph Transformer for Urban Spatio-temporal Prediction
Authors: Xu Geng
Lingyu Zhang
Shulin Li
Yuanbo Zhang
Lulu Zhang
Leye Wang
Qiang Yang
Hongtu Zhu
Jieping Ye
Venue:  (2020)
Keywords: Unsmooth spatiotemporal forecasting
Clustered graph neural network
Graph-Transformer
Urban computing

Abstract: Deep learning based approaches have been widely used in various urban spatio-temporal forecasting problems, but most of them fail to account for the unsmoothness issue of urban data in their architecture design, which significantly deteriorates  their prediction performance. The aim of this paper is to develop a novel clustered graph transformer framework that integrates both graph attention network and transformer under an encoder-decoder architecture to address such unsmoothness issue.  Specifically,  we propose two novel structural components to refine the architectures of those existing deep learning models. In spatial domain, we propose a gradient-based clustering method to distribute different feature extractors to regions in different contexts. In temporal domain, we propose to use multi-view position encoding to address the periodicity and closeness of urban time series data. Experiments on real datasets obtained from a ride-hailing business show that our method can achieve 10\%-25\% improvement than many state-of-the-art baselines. 

Under review as a conference paper at ICLR 2020
CGT: CLUSTERED GRAPH TRANSFORMER FOR URBAN PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
Deep learning based approaches have been widely used in various urban spatio- temporal forecasting problems, but most of them fail to account for the unsmooth- ness issue of urban data their architecture design, which significantly deterio- rates their prediction performance. The aim of this paper is to develop a novel clustered graph transformer framework that integrates both graph attention net- work and transformer under an encoder-decoder architecture to address such un- smoothness issue. Specifically, we propose two novel structural components to refine the architectures of those existing deep learning models. In spatial domain, we propose gradient-based clustering method to distribute different feature ex- tractors to regions in different contexts. In temporal domain, we propose to multi-view position encoding to address the periodicity and closeness of urban time series data. Experiments ness show that our method can achieve 10%-25% improvement than many state- of-the-art art baselines.
real datasets obtained from ride-hailing busi-
INTRODUCTION
The aim of this paper is use urban data to study spatio-temporal prediction problems, whose goal is to forecast region-based spatial distribution in the future (Shi & Yeung 2018; Wang et al.] 2019). Recently, region-based spatio-temporal forecasting has been extensively studied with var- ious applications in traffic (Li et al., 2018b), ride-hailing services (Zhang et al. 2018 Li & Zheng 2019). environment (Liang et al. 2018). resources (Li & Zheng 2019). and human flows Waan et al. 2019 Shi et al.) (2020). among others. Such spatio-temporal forecasting is cru- cial for various tasks, such as dispatching and pricing, in urban computing (Zheng al. 2014).
Accurate spatiotemporal forecasting methods may not only lower the barrier for decision mak- ing, but also improve the quality of various services including traffic management, transporta- tion, and air quality management. However, achieving accurate prediction represents major chal- lenges for many existing methods due to some difficulties in urban data, such as unsmoothness.
As an illustration, we will show throughout the paper that ur- ban spatio-temporal prediction task suffers from not appropri- ately handing spatial and temporal unsmoothness. Unsmooth- ness is common phenomenon in many spatial and/or tempo- ral data sets. The observations in some locations/time-steps differ substantially from the observations in their neighboring locations/time-steps For example, rush-hour traffic surge is temporally unsmoothed, while it occurs many regions city. Figure [ shows potential spatial unsmoothness due to different points of interest. Specifically, Si, S2 and SB de- note business, residential, and recreational areas, respectively, whereas U1. U2, and U3 are their boundaries. Traffic observa- tions in the interior areas of Si, S2 and Sa may be quite close to each other, whereas observations along U1. U2, and U3 may have much larger variations. Figure 2 shows 24-hour tempo- ral patterns in the spatial neighborhood of a selected smooth region (2a and selected unsmooth region 2b The temporal patterns for the smooth region
Business
$1
S3
Recreational
52
Residential
An example of spa- tial distribution for illustrating spa- tial smoothness and unsmoothness, where S1, S2 and S₃ are smooth function areas and U1. Uz and U3. are on their boundaries.
Figure 1:

---

Under review as a conference paper at ICLR 2020
pattern
target regior neighborl
target region
naigt
Ride halling demanc
neighbore neighbors neighbord
demane
neight bord
neigh bort
bord
x.
Time
(a) The close neighborhood of smooth region (blue line)
(b) The close neighborhood of an unsmooth region (blue line)
Figure 2: The 24-hour demand patterns of selected smooth and unsmooth regions (in blue lines) and their close neighbors (in other colors). In Figure 2a the target region and neighbor 2-4 are smooth regions, while the neighbor is an unsmooth region. In Figure 2b] the target region is an unsmooth region. As shown in an interpretation in Section 4.3] the spatial learner assigns neighbor 1 in Figure 2a and target region Figure 2b to the same cluster. Remaining regions in Figure 2a are assigned to another cluster.
close those of most of their closest neighbors except the region with yellow color in terms of shape and magnitude. The temporal patterns of the unsmooth region differ more significantly from those in all his closest neighbors. Moreover, we also discover some sharp surge and drop patterns in the unsmooth region. For more quantitative metrics for measuring unsmoothness, please refer to Appendix(A for details.
However, the attempt to handling unsmoothness is quite limited both at model and feature levels. Many existing prediction methods share their spatial and temporal feature extractors universally across all time-steps and/or all regions. Thus, it greatly limits the expressiveness of those prediction methods, since they are greatly vulnerable to the presence of unsmoothness urban data.
In this paper, we develop novel Clustered Graph Transformer (CGT) framework by integrating Graph Attention Network (GAT) (Veličković et al. 2018) with Transformer (Vaswani et al. 2017) based unified attention encoder-decoder architecture (Bahdanau et al. 2015). The GAT and Transformer are effective methods for spatial and temporal feature extraction based on attention mechanism. To handle spatial unsmoothness, develop novel Clustered Graph Attention Net- work (CGAT) by leveraging different graph attention kernels on different regions. The CGAT learns clustering assignments for each region according their temporal patterns so that regions with dif- ferent temporal patterns are assigned with different spatial feature extractors. To handle temporal unsmoothness, we propose Additivity-preserve multi-view position encoding (MVPE) by charac- terizing different kinds of temporal relationship including weekly or daily periodicity and temporal closeness (Zhang al. 2017). In summary, our major contributions are summarized as follows:
To the best our knowledge, CGT is the first of kind using graph attention network with transformer under encoder-decoder architecture for long sequence spatiotemporal forecasting.
We propose to improve the expressiveness of spatial feature extraction using the clustering technique. We also handle the temporal unsmoothness by using MVPE, which a tailored position encoding for urban computing.
Compared with most existing baseline models, CGT achieves improvement ranging from 10% to 25% in spatiotemporal prediction on various tasks, datasets and cities. Error reduc- tion on unsmooth areas dominates this improvement.
Except region 1. Details in section 4.3
2

---

Under review as a conference paper at ICLR 2020
2
METHODOLOGY
We formalize the learning problem of urban spatiotemporal forecasting and introduce the CGT framework capture the spatial and temporal dependencies and unsmoothness of urban data.
2.1 URBAN SPATIOTEMPORAL FORECASTING
We consider urban spatiotemporal forecasting problems in city. Suppose that the city can be divided into V disjoint regions and we observe temporal information across T time intervals within each region. Our spatio-temporal data is represented as [V] T matrix. As an example, we consider taxi demand forecasting problem and each entry in the matrix represents the total number of taxi orders inside a region within specific time interval. Our prediction problem is formulated as f R IVI + V''',, such that we use the historical data within time period Tp to predict the future trend time period Ty, that is,
=
J6-
X X41184T, = [X [
X
(1)
Problem is one-step prediction problem T, 1 and multi-step prediction problem as Ty > 1. We regard the whole city as graph with V regions (or vertices), each of which has both demand and supply numbers across T time intervals.
Deccoer layer (44
function
atwedionn
Target
Feed stantion
Bel-aterton
Maros set-
Output
CGAT
Enooder
Conclttnncy
Decader
Cluster
Buanndg
Cluster
Inpue
MYPC
Macoed targe
MVFC
Figure 3: The CGT framework overview. CGT is an encoder-decoder architecture. The encoder and decoder consists of stacked CGATs and temporal attentions. The encoder and decoder are linked with attention. The loss function contains the prediction error, as well as three regularizers to ensure the efectiveness of clustering.
2.2 FRAMEWORK OVERVIEW
Figure shows the overview of the CGT framework. The CGT is an encoder-decoder based ar- chitecture. The clustered graph attention layers (CGAT) and temporal attention layers are stacked alternately in the encoder and decoder. For prediction problem ( the input tensor to encoder is X1-T,+1:1- In the training phase, the whole target sequence X is input to the decoder and the masked attention controls the visibility of target sequence in each step of decoding. The X, used to initialize the decoder state at the initial decoding step. In the testing phase, at step T, (T, S Ty), the decoder input tensor is [X, X++1++T,] where X represents predicted values from previous steps. The multi-view position encoding (MVPE) is added to input tensors in both encoder and decoder.
Our key novel contribution in CGT lies in the design and use of CGAT and MVPE. On spatial mode, CGAT integrates self-clustering and a set graph attention kernels (GAT), which are used for region clustering and spatial feature extraction, respectively. CGATs firstly generate soft clus- tering assignment scores for regions according to their temporal pattern in the sampled period. The cluster scores are used as the weights apply different GAT' each region. On temporal mode,
3

---

Under review as a conference paper at ICLR 2020
multi-view position encoding (MVPE) manually inject ordering information, periodicity information, and closeness information to the model.
2.3 MULTI-VIEW POSITION ENCODING
We design niew position encoding mechanism (MVPE) for urban computing. The position encoding consists of five channels follows. There are channels for weekly periodicity given by
( (1 sin(2mx/77)/2 and ())(2) = (1 cos((22
There are two channels for daily periodicity given by
(1) = (1 - sin(2xx))/2 and PFa(") = (1 +1++0 cos((2rx))
There is one channel for closeness given by
(f)(#) exp(-ax²).
where x represents the temporal difference between a fixed timestep (e.g. the time t + in formula- tion (I) and specific timestep in inputs outputs.
The MVPE mechanism injects ordering information and relevance information to the temporal mod- eling. First, MVPE provides unique identifiers for all time-steps in temporal sequences, which is critical for non-recurrent to model sequences. Second, according to Zhang et al. (2016). the weekly and daily periodicity, as well as closeness are considered to be highly important in modeling urban time series. The periodicity channels provide constant crest/trough values for timesteps spanned by an integer number of periods. The closeness channel provides surging value for the nearby timesteps. The model will be trained to make predictions according to different temporal relevance to avoid making over-smoothing ones.
2.4 CLUSTERED GRAPH ATTENTION
We propose the clustered graph attention layer (CGAT) for spatial feature extraction. learns differ- ent graph attention kernels for different regions based on tradient-based self-clustering assignment such that different regions are treated diffcrently in spatial dependency modeling.
First, vertex-level soft-assigmment to K clusters is learnt from the temporal pattern of each vertex:
C' = (x, (X)
(2)
where C' is the cluster assignment score for each vertex to K clusters. W, y and W, are parameters for linear layers on the feature mode and temporal mode, respectively, and or, and a, represent the relu and softmax activation functions. The feature dimension of input tensor Xy is first squeezed to using W,. in order to provide summarized temporal pattern at each vertex. The W, is further applied to the temporal pattern to calculate K dimensiionl cluster assignment score.
Second, we use Cz to denote the assignment score of C for assigning vertex 2; to cluster k. Then, is used re-weight the latent output by using K GATs as follows:
K
hg. Σ ((x,)c2,Kk
(3)
where ha; is the summed output for CGAT layer at vertex and Go, (ar)) is the output for each GAT at vertex 2;. We use the GAT defined by Veličković et al. (2018). The attention between two vertex and zy is calculated as:
exp(o Wa, Wxy)))
0 Σ EEMEN exp(o((" Wa, [W,,)))
(4)
where W and a² are network parameters, N₂. represents the neighborhood of region 2;, [-]]-] is the concatenation operation, and (-)) is the Leakyrelu activation function. The aggregation function for GAT is given by ((x:) (EEeeN, 04, Wr,)).
4

---

Under review as a conference paper at ICLR 2020
2.5
TEMPORAL ATTENTION AND TRANSFORMER
The temporal features are extracted by adapting the multi-head attention model to the temporal mode
of spatio-temporal tensor (Vaswani al. 2017). Detailed information is included in Appendix C
2.6
OPTIMIZATION OBJECTIVE
The optimization objective of CGT consists of two major components. The first component is the prediction error given by
Lpred |yy 8112,
(5)
where || 2l2 the standard Le norm and y and y are, respectively, the tensor of predicted values and that of true values.
The second component consists of three constraints for obtaining a good vertex cluster schema C. Let the clustering result C (C).,,,k)) be tensor of order (B, L, V, K),, where B is the batch size, is the number of all CGAT layers, V is the number of vertices, and K is the cluster number. Thus, Cb ... denotes V K matrix and Cb.l,e,- is K vector. The three constraints including consistency, purity, and diversity are summarized below.
The clustering assignment is generated through the temporal pattern of each vertex such that C' should be temporally (batch) invariant and layer invariant. The loss function for ensuring consistency is defined as
Consistency:
B-1
B
L-1
L
The BL - - 1)V2
4
ΣΣΣΣ
Σ
Σ
Σ
Σ
Cba. 12
(6)
b. 12
Purity:
Ideally, each vertex should be only assigned to one cluster. The loss function for encouraging purity given by
B L
I, BLV
1
ΣΣΣ (1
b=1i=1
Diversity: To balance the cluster size, we define the loss function for ensuring diversity as follows:
La
BLV2 ΣΣ Co.
The overall minimization goal is to minimize the following object function
Lpred + a(L. + Lp + La),
where a is tuning parameter: It is worth noting that each batch consists of training snippets within continuous short period of time (e.g., several hours). Keeping the clusters stable by using above regularizations allows to capture the short-term stability of urban time series data, while it does not violate the daily or long-term variability. We verify this short-term stability feature in Appendix
2.7 APPLICATION- ADAPTATIONS
Several state-of-the-ar techniques need to be considered for improving our implementation and evaluation on real-world applications. First, we construct a set of graphs G = (G1, G2, G3) as the graph adjacency matrix, each of which represents region-wise distance. region-wise functional similarity and region- wise road connectivity, respectively. follows the idea of Geng et al. (2019) in order to encode multiple region-wise relationships as multi-graphs for the better utilization of auxiliary data. Second, following Child et al.] (2019). we implement spatial atrous attention to reduce the extra large spatial dimension in order reduce the computational cost for CGATs.
Denoting atrous stride and atrous offset as and a, the atrous version of eq.44 is revised to be:
exp(o (a W D.
(10)
EXmEN, EN., explo g. W Wmfm))) T) (m)
5

---

Under review as a conference paper at ICLR 2020
where a7 and Wg denote the feature transformation operation for graph g. Is,(') is masking function for atrous attention:
l2) - { 1 if j mod s = a; otherwise.
To cover all the regions, atrous-GAV needs to traverse 0, 0,....... by in-layer aggregation. The aggregated atrous-GAT output under single graph g is calculated as:
gg(x,) o
sijaas Way).
(11)
Each GAT outputs an aggregated output over -raith given by (z;)) = = (y(z:)/IG). The regularizers eq. (8) are calculated separately for multi- multi-graph and finally averaged for calculat- ing the loss.
3 RELATED WORK
3.1 SPATIOTEMPORAT DEEP LEARNING
There have been various ways to model the spatial and temporal dependencies with deep neural
networks. The combination CNN and RNN is the most intuitive one (Yao al., 2018 Zhang
et al. 2017; Wang et al., 2017 Liao et al. 2018 Rodrigues et al. (019). However, CNN-based architectures are unable handle non-Euclidaal region-wise relationships. The STMGCN (Geng
et al. 2019). DCRNN (Li et al. 201 and GCNN-DDGF (Lin et al.) (0188) use ChebNet (Kipf & Welling 2017) and diffusion convolution to extract spatial features based on rather complex graph representations. The graph attention network (Veličković et al. 2018) is the third generation of spatial modeling. which has more complex structures for discovering region- wise relationships. The ST-MetaNNt (Pan al. 2019) is representative work of GAT with application in urban computing. In temporal feature extraction, attention- -based methods outperform RNNs in terms both accuracy and efficiency. The ASTGCN (Guo et al.] 2019) is attention- based model both spatial and temporal domains. The GeoMAN (Liang et al.) 2018)) further adapts the attention-based model to encoder-decoder architecture for long sequence prediction. However, existing methods neglect the unsmoothness problem of urban data.
3.2 GRAPH NEURAL NETWORKS
The research works on graph neural networks could be summarized several categories (Zhou
et al. 2018). Spectral graph convolution (Hammond et al. 2011; Kipf & Welling 2017) defines
convolution operator by the eigen- secenddcoopositioon
the graph laplacian based on the graph spectral
thcorem. Non-spectral graph convolution (Duvenaud et al. 2015 Atwood & Towsley 2016 Hamil-
ton et al. 2017) directly operates the spatial domain by definig a set of neighborhood aggregation and updating rules vertex-level. Moreover, the gated GNN (Li al. 2015) leverages the gat- ing technique from LSTM, which provides more complex operations for neighborhood aggregation.
GAT (Veličković al. 2018) is using attention mechanism discover the vertex-wise similarities in graph. In urban computing, the intuition is confident that regions are strongly correlated. The
GAT might help discover this intrinsic structure in the region-based spatial distribution.
3.3 VERTEX CLUSTERING AND GRAPH POOLING
Vertex clustering is frequently used method for graph pooling (Ying et al. 2018). which is common technique migrated from computer vision. However, the intuition for graph pooling and vertex clustering proposed in this work is different. Graph pooling is to reduce the number of vertices such that vertices in each cluster are regarded sampled into one in the next layer. Downscaling the graph layer would improve model generality and reduce computational cost. The vertex clustering in our approach to discover a set of vertices sharing similar patterns, which is used for better feature extraction.

---

Under review as a conference paper at ICLR 2020
3.4 HANDLING UNSMOOTHNESS IN MACHINE LEARNING
Non-local mean methods are proposed for image denoising and restoration by investigating the pixel-level self-similarity in images. Buades et al. (2005) proposed calculate pixel pair- sim- ilarity within large search window. The densities for noisy pixels are interpolated by those similar pixels. Wang et al. (2018b [iu et al. (2018) designed non-local neural networks to incorporate the similar idea in deep learning. However, these non-local mean methods are applied to fix part of the spatial distribution at corrupted areas, which is different from our problem formulation. In time se- ries prediction, Ding et al. 2018) proposed a catteme--vaute loss (EVL) for extreme event prediction in time series data. Compared with our method, EVL has no model-level effort to extract unsmooth features from data, which totally relies the success of optimization algorithm.
4 EXPERIMENTS
We conduct experiments demand/supply dataset for ride-hailing services in two cities A and from Jun 2017 to Jun 2019. The demand data is the number of ride-hailing orders in each region within each time interval. The supply data is the summation of online duration for all ride- hailing drivers. The spatial and temporal distribution is sampled and aggregated into 1km x 1km map gridding and 30 minutes temporal sampling interva For both datasets, cut off the long tail on the 95th quantile and normalize them by min-max normalization. The duration of the dataset is two years, where first 70% used for training, 10% for validation, and remaining 20% for test. The multi-graph G involves region-wise relationships including neighborhood, functional similarity, and road connectivity, which are identical to those used in (Geng al. 2019). The functional similarity graph is binarized on 0.9 so that three graphs have almost equal graph sparsity
4.1 OVERALL PERFORMANCE
In this section, the encoder and decoder both stack 3 CABs and TABs alternately, with 16 hidden states each. The default clustering number is 3, with atrous stride being equal to 2. The number of heads in TAB multi-head attention is set to be 3. The network optimized by using scheduled Adam on PyTorc (Paszke et al. 2017).
We conduct both one-step prediction and multi-step prediction experiments. To avoid the long sequence problem, which may exhaust computational resource, we sampled 10 input time-steps for each output time-step Ti. The input time-steps include 3 closest time-steps (I Ts-2, Tk-3), 2 from daily periodicity (Tk-4s, Th 96) and I from weekly periodicity (TR-48.7). For multi-step prediction, the default output length 8, the encoder input length is 27.
One-step prediction
Multi-step prediction Demand
Data
Demand
Supply
City
City
City
City
City
City A
HA
12.20 0.261
12.35 / 0.298
0.14/
2.68 0.16
18.07 / 0.345
LR
10.61 0.247
10.08 / 0.288
3.22 / 0.148
2.12/
14.31 / 0.286
XGBoost
10.28 0.233
10.25 / 0.268
3.12/0.16
2.07 0..10
13.03 / 0.240
M-DMVST
9.8570.223
9.459
2.88/ 0.152
1.94/0.137
12.22 / 0.270
m- DCRNNN
0.217
9.34 0.249
2.88 0.149
1.97 148
10.74 / 0.245
ST-MGCN
/
0.259
2.85 0.136
0.199
12.29 / 0.276
m- n-GeoMAN
0.232
9.86 0.291
2.94 / 0.145
/
11.29 / 0.259
m-ASTGNN
10.23 / 0.273
9.48 0.252
/
1.98 0.140
12.66 / 0.262
DST-GCNN
9.72 / 0.226
m-GAT+RNN
9.54 0.224
Our Method
7 0.200
7.66 / 0.232
2.66 / 0.131 1.73/0.124
7.49/ 0.155
Table 1: The error metrics (RMSE/MAPE) based on the demand-supply datasets in two cities.
Anonymous for blind review.
DDssgneed according to industrial practices
II city A, it is roughly 12000 edges for 1296 vertices.
7

---

Under review as a conference paper at ICLR 2020
Baselines built based following research works. For the sake of information faimesss all deep learning models are slightly revised to take in multiple graphs.
HA, LR and XGBoost: Historical average, linear regression and XGBoost. The model pa- rameters for LR and XGBoost are shared across space and time. XGBoost is implemented using LightGBM (Ke al. 2017).
DMVST Yao et al., 2018); CNN-RNN based model. Region-wise relationships are
encoded as external inputs.
DCRNN Li ct al. 2018b) and ST-MGCN (Geng et al. 2019): Both are GCN-RNN based model for spatiotemporal feature extraction. DCRNN is based on encoder-decoder archi- tecture for multi-step traffic prediction.
GeoMAN (Liang et al. 2018): The spatial and temporal features are extracted by spatial
attention and RNN. It based the encoder-decoder architecture.
ASTGCN (Guo et al. 2019) attention- model spatial and temporal domain. Spatiotemporal features are extracted using GCN and convolution. ASTGCN don't t have position encoding.
DST-GCNN Wang al. 2018a): : A two stream framework for traffic prediction. The first stream constructs dynamic affinity graph. The second stream use the graph and spatiotem- poral convolution for feature extraction.
GAT+RNN Xu & Li 2019) GAT+RNN structure for traffic forecasting.
Table 1 summarizes the RMSEs and MAPEs of CGT and state-of-the-arr baselines. There are few observations from above experiment results. First, the proposed CGT model outperforms all base- lines in all prediction tasks by large margin. Second, all encoder-decoder based models, including m-DCRNN, m-GeoMAN and CGT, perform well in all multi-step prediction tasks.
4.2 ABLATION STUDY AND PARAMETER TEST
In this section, we show the experiment results of CGT on the one-step demand prediction problem in city A with varying model configurations to show the parameter effect. As shown by figure the optimal result achieved when specific component is removed from CGT. The ablation study shows the significance of our novel contribution.
and k 3. Table shows the model performance when
Parameter cluster
clusters
180
180
Teal etror (RMSE)
Text Teal enrur (RMSE))
05
1
Number of cluster
Figure 4: Parameter test for the trade-off parameter a and number of clusters k.
Ablation
MVPE and
MVPE
Clustering
component RMSE
Clustering
8.69
9.87
10.11
Table 2: Ablation study for novel components
"Made-up experiment according reviewer's comments. Unfinished due time limit.
8

---

Under review as a conference paper at ICLR 2020
Our model
Baseline model
Error reduction
Our model
Baseline model
Error reduction
Spatial
Temporal
0-50%
0-50%
2.64
3.77
30%
3.54
4.68
24%
(smooth)
(smooth)
50-75%
6.29
9.30
32%
50-75%
7.08
9.98
29%
75-100%
75-100%
11.51
17.65
35%
10.98
16.83
35%
(unsmooth)
(unsmooth)
Table 3: Error reduction ratio on different quantiles of unsmoothness in spatial and temporal do- mains.
4.3 MODEL INTERPRETATION
We claim that the prediction error is reduced due to more accurate predictions in spatial and temporal unsmooth areas. To validate this, we explore the relationship between the unsmoothness and the error reduction from baseline model. The baseline model is designed by removing clustering and MVPE from CGT. Figure [ shows a positive relationship between unsmoothness and error reduction. Table 3 shows the error reduction from the baseline model on different quantiles of spatial and temporal unsmooth areas. Conclusions could be drawn that CGT could improve the prediction performance, especially in spatial and temporal unsmooth areas by significant margins.
RMSE improvement under spatial unsmoothness
RMSE improvement under temporal unsmoothness
RMSE improvement
RMSE Improvement
200 spatial unsmoothness
temporal unsmoothness
Figure 5: Error reduction and unsmoothness spatial and temporal domains.
Above improvements on spatial mode may be caused by the effective clustering technique, which applies different spatial kernels regions with different patterns. Referring back Figure 2 the neighbor I (yellow line) in Figure 2a is an unsmooth point in the neighborhood. It is assigned to the same cluster with the target region (blue line) in Figure 2b which also an unsmooth region. The remaining regions in Figure 2 stay in the same cluster. According to above case study, the temporal pattern for inner-cluster regions are similar, while the temporal pattern for intra-cluster regions are diverse. It is sensible to apply shared operator to target regions in single cluster, since the relationship between target regions and their neighbors are more stable within a cluster. smoothing operator to different clusters does not work very well, since it may introduce some artifacts. In conclusion, discriminating smooth and unsmooth patterns may greatly improve the local spatial feature extraction.
However, applying
5 CONCLUSION AND FUTURE WORK
In this paper, we proposed Clustered Graph-Trame Transforannormmrr for spatiotemporal prediction problems urban computing. To handle the spatial and temporal unsmoothness problem, we use a gradient- based clustering technique in GAT to construct spatial feature extractor. We use MVPE and attention to handle temporal unsmoothness by providing a urban-specific temporal encoding to the temporal mode. The experiment has shown the effectiveness of the proposed model. In the future, we plan to explore long seq2seq problem using atrous transformer (Child et al. 2019) as well as explore the explanation for the attention and clustering assignments.

---

Under review as a conference paper at ICLR 2020
REFERENCES
James Atwood and Don Towsley. Diffusiinn-oovolurional neural networks. In Advances in Neural Information Processing Systems, pp. 1993-2001, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. International Conference on Learning Representations (ICLR), 2015.
Antoni Buades, Bartomeu Coll, and J-M Morel. non-local algorithm for image denoising. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 005, volume 2. pp. 60-65, IEEE, 2005.
Rewon Child, Scott Gray, Alec Radford, and Πya Sutskever. Generating long sequences with sparse transformers. CoRR, abs/1904. 10509, 2019. http://///aaxi org httg///b//1904.. 100.9944.155050 10509.
Daizong Ding, Mi Zhang. Xudong Pan, Min Yang, and Xiangnan He. Modeling extreme events in time series prediction. In Advances in Neural Information Processing Systems, 2018
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Gust Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. In Advances neural information processing systems. pp. 2224-223333 2015.
Xu Geng, Yaguang Li, Leye Wang, Lingyu Zhang, Qiang Yang. Jieping Ye, and Yan Liu. Spatiotem- poral multi-gran -gaph convolution network for ride-hailing demand forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3656-3663, 2019.
Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based spatial- temporal graph convolutional networks for traffic flow forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 922-929, 2019.
Will Hamilton, Zhitao Ying. and Jure Leskovee. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, pp. 1024-1034, 2017.
David K Hammond, Pierre Vandergheynst, and Rémi Gribonval. Wavelets on graphs via spectral graph theory. Applied and Computational Harmonic Analysis, 1002):12-1500, 2011.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems, pp. 3146-3154, 2017.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. International Conference Learning Representations (ICLR), 2017.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3538-3545 2018a.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural net- work: Data-driven traffic forecasting. International Conference on Learning Representations (ICLR), 2018b.
Yexin Li and Yu Zheng, Citywide bike usage prediction in bike-sharing system. IEEE Transactions on Knowledge and Data Engineering, 2019.
Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. arXiv preprint
2015.
Yuxuan Liang. Songyu Ke, Junbo Zhang Xiuwen Yi, and Yu Zheng. Geoman: Multi-level attention 2018.
networks for geo-sensory time series prediction. In IJCAI, pp.
Binbing Liao, Jingqing Zhang, Ming Cai, Siliang Tang. Yifan Gao, Chao Wu, Shengwen Yang, Wenwu Zhu, Yike Guo, and Fei Wu. Dest-resnet: A deep spatiotemporal residual network for hotspot traffic speed prediction. In 2018 ACM Multimedia Conference on Multimedia Conference, pp. 1883-1891. ACM, 2018.
10

---

Under review as a conference paper at ICLR 2020
Lei Lin, Zhengbing He, and Srinivas Peeta. Predicting station-level hourly demand in a large-scale bike-sharing network: A graph convolutional neural network approach. Transportation Research Part C: Emerging Technologies, 97:258-2766 2018.
Ding Liu, Bihan Wen, Yuchen Fan, Chen Change Loy, and Thomas S Huang. Non-local recurrent network for image restoration. In Advances in Neural Information Processing Systems, pp. 1673- 1682, 2018
Zheyi Pan, Yuxuan Liang, Weifeng Wang, Yong Yu, Yu Zheng, and Junbo Zhang. Urban traffic prediction from spatio-temporal data using deep meta learning. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. 1720-1732 ACM, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Filipe Rodrigues, Ioulia Markou, and Francisco C Percira. Combining time-series and textual data for taxi demand prediction in event areas: deep learning approach. Information Fusion, 49: 120-129, 2019.
Pavel Senin. Dynamic time warping algorithm review. Information and Computer Science Depart- ment University of Hawaii at Manoa Honolulu, USA, 855(1-23):40, 2008
Hongzhi Shi, Quanming Yao, Qi Guo, Yaguang Li, Lingyu Zhang, Jieping Ye, Yong Li, and Yan Liu. Predicting origin-destination flow via multi-perspective graph convolutional network. In 2020 IEEE 36th International Conference on Data Engineering (ICDE), 2020.
Xingjian Shi and Dit-Yan Yeung. Machine learning for spatiotemporal sequence forecasting: A survey. arXiv preprint 1aep::888..0.555 2018
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information 2017.
processing systems, pp.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. International Conference Learning Representations (ICLR), 2018.
Dong Wang. Wei Cao. Jian Li, and Jieping Ye. Deepsd: supply-demand prediction for online car- hailing services using deep neural networks. In 2017 IEEE 33rd International Conference Data Engineering (ICDE), pp. 243-254. IEEE, 2017.
Menglin Wang, Baisheng Lai, Zhongming Jin, Xiaojin Gong, Jianqiang Huang, and Xian- sheng Hua. Dynamic spatio-temporal graph-based cnns for traffic prediction. arXiv preprint 1812..1812.022 2018a.
Senzhang Wang, Jiannong Cao, and Philip S Yu. Deep learning for spatio-temporal data mining: A survey. arXiv preprint arXiv: 100.:29906.04928, 2019.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794- 7803, 2018b.
Ying Xu and Dongsheng Li. Incorporating graph attention and recurrent architectures for city-wide taxi demand prediction. ISPRS International Journal of Geo-Information, 80)):44, 2019.
Huaxiu Yao, Fei Wu, Jintao Ke, Xianfeng Tang, Yitian Jia, Siyu Lu, Pinghua Gong, Jieping Ye, and Zhenhui Li. Deep multi-veww spatial-temporal network for taxi demand prediction. In Thirty- Second AAAI Conference Artificial Intelligence, 2018.
Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hi- erarchical graph representation learning with differentiable pooling. In Advances in Neural Infor- mation Processing Systems, pp. 4800-4810, 2018

---

Under review as a conference paper at ICLR 2020
Junbo Zhang. Yu Zheng, Dekang Qi, Ruiyuan Li, and Xiuwen Yi. Dnn-based prediction model for spatio-tempora data. In Proceedings of the 24th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, pp. 92. ACM, 2016.
Junbo Zhang, Yu Zheng, and Dekang Qi. Deep spatio-temporal residual networks for citywide crowd flows prediction. Thirty- First AAAI Conference on Artificial Intelligence, 2017.
Lingyu Zhang, Wei Ai, Chuan Yuan, Yuhui Zhang, and Jieping Ye. Taxi hitchhiking: Predicting passenger's preferred service on ride sharing platforms. In The 4 International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1041-1044. ACM, 2018.
Yu Zheng, Licia Capra, Ouri Wolfson, and Hai Yang. Urban computing: concepts, methodologies, and applications. ACM Transactions on Intelligent Systems and Technology ( 2014.
Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications, 2018.
A PRELIMINARY ANALYSIS
In this section, we conduct a preliminary error analysis for a baseline GNN+RNN based model on spatial and temporal unsmooth areas in order to prove our assumptions and motivate design methodology.
For input spatial temporal signal X, denote X, and Xx as its spatial and temporal modes, respectively, and L as the graph laplacian matrix for spatial distribution. The C is the loss metric. The spatial and temporal unsmooth- nesses could be represented by the difference between a convoluted value and its true value. We define the spatial convolution kernel as aT., which is a linear combination of the features from all neighboring locations. The tem- poral convolution kernel is defined as W**, which a convolution along temporal mode. The spatial and tempo- ral unsmoothnesses are calculated as Φ, L(X, aLX,) and Φ = L(X, W, Xt), respectively. The a and W are optimized using the same training set as the baseline model. Spatial and temporal areas with larger unsmooth- ness means that they are hard to be represented by using linear combination of observations neighboring locations.
Figure 6: The averaged 24 hour tempo- ral pattern for regions in each cluster, shaded within I standard deviation.
RMSE under spatial unsmoothness
RMSE under temporal unsmoothness
RMSE
RMSE
spatial unsmoothness
temporal unsmoothness
(a) Positive relationship between spatial unsmoothness(b) Positive relationship between temporal unsmooth-
(X) and prediction error (Y) for the baseline model. ness (X) and prediction error (Y) for the baseline model.
Figure 7: Baseline prediction error and unsmoothness
Figure 7 shows the positive relationship between baseline prediction enror and spatial/temporal un- smoothness. There several reasons for this. First, the expressiveness of GNNs are limited. Using
12

---

Under review as a conference paper at ICLR 2020
ChebNet as example, it treats all neighbors in convolution kernel homogeneously and shares its weighted transformation on feature dimension all neighboring vertices. Thus, ChebNet lacks discriminativity within the neighborhood. [i et al. (2018a) also proves that the usage of laplacian smoothing will cause serious problem in deep-stacked GCN models. Second, spatial feature ex- tractors shared among all regions, that is, it is assumed that spatial feature extraction rule is universally applied to the whole city. In Figure 6 we clustered the regions in the city into 5 clusters according to their similarities by using DTW (Senin) (0088) of temporal pattern short snippet and print averaged 24 hour temporal pattern for them. The clustering result shows that the magnitudes for regions from different clusters are different with high confidence. The overall temporal patterns for different clusters are quite diverse.
B LOCAL STABILITY ON THE TEMPORAL MODE
In this section, we prove in details that the temporal patterns are stable within each batch, compared with its large variation across the whole dataset. The average value for batch-level standard deviation on temporal mode is 658.57. The standard-deviation 1074.84, which is significantly larger. The batch-size the same with our experiment setting.
temporal mode for the whole dataset is
C CALCULATION FOR TEMPORAL ATTENTION IN TRANSFORMER
The temporal attention calculated as follows:
QKT
Attn(Q. K, K) = softmar
VV,
dd
where QKT is self-attention to represent relationships among different time-steps. Such relation- ship is further applied on V by transforming its temporal mode.
The feed forward layer is linear transformation the feature mode as follows:
(x)n(X) = W.,
encoder, the self-attention used to discover temporal dependencies the latent feature vector h such that the attention is defined as
Q = hW g: K = hW2, and V = hW
(12)
In decoder, the self-attention performs similarly, but a mask is applied on the input sequence so that the feature extraction at time could not view the temporal features later than it. The mask is implemented by an upper triangular matrix. The encoder-decoder attention transforms the temporal mode of decoder latent feature based temporal dependencies from the encoder follows:
Q Wew.eeWWWW.. W K WteneWW Wk, and V = h...
(13)
where h is the decoder latent feature vector from previous decoder layers and hene is the encoder feature vector which denotes the encoder state. this application, the initial state for the decoder input sequence (target sequence) is copied from the last step of the input sequence.
D VISUALIZATION FOR PERFORMANCE ON TEMPORAL UNSMOOTHNESS
Figure shows the comparison between ground truth and machine learning models based on 24- hour ride-hailing demand values in selected regions. According to the visualization, CGT model is more close to ground truth, especially in those temporal unsmoothed areas.
13

---

Under review as a conference paper at ICLR 2020
Prediction
Ride trailing demand
mouh
Ride- alling demand
Grond
Time
Time
Prediction of derand Gid ID:342
Prediction of demand - Gid ID:1198
OCT
Doseina
hailing demand
Ride-ha hide-hailing demand
Baselie
Time
Time
Figure 8: Plot 24-hour region demand values to compare ground truth with prediction result of CGT and baseline
14


Reviews:
[{"review_id": "r1g2jGL6FB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors developed a neural network architecture to address the spacial and temporal unsmoothness problem, which was claimed to be neglected by existing works. The proposed model, CGT, has an encoder-decoder structure, and is characterized by clustering modules for spacial regions based on their temporal patterns. To handle temporal unsmoothness, additivity-preserved multi-view position encoding was proposed to characterize different temporal relationships. The experimental results on real ride-hailing datasets demonstrate the effectiveness of the proposed method to some extent.\n\nThe major concern is the presentation of this paper. There are many unclear points by going through the current paper, which prevents full judgement of the merits of the proposed method. First, the key problem to address is claimed to be the spacial and temporal unsmoothness. From the introduction, it is hard to see how important the problem is. It is better to use real data statistics to show the prevalence and concrete examples of such unsmoothness. Second, the technical sections presents the methods with few intuition on how does each component solve the unsmoothness problem. Figures such as fig 2 and 4 are very dense with few annotations, neither in captions nor main texts, thus are hard to understand. Finally, in the experiments, it is good to see some results for model interpretation. However, it is not clear on how to measure the spatial and temporal unsmoothness in fig 6 on the x axis.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1xBWDqB5H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to address the problem of spatio-temporal forecasting in urban data, in a way that can accommodate regions with highly distinct characteristics.\n\nOn the spatial side, they make use of Graph Attention Networks (GAT), a very recent technique for spatial feature extraction using graph attention as a form of reweighting. The authors modify the GAT to accommodate a masking that allows for selection. For some parameter K, a collection of K GATs is then combined with the masking used so that only one of them can be active at any given time. This architecture (called MGAAT) then encourages a form of clustering,\nwith each cluster associated with a single GAT. This modification one of the two essential contributions of the paper.\nAlthough the description is not easy to follow, it does appear to have the potential to encourage clustering as claimed by the authors.\n\nOn the temporal side, the autoencoder-based Transformer architecture of Vaswani et al is imposed on top of the MGAAT architecture. Very few details are given in the main paper - as it stands now, without the hints on Transformer that appear only in the supplement, the overall workings of the paper cannot be easily understood. No insight is given as to how the overall architecture solves the main motivating problem for this paper.\n\nFor their experimentation, the authors compare against a good number of competing methods. However, three of them - DCRNN, GeoMAN, and ASTGCN - use important elements of the authors' own design, namely attention-based  models and encoder-decoder architectures (GeoMan uses both). However, the authors fail to differentiate their design from these approaches.\n\nOverall, the machinery is rather complex, underexplained, and undermotivated. The paper has major omissions and other serious presentational issues that make it very difficult to follow. The authors do not take care to point out which parts of their design are original and which are borrowed - it took much sleuthing to determine that the MGAAT differs from the GAT only in its introduction of a masking factor. As someone not previously familiar with GATs and atrous graph attention (as I suspect most of the audience would be), I found the paper very difficult going. A total overhaul of the paper would be needed in order to properly explain and motivate this work. \n\nOverall, in its current state (not least due to presentational issues) the paper appears to be significantly below the acceptance threshold.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJxr_thUKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes a clustering attention-based approach to handle the problem of unsmoothness while modeling spatio-temporal data, which may be divided into several regions with unsmooth boundaries. With the help of a graph attention mechanism between vertices (which correspond to different regions), the CGT model is able to model the (originally unsmooth) cross-region interactions just like how Transformers are applied in NLP tasks (where words are discrete). Experiments seem to suggest a big improvement when compared to baselines.\n\nPros:\n+This should be one of the first works that apply a graph transformer alike method in this domain, and specifically on the unsmoothness problem.\n+ Since the dataset is not publically available, there aren't many prior works to compare the CGT to. However, at least compared to the one prior work [1] that the authors point to in Section 4, the RMSE results achieved CGT does seem to be significantly better.\n\n========================================\n\nHowever, I still have some questions/concerns on the paper, detailed below.\n\n1) The current organization of the paper, as well as its clarity, can (and should) be significantly improved. I didn't completely understand the approach on my first two passes, and I **had** to read the code published by the authors. Here are some issues that I found:\n\n  - For one, Figure 2 is not quite helpful as it's too messy with font size too small. A similar problem is with Figure 4 which, without further clarification (e.g., of what \"atrous aggregation\" exactly mean), is very hard to interpret. \n\n  - The notations are very inconsistent and messy:\n      i) In Eq. (1), you should use a symbol different from $\\mathbf{X}$ to refer to the \"predictions\". Since you are applying $f(\\cdot)$ on $\\mathbf{X}_{t-T_x+1:t}$, you should not get the \"exact same\" target sequence. That's your target. Maybe use $\\hat{\\mathbf{y}}$, which you used in Eq. (8).\n\n      ii) In Figure 3, what is the orange line? In addition, I only saw two blue lines in the figure, but the legend seems to suggest there are four of them...\n\n      iii) The notations used in Figure 4 are somewhat confusing. For example, what does \"f->1\" mean? (I later found through Eq. (2) that it means transform to 1 dimension; but the small plots in Figure 4 suggest f is a \"magnitude\" of the feature.) In addition, there are two $H_1$ in Figure 4 with clearly different definitions.\n\n      iv) The authors used $\\mathcal{G}_{\\theta_k}(x_i)$ in Eq. (3) without defining it. The definition actually came much later in the text in Eq. (6). I suggest moving the usage of the clustering assignment (i.e., Eq. (3)) to after Eq. (6).\n\n      v) What does $[\\cdot || \\cdot]$ mean (cf. Eq. (4))? (The code seems to suggest it's concatenation?)\n\n      vi) The authors first used $h_{x_i}$ in Eq. (3) to denote the output of the CAB module. Then letter $h$ is then re-used in Eq. (4) and (5) with completely different meanings. For instance, the $W_kh_i$ in Eq. (4) correspond to line 48 of the code \"model.py\". (By the way, nowhere around Eq. (4) did the authors explain how $h_i$ is produced, such as taking the mean over the batch dimension, etc.). \n\n      vii) In Section 2.6, you denote the \"optimal vertex cluster scheme\" with letter $C$, which is used in Eq. (2). Similar for parameter $a_k$ and atrous offset $a$.\n  \n  - This not a very big problem (as it seems somewhat inevitable), but I think there are too many acronyms in the paper.\n  \n  I think it'd be great if the authors can take care of these issues, as clarity in math and descriptions are critical to the presentation of such an involuted method. It would also be useful to clearly define the dimensionality of all the variables (e.g., you defined $V$ in Section 2.1, but never used it again in later subsections).\n\n2) Regarding the usage of the multi-view position encoding, the authors claimed that it \"provides unique identifiers for all time-steps in temporal sequences\". However, if you consider $x=7$ and $x=14$, then $PE_i(7)=PE_i(14)$ for all $i=1,2,3,4$ with $PE_5(7) \\approx PE_5(14)$. Doesn't this invalidate the authors' claim? Also, doesn't this mean that the proposed MVPE only works on sequences with length <= 7? (In comparison, the design of positional encoding in the original Transformer doesn't have this problem.)\n\n(You didn't show how you implemented and initialized the position encoding in the uploaded code, so I may be missing some assumptions here.)\n\n3) In line 48 of the code (https://github.com/CGT-ICLR2020/CGT-ICLR2020/blob/master/model.py#L48), why did you take the mean over the batch dimension? Shouldn't different samples in a minibatch be very different? Does a (potentially completely independent) sample in a batch affect another sample? A similar problem occurs for Eq. (9): Why do you require clusterings of two different samples $b_1, b_2$ to be similar? (Where these samples can come from quite different times and years of the data?) \n\n4) In the experiments, you \"sampled 10 input time-steps\" due to computational resources. Typically, in Transformer-based NLP tasks the sequence lengths can be over 500, with much higher dimensionality (e.g., 512); but you are only using sequence length 10 and dimensions <= 16 (in your code, you used \"self.dec_io_list = [[5,8,16,16],[16,16,16,16],[16,8,8,8]]\"). What is the bottleneck for the computation of your approach? (I noticed there are more than 1K vertices in city A, which may be a costly factor indeed.) How much memory/compute does the CGT method consume? How does using a longer sequence affect the performance of CGT?\n\n5) You performed an ablation study on MVPE. Did you simply remove MVPE, or did you use the conventional PE from the original Transformer paper (Vaswani et al. 2017)? (If the latter, I'm very surprised that MVPE is so much better than PE. In that case, you may want to try MVPE on NLP tasks to see if it also improves SOTA.)\n\n6) How did you measure unsmoothness in Figure 6? It doesn't seem like a quantifiable property to me. You should discuss this in the experiment section.\n\n-----------------------------------\n\nMinor questions/issues that did not affect the score:\n\n7) There are some strange phrases/sentences in the paper. For example, the first sentence of the 2nd paragraph of Section 1: \"we will show **throughout the paper** that urban spatiotemporal prediction task suffers from...\"\n\n8) Why use an encoder-decoder architecture at all? Why can't we train the model like in language modeling tasks, where we want to predict the next token? In other words, you can simply use a decoder-side CGT, and mask the temporal self-attention as in the Transformers.\n\n-----------------------------------\n\nIn general, I think this paper proposed a valuable approach that seems to work very well on the spatio-temporal dataset they used (which unfortunately is private). However, as I pointed out above, I still have numerous issues with the paper's organization and clarity, as well as some doubts over the methodology and the experiment. I'm happy to consider adjusting my score if the authors can resolve my concerns satisfactorily.\n\n\n[1] http://www-scf.usc.edu/~yaguang/papers/aaai19_multi_graph_convolution.pdf\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach to handle the problem of unsmoothness while modeling spatio-temporal urban data. However all reviewers have pointed major issues with the presentation of the work, and whether the method's complexity is justified. 
Decision: Reject