Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proposes to address the problem of spatio-temporal forecasting in urban data, in a way that can accommodate regions with highly distinct characteristics.

On the spatial side, they make use of Graph Attention Networks (GAT), a very recent technique for spatial feature extraction using graph attention as a form of reweighting. The authors modify the GAT to accommodate a masking that allows for selection. For some parameter K, a collection of K GATs is then combined with the masking used so that only one of them can be active at any given time. This architecture (called MGAAT) then encourages a form of clustering,
with each cluster associated with a single GAT. This modification one of the two essential contributions of the paper.
Although the description is not easy to follow, it does appear to have the potential to encourage clustering as claimed by the authors.

On the temporal side, the autoencoder-based Transformer architecture of Vaswani et al is imposed on top of the MGAAT architecture. Very few details are given in the main paper - as it stands now, without the hints on Transformer that appear only in the supplement, the overall workings of the paper cannot be easily understood. No insight is given as to how the overall architecture solves the main motivating problem for this paper.

For their experimentation, the authors compare against a good number of competing methods. However, three of them - DCRNN, GeoMAN, and ASTGCN - use important elements of the authors' own design, namely attention-based  models and encoder-decoder architectures (GeoMan uses both). However, the authors fail to differentiate their design from these approaches.

Overall, the machinery is rather complex, underexplained, and undermotivated. The paper has major omissions and other serious presentational issues that make it very difficult to follow. The authors do not take care to point out which parts of their design are original and which are borrowed - it took much sleuthing to determine that the MGAAT differs from the GAT only in its introduction of a masking factor. As someone not previously familiar with GATs and atrous graph attention (as I suspect most of the audience would be), I found the paper very difficult going. A total overhaul of the paper would be needed in order to properly explain and motivate this work. 

Overall, in its current state (not least due to presentational issues) the paper appears to be significantly below the acceptance threshold.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors developed a neural network architecture to address the spacial and temporal unsmoothness problem, which was claimed to be neglected by existing works. The proposed model, CGT, has an encoder-decoder structure, and is characterized by clustering modules for spacial regions based on their temporal patterns. To handle temporal unsmoothness, additivity-preserved multi-view position encoding was proposed to characterize different temporal relationships. The experimental results on real ride-hailing datasets demonstrate the effectiveness of the proposed method to some extent.

The major concern is the presentation of this paper. There are many unclear points by going through the current paper, which prevents full judgement of the merits of the proposed method. First, the key problem to address is claimed to be the spacial and temporal unsmoothness. From the introduction, it is hard to see how important the problem is. It is better to use real data statistics to show the prevalence and concrete examples of such unsmoothness. Second, the technical sections presents the methods with few intuition on how does each component solve the unsmoothness problem. Figures such as fig 2 and 4 are very dense with few annotations, neither in captions nor main texts, thus are hard to understand. Finally, in the experiments, it is good to see some results for model interpretation. However, it is not clear on how to measure the spatial and temporal unsmoothness in fig 6 on the x axis.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Summary: This paper proposes a clustering attention-based approach to handle the problem of unsmoothness while modeling spatio-temporal data, which may be divided into several regions with unsmooth boundaries. With the help of a graph attention mechanism between vertices (which correspond to different regions), the CGT model is able to model the (originally unsmooth) cross-region interactions just like how Transformers are applied in NLP tasks (where words are discrete). Experiments seem to suggest a big improvement when compared to baselines.

Pros:
+This should be one of the first works that apply a graph transformer alike method in this domain, and specifically on the unsmoothness problem.
+ Since the dataset is not publically available, there aren't many prior works to compare the CGT to. However, at least compared to the one prior work [1] that the authors point to in Section 4, the RMSE results achieved CGT does seem to be significantly better.

========================================

However, I still have some questions/concerns on the paper, detailed below.

1) The current organization of the paper, as well as its clarity, can (and should) be significantly improved. I didn't completely understand the approach on my first two passes, and I **had** to read the code published by the authors. Here are some issues that I found:

  - For one, Figure 2 is not quite helpful as it's too messy with font size too small. A similar problem is with Figure 4 which, without further clarification (e.g., of what "atrous aggregation" exactly mean), is very hard to interpret. 

  - The notations are very inconsistent and messy:
      i) In Eq. (1), you should use a symbol different from $\mathbf{X}$ to refer to the "predictions". Since you are applying $f(\cdot)$ on $\mathbf{X}_{t-T_x+1:t}$, you should not get the "exact same" target sequence. That's your target. Maybe use $\hat{\mathbf{y}}$, which you used in Eq. (8).

      ii) In Figure 3, what is the orange line? In addition, I only saw two blue lines in the figure, but the legend seems to suggest there are four of them...

      iii) The notations used in Figure 4 are somewhat confusing. For example, what does "f->1" mean? (I later found through Eq. (2) that it means transform to 1 dimension; but the small plots in Figure 4 suggest f is a "magnitude" of the feature.) In addition, there are two $H_1$ in Figure 4 with clearly different definitions.

      iv) The authors used $\mathcal{G}_{\theta_k}(x_i)$ in Eq. (3) without defining it. The definition actually came much later in the text in Eq. (6). I suggest moving the usage of the clustering assignment (i.e., Eq. (3)) to after Eq. (6).

      v) What does $[\cdot || \cdot]$ mean (cf. Eq. (4))? (The code seems to suggest it's concatenation?)

      vi) The authors first used $h_{x_i}$ in Eq. (3) to denote the output of the CAB module. Then letter $h$ is then re-used in Eq. (4) and (5) with completely different meanings. For instance, the $W_kh_i$ in Eq. (4) correspond to line 48 of the code "model.py". (By the way, nowhere around Eq. (4) did the authors explain how $h_i$ is produced, such as taking the mean over the batch dimension, etc.). 

      vii) In Section 2.6, you denote the "optimal vertex cluster scheme" with letter $C$, which is used in Eq. (2). Similar for parameter $a_k$ and atrous offset $a$.
  
  - This not a very big problem (as it seems somewhat inevitable), but I think there are too many acronyms in the paper.
  
  I think it'd be great if the authors can take care of these issues, as clarity in math and descriptions are critical to the presentation of such an involuted method. It would also be useful to clearly define the dimensionality of all the variables (e.g., you defined $V$ in Section 2.1, but never used it again in later subsections).

2) Regarding the usage of the multi-view position encoding, the authors claimed that it "provides unique identifiers for all time-steps in temporal sequences". However, if you consider $x=7$ and $x=14$, then $PE_i(7)=PE_i(14)$ for all $i=1,2,3,4$ with $PE_5(7) \approx PE_5(14)$. Doesn't this invalidate the authors' claim? Also, doesn't this mean that the proposed MVPE only works on sequences with length <= 7? (In comparison, the design of positional encoding in the original Transformer doesn't have this problem.)

(You didn't show how you implemented and initialized the position encoding in the uploaded code, so I may be missing some assumptions here.)

3) In line 48 of the code (https://github.com/CGT-ICLR2020/CGT-ICLR2020/blob/master/model.py#L48), why did you take the mean over the batch dimension? Shouldn't different samples in a minibatch be very different? Does a (potentially completely independent) sample in a batch affect another sample? A similar problem occurs for Eq. (9): Why do you require clusterings of two different samples $b_1, b_2$ to be similar? (Where these samples can come from quite different times and years of the data?) 

4) In the experiments, you "sampled 10 input time-steps" due to computational resources. Typically, in Transformer-based NLP tasks the sequence lengths can be over 500, with much higher dimensionality (e.g., 512); but you are only using sequence length 10 and dimensions <= 16 (in your code, you used "self.dec_io_list = [[5,8,16,16],[16,16,16,16],[16,8,8,8]]"). What is the bottleneck for the computation of your approach? (I noticed there are more than 1K vertices in city A, which may be a costly factor indeed.) How much memory/compute does the CGT method consume? How does using a longer sequence affect the performance of CGT?

5) You performed an ablation study on MVPE. Did you simply remove MVPE, or did you use the conventional PE from the original Transformer paper (Vaswani et al. 2017)? (If the latter, I'm very surprised that MVPE is so much better than PE. In that case, you may want to try MVPE on NLP tasks to see if it also improves SOTA.)

6) How did you measure unsmoothness in Figure 6? It doesn't seem like a quantifiable property to me. You should discuss this in the experiment section.

-----------------------------------

Minor questions/issues that did not affect the score:

7) There are some strange phrases/sentences in the paper. For example, the first sentence of the 2nd paragraph of Section 1: "we will show **throughout the paper** that urban spatiotemporal prediction task suffers from..."

8) Why use an encoder-decoder architecture at all? Why can't we train the model like in language modeling tasks, where we want to predict the next token? In other words, you can simply use a decoder-side CGT, and mask the temporal self-attention as in the Transformers.

-----------------------------------

In general, I think this paper proposed a valuable approach that seems to work very well on the spatio-temporal dataset they used (which unfortunately is private). However, as I pointed out above, I still have numerous issues with the paper's organization and clarity, as well as some doubts over the methodology and the experiment. I'm happy to consider adjusting my score if the authors can resolve my concerns satisfactorily.


[1] http://www-scf.usc.edu/~yaguang/papers/aaai19_multi_graph_convolution.pdf


review_title: Official Blind Review #2

Meta-review: This paper proposes an approach to handle the problem of unsmoothness while modeling spatio-temporal urban data. However all reviewers have pointed major issues with the presentation of the work, and whether the method's complexity is justified. 
Decision: Reject