Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: 
Review Summary
--------------
While I think there are some interesting innovations here, I don't think this is ready for ICLR. I have concerns that the evaluation doesn't focus enough on application-relevant scenarios (why exclude cells that are not in the top 50? why not use 7-day windows to capture day-of-week effects?), and that the evaluations don't quantify uncertainty (and thus the claimed improvements due to attention may not be significant). The presentation quality of the method details in Sec. 3 needs a significant rewrite to improve clarity. I do think the non-standard convolution operator is a nice idea.

Paper Summary
-------------
This paper addresses the forecasting of origin-destination demand data, with a focus on ride-sharing transportation applications. We assume an urban area has been divided up into N cells (known in advance). The problem is to forecast demand for *directed* rides from cell i to cell j, using historical demand data.

The paper's first contribution is the design of a convolutional neural net architecture that uses non-rectangular receptive fields more suitable to origin-destination data. Instead of assuming that cells in a given 2D euclidean neighborhood are correlated, it assumes that all journeys that overlap with the one of the cells in the current origin-destination pair are similar. They call this the "SACN" architecture.

The second contribution is developing an "ST-Conv" block that combines the above SACN convolution with gated temporal convolutions, so that both space and time can be summarized via convolution operators. A periodically-shifting attention mechanism is further used to measure long-term data's similarities to short-term data and weight accordingly.

Overall, these two ideas (SACN convolutions and ST-Conv blocks with attention) are combined into what they call their Latent Spatio-Temporal Origin-Destination or "LSTOD" architecture, which is claimed to be the first to use both short-term and long-term features in prediction.

Evaluation examines demand for ride-sharing in two major cities, A and B (presumably masked because they are proprietary). RMSE error comparing to several classic (e.g. ARIMA) and deep feature learning (e.g. LSTMs and SRCN) baselines, with primary results in Table 1. A few side experiments examine superiority over standard convolutions (Fig 3) and experiments without the attention and long-term bits of the model (Table 2). 


Novelty & Significance
-----------------------
The current paper is quite niche when it comes to significance; I think it may be a bit too specialized for most ICLR readers. Solving this kind of forecasting problem seems important to ride-sharing applications, but is highly specialized for origin-destination demand data. Would be nice to see the paper attempt to connect to other problems beyond ride-sharing (maybe animal migration? maybe package logistics?).

That said, I think there's sufficient novelty. The architectural design contributions here do appear new to me (though I don't follow this kind of data closely). 


Method Concerns
------------------

## M1: Complexity analysis missing, scaling could be a problem

When comparing the current square receptive field architecture for CNNs with the proposed SACN, I felt there was an opportunity to clarify how both scale with N and other key problem size parameters in terms of number of parameters or execution time. I'm concerned that it will be non-scalable given the O(N) cost in terms of number of parameters and the O(N^2) cost of runtime (you need to execute Eq. 2 for each of the N^2 entries at each layer). I'd like to see some careful breakdown of this compared to other approaches. 

Experimental Concerns
---------------------

## E1: Why not use a 7-day past history?

Shouldn't ride share demand have a day-of-week trend? Why wouldn't we use last Sunday's demand to predict this Sunday's demand? I find it quite odd that for the Historical Average baseline, only the last 5 days (rather than last 7) are used, and for the presented method, only the last 3 days are used. I'd be happy to be proven wrong, but I'd guess including a 7-day window would lead to noticeably better results.

## E2: Lack of error bars / uncertainty quantification

A natural question is, can we reliably tell that the difference between (for example) RMSE 2.49 and 2.54 is significant and not noise?  I'd like to see some attempt at quantifying the uncertainty for measurements in Table 1 (perhaps taking each full day in test set as its own "mini" test set that produces one RMSE score, then reporting average as well as 2.5th and 97.5th percentiles or something across each day). Without this, I think the claim that attention is useful here is unproven, since the change in performance is so small (less than 0.1 RMSE). 

## E3: Why focus only on the N=50 most common cells?

I would think to really assess demand forecasting, you want to know when to task drivers to visit less-common cells. The focus on the cells with only the top 80% of journeys means that 1 of every 5 rides would not be covered by this prediction system. I wonder if part of the reason is that using N much larger than 50 is problematic due to the scaling mentioned earlier.


Presentation Concerns
---------------------

## P1: Need to simplify notation

I found the descriptions of the neural net architecture throughout Sec. 3 quite hard to parse. I think there's an overreliance on math notation and there could be more simple description of high level intent and motivation.

For example, Eq. 2 could be simplified to avoid the channel "m" and layer "l" notation and thus let the reader focus on what matters, which is how any part of input A that involves the origin node i or the destination node j is included in the weighted sum.  Words can be used around this to clarify this same operation can happen across channels and layers. 

Similarly, Eq. 3 and Eq. 4 could perhaps be replaced by a good diagram. Currently, Eq. 3 both P and Q are undefined and quite confusing.

## P2: Name of the convolution operation

I'm not sure "SACN" is the best name. The receptive field of a standard CNN looks much more "spatially adjacent" to me (a compact rectangle surrounding the target cell). Instead, you might call it "topologically adjacent" or even something like "vertex adjacent". You want to emphasize that you are getting strength by finding all journeys whose start or end overlaps with one of the endpoints of the current journey.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary: This paper proposes a latent spatial-temporal origin-destination model to address the OD flow prediction problem. 

The main contributions are summarized as follows:

1.	The authors propose a purely convolutional framework to learn both short-term and long-term spatio-temporal features simultaneously from dynamic origin-destination flow data.
2.	The authors propose a novel SACN architecture to capture the relevance of OD flows by modeling each OD flow map as an adjacency matrix.
3.	The authors design a periodically shift attention mechanism to model the long-term periodicity.
4.	The results demonstrate that the proposed model outperforms state-of-the-art methods in OD flow prediction, with 6.5% to 15.0% improvement of testing RMSE.

However, my major concerns are as follows:

1. On Page 4, the authors explain the reasons why they use TGCNN instead of RNN-based architectures to capture the temporal representation. However, it would be more convincing if quantitative analysis or empirical results are provided.

2. On Page 4, readers might be confused with the symbols in the formulation (3) that are not defined clearly.

3. On Page 7, the experiment only considers one metric, i.e., RMSE. The efficiency comparisons between the proposed model and the baselines are missing.
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: The paper proposes a  deep learning architecture for forecasting Origin-Destination (OD) flow. The model integrates several existing modules including spatiotemporal graph convolution and periodically shifted attention mechanism. Experiments on a large-scale ride-sharing demand dataset demonstrate improved forecasting accuracy.

+ The spatiotemporal forecasting problem is an important task in the context of ride-sharing platforms.
+ The description of the model is clear and the real-world dataset experiments are interesting.

- The novelty of the model is rather limited. Spatiotemporal graph convolution has been previously proposed in [Yu et al. 2018] and later studies. Periodically shifted attention is a slight modification of the earlier work.
- The mathematical notations are messy and redundant. For example, the superscript index i, j can be removed and write all equations in matrix form. The subscript index can be replaced with a range operator. In Eqn (2), r_1 and c_2 are non-identifiable. 
- The experiments are not convincing. The paper is missing the following important baselines:

Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, 3634-3640. AAAI Press, 2018.

Yaguang Li, Rose Yu, Cyrus Shahabi, Yan Liu. "Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting." International Conference on Learning Representations (ICLR), 2018.

The authors should provide confidence intervals and visualizations for the predictions.



review_title: Official Blind Review #2

Meta-review: The paper proposes a  deep learning architecture for forecasting Origin-Destination (OD) flow. The model integrates several existing modules including spatiotemporal graph convolution and periodically shifted attention mechanism. 

The reviewers agree that the paper is not written well, and the experiments are also not executed well. Overall, we recommend rejection.
Decision: Reject