Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
This paper proposes a new framework of casual spatial-temporal prediction with high interpretability. Specifically, it first proposes a casual transformer including fast spatial graph fusion, casual attention, and temporal attention units. The authors conduct extensive experiments from different domains (electricity, traffic, etc.) and show the effectiveness, efficiency, and interpretability of their proposed method. 

Strength:

+ The authors propose to reduce the complexity of the computation of the attention module. 

+ The authors first propose a causal attention method for HTE in large-scale spatio-temporal prediction problems.

+ The experiments on datasets from various domains are adequate, which can support the authors' claim. 

Weakness: 

- Both equations (1) and (2) are based on the authors' assumptions. Do such assumptions have any support (either from previous literature or from the data)? In my opinion, some of the assumptions do not make sense, for example, one assumption is that $x_v(t + 1)$ is primarily affected by historical demands in $x_v(: t)$ and external covariates in $z$ without historical supply $y$. However, if the historical supply y is not enough, then the demand may raise because more and more demand accumulates. 

- It seems that the authors' proposed approximate Taylor's expansion attention can be used for attention models or even softmax functions in any scenarios. Does it have any limitations? For example, some of the coefficients are very large so that the higher-order Taylor terms cannot be overlooked. Please explain why such problems do not exist in the spatial-temporal prediction scenario in this paper. 
review_title: Interesting paper, but some designs need more explanation.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a Transformer-like framework, named CausalTrans, to tackle the demand and supply problem in the ride-hailing market. The problem is formulated by training two probabilistic models which forecasts collaborative demand and supply, by given historical observations and dynamic covariates. The paper leveragesTransformer encoder-decoder architecture, and proposes submodule (Fast S.F., C.A. and T.A.) for different functionalities. Many experiments and ablation analysis are constructed to show that the proposed method outperforms state-of-the-art. The paper also provides good visualization regarding the casual attention model, facilitating to understand the proposed idea.

+ overall the paper is well organized and easy to read.
+ the proposed architecture is with merit: Fast S.F. with an approximate Taylor’s expansioninstead of using softmax function, showing lower computational cost with little performance dropped; C.A. is proposed to deal with HTE problem in large-scale spatio-temporal forecasting problems, where a DML algorithm is also proposed to learn the C.A. model.
+ the experimental results are comprehensive and promising.

Still, I have some questions:
- The ablation analysis shows that spatial fusion shows tiny improvement (+0.3% on average). I’d like the author to elaborate more on the reason. If there are hyperparameters for the spatial fusion, please do some ablation analysis in this regard.

- From experiments the fast attention improves computation a lot with only 0.2% performance dropped in multi-horizon methods. Is it the same in multi-horizon methods? 

- From table 1 the results show that, in case of Electricity, the proposed method can’t outperform the state-of-the-art (TFT) due to lack of covariates and spatial information. I’d like to see the ablation analysis (like table 3) in case of Traffic, showing that in case of the iterative method, the performance can be gained by each submodule.

- Lack of experimental details. For example, learning rate/strategy, batch size, optimizer, architecture settings...etc. If the author(s) plan not to release the code in the future, it’s better to list the experimental details in the appendix, for reproducing the performance. 

Overall I think that the proposed causal attention is valuable. By adapting the C.A. to the transformer architecture, the proposed method is comparable to state-of-the-art. Moreover, sufficient experiments demonstrate the proposed method can achieve decnet performance in the demand-supply problem. If the paper can better clarify the points mentioned above, I’ll vote for positive.
review_title: Causal attention improves performance a lot.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose an interpretable spatio-temporal fusion transformer for predicting supply and demand in ride-haling platforms.  More generally, the authors claim that their approach extends to other two-sided markets such as electric grids, retail etc by showing empirical results of their approach using data from these markets.  The paper is well-written and easy to follow. 

The assumption that the supply is always conditioned on the demand x_v(, t + \tau_max) is too strong.  Is there a smoother of this assumption where the demand is dependent on a moving window over the past and future supply?  The experimental results show the efficacy of the proposed ML architecture.  However, one result that will greatly help the conclusions are results that clearly show the interpretability of the predictions, given that the authors state this as one of the main differences of their proposed solution.

The use of higher-order Taylor terms to approximate the attention procedure is interesting but the time complexity reductions are obvious and therefore does not meet the novelty bar for an ICLR submission.

Other suggestions - make the use of the word 'collaborative' in the model more clear.  The significance of the results is not completely clear and how the interpretability is helping understanding of the results.  Adding clarifications will help in the exposition.

I still have some concerns regarding the paper. The lack of baseline comparisons with spatio-temporal data (as also observed by fellow reviewers). My other concern also remains - from the authors' response, it is not clear how once can clearly attribute explainability of the results from their analysis of the model.
review_title: Spatio-temporal fusion transformers for predicting supply and demand in two-sided markets.

final_score: rating: 2: Strong rejection
review_content: The paper is interested with multivariate probabilistic forecasting applied in the context of ride-hailing forecast.
The goal is to be able to handle the spatial aspect, causal effects of external covariates (e.g. the causal impact of rain, Christmas on supply/demand) and dependency between supply and demand. To this end, the authors propose a causal attention mechanism	(to not just detect correlation with covariates) and a custom transformer architecture where the quadratic attention cost is avoided. Experiments are performed on public and private datasets against a set of (mostly) univariate baselines.

Strong points:
+ highly-relevant problem
+ novel consideration of handling causality instead of simple covariate correlation in this context

Weak points:
- very hard to read, many details missing and notation are not introduced
- lack of relevant baselines (e.g. baselines using spatial information)
- lack of relevant metrics to illustrate the benefit of the contribution
- missing related work discussion for efficient attention computation

I recommend a reject for this paper.

While the problem presented by the paper is highly relevant for the community, the paper has several issues that makes it not ready for publication. The first issue comes in the clarity of the description and in particular section 3.3 where many terms and notation are not introduced at all making the paper very hard to read (see detailed comments). 

Experiments are also problematic as many details were unclear (see detailed comments) and would be far from being reproducible. But the most problematic bit is in its design: key aspects introduced in the paper are not asserted in the experiments.1) While the method claims ~5% error improvement on the ride-hailing benchmark, only one baseline have access to spatial information while numerous methods have been proposed to handle spatio-temporal forecast, only one baseline (TFT) is provided with spatial information (with no detail). 2) Since you are interested in the probabilistic forecast of the joint demand/supply targets (collaborative), your experimental setup should consider a joint metric (CRPS-sum for instance [1,2]) rather than the average of R50/R90 metrics of demand/supply targets which is "blind" to correlation between the two demand and supply targets.

Finally, I had issue with the Taylor expansion as its description was not clear (see detailed comments) but also I found description of related work missing in this aspect. Attention cost has been decreased to $N\sqrt{N}$, $N\log{N}$ and $N$ respectively in [3, 4, 5], this discussion is clearly missing to relate your contribution.

# Additional questions to the author
- 3.1 at no point in the paper you say explicitly what is the loss that you are minimizing, this makes it harder to read. I would recommend to specify in 3.1 directly that you minimize NLE with poisson distribution (which is what I understood) and specify also clearly how predictions are made.
- 3.3 the beginning of the section mix model description and related work
- 3.3 Eq. (3) has two unintroduced notation, what is the \bar? what is "batch"?
- 3.3 Eq. (6) is so confusing, you are mixing norms and vectors (unintroduced), $T$ is not explicitly introduced, there is no clear explanation why the approximation holds also (e.g. why a^T W would be small)
- 4.2 DeepState is *not* an auto-regressive model, it belongs to the second category.
- 4.3: "CausalTrans outperforms all other competing methods primarily due to the use of the causal estimator DML and spatial information". This is very problematic as you dont compare with methods designed to handle spatial information. 

# Additional feedback (not part of the decision assessment)
- 3.1 eq (1) (2) makes an assumption that only demand impacts supply: it would be good to discuss it at least. One could imagine where it breaks.
- 3.1 weather features: you should precise whether they are known in advance
- 3.3 The section will be easier to read if you indicate variable dimensions

Finally, I would recommend to use a public dataset rather than the private one for your benchmark (e.g. NY taxi dataset or Uber), you also have covariates such as date or weather forecast and this would make your work comparable and reproducible in the future.



[1] High-dimensional multivariate forecasting with low-rank Gaussian Copula Processes http://papers.nips.cc/paper/8907-high-dimensional-multivariate-forecasting-with-low-rank-gaussian-copula-processes

[2] Multi-variate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows https://arxiv.org/abs/2002.06103

[3] Generating Long Sequences with Sparse Transformers https://arxiv.org/pdf/1904.10509.pdf

[4] Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention https://arxiv.org/abs/2006.16236

[5] Reformer: The Efficient Transformer https://arxiv.org/abs/2001.04451


review_title: Recommendation to Reject

Meta-review: The paper presents a spatial-temporal prediction framework with causal effects of predictors for better interpretability. The idea is interesting and the touch on modeling causal relations could be useful in practical applications. The paper receives mixed ratings and therefore there has been extensive discussion. We agree that while the paper has some merits, it falls short on the following aspects: 

1, One central issue pointed out by all reviewers is the evaluation. For example, the contribution on efficient attention was not compared to any previous work; most of the baselines do no have access to the spatial information, which makes the comparison unfair. The authors did add two more baselines with access to spatial information. However, there are not enough details and discussions to make the results convincing; In addition, other stronger baselines should be added. 

2. The notation and technical presentation was extremely lacking in the submitted version, the amount of unintroduced notations. Even in their core contribution equations had major issues with norm and vectors mixed together (see the difference between the corrected equation in the Taylor equation and the one in the original submission)

After the discussion, all reviewers agree that the paper fails to provide a fair and convincing evaluation, and the ratings will be adjusted to reflect the discussion. We hope that the reviews can help the authors improve the draft for a stronger submission in the future. 
Decision: Reject