Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: (Note: I am not well-versed in the forecast modeling literature but I am reasonably so in the use of Transformer models in NLP tasks)

The paper proposes MQTransformer, an improvement on MQRNN (Wen 2017) for multi-horizon forecast prediction that leverages the Transformer architecture. Their contributions are:
1) using learnable positionable embedding from event indicators
2) using an attention head for each of the k horizons that need to be forecasted
3) applying decoder self-attention so that the model can use one horizon prediction to improve a later one

The technical / modeling contributions aren't very novel or profound, so the ICLR community at large may not find them that inspiring. However, as backed by the experiments, these contributions do translate into non-trivial improvements on real-world datasets, so I expect the practical impact on real-world forecasting tasks to be high. I marginally support accepting this paper for this reason.

Pros:
* Well written overall
* Non-negligible improvement over baseline methods on "RETAIL" dataset
* Likely to be used in practice

Cons:
* Datasets / tasks are limited. Paper could benefit from a few more.
* Modeling novelty is limited

Suggestions for improvement:
* In equation 2, it's more clear to write as (x_s + r_s)^T W_q^h W_k^h (x_t + r_t), no?


review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: After rebuttal:
I appreciate authors' detailed responses and an updated version of the paper. The new version is a lot clearer. After reading other reviews, I agree that the algorithmic novelty is limited, but the model is well-adapted for multi-horizon forecasting problem. Overall, I increase my score to 6. marginally above acceptance threshold. 

----------------------------------------------------------
Summary:
This paper introduces a new model for multi-horizon forecasting. The proposed model is an extension of MQRNN with two new modules: task specific attention and decoder self-attention. The experiments on the large-scale demand forecasting dataset and other publicly available datasets show that the proposed model outperforms or is comparable with the CNN, RNN-based models as well as the transformer-based model. 
--------------------------------------
Pros:
+ Authors adapted the attention mechanism for a challenging problem (multi-horizon forecasting). 
+ The proposed model is evaluated on a large-scale dataset as well as existing public datasets. 
+ Additional experiments and Figure 4 on Appendix are helpful. 
--------------------------------------
Cons:
I found this paper lacks clarity. I list the issues below:

1. Authors stated that 'Our horizon-specific attention mechanism can be viewed as a multi-headed attention mechanism. Each head corresponds to a different horizon.'. I don't think this is true. Multi-head attention is an ensemble of attentions from the same input to attend different positions by incorporating different representations. The purpose of the horizon-specific attention in this paper is to merge the encodings of multiple horizon. I ask authors to clarify it.

2. Design choice: the main contribution of this paper is attention mechanisms (horizon-specific attention between encoder-decoder and decoder self-attention) for multi-horizon forecasting. 
Questions:
    1. Since the proposed model architecture is similar to MQRNN [Wen et al. 2017], it should be the baseline. What is the reason MQCNN is used as a baseline? 
    2. The proposed model performs better than MQRNN for public datasets. Which module in the model has a big role between horizon specific attention and decoder self-attention? Or is attention specifically handling better for multi-horizon forecasting problems than RNNs? Adding MQRNN as another baseline and the result without horizon specific attention for the large-scale demand forecasting would be helpful (in Table 3 and Figure 2). 

3. TFT [Lim2019] is an existing transformer-based model for Multi-horizon Time Series Forecasting. In my understanding, the major differences are horizon specific attention in the proposed model and a different design of decoder in TFT. Could you clarify the difference and similarities between these two models in the paper?
Also, authors stated that 'We were unable to compare to TFT (the prior state of the art on several public datasets) as it does not scale-up'. What does this mean? I assume TFT cannot be easily applied for large-scale demand forecasting. This needs more explanations what are the major difficulties to scale-up TFT for the problem. 

3. A description of experiment setup and model specification is missing (for both the proposed model and baseline). The results can not be easily reproduced. 
    - Hyper-parameters: the number of layers and hidden units, learning rate, optimizer, etc. 
    - Preprocessing if there is any.
--------------------------------------
Minor comments:
- A description of P50 and P90 in Figure 2, 3, and Table 4 is missing. 
- A description of LTSP in Section 4 is missing. 
- The baseline (MQCNN) has no reference. 
review_title: This paper lacks clarity.

final_score: rating: 5: Marginally below acceptance threshold
review_content: 

This paper aims at improving accuracy of multi horizon univariate time series forecasting. The authors propose an encoder-decoder attention-based architecture for multi-horizon quantile forecasting. The model encodes a distinct representation of the past for each requested horizon.

#### Strong points

+ The encoding of the holidays and special event specific to the time series is elegant.
+ The idea of explicitly using forecast error feedback is interesting.
+ Ablation study of the architecture's innovations on a large scale forecasting dataset mainly outlines the importance of the horizon specific component of the architecture.

#### Weak points

- The evaluation of the new method is conducted on only two public datasets and the new methods outperform TFT only on one. There exists other common datasets (DeepAR evaluates on three-parts and traffic, TFT evaluates on traffic and volatility) that should be considered to place this method in the literature. 
- Both manuscripts of Fine and Foster (2020 al b) are not published and I could not find them online. The summary of Fine and Foster (2020 a; b) in Section 2.3 is not enough for me to judge the relevance of the results in Fig 2 and 3.
- The strong claims at the end of the introduction are made compared to the ablated model (MQCNN) on the private dataset, not against alternative methods such as MQRNN or DeepAR which could scale to this dataset size.


#### Decision

I would tend to reject this submission. The proposed model exhibits none of: significant quantitative improvement (on public dataset), speed up, improved simplicity over alternative methods.

Additionally to the weak points mentioned above, the contribution of this paper is undermined by the following points: 
- The contribution of “Positional Encoding from Event Indicators” is rather incremental considering BERT encoding of input segments (e.g. Sentence A, Sentence B, Question, Answers).
- The Horizon-Specific encoding is interesting but does not allow to forecast at inference a horizon that has not been trained on (as parametric method such as DeepAR have).
- The code of the submission is not submitted and there is no mention of future release.

#### Questions

- Is your positional encoding method a superset of relative positional encoding?
- Can you provide runtimes of the MQTransformers? How does it compare to (Lim et al. 2019) ?
- In appendix B: What is the percentage of unseen object in the test/valid dataset that would be harder to forecast by TFT?
- Why do you use the large scale private dataset for the ablation of your method and not a public dataset? Are the architectural innovations only useful in the high data regime?

#### Additional feedback

- Figure 4 in appendix D should be mentioned in the main text as it is helpful for the reader.
- Typo in Table 1, eq (4), third line: $c_{t,1}^{a}$ should be $c_{t,1}^{hs}$ I believe.
- Section 3.3: what do you call a *bidirectional* 1-D convolution? (As opposed to unidirectional?)
- Caption of Figure 3: Try to be consistent on the short name of retail vs Favorita dataset.
- Eq (2): Consider providing the factorised form which is easier to parse.


--- 
**Update:** I would like to thank the authors for their answer. I acknowledge the improvement of the manuscript after the review process:

- Added clarifications on the baselines,
- Added helpful precisions for reproducibility (even though the code cannot be open sourced),
- Evaluation on 2 requested public datasets with good results.

However, I am still not confident to raise my score to 6 (marginally above the acceptance threshold) given the missing public manuscripts (or Appendix) to explain the martingale diagnostic tools. This hinder one of the main selling point of the paper that their model reduces the volatility of the forecast.
review_title: Interesting architecture but evaluation should be more thorough to compare with the literature

Meta-review: The paper proposes and uses a fairly involved attention based architecture to perform time series forecasting. The idea of transformers is raised, but, given how sequence embedding is often convolutional, and position encoding input is provided to the model (albeit implictly in the form of features having to do with qualitative things such as promotions, etc among other things), I'm  of the opinion it is closer to the paper "Convolutional Sequence to Sequence Learning" https://arxiv.org/abs/1705.03122 than it is to transformers per se... Also the connection to sequence to sequence is not clear, since the chain rule of probability isn't really stressed on much. 

The paper proposes some interesting ideas, but I feel that it failed to convince the reviewers of the utility and the novelty. Part of it has to do with the clarity of presentation, and part of it, I think has to do with the fact that paper jam packs a bunch of different ideas together, without carefully ablating the influence of their various ideas. For example, transfomers have been applied for time sequences (https://arxiv.org/pdf/2001.08317.pdf), and its not clear in what ways this paper improves on them -- is it the complicated attention model scheme ? or is it the multi-horizon context schemes ?  

That being said, the results shows are relatively decent and the reviewers liked that aspect. Had the paper been easier to follow and the ideas presented with a little more insight it would be been a better fit for ICLR. As it stands, I have to give it a weak reject.
Decision: Reject