Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an improvement over previously published MQ-forecaster model for multi-horizon forecasts in time series data.  The proposed approach aims to improve the forecast accuracy as well as achieve a lower variability in forecasts made at various times for a specific time point in the future (excess forecast variability).  A number of model improvements are proposed, including a new way of handling positional encoding (via learnt embedding of event indicators), separate encodings for different horizons, and decoder attention on previous forecast errors to allow decoder to be aware of excess forecast variability and reduce it.  Empirical evaluation is carried out for the task of demand forecasting on a large but proprietary dataset, and also for tasks of retail sales, electricity load, securities volatility, and traffic forecasting on publicly available datasets.
main_review: Pros:

a) Significant gains in forecast accuracy on the proprietary data, and on the public retail forecasting task

b) Including forecast volatility as part of the model leads to a significant reduction in excess volatility over the MQ-CNN model

Cons:

a) A large part of the empirical analysis is conducted on a proprietary dataset, and thus those results are not reproducible or verifiable.  Furthermore, in these results the authors mask absolute values and only present relative gains, making it even harder to assess performance.

b) For results on the public task, a more direct comparison with MQ-CNN model is needed as the proposed model aims to improve over this model.  Since previously published results from the MQ-CNN model are on the GEFCom2014 forecasting task, could this be used to compare the proposed model?

c) Ablation studies are needed to understand key contributors to the gains observed.  For instance, on the ‘retail’ public task, a 38% accuracy gain is observed over the TFT model, yet excess variability studies show that TFT model has a lower excess variability as compared to the MQ transformer model (Fig. 6 in appendix D.5).  This raises the question if less focus on reducing excess variability will lead to further accuracy gains on this task.

d) Reproducibility of results is questionable.  Implementation details are shared to some extent, but lack of code, large focus on internal datasets that can not be released and numbers on those can not be disclosed, and a lack of detailed ablations will make it difficult to reproduce in my opinion.

Minor edits:
* subscript ‘I’ missing from x^{(s)} after Eq. (1)
* what is s_i in the reformulated version of (1) on page 4
* ‘where c_{:t} denotes … but there is no c_{:t} on page 5

summary_of_the_review: My rating is due to concerns about reproducibility (no code and bunch of results on proprietary dataset), and lack of comparison with previously published results for the MQ-CNN model which this paper aims to improve over.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper looks into applying Transformer with encoder-decoder architecture for forecasting. The work follows the previous work of multi-horizon quantile forecasts that predict quantile for each horizon. The framework applies a 1D-conv and a MLP network on time series feature to model positional information, and apply an attention head for each horizon. The authors also show that limiting the context of the attention provides better bias to improve the prediction quality. The authors also discussed that having access to past context allows reducing the volatility. The proposed framework showed some improvement on demand forecasting tasks.
main_review: This work applies Transformer architecture on forecasting tasks, and shows improvement over existing work, including the previous work that used Transformer for decoder. The overall framework follows the conventional Transformer design with minor additions, and therefore provides limited technical novelty. There is some room for improvement on the clarity, as some notations are not clearly defined or used to indicate different meanings at different paragraphs.

summary_of_the_review: The proposed approach shows some improvement compared to existing work, but it’s technical technical novelty is a bit limited.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This study proposes a decoder-encoder attention mechanism, a positional encoding mechanism and a decoder self-attention scheme to improve the forecast accuracy and minimise the excess variations in time series forecasting applications. As highlighted in the introduction section of the paper, both improving the forecast accuracy and minimising the unnecessary forecast volatility are important endeavours in the time series forecasting domain. Hence the motivations of this study are clear and strong. The proposed MQTransformer framework aims to obviate the limitations of MQ-Forecaster, which is also a state-of-the-art framework in the recent time series forecasting literature. 
main_review: Strengths

- A clear formal introduction to the problem statement and concise descriptions of the three contributions. The overall stricture and readability of the paper are good.

- Authors have attested their framework against state-of-the-art deep learning time series forecasting methods and evaluated using 5 benchmark datasets, including a one retail dataset, and four publicly available datasets. The benchmark methods and datasets used are well-established yardsticks in the multivariate forecasting (or global models where sets of many time series are available) domain. So, the experiments are strong and well empirically supported.

- The use of an ablation study to measure the effect of the proposed attention units and self-attention scheme. 

- Have also reported the computational cost of the framework and compared against the current state-of-the-art (Lim et al., 2019)


Weaknesses

- Only the ablated based variants of MQTransformer are evaluated and compared on the large-scale retail dataset. What is the reason for excluding other forecasting benchmarks, such as DeepAR, TFT, MQ-RNN in Table 3 ?

- The formula used to calculate the quantile loss (quantile forecast error) is not defined. Please add this formula to the appendix or main text to improve the clarity of the error measures summarised in the tables.

- I also strongly recommend the authors to include state-of-the-art Informer framework [1] as a benchmark in this study. This is because, Informer framework also follows the self- attention architecture, thus can be directly comparable against the proposed MQTransformer.

 [1] Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H. and Zhang, W., 2021, May. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of AAAI.

summary_of_the_review: Given the significance of the three contributions, supported by the strong experimental setup (well established benchmarks and datasets) and empirical evidence (statistical significance), I am recommending to accept this paper.
review_title: null

Meta-review: This paper proposes a number of improvements to the previously-published transformer-based MQ-forecaster model for multi-horizon forecasts on time series data. They show strong empirical improvements in terms of accuracy and excess forecast variability on a large proprietary dataset, as well as four public datasets. Concerns were raised about the relatively incremental changes to the MQ-forecaster model this work is based on, lack of ablations on public data and, relatedly, inability to reproduce results on the proprietary data.
Decision: Reject