MQTransformer: Multi-Horizon Forecasts with Context Dependent and Feedback-Aware Attention

Abstract: Recent advances in neural forecasting have produced major improvements in accuracy for probabilistic demand prediction. In this work, we propose novel improvements to the current state of the art by incorporating changes inspired by recent advances in Transformer architectures for Natural Language Processing. We develop a novel decoder-encoder attention for context-alignment, improving forecasting accuracy by allowing the network to study its own history based on the context for which it is producing a forecast. We also present a novel positional encoding that allows the neural network to learn context-dependent seasonality functions as well as arbitrary holiday distances. Finally we show that the current state of the art MQ-Forecaster (Wen et al., 2017) models display excess variability by failing to leverage previous errors in the forecast to improve accuracy. We propose a novel decoder-self attention scheme for forecasting that produces significant improvements in the excess variation of the forecast.

Reviews and author responses:



This paper aims at improving accuracy of multi horizon univariate time series forecasting. The authors propose an encoder-decoder attention-based architecture for multi-horizon quantile forecasting. The model encodes a distinct representation of the past for each requested horizon.

#### Strong points

+ The encoding of the holidays and special event specific to the time series is elegant.
+ The idea of explicitly using forecast error feedback is interesting.
+ Ablation study of the architecture's innovations on a large scale forecasting dataset mainly outlines the importance of the horizon specific component of the architecture.

#### Weak points

- The evaluation of the new method is conducted on only two public datasets and the new methods outperform TFT only on one. There exists other common datasets (DeepAR evaluates on three-parts and traffic, TFT evaluates on traffic and volatility) that should be considered to place this method in the literature. 
- Both manuscripts of Fine and Foster (2020 al b) are not published and I could not find them online. The summary of Fine and Foster (2020 a; b) in Section 2.3 is not enough for me to judge the relevance of the results in Fig 2 and 3.
- The strong claims at the end of the introduction are made compared to the ablated model (MQCNN) on the private dataset, not against alternative methods such as MQRNN or DeepAR which could scale to this dataset size.


#### Decision

I would tend to reject this submission. The proposed model exhibits none of: significant quantitative improvement (on public dataset), speed up, improved simplicity over alternative methods.

Additionally to the weak points mentioned above, the contribution of this paper is undermined by the following points: 
- The contribution of “Positional Encoding from Event Indicators” is rather incremental considering BERT encoding of input segments (e.g. Sentence A, Sentence B, Question, Answers).
- The Horizon-Specific encoding is interesting but does not allow to forecast at inference a horizon that has not been trained on (as parametric method such as DeepAR have).
- The code of the submission is not submitted and there is no mention of future release.

#### Questions

- Is your positional encoding method a superset of relative positional encoding?
- Can you provide runtimes of the MQTransformers? How does it compare to (Lim et al. 2019) ?
- In appendix B: What is the percentage of unseen object in the test/valid dataset that would be harder to forecast by TFT?
- Why do you use the large scale private dataset for the ablation of your method and not a public dataset? Are the architectural innovations only useful in the high data regime?

#### Additional feedback

- Figure 4 in appendix D should be mentioned in the main text as it is helpful for the reader.
- Typo in Table 1, eq (4), third line: $c_{t,1}^{a}$ should be $c_{t,1}^{hs}$ I believe.
- Section 3.3: what do you call a *bidirectional* 1-D convolution? (As opposed to unidirectional?)
- Caption of Figure 3: Try to be consistent on the short name of retail vs Favorita dataset.
- Eq (2): Consider providing the factorised form which is easier to parse.


--- 
**Update:** I would like to thank the authors for their answer. I acknowledge the improvement of the manuscript after the review process:

- Added clarifications on the baselines,
- Added helpful precisions for reproducibility (even though the code cannot be open sourced),
- Evaluation on 2 requested public datasets with good results.

However, I am still not confident to raise my score to 6 (marginally above the acceptance threshold) given the missing public manuscripts (or Appendix) to explain the martingale diagnostic tools. This hinder one of the main selling point of the paper that their model reduces the volatility of the forecast.

After rebuttal:
I appreciate authors' detailed responses and an updated version of the paper. The new version is a lot clearer. After reading other reviews, I agree that the algorithmic novelty is limited, but the model is well-adapted for multi-horizon forecasting problem. Overall, I increase my score to 6. marginally above acceptance threshold. 

----------------------------------------------------------
Summary:
This paper introduces a new model for multi-horizon forecasting. The proposed model is an extension of MQRNN with two new modules: task specific attention and decoder self-attention. The experiments on the large-scale demand forecasting dataset and other publicly available datasets show that the proposed model outperforms or is comparable with the CNN, RNN-based models as well as the transformer-based model. 
--------------------------------------
Pros:
+ Authors adapted the attention mechanism for a challenging problem (multi-horizon forecasting). 
+ The proposed model is evaluated on a large-scale dataset as well as existing public datasets. 
+ Additional experiments and Figure 4 on Appendix are helpful. 
--------------------------------------
Cons:
I found this paper lacks clarity. I list the issues below:

1. Authors stated that 'Our horizon-specific attention mechanism can be viewed as a multi-headed attention mechanism. Each head corresponds to a different horizon.'. I don't think this is true. Multi-head attention is an ensemble of attentions from the same input to attend different positions by incorporating different representations. The purpose of the horizon-specific attention in this paper is to merge the encodings of multiple horizon. I ask authors to clarify it.

2. Design choice: the main contribution of this paper is attention mechanisms (horizon-specific attention between encoder-decoder and decoder self-attention) for multi-horizon forecasting. 
Questions:
    1. Since the proposed model architecture is similar to MQRNN [Wen et al. 2017], it should be the baseline. What is the reason MQCNN is used as a baseline? 
    2. The proposed model performs better than MQRNN for public datasets. Which module in the model has a big role between horizon specific attention and decoder self-attention? Or is attention specifically handling better for multi-horizon forecasting problems than RNNs? Adding MQRNN as another baseline and the result without horizon specific attention for the large-scale demand forecasting would be helpful (in Table 3 and Figure 2). 

3. TFT [Lim2019] is an existing transformer-based model for Multi-horizon Time Series Forecasting. In my understanding, the major differences are horizon specific attention in the proposed model and a different design of decoder in TFT. Could you clarify the difference and similarities between these two models in the paper?
Also, authors stated that 'We were unable to compare to TFT (the prior state of the art on several public datasets) as it does not scale-up'. What does this mean? I assume TFT cannot be easily applied for large-scale demand forecasting. This needs more explanations what are the major difficulties to scale-up TFT for the problem. 

3. A description of experiment setup and model specification is missing (for both the proposed model and baseline). The results can not be easily reproduced. 
    - Hyper-parameters: the number of layers and hidden units, learning rate, optimizer, etc. 
    - Preprocessing if there is any.
--------------------------------------
Minor comments:
- A description of P50 and P90 in Figure 2, 3, and Table 4 is missing. 
- A description of LTSP in Section 4 is missing. 
- The baseline (MQCNN) has no reference. 

Continued from previous comment

-------
-------

6. "The code of the submission is not submitted and there is no mention of future release."

Unfortunately we were unable to obtain permission to release our code. However, we have added a significant amount of detail (see Appendices B and C) regarding the exact architecture and parameters used, so that our experiments can be reproduced by readers who are interested in doing so.

----

Questions:
1. "Is your positional encoding method a superset of relative positional encoding?"

It is not a superset of relative position encoding (Dai et al. 2019 - https://arxiv.org/pdf/1901.02860.pdf), but it is similar in spirit. By encoding event indicators (such as seasonal holiday indicators) with a convolution filter, has a similar inductive bias to that imposed by relative position encoding and similarly avoids the issues associated with using an absolute position encoding.

2. "Can you provide runtimes of the MQTransformers? How does it compare to (Lim et al. 2019) ?"

Thanks for this suggestion, we have added a note on computational efficiency to Section 4. On the public retail forecasting task, TFT requires 5 hours (5 epochs) to train a model using a single GPU, whereas our model only requires 15 minutes (5 epochs). Further, Lim et al. (2019) need to restrict their model to train on only 450K of 20M available trajectories during the train period, whereas our model is trained on all 20M trajectories.

3. "In appendix B: What is the percentage of unseen object in the test/valid dataset that would be harder to forecast by TFT?"

In this specific case, only about 2% of items would be harder to forecast. But in a real-world setting, this would occur much more frequently for because the retail task is only evaluated on a single forecast creation time, 30 days after the end of the training period. For real-world use cases, the model would be used for inference over a longer period of time, and the percentage of items being forecasted that were not seen in training would increase.

----

Additional Feedback:
1. "Figure 4 in appendix D should be mentioned in the main text as it is helpful for the reader."

Thanks for pointing this out, we now reference Figure 4 in the main text.

2. "Typo in Table 1, eq (4)," 

Thanks, this is fixed in the revised draft.

3. "Section 3.3: what do you call a bidirectional 1-D convolution? (As opposed to unidirectional?)"

We have now clarified this in Section 3.3. By bi-directional convolution we mean one which looks both forward and backward in time, as opposed to one which only looks backwards. For covariates which are not known a priori, the convolution must be backwards looking so that the model does not have access to the target it is supposed to predict.

4. "Caption of Figure 3: Try to be consistent on the short name of retail vs Favorita dataset."

Thanks for pointing this out. We now consistently use "retail" to refer to this dataset.


5. "Eq (2): Consider providing the factorised form which is easier to parse."

Thanks for the suggestion, we have updated equation (2) to use the factorized form.

Thank you for the time and effort spent reviewing our paper; the critical feedback has been very helpful in terms of strengthening this work, and we hope the revised draft satisfactorily addresses the concerns you raised. In particular, the revised draft shows (see below) that our method achieves substantial improvements in terms of quantile loss on the most challenging public dataset (retail) and is significantly more efficient to train than previous transformer models for forecasting (TFT) because we have made our architecture innovations compatible with the forking sequences training scheme.

Please find below the point by point responses to specific questions raised and suggestions made in the review. 

--------
1. "The evaluation of the new method is conducted on only two public datasets and the new methods outperform TFT only on one. There exists other common datasets (DeepAR evaluates on three-parts and traffic, TFT evaluates on traffic and volatility) that should be considered to place this method in the literature."

Thanks for bringing this up, and we agree that additional empirical results would help better place our method in the literature. We have added evaluation of both MQ-CNN (our baseline model) and MQTransformer on two additional tasks: Traffic and Volatility. As can be seen in Section 4 of the revised draft, our model provides significant improvements over the state of the art on the hardest public task (retail), and matches or provides small improvements on the simpler tasks (volatility, traffic and electricity). Specifically, our MQTransformer model outperforms the previously reported state-of-the-art (TFT) by 38%. Compared to the baseline MQ-CNN model we trained on the public task, it achieves 10% improvement in quantile loss. 

2. "Both manuscripts of Fine and Foster (2020 al b) are not published and I could not find them online. The summary of Fine and Foster (2020 a; b) in Section 2.3 is not enough for me to judge the relevance of the results in Fig 2 and 3."

We have communicated with the authors since receiving the review and they intend to publish soon. If it is necessary for the evaluation of our work, the authors have indicated we may add a supplementary appendix that details their work and provides its importance.

To provide a little more context as to why forecast volatility matters: the primary effect in Supply Chains is that it reduces what is known as "bull-whip" effect. Though the cited work (http://faculty.haas.berkeley.edu/ned/AugenblickRabin_MovementUncertainty.pdf) of Augenblick and Rabin connects martingality to internal consistency of the forecast, bull-whip effects are one of the largest cost effects for supply chains (https://sloanreview.mit.edu/article/the-bullwhip-effect-in-supply-chains/) and reducing the forecasts effect on it by reducing forecast volatility is a major focus for research in supply chains. Our architectures produce anywhere from a ~10%-50% reduction in volatility without adding an auxiliary loss or creating a trade-off with accuracy.

3. "The strong claims at the end of the introduction are made compared to the ablated model (MQCNN) on the private dataset, not against alternative methods such as MQRNN or DeepAR which could scale to this dataset size."

Thanks for raising this point -- we agree we did not make it sufficiently clear in the original draft why MQ-CNN is used as the baseline (rather than models such as MQ-RNN or DeepAR). The reason we do not compare to MQ-RNN or DeepAR is prior work (Wen et al. 2017) showed that on the same private dataset (Figure 3 here - https://arxiv.org/pdf/1711.11053.pdf), MQ-CNN does outperform both MQ-RNN and DeepAR (the model Seq2SeqC in Wen et al (2017) is a comparison to an improvement of DeepAR).

4. "The contribution of “Positional Encoding from Event Indicators” is rather incremental considering BERT encoding of input segments (e.g. Sentence A, Sentence B, Question, Answers)."

We believe our position encoding scheme is rather different than the BERT encoding of input segments. The BERT encoding is still a matrix embedding scheme (like traditional position encodings), whereas ours is a more general mapping from an arbitrary sequence of indicators to a position representation. 


5. "The Horizon-Specific encoding is interesting but does not allow to forecast at inference a horizon that has not been trained on (as parametric method such as DeepAR have)."

Thanks for raising this point. We do agree that this is a drawback of all direct models -- they can only be used for inference at horizons they have been trained on. In practice, however, the horizons at which forecasts are required are typically known ahead of time, and the gains in performance (accuracy) outweigh the loss in flexibility. 

----------
----------
Reply continued in next comment

Continued from previous comment

---------
----------
5. "Also, authors stated that 'We were unable to compare to TFT (the prior state of the art on several public datasets) as it does not scale-up'. What does this mean? I assume TFT cannot be easily applied for large-scale demand forecasting. This needs more explanations what are the major difficulties to scale-up TFT for the problem."

Thanks for this suggestion -- we agree some more explanation of the difficulty in scaling TFT is needed. We have added a paragraph on computational efficiency and training schemes to Section 4.

In terms of how TFT scales, it's training scheme treates each entity, time pair as a distinct sample -- TFT does not support training using the forking sequences scheme. For the public retail task, Lim et al. (2019) restrict their training to just 450K out of 20M available trajectories and still requires 13 minutes per epoch (with the optimal model found after 6 epochs for the results reported in Lim et al. 2019) to train a model. By contrast, it takes 5 minutes per epoch (with the minimum validation error achieved in 5 epochs) to train our model on the public retail task, but we train on all 20M trajectories. For the non-public dataset on which we trained our model, TFT would need to be trained on 234M trajectories, and this is clearly intractable. 

6. "A description of experiment setup and model specification is missing (for both the proposed model and baseline). The results can not be easily reproduced."

We have added substantially more detail in Appendices B and C so that our experiments can be reproduced.

---------
Reply to minor comments
1. "A description of P50 and P90 in Figure 2, 3, and Table 4 is missing."

Thanks -- we have added these to the table captions.

2. "A description of LTSP in Section 4 is missing."

Thanks -- the definition can be found at the start of section 4.

3. "The baseline (MQCNN) has no reference."

MQCNN was defined in Wen et al (2017). Figure 3 here - https://arxiv.org/pdf/1711.11053.pdf - contains results comparing MQRNN to MQCNN. Section 4 also discusses improvements using different encoders. We have made it clearer in the revised draft which prior work (Wen et al. 2017) is the reference for the MQ-CNN architecture.

Thank you for the time spent reviewing our paper and for providing such detailed feedback. This has been very helpful to us and has strengthened the work in the revised draft. Below we respond to your detailed suggestions point by point.

------
Point by point:

1. "Authors stated that 'Our horizon-specific attention mechanism can be viewed as a multi-headed attention mechanism. Each head corresponds to a different horizon.'. I don't think this is true. Multi-head attention is an ensemble of attentions from the same input to attend different positions by incorporating different representations. The purpose of the horizon-specific attention in this paper is to merge the encodings of multiple horizon. I ask authors to clarify it."

We are happy to clarify this point, and agree that our attention mechanism serves a different purpose than a traditional multi-headed attention. The text now reads:
> Our horizon-specific attention mechanism is a multi-headed attention mechanism where the projection weights are shared across all horizons. Each head corresponds to a different horizon. It differs from a traditional multi-headed attention mechanism in that its purpose is to attend over representations of past time points to produce a representation specific to the target period. 

2. "Since the proposed model architecture is similar to MQRNN [Wen et al. 2017], it should be the baseline. What is the reason MQCNN is used as a baseline?"

Thank you for bringing this to our attention -- we agree that we did not clearly explain why MQ-CNN is used as the baseline. MQCNN was also developed by Wen et al (2017) and named 'MQCNN_Wave' (See Figure (3) here - https://arxiv.org/pdf/1711.11053.pdf). It dramatically outperforms MQRNN on the private dataset they use. Our MQTransformer architecture is similar to both MQ-RNN and MQ-CNN (both of which are proposed by Wen et al. (2017)). We have changed some wording in the paper to emphasize that MQ-RNN and MQ-CNN are almost the exact same architecture, and the differ only in choice of the encoder. We have also added the following clarification to the empirical results section to explain the choice of baseline:

> MQ-CNN is selected as the baseline since prior work [Wen et al. 2017, Figure 3 shows MQ-CNN (labeled "MQ_CNN_wave") outperforms MQ-RNN (all variants) and DeepAR (labeled "Seq2SeqC") on the test set] demonstrates that MQ-CNN outperforms MQ-RNN and DeepAR on this dataset, and as can be seen in Table 4, MQ-CNN similarly outperforms MQ-RNN and DeepAR on public datasets.

3. "The proposed model performs better than MQRNN for public datasets. Which module in the model has a big role between horizon specific attention and decoder self-attention? Or is attention specifically handling better for multi-horizon forecasting problems than RNNs? Adding MQRNN as another baseline and the result without horizon specific attention for the large-scale demand forecasting would be helpful (in Table 3 and Figure 2)."

Thanks for raising this concern -- similar to point 1, above, we agree that we did not clearly establish what architecture is appropriate to use as a baseline. MQ-CNN is the appropriate baseline because MQ-CNN outperforms MQ-RNN on every single dataset on which we evaluated. We have added results for MQ-CNN on the public dataset to illustrate this. In terms of which attention units made the most difference, they both are indeed necessary which we demonstrate in the ablation analysis on the non-public dataset in Section 4.1.

4. "In my understanding, the major differences are horizon specific attention in the proposed model and a different design of decoder in TFT. Could you clarify the difference and similarities between these two models in the paper?"

Thanks for bringing this up -- there are in fact other important differences too, including the design of the encoder. We have placed greater emphasis on the differences between TFT and MQ-Forecasters in our revised draft (see response to the next concern raised).

--------------
--------------
Reply continued in next comment

Thank you for the time and effort spent reviewing our paper; the critical feedback has been invaluable and we hope the revised draft addresses the concerns raised. We respond to them point by point below.

1. "Datasets / tasks are limited. Paper could benefit from a few more."

Thank you for this suggestion -- we agree that a more thorough empirical evaluation would be valuable. We have added evaluations on two additional tasks: traffic and volatility.

2. "In equation 2, it's more clear to write as (x_s + r_s)^T W_q^h W_k^h (x_t + r_t), no?"

Thanks for this suggestion, we have changed equation (2) to use the factorized form. 

3. "Modeling novelty is limited"

Thanks for the feedback on this point -- in our opinion (and we may not have emphasized this enough in the original submission), something that is quite novel about this work is that we consider the impact of architecture design on reducing excess forecast volatility. To the best of our knowledge, all previous work to consider the volatility of forecasts as they evolve did so by adding a penalty term to the objective function. Implicit in that formulation is there is a tradeoff to be made between greater accuracy and lower forecast volatility. What we show in this work is that by considering the impact of architecture design on forecast evolution, it is possible to both reduce excess volatility and increase accuracy.

Furthermore, we believe that the horizon specific context awareness has applicability beyond our problem. In many tasks (such as finance, weather prediction) future information is adapted to the current filtration - examples including earnings dates and option prices in finance, or rare meteorological phenomena in weather. Using that future information to specify the encoding of the past is a novel application that we have not seen in the literature.


We want to thank all the reviewers for the time and effort spent reviewing our paper.

Before responding to each review individually, we want to give a a high-level summary of the changes made and address some concerns common to all the reviews:

1. A common theme in the reviews was that we did not explain clearly why MQ-CNN was used as the baseline on the private dataset, and that a citation was not provided for the model. We agree that we did not make this clear, and we apologize for the confusion. Wen et al. 2017 propose both MQ-RNN and MQ-CNN (although they place greater emphasis on MQ-RNN). In their empirical work, they use the exact same private dataset we do in this paper, and they showed that MQ-CNN outperforms both MQ-RNN and DeepAR on that dataset (see Figure 3 - https://arxiv.org/pdf/1711.11053.pdf - of their paper, MQ-CNN is labeled "MQ_CNN_wave" and an improvement of DeepAR is labeled "Seq2SeqC"). For this reason, we use MQ-CNN as the baseline. We have updated Section 4 of our paper to emphasize this point.

2. Appendices B and C in the revised draft contain a detailed description of the architectures, parameters and training procedures used in each experiment. We agree that the original draft lacked the necessary information to reproduce our results, and we hope that these changes address that concern. Although we are unable to obtain permission to release the source code used, there are now sufficient details so that other researchers can reproduce the results.

3. Section 4 of the revised draft contains evaluations of MQ-CNN and MQTransformer on two additional, public tasks from the literature: traffic and volatility forecasting. We demonstrate on these datasets as well that MQTransformer matches or exceeds previous state of the art performance. Furthermore, in the original submission, we did not apply z-score normalization to either input or target variables as our model did not require this additional pre-processing to achieve state of the art performance. Based on feedback received, we re-ran the public data experiments using the same z-score normalization across entities as prior work (e.g. Lim et al 2019). After doing this, we found that on the public retail task, MQTransformer outperforms the previous reported state-of-the-art (TFT) by 38\%. Compared to the baseline experiment we ran using MQ-CNN, the gains are 5\% at P50 and 11\% at P90.

Again, thank you for all the helpful feedback -- we have found it to be very valuable, and we hope that the revisions address the concerns raised.


(Note: I am not well-versed in the forecast modeling literature but I am reasonably so in the use of Transformer models in NLP tasks)

The paper proposes MQTransformer, an improvement on MQRNN (Wen 2017) for multi-horizon forecast prediction that leverages the Transformer architecture. Their contributions are:
1) using learnable positionable embedding from event indicators
2) using an attention head for each of the k horizons that need to be forecasted
3) applying decoder self-attention so that the model can use one horizon prediction to improve a later one

The technical / modeling contributions aren't very novel or profound, so the ICLR community at large may not find them that inspiring. However, as backed by the experiments, these contributions do translate into non-trivial improvements on real-world datasets, so I expect the practical impact on real-world forecasting tasks to be high. I marginally support accepting this paper for this reason.

Pros:
* Well written overall
* Non-negligible improvement over baseline methods on "RETAIL" dataset
* Likely to be used in practice

Cons:
* Datasets / tasks are limited. Paper could benefit from a few more.
* Modeling novelty is limited

Suggestions for improvement:
* In equation 2, it's more clear to write as (x_s + r_s)^T W_q^h W_k^h (x_t + r_t), no?



Meta-review: The paper proposes and uses a fairly involved attention based architecture to perform time series forecasting. The idea of transformers is raised, but, given how sequence embedding is often convolutional, and position encoding input is provided to the model (albeit implictly in the form of features having to do with qualitative things such as promotions, etc among other things), I'm  of the opinion it is closer to the paper "Convolutional Sequence to Sequence Learning" https://arxiv.org/abs/1705.03122 than it is to transformers per se... Also the connection to sequence to sequence is not clear, since the chain rule of probability isn't really stressed on much. 

The paper proposes some interesting ideas, but I feel that it failed to convince the reviewers of the utility and the novelty. Part of it has to do with the clarity of presentation, and part of it, I think has to do with the fact that paper jam packs a bunch of different ideas together, without carefully ablating the influence of their various ideas. For example, transfomers have been applied for time sequences (https://arxiv.org/pdf/2001.08317.pdf), and its not clear in what ways this paper improves on them -- is it the complicated attention model scheme ? or is it the multi-horizon context schemes ?  

That being said, the results shows are relatively decent and the reviewers liked that aspect. Had the paper been easier to follow and the ideas presented with a little more insight it would be been a better fit for ICLR. As it stands, I have to give it a weak reject.

Decision: withdrawn-rejected-submissions