MQTransformer: Multi-Horizon Forecasts with Context Dependent and Feedback-Aware Attention
Authors: Carson Eisenach
Yagna Patel
Dhruv Madeka
Venue:  (2021)

Abstract: Recent advances in neural forecasting have produced major improvements in accuracy for probabilistic demand prediction. In this work, we propose novel improvements to the current state of the art by incorporating changes inspired by recent advances in Transformer architectures for Natural Language Processing. We develop a novel decoder-encoder attention for context-alignment, improving forecasting accuracy by allowing the network to study its own history based on the context for which it is producing a forecast. We also present a novel positional encoding that allows the neural network to learn context-dependent seasonality functions as well as arbitrary holiday distances. Finally we show that the current state of the art MQ-Forecaster (Wen et al., 2017) models display excess variability by failing to leverage previous errors in the forecast to improve accuracy. We propose a novel decoder-self attention scheme for forecasting that produces significant improvements in the excess variation of the forecast.

Under review as a conference paper at ICLR 2021
MQTRANSFORMER: MULTI-HORIZON FORECASTS WITH CONTEXT DEPENDENT AND FEEDBACK-AWARY ATTENTION
Anonymous authors Paper under double-blind review
ABSTRACT
Recent advances in neural forecasting have produced major improvements in accuracy for probabilistic demand prediction. In this work, we propose novel improvements to the current state of the art by incorporating changes inspired by recent advances in Transformer architectures for Natural Language Processing. We develop novel recoder-encoder attention for context-alignment, improving forecasting accuracy by allowing the network to study its own history based on the context for which it is producing a forecast. We also present a novel positional encoding that allows the neural network learn context-dependent seasonality functions as well as arbitrary holiday distances. Finally we show that the current state of the MQ-Forecaster (Wen et al., 2017) models display excess variability by failing to leverage previous errors in the forecast to improve accuracy. We pro- pose novel decoder-self attention scheme for forecasting that produces significant improvements in the excess variation of the forecast.
INTRODUCTION
Time series forecasting is a fundamental problem in machine learning with relevance to many application domains including supply chain management, finance, healthcare analytics, and more. Modern forecasting applications require predictions of many correlated time series over multiple horizons. In multi-horizon forecasting, the learning objective is produce forecasts for multiple future horizons at each time-step. Beyond simple point estimation, decision making problems require measure of uncertainty about the forecasted quantity. Access to the full distribution is usually unnecessary. and several quantiles are sufficient (many problems in Operations Research use the 50th and 90th percentiles, for example).
As motivating example, consider a large e-commerce retailer with a system to produce forecasts of the demand distribution for a set of products at a target time T. Using these forecasts as input, the retailer can then optimize buying and placement decisions to maximize revenue and/or customer value. Accurate forecasts are important, but - perhaps less obviously - forecasts that don't 't exhibit excess volatility as a target date approaches minimize costly, bull-whip effects in a supply chain (Chen et al., 2000; Bray and Mendelson, 2012).
Recent work applying deep learning to time-series forecasting focuses primarily on the use of recurrent and convolutional architectures (Nascimento et al., 2019; Yu et al., 2017; Gasparin et al., 2019; Mukhoty et al., 2019; Wen et al., 2017)' These are Seq2Seq architectures (Sutskever et al., 2014) - which consist of an encoder which takes an input sequence and summarizes it into a fixed- length context vector, and decoder which produces Seq2Seq models suffer from an information bottleneck by transmitting information from encoder to decoder via single hidden state. To address this Bahdanau et al. (2014) introduces method called attention, allowing the decoder to take as input weighted combination relevant latent encoder states at each output time step, rather than using single context to produce all decoder outputs. While NLP the predominate application of attention architectures, this paper we show how novel attention modules and positional embeddings can be used to introduce proper inductive biases for probabilistic time-series forecasting to the model architecture.
output sequence. It is well known that
For complete overview see Benidis al. (2020)

---

Under review as a conference paper at ICLR 2021
Even with these shortcomings, this line of work has lead to major advances in forecast accuracy for complex problems, and real-world forecasting systems increasingly rely on neural nets. Accordingly, need for black-box forecasting system diagnostics has arisen. Stine and Foster (2020b:a) use probabilistic martingales to study the dynamics of forecasts produced by an arbitrary forecasting system. They can be used to detect the degree to which forecasts adhere to the martingale model of forecast evolution (Heath and Jackson, 1994) and to detect unnecessary volatility (above and beyond any inherent uncertainty) in the forecasts produced. Thus, Stine and Foster (2020b:a) describe way to connect the excess variation of a forecast accuracy misses against the realized target.
While, multi-horizon forecasting networks such as (Wen et al., 2017; Madeka et al., 2018), minimize quantile loss the network architectures do not explicitly handle excess variation, since forecasts on any particular date are not made aware of errors in the forecast for previous dates. In short, such tools can be used detect flaws in forecasts, but the question of how to incorporate that information into model design is unexplored.
Our Contributions In this paper, reducing excess forecast volatility. We present a set of novel architectures that seek to remedy some of inductive biases that are currently missing in state of the art MQ- NQ-Forecastters (Wen et al., 2017). The major contributions of this paper are
concerned with both improving forecast accuracy and
1. Positional Encoding from Event Indicators: Current MQ-Forecasters use explicitly engineered holiday "distances" to provide the model with information about the seasonality of the time series, We introduce novel positional encoding mechanism that allows the network learn a seasonality function depending on other information of the time series being forecasted, and demonstrate that its a strict generalization of conventional position encoding schemes.
2. Horizon-Specific Decoder-Encoder Attention: Wen et al. (2017); Madeka et al. (2018) and other MQ-Forecssters learn single encoder representation for all future dates and periods being forecasted. We present a novel horizon-specific decoder-encoder attention scheme that allows the network to learn a representation of the past that depends which period is being forecasted.
3. Decoder Self-Attention for Forecast Evolution: work to consider the impacts of network architecture design on forecast evolution. Importantly, we accomplish this by using attention mechanisms to introduce the right inductive biases, and not by explicitly penalizing measure of forecast variability. This allows us to maintain a single objective function without needing to make trade-offs between accuracy and volatility.
To the best of our knowledge, this is the first
By providing MQ- Foreccasssssss with the structure necessary to learn context information dependent encodings, we observe major increases in accuracy (3.9% in overall P90 quantile loss throughout the year, and up to 60% during peak periods) on our demand forecasting application along with a signifi- cant reduction in excess volatility (52% reduction in excess volatility at P50 and 30% at P90). We also apply MQTransformer to four public datasets, and show parity with the state-of-the-an on simple, univariate tasks. On a substantially more complex public dataset (retail forecasting) we demonstrate a 38% improvement over the previously reported sfate-of-the-art, and 5% improvement in P50 QL, 1 11% in P90 QL versus our baseline. Because our innovations are compatible with efficient training schemes, our architecture also achieves significant speedup (several orders of magnitude greater throughput) over earlier transformer models for time-series forecasting.
2 BACKGROUND AND RELATED WORK
2.1 TIME SERIES FORECASTING
Formally, the task considered in our work is the high-dimensional regression problem
(h) p 1e X
(1)
where Vt-te, Vit.i X E. (*) denote future observations of the target time series i, observations
(h)
(
of the target time series observed up until time t, the past covariates, known future information, and static covariates, respectively.
For sequence modeling problems, Seq2Seq (Sutskever et al., 2014) is the canonical deep learning framework and although applied this architecture to neural machine translation (NMT) tasks, it has
2

---

Under review as a conference paper at ICLR 2021
since been adapted to time series forecasting (Nascimento al., 2019; Yu et al., 2017; Gasparin et al., 2019; Mukhoty et al., 2019; Wen et al., 2017; Salinas et al., 2020; Wen and Torkkola, 2019). The MQ-Forecaster framework (Wen et al., 2017) solves (1) above by treating each series i as sample from joint stochastic process and feeding into a neural network which predicts Q quantiles for each horizon. These types of models, however, inherit from the Seq2Seq architecture the limited contextual information available to the decoder as produces each estimate V the tth quantile of the distribution of the target at time t + s s/t+,,. Seq2Seq models rely on a single encoded context to produce forecasts for all horizons, imposing an information bottleneck and making it difficult for the model understand long term dependencies.
Our MQTransformer architecture, like other MO-Forecasters, uses the direct strategy: the model outputs the quantiles of interest directly, rather than the parameters of distribution from which samples are to be generated. This has been shown (Wen et al., 2017) to outperform parametric models, like DeepAR (Salinas et al., 2020), on a wide variety of tasks. Recently, Lim et al. (2019) consider an application of attention to multi-horizon forecasting, but their method still produces a single context for all horizons. Furthermore, by using an RNN decoder their models do not enjoy the same scaling properties as MQ-Forecaster models. To the best of our knowledge, our work is the first to devise attention mechanisms for this problem that readily scale.
2.2 ATTENTION MECHANISMS
Bahdanau et al. (2014) introduced the concept of an attention mechanism to solve the information bottleneck and sequence alignment problems in Seq2Seq architectures for NMT. Recently, attention has enjoyed success across a diverse range of applications including natural language processing (NLP), computer vision (CV) and time-series forecasting tasks (Galassi et al., 2019; Xu et al., 2015; Shun-Yao Shih and Fan- Keng Sun and Hung-yi Lee, 2019; Kim and Kang, 2019; Cinar et al., 2017; Li et al., 2019; Lim et al., 2019). Many variants have been proposed including self attenttnnoon and dot-produce attention (Luong et al., 2015; Cheng et al., 2016; Vaswani et al., 2017: Devlin et al., 2019), and transformer architectures (end-to-end attention with no recurrent layers) achieve otate-of-the-art performance on most NLP tasks.
Time series forecasting applications exhibit seasonal trends and the absolute position encodings commonly used in the literature cannot be applied. Our work differs from previous work on relative position encodings (Dai et al., 2019; Huang et al., 2018; Shaw et al., 2018) in that we learn a representation from a time series of indicator variables which encode events relevant to the target application (such as holidays and promotions). If event indicators relevant to the application are provided, then this imposes a strong inductive bias that will allow the model to generalize well to future observations. Existing encoding schemes either involve feature engineering (e.g. sinusoidal encodings) or have a maximum input sequence length, ours requires no feature engineering the model learns it directly from raw data and it extends to arbitrarily long sequences.
In the vanilla transformer (Vaswani et al., 2017), sinusoidal position embedding is added to the network input and each encoder layer consists of multi-headed attention block followed by a feed-forward sub-layer. For each head i, the attention score between query qs and key k, is defined as follows for the input layer
A", = (x.+ T.. W +r.)
(2)
where x., rs are the observation of the time series and the position encoding, respectively, at time s. Section 3 introduces attention mechanisms that differ in their treatment of the position dependent biases. See Appendix A for additional discussion of attention mechanisms.
2.3 MARTINGALE DIAGNOSTICS
Originally the martingale model forecast evolution (MMFE) was conceived as way to simulate demand forecasts used in inventory planning problems (Heath and Jackson, 1994). Denoting by Prye the forecast for Yr made at time S T. the MMFE assumes that the forecast process (frie is martingale. Informally, a martingale captures the notion that a forecast should use all information available the forecasting system time I. Mathematically, discrete time martingale is stochastic process (X,) such that I XX.......... X1] = X,. We assume a working knowledge of martingales and direct the reader to Williams (1991) for a thorough coverage in discrete time.
3

---

Under review as a conference paper at ICLR 2021
Taleb (2018) describe how martingale forecasts correspond to rational updating, then expanded by Augenblick and Rabin (2019). Taleb (2018), Taleb and Madeka (2019) and Augenblick and Rabin (2019) go on to develop tests for forecasts that rule out martingality and indicate irrational or
predictable updating for binary bets. Stine and Foster (2020a;b) further extend these ideas to quantile forecasts. Specifically, they consider the coverage probability process p. == P[YT S </ S () B[T(Yr S <V, I t), where denotes the forecast announced in the first period = 0. Because
{p.} is also a martingale, the authors show that E[(rT - ] =It= El(p: pt-1)²] = m -)), where Po is the expected value of Pr. Bernoulli random variable, across realizations of the coverage process. In the context of quantile forecasting, I is simply the quantile forecasted. The measure of excess volatility proposed is the quadratic variation process associated with {pi }. Q, : -i= (pr Pt- While this process is not a martingale, we do know that under the MMFE assumption, E[Qr] x1-xxt). x).
Q. second quantity of interest is the martingale V; = (p:-paar =) which follows the typical structure of subtracting the compensator to turn a sub-martingale into martingale. In Section 4 we leverage the properties of {Vi} and [Qr} compare the dynamics of forecasts produced by variety of models, demonstrating that our feedback-aware decoder self self-attention units reduce excess forecast volatility.
3 METHODOLOGY
As mentioned, this work is motivated in part by the needs of the consumers of forecasting systems. We therefore care about whether not innovations can be used in practice. Our methodology must scale to forecasting tens of thousands or millions of signals, at hundreds of horizons. We extend the MQ-Forecaster family of models (Wen et al., 2017) because it, unlike many other architectures considered in the literature, can be applied at large-scale (millions samples) due to its use of forking sequences technique to dramatically increase the effective batch size during training and avoid expensive data augmentation. In this section we present our MQTransformer architecture, building upon the MQ-Forecaster framework.
For ease of exposition, we reformulate the generic probabilistic forecasting problem in (I) as xx,xx(1) x(s),,)), where X.t, are past observations of all covariates, x je are known covariates specific to time-series x(9) (x!") ) are the global, known covariates. In this setting, known signifies that the model has access to (potentially noisy) observations of past and future values. Note that this formulation is equivalent to (1), and that known covariates can be included in the past covariates X.,. When it can be inferred from context, the time series index i is omitted.
plyc+1,i..... Mt Xa.i, (1)
3.1 LEARNING OBJECTIVE
We train quantile regression model to minimize the quantile loss, summed over all forecast creation dates and quantiles E, E, Σn Ly (( where Ly f) 4 g)+ + (1
( y)++
(++ the positive part operator, q denotes a quantile, and denotes the horizon.
3.2 NETWORK ARCHITECTURE
The design of the architecture is similar to MQ-RNN (Wen et al., 2017), and consists encoder, decoder and position encoding blocks (see Figure 4 in Appendix B). The position encoding outputs, for each time step t, are a representation of global position information, x PE..)
(9) (9) as well as time- series specific context information, r PE. ())). Intuitively, r,
(x
(9) captures position
information that is independent of the time-series i (such as holidays), whereas (1)) encodes time- series specific context information (such promotions). In both cases, the inputs indicator variables and require no feature- eratuei-nngineering or handcrafted functioon
time series of
The encoder then summarizes past observations of the covariates into sequence of hidden states h := encoder(y :t, Xx,, (0))) rr((), s). Using these representations, the decoder produces an x Q matrix of forecasts Y, = decoder(h,; h.,, r(s), p())). Note that in the decoder, the model has access to position encodings.
4

---

Under review as a conference paper at ICLR 2021
Figure 1: Position encoding learned from daily-grain event indicators
In this section we focus on our novel attention blocks and position encodings; the reader is directed to Appendix B for the other architecture details.
MQTransformer Now we describe design, evaluated in Section following the generic pattern given above. We define the combined position encoding as r (,(9)r(()) In the encoder a stack of dilated temporal convolutions (van den Oord et al., 2016; Wen et al., 2017) to encode historical time-series and multi-layer perceptron encode the statie features (3).
Table I: MQTransformer encoder and decoder
ENCODER
DECODER CONTEXTS
h CENV:
(3)
Cτ ISATTENTION(L....
(4)
h FEEDFORWARD(S)
C FoEDDORWARD(h,.r)
h, [h]:
[ci,i :c
C.. DSATTENTION(C:; ha.r),
Our decoder incorporates our horizon specific and decoder self-attention blocks, and consists of two branches. The first (global) branch summarizes the encoded representations into horizon-specific (ct.) and horizon agnostic (cf) contexts. Formally, the global branch := mm(-) is given by (4).
The output branch consists of self-attention block followed by local MI.,, which produces outputs using the same weights for each horizon. For FCT and horizon h, the output is given by 2Q
(We+h>--. ... ( (cf c/ Cr, Te+h), where c, denotes the output of the global branch, up through the FCT L. Next describe the specifies of our position encoding and attention blocks.
3.3 LEARNING POSITION AND CONTEXT REPRESENTATIONS FROM EVENT INDICATORS
Prior work typically uses variant on one two approaches to provide attention blocks with position information: (1) handerafted representation (such as sinusoidal encodings) (2) matrix M E f. of position encoding where is the maximum sequence length and each row corresponds to the position encoding for time point.
In contrast, our novel encoding scheme maps sequences indicator variables to a d-dimensional representations. For demand forecasting, this enables our model to learn an arbitrary function of events (like holidays and promotions) to encode position information. As noted above, our model
includes two position encodings: 7(9) = PE(9) (x(9)) and r == PE! (1) (x)"), ), one that is shared among all time-series i and one that is specific. For the design we use in Section 4, PE is implemented bidirectional 1-D convolution (looking both forward and backward in time) and (E()) is MLP applied separately each time step. Figure shows example of P PE(*) learned from holiday indicator variables. For reference, MQ-(C)RNN (Wen et al., 2017) uses linear holiday and promotion distances to represent position information.
Connection to matrix embeddings Another way to view our position encoding scheme is as a form of set based indexing into rows of an infinite dimensional matrix. We note that the traditional method of learning a matrix embedding M can be recovered as a special case of our approach. Consider sequence length L, and take x := (e....., ., ez] where e, is used denote the vector in TR with in the th position and Os elsewhere. To recover the matrix embedding scheme,

---

Under review as a conference paper at ICLR 2021
( M. Thus see that scheme is strict generalization of the matrix embedding approach commonly used in the NLP community.
define (x(=)))(())) x
3.4
CONTEXT DEPENDENT AND FEEDBACK-AWARE ATTENTION
Table 2: Attention weight and output computations for blocks introduced in Section 3.4
BLOCK
ATTENTION WEIGHTS
OUTPLT
DECODER-ENCODEN
A q T. W. Weke
(5)
c(h =
Σ
Al..v
(6)
ATTENTION
q herATe+A
h.:r.]
V. h.
DECODER SELF-ATTENTION
A Ser qrh
T Wh. T (7)
E
A Verr
(8)
qr [bg lhgei,ii ha
(t.))E
H(t,, = ((+))
k.. (ch,ri
he Var C..r
Horizon-Speciffe Decoder-Encoder Attention Our corizon-specific attention mechanism multi-headed attention mechanism where the projection weights are shared across all horizons. Each head corresponds to a different horizon. It differs from a traditional multi-headed attention mechanism in that its purpose is to attend over representations of past time points to produce a representation specific to the target period. In our architecture, the inputs to the block are the encoder hidden states and position encodings. Mathematically, for time s and horizon h, the attention weight for the value time is computed as (5).
Observe that there are two key differences between these attention scores and those the vanilla transformer architecture: (a) projection weights are shared by all H heads, (b) the addition of the position encoding of the target horizon h to the query. The output our horizon specific decoder- encoder attention block, ch is obtained by taking weighted sum of the encoder hidden contexts, up to maximum look-back of L periods in (6).
Decoder Self-Attention The martingale diagnostic tools developed in (Stine and Foster, 2020b) indicate a deep connection between accuracy and volatility. We leverage this connection to develop a novel decoder self-attention scheme for multi-horizon forecasting. To motivate the development, consider a model which forecasts values of 40, 60 when the demand has constantly been 50 units. We would consider this model have excess volatility. Similarly, a model forecasting 40, 60 when demand jumps between 40 and 60 units would not be considered to have excess volatility. This is because the first model fails learn from its past forecasts continues jumping between 40, 60 when the demand is 50 units.
In order to ameliorate this, we need to pass the information of the previous forecast errors into the current forecast. For each FCT and horizon h. the model attends on the previous forecasts using a query containing the demand information for that period. The attention mechanism has separate for each horizon.
Rather than attend on the demand information and prior outputs directly, richer representations of the same information is used: the demand information at time t incorporated via the encoded context h, and previous forecasts are represented via the corresponding horizon-spectis context chs the absence of decoder-self attention che would be passed through the local MLP to generate the forecasts. Formally, the attention scores are given by (7). The horizon-specific and feedback-aware outputs, c. are given by (8). Note how sum only over previous forecasts of the same period.

---

Under review as a conference paper at ICLR 2021
Table Aggregate Quantile Loss Metries
SEASONAL PEAK
POST- RAMPDOWN
PROMOTION
MODEL
ALL LTSP
LTSP 0/4
TYPE 1
BASELINE
1,000
1.000
1,000
1.000
1.000
DEC-ENC
0.984
0.931
0.748
0.712
0.706
DEC-ENC + DEC-SELF
0.989
0.908
0.698
0.639
0.670
4 EMPIRICAL RESULTS
4.1 -CAGE-SCALE DEMAND FORECASTING
First, we evaluate our architecture on demand forecasting problem for large-scale e-commerce retailer with the objective of producing multi-horizon forecasts that span up to one year. Each horizon is specified by a lead time (LT), number of periods from the FCT to the start of the horizon, and a span (SP), number of periods covered by the forecast, combination. To assess the effects of each innovation, ablate by removing components one time. MQTransfrrmer is denoted as Dec-Enc & Dec-Self Att, Dec-Enc Att - which contains only the horizon- decoder-encoder unit- and Baseline the vanilla MQ-CNN model. MQ-CNN is selected the baseline since prior work" demonstrate that MQ-CNN outperforms MQ-RNN and DeepAR on this dataset, and as can be seen in Table 4, MQ-CNN similarly outperforms MQ-RNN and DeepAR on public datasets.
We conduct our experiments on subset of products (~ 2 million products) in the US store. Each model is trained using single machine with NVIDIA V100 Tensor Core GPUs, on three years of demand data (2015-2018); one year (2018-2019) is held out for back-testing.
Forecast Accuracy Table summarizes several key metries that demonstrate the accuracy im- provements achieved by adding our proposed attention mechanisms to the MQ-CNN architecture the full set of results can be found in Appendix C. Introducing the Horizon-Speeifhe Decoder- Encceer attention alone yields improvements along all metrics evaluated. Overall we see 1.6% improvement in P50 QL and a 3.9% improvement in P90 QL. Notably, the attention mechanism yields significant improvements on short LT-SP (LT-SP 0/4).
Further, Table 3 demonstrates improved performance on seasonal peaks and promotions. Observe that while MQ-CNN performs well on some seasonal peaks, also is misaligned and fails ramp-down post-peak post-peak ramp-down issues occur when the model continues to forecast high for target weeks after the peak week. By including MQTransformer'' attention mechanisms in the architecture, we see a 43% improvement for Seasonal Peak and a 56% improvement on Post-Peak Rampdown. In retail, promotions able to react to the upcoming promotion and forecast an accurate lift in demand for the target weeks in which the promotion is placed. From Table 3 see that MQTransformer achieves items with Promotion Type versus the baseline.
used to provide demand lift for products. Accordingly, model should be
49% on
Forecast Volatility We study the effect of our proposed attention mechanisms excess forecast volatility using diagnostic tools recently proposed by Stine and Foster (2020b;a). Figure plots the process {Vi} (see Section 2). In the plot, the lines should appear horizontal under the MMFE. Any deviation above this (on an aggregate level) indicates excess volatility in the forecast evolution. We can observe that while none of the models produce ideal forecasts, both attention models outperform the Baseline with the attention model with both proposed attention mechanisms performing the best in terms of these evolution diagnostics.
The green line corresponds to the attention model with only the horizon- horizon-specific decoder-encoder attention. We can see that compared to the baseline, this model achieves up to 27% reduction in excess volatility at P50 and 7% P90. By also adding decoder-selt attention see further reduction in excess volatility of an additional 20% at P50 and 21% at P90.
'ee al. (2017). Figure shows MQ-CNN (labeled "MaveeN_ outperforms MQ-RNN (all variants) and DeepAR (labeled "Seq2SegC") on the test set.
7

---

Under review as a conference paper at ICLR 2021
Model
0.150
Horizon-Speclic Decoder-Encoder + Decoder-Sel Attertion
Horizon-Seeclic Recoder-Encodee Attention
QV Compensator
PSO
0.100
0.075
0.050
0.025
0.000
10
15
20
25
30
Time
Figure Martingale diagnostic process {V/ averaged over all weeks in test period (2018-2019)
model
gV- Compensstoo
0.05
MOCNN
0.001
TFT
MOTranstormer
Quanntie Loss
0.000
0.03
percentile
P90
-0.002
0.01
-0.004
10
12
14
16
2
10
Time
Lead Time
Figure 3: Forecast evolution analysis on the retail dataset. Left: Martingale Diagnostic Process [Vi). Right: QL by lead time, averaged over target dates from 2016-03-01 through 2016-05-01; QL trajectories are centered around O.
4.2 PUBLICLY AVAILABLE DATASETS
Following Lim et al. (2019), we consider applications to brick-and-mortar retail sales, electricity load, securities volatility and traffic forecasting. For the retail task, predict the next 30 days of sales, given the previous 90 days of history. This dataset contains a rich set of static, time series, and known features. At the other end of the spectrum, the electricity load dataset is univariate. See Appendix D for additional information about these tasks. Table 4 compares MQTransformer' s performance with other recent works De (Salinas et al., 2020), ConvTram (Li et al., 2019), MQ-RNN (Wen et al., 2017), and TFT (Lim et al., 2019).
Our MQTransformer architecture is competitive with or beats the state-of-the-art on the electricity load, volatility and traffic prediction tasks, as shown in Table On the most challenging task, dramatically outperforms the previously reported state of the art by 38% and the MQ-CNN baseline by 5% at P50 and 11 P90. Because MQ-CNN and MQTransformer are trained using forking sequences, we can use the entire training population, rather than downsample as is required to train TFT (Lim al., 2019)- see Appendix D. To ascertain what portion of the gain is due learning from more trajectories, versus our innovations alone, we retrain the optimal MQTransformer architecture using a random sub-sample of 450K trajectories (the same sampling procedure TFT) and without using forking sequences- - the results are indicated in parentheses in Table 4. We can observe that MQTransformer still dramatically outperforms TFT, and its performance is similar the MQ-CNN baseline trained on all trajectories.
Furthermore, the Favorita retail forecasting task, Figure 3 shows that expected. MQTransformer substantially reduces excess volatility in the forecast evolution compared to the MQ-CNN bascline. Somewhat surprisingly, TFT exhibits much lower volatility than does MQTransformer. In Figure 3, the right hand plot displays quantile loss as the target date approaches trajectories for each model are zero centered to emphasize the trends exhibited. While TFT is less volatile, it is also less
RRssults for TFT. MQ-RNN, DeepAR and ConvTrans from Lim al. (2019). For MQ-CNN and MQTransformer, their pre-processing and evaluation code ensure parity
8

---

Under review as a conference paper at ICLR 2021
Table 4: P50 (50th percentile) and P90 (90th percentile) QL on electricity and retail datasets with the best results on each task emphasized. For the retail task, MQTransformer has results in parentheses, which correspond training without forking sequences 450K trajectories only.
P50 QL
TASK
DEEPAR
CONVTRANS
MQ-RNN
MQ-CNN
TFT
MQTRANSFORMER
ELECTRICITY
0.075
0.059
0.077
0.076
0.055
0.057
RETAIL
0.574
0,429
0.379
0.269
0.354
0.256 (0.2645)
VOLATILITY
0.050
0.047
0.042
0.042
0.039
0.039
TRAFFIC
0.161
0.122
0.117
0.115
0.095
0.101
P90 QL
TASK
DEEPAR
CONVTRANS
MQ-RNN
MQ-CNN
TFT
MQTRANSFORMER
ELECTRICITY
0.040
0.034
0.036
0.035
0.027
0.027
RETAIL
0.230
0.192
0.152
0.118
0.147
0.106 (0. (0.109)
VOLATILITY
0.024
0.024
0.021
0.020
0.020
0.019
TRAFFIC 0.099
0.081
0.082
0.077
0.070
0.068
accurate as it fails to incorporate newly available information. By contrast, MQTransformer is both less volatile and more accurate when compared with MQ-CNN, See Appendix D for more details the experiment setup and training procedure used.
Computational Efficiency For purposes of illustration, consider the Retail dataset. Prior work (Lim et al., 2019) was only able to make use of 450K out 20M trajectories, and the optimal TFT architecture required 13 minutes per epoch (minimum validation error at epoch 6)* using single V100 GPU. By contrast, our innovations are compatible with forking sequences, and thus our architecture make of all available trajectories. To train, MQTransformer requires only minutes per epoch (on 20M trajectories) using a single V100 GPU (minimum validation error reached after 5 epochs). Of course, some of the differences in runtime can be attributed to use of different deep learning frameworks, but clear that MQTransformer (and other MQForecasters) can be trained much more efficiently than models like TFT and DeepAR.
5 CONCLUSIONS AND FUTURE WORK
In this work, we present three novel architecture enhancements that improve bottlenecks in state of the art MQ-Forecasters. We presented series of architectural innovations for probabilistic time-series forecasting including a novel alignment decoder-encoder attention, as well as decoder self-attention scheme tailored the problem of multi-horizon forecasting. To the best of our knowledge, this is the first work to consider the impact of model architecture on forecast evolution. We also demonstrated how position embeddings can be learned directly from domain-specific event indicators and horizon- specific contexts can improve performance peaks. Together, these innovations produced significant improvements in the excess variation of the forecast and accuracy across different dimensions. Finally, we applied our model to several public datasets, where it outperformed the baseline architecture by 5% at P50, 11% at P90 and the previous reported state-of- ofie-oofthhe of-the-art (TFT) by 38% on the most complex task. On the three less complex public datasets, our architecture achieved parity or slightly exceeded previous state of the art results. Beyond accuracy gains, by making our architecture innovations compatible with forking sequences, our model achieves massive increases in throughput compared to existing transformer architectures for time-series forecasting. An interesting direction we intend to explore future work is incorporating encoder self-altention so that the model can leverage arbitrarily long historical time series, rather than the fixed length consumed by the convolution encoder.
difficult sub-problems such as promotions or seasonal
"Timing results obtained by running the source code provided by Lim al. (2019)

---

Under review as a conference paper at ICLR 2021
REFERENCES
AUGENBLICK, N. and RABIN, M. (2019). Belief movement, uncertainty reduction, and rational updating. Tech. rep., Haas School of Business, University of California, Berkeley.
BAHDANAU, D., CHO, K. and BENGIO, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv: 1409. 0473.
BENIDIS, K., RANGAPURAM, S., FLUNKERT, V., WANG, B., MADDIX, D., TURKMEN, C., GASTHAUS, J., SOOILKEESCHNEIDDAR M., SALINAS, D., STELLA, L. ET AL. (2020). Neural forecasting: Introduction and literature overview.
BRAY, R. L. and MENDELSON, H. (2012). Information Transmission and the Bullwhip Effect: An Empirical Investigation, Management Science 58 860-875.
CHEN, F., DREZNER, Z., RYAN, J. K. and SIMCHI-LEVI, D. (2000). Quantifying the Bullwhip Effect in a Simple Supply Chain: The Impact of Forecasting, Lead Times, and Information. Management Science 46 436-443.
CHENG, J., DONG, L. and LAPATA, M. (2016). Long short-term memory-networks for machine reading. arXiv: 1601. 06733.
CINAR, Y. G., MIRISAEE, H., GOSWAMI, P., GAUSSIER, AIT-BACHIP and STRIOV, V. (2017). Position-based content attention for time series forecasting with sequence-to-seqquene mns. In ICONIP.
DAI, Z., YANG, Z., YANG. Y., CARBONELL, J., LE. Q. V. and SALAKHUTDINOV, R. (2019). T: Attentive Language Models Beyond Fixed-Length Context. In ACI..
DEVLIN, J., CHANNG, M.-W., LEE, K. and TOUTANOVA, K. (2019). BERT: Pre-training of Deep
Bidirectional Transformers for Language Understanding. In NAACL-HILT.
GALASSI, A., LIPPI, M. and TORRONI, (2019). Attention, please! critical review of neural attention models in natural language processing. a arXiv: 902. 0200011111
GASPARIN, A., LUKOVIC, S. and ALIPPI, C. (2019). Deep Learning for Time Series Forecasting: The Electric Load Case. arXiv: 1907. 09207.
HEATH, D. C. and JACKSON, P. L. (1994). Modeling the evolution of demand forecasts with application to safety stock analysis in productionn/dsstiiutiin systems. IIE transactions 26 17-30.
HUANG, C.-Z. A., VASWANI, A., USZKOREIT, J., SHAZEER, N., SIMON, I., HAWTHORNE, C.,
DAI, A. M., HOFFMAN, M. D., DINCULESCU, M. and ECK, D. (2018), Music Transformer: Generating Music with Long-Teri Structure. arXiv: 110::1000 04281.
KIM, and KANG, arXiv: 10877.
(2019). Financial series prediction using Attention LSTM.
KINGMA, D. P. and BA, J. (2015). Adam: A Method for Stochastic Optimization. In ICLR.
LI, S., JIN, X., XUAN, Y., ZHOU, X., CHEN, W., WANG, Y.-X. and YAN., X. (2019). Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In NIPS.
LIM, B., ARIK, S. O.. LOFFF, N. and PFISTER, T. (2019). Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting. arXiv: 1912. 09363.
LUONG, M.-T., PHAM, H. and MANNING, C. D. (2015). Effective approaches attention- based neural machine translation. arXiv: 1508. 04025.
MADEKA, D., SWINIARSKI, L., FOSTER, D.,. RAZOUMOV, L., TORKKOLA, K. and WEN, R. (2018). Sample path generation for probabilistic demand forecasting. In ICML workshop on Theoretical Foundations and Applications of Deep Generative Models.
MUKHOTY, B. P., MAURYA, V. and SHUKLA, S. K. (2019). Sequence to sequence deep learning models for solar irradiation forecasting. In IEEE Milan PowerTech.
10

---

Under review as a conference paper at ICLR 2021
NASCIMENTO, R. C., SOUTO, Y. M., OGASAWARA, E., PORTO, F. and BEZERRA, E. (2019). STConvS2S: Spatiotemporal Convolutional Sequence to Sequence Network for weather forecasting. arXiv:1912 00134.
SALINAS, D., FLUNKERT, V., GASTHIAUS, J. and JANUSCHIOWKKL T. (2020). Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting 36 1181-1191.
SHAW, P., USZKOREIT, J. and VASWANI, A. (2018). Self-Attention with Relative Position Repre- sentations. In NAACL.
SHUN- YAO SHIH AND FAN-KENG SUN AND HUNG-YL YI LEE (2019). Temporal pattern attention for multivariate time series forecasting. Machine Learning 108 1421-1441.
STINE, R. and FOSTER, D. (2020a). Martingale Descriptions of the Evolution of Forecasts. Tech. rep., Manuscript in preparation.
STINE, R. and FOSTER, D. (2020b). Martingale Diagnostics. Tech. rep., Manuscript preparation.
SUTSKEVER, I., VINYALS, O. and LE, Q. V. (2014). Sequence to sequence learning with neural networks. In NIPS.
TALEB, N. N. (2018). Election predictions martingales: arbitrage approach. Quantitative Finance 18 1-5.
TALEB, N. N. and MADEKA, D. (2019). All roads lead to quantitative finance. Quantitative Finance 19 1775-1776.
VAN DEN OORD, A., DIELEMAN, S., ZEN, H., SIMONYAN, K., VINYAL., O., GRAVES, A.,
KALCHBRENNER, N., SENIOR, A. and KAVUKCUOGLU, K. (2016). Wavenet: A generative model for raw audio. arXiv:1609. 03499.
VASWANI, A., SHAZEER, N., PARMAR, N., USZKOREIT, J., JONES, L., GOMEZ, A. N., KAISER, L., and POLOSUKHIN, I. (2017). Attention all you need. NIPS.
WEN, R. and TORKKOLA, K. (2019). Deep Generative Quantile-Copula Models for Probabilistic Forecasting. In ICML Time Series Workshop.
WEN, R., TORKKOLA, K., NARAYANASWAMY, B. and MADEKA, D. (2017). A multi-horizon quantile recurrent forecaster. In NIPS Time Series Workshop.
WILLIAMS, D. (1991). Probability with Martingales. Cambridge University Press.
XU, K., BA, J., KIROS, R., CIIO, K., COURVILLE, A., SALAKIUDINOY, R., ZEMEL, R. and BENGIO, Y. (2015). Show, attend and tell: Neural image caption generation with visual attention.
In ICML.
YU, R., ZHENG, S., ANANDKUMAR, A. and YUE, Y. (2017). Long-term Forecasting using Higher Order Tensor RNNs. arXiv: 1711. 00073.

---

Under review as a conference paper at ICLR 2021
A ADDITIONAL BACKGROUND AND RELATED WORK
A.1 ATTENTION MECHANISMS
be viewed form of content based addressing, that computes alignment between a set of queries and keys to extract a value. Formally, let qi..... qt, k1 ki..... .......,, and v1, Vi....: Ve be series of queries, keys and values, respectively. The 5t attended value is defined C2
Attention mechanisms
score 9s kg Vv. where score is scoring function commonly score(u, v) u v. In the vanilla transformer model, qa = kg Ve hg, where h2 is the hidden state at time s. Because attention mechanisms have concept of absolute relative position, some sort of position information must be provided. Vaswani al. (2017) uses a sinusoidal positional encoding added to the input to attention block. providing each token's 's position in the input time series.
B MOTRANSFORMER ARCHITECTURE DETAILS
In this section we describe in detail the layers in our MQTransformer architecture, which is based off of the MQ-Forecaster framework (Wen et al., 2017) and uses wavenet encoder (van den Oord et al., 2016) for time-series covariates. Before describing the layers in each component, Figure outlines the MQTransformer architecture. On different datasets, we consider the following variations: choice of encoding for categorical variables, a tunable parameter d, (dimension of hidden layers), dropout rate Parop- list of dilation rates for the wavenet encoder, and list of dilation rates for the position encoding. The ReLU activation function used throughout the network.
B. INPUT EMBEDDINGS AND POSITION ENCODING
Static categorical variables encoded using either one-hot encoding or an embedding layer. Time- series categorical variables are one-hot encoded, and then passed through a single feed-forward layer of dimension d,.
The global position encoding module takes as input the known time-series covariates, and consist of stack of dilated, bi-directional 1-D convolution layers with d2 filters. After each convolution is a ReLU activation, followed by a dropout layer with rate Parops and the local position encoding is implemented as single dense layer of dimension da.
B.2 ENCODER
After categorical encodings are applied, the inputs are passed through the encoder block. The encoder consists of two components: single dense layer to encode the static features, and a stack of dilated, temporal convolutions. The the time-series covariates are concatenated with the position encoding to form the input to the convolution stack. The output of the encoder block is produced by replicating encoded static features across all time steps and concatenating with the output the convolution.
B.3 DECODER
Please see Table for a description of the blocks in the decoder. The dimension of each head's in both horizon-specific and decoder self-attention blocks is [d,/8]] The dense layer used compute of has dimension dn. The output block is two layer MLP with hidden layer dimension d,, and weights are shared across all time points and horizons. The output layer has one output per horizon, quantile pair.
12

---

Under review as a conference paper at ICLR 2021
Decoder
Eneoder
F+11
hee
Decoder
y/
4
b
Hotinot- Hottiob-AAuuulle Agunstic MLP
MLP
P Podisional Recoding
Tepaa
(P)
Globai
Decoder Self- Attention
Leppal
C+2
Lp
Positional Eacoding
C++
Decoden-Eie Enooder
Horiaon Specifie
Figure 4: MQTransformer architecture with learned global/local positional encoding, Corizonnsspciiff decoder-encodee attention, and decoder self attf-attettion
13

---

Under review as a conference paper at ICLR 2021
C LARGE SCALE DEMAND FORECASTING EXPERIMENTS
C.I EXPERIMENT SETUP
In this section we describe the details the model architecture and training procedure used in the experiments the large-scale demand forecasting application.
TRAINING PROCEDURE
Because did not have enough history available set aside holdout set, all models trained for 100 epochs, and the final model is evaluated on the test set. For the same reason, no hyperparameter tuning was performed.
ARCHITECTURE AND HYPERPARAMETERS
The categorical variables consist of static features of the item, and the timeseries categorical variables are event indicators (e.g. holidays). The parameters are summarized in Table 5.
Table 5: Parameter settings for Large Scale Demand Forecasting Experiments
Parameter
Value
Encoder Convolution Dilation Rates
[1.2,4.8,16.32]
Position Encoding Dilation Rates
[1,2,4.8,16,20]
Static Categorical
One-Hot
Time-Series Categorical
One-Hot
Static Encoder Dimension
64
Convolution Filters
64
Attention Block Heads
52
Attention Block Head Dimension
8
Dropout Rate
0.15
Activation Function
ReLU
C.2 RESULTS
Tables 6, 7, and contain the full set of results on the large scale demand forecasting task. Per the discussion in Section 4, we use MQ-CNN as the baseline model and we did not compare to TFT due scaling issues and the fact that on the public task that was most similar (retail), MQ-CNN significantly outperformed TFT.
Quantile loss by horizon Table 6 demonstrates how the attention mechanism yields significant improvements in shorter LTSP (e.g. LTSP 3/1 and LTSP 0/4), 7% improvement in P90 QL for LTSP 3/1 and 7.6% improvement in P90 QL for LTSP 0/4, We still see improvements for longer LTSP, but they are less substantial: 3.8% improvement in P90 QL for LTSP 12/3 and 3.9% improvement in P90 QL for LTSP 0/33. By also adding decoder-self attention, we continue to see improved results for shorter LTSP compared only longer LTSP when comparing to decoder-encoder attention.
attention, but we do see slight degradations for
Promotions Performance Table we see that MQTransformer outperforms the prior state of the art on all promotion types. After adding the horizon-specific decoder-encoder and decoder-self attentions, versus the baseline, we see 49% improvement for Promotion Type products, a 31 % improvement for Promotion Type 2 products, and a 17% improvement for Promotion Type 3 products.
Peak Performance Table illustrates that while MQCNN performs well on some seasonal peaks, it also misaligned and fails to rampdown post-peak ramp-down issues occur when the model continues to forecast high for target weeks after the peak week. By including MQTransformer's attention mechanisms in the architecture,
43% improvement for Seasonal Peak 1, 21 %
14

---

Under review as a conference paper at ICLR 2021
improvement for Seasonal Peak 2, 7% improvement for Seasonal Peak 3, and 56% improvement on Post-Peak Rampdown.
Table 52-week aggregate quantile loss metrics with for set of representative lead times and spans
MODEL
ALL LTSP
LTSP 3/1
LTSP 0/4
P50
P90
P50
P90
P50
P90
BASELINE
1.000
1.000
1.000
1.000
1.000
1,000
DEC-ENC
0.984
0.960
0.950
0.927
0.963
0.931
DEC-ENC & DEC-SELF
0,989
0.984
0.934
0.911
0.948
0.908
MODEL
LTSP
12/3
LTSP
0/33
P50
P90
P90
BASELINE
1.000
1.000
1.000
1.000
DEC-ENC
0.975
0.957
0.982
0.963
DEC-EN & DEC-SELF
0.960
0.964
0.982
0.981
Table 7: P90 quantile loss metrics seasonal peak target weeks
SEASONAL PEAK
SEASONAL PEAK 2
SEASONAL PEAK 3
POST-PEAK
MODEL
RAMPDOWN
BASELINE
1,000
1.000
1.000
1.000
DEC-ENC
0.748
0.817
0.962
0.712
DEC-ENC DEC-SELF
0.698
0.826
0.931
0.639
Table 8: P90 quantile loss metrics item, weeks with promotions
PROMOTION TYPE
PROMOTION TYPE
PROMOTION
MODEL
TYPE
BASELINE
1.000
1.000
1,000
DEC-ENC
0.706
0.769
0.865
DEC-ENC DEC-SELF
0.670
0.763
0.851
15

---

Under review as a conference paper at ICLR 2021
D EXPERIMENTS ON PUBLIC DATASETS
In this section we describe the experiment setup used for the public datasets in Section 4,
D.I
DATASETS
We evaluate our MQTransformer on four public datasets. We summarize the datasets and preprocess- ing logic below; the reader is referred to Lim et al. (2019) for more details.
RETAIL
This dataset provided by the Favorita Corporacion major Grocery chain in Ecuador) part of Kaggle² to predict sales for thousands of items at multiple brick-and-mortar locations. In total there treated as distinct entities), and the dataset contains variety of features including: local, regional and national holidays; static features about each item; total sales volume at each location. The task is predict log-sales for each (item, store) combination over the next 30 days, using the previous 90 days of history. The training period is January 1, 2015 through December 2015. The following 30 days used as validation set, and the 30 days after that as the test set. These 30 day windows correspond to a single forecast creation time. While Lim et al. (2019) extract only 450K samples from the histories during the train window, there are in fact 20M trajectories avalaible for training - because our models can produce forecasts for multiple trajectories (FCDs) simultaneously, we train using all available data from the training window.
135K items (item, store combinations
For the volatility analysis presented Figure 3, we used 60 day validation window (March 1, 2016 through May 1, 2016), which corresponds to 30 forecast creation times.
ELECTRICITY
This dataset consists of time series for 370 customers of at an hourly grain. The univariate data is augmented with a day-of-week, hour-of-day and offset from a fixed time point. The task is to predict hourly load over the next 24 hours for each customer, given the past seven days of usage. From the training period (January 1, 2014 through September, 2019) 500K samples are extracted.
TRAFFIC
This dataset consists of lane occupancy information for 440 San Francisco area freeways. The data is aggregated to an hourly grain, and the task is to predict the hourly occupancy over the next 24 hours given the past seven days. The training period consist of all data before 2008-06-15 15, with the final 7 days used as a validation set. The days immediately following the training window is used for evaluation. The model takes input lane occupancy, hour of day, day of week, hours from start and an entity identifier.
VOLATILITY
The volatility dataset consists of minute sub-sampled realized volatility measurements from 2000- 01-03 to 2019-06-28. Using the past one year's worth daily measurements, the goal is to predict the next week's (5 business days) volatility. The period ending on 2015-12-31 is used as the training set, 2016-2017 as the validation set, and 2018-01-01 through 2019-06-28 as the evaluation set. The region identifier is provided as a static covariate, along with time-varying covariates daily returns, day-of-week, week-of-yeal and month. A log transformation applied to the target.
D.2 TRAINING PROCEDURE
We only consider tuning two hyper- paranettras,sss size hidden layer d, E {32, 64, 128) and learning rate a E {1 x 10 1 10 x 10 10-4). The model is trained using the ADAM optimizer Kingma and Ba (2015) with parameters 31 = 0.9, B2 = 0.999, = 1e - 8 and a minibatch size of 256, for a maximum of 100 epochs and an early stopping patience of epochs.
TThe original competition be found here.
16

---

Under review as a conference paper at ICLR 2021
We train model for each hyperparameter setting in the search grid (6 combinations), select the one with the minimal validation loss and report selected model's 's test-set error Table
D.3 ARCHITECTURE DETAILS
Our MQTransformer architecture used for these experiments contain single tune-able hyperparame- ter - hidden layer dimension d,. Dataset specific settings are used for dilation rates. For static categorical covariates we use embedding layer with dimension da and use one-hot encoding for time-series covariates. A dropout rate 0.15 and ReLU activations are used throughout the network. The only difference between this variant and the one used for the non-public large scale demand forecasting task the use of an embedding layer for static, categorical covariates rather than one-hot encoding.
D.4 REPORTED MODEL PARAMETERS
The optimal parameters for each task are given Table 9.
Table 9: Parameter settings of reported MQTransformer model on each public dataset.
Name
dp
@
Enc. Dilation Rates
Pos. Dilation Rates
Electricity
128
1 X 10-3
[1,2,4.8,16.32]
[1,2,4.8.8] [1,2,4,8.8]
Traffic
64
1 x 10-3
[1,2,4.8,16,32] [1,2,4
Volatility
128
1 1x 10-3
[1,1,2]
Retail
64
1 x 10-1
[1,2,4.8,16,22]
[1,2,4,8,44]
17


Reviews:
[{"review_id": "jDZKUIgJFKo", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After rebuttal:\nI appreciate authors' detailed responses and an updated version of the paper. The new version is a lot clearer. After reading other reviews, I agree that the algorithmic novelty is limited, but the model is well-adapted for multi-horizon forecasting problem. Overall, I increase my score to 6. marginally above acceptance threshold. \n\n----------------------------------------------------------\nSummary:\nThis paper introduces a new model for multi-horizon forecasting. The proposed model is an extension of MQRNN with two new modules: task specific attention and decoder self-attention. The experiments on the large-scale demand forecasting dataset and other publicly available datasets show that the proposed model outperforms or is comparable with the CNN, RNN-based models as well as the transformer-based model. \n--------------------------------------\nPros:\n+ Authors adapted the attention mechanism for a challenging problem (multi-horizon forecasting). \n+ The proposed model is evaluated on a large-scale dataset as well as existing public datasets. \n+ Additional experiments and Figure 4 on Appendix are helpful. \n--------------------------------------\nCons:\nI found this paper lacks clarity. I list the issues below:\n\n1. Authors stated that 'Our horizon-specific attention mechanism can be viewed as a multi-headed attention mechanism. Each head corresponds to a different horizon.'. I don't think this is true. Multi-head attention is an ensemble of attentions from the same input to attend different positions by incorporating different representations. The purpose of the horizon-specific attention in this paper is to merge the encodings of multiple horizon. I ask authors to clarify it.\n\n2. Design choice: the main contribution of this paper is attention mechanisms (horizon-specific attention between encoder-decoder and decoder self-attention) for multi-horizon forecasting. \nQuestions:\n    1. Since the proposed model architecture is similar to MQRNN [Wen et al. 2017], it should be the baseline. What is the reason MQCNN is used as a baseline? \n    2. The proposed model performs better than MQRNN for public datasets. Which module in the model has a big role between horizon specific attention and decoder self-attention? Or is attention specifically handling better for multi-horizon forecasting problems than RNNs? Adding MQRNN as another baseline and the result without horizon specific attention for the large-scale demand forecasting would be helpful (in Table 3 and Figure 2). \n\n3. TFT [Lim2019] is an existing transformer-based model for Multi-horizon Time Series Forecasting. In my understanding, the major differences are horizon specific attention in the proposed model and a different design of decoder in TFT. Could you clarify the difference and similarities between these two models in the paper?\nAlso, authors stated that 'We were unable to compare to TFT (the prior state of the art on several public datasets) as it does not scale-up'. What does this mean? I assume TFT cannot be easily applied for large-scale demand forecasting. This needs more explanations what are the major difficulties to scale-up TFT for the problem. \n\n3. A description of experiment setup and model specification is missing (for both the proposed model and baseline). The results can not be easily reproduced. \n    - Hyper-parameters: the number of layers and hidden units, learning rate, optimizer, etc. \n    - Preprocessing if there is any.\n--------------------------------------\nMinor comments:\n- A description of P50 and P90 in Figure 2, 3, and Table 4 is missing. \n- A description of LTSP in Section 4 is missing. \n- The baseline (MQCNN) has no reference. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "h9EezZWBsTa", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "(Note: I am not well-versed in the forecast modeling literature but I am reasonably so in the use of Transformer models in NLP tasks)\n\nThe paper proposes MQTransformer, an improvement on MQRNN (Wen 2017) for multi-horizon forecast prediction that leverages the Transformer architecture. Their contributions are:\n1) using learnable positionable embedding from event indicators\n2) using an attention head for each of the k horizons that need to be forecasted\n3) applying decoder self-attention so that the model can use one horizon prediction to improve a later one\n\nThe technical / modeling contributions aren't very novel or profound, so the ICLR community at large may not find them that inspiring. However, as backed by the experiments, these contributions do translate into non-trivial improvements on real-world datasets, so I expect the practical impact on real-world forecasting tasks to be high. I marginally support accepting this paper for this reason.\n\nPros:\n* Well written overall\n* Non-negligible improvement over baseline methods on \"RETAIL\" dataset\n* Likely to be used in practice\n\nCons:\n* Datasets / tasks are limited. Paper could benefit from a few more.\n* Modeling novelty is limited\n\nSuggestions for improvement:\n* In equation 2, it's more clear to write as (x_s + r_s)^T W_q^h W_k^h (x_t + r_t), no?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "RRCxNpya2qY", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\nThis paper aims at improving accuracy of multi horizon univariate time series forecasting. The authors propose an encoder-decoder attention-based architecture for multi-horizon quantile forecasting. The model encodes a distinct representation of the past for each requested horizon.\n\n#### Strong points\n\n+ The encoding of the holidays and special event specific to the time series is elegant.\n+ The idea of explicitly using forecast error feedback is interesting.\n+ Ablation study of the architecture's innovations on a large scale forecasting dataset mainly outlines the importance of the horizon specific component of the architecture.\n\n#### Weak points\n\n- The evaluation of the new method is conducted on only two public datasets and the new methods outperform TFT only on one. There exists other common datasets (DeepAR evaluates on three-parts and traffic, TFT evaluates on traffic and volatility) that should be considered to place this method in the literature. \n- Both manuscripts of Fine and Foster (2020 al b) are not published and I could not find them online. The summary of Fine and Foster (2020 a; b) in Section 2.3 is not enough for me to judge the relevance of the results in Fig 2 and 3.\n- The strong claims at the end of the introduction are made compared to the ablated model (MQCNN) on the private dataset, not against alternative methods such as MQRNN or DeepAR which could scale to this dataset size.\n\n\n#### Decision\n\nI would tend to reject this submission. The proposed model exhibits none of: significant quantitative improvement (on public dataset), speed up, improved simplicity over alternative methods.\n\nAdditionally to the weak points mentioned above, the contribution of this paper is undermined by the following points: \n- The contribution of “Positional Encoding from Event Indicators” is rather incremental considering BERT encoding of input segments (e.g. Sentence A, Sentence B, Question, Answers).\n- The Horizon-Specific encoding is interesting but does not allow to forecast at inference a horizon that has not been trained on (as parametric method such as DeepAR have).\n- The code of the submission is not submitted and there is no mention of future release.\n\n#### Questions\n\n- Is your positional encoding method a superset of relative positional encoding?\n- Can you provide runtimes of the MQTransformers? How does it compare to (Lim et al. 2019) ?\n- In appendix B: What is the percentage of unseen object in the test/valid dataset that would be harder to forecast by TFT?\n- Why do you use the large scale private dataset for the ablation of your method and not a public dataset? Are the architectural innovations only useful in the high data regime?\n\n#### Additional feedback\n\n- Figure 4 in appendix D should be mentioned in the main text as it is helpful for the reader.\n- Typo in Table 1, eq (4), third line: $c_{t,1}^{a}$ should be $c_{t,1}^{hs}$ I believe.\n- Section 3.3: what do you call a *bidirectional* 1-D convolution? (As opposed to unidirectional?)\n- Caption of Figure 3: Try to be consistent on the short name of retail vs Favorita dataset.\n- Eq (2): Consider providing the factorised form which is easier to parse.\n\n\n--- \n**Update:** I would like to thank the authors for their answer. I acknowledge the improvement of the manuscript after the review process:\n\n- Added clarifications on the baselines,\n- Added helpful precisions for reproducibility (even though the code cannot be open sourced),\n- Evaluation on 2 requested public datasets with good results.\n\nHowever, I am still not confident to raise my score to 6 (marginally above the acceptance threshold) given the missing public manuscripts (or Appendix) to explain the martingale diagnostic tools. This hinder one of the main selling point of the paper that their model reduces the volatility of the forecast.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes and uses a fairly involved attention based architecture to perform time series forecasting. The idea of transformers is raised, but, given how sequence embedding is often convolutional, and position encoding input is provided to the model (albeit implictly in the form of features having to do with qualitative things such as promotions, etc among other things), I'm  of the opinion it is closer to the paper "Convolutional Sequence to Sequence Learning" https://arxiv.org/abs/1705.03122 than it is to transformers per se... Also the connection to sequence to sequence is not clear, since the chain rule of probability isn't really stressed on much. 

The paper proposes some interesting ideas, but I feel that it failed to convince the reviewers of the utility and the novelty. Part of it has to do with the clarity of presentation, and part of it, I think has to do with the fact that paper jam packs a bunch of different ideas together, without carefully ablating the influence of their various ideas. For example, transfomers have been applied for time sequences (https://arxiv.org/pdf/2001.08317.pdf), and its not clear in what ways this paper improves on them -- is it the complicated attention model scheme ? or is it the multi-horizon context schemes ?  

That being said, the results shows are relatively decent and the reviewers liked that aspect. Had the paper been easier to follow and the ideas presented with a little more insight it would be been a better fit for ICLR. As it stands, I have to give it a weak reject.
Decision: Reject