MQTransformer: Multi-Horizon Forecasts with Context Dependent and Feedback-Aware Attention
Authors: Carson Eisenach
Yagna Patel
Dhruv Madeka
Venue: ICLR 2022 Submitted (2022)

Abstract: Recent advances in neural forecasting have produced major improvements in accuracy for probabilistic demand prediction. In this work, we propose novel improvements to the current state of the art by incorporating changes inspired by recent advances in Transformer architectures for Natural Language Processing. We develop a novel decoder-encoder attention for context-alignment, improving forecasting accuracy by allowing the network to study its own history based on the context for which it is producing a forecast. We also present a novel positional encoding that allows the neural network to learn context-dependent seasonality functions as well as arbitrary holiday distances. Finally we show that the current state of the art MQ-Forecaster (Wen et al., 2017) models display excess variability by failing to leverage previous errors in the forecast to improve accuracy. We propose a novel decoder-self attention scheme for forecasting that produces significant improvements in the excess variation of the forecast.

Under review as a conference paper at ICLR 2022
MQTRANSFORMER: MULTI-HORIZON FORECASTS WITH CONTEXT DEPENDENT AND FEEDBACK-AWARY ATTENTION
Anonymous authors Paper under double-blind review
ABSTRACT
Recent advances in neural forecasting have produced major improvements accuracy for probabilistic prediction. In this work, we propose novel improvements to the current state of the art by incorporating changes inspired by recent advances in Transformer architectures for Natural Language Processing. We develop novel decoder-encoder attention for context-alins aalieett-llgmment improving forecasting accuracy by allowing the network to study its history based the context for which it is producing a forecast. We also present a novel positional encoding that allows the neural network learn context-dependent seasonality functions as well arbitrary holiday distances. Finally, we show that the current state of the art MQ- Forecaster (Wen al., 2017) models display excess variability by failing leverage previous errors in the forecast to improve accuracy. We propose novel decoder- self attention scheme for forecasting that produces significant improvements in the excess variation of the forecast.
INTRODUCTION
Time series forecasting is a fundamental problem in machine learning with relevance to many application domains including supply chain management, finance, healthcare analytics, and more. Modern forecasting applications require predictions of many correlated time series over multiple horizons. In multi-horizon forecasting, the learning objective is to produce forecasts for multiple future horizons at each time-step. Beyond simple point estimation, decision making problems require measure of uncertainty about the forecasted quantity. Access to the full distribution is usually unnecessary, and several quantiles are sufficient (many problems in Operations Research use the 50th and 90th percentiles, for example).
As motivating example, consider a large e-commerce retailer with a system to produce forecasts of the demand distribution for a of products at target time T. Using these forecasts as input, the retailer can then optimize buying and placement decisions to maximize revenue and/or customer value. Accurate forecasts are important, but perhaps less obviously - forecasts that don't 't exhibit excess volatility as a target date approaches minimize costly, bull-whip effects in a supply chain (Chen et al., 2000; Bray and Mendelson, 2012).
Recent work applying deep learning to time-series forecasting focuses primarily on the use of recurrent and convolutional architectures (Nascimento et al., 2019; Yu et al., 2017; Gasparin et al., 2019; Mukhoty et al., 2019; Wen et al., 2017)!. These are Seq2Seq architectures (Sutskever et al., 2014) - which consist of an encoder that summarizes an input sequence into a fixed-length context vector, and a decoder which produces an output sequence. This line of work has lead to major advances in forecast accuracy for complex problems, and real-world forecasting systems increasingly rely on neural nets. Accordingly, need for black-box forecasting system diagnosties has arisen. Foster and Stine (2021) use probabilistic martingales to study the dynamics of forecasts produced by an arbitrary forecasting system. They can be used to detect the degree which forecasts adhere to the martingale model of forecast evolution (Heath and Jackson, 1994) and to detect unnecessary volatility (above and beyond any inherent uncertainty) in the forecasts produced that is to say, Foster
For complete overview see Benidis al. (2020)

---

Under review as a conference paper at ICLR 2022
and Stine (2021) describe way connect the excess variation of forecast to accuracy misses against the realized target. While tools such as martingale diagnostics can be used to detect flaws in forecasts, the question of how to incorporate that information into model design is unexplored. Existing multi-horizon forecasting architectures (Wen et al., 2017; Madeka et al., 2018) are designed to minimize quantile loss, but they do not explicitly handle excess variation, since forecasts any particular date are not made aware of errors in the forecast for previous dates.
Another limitation in many existing architectures is the well known information bottleneck, where then encoder transmits information to the decoder via a single hidden state. To address this, Bahdanau al. (2014) introduce a method called attention, allowing the decoder take as input a weighted combination of relevant latent encoder states at each output time step. rather than using a single context to produce all decoder outputs. While NL.P is the predominate application of attention architectures to date, in this paper we use novel attention modules and positional embeddings to introduce the proper inductive biases for probabilistic time-series forecasting to the model architecture.
Summary of Contributions In this paper, we are concerned with both improving forecast accuracy and reducing excess forecast volatility. We present set of novel architectures that seek to remedy some of inductive biases that are currently missing in state of the art MQ-Forecasters (Wen et al., 2017). Our main contributions are
1. Positional Encoding from Event Indicators: Current NO--oocccssters use explicitly engineered holiday "distances" to provide the model with information about the seasonality of the time series. We introduce novel positional encoding mechanism that allows the network learn context-dependent seasonality function specific that is more general than conventional position encoding schemes.
Horizon-Specifiee Decoder-Eneodes Attention: Wen et al. (2017); Madeka et al. (2018) and other MQ-Forecasters learn single encoder representation for all future dates and periods being forecasted. We present a novel horizon-speciff decoder-encoder attention scheme that allows the network to learn a representation of the past that depends on the period being forecasted.
3. Decoder Celf-Attention for Forecast Evolution: To the best of our knowledge, this is the first work to consider the impacts of network architecture design on forecast evolution. Importantly, accomplish this by using attention mechanisms introduce the right inductive biases, and not by explicitly penalizing a measure of forecast variability. This allows us to maintain a single objective function without needing make trade-off's between accuracy and volatility.
By providing MQ-Forecasters with the structure necessary to learn context information dependent encodings, observe major increases in accuracy (5.5% in overall P90 quantile loss throughout the year. and up to 33 % during peak periods) on our demand forecasting application, along with a statistically significant reduction in excess volatility (67 reduction at P50 and 95% at P90). We also apply MQTransformer to four public datasets, and show parity with the of-te-ofthh-arrr on simple, univariate tasks. On a substantially more complex public dataset (retail forecasting) we demonstrate a 38% % improvement over the previously reported state-of-the- 1 11% in P90 QL versus our baseline. Because our innovations are compatible with efficient training schemes, our architecture also achieves significant speedup (several orders of magnitude greater throughput) over earlier transformer models for time-series forecasting.
and a 5% improvement in P50 QL,
BACKGROUND AND RELATED WORK
2.1 TIME SERIES FORECASTING
Formally, the task considered in our work is the high-dimensional regression problem
H. p(Uet1.......
(h) X
(1)
where Ve Ya.. x x K x, denote future observations of the target time series i, observations of the target time series observed up until time t, the past covariates, known future information, and static covariates, respectively.
For sequence modeling problems, Seq2Seq (Sutskever et al., 2014) is the canonical deep learning framework and although they applied this architecture to neural machine translation (NMT) tasks, has since been adapted to time series forecasting (Nascimento et al., 2019; Yu et al., 2017; Gasparin
2

---

Under review as a conference paper at ICLR 2022
et al., 2019; Mukhoty et al., 2019; Wen al., 2017; Salinas et al., 2020; Wen and Torkkola, 2019). The MQ-Foue (Q-Forecaster framework (Wen et al., 2017) solves (1) above by treating each series as a sample from a joint stochastic process and feeding into neural network which predicts Q quantiles for each horizon. These types of models have limited contextual information available to the decoder drawback inherited from the Seq2Seq architecture
it produces each estimate vi+,..> the 4th quantile of the distribution of the target at time i+ s, a Seq2Seq models rely on a single encoded context produce forecasts for all horizons, imposing an information bottleneck and making difficult for the model to understand long term dependencies.
Our MQTransformer architecture uses the direct strategy: the model outputs the quantiles of interest directly, rather than the parameters of a distribution from which samples are to be generated. This has been shown (Wen et al., 2017) to outperform parametric models, like DeepAR (Salinas et al., 2020), on wide variety of tasks. Recently, Lim et al. (2019) consider an application of attention to multi-horizon forecasting, but their method still produces a single context for all horizons and by using an RNN decoder, does not enjoy the same scaling properties as MQ-Forecaster models.
2.2 ATTENTION MECHANISMS
Bahdanau et al. (2014) introduced the concept of an attention mechanism to solve the information bottleneck and sequence alignment problems in Seq2Seq architectures for NMT. Recently, attention has enjoyed success across diverse range of applications including natural language processing (NLP), computer vision (CV) and time-series forecasting tasks (Galassi et al., 2019; Xu et al., 2015;
Yhun-Yao Shih and Pan-Keng Sun and Hung-yi Lee, 2019; Kim and Kang, 2019; Cinar et al., 2017; Li et al., 2019; Lim et al., 2019). Many variants have been proposed including self-attention and dot-product attention (Luong al., 2015; Cheng et al., 2016; Vaswani al., 2017: Devlin et al., 2019), and transformer architectures (end-to-end attention with no recurrent layers) achieve state-of-the-art performance on most NLP tasks.
Time series forecasting applications exhibit seasonal trends and the absolute position encodings commonly used in the literature cannot be applied. Our work differs from previous work on relative position encodings (Dai et al.. 2019; Huang et al., 2018; Shaw et al., 2018) in that we learn representation from a time series of indicator variables which encode events relevant to the target application (such as holidays and promotions). This imposes strong inductive bias that allows the model to generalize well to future observations. Existing encoding schemes either involve feature engineering (e.g. sinusoidal encodings) or have maximum input sequence length; ours requires no feature engineering the model learns it directly from raw data- and it extends to arbitrarily long sequences. See Appendix A for additional background on attention mechanisms.
2.3 MARTINGALE DIAGNOSTICS
Originally the martingale model of forecast evolution (MMFE) was conceived way to simulate demand forecasts used in inventory planning problems (Heath and Jackson, 1994). Denoting by Yrie the forecast for Yr made at time / < T, the MMFE assumes that the forecast process { & is martingale. Informally, martingale captures the notion that forecast should use all information available to the forecasting system at time t. Mathematically, discrete time martingale is stochastic process } such that
[.X.41Xx,.. X1] X.-
We assume working knowledge of martingales and direct the reader to Williams (1991) for thorough coverage in discrete time.
Taleb (2018) describe how martingale forecasts correspond to rational updating, then expanded by Augenblick and Rabin (2019). Taleb (2018), Taleb and Madeka (2019) and Augenblick and Rabin (2019) go on develop tests for forecasts that rule martingality and indicate irrational or predictable updating for binary bets. Foster and Stine (2021) ) further extend these ideas to quantile forecasts. They consider the coverage probability process Pt := PP|rr S T|Y., 3. s S t] = E[I I T))Y,, s < t). where t denotes the forecast announced in the first period t = 0. Because {pr} is )r
also martingale, the authors show that E((pr - ΣT E[(p. P==1)²1 = a T). where т Po is the expected value of pr, a Bernoulli random variable, across realizations of the coverage process. In the context of quantile forecasting, π is simply the quantile forecasted.
3

---

Under review as a conference paper at ICLR 2022
The measure of excess volatility proposed is the quadratic variation process associated with {pr J. Q. := (p= - Pt-1)². While this process is not a martingale, we do know that under the MMFE assumption, [[Qr] = ( 7). The martingale Vi := Q: - (p. - ))² is obtained from this process in the usual way by subtracting the compensator. In Section we leverage the properties of [Vi] to compare the dynamics of forecasts produced by a variety of models, demonstrating that our feedback-aware decoder self-attention units reduce excess forecast volatility.
Notation We denote by H and Q the number of horizons and quantiles being forecast, respectively. Bolded characters are used to indicate vector and matrix values. The concatenation of two vectors and u is denoted as [u; v].
3 METHODOLOGY
In this section we present our MQTransformer architecture, building upon the MQ-Forecaster framework (Wen et al., 2017). We extend the MQ-Forecaster family of models because it, unlike many other architectures considered in the literature, can be applied at a large-scale (millions of samples) due to its use of forking sequences a technique to dramatically increase throughput during training and avoid expensive data augmentation.
For ease of exposition, reformulate the generic probabilistic forecasting problem in (1)
xxxx'))) x()
NUC+L.........HHH.... X,,,,
where X,,,, past observations of all covariates, x (1)
11) 100 are known covariates specific fxx 2,2
to time-series i, x(9) { ja the global, known covariates. In this setting, known signifies that the model has access to (potentially noisy) observations of past and future values. Note that this formulation equivalent (1), and that known covariates can be included in the past covariates x,-- When it can be inferred from context, the time series index is omitted.
Learning Objective We train quantile regression model to minimize the quantile loss, summed over all forecast creation times (FCTs) and quantiles
ΣΣ>
Σ Li (JetR. Ditkk
where (y (7 - 4)+, (+)+ is the positive part operator, q denotes quantile, and l. denotes the horizon.
3.1 NETWORK ARCHITECTURE
The design of the architecture is similar to MQ-(C) (Wen et al., 2017), and consists of encoder, decoder and position encoding blocks (see Figure Appendix B). The position encoding outputs, for each time step f, are representation of global position information, (g) -PE (x as well as time-series specific context information, ,0()) PE. ()). Intuitively, r(()) captures position information that is independent of the time-series i (such holidays), whereas r encodes time- series specific context information (such as promotions). In both cases, the inputs are a time series of indicator variables and require no Ceature-enggnnering handcrafted functions.
The encoder then summarizes past observations the covariates into a sequence of hidden states h := encoder(y: :t, Xx,, (g) ,rr,,,,, s). Using these representations, the decoder produces an H x Q matrix of forecasts Y,- = decoder(h... r(9), r()).. Note that in the decoder, the model has access to position encodings.
In this section focus our novel attention blocks and position encodings; the reader is directed to Appendix B for additional architecture details.
MQTransformer Following the generic pattern given above, we present the MQTranformer archi- tecture. First, define the combined position encoding
= (r] In the encoder use stack of dilated temporal convolutions (van den Oord et al., 2016; Wen et al., 2017) to encode historical time-series and multi-layer perceptron to encode the static features (2).
4

---

Under review as a conference paper at ICLR 2022
Table MQTransformer encoder and decoder
ENCODER
DECODER CONTEXTS
h: TEMPORALCONV((.......
(2)
Ci,4 (SAATENNTON(L,..2
(3)
h FEEDFORWARD(S)
C/ FEEDFORWARD(h,,))
h: = [bh;; hi),
[c..1:-.. ;c,, of)
C, DSATTENTION(C:; h.,,
Our decoder incorporates our horizon specific and decoder self-attention blocks, and consists of two branches. The first (global) branch summarizes the encoded representations into horizon-specific (Ct,A) and horizon agnostic (cf) contexts. Formally, the global branch c₁ := mm(-) is given by (3).
The output branch consists of self-attention block followed by local MLP, which produces outputs using the same weights for each horizon. For FCT and horizon h, the output is given by (o Ve++k) = mr Cr,h: Ct,h, Fr+h). Next we describe the specifics of our position encoding and attention blocks.
3.2 LEARNING POSITION AND CONTEXT REPRESENTATIONS FROM EVENT INDICATORS
Prior work typically uses variant on one of two approaches provide attention blocks with position information: (1) handcrafted representation (such as sinusoidal encodings) or (2) matrix M E RLxd of position encoding where is the maximum sequence length and each row corresponds to the position encoding for time point.
In contrast, our novel encoding scheme maps sequences indicator variables to a d-dimensional representations. For demand forecasting, this enables our model to learn an arbitrary function of events (like holidays and promotions) to encode position information. As noted above, our model
includes two position encodings: v.0) P PE(") (x(()) and r, PE," (1) (x(*),, one that is shared among all time-series i and one that is specific. For the design we use in Section 4, PE is implemented as a bidirectional (looking both forward and backward time) 1-D convolution and Pg is an MLP applied separately at each time step. For reference, MQ-(C)RNN (Wen al., 2017) uses linear holiday and promotion distances to represent position information.
By using 1-D convolutions and MLPs over time-series of indicator variables, our approach is more flexible and can be used to learn a position representation that is context specific. In the demand forecasting application, consider two products during holiday season where one has promotion and the other does not. These two products need different position encodings, else the decoder-encoder attention (Section 3.3) will not able to align target horizons with past contexts. addition, note that the classical method of learning matrix embedding M can be recovered approach. Consider sequence of length L, and take x(9) := [ci...., ez), where c, is used to denote the vector R with 1 in the 4th position and Os elsewhere. To recover the matrix embedding scheme, we define PEmatrixx (x())) == ==x(0); (g), M.
special case of our
3.3 CONTEXT DEPENDENT AND FEEDBACK AWARE ATTENTION
HORIZOO-SSECIFIC EECODER-ENCOOEE ATTENTION
To motivate the porizon-specific attention unit, consider retail demand forecasting task. At each FCT T, the forecaster produces forecasts for multiple target horizons, which may contain different events such promotions or holidays. Prior work (Wen et al., 2017) incorporated horizon-specific contexts with the functional form Cr = [h,, Tt+h) for each FCT, target horizon. Because events like promotions holidays which are strong predictors of observed demand are fairly sparse, this functional form is insufficient. Instead, a function of the form C₁ = f(h T hes rey reth) allows the model to information from many past periods, which is valuable due the sparsity of relevant events. Using an attention mechanism to align target horizons enables this. Figure depicts the difference between prior work and the horion-specific attention in MQTransformer.
5

---

Under review as a conference paper at ICLR 2022
Table 2: Attention weight and output computations for blocks introduced in Section 3.3
BLOCK
ATTENTION WEIGHTS
OUTPLT
RECODER-ENCODER
A & W Weke
(4)
CLN = Σ
A..Woo.
(5)
ATTENTION
qi DergTenN
k. h.;
he
DECODER SELF-ATTENTIN
a T W Waker (6)
Ō = E
A Wover: (7)
[hq TLRTTENNN
(a.))ER(t.N)
k. (C,ri (aiTtr)
H(t,h) ((,,r)a
Var Cs.r
Decoder (FCT T)
Decoder (FCT T)
Vr+1
YT+2
Fr+a
Yr+1
Vr+2
Fr+
Promotion
Promotion
Encoder
Encoder
hT
hr-3
hr-
hor
br
hor-4
hr-a
hr
(a) MQTransformer decoder FCT T.
(b) MQCNN decoder at FCT T
Figure 1: Example of a demand forecasting task where periods T 3 and T + both have promotions. The encoded context hr-3, the last time the item had promotion, contains useful information for forecasting for target periods that also have a promotion. The horizon-specific attention aligns past encoded contexts with the target horizon.
To do this, we introduce the horizon-specific attention mechanism, which can be viewed is multi- headed attention mechanism where the projection weights are shared across all horizons. Each head corresponds to different horizon. It differs from traditional multi-headed attention mechanism that its purpose is to attend over representations of past time points to produce a representation specific the target period. In architecture, the inputs to the block the encoder hidden states and position encodings. Mathematically, for time s and horizon h, the attention weight for the value time is computed as (4).
Observe that there are two key differences between these attention scores and those in the vanilla transformer architecture: (a) projection weights are shared by all H heads, (b) the addition of the position encoding of the target horizon h to the query. The output of our horizon specific decoder- encoder attention block, Cr is obtained by taking weighted sum of the encoder hidden contexts, up to maximum look-back of L periods as in (5).
DECODER AETENNITEN
The martingale diagnostic tools developed in (Foster and Stine, 2021) indicate deep connection between accuracy and volatility. We leverage this connection to develop a novel decoder self-attention scheme for multi-horizon forecasting. To motivate the development, consider model which forecasts values of 40, 60 when the demand has constantly been units. We would consider this model to have excess volatility. Similarly, model forecasting 40, 60 when demand jumps between 40 and 60 units would not be considered to have excess volatility. This is because the first model fails to learn from its past forecasts it continues jumping between 40, 60 when the demand is 50 units.
In order to ameliorate this, we need to pass information of the previous forecast errors into the current forecast. For each FCT and horizon h, the model attends on the previous forecasts using query containing the demand information for that period. The attention mechanism has a separate head for each horizon.

---

Under review as a conference paper at ICLR 2022
Past forecasts
Actual Forecast
V++1
Value
8443
Peta
Ve+3
9442
P44
Vi42
Time
8141
Pet1
Y++1
(b) Forecasts fail to adjust as new information is observed
Decoder t- 1 2
Decoder t=1--
Decoder
t= T
Value
Feedback
2rr-2
hr-1
hT
Time
Encoder
(c) Forecast fluctuating even though observed demand is constant
(a) Decoder self-attention
Figure 2: The decoder attends over past forecasts for the same target horizon - the context he contains feedback information (demand and other encoded signals), allowing the model to adjust forecasts that are either too volatile or not volatile enough as the target date approaches.
Rather than attend on the demand information and prior outputs directly, a richer representation of the same information is used: the demand information time is incorporated via the encoded context h, and previous forecasts are represented via the corresponding horizon-specife context Ca,r in absence of decoder-self attention Ca,r would be passed through the local MLP to generate the forecasts. Formally, the attention scores are given by (6). The horizon-specific and feedback-aware outputs, Cr are given by (7). Note how we sum only over previous forecasts of the same period. In Section we demonstrate the importance of this inductive bias by ablating against more traditional decoder-self attention unit that attends over all past forecasts.
4 EMPIRICAL RESULTS
4.1 LARGE-SCALE DEMAND FORECASTING
First, we evaluate architecture demand forecasting problem for large-scale e-commerce retailer with the objective of producing multi-horizon forecasts for the next 52 weeks. We conduct experiments subset of products (~ million products) in the US store. Each model is trained using a single machine with 8 NVIDIA V100 Tensor Core GPUs, on three years of demand data (2015-2018) 8); year (2018-2019) is held out for back-testing.
To assess the effects of each innovation, we ablate by removing components one at a time. The architectures we compare are the baseline (MQ-CNN), MQTransformer (MQT), MQTransformer without decoder self-attention (MQT-NoDS), and MQ-TMame To-TTransformer with decoder self-attention unit that attends over all past forecasts (MQT-AII). MQ-CNN is selected the baseline since prior work? demonstrate that MQ-CNN outperforms MQ-RNY and DeepAR on this dataset, and as can be seen Table 4, MQ-CNN similarly outperforms MQ-RNN and DeepAR on the public retail forecasting dataset. For additional details see Appendix C. (Table 5 describes the decoder-self attention unit used in MQT-AIl).
Forecast Accuracy Table 3 summarizes several key metrics that demonstrate the accuracy improve- ments achieved by adding our proposed attention mechanisms to the MQ-CNN architecture. We consider overall quantile loss, well quantile loss for specific target periods (seasonal peaks, promotions). We also measure post-peak ramp-down performance - models that suffer issues with aligmment will continue to forecast high for target weeks after seasonal peak. By including MQ-
Wen et al. (2017), Figure 3 shows MQ-CNN (labeled "MQ_CNN_wave"s outperforms MQ-RNN (all variants) and DeepAR (labeled "Seq2SeqC") the test set.

---

Under review as a conference paper at ICLR 2022
Table 3: P50 (50th percentile) and P90 (90th percentile) quantile loss on the backtest year along different dimensions. Values indicate relative performance versus the baseline.
MQ-CNN
MQT
MQT-NoDS
MQT-ALL
DIMENSION
P50
P90
P50
P90
P50
P90
P50
P90
OVERALL
1.000
1.000
0.977
0.945
0.983
0.963
0.977
0.957
SEASONAL PEAK
1.000
1.000
0.865
0.667
0.881
0.688
0.908
0.729
SEASONAL PEAK
1.000
1.000
0.957
0.884
0.984
0.946
0.953
0.931
SEASONAL PEAK 3
1.000
1.000
0.838
0.845
0.837
0.892
0.851
0.870
POST-PEAK RAMPDOWN
1.000
1.000
0.987
0.978
0.985
0.997
0.990
0.991
PROMOTION TYPE
1.000
1.000
0.945
0,800
0.969
0.824
0.953
0.842
PROMOTION TYPE
1.000
1.000
0.868
0.751
0.895
0.762
0.914
0.808
PROMOTION TYPE
1.000
1.000
0.949
0.837
1.075
1.004
0.960
0.911
0.20
Model
MQ-CNN
0.150
MG-GNN
QV Compensator
0.15
Compensator
MOT
MOT-AII
MOT-AII
MOT-NoDS
0.100
MOT-NoDS
0.10
0.075
0.050
0.025
0.00
10
15 Time
25
30
Tim
Figure 3: Martingale diagnostic process {V:) for P50 (left) and P90 (right) forecasts. Trajectories are femand-wwightee and the results are averaged over all items, target weeks in the test period (2018-2019) Closer to zero is better.
Transformer's s attention mechanisms in the architecture, we see up to 33% improvements for seasonal peaks and 24.9% improvements on promotions versus MQ-CNN. Further, the ablation analysis (against MQT-NoDS and MQT-AII) shows that both novel attention units are important to improving accuracy, and that our decoder-self attention scheme (where we attend only over past forecasts for the same target period) introduces powerful inductive bias. Table 8 in Appendix C shows the metrics computed on only short horizons, where we see even greater gains in accuracy.
Forecast Volatility We study the effect of our proposed attention mechanisms on excess forecast volatility using diagnostic tools recently proposed by Foster and Stine (2021). Coverage probabilities computed by fitting gamma distribution to the two quantiles produced for each FCT, target horizon. Figure 3 shows {V.} (see Section 2) for the 4 models we consider, along with bootstrapped 95% confidence intervals. The results weighted by demand in the context of bull-whip effects in supply chain, excess volatility will have the largest impact for items with high demand but similar results are obtained if we weight all items equally. Under the MMFE, the lines should appear horizontal and any deviation above this (on an aggregate level) indicates excess volatility in the forecast evolution. While none of the models produce ideal forecasts, MQT has the least amount of excess volatility it shows statistically significant reductions over all other models and achieves 67% reduction over baseline at P50 and 95% at P90.
4.2 PUBLICLY AVAILABLE DATASETS
Following Lim al. (2019), we consider applications to brick-and-morte retail sales, electricity load, securities volatility and traffic forecasting. In each task, we report the quantile loss summed over all forecast creation times, normalized by the target values:
2 3.. Lg Vl+k: Wetk
E,
ΣE Bett
8

---

Under review as a conference paper at ICLR 2022
Table Quantile loss metrics with the best results on each task emphasized. Results in parentheses correspond to training MQTransformer without forking sequences on 450K trajectories only.
TASK
DEEPAR
CONVTRANS
MQ-RNN
MQ-CNN
TFT
MQTRANSFORMER
P50 QL
ELECTRICITY
0.075
0.059
0.077
0.076
0.055
0.057
RETAIL
0.574
0.429
0.379
0.269
0.354
0.256 (0.2645)
VOLATILITY
0.050
0.047
0.042
0.042
0.039
0.039
TRAFFIC
0.161
0.122
0.117
0.115
0.095
0.101
TASK
DEEPAR
CONYTRANS
MQ-RNN
MQ-CNN
TFT
MQTR MQTRANSFORMER
P90 QL
ELECTRICITY
0.040
0.034
0.036
0.035
0.027
0.027
RETAIL
0.230
0.192
0.152
0.118
0.147
0.106 (0.109)
VOLATILITY
0.024
0.024
0.021
0.020
0.020
0.019
TRAFFIC
0.099
0.081
0.082
0.077
0.070
0.068
For the retail task, we predict the next 30 days of sales, given the previous 90 days of history. This dataset contains a rich set of static, time series, and known features. At the other end of the spectrum. the electricity load dataset is univariate. See Appendix D for additional information about these tasks. Table 4 compares MQTransformer's performance with other recent works DeepAR (Salinas et al., 2020), Conv (Li ct al., 2019), MQ-RNN (Wen et al., 2017), and TFT (Lim et al., 2019).
Our MQTransformer architecture is competitive with or beats the state-of-the-ee on the electricity load, volatility and traffic prediction tasks, as shown in Table 4. On the most challenging task, dramatically outperforms the previously reported state of the art by 38% and the MQ-CNN baseline by 5% at P50 and 11 % at P90. Because MQ-CNN and MQTransformer are trained using forking sequences, we can use the entire training population, rather than downsample as is required to train TFT (Lim et al., 2019). To ascertain what portion of the gain due to learning from more trajectories, versus our innovations alone, we retrain the optimal MQTransformer architecture using a random sub-sample of 450K trajectories (the same sampling procedure TFT) and without using forking sequences - the results are indicated in parentheses in Table 4. We can observe that MQTransformer still dramatically outperforms TFT, and its performance similar the MQ-CNN baseline trained on all trajectories. See Appendix D for more details and results on the Favorita forecasting task.
Computational Efficiency For purposes of illustration, consider the Retail dataset. Prior work (Lim et al., 2019) was only able to make use of 450K out of 20M trajectories, and the optimal TFT architecture required 13 minutes per epoch (minimum validation error at epoch 6)* using a single V100 GPU. Our innovations are compatible with forking sequences, and thus our architecture can make use of all available trajectories. To train, MQTransformer requires only 5 minutes per epoch (on 20M trajectories) using single V100 GPU (minimum validation error reached after 5 epochs)) Somee of the differences in runtime can be attributed to use of different deep learning frameworks, but is clear that MQTransformer can be trained much more efficiently than models like TFT and DeepAR.
5 CONCLUSIONS AND FUTURE WORK
In this work, present three novel architecture enhancements that improve bottlenecks in state of the art MQ- MQ-Foreccaters. To the best of our knowledge, this is the first work to consider the impact of model architecture on forecast evolution. On large scale moonand-orreasttng task demonstrated how position embeddings can be learned directly from domain-specific event indicators and horizon-specific contexts improve performance for difficult sub-problems such as promotions or seasonal peaks. Together, these innovations produced significant improvements in accuracy across different dimensions and in the excess variation of the forecast we also saw there was no tradeoff between these two desiderata, but rather they are closely linked. On a public retail forecasting task MQTransformer outperformed the baseline architecture by 5% P50, 11% at P90 and the previous reported state-of oftie-----h--rrr the-art (TFT) by 38%. Beyond accuracy gains, our model achieves massive increases throughput compared to existing transformer architectures for time-series forecasting.
'Results for TFT, MQ-RNN, DeepAR and ConvTrras are from Lim et al. (2019). "Timigg results obtained by running the source code provided by Lim et al. (2019)

---

Under review as a conference paper at ICLR 2022
An interesting direction we intend to explore in future work is incorporating encoder self-attentio so that the model can leverage arbitrarily long historical time series, rather than the fixed length consumed by the convolution encoder.
10

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
Our work included experiments on both public and proprietary datasets. Although, by definition, the experiments on the proprietary dataset cannot be reproduced by others, we did perform experiments using a public dataset with very similar features (Favorita forecasting task). In terms of reproducing our results on the public datasets, we used publicly available code to preprocess the data and construct model inputs (see Footnote 3). Since we were unable to obtain permission to release the source code for our model architecture, we instead included sufficient detail such that it will be straightforward for others to re-implement. See Table I, Table and Appendix B for details about the exact structure of the MQTransformer architecture. See Table and Appendix D for details about the hyperparameters used for each of the public datasets.

---

Under review as a conference paper at ICLR 2022
REFERENCES
AUGENBLICK, N. and RABIN, M. (2019). Belief movement, uncertainty reduction, and rational updating. Tech. rep., Haas School of Business, University of California, Berkeley.
BAHDANAU, D., CHO, and BENGIO, (2014). Neural machine translation by jointly learning to align and translate. arXiv: 1409. ....99.00000 0473.
BENIDIS, K., RANGAPURAM, S. S., FLUNKERT, V., WANG, B., MADDIX, D., TURKMEN, C., GASTHAUS, J., WOOLKE-SCHINEDEN M., SALINAS, D., STELLA, L. ET AL. (2020). Neural forecasting: Introduction and literature overview. 2004.
BRAY, R. L. and MENDELSON, H. (2012). Information Transmission and the Bullwhip Effect: An Empirical Investigation. Management Science 58 860-875.
CHEN, F., DREZNER, Z., RYAN, J. K. and SIMCHI-LEVI, D. (2000). Quantifying the Bullwhip Effect in Simple Supply Chain: The Impact of Forecasting, Lead Times, and Information. Management Science 46 436-443.
CHENG, J., DONG, I.. and LAPATA, M. (2016). Long short-term memory-networks for machine reading. arxiv: :::::66000 1601. 06733.
CINAR, Y. G., MIRISAEE, H., GOSWAMI, P., GAUSSIER, E., ArT- BACHIR A. and STRUOV, (2017). Position-basee content attention for time series forecasting with seguence-to-sequenee mns. ICONIP.
DAI, Z., YANG, Z., YANG, Y., CARBONELL, J., LE, Q. V. and SALAKHUTDINOV, R. (2019). Tr Attentive Language Models Beyond Fixed-Length Context. In ACL.
DEVLIN, J., CHANG,, M.-W., LEE, and TOUTANOVA, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT.
DUA, D. and GRAFF, C. (2017). UCI machine learning repository.
URL
ics uci. edu/mm
FOSTER, D. and STINE, R. (2021). Threshold Martingales and the Evolution of Forecasts. arXiv:2105.c 06834.
GALASSI, A., LIPPI, M. and TORRONI, P. (2019). Attention, please! critical review of neural attention models in natural language processing. a arXiv:1900 021 81.
GASPARIN, A., LUKOVIC, S. and ALIPPI, C. (2019). Deep Learning for Time Series Forecasting: The Electric Load Case. arXiv: 1907. 09207.
GERD HEBER, N. S., ASGER LUNDE and SHEPPARD, (2009). Oxford-man institute's 's realized library.
HEATH. D. C. and JACKSON, P. L. (1994). Modeling the evolution of demand forecasts with application to safety stock analysis in production/distriittiin systems. IIE transactions 26 17-30.
HUANG, C.-Z. A., VASWANI, A., USZKOREIT, J., SHAZEER, N., SIMON, I., HAWTHORNE, C.,
DAI, A. M., HOFFMAN, M. D., DINCULESCU, M. and ECK, D. (2018). Music Transformer: Generating Music with Long-Term Structure. arxiv: 1898 04281.
KIM, S. and KANG, M. (2019). Financial series prediction using Attention LSTM.
arXiv: 1.VVv19000 1902 10877.
KINGMA, D. and BA, (2015). Adam: A Method for Stochastic Optimization. ICLR.
LI, S., JIN, X., XUAN, Y., ZHOU, X., CHEN, W., WANG, Y.-X. and YAN., (2019). Enhancing the locality and breaking the memory bottleneck of transformer time series forecasting. In NIPS.
LIM, B., ARIK, S. O., LOFFF, N. and PFISTER, T. (2019). Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting. arXiv: 91 09363.
12

---

Under review as a conference paper at ICLR 2022
LUONG, M.-T., PHAM, H. and MANNING, C. D. (2015). Effective approaches attention based neural machine translation. arxiv:1508 04025.
MADEKA, D., SWINIARSKI, L.. FOSTER, D., RAZOUMOV, L., TORKKOLA, K. and WEN, R. (2018). Sample path generation for probabilistic demand forecasting. In ICML workshop on Theoretical Foundations and Applications of Deep Generative Models.
MUKHOTY, B. P., MAURYA, V. and SHUKLA, S. K. (2019). Sequence to sequence deep learning models for solar irradiation forecasting. In IEEE Milan PowerTech.
NASCIMENTO, R. C., SOUTO, M., OGASAWARA, E., PORTO, and BEZERRA, E. (2019). STConvS2S: Spatiotemporal Convolutional Sequence to Sequence Network for weather forecasting.
arxiv:1912.0013:
SALINAS, D., FLUNKERT, V., GASTHAUS, J. and JANUSCHOWSKI, T. (2020). Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting 36 1181-1191.
SHAW, P., USZKOREIT, J. and VASWANT, A. (2018). Self-Attention with Relative Position Repre- sentations. In NAACI..
SHUN- YAO SHIH AND FAN-KENG SUN AND HUNG-YI LEE (2019). Temporal pattern attention for multivariate time series forecasting. Machine Learning 108 1421-1441.
SUTSKEVER, I., VINYALS, O. and LE, Q. V. (2014). Sequence to sequence learning with neural networks. In NIPS.
TALEB, N. N. (2018). Election predictions as martingales: an arbitrage approach. Quantitative Finance 18 1-5.
TALEB, N. N. and MADEKA, D. (2019). All roads lead to quantitative finance. Quantitative Finance 19 1775-1776.
VAN DEN OORD, A., DIELEMAN, S., ZEN, H., SIMONYAN, K., VINYALS, O., GRAVES, A.,
KALCHBRENNER. N., SENIOR, A. and KAVUKCUOGLU, K. (2016). Wavenet: A generative model for raw audio. arXiv:1609, 03499.
VASWANI, A., SHAZEER, N., PARMAR, N., USZKOREIT, J., JONES, L., GOMEZ, N., KAISER, L., and POLOSUKHIN, I. (2017). Attention is all you need. In NIPS.
WEN, R. and TORKKOLA, K. (2019). Deep Generative Coppla Models for Probabilistic Forecasting. In JCML Time Series Workshop.
WEN, R., TORKKOLA, K., NARAYANASWAMY, B. and MADEKA, D. (2017). A multi-horizon quantile recurrent forecaster. In NIPS Time Series Workshop.
WILLIAMS, (1991). Probability with Martingales. Cambridge University Press.
XU, K., BA, J., KIROS, R., CHO, K., COURVILLE, A., SALAKHUDINOV, R., ZEMEL, R. and BENGIO, Y. (2015). Show, attend and tell: Neural image caption generation with visual attention. In ICML.
YU, R., ZHENG, S., ANANDKUMAR, A. and YUE, Y. (2017). Long-term Forecasting using Higher Order Tensor RNNs. arXiv: 00073.
13

---

Under review as a conference paper at ICLR 2022
A ADDITIONAL BACKGROUND AND RELATED WORK
A.1 ATTENTION MECHANISMS
Attention mechanisms can be viewed as form of content based addressing, that computes an alignment between set of queries and keys to extract value. Formally, let qu..... Qr, . and Vi,.... Vt. be a series queries, keys and values, respectively. The 4t attended value is defined q.. as C2 = Ei=1 score qs ky Vv. where score is a scoring function commonly score(u, v) := u v. In the vanilla transformer model, 9s k, V, hg. where h, is the hidden state time Because attention mechanisms have no concept of absolute or relative position, some sort of position information must be provided. Vaswani al. (2017) uses sinusoidal positional encoding added to the input to an attention block, providing each token's 's position in the input time series.
In the vanilla transformer (Vaswani al., 2017), sinusoidal position embedding is added to the network input and each encoder layer consists of multi-headed attention block followed by a werdaro sub-layer. For each head i, the attention score between query qs and key is defined as follows for the input layer
A (x, rss) T.
W
+ r2)
where Xs, are the observation of the time series and the position encoding, respectively, time 8. In Section we introduced attention mechanisms that differ in their treatment the position dependent biases
B MQTRANSFORMER ARCHITECTURE DETAILS
In this section we describe in detail the layers in our MQTransformer architecture, which is based off of the MQ-Forecaster framework (Wen et al., 2017) and uses a wavenet encoder (van den Oord et al., 2016) for time-series covariates. Before describing the layers in each component, Figure outlines the MQTransformer architecture. On different datasets, we consider the following variations: choice of encoding for categorical variables, a tunable parameter d, (dimension of hidden layers), dropout rate Pårops list of dilation rates for the wavenet encoder, and list of dilation rates for the position encoding. The ReLU activation function used throughout the network.
B. INPUT EMBEDDINGS AND POSITION ENCODING
Static categorical variables are encoded using either one-hot encoding or an embedding layer. Time- series categorical variables are one-hot encoded, and then passed through a single feew-forward layer of dimension d..
The global position encoding module takes as input the known time-series covariates, and consist of a stack of dilated, bi-directional 1-D convolution layers with d, filters. After cach convolution is ReLU activation, followed by dropout layer with rate Pdrop, and the local position encoding is implemented as a single dense layer of dimension dn Figure 4b shows an example of the position encoding from model trained on the large-scale demand forecasting task.
B.2 ENCODER
After categorical encodings applied, the inputs are passed through the encoder block. The encoder consists of two components: single dense layer to encode the static features, and a stack of dilated, temporal convolutions. The time-series covariates form the input to the convolution stack. The output of the encoder block is produced by replicating the encoded static features across all time steps and concatenating with the output of the convolution.
concatenated with the position encoding to
B.3 DECODER
Please see Table 1 for description of the blocks in the decoder. The dimension of each head in both the horizon-specific and decoder self-attention blocks d,//2. /2. The dense layer used to compute ef has dimension d2/22. The output block is two layer MI.P with hidden layer dimension d,/2,
14

---

Under review as a conference paper at ICLR 2022
Seasonal Peaks
Dropout
Linear
Attention Cell
LayerNorm (x3)
Q
K
10
20
30
40
50
Time
(h) Position encoding learned from event indicators in the large scale demand forecasting task (Section 4)
(a) Basic structure of our time-series attention blocks
Figure 4: Components of the MQTransformer architecture
and weights shared across all time points and horizons. The output layer has output per horizon, quantile pair. Figure 4a shows the structure of both our attention blocks, where the sub-layer "Attention Cell" is replaced with either the decoder-eneoder decoder-self attention.
15

---

Under review as a conference paper at ICLR 2022
h=--
Decodor f-2
Eacoder
XL
lo-1
A
Docodar
y/+
+
b
Hetioos
MLP
MLP
Posttimal Thooding
(mm))
Decodae Self Attention
(im-4)
1p
Postiional
CH1
Ehooding
Decnder Enccode
Horben- Wortaen-Sppectt Speetffic
Atlectioe
Figure 5: MQTransformet architecture with learned global/local positional encoding, horizon-specifie Pigndder-encoode altention, and decoder self-attention
16

---

Under review as a conference paper at ICLR 2022
C LARGE SCALE DEMAND FORECASTING EXPERIMENTS
C.I ABLATION STUDY
In the ablation study, the model MQT-AIl uses a standard multi-heat attention applied the concate- nated contexts for all horizons at each period t. A separate head is used for each output horizon (52 heads on the private dataset),
Table Attention weight and output computations for MQT-AIl decoder self-attention
ATTENTION WEIGHTS
OUTPUT
A q. W Wek,
ट = Σ Aln Wh
y
= [ea,1) Ca hard)
k,- [c,,13-* Ts
V. [0, ;c,,H]
Table 6 gives the number of parameters in each trained model.
Table Parameter counts in trained models for the large scale demand-forecasting task.
MODEL
NUMBER OF PARAMETERS
MQ-CNN
9.17 x 10a
MQT
9.25 10
MQT-NoDS
9.09 x 10
MQT-ALL
2.82 10
C.2
EXPERIMENT SETUP
In this section we describe the details the model architecture and training procedure used in the experiments the large-scale demand forecasting application.
TRAINING PROCEDURE
Because did not have enough history available set aside true holdout set, all models trained for 100 epochs, and the final model is evaluated on the test set. For the same reason, no hyperparameter tuning was performed.
ARCHITECTURE AND HYPERPARAMETERS
The categorical variables consist of static features of the item, and the timeseries categorical variables are event indicators (e.g. holidays). The parameters are summarized in Table 7.
C.3 RESULTS FOR SHORTER HORIZONS
Table 8 shows results along various dimensions for horizons up to 6 weeks. Similar to the results in Section 4, MQTransformer significantly outperforms the baseline.
D EXPERIMENTS PUBLIC DATASETS
In this section describe the experiment setup used for the public datasets in Section 4. As mentioned in Section 4.2, we display the baseline results published in Lim et al. (2019). For MQ-CNN and MQTransformer, use their pre-processing and evaluation code ensure parity.
17

---

Under review as a conference paper at ICLR 2022
Table Parameter settings for Large Scale Demand Forecasting Experiments
Parameter
Value
Encoder Convolution Dilation Rates
[1,2,4,8.16,32]
Position Encoding Dilation Rates
[1,2,4,8,16,20]
Static Categorical
One-Hot
Time-Series Categorical
One-Hot
Static Encoder Dimension
64
Convolution Filters
32
Attention Block Head Dimension
16
Dropout Rate
0.15
Activation Function
ReLU
Table 8: Quantile loss on the backtest year along different dimensions. Values are relative performance versus baseline, with the best result for each dimension emphasized.
MQ-CNN
MQT
MQT-NoDS
MQT-ALL
DIMENSION
P50
P90
P50
P90
P90
OVERALL
1,000
1.000
0.962
0.915
0.983
0.963
0.977
0.957
SEASONAL PEAK
1,000
1.000
0.865
0.667
0.881
0.688
0.908
0.729
SEASONAL PEAK
1.000
1.000
0.925
0.850
0.972
0.910
0.932
0.886
SEASONAL PEAK 3
1,000
1.000
0.798
0.804
0.803
0.872
0.815
0.830
POST-PEAK RAMPDOWN
1,000
1,000
0.968
0.956
0.983
0.996
0.967
0.975
PROMOTION TYPE
1,000
1,000
0.915
0.601
0.929
0.616
0.921
0.628
PROMOTION TYPE 2
1,000
1.000
0.764
0.537
0.805
0.561
0.861
0.640
PROMOTION TYPE 3
1.000
1.000
0.901
0.746
1.075
0.942
0.941
0.848
18

---

Under review as a conference paper at ICLR 2022
D.1 DATASETS
We evaluate our MQTransformer on four public datasets. We summarize the datasets and preprocess ing logic below; the reader is referred to Lim et (2019) for more details. Lim et (2019) released their code under the Apache 2.0 License. Favorita's terms of use for their retail dataset allows for it to be used for scientific research. The Electricity and Traffic datasets are provided by the UCI machine learning repository (Dua and Graff, 2017). The volatility dataset is sourced from the Oxford-Man Institute's 's realized library v0.3 (Gerd Heber and Sheppard, 2009).
RETAIL
This dataset is provided by the Favorita Corporacion (a major Grocery chain in Ecuador) part of Kaggle² to predict sales for thousands of items at multiple brick-and-morta. locations. In total there treated as distinct entities), and the dataset contains variety of features including: local, regional and national holidays; static features about each item; total sales volume at each location. The task is predict log-sales for each (item, store) combination over the next 30 days, using the previous 90 days of history. The training period is January 1, 2015 through December 2015. The following 30 days are used as a validation set, and the 30 days after that as the test set. These 30 day windows correspond to single FCT. While Lim et al. (2019) extract only 450K samples from the histories during the train window, there are in fact 20M trajectories avalaible for training- because our models can produce forecasts for multiple trajectories (FCDs) simultaneously, we train using all available data from the training window.
135K items (item, store combinations
For the volatility analysis presented in Figure 6, we used 60 day validation window (March 1, 2016 through May 1, 2016), which corresponds to 30 FCTs.
ELECTRICITY
This dataset consists of time series for 370 customers of at hourly grain. The univariate data is augmented with a day-of- week, hour-of-day and offset from a fixed time point. The task to predict hourly load over the next 24 hours for each customer, given the past seven days of usage. From the training period (January 1, 2014 through September, 2019) 500K samples are extracted.
TRAFFIC
This dataset consists of lane occupancy information for 440 San Francisco area freeways. The data is aggregated to an hourly grain, and the task is to predict the hourly occupancy over the next 24 hours given the past seven days. The training period consist of all data before 2008-06-15, 15, with the final 7 days used as a validation set. The days immediately following the training window is used for evaluation. The model takes input lane occupancy, hour of day. day of week, hours from start and an entity identifier.
VOLATILITY
The volatility dataset consists minute sub-sampled realized volatility measurements from 2000- 01-03 to 2019-06-28. Using the past one year's worth of daily measurements, the goal is to predict the next week's (5 business days) volatility. The period ending on 2015-12-31 is used as the training set, 2016-2017 as the validation set, and 2018-01- through 2019-06-28 as the evaluation set. The region identifier is provided as a static covariate, along with time-varying covariates daily returns, day-of-week, week-of-yen and month. A log transformation is applied to the target.
D.2 TRAINING PROCEDURE
We only consider tuning two hyper-parameters, size hidden layer d, {32, 64, 128) and learning rate a E {1 x 10 1 10-3 x 10-4). The model is trained using the ADAM optimizer Kingma and Ba (2015) with parameters 31 = 0.9, B2 = 0.999, = 1e - 8 and a minibatch size of 256, for a maximum of 100 epochs and an early stopping patience of epochs.
Thhe original competition can be found here.
19

---

Under review as a conference paper at ICLR 2022
We train model for each hyperparameter setting in the search grid (6 combinations), select the one with the minimal validation loss and report the selected model's 's test-set error in Table 4.
D.3 ARCHITECTURE DETAILS
Our MQTransformer architecture used for these experiments contain single tune-able hyperparame- ter hidden layer dimension d, Dataset specific settings are used for the dilation rates. For static categorical covariates we use an embedding layer with dimension dą and use one-hot encoding for time-series covariates. A dropout rate of 0. 0.15 and ReLU activations are used throughout the network. The only difference between this variant and the one used for the non non-public large scale demand forecasting task is the use of an embedding layer for static, categorical covariates rather than one-hot encoding.
D.4 REPORTED MODEL PARAMETERS
The optimal parameters for each task are given in Table 9.
Table 9: Parameter settings of reported MQTransformer model on each public dataset.
Name
d2
Enc. Dilation Rates
Pos. Dilation Rates
Electricity
128
1 X 10-3
[1,2,4.8,16.32]
[1,2,4,8.8] [1,2,4,8,8
Traffic
64
1 x 10-3
[1,2,4,8,16.22]
Volatility
128
1 x 10-3
11,2,48.16,32264]
[1,1,2]
Retail
64
1 x 10-1
[1,2,48,16,322]
[1,2,4,8,14]
D.5 MARTINGALE DIAGNOSTICS ON FAVORITA FORECASTING TASK
model
0.04
MCDTranstormer
Quantile Lose
0.000
0.03
lile
-0.001
PSO
0.02
0.01
-0.003
0.00
8 Time
10
12 14
16
Lead Time
Figure 6: Forecast evolution analysis on the retail dataset. Left: Martingale Diagnostic Process (V.). Right: QL by lead time, averaged over target dates from 2016-03-01 through 2016-05-01; QL trajectories are centered around 0.
On the Favorita retail forecasting task, Figure shows that as expected, MQTransformer substantially reduces excess volatility in the forecast evolution compared to the MQ-CNN baseline, Somewhat surprisingly, TFT exhibits much lower volatility than does MQTransformer. hand plot displays quantile loss as the target date approaches trajectories for each model are 7ero centered to emphasize the trends exhibited. While TFT is less volatile, it is also less accurate as fails to incorporate newly available information. By contrast, MQTransformer is both less volatile and more accurate when compared with MQ-CNN.
Figure 6, the right
20


Reviews:
[{"review_id": "eV0e069J1q1", "reviewer": "Reviewer_DW35", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work applies Transformer architecture on forecasting tasks, and shows improvement over existing work, including the previous work that used Transformer for decoder. The overall framework follows the conventional Transformer design with minor additions, and therefore provides limited technical novelty. There is some room for improvement on the clarity, as some notations are not clearly defined or used to indicate different meanings at different paragraphs.\n", "summary_of_the_paper": "This paper looks into applying Transformer with encoder-decoder architecture for forecasting. The work follows the previous work of multi-horizon quantile forecasts that predict quantile for each horizon. The framework applies a 1D-conv and a MLP network on time series feature to model positional information, and apply an attention head for each horizon. The authors also show that limiting the context of the attention provides better bias to improve the prediction quality. The authors also discussed that having access to past context allows reducing the volatility. The proposed framework showed some improvement on demand forecasting tasks.", "summary_of_the_review": "The proposed approach shows some improvement compared to existing work, but it’s technical technical novelty is a bit limited.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KwxZjSbX-NE", "reviewer": "Reviewer_3wZQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\na) Significant gains in forecast accuracy on the proprietary data, and on the public retail forecasting task\n\nb) Including forecast volatility as part of the model leads to a significant reduction in excess volatility over the MQ-CNN model\n\nCons:\n\na) A large part of the empirical analysis is conducted on a proprietary dataset, and thus those results are not reproducible or verifiable.  Furthermore, in these results the authors mask absolute values and only present relative gains, making it even harder to assess performance.\n\nb) For results on the public task, a more direct comparison with MQ-CNN model is needed as the proposed model aims to improve over this model.  Since previously published results from the MQ-CNN model are on the GEFCom2014 forecasting task, could this be used to compare the proposed model?\n\nc) Ablation studies are needed to understand key contributors to the gains observed.  For instance, on the ‘retail’ public task, a 38% accuracy gain is observed over the TFT model, yet excess variability studies show that TFT model has a lower excess variability as compared to the MQ transformer model (Fig. 6 in appendix D.5).  This raises the question if less focus on reducing excess variability will lead to further accuracy gains on this task.\n\nd) Reproducibility of results is questionable.  Implementation details are shared to some extent, but lack of code, large focus on internal datasets that can not be released and numbers on those can not be disclosed, and a lack of detailed ablations will make it difficult to reproduce in my opinion.\n\nMinor edits:\n* subscript ‘I’ missing from x^{(s)} after Eq. (1)\n* what is s_i in the reformulated version of (1) on page 4\n* ‘where c_{:t} denotes … but there is no c_{:t} on page 5\n", "summary_of_the_paper": "The paper proposes an improvement over previously published MQ-forecaster model for multi-horizon forecasts in time series data.  The proposed approach aims to improve the forecast accuracy as well as achieve a lower variability in forecasts made at various times for a specific time point in the future (excess forecast variability).  A number of model improvements are proposed, including a new way of handling positional encoding (via learnt embedding of event indicators), separate encodings for different horizons, and decoder attention on previous forecast errors to allow decoder to be aware of excess forecast variability and reduce it.  Empirical evaluation is carried out for the task of demand forecasting on a large but proprietary dataset, and also for tasks of retail sales, electricity load, securities volatility, and traffic forecasting on publicly available datasets.", "summary_of_the_review": "My rating is due to concerns about reproducibility (no code and bunch of results on proprietary dataset), and lack of comparison with previously published results for the MQ-CNN model which this paper aims to improve over.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "F9RCZoi_f0", "reviewer": "Reviewer_tUKV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n- A clear formal introduction to the problem statement and concise descriptions of the three contributions. The overall stricture and readability of the paper are good.\n\n- Authors have attested their framework against state-of-the-art deep learning time series forecasting methods and evaluated using 5 benchmark datasets, including a one retail dataset, and four publicly available datasets. The benchmark methods and datasets used are well-established yardsticks in the multivariate forecasting (or global models where sets of many time series are available) domain. So, the experiments are strong and well empirically supported.\n\n- The use of an ablation study to measure the effect of the proposed attention units and self-attention scheme. \n\n- Have also reported the computational cost of the framework and compared against the current state-of-the-art (Lim et al., 2019)\n\n\nWeaknesses\n\n- Only the ablated based variants of MQTransformer are evaluated and compared on the large-scale retail dataset. What is the reason for excluding other forecasting benchmarks, such as DeepAR, TFT, MQ-RNN in Table 3 ?\n\n- The formula used to calculate the quantile loss (quantile forecast error) is not defined. Please add this formula to the appendix or main text to improve the clarity of the error measures summarised in the tables.\n\n- I also strongly recommend the authors to include state-of-the-art Informer framework [1] as a benchmark in this study. This is because, Informer framework also follows the self- attention architecture, thus can be directly comparable against the proposed MQTransformer.\n\n [1] Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H. and Zhang, W., 2021, May. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of AAAI.\n", "summary_of_the_paper": "This study proposes a decoder-encoder attention mechanism, a positional encoding mechanism and a decoder self-attention scheme to improve the forecast accuracy and minimise the excess variations in time series forecasting applications. As highlighted in the introduction section of the paper, both improving the forecast accuracy and minimising the unnecessary forecast volatility are important endeavours in the time series forecasting domain. Hence the motivations of this study are clear and strong. The proposed MQTransformer framework aims to obviate the limitations of MQ-Forecaster, which is also a state-of-the-art framework in the recent time series forecasting literature. ", "summary_of_the_review": "Given the significance of the three contributions, supported by the strong experimental setup (well established benchmarks and datasets) and empirical evidence (statistical significance), I am recommending to accept this paper.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a number of improvements to the previously-published transformer-based MQ-forecaster model for multi-horizon forecasts on time series data. They show strong empirical improvements in terms of accuracy and excess forecast variability on a large proprietary dataset, as well as four public datasets. Concerns were raised about the relatively incremental changes to the MQ-forecaster model this work is based on, lack of ablations on public data and, relatedly, inability to reproduce results on the proprietary data.
Decision: Reject