MQTransformer: Multi-Horizon Forecasts with Context Dependent and Feedback-Aware Attention

1 INTRODUCTION . Time series forecasting is a fundamental problem in machine learning with relevance to many application domains including supply chain management , finance , healthcare analytics , and more . Modern forecasting applications require predictions of many correlated time series over multiple horizons . In multi-horizon forecasting , the learning objective is to produce forecasts for multiple future horizons at each time-step . Beyond simple point estimation , decision making problems require a measure of uncertainty about the forecasted quantity . Access to the full distribution is usually unnecessary , and several quantiles are sufficient ( many problems in Operations Research use the 50th and 90th percentiles , for example ) . As a motivating example , consider a large e-commerce retailer with a system to produce forecasts of the demand distribution for a set of products at a target time T . Using these forecasts as an input , the retailer can then optimize buying and placement decisions to maximize revenue and/or customer value . Accurate forecasts are important , but – perhaps less obviously – forecasts that don ’ t exhibit excess volatility as a target date approaches minimize costly , bull-whip effects in a supply chain ( Chen et al. , 2000 ; Bray and Mendelson , 2012 ) . Recent work applying deep learning to time-series forecasting focuses primarily on the use of recurrent and convolutional architectures ( Nascimento et al. , 2019 ; Yu et al. , 2017 ; Gasparin et al. , 2019 ; Mukhoty et al. , 2019 ; Wen et al. , 2017 ) 1 . These are Seq2Seq architectures ( Sutskever et al. , 2014 ) – which consist of an encoder that summarizes an input sequence into a fixed-length context vector , and a decoder which produces an output sequence . This line of work has lead to major advances in forecast accuracy for complex problems , and real-world forecasting systems increasingly rely on neural nets . Accordingly , a need for black-box forecasting system diagnostics has arisen . Foster and Stine ( 2021 ) use probabilistic martingales to study the dynamics of forecasts produced by an arbitrary forecasting system . They can be used to detect the degree to which forecasts adhere to the martingale model of forecast evolution ( Heath and Jackson , 1994 ) and to detect unnecessary volatility ( above and beyond any inherent uncertainty ) in the forecasts produced – that is to say , Foster 1For a complete overview see Benidis et al . ( 2020 ) and Stine ( 2021 ) describe a way to connect the excess variation of a forecast to accuracy misses against the realized target . While tools such as martingale diagnostics can be used to detect flaws in forecasts , the question of how to incorporate that information into model design is unexplored . Existing multi-horizon forecasting architectures ( Wen et al. , 2017 ; Madeka et al. , 2018 ) are designed to minimize quantile loss , but they do not explicitly handle excess variation , since forecasts on any particular date are not made aware of errors in the forecast for previous dates . Another limitation in many existing architectures is the well known information bottleneck , where then encoder transmits information to the decoder via a single hidden state . To address this , Bahdanau et al . ( 2014 ) introduce a method called attention , allowing the decoder to take as input a weighted combination of relevant latent encoder states at each output time step , rather than using a single context to produce all decoder outputs . While NLP is the predominate application of attention architectures to date , in this paper we use novel attention modules and positional embeddings to introduce the proper inductive biases for probabilistic time-series forecasting to the model architecture . Summary of Contributions In this paper , we are concerned with both improving forecast accuracy and reducing excess forecast volatility . We present a set of novel architectures that seek to remedy some of inductive biases that are currently missing in state of the art MQ-Forecasters ( Wen et al. , 2017 ) . Our main contributions are 1 . Positional Encoding from Event Indicators : Current MQ-Forecasters use explicitly engineered holiday “ distances '' to provide the model with information about the seasonality of the time series . We introduce a novel positional encoding mechanism that allows the network to learn a context-dependent seasonality function specific that is more general than conventional position encoding schemes . 2 . Horizon-Specific Decoder-Encoder Attention : Wen et al . ( 2017 ) ; Madeka et al . ( 2018 ) and other MQ-Forecasters learn a single encoder representation for all future dates and periods being forecasted . We present a novel horizon-specific decoder-encoder attention scheme that allows the network to learn a representation of the past that depends on the period being forecasted . 3 . Decoder Self-Attention for Forecast Evolution : To the best of our knowledge , this is the first work to consider the impacts of network architecture design on forecast evolution . Importantly , we accomplish this by using attention mechanisms to introduce the right inductive biases , and not by explicitly penalizing a measure of forecast variability . This allows us to maintain a single objective function without needing to make trade-offs between accuracy and volatility . By providing MQ-Forecasters with the structure necessary to learn context information dependent encodings , we observe major increases in accuracy ( 5.5 % in overall P90 quantile loss throughout the year , and up to 33 % during peak periods ) on our demand forecasting application , along with a statistically significant reduction in excess volatility ( 67 % reduction at P50 and 95 % at P90 ) . We also apply MQTransformer to four public datasets , and show parity with the state-of-the-art on simple , univariate tasks . On a substantially more complex public dataset ( retail forecasting ) we demonstrate a 38 % improvement over the previously reported state-of-the-art , and a 5 % improvement in P50 QL , 11 % in P90 QL versus our baseline . Because our innovations are compatible with efficient training schemes , our architecture also achieves a significant speedup ( several orders of magnitude greater throughput ) over earlier transformer models for time-series forecasting . 2 BACKGROUND AND RELATED WORK . 2.1 TIME SERIES FORECASTING . Formally , the task considered in our work is the high-dimensional regression problem p ( yt+1 , i , . . . , yt+H , i|y : t , i , x ( h ) : t , i , x ( f ) : t , i , x ( s ) i ) , ( 1 ) where yt+s , i , y : t , i , x ( h ) : t , i , x ( f ) t : ,i , x ( s ) i denote future observations of the target time series i , observations of the target time series observed up until time t , the past covariates , known future information , and static covariates , respectively . For sequence modeling problems , Seq2Seq ( Sutskever et al. , 2014 ) is the canonical deep learning framework and although they applied this architecture to neural machine translation ( NMT ) tasks , it has since been adapted to time series forecasting ( Nascimento et al. , 2019 ; Yu et al. , 2017 ; Gasparin et al. , 2019 ; Mukhoty et al. , 2019 ; Wen et al. , 2017 ; Salinas et al. , 2020 ; Wen and Torkkola , 2019 ) . The MQ-Forecaster framework ( Wen et al. , 2017 ) solves ( 1 ) above by treating each series i as a sample from a joint stochastic process and feeding into a neural network which predicts Q quantiles for each horizon . These types of models have limited contextual information available to the decoder – a drawback inherited from the Seq2Seq architecture – as it produces each estimate yqt+s , i , the q th quantile of the distribution of the target at time t+s , yt+s , i . Seq2Seq models rely on a single encoded context to produce forecasts for all horizons , imposing an information bottleneck and making it difficult for the model to understand long term dependencies . Our MQTransformer architecture uses the direct strategy : the model outputs the quantiles of interest directly , rather than the parameters of a distribution from which samples are to be generated . This has been shown ( Wen et al. , 2017 ) to outperform parametric models , like DeepAR ( Salinas et al. , 2020 ) , on a wide variety of tasks . Recently , Lim et al . ( 2019 ) consider an application of attention to multi-horizon forecasting , but their method still produces a single context for all horizons and by using an RNN decoder , does not enjoy the same scaling properties as MQ-Forecaster models . 2.2 ATTENTION MECHANISMS . Bahdanau et al . ( 2014 ) introduced the concept of an attention mechanism to solve the information bottleneck and sequence alignment problems in Seq2Seq architectures for NMT . Recently , attention has enjoyed success across a diverse range of applications including natural language processing ( NLP ) , computer vision ( CV ) and time-series forecasting tasks ( Galassi et al. , 2019 ; Xu et al. , 2015 ; Shun-Yao Shih and Fan-Keng Sun and Hung-yi Lee , 2019 ; Kim and Kang , 2019 ; Cinar et al. , 2017 ; Li et al. , 2019 ; Lim et al. , 2019 ) . Many variants have been proposed including self-attention and dot-product attention ( Luong et al. , 2015 ; Cheng et al. , 2016 ; Vaswani et al. , 2017 ; Devlin et al. , 2019 ) , and transformer architectures ( end-to-end attention with no recurrent layers ) achieve state-of-the-art performance on most NLP tasks . Time series forecasting applications exhibit seasonal trends and the absolute position encodings commonly used in the literature can not be applied . Our work differs from previous work on relative position encodings ( Dai et al. , 2019 ; Huang et al. , 2018 ; Shaw et al. , 2018 ) in that we learn a representation from a time series of indicator variables which encode events relevant to the target application ( such as holidays and promotions ) . This imposes a strong inductive bias that allows the model to generalize well to future observations . Existing encoding schemes either involve feature engineering ( e.g . sinusoidal encodings ) or have a maximum input sequence length ; ours requires no feature engineering – the model learns it directly from raw data – and it extends to arbitrarily long sequences . See Appendix A for additional background on attention mechanisms . 2.3 MARTINGALE DIAGNOSTICS . Originally the martingale model of forecast evolution ( MMFE ) was conceived as a way to simulate demand forecasts used in inventory planning problems ( Heath and Jackson , 1994 ) . Denoting by ŶT |t the forecast for YT made at time t ≤ T , the MMFE assumes that the forecast process { ŶT |t } t is martingale . Informally , a martingale captures the notion that a forecast should use all information available to the forecasting system at time t. Mathematically , a discrete time martingale is a stochastic process { Xt } such that E [ Xt+1|Xt , . . . , X1 ] = Xt . We assume a working knowledge of martingales and direct the reader to Williams ( 1991 ) for a thorough coverage in discrete time . Taleb ( 2018 ) describe how martingale forecasts correspond to rational updating , then expanded by Augenblick and Rabin ( 2019 ) . Taleb ( 2018 ) , Taleb and Madeka ( 2019 ) and Augenblick and Rabin ( 2019 ) go on to develop tests for forecasts that rule out martingality and indicate irrational or predictable updating for binary bets . Foster and Stine ( 2021 ) further extend these ideas to quantile forecasts . They consider the coverage probability process pt : = P [ YT ≤ τ |Ys , s ≤ t ] = E [ I ( YT ≤ τ ) |Ys , s ≤ t ] , where τ denotes the forecast announced in the first period t = 0 . Because { pt } is also a martingale , the authors show that E [ ( pT − π ) 2 ] = ∑T t=1 E [ ( pt − pt−1 ) 2 ] = π ( 1− π ) , where π = p0 is the expected value of pT , a Bernoulli random variable , across realizations of the coverage process . In the context of quantile forecasting , π is simply the quantile forecasted . The measure of excess volatility proposed is the quadratic variation process associated with { pt } , Qs : = ∑s t=0 ( pt − pt−1 ) 2 . While this process is not a martingale , we do know that under the MMFE assumption , E [ QT ] = π ( 1− π ) . The martingale Vt : = Qt − ( pt − π ) 2 is obtained from this process in the usual way by subtracting the compensator . In Section 4 we leverage the properties of { Vt } to compare the dynamics of forecasts produced by a variety of models , demonstrating that our feedback-aware decoder self-attention units reduce excess forecast volatility . Notation We denote by H and Q the number of horizons and quantiles being forecast , respectively . Bolded characters are used to indicate vector and matrix values . The concatenation of two vectors v and u is denoted as [ u ; v ] .

Summary: This paper looks into applying Transformer with encoder-decoder architecture for forecasting. The work follows the previous work of multi-horizon quantile forecasts that predict quantile for each horizon. The framework applies a 1D-conv and a MLP network on time series feature to model positional information, and apply an attention head for each horizon. The authors also show that limiting the context of the attention provides better bias to improve the prediction quality. The authors also discussed that having access to past context allows reducing the volatility. The proposed framework showed some improvement on demand forecasting tasks.