Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows

1 INTRODUCTION . Classical time series forecasting methods such as those in Hyndman & Athanasopoulos ( 2018 ) typically provide univariate forecasts and require hand-tuned features to model seasonality and other parameters . Time series models based on recurrent neural networks ( RNN ) , like LSTM ( Hochreiter & Schmidhuber , 1997 ) , have become popular methods due to their end-to-end training , the ease of incorporating exogenous covariates , and their automatic feature extraction abilities , which are the hallmarks of deep learning . Forecasting outputs can either be points or probability distributions , in which case the forecasts typically come with uncertainty bounds . The problem of modeling uncertainties in time series forecasting is of vital importance for assessing how much to trust the predictions for downstream tasks , such as anomaly detection or ( business ) decision making . Without probabilistic modeling , the importance of the forecast in regions of low noise ( small variance around a mean value ) versus a scenario with high noise can not be distinguished . Hence , point estimation models ignore risk stemming from this noise , which would be of particular importance in some contexts such as making ( business ) decisions . Finally , individual time series , in many cases , are statistically dependent on each other , and models need the capacity to adapt to this in order to improve forecast accuracy ( Tsay , 2014 ) . For example , to model the demand for a retail article , it is important to not only model its sales dependent on its own past sales , but also to take into account the effect of interacting articles , which can lead to cannibalization effects in the case of article competition . As another example , consider traffic flow in a network of streets as measured by occupancy sensors . A disruption on one particular street will also ripple to occupancy sensors of nearby streets—a univariate model would arguably not be able to account for these effects . In this work , we propose end-to-end trainable autoregressive deep learning architectures for probabilistic forecasting that explicitly models multivariate time series and their temporal dynamics by employing a normalizing flow , like the Masked Autoregressive Flow ( Papamakarios et al. , 2017 ) or Real NVP ( Dinh et al. , 2017 ) . These models are able to scale to thousands of interacting time series , we show that they are able to learn ground-truth dependency structure on toy data and we establish new state-of-the-art results on diverse real world data sets by comparing to competitive baselines . Additionally , these methods adapt to a broad class of underlying data distribution on account of using a normalizing flow and our Transformer based model is highly efficient due to the parallel nature of attention layers while training . The paper first provides some background context in Section 2 . We cover related work in Section 3 . Section 4 introduces our model and the experiments are detailed in Section 5 . We conclude with some discussion in Section 6 . The Appendix contains details of the datasets , additional metrics and exploratory plots of forecast intervals as well as details of our model . 2 BACKGROUND . 2.1 DENSITY ESTIMATION VIA NORMALIZING FLOWS . Normalizing flows ( Tabak & Turner , 2013 ; Papamakarios et al. , 2019 ) are mappings from RD to RD such that densities pX on the input space X = RD are transformed into some simple distribution pZ ( e.g . an isotropic Gaussian ) on the space Z = RD . These mappings , f : X 7→ Z , are composed of a sequence of bijections or invertible functions . Due to the change of variables formula we can express pX ( x ) by pX ( x ) = pZ ( z ) ∣∣∣∣det ( ∂f ( x ) ∂x ) ∣∣∣∣ , where ∂f ( x ) /∂x is the Jacobian of f at x. Normalizing flows have the property that the inverse x = f−1 ( z ) is easy to evaluate and computing the Jacobian determinant takes O ( D ) time . The bijection introduced by Real NVP ( Dinh et al. , 2017 ) called the coupling layer satisfies the above two properties . It leaves part of its inputs unchanged and transforms the other part via functions of the un-transformed variables ( with superscript denoting the coordinate indices ) { y1 : d = x1 : d yd+1 : D = xd+1 : D exp ( s ( x1 : d ) ) + t ( x1 : d ) , where is an element wise product , s ( ) is a scaling and t ( ) a translation function from Rd 7→ RD−d , given by neural networks . To model a nonlinear density map f ( x ) , a number of coupling layers which map X 7→ Y1 7→ · · · 7→ YK−1 7→ Z are composed together all the while alternating the dimensions which are unchanged and transformed . Via the change of variables formula the probability density function ( PDF ) of the flow given a data point can be written as log pX ( x ) = log pZ ( z ) + log |det ( ∂z/∂x ) | = log pZ ( z ) + K∑ i=1 log |det ( ∂yi/∂yi−1 ) | . ( 1 ) Note that the Jacobian for the Real NVP is a block-triangular matrix and thus the log-determinant of each map simply becomes log |det ( ∂yi/∂yi−1 ) | = log | exp ( sum ( si ( y1 : di−1 ) ) | , ( 2 ) where sum ( ) is the sum over all the vector elements . This model , parameterized by the weights of the scaling and translation neural networks θ , is then trained via stochastic gradient descent ( SGD ) on training data points where for each batch D we maximize the average log likelihood ( 1 ) given by L = 1 |D| ∑ x∈D log pX ( x ; θ ) . In practice , Batch Normalization ( Ioffe & Szegedy , 2015 ) is applied as a bijection to outputs of successive coupling layers to stabilize the training of normalizing flows . This bijection implements the normalization procedure using a weighted moving average of the layer ’ s mean and standard deviation values , which has to be adapted to either training or inference regimes . The Real NVP approach can be generalized , resulting in Masked Autoregressive Flows ( Papamakarios et al. , 2017 ) ( MAF ) where the transformation layer is built as an autoregressive neural network in the sense that it takes in some input x ∈ RD and outputs y = ( y1 , . . . , yD ) with the requirement that this transformation is invertible and any output yi can not depend on input with dimension indices ≥ i , i.e . x≥i . The Jacobian of this transformation is triangular and thus the Jacobian determinant is tractable . Instead of using a RNN to share parameters across the D dimensions of x one avoids this sequential computation by using masking , giving the method its name . The inverse however , needed for generating samples , is sequential . By realizing that the scaling and translation function approximators don ’ t need to be invertible , it is straight-forward to implement conditioning of the PDF pX ( x|h ) on some additional information h ∈ RH : we concatenate h to the inputs of the scaling and translation function approximators of the coupling layers , i.e . s ( concat ( x1 : d , h ) ) and t ( concat ( x1 : d , h ) ) which are modified to map Rd+H 7→ RD−d . Another approach is to add a bias computed from h to every layer inside the s and t networks as proposed by Korshunova et al . ( 2018 ) . This does not change the log-determinant of the coupling layers given by ( 2 ) . More importantly for us , for sequential data , indexed by t , we can share parameters across the different conditioners ht by using RNNs or Attention in an autoregressive fashion . For discrete data the distribution has differential entropy of negative infinity , which leads to arbitrary high likelihood when training normalizing flow models , even on test data . To avoid this one can dequantize the data , often by adding Uniform [ 0 , 1 ) noise to integer-valued data . The log-likelihood of the resulting continuous model is then lower-bounded by the log-likelihood of the discrete one as shown in Theis et al . ( 2016 ) . 2.2 SELF-ATTENTION . The self-attention based Transformer ( Vaswani et al. , 2017 ) model has been used for sequence modeling with great success . The multi-head self-attention mechanism enables it to capture both longand short-term dependencies in time series data . Essentially , the Transformer takes in a sequence X = [ x1 , . . . , xT ] ᵀ ∈ RT×D , and the multi-head self-attention transforms this into H distinct query Qh = XW Q h , key Kh = XW K h and value Vh = XW V h matrices , where the W Q h , W K h , and W V h are learnable parameters . After these linear projections the scaled dot-product attention computes a sequence of vector outputs via : Oh = Attention ( Qh , Kh , Vh ) = softmax ( QhK ᵀ h√ dK ·M ) Vh , where a mask M can be applied to filter out right-ward attention ( or future information leakage ) by setting its upper-triangular elements to −∞ and we normalize by dK the dimension of the WKh matrices . Afterwards , all H outputs Oh are concatenated and linearly projected again . One typically uses the Transformer in an encoder-decoder setup , where some warm-up time series is passed through the encoder and the decoder can be used to learn and autoregressively generate outputs . 3 RELATED WORK . Related to this work are models that combine normalizing flows with sequential modeling in some way . Transformation Autoregressive Networks ( Oliva et al. , 2018 ) which model the density of a multi-variate variable x ∈ RD as D conditional distributions ΠDi=1pX ( xi|xi−1 , . . . , x1 ) , where the conditioning is given by a mixture model coming from the state of a RNN , and is then transformed via a bijection . The PixelSNAIL ( Chen et al. , 2018 ) method also models the joint as a product of conditional distributions , optionally with some global conditioning , via causal convolutions and self-attention ( Vaswani et al. , 2017 ) to capture long-term temporal dependencies . These methods are well suited to modeling high dimensional data like images , however their use in modeling the temporal development of data has only recently been explored for example in VideoFlow ( Kumar et al. , 2019 ) in which they model the distribution of the next video frame via a flow where the model outputs the parameters of the flow ’ s base distribution via a ConvNet , whereas our approach will be based on conditioning of the PDF as described above . Using RNNs for modeling either multivariate or temporal dynamics introduces sequential computational dependencies that are not amenable to parallelization . Despite this , RNNs have been shown to be very effective in modeling sequential dynamics . A recent work in this direction ( Hwang et al. , 2019 ) employs bipartite flows with RNNs for temporal conditioning to develop a conditional generative model of multivariate sequential data . The authors use a bidirectional training procedure to learn a generative model of observations that together with the temporal conditioning through a RNN , can also be conditioned on ( observed ) covariates that are modeled as additional conditioning variables in the latent space , which adds extra padding dimensions to the normalizing flow . The other aspect of related works deals with multivariate probabilistic time series methods which are able to model high dimensional data . The Gaussian Copula Process method ( Salinas et al. , 2019a ) is a RNN-based time series method with a Gaussian copula process output modeled using a low-rank covariance structure to reduce computational complexity and handle non-Gaussian marginal distributions . By using a low-rank approximation of the covariance matrix they obtain a computationally tractable method and are able to scale to multivariate dimensions in the thousands with state-of-the-art results . We will compare our model to this method in what follows .

Summary: The paper proposes a method to provide probabilistic forecasts of multivariate time-series taking dependencies between series into account even for large dimensions. The approach consists in using a normalizing flow to model the distribution of observations at a time-step condition on a state that can be obtained either with a RNN or a transformer. The motivation of using the normalizing flow is to be able to model various type of distributions without having to make specific hypothesis on the distribution which could hinder accuracy. Experiments are performed both on a synthetic task and on real-world datasets where accuracy is shown to outperform previous methods.