Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows

1 INTRODUCTION . Classical time series forecasting methods such as those in Hyndman & Athanasopoulos ( 2018 ) typically provide univariate forecasts and require hand-tuned features to model seasonality and other parameters . Time series models based on recurrent neural networks ( RNN ) , like LSTM ( Hochreiter & Schmidhuber , 1997 ) , have become popular methods due to their end-to-end training , the ease of incorporating exogenous covariates , and their automatic feature extraction abilities , which are the hallmarks of deep learning . Forecasting outputs can either be points or probability distributions , in which case the forecasts typically come with uncertainty bounds . The problem of modeling uncertainties in time series forecasting is of vital importance for assessing how much to trust the predictions for downstream tasks , such as anomaly detection or ( business ) decision making . Without probabilistic modeling , the importance of the forecast in regions of low noise ( small variance around a mean value ) versus a scenario with high noise can not be distinguished . Hence , point estimation models ignore risk stemming from this noise , which would be of particular importance in some contexts such as making ( business ) decisions . Finally , individual time series , in many cases , are statistically dependent on each other , and models need the capacity to adapt to this in order to improve forecast accuracy ( Tsay , 2014 ) . For example , to model the demand for a retail article , it is important to not only model its sales dependent on its own past sales , but also to take into account the effect of interacting articles , which can lead to cannibalization effects in the case of article competition . As another example , consider traffic flow in a network of streets as measured by occupancy sensors . A disruption on one particular street will also ripple to occupancy sensors of nearby streets—a univariate model would arguably not be able to account for these effects . In this work , we propose end-to-end trainable autoregressive deep learning architectures for probabilistic forecasting that explicitly models multivariate time series and their temporal dynamics by employing a normalizing flow , like the Masked Autoregressive Flow ( Papamakarios et al. , 2017 ) or Real NVP ( Dinh et al. , 2017 ) . These models are able to scale to thousands of interacting time series , we show that they are able to learn ground-truth dependency structure on toy data and we establish new state-of-the-art results on diverse real world data sets by comparing to competitive baselines . Additionally , these methods adapt to a broad class of underlying data distribution on account of using a normalizing flow and our Transformer based model is highly efficient due to the parallel nature of attention layers while training . The paper first provides some background context in Section 2 . We cover related work in Section 3 . Section 4 introduces our model and the experiments are detailed in Section 5 . We conclude with some discussion in Section 6 . The Appendix contains details of the datasets , additional metrics and exploratory plots of forecast intervals as well as details of our model . 2 BACKGROUND . 2.1 DENSITY ESTIMATION VIA NORMALIZING FLOWS . Normalizing flows ( Tabak & Turner , 2013 ; Papamakarios et al. , 2019 ) are mappings from RD to RD such that densities pX on the input space X = RD are transformed into some simple distribution pZ ( e.g . an isotropic Gaussian ) on the space Z = RD . These mappings , f : X 7→ Z , are composed of a sequence of bijections or invertible functions . Due to the change of variables formula we can express pX ( x ) by pX ( x ) = pZ ( z ) ∣∣∣∣det ( ∂f ( x ) ∂x ) ∣∣∣∣ , where ∂f ( x ) /∂x is the Jacobian of f at x. Normalizing flows have the property that the inverse x = f−1 ( z ) is easy to evaluate and computing the Jacobian determinant takes O ( D ) time . The bijection introduced by Real NVP ( Dinh et al. , 2017 ) called the coupling layer satisfies the above two properties . It leaves part of its inputs unchanged and transforms the other part via functions of the un-transformed variables ( with superscript denoting the coordinate indices ) { y1 : d = x1 : d yd+1 : D = xd+1 : D exp ( s ( x1 : d ) ) + t ( x1 : d ) , where is an element wise product , s ( ) is a scaling and t ( ) a translation function from Rd 7→ RD−d , given by neural networks . To model a nonlinear density map f ( x ) , a number of coupling layers which map X 7→ Y1 7→ · · · 7→ YK−1 7→ Z are composed together all the while alternating the dimensions which are unchanged and transformed . Via the change of variables formula the probability density function ( PDF ) of the flow given a data point can be written as log pX ( x ) = log pZ ( z ) + log |det ( ∂z/∂x ) | = log pZ ( z ) + K∑ i=1 log |det ( ∂yi/∂yi−1 ) | . ( 1 ) Note that the Jacobian for the Real NVP is a block-triangular matrix and thus the log-determinant of each map simply becomes log |det ( ∂yi/∂yi−1 ) | = log | exp ( sum ( si ( y1 : di−1 ) ) | , ( 2 ) where sum ( ) is the sum over all the vector elements . This model , parameterized by the weights of the scaling and translation neural networks θ , is then trained via stochastic gradient descent ( SGD ) on training data points where for each batch D we maximize the average log likelihood ( 1 ) given by L = 1 |D| ∑ x∈D log pX ( x ; θ ) . In practice , Batch Normalization ( Ioffe & Szegedy , 2015 ) is applied as a bijection to outputs of successive coupling layers to stabilize the training of normalizing flows . This bijection implements the normalization procedure using a weighted moving average of the layer ’ s mean and standard deviation values , which has to be adapted to either training or inference regimes . The Real NVP approach can be generalized , resulting in Masked Autoregressive Flows ( Papamakarios et al. , 2017 ) ( MAF ) where the transformation layer is built as an autoregressive neural network in the sense that it takes in some input x ∈ RD and outputs y = ( y1 , . . . , yD ) with the requirement that this transformation is invertible and any output yi can not depend on input with dimension indices ≥ i , i.e . x≥i . The Jacobian of this transformation is triangular and thus the Jacobian determinant is tractable . Instead of using a RNN to share parameters across the D dimensions of x one avoids this sequential computation by using masking , giving the method its name . The inverse however , needed for generating samples , is sequential . By realizing that the scaling and translation function approximators don ’ t need to be invertible , it is straight-forward to implement conditioning of the PDF pX ( x|h ) on some additional information h ∈ RH : we concatenate h to the inputs of the scaling and translation function approximators of the coupling layers , i.e . s ( concat ( x1 : d , h ) ) and t ( concat ( x1 : d , h ) ) which are modified to map Rd+H 7→ RD−d . Another approach is to add a bias computed from h to every layer inside the s and t networks as proposed by Korshunova et al . ( 2018 ) . This does not change the log-determinant of the coupling layers given by ( 2 ) . More importantly for us , for sequential data , indexed by t , we can share parameters across the different conditioners ht by using RNNs or Attention in an autoregressive fashion . For discrete data the distribution has differential entropy of negative infinity , which leads to arbitrary high likelihood when training normalizing flow models , even on test data . To avoid this one can dequantize the data , often by adding Uniform [ 0 , 1 ) noise to integer-valued data . The log-likelihood of the resulting continuous model is then lower-bounded by the log-likelihood of the discrete one as shown in Theis et al . ( 2016 ) . 2.2 SELF-ATTENTION . The self-attention based Transformer ( Vaswani et al. , 2017 ) model has been used for sequence modeling with great success . The multi-head self-attention mechanism enables it to capture both longand short-term dependencies in time series data . Essentially , the Transformer takes in a sequence X = [ x1 , . . . , xT ] ᵀ ∈ RT×D , and the multi-head self-attention transforms this into H distinct query Qh = XW Q h , key Kh = XW K h and value Vh = XW V h matrices , where the W Q h , W K h , and W V h are learnable parameters . After these linear projections the scaled dot-product attention computes a sequence of vector outputs via : Oh = Attention ( Qh , Kh , Vh ) = softmax ( QhK ᵀ h√ dK ·M ) Vh , where a mask M can be applied to filter out right-ward attention ( or future information leakage ) by setting its upper-triangular elements to −∞ and we normalize by dK the dimension of the WKh matrices . Afterwards , all H outputs Oh are concatenated and linearly projected again . One typically uses the Transformer in an encoder-decoder setup , where some warm-up time series is passed through the encoder and the decoder can be used to learn and autoregressively generate outputs . 3 RELATED WORK . Related to this work are models that combine normalizing flows with sequential modeling in some way . Transformation Autoregressive Networks ( Oliva et al. , 2018 ) which model the density of a multi-variate variable x ∈ RD as D conditional distributions ΠDi=1pX ( xi|xi−1 , . . . , x1 ) , where the conditioning is given by a mixture model coming from the state of a RNN , and is then transformed via a bijection . The PixelSNAIL ( Chen et al. , 2018 ) method also models the joint as a product of conditional distributions , optionally with some global conditioning , via causal convolutions and self-attention ( Vaswani et al. , 2017 ) to capture long-term temporal dependencies . These methods are well suited to modeling high dimensional data like images , however their use in modeling the temporal development of data has only recently been explored for example in VideoFlow ( Kumar et al. , 2019 ) in which they model the distribution of the next video frame via a flow where the model outputs the parameters of the flow ’ s base distribution via a ConvNet , whereas our approach will be based on conditioning of the PDF as described above . Using RNNs for modeling either multivariate or temporal dynamics introduces sequential computational dependencies that are not amenable to parallelization . Despite this , RNNs have been shown to be very effective in modeling sequential dynamics . A recent work in this direction ( Hwang et al. , 2019 ) employs bipartite flows with RNNs for temporal conditioning to develop a conditional generative model of multivariate sequential data . The authors use a bidirectional training procedure to learn a generative model of observations that together with the temporal conditioning through a RNN , can also be conditioned on ( observed ) covariates that are modeled as additional conditioning variables in the latent space , which adds extra padding dimensions to the normalizing flow . The other aspect of related works deals with multivariate probabilistic time series methods which are able to model high dimensional data . The Gaussian Copula Process method ( Salinas et al. , 2019a ) is a RNN-based time series method with a Gaussian copula process output modeled using a low-rank covariance structure to reduce computational complexity and handle non-Gaussian marginal distributions . By using a low-rank approximation of the covariance matrix they obtain a computationally tractable method and are able to scale to multivariate dimensions in the thousands with state-of-the-art results . We will compare our model to this method in what follows .

Summary: This work explores combining an RNN and a neural density estimator for forecasting in multivariate time series. RNN is stacked with a density estimator, MAF for best results, to forecast density of a multivariate time series at future time steps. In addition, variations of the architecture with attention and other density estimators are examined. The architecture, RNN+MAF and variants, is evaluated by CRPS score on several datasets.