Hierarchically Regularized Deep Forecasting

1 INTRODUCTION . Multivariate time series forecasting is a key problem in many domains such as retail demand forecasting ( Böse et al. , 2017 ) , financial predictions ( Zhou et al. , 2020 ) , power grid optimization ( Hyndman & Fan , 2009 ) , road traffic modeling ( Li et al. , 2017 ) , and online ads optimization ( Cui et al. , 2011 ) . In many of these setting , the problem involves simultaneously forecasting a large number of possibly correlated time series for various downstream applications . In the retail domain , the time series may capture sales of items in a product inventory , and in power grids , the time series may correspond to energy consumption in a household . Often , these time series are arranged in a natural multi-level hierarchy - for example in retail forecasting , items are grouped into subcategories and categories , and arranged in a product taxonomy . In the case of power consumption forecasting , individual households are grouped into neighborhoods , counties , and cities . The hierarchical structure among the time series can usually be represented as a tree , with the leaf nodes corresponding to time series at the finest granularity , and the edges representing parent-child relationships . Figure 1 illustrates a typical hierarchy in the retail forecasting domain for time series of product sales . In such settings , it is often required to obtain good forecasts , not just for the leaf level time-series ( fine grained forecasts ) , but also for the aggregated time-series corresponding to higher level nodes ( coarse gained forecasts ) . Furthermore , for interpretability and business decision making purposes , it is often desirable to obtain predictions that are roughly coherent or consistent ( Hyndman et al. , 2011 ) with respect to the hierarchy tree . This means that the predictions for each parent time-series is equal to the sum of the predictions for its children time-series . More importantly , incorporating coherence constraints in a hierarchical forecasting model captures the natural inductive bias in most hierarchical datasets , where the ground truth parent and children time series indeed adhere to additive constraints . For example , total sales of a product category is equal to the sum of sales of all items in that category . Some standard approaches for hierarchical forecasting include bottom-up aggregation , or reconciliation-based approaches . Bottom-Up aggregation involves training a model to obtain predictions for the leaf nodes , and then aggregate up along the hierarchy tree to obtain predictions for higher-level nodes . Reconciliation methods ( Ben Taieb & Koo , 2019 ; Taieb et al. , 2017 ; Van Erven & Cugliari , 2015 ; Hyndman et al. , 2016 ; Wickramasuriya et al. , 2015 ; 2020 ; Panagiotelis et al. , 2020 ) make use of a trained model to obtain predictions for all nodes of the tree , and then , in a separate post-processing phase , reconcile ( or modify ) them using various optimization formulations to obtain coherent predictions . Both of these approaches suffer from shortcomings in term of either aggregating noise as one moves to higher level predictions ( bottom-up aggregation ) , or not jointly optimizing the forecasting predictions along with the coherence constraints ( for instance , reconciliation ) . At the same time , there have been several recent advances on using Deep Neural Network models for multivariate forecasting , including Recurrent Neural Network ( RNN ) , Convolutional Neural Network ( CNN ) architectures ( Salinas et al. , 2020 ; Oreshkin et al. , 2019 ; Rangapuram et al. , 2018 ; Benidis et al. , 2020 ) , and models designed for multivariate time series based on dimensionality reduction techniques ( Sen et al. , 2019 ; Wang et al. , 2019 ; Nguyen & Quanz , 2021 ; Salinas et al. , 2019 ; Rasul et al. , 2020 ; de Bézenac et al. , 2020 ) , that have been shown to outperform classical time-series models such as autoregressive and exponential smoothing models ( McKenzie , 1984 ; Hyndman et al. , 2008 ; Hyndman & Athanasopoulos , 2018 ) , especially for large datasets . However , most of these approaches do not explicitly address the question of how to model the hierarchical relationships in the dataset . Deep forecasting models based on Graph Neural Networks ( GNN ) ( Bai et al. , 2020 ; Cao et al. , 2020 ; Yu et al. , 2017 ; Li et al. , 2017 ; Wu et al. , 2020 ) do offer a general framework for learning on graph-structured data . However it is well known ( Bojchevski et al. , 2020 ) that GNNs are hard to scale for learning on graphs with a very large number of nodes , which in real-world settings such as retail forecasting , could involve hundreds of thousands of time series . More importantly , a desirable practical feature for multi-variate forecasting models is to let the prediction of future values for a particular time series only require as input historical data from that time series ( along with covariates ) , without requiring access to historical data from all other time series in the hierarchy . This allows for scalable training and inference of such models using mini-batch gradient descent , without requiring each batch to contain all the time series in the hierarchy . This is often not possible for GNN-based forecasting models , which require batch sizes of the order of the number of time series . Problem Statement : Based on the above motivations , our goal is to design a hierarchical forecasting model with the following requirements : 1 ) The model can be trained using a single-stage pipeline on all the time series data , without any separate post-processing , 2 ) The model captures the additive coherence constraints along the edges of the hierarchy , 3 ) The model is efficiently trainable on large datasets , without requiring , for instance , batch sizes that scale with the number of time series . We propose a principled methodology to address all these above requirements for hierarchical forecasting . Our model comprises of two components , both of which can support coherence constraints . The first component is purely a function of the historical values of a time series , without distinguishing between the individual time series themselves in any other way . Coherence constraints on such a model correspond to imposing an additivity property on the prediction function - which constrains the model to be a linear autoregressive model . However , crucially , our model uses timevarying autoregressive coefficients that can themselves be nonlinear functions of the timestamp and other global features ( linear versions of time-varying AR have been historically used to deal with non-stationary signals ( Sharman & Friedlander , 1984 ) ) . We will refer to this component as the time-varying autoregressive model . The second component focuses on modeling the global temporal patterns in the dataset through identifying a small set of temporal global basis functions . The basis time-series , when combined in different ways , can express the individual dynamics of each time series . In our model , the basis time-series are encoded in a trained seq-2-seq model ( Sutskever et al. , 2014 ) model in a functional form . Each time series is then associated with a learned embedding vector that specifies the weights for decomposition along these basis functions . Predicting a time series into the future using this model then just involves extrapolating the global basis functions and combining them using its weight vector , without explicitly using the past values of that time series . The coherence constraints therefore only impose constraints on the embedding vectors of each time series , which can be easily modeled by a hierarchical regularization function . We call this component a basis decomposition model . As we will see , this part of the model is only approximately coherent unless the embedding constraints hold exactly . In particular , in this paper , we focus on improving model accuracy rather than preserving exact coherency . In Section A.2 , we also provide theoretical justification for how such hierarchical regularization using basis decomposition results in improved prediction accuracy . We experimentally evaluate our model on multiple publicly available hierarchical forecasting datasets . We compare our approach to state-of-the-art ( non-hierarchical ) deep forecasting models , GNN-based models and reconciliation models , and show that our approach can obtain consistently more accurate predictions at all levels of the hierarchy tree . 2 RELATED WORK ON DEEP HIERARCHICAL MODELS . In addition to the works referenced in the previous section , we now discuss a few papers that are more relevant to our approach . Specifically , we discuss some recent deep hierarchical forecasting methods that do not require a post-processing reconciliation step . Hierarchical forecasting methods can be roughly divided into two categories : point forecasters and probabilistic forecasters . Mishchenko et al . ( 2019 ) propose a point-forecasting approach which imposes coherency on a base model via ` 2 regularization on the predictions . Gleason ( 2020 ) extend the idea further to impose the hierarchy on an embedding space rather than the predictions directly . SHARQ ( Han et al. , 2021 ) follows a similar ` 2 regularization based approach as Mishchenko et al . ( 2019 ) , and also extends the idea to probabilistic forecasting . Their model is trained separately for each of the hierarchical levels starting from the leaf level , thus requiring a separate prediction model for each level . Probabilistic forecasting methods include , Hier-E2E ( Rangapuram et al. , 2021 ) which produces perfectly coherent forecasts by using a projection operation on base predictions from a DeepVAR model ( Salinas et al. , 2019 ) . It requires the whole hierarchy of time series to be fed as input to the model leading to a large number of parameters , and hence does not scale well to large hierarchies . Yanchenko et al . ( 2021 ) take a fully Bayesian approach by modelling the hierarchy using conditional distributions . 3 PROBLEM SETTING . We are given a set of N coherent time series of length T , arranged in a pre-defined hierarchy consisting of N nodes . At time step t , the time series data can be represented as a vector yt ∈ RN denoting the time series values of all N nodes . We compactly denote the set of time series for all T steps as a matrix Y = [ y1 , · · · , yT ] > ∈ RT×N . Also define y ( i ) as the ith column of the matrix Y denoting all time steps of the i th time series , and y ( i ) t as the t th value of the i th time series . We compactly denote the H-step history of Y by YH = [ yt−H , · · · , yt−1 ] > ∈ RH×N and the H-step history of y ( i ) by y ( i ) H = [ y ( i ) t−H , · · · , y ( i ) t−1 ] ∈ RH . Similarly define the F -step future of Y as YF = [ yt , · · · , yt+F−1 ] > ∈ RF×N . We use the ·̂ notation to denote predicted values , for example ŶF , ŷF and ŷt . Time series forecasts can often be improved by using features as input to the model along with historical time series . The features often evolve with time , for example , categorical features such as type of holiday , or continuous features such as time of the day . We denote the matrix of such features byX ∈ RT×D , where the t th row denotes the D-dimensional feature vector at the t time step . For simplicity , we assume that the features are global , meaning that they are shared across all time series . We similarly defineXH andXF as above . Hierarchically Coherent Time Series . We assume that the time series data are coherent , that is , they satisfy the sum constraints over the hierarchy . The time series at each node of the hierarchy is the equal to the sum of the time series of its children , or equivalently , equal to the sum of the leaf time series of the sub-tree rooted at that node . Figure 1 shows an example of a sub-tree rooted at a node . As a result of aggregation , the data can have widely varying scales with the values at higher level nodes being magnitudes higher than the leaf level nodes . It is well known that neural network training is more efficient if the data are similarly scaled . Hence , in this paper , we work with rescaled time series data . The time series at each node is downscaled by the number of leaves in the sub-tree rooted at the node , so that now they satisfy mean constraints rather than sum constraints described above . Denote by L ( p ) , the set of leaf nodes of the sub-tree rooted at p. Hierarchically coherent data satisfy the following data mean property , y ( p ) = 1 |L ( p ) | ∑ i∈L ( p ) y ( i ) ( Data Mean Property ) . ( 1 )

Summary: The paper introduces a method for hierarchical time series forecasting. The problem setting is: given historical hierarchical univariate time series  data and given historical and future features (like holidays, etc..), try to  predict future values for all time series, while keeping the coherence  constraints of the hierarchy. The forecasting is broken down into an autoregressive part with shared  parameters for all time series, and a basis decomposition with different  weights for each time series (regularized by the hierarchy structure). The method is then being compared to different baseline methods on three  datasets.