Hierarchically Regularized Deep Forecasting

1 INTRODUCTION . Multivariate time series forecasting is a key problem in many domains such as retail demand forecasting ( Böse et al. , 2017 ) , financial predictions ( Zhou et al. , 2020 ) , power grid optimization ( Hyndman & Fan , 2009 ) , road traffic modeling ( Li et al. , 2017 ) , and online ads optimization ( Cui et al. , 2011 ) . In many of these setting , the problem involves simultaneously forecasting a large number of possibly correlated time series for various downstream applications . In the retail domain , the time series may capture sales of items in a product inventory , and in power grids , the time series may correspond to energy consumption in a household . Often , these time series are arranged in a natural multi-level hierarchy - for example in retail forecasting , items are grouped into subcategories and categories , and arranged in a product taxonomy . In the case of power consumption forecasting , individual households are grouped into neighborhoods , counties , and cities . The hierarchical structure among the time series can usually be represented as a tree , with the leaf nodes corresponding to time series at the finest granularity , and the edges representing parent-child relationships . Figure 1 illustrates a typical hierarchy in the retail forecasting domain for time series of product sales . In such settings , it is often required to obtain good forecasts , not just for the leaf level time-series ( fine grained forecasts ) , but also for the aggregated time-series corresponding to higher level nodes ( coarse gained forecasts ) . Furthermore , for interpretability and business decision making purposes , it is often desirable to obtain predictions that are roughly coherent or consistent ( Hyndman et al. , 2011 ) with respect to the hierarchy tree . This means that the predictions for each parent time-series is equal to the sum of the predictions for its children time-series . More importantly , incorporating coherence constraints in a hierarchical forecasting model captures the natural inductive bias in most hierarchical datasets , where the ground truth parent and children time series indeed adhere to additive constraints . For example , total sales of a product category is equal to the sum of sales of all items in that category . Some standard approaches for hierarchical forecasting include bottom-up aggregation , or reconciliation-based approaches . Bottom-Up aggregation involves training a model to obtain predictions for the leaf nodes , and then aggregate up along the hierarchy tree to obtain predictions for higher-level nodes . Reconciliation methods ( Ben Taieb & Koo , 2019 ; Taieb et al. , 2017 ; Van Erven & Cugliari , 2015 ; Hyndman et al. , 2016 ; Wickramasuriya et al. , 2015 ; 2020 ; Panagiotelis et al. , 2020 ) make use of a trained model to obtain predictions for all nodes of the tree , and then , in a separate post-processing phase , reconcile ( or modify ) them using various optimization formulations to obtain coherent predictions . Both of these approaches suffer from shortcomings in term of either aggregating noise as one moves to higher level predictions ( bottom-up aggregation ) , or not jointly optimizing the forecasting predictions along with the coherence constraints ( for instance , reconciliation ) . At the same time , there have been several recent advances on using Deep Neural Network models for multivariate forecasting , including Recurrent Neural Network ( RNN ) , Convolutional Neural Network ( CNN ) architectures ( Salinas et al. , 2020 ; Oreshkin et al. , 2019 ; Rangapuram et al. , 2018 ; Benidis et al. , 2020 ) , and models designed for multivariate time series based on dimensionality reduction techniques ( Sen et al. , 2019 ; Wang et al. , 2019 ; Nguyen & Quanz , 2021 ; Salinas et al. , 2019 ; Rasul et al. , 2020 ; de Bézenac et al. , 2020 ) , that have been shown to outperform classical time-series models such as autoregressive and exponential smoothing models ( McKenzie , 1984 ; Hyndman et al. , 2008 ; Hyndman & Athanasopoulos , 2018 ) , especially for large datasets . However , most of these approaches do not explicitly address the question of how to model the hierarchical relationships in the dataset . Deep forecasting models based on Graph Neural Networks ( GNN ) ( Bai et al. , 2020 ; Cao et al. , 2020 ; Yu et al. , 2017 ; Li et al. , 2017 ; Wu et al. , 2020 ) do offer a general framework for learning on graph-structured data . However it is well known ( Bojchevski et al. , 2020 ) that GNNs are hard to scale for learning on graphs with a very large number of nodes , which in real-world settings such as retail forecasting , could involve hundreds of thousands of time series . More importantly , a desirable practical feature for multi-variate forecasting models is to let the prediction of future values for a particular time series only require as input historical data from that time series ( along with covariates ) , without requiring access to historical data from all other time series in the hierarchy . This allows for scalable training and inference of such models using mini-batch gradient descent , without requiring each batch to contain all the time series in the hierarchy . This is often not possible for GNN-based forecasting models , which require batch sizes of the order of the number of time series . Problem Statement : Based on the above motivations , our goal is to design a hierarchical forecasting model with the following requirements : 1 ) The model can be trained using a single-stage pipeline on all the time series data , without any separate post-processing , 2 ) The model captures the additive coherence constraints along the edges of the hierarchy , 3 ) The model is efficiently trainable on large datasets , without requiring , for instance , batch sizes that scale with the number of time series . We propose a principled methodology to address all these above requirements for hierarchical forecasting . Our model comprises of two components , both of which can support coherence constraints . The first component is purely a function of the historical values of a time series , without distinguishing between the individual time series themselves in any other way . Coherence constraints on such a model correspond to imposing an additivity property on the prediction function - which constrains the model to be a linear autoregressive model . However , crucially , our model uses timevarying autoregressive coefficients that can themselves be nonlinear functions of the timestamp and other global features ( linear versions of time-varying AR have been historically used to deal with non-stationary signals ( Sharman & Friedlander , 1984 ) ) . We will refer to this component as the time-varying autoregressive model . The second component focuses on modeling the global temporal patterns in the dataset through identifying a small set of temporal global basis functions . The basis time-series , when combined in different ways , can express the individual dynamics of each time series . In our model , the basis time-series are encoded in a trained seq-2-seq model ( Sutskever et al. , 2014 ) model in a functional form . Each time series is then associated with a learned embedding vector that specifies the weights for decomposition along these basis functions . Predicting a time series into the future using this model then just involves extrapolating the global basis functions and combining them using its weight vector , without explicitly using the past values of that time series . The coherence constraints therefore only impose constraints on the embedding vectors of each time series , which can be easily modeled by a hierarchical regularization function . We call this component a basis decomposition model . As we will see , this part of the model is only approximately coherent unless the embedding constraints hold exactly . In particular , in this paper , we focus on improving model accuracy rather than preserving exact coherency . In Section A.2 , we also provide theoretical justification for how such hierarchical regularization using basis decomposition results in improved prediction accuracy . We experimentally evaluate our model on multiple publicly available hierarchical forecasting datasets . We compare our approach to state-of-the-art ( non-hierarchical ) deep forecasting models , GNN-based models and reconciliation models , and show that our approach can obtain consistently more accurate predictions at all levels of the hierarchy tree . 2 RELATED WORK ON DEEP HIERARCHICAL MODELS . In addition to the works referenced in the previous section , we now discuss a few papers that are more relevant to our approach . Specifically , we discuss some recent deep hierarchical forecasting methods that do not require a post-processing reconciliation step . Hierarchical forecasting methods can be roughly divided into two categories : point forecasters and probabilistic forecasters . Mishchenko et al . ( 2019 ) propose a point-forecasting approach which imposes coherency on a base model via ` 2 regularization on the predictions . Gleason ( 2020 ) extend the idea further to impose the hierarchy on an embedding space rather than the predictions directly . SHARQ ( Han et al. , 2021 ) follows a similar ` 2 regularization based approach as Mishchenko et al . ( 2019 ) , and also extends the idea to probabilistic forecasting . Their model is trained separately for each of the hierarchical levels starting from the leaf level , thus requiring a separate prediction model for each level . Probabilistic forecasting methods include , Hier-E2E ( Rangapuram et al. , 2021 ) which produces perfectly coherent forecasts by using a projection operation on base predictions from a DeepVAR model ( Salinas et al. , 2019 ) . It requires the whole hierarchy of time series to be fed as input to the model leading to a large number of parameters , and hence does not scale well to large hierarchies . Yanchenko et al . ( 2021 ) take a fully Bayesian approach by modelling the hierarchy using conditional distributions . 3 PROBLEM SETTING . We are given a set of N coherent time series of length T , arranged in a pre-defined hierarchy consisting of N nodes . At time step t , the time series data can be represented as a vector yt ∈ RN denoting the time series values of all N nodes . We compactly denote the set of time series for all T steps as a matrix Y = [ y1 , · · · , yT ] > ∈ RT×N . Also define y ( i ) as the ith column of the matrix Y denoting all time steps of the i th time series , and y ( i ) t as the t th value of the i th time series . We compactly denote the H-step history of Y by YH = [ yt−H , · · · , yt−1 ] > ∈ RH×N and the H-step history of y ( i ) by y ( i ) H = [ y ( i ) t−H , · · · , y ( i ) t−1 ] ∈ RH . Similarly define the F -step future of Y as YF = [ yt , · · · , yt+F−1 ] > ∈ RF×N . We use the ·̂ notation to denote predicted values , for example ŶF , ŷF and ŷt . Time series forecasts can often be improved by using features as input to the model along with historical time series . The features often evolve with time , for example , categorical features such as type of holiday , or continuous features such as time of the day . We denote the matrix of such features byX ∈ RT×D , where the t th row denotes the D-dimensional feature vector at the t time step . For simplicity , we assume that the features are global , meaning that they are shared across all time series . We similarly defineXH andXF as above . Hierarchically Coherent Time Series . We assume that the time series data are coherent , that is , they satisfy the sum constraints over the hierarchy . The time series at each node of the hierarchy is the equal to the sum of the time series of its children , or equivalently , equal to the sum of the leaf time series of the sub-tree rooted at that node . Figure 1 shows an example of a sub-tree rooted at a node . As a result of aggregation , the data can have widely varying scales with the values at higher level nodes being magnitudes higher than the leaf level nodes . It is well known that neural network training is more efficient if the data are similarly scaled . Hence , in this paper , we work with rescaled time series data . The time series at each node is downscaled by the number of leaves in the sub-tree rooted at the node , so that now they satisfy mean constraints rather than sum constraints described above . Denote by L ( p ) , the set of leaf nodes of the sub-tree rooted at p. Hierarchically coherent data satisfy the following data mean property , y ( p ) = 1 |L ( p ) | ∑ i∈L ( p ) y ( i ) ( Data Mean Property ) . ( 1 )

Summary: The paper considers point forecasting of hierarchical time series, i.e. multivariate time series with hierarchical aggregation constraints. The authors propose a new approach based on decomposing the series along a global set of basis time series where (approximate) hierarchical constraints are applied on the coefficients of the basis decomposition. Forecasts are produced using a dynamic linear autoregressive model. Compared to existing state-of-the-art hierarchical models, the proposed approach improved overall performance on forecasts at different levels of the hierarchy on several public datasets.