When Rigid Coherency Hurts: Distributional Coherency Regularization for Probabilistic Hierarchical Time Series Forecasting
Authors: Harshavardhan Kamarthi
Lingkai Kong
Alexander Rodríguez
Chao Zhang
B. Aditya Prakash
Venue: Submitted to ICLR 2023 (2023)
Keywords: Hierarchical Forecasting
Time Series Forecasting
Deep Probabilistic Models

Abstract: Probabilistic hierarchical time-series forecasting is an important variant of time-series forecasting, where the goal is to model and forecast multivariate time-series that have hierarchical relations. Previous works all assume rigid consistency over the given hierarchies and do not adapt to real-world data that show deviation from this assumption. Moreover, recent state-of-art neural probabilistic methods also impose hierarchical relations on point predictions and samples of distribution. This does not account for full forecast distributions being coherent with the hierarchy and leads to poorly calibrated forecasts. We close both these gaps and propose PROFHIT, a probabilistic hierarchical forecasting model that jointly models forecast distributions over the entire hierarchy. PROFHIT (1) uses a flexible probabilistic Bayesian approach and (2) introduces soft distributional coherency regularization that enables end-to-end learning of the entire forecast distribution leveraging information from the underlying hierarchy. This enables robust and calibrated forecasts as well as adaptation to real-life data with varied hierarchical consistency. PROFHIT provides 41-88% better performance in accuracy and 23-33% better calibration over a wide range of dataset consistency. Furthermore, PROFHIT can robustly provide reliable forecasts even if up to 10% of input time-series data is missing, whereas other methods’ performance severely degrade by over 70%.

WHEN RIGID COHERENCY HURTS: DISTRIBUTIONAL COHERENCY REGULARIZATION FOR PROBABILISTIC HIERARCHICAL TIME SERIES FORECASTING
Anonymous authors Paper under double-blind review
ABSTRACT
Probabilistic hierarchical time-series forecasting an important variant of time- series forecasting, where the goal is to model and forecast multivariate time-series that have hierarchical relations. Previous works all assume rigid consistency over the given hierarchies and do not adapt to real-world data that show deviation from this assumption. Moreover, recent state-of-arr neural probabilistic methods also impose hierarchical relations on point predictions and samples of predictive distribution. This does not account for full forecast distributions being coherent with the hierarchy and leads to poorly calibrated forecasts. We close both these gaps and propose PROFHIT, a probabilistic hierarchical forecasting model that jointly models forecast distributions over the entire hierarchy. PROFHIT (1) uses a flexible probabilistic Bayesian approach and (2) introduces soft distributional coherency regularization that enables end-to-ent learning of the entire forecast distribution leveraging information from the underlying hierarchy. This enables robust and calibrated forecasts well as adaptation to real-life data with varied hierarchical consistency. PROFHIT provides 41-88% better performance in accuracy and 23-33% better calibration over wide range of dataset consistency. Furthermore, PROFHIT can robustly provide reliable forecasts even if to 10% of input time- series data missing, whereas other methods' performance severely degrade by over 70%.
INTRODUCTION
Time-series forecasting is an important problem that impacts decision-making in a wide range of applications. In many real-world situations, the time-series have inherent hierarchical relations and structures. Examples include forecasting time-series of employment (Taieb et al.) 2017) measured at different geographical scales; epidemic forecasting (Reich et al. 2019) county, state and country, etc. Given time-series dataset with underlying hierarchical relations, the goal of Hierarchical Time-series Forecasting (HTSF) to generate accurate forecast for all time-series leveraging the hierarchical relations between time-series (Hyndman al. 2011).
Most previous methods do not provide well-calibrated forecasts for both so-called "strong" " and "weakly" consistent datasets. Previous HTSF methods assume that the time-series values of datasets strictly satisfy the underlying hierarchical constraints and impose rigid coherency on generated forecasts i.c., forecasts strictly satisfy the hierarchical relations of dataset. These methods datasets generated (Taieb et al. 2017) by first collecting data for time-series of the leaf level nodes and deriving time-series for higher-level nodes. We call such data strongly consistent. For example, classical HTSF methods (Hyndman & Athanasopoulos (2088) use bottom-up or top-down approach where all time-series levels are derived using the aggregation function governing the hierarchy. In contrast, many real-world datasets are weakly consistent, i.c., they do not follow the strict constraints of the hierarchy Such data have an underlying data generation process that may follow hierarchical set of constraints but may contain some deviations. These deviations can be caused by factors such as measurement or reporting error, asynchrony in data aggregation and revision pipeline, etc, as frequently observed in
model
single level of hierarchy
modeled independently and the values of other
Note that describe consistency over dataset and coherency over model forecasts.

---

epidemic forecasting (Adhikari et al. 2019). Most fatte-of-theaard HTSF methods designed for strongly consistent datasets and impose rigid coherency constraints - they thus may not adapt to such deviations and can provide poor forecasts for weakly consistent datasets.
Moreover, previous methods do not focus on pro- viding calibrated forecasts with precise uncertainty measures. Traditional methods focus on point predic- tions only. Recent methods like MINT (Wickrama- suriya et al. 2019). ERM (Ben Taieb & Koo 2019)) and PEMBU (Taieb al., 2017) refine raw indepen- dent forecast distribution as a post-processee
Raw
Puod
Plani
Pian)
step.
This does not enable the models generating the raw forecasts to leverage underlying hierarchical relations across time-series. End-to-end learning neural meth- ods directly leverage hierarchical relations as part of the model architecture or learning algorithm like
Σ 6
Maa
Hlerarchical Relations
Distributional Coherency
Figure 1: Regularizing forecasts using Distri- butional Coherency.
HIERE2E [Rangapram et al., 2021) and SHARQ
(Han et al. 2021). They do this and usually outperform post-processing methods by imposing hierarchical constraints on the mean fixed quantiles of the forecast distributions. However, these methods do not enforce hierarchical coherency on the full distributions. Therefore, the forecasts may not be well-callbrated (Kuleshov al., 2018 i.e., they produce unreliable prediction intervals that may not match observed probabilities from ground truth (Fisch et al., 2022).
In this work, we fill this gap of learning well-calibrated and accurate forecasts for both strong and weakly consistent datasets leverag- ing underlying hierarchical relations. We propose PROFHIT (Probabilistic Robust Forecasting for Hierarchical Time-series), a neural probabilistic HTSF
Table 1: Comparison of PROFHIT with of the- methods.
2-step methods
MINT erm
PEMBU
HIFRE2K
SHARQ
PROFHIT
This paper)
Probabilistic
Forecasts
x
/
Strong & Weak
Consistency
x
V
Distributinal
Coherency
x
x
x
x
End-to-end Learning
X
4 <
Robust to missing data
X X
method that provides an end-to-end Bayesian approach to model the distributions of forecasts of all time-series together (see Table [ for comparison). Specifically, we introduce a novel Soft Distributional Coherency Regularization (SDCR) to tackle the challenge. First, SDCR enables PROFHIT to leverage hierarchical relations over entire forecast distributions to generate calibrated forecast distríbutions by encouraging forecast distribution of any parent node be similar to aggregation of children nodes' forecast distribution (Figure []. Second, since SDCR is soft constraint, our model trained to adapt to datasets with varying hierarchical consistency that allows the model to trade-off coherency for better accuracy and calibration on weakly consistent datasets. Our main contributions are:
(1) Accurate and Calibrated Probabilistic Hierarchical Time-Series Forecasting: We propose PROFHIT, a deep probabilistic framework for modeling the distributions of each time-series together using soft distributional coherency regularization (SDCR). PROFHIT leverages probabilistic deep-learning models to learn priors of individual time-series and refines the priors of all time-series leveraging the hierarchy provide accurate and well-calibrated forecasts.
(2) Adaptation to Strong and Weak Consistency via Soft Distributional Coherency Regulariza- tion: SDCR imposes soft hierarchical constraints on the full forecast distributions to help adapt the model to varying levels of hierarchical consistency. We build a novel refinement module over raw forecast priors and leverage multi-task learning over shared parameters that enable PROFHIT to perform consistently well across the hierarchy.
(3) Evaluation Across Multiple Datasets and with Missing Data:
We show that our method PROFHIT outperforms wide variety of state-of-the-an baselines on both accuracy and calibration, at all levels of the hierarchy, for both strong and weakly consistent datasets. We also show training using SDCR enables PROFHIT to leverage hierarchical relations to provide robust predictions that can handle missing data values in the time-series.
2

---

2 PROBLEM STATEMENT
Consider the dataset of N time-series over the time horizon 1,2,.... T. Let E R be time- series and v¹") its value time t. The time-series have hierarchical relationship denoted as T (GT, H where GT is a tree of nodes rooted at time-series 1. For a non-leaf node (time- series) i, denote its children as C, The node values are related via set of relations Hy of form H = {y: = Ejec, oiyy : Vi E (1, C,| > 0} where values of diş are known and -nneiindeenndent real-valued constants.
Definition 1 (Consistency Error CE), Given dataset D of N time-series over the time horizon
1, . and aggregation relations HT as above, the dataset consistency error (CE) is defined as
2
(T(D)
Σ
y1 Σ JEC.
Oisy3s
(1)
.E(1,2...N) C,
(Intuitively, datasets with lower CE have time-series values which more strictly follow relations H Definition (Strong and weak consistency). A dataset 'D strongly consistent if For (D) = Otherwise, D is said to be weakly consistent.
Let current time-step be I. For any 1 1 1 < S I, we denote y; (frita) = {v, y
(b1) (1++1) +1)
(1a)
Vi
Given the data D lt yy ...... ... 1:t and hierarchical relations Hy. model M is trained to predict the marginal forecast distributions at time t + T for all time-series of hierarchy leveraging past values of all time-series: {pM(wv (y/"-> ..... Pas pMr (D))) Along with accuracy of probabilistic forecasts we also evaluate forecast distributions for calibration. We define calibration of model forecasts based on previous works (Kamarthi al., 2021; Kuleshov al. 2018):
Definition (Calibration Score of Model) Given a model M we define a calibration function kM [0, [0, 1] as follows: Given confidence e, km(()) is the fraction of the predictions for which the ground truth lies within e-confidence interval. The calibration score (SM ) is the total deviation between e and (c)(c): CS(M) = So [kx(c) - c/dc. A perfectly calibrated model is such that Vo: p (c) Fs c.
Given a dataset D with underlying hierarchical relations HT, the goal of Calibrated Probabilistic Hierarchical Forecasting is to design model M that provides accurate and well-calibrated forecast distributions (pas (v) (y) (++T)) (D)) "'D'),...PM(CC .....-M(Ul (MN (t+47) (t+7)
|D')} across all levels of the hierarchy for both weakly
and strongly consistent datasets.
METHODOLOGY
Soft Distributional Coherency Regularization
Level
Level 2
He. dra
TSFNP
&
y3
y4
A4 C'
JASx IT'S
J4- CT
ies
Bay, Rx
4/1
Raw
Hierarchy of Input Time-Series
Forecast Distribution
Refinement
Refined Forecasts
Likelihood
Module
Figure 2: Overview of pipeline of PROFHIT. The input time series is ingested by TSFNP, a Neural Gaussian Process based probabilistic forecasting model to output the raw forecast distribution. The parameters of raw forecasts are refined by the Refinement module using predictions from all time- series. The training is driven by a likelihood loss that learns from ground truth and Soft Distributional Coherency Regularization that regularizes the forecast distribution to follow the hierarchical relations.
3

---

Overview PROFHIT models the forecast distributions (P(y, (t+T) of the hierarchy by leveraging the relations from the hierarchy to provide accurate and well-calibrated forecasts that are adaptable varying hicrarchical consistency. Most existing methods do not attempt to model entire probabilistic distribution but focus on coherency of point forecasts or samples or
[D')] of all time-series nodes
fixed quantiles of the distribution (Rangapuram et al. 2021 Han et al. 202T). This approach does not fully capture the uncertainty of the forecasts and in turn does not provide calibrated predictions. Moreover, most methods operate on datasets that are strongly consistent over hierarchical relations. However, many real-world datasets are weakly consistent with time-series values of all nodes of hierarchy observed simultaneously and may not follow the hierarchical relations strictly due to noise and discrepancies in collecting data at different levels. Therefore, most previous works may not adapt well to such deviations from these constraints.
PROFHIT, on the other hand, reconciles the need to model coherency between entire forecast distributions as well as induce soft adaptable constraint to enforce coherency via a two-step stochastic process that is trained in end-to-end manner. PROFHIT first produces raw forecast distribution for each node parameterized by (()),)))) by using past values of time-series via neural probabilistic forecasting model. Raw forecasts of all nodes used priors to derive refined set of forecast distributions parameterized by [[us { Ti=l via the refinement module. The full probabilistic process of PROFHIT is depicted in Figure and formally summarized as:
P( (((tt) ( 1 f P(ul(y) (
II
N
P(z., u (a , /2,,, u.)
TSPNP
(2)
N
II Plun (1 (dd
Refinement Module
where Z,, U. are intermediate latent variables of our probabilistic raw forecasting model TSFNP (Section 3.1 PROFHIT's SDCR regularizes the parameters {(44 o to leverage the hierarchi- cal relations by minimizing the Distributional Coherency Error (DCE) defined as follows:
Definition 4. (Distributional Coherency Error DCE) Given the forecasts at time + t as (pM(1) (1+7) (D')....PPl PM
(t+T))
(D')) distributional coherency error (DCE) is defined as
Σ
Dist
PMs Σ
(3)
C.O....,,,CC.t NyC
sEC,
where Dist is a distributional distance metric.
Leveraging distributional coherency well-calibrated while adaptively adhering to hicrarchical relations of the dataset.
soft regularizer enforces forecast distributions to be
3.1 Raw FORECAST DISTRIBUTIONS FROM NGPs
NGPs (Neural Gaussian Process) Louizos al. 2019) are novel class of probabilistic neural models oftt-offthh-art accurate and calibrated predictions. We, therefore, use a modified form of a state- of-the-art NGP model for time-series forecasting (Kamarthi al. 2021) which call TSFNP to derive raw forecast distributions for each time-series.
We briefly describe TSFNP's three components: 1) Probabilistic Neural Encoder: It encodes the input univariate time-series into latent stochastic embedding via GRU (Cho et al. 2014) followed by self-attention layer (Vaswani al. 2017):
(4) Stochastic Data Correlation Graph: We further leverage similar patterns of past time-series data and aggregate them as local latent variable. Unlike EPIENP which uses past time-series information from same node, in our multi-variate case TSENP uses past information from all nodes. Formally,
(og GGRf-Atten(GRU(y ((")))) ~ ( (u)))
for input sequence y: (t'st) and each of the past sequence y, where {1...., N), we sample yj with probability exp(-7/ lu; uyll²) into N,. Then, derive the local latent variable as
2:
Σ (x(u)),exp( Σ O2(u,)) JEN,
(5)
JEN,
4

---

where 1 and O2 are feed-forward networks.
3) Predictive Distribution Decoder: Finally, we combine the latent embedding of input time-series, local latent variable and combined information of all past sequences to derive the parameters of the output distribution via a simple feed-forve sed-ffrwwrr network. We first derive a global latent variable that combines the information from latent embeddings of all past sequences via self-attention:
N
(3) Self-Alten ((u.)
Σ B.u:
(6)
Finally, we combine the latent embedding of input time-series, local latent variable and global latent variable derive the raw forecast distribution modelled Gaussian N (fz, ds) as:
concat(u, 2,, z), [0,, log 0.] = @s(e)
where θ₃ is feed forward network.
3.2 REFINEMENT MODULE
The refinement module leverages the raw distributions of all nodes of hierarchy to produce refined forecast distributions using hierarchical relations. Given the parameters of raw forecast distributions i= derived from TSFNP for all time-series {yr {y( Ji=1 the refinement module derives the
{er, G refined forecast distributions denoted by parameters {ng, Ji as functions of parameters of raw forecasts of all time-series. The refined forecasts are optimized to be more coherent using SDCR. Since we drive the full distributions of refined forecasts to be coherent, rather than just the samples or mean statistics, the refined distributions' calibration also consistent with the hierarchical relations.
(t':t)
Let j = [ AuN] and = - åN] be vectors of means and standard deviations of raw distributions. We model the refined mean as a function of the raw means of all the nodes. Formally, we derive the mean ii of refined distribution series, and b) linear combination of all raw mean of all time-series:
weighted sum of two terms: a) jás the mean of raw
74 sigmoid( (w:)), μ4 7:p + (1 - nwwT A.
(8)
[ and (wi]e are both learnable set of parameters of the model. sigmoid(.) denotes the sigmoid function. % helps model the trade-off between the influence of the raw distribution of node i and the influence of the other nodes of the hierarchy. This is useful for the model to automatically adapt datasets with varying hierarchical consistency.
Similarly, we assume the variance the refined distribution depends on the raw mean and variance of all the time-series The variance parameter a, of the refined distribution is derived from the raw distribution parameters fi and & as
a; = sigmoodd vIA + v + bi)
(9)
where (vir) (v2r) N and (b; N are parameters and is a positive constant hyperparameter.
3.3 LIKELIHOOD LOSS AND REGULARIZATION OVER HIERARCHY
We optimize the probabilistic process of Equation for accuracy and calibration by leveraging hierarchical relations by training on likelihood loss on ground truth training data as well as SDCR.
To maximise the likelihood of forecasts over ground truth (((y!t>)) (,, (D,) we use variational inference by approximating the posterior II P(z,, v.6)) (D.) with the varia- )):))))()(
Likelihood Loss
tional distribution II (y (e'4t) )q:((,, (u, ( (t'()) where qi is feed-forward network over GRU hidden embeddings of Probabilistic Neural Encoder that parameterizes the Gaussian distribution of q. (u:ly!(' (e.)))
We derive the ELBO (detailed derivation in Appendix)
N
L1 EII, aitk u.)) (((y((4)))) y [u; 2.
log P(z./u, {u, log qu (uily!"
(10)
Soft Distributional Coherency Regularization PROFHIT leverages the hierarchy relations in T and regularizes the refined distributions to be coherent. Since PROFHIT aims to leverage hicrarchical coherency for improved robustness and calibration, we regularize over the full distributions by using distributional coherency error as part of the loss function. We use the Jensen-Shannon Divergence
(Endres & Schindelin, 2003) (JSD) the distance metric since it is symmetric and bounded
5

---

variant of the popularly used KL-Divergence distance and assumess closed form for many widely used distributions. We derive the distributional coherency enror on {(pt, 0,) as
S5y (++ (n). )) +
+1+1
=
JSD
P(u. (++T)) ps: o.).
P
(II)
JEC.
Computation of JSD is generally intractable. However, in our case, due to parameterization of each time-series distribution as Gaussian we get closed-form differentiable expression:
(m- Σjec, 0sspes) 2Ijec, 02
N Esec 02 (Is EEEc 01515)
L2
(12)
20
121
We provide the derivation of Equation 12 in the Appendix. We use the distributional coherency error as soft regularization term to enable PROFHIT to leverage constraints Hy when generating forecast distributions. Thus, the total loss for training is given as C = L1 + AC2 where the hyperparameter A controls the trade off between data likelihood and cohereney. We also use the reparameterization trick to make the sampling process differentiable and we learn the parameters of all training modules via Stochastic Variational Bayes (Kingma & Welling 2013). The full pipeline of PROFHIT is summarized in Figure 2
3.4 DETAILS ON TRAINING
Parameter sharing across nodes Since PROFHIT's TSFNP module forecasts for multiple nodes, we leverage the hard-parameter sharing paradigm of multi-task learning (Caruana 1997) and use different set of parameters for Predictive Distribution Decoder (i.e., weights of θ2 for each time- series is different) whereas the parameters of other components of TSFNP are shared across all nodes (Figure 2. Sharing parameters for Probabilistic Neural Encoder drastically lowers the number of learnable parameters since datasets can have large number of nodes (up to 512 nodes in our experiments).
Pre-training on individual time-series Before we start training for refined forecasts, we pre-train the parameters of TSFNP on given training dataset to model raw forecast distribution accurately. We train using only a log likelihood loss to learn parameters {ū;, 6; JN similar to Equation 10
4 EXPERIMENTS
We evaluate PROFHIT over multiple datasets and compare it with state-of the--rtertttt baselines
4.1 SETUP
Baselines: We compare PROFHIT's performance against state-of-the-ars HTSF methods. We also compare against state-of-the-art general probabilistic forecasting methods to study the importance of modeling the hierarchy for both weak and strongly consistent datasets. (1) TSFNP (Kamarthi et al.,
202I) and (2) DEEPAR (Salinas et al. 2020 as state-of-the-arr deep probabilistic forecasting models which do not exploit hierarchy relations. (3) MINT (Wickramasuriya et al. 2019) and (4) ERM (Ben Taieb & Koo| 2019) are methods that convert incoherent forecasts as post-processing step by framing as an optimization problem. Since TSFNP provided better evaluation scores compared to DEEPAR, we performed ERM and MINT on Monte Carlo samples of TSFNP predictive distribution. (5) HIEREEF (Rangapuram et al. 2021) is recent state-of-the-arr deep-learning based approach that projects the raw predictions onto a space of coherent forecasts and trains the model in an end-to-end manner. (6) SHARQ (Han et al., 202I) is another state-of-the-an deep learning based approach that reconciles forecast distributions by using quantile regressions and making the quantile values coherent. (7) PEMBU (Taieb et al., 2017) is a post-processing method that refines raw forecasts to be distributionally coherent. We use the mean forecast from MINT and ERM as input forecasts for PEMBU. Note that we fine-tune the hyperparameters of PROFHIT and each baseline specific to each benchmark. More details on hyperparameters are in Appendix.
We also evaluate the efficacy and contribution of our various modeling choices by performing an ablation study using the following variants of PROFHIT: (7) P-GLOBAL: We study the effect of our multi-tasking hard-parameter sharing approach (Section 3.4) by training a variant where all the parameters shared across all the nodes. (8) P-FINETUNE: We also look at the efficacy of
Code and datasets: http:::t/////tt hcttps://anooymouss.ee 4
6P2

---

our soft regularization using both losses that adapts to optimize for both coherency and training accuracy by comparing it with a variant where the predictive distribution decoder parameters are further fine-tuned for individual nodes using only the likelihood loss. (9) P-DEEPAR: R: We evaluate our choice of using TSFNP, a previous state-of-the-an univariate forecasting model for accurate and calibrated forecasts with DR, another popular probabilistic forecasting model that was used by HIERE2E. (10) P-NOCORERENT: This variant is trained by completely removing the SDCR from the training. Note that unlike P-FINETUNE which was initially trained with SDCR before fine-tuning, P-NOCOHERENT never uses the SDCR at any point of training routine. Therefore P-NOCOHERENS measures the importance of explicitly regularizing over the information from the hierarchy.
Datasets: We evaluate on a diverse set of publicly available datasets (Table 2 from different domains with varied hierarchical relations and consistency. The bechmark- ing dataset and evaluation setup including forecast horizon is replicated from recent and past literature related to general HTSF as well as epidemic forecasting. (1) Labour dataset contains monthly employment data from Feb 1978 to Dec 2020 collected from Australian Bureau of Statistics. (2) Touri L (Wickramasuriya et al. 2019) contains tourism flows in different regions in Australia grouped via region and demo- graphic. It has two sets of hierarchy (with four and five levels), one for the mode of travel and the other for geography with the top node being the only common node of both hierarchies. (3) Wiki dataset collects the number of daily views of 145000 Wikipedia articles aggregated into 150 groups (Taieb al.) 2017). These 150 groups are leaf nodes of four-level hierarchy with groups of similar topics aggregated together. (4) Flu- Sympt oms contains flu incidence values called weighted inftuenza-like llke incidence (wILI) values (Reich al. 2019) at multiple spatial scales for USA for period of 2004-2020. The scales used are states, HHS and National level (US states are grouped into 10 HHS regions by CDC). (5) FB-Survey provides aggregated anonymized daily indicator for the prevalence of Covid- 19 symptoms based on online surveys conducted on Facebook (Delphi Research Group 202T) from Dec 2020 to Aug 2021 for each state and national level. We use the state-level values find aggregates at HHS levels.
Table 2: Dataset Characteristics and Consistency
Dataset
No. of Nodes
Levels of Hierarshy
Obs. per node
Consistency
(CB)
Touriam-1 am-L
555
4,3
12
228
Strong:
Tabour
57
8
514
Strong())
Wikt
207
366
Strong(
Plu-Sympt.oms
61
544
Weak(3 37)
F2-Survey
61
4
257
Weak(2.44)
Tourism-L, Labour and Wiki are constructed by collecting values of leaf nodes and deriving the values of time-series of other nodes of the hierarchy. Hence, they are strongly consistent with zero CE (Definition D The values of each node of hierarchy in case of Pll-Sympt.cm oms and FB-Survey directly collected or measured. For example, the values of Flu-Symptoms dataset are collected from public health agencies at the state, HHS and national levels and aggregated by CDC. Due to factors like reporting discrepancies and noise they contain values in time-series that may deviate from the given hierarchical relations (Chakraborty et al. 2018). Therefore, these datasets are weakly consistent with significant CE (Table 2p.
Evaluation metrics For a ground truth y((), let the predicted probability distribution be Pyce with mean g(t) Also let F Fgt be the CDF. We evaluate model and basclines using carefully chosen metrics that are widely used in literature to measure accuracy and calibration. 1. Mean Absolute Percentage Error (MAPE) is commonly used score for point-predictions calculated as
MAPE = N Eie,, 1000gg00 3. Log Score (LS) is a standard score used to measure accuracy of probabilistic forecasts in epidemiology (Reich et al. 2019). LS measures the negative log likelihood of a fixed size interval around the ground truth under the predictive distribution: LS(py, y) = - log Py di) Similar to (Reich et al. 2019), log likelihood of forecast is capped at -10. 4. Calibration Score (CS): To measure calibration of forecasts, we the calibration score defined in Section 2 2. Cumulative Ranked Probability Score (CRPS) is a widely used standard metric for evaluation of probabilistic forecasts that measures both accuracy and calibration. Given ground truth y and the predicted probability distribution py. let Fg be the CDF. Then, CRPS is defined as: CRPS( FF y) = [_ (t) - (0 > ddj. We approximate F, as a Gaussian distribution formed from samples of model to derive CRPS. 5. Distributional Coherency Error (DCE): We calculate the Distributional Coherency Error (Equation [1] on output forecast distributions during inference to study how PROFHIT and baselines leverage SDCR to learn from hierarchical relations across datasets of varying consistency and trade-off coherency, calibration and accuracy, especially for weakly consistent data (Section 4.2 Q3).
y(t)
7

---

4.2 RESULTS
We comprehensively evaluate PROFHIT through the following questions: Q1: Does PROFHIT predict accurate calibrated forecasts? Q2: Does PROFHIT provide consistently better performance across all levels of the hierarchy? Q3: Does SDCR help PROFHIT outperform baselines on both strongly and weakly consistent datasets? Q4: How does improved calibration affect robustness of PROFHIT's forecasts?
Table 3: Average scores (across 5 runs) across all levels of hierarchy for all baselines, PROFHIT and its variants. PROFHIT provides 54% better accuracy and 32% better calibration.
Tourism-
Labour
Modek/Data
MAPES
CRPS
LS
CS
DCE
MAPES
CRPS
LS
CS
DCE
MAPER
CRPS
LS
CS
DCE
DEEPAK
3.12
0.17
0.81
0.19
0.32
18.27
0.015
0.75
0.23
0.34
16.52
0.232
0.83
0.27
0.25
TSENP
2.28
0.21
1.19
1.41
ISENP-MiTT
1.17
0.5
0.58
0.15
0.24
16.46
0.045
4.12
0.26
0.12
13.79
0.243
0.78
0.18
0.18
TSENP-FRM
1.42
0.56
0.53
0.11
0.18
13.57
0.045
3.63
0.23
0.19
17.74
0.221
0.74
0.19
0.21
HIEREZE
0.15
0.38
0.25
SHARQ
1.63
0.17
0.41
0.12
0.13
14.21
0.054
0.47
0.18
0.09
16.13
0.241
0.52
0.16
0.16
PEMBU-MINT
1.77
0.15
0.46
0.24
0.03
13,55
0.009
0.56
0.22
0.11
14.66
0.279
0.58
0:21
0.05
PEMBU-ERM
0.16
0.042
0.25
PROPHIT
127
0.12
0.33
0.09
0.02
12.79
0.026
0.21
0.14
0.03
12:47
0.184
0.35
0.13
0.04
I-FINETUNE
132
0.16
0.39
0.14
0.25
14.36
0,031
0.36
0.21
0.13
13:22
0.2116
0.39
0.21
0.08
0.
P-DEEPAR
145
0.13
0.52
0.12
0.04
13:44
0.029
0.58
0.17
0.08
12.89
0.201
0.48
0.24
0.07
P-NOCOBERENT
1.82
0.18
0.37
0.21
0.35
16.97
0.043
0.45
0.26
0.17
17.44
0.227
0.47
0.35
0.14
Plo-Sympt.omm
Survey
Modes/Data
MAPES
CRPS
LS
CS
DCE
MAPE
CRPS
LS
CS
DCE
DESPAR
31.27
0.610
3:25
0.065
0.31
17.39
7.32
5.32
0.17
0.28
TSFNP
0.42
15.35
5.53
7.54
TSENP-MINT
10.56
0.630
3.18
0.082
0.18
12:24
5.39
6.35
0.14
0.24
TSENP-ERM
11.85
0.620
2.75
0.075
0.12
13.16
6.14
4.23
0.12
0.19
HIERESF
15.67
0.420
0.81
0.12
0.32
12.63
4.12
1.13
0.19
0.26
SHARQ
18.34
0.470
1.42
0.071
0.21
12.82
3.12
0.81
0.15
0.19
PEMBU- MinT
15.44
0.621
2.58
0.18
0.05
13.75
5.78
4:22
0.22
0.07
PEMBU-ERM
17.57
0.688
2.74
0,15
0.07
12.99
6.31
5.18
0.18
0.1
PROPHIT
8.85
0.250
0.28
0.042
0.14
9.67
1,43
0.45
0.08
D.16
I-FINETUNE
10.44
0.210
0.3
0,039
0.17
9.33
1.18
0.72
0.07
0.19
P-GLOSAL
14.27
0.350
0.47
0.086
0.09
12.11
2.64
1.39
0.14
0.11
P.DREPAR
17.43
0.361
0.083
0.15
11.89
2.13
0.75
0.18
0.15
P-NOCOHERENT
9.17
0.248
0.36
0.16
0.22
13.99
1.17
0.84
0.24
0.22
Accuracy and calibration performance (Q1) We evaluate all baselines, PROFHIT and its variants for all the datasets over 5 independent runs. The average scores across all levels hierarchy are shown in Tables PROFHIT significantly outperforms all baselines in MAPE score by 13% and LS by 14%-550% In terms of calibration, we observe average of 32% lower CS scores. Finally, PROFHIT shows 41-88% better CRPS scores. Thus, PROFHIT adapts well to varied kinds of datasets and outperforms all baselines in both accuracy and calibration. Performing t-test with significance a = 1% we find that all the CRPS, LS and CS scores are statistically significant compared to baselines. On comparing the performance of PROFHIT with the variants, PROFHIT is comparable to or better than the best-performing variant in most benchmarks. This shows that all the important model design choices (multi-task parameter sharing, distributional coherency, and joint training on both losses) of PROFHIT are important for its consistently superior performance.
Performance across the hierarchy (Q2) Next, we look at the performance of all models across each level of hierarchy. We compared the performance of PROFHIT with best performing baselines HIERE2E and SHARQ for all datasets. PROFHIT significantly outperforms the best baselines. At the leaf nodes, which contain most data, PROFHIT outperforms best baselines by 7% in Wi k to 100% in FB-Survey. For the top node of time-series the performance improvement is largest at 35% (wiki) wikkieer to 962% (UBB-Surey)) Similarly, for calibration score, we observe an average improvement of 12% for top nodes and 18% for bottom nodes. We show detailed results in Appendix. PROFHIT also performs better than the variants in most higher levels of hierarchy and its performance is comparable to the best variant P--FINTTUE and P-GLOBAL) at leaf nodes as well. P-NOCOHERENT performs most poorly compared to all variant and PROFHIT, proving that SDCR is very important contributor for consistent performance across the hierarchy all datasets.
SDCR leads to consistently better performance across varying data consistency (Q3) As discussed in Section 4.2, we evaluated on both strong and weakly consistent datasets. Since most previous state-of-ther the-----tth-tttt- art models assume datasets to be strongly consistent, deviations from this assumptions can cause under- munder-prformance when used with weakly consistent datasets. This is evidenced in Table 3 where most of the baselines explicitly optimize for hierarchical coherency
8

---

hard constraint the forecasts. For example, PEMBU's forecasts have better distributional coherency error (DCE) for weakly consistent datasets. However, they perform much worse both accuracy and calibration than even TSFNP, which does not even leverage hierarchical relations. Since we use SDCR as soft learning constraint, PROFHIT can learn to trade-off coherency for accuracy and calibration. Therefore, PROFHIT provides 93% better CRPS and 33% better calibration scores over best HTSF baselines. These improvements are more pronounced at non-leaf nodes of hierarchy where PROFHIT improves by 2.8 times for F Pluu-yypee S-mmptomm and 9.2 times for Survey. In case of strongly consistent datasets, PROFHIT provides 54% better CRPS and 23% better calibration scores while having comparable DCE to PEMBU. We observed that soft coherency regularization and parameter sharing across nodes are vital for PROFHIT's adaptability to varying levels of consistency. We provide detailed analysis of these observations in the Appendix.
THAPP
THAP
MASRI
Duta miaang
Batn misin
Bar miSang
Pacxa mining
Bsta Ring
(a) Tourism-L
(b) Labour
(c) Miki
(d) F Fll--Sa Sluu-Sypttom oms
(e) (D)FB-Surl -B--urvve
Figure 3: % increase in CRPS for all models with increase in proportion of missing data.
Calibration enables robustness (Q4) Accurate and well-calibrated models that can effectively leverage knowledge of the hierarchy can intuitively allow models to better adapt to noise/missing data. Hence we introduce the task of Hierarchical Forecasting with Missing Values to study the robustness of models when there are missing values in time-series. We model situation that is encountered in many weal-world applications such as Epidemic Forecasting where the past few values of time-series missing due to various factors like data reporting delays (Chakraborty et al., 2018).
Formally, at time time-period t, we are given full data up to time t é p. We set p = 5 since it is the average forecast horizon of all datasets. For sequence values in time period between remove 1% of these values across all time-series. The models are trained on complete time-series dataset till time t' t- p. Models' predictions are then used to fill in missing values for time t' to t. Finally, we input the filled time-series to generate forecasts for the future time-steps.
p and i, we randomly
We measure relative decrease in performance with increase in percentage of missing data &: (Figures 3). We observe that PROFHIT's performance decrease with larger fraction of missing values is much slower compared to other baselines. Even k 10%, PROFHIT's performance decreases by 10.45-26.89 8% compared to other baselines that typically decrease by over 70%. Thus, PROFHIT effectively uses coherency to generate robust predictions on strong and weakly consistent datasets.
CONCLUSION AND DISCUSSION
We introduced PROFHIT, a probabilistic hierarchical forecasting model that produces accurate and well-calibrated forecasts using soft distributional coherency regularization (SDCR) which enables adaptablity to datasets with varying levels of hierarchical consistency. We evaluated PROFHIT against previous -fate-of-the-ar hierarchical forecasting baselines over wide variety of datasets and observed 41-88 improvement average improvement in accuracy and 23-33% better calibration. PROFHIT provided best performance across the entire hierarchy as well as significantly outperformed other models in providing robust predictions when it encountered missing data where other baselines' performance degraded by over 70%.
Our work opens new possibilities like extending to various domains where time-series values across the hierarchy may not be continuous real numbers, can not be modelled as Gaussian distributions or may have different sampling rates. We can also explore modeling more complex structures between time-series with different aggregation relations. PROFHIT can also be used to study anomaly detection in time-series, especially in time-periods where there are deviations from assumed coherency relations. Similar to Kamarthi et al. (2022). we can extend our work to include multiple sources of features and modalities of data both specific to each time-series and global to the entire hierarchy.

---

REFERENCES
Bijaya Adhikari, Xinfeng Xu, Naren Ramakrishnan, and B. Aditya Prakash. Epideep: Exploiting embeddings for epidemic forecasting- In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '19, pp. 577-586, New York, NY, USA, 2019, Association for Computing Machinery.
Souhaib Ben Taieb and Bonsoo Koo. Regularized regression for hierarchical forecasting without unbiasedness conditions. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1337-1347, 2019.
6 12
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Prithwish Chakraborty, Bryan Lewis, Stephen Eubank, John S Brownstein, Madhav Marathe, and Naren Ramakrishnan. What know before forecasting the flu. PLOS Computational Biology, 14 (10):e1005964, 2018.
7 17
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv: 1259, 2014.
Giorgio Corani, Dario Azzimonti, Joao PSC Augusto, and Marco Zaffalon. Probabilistic reconciliation of hierarchical forecast via bayes' rule. In ECML/PKDD (3), pp. 2112211121 211-226, 2020. 12
CMU Delphi Research Group. Covid-19 trends and impact survey. 2021. URL https
7/ //cmuddelphh github. 1o/de / httpt//aaapp/cooooccsssssssss si ginggnn s/
fb Survey.html
Dominik Maria Endres and Johannes E Schindelin. A new metric for probability distributions. IEEE
Transactions Information theory, 49(7): 1858-1888666 2003. 5
Tim van Erven and Jairo Cugliari. Game-theoretically optimal reconciliation of contemporaneous hierarchical time series forecasts. In Modeling and stochastic learning for forecasting in high dimensions, pp. 297-317. Springer, 2015. 12
Adam Fisch, Tommi Jaakkola, and Regina Barzilay. Calibrated selective classification. arXiv preprint
arXiv:2208. /2084, 2022. 2
Xing Han, Sambarta Dasgupta, and Joydeep Ghosh. Simultaneously reconciled quantile forecasting of hierarchically related time series. In International Conference on Artificial Intelligence and Statistics, pp. 190-198. PMLR, 2021.
12
Rob J Hyndman and George Athanasopoolos. Forecasting: principles and practice, OTexts, 2018.
12
Rob Hyndman, Roman A Ahmed, George Athanasopoulos, and Han Lin Shang. Optimal com- bination forecasts for hierarchical time series. Computational statistics data analysis, 55(9): 2579-2589, 2011.
Harshavardhan Kamarthi, Lingkai Kong, Alexander Rodriguez, Chao Zhang, and Aditya Prakash. When in doubt: Neural paraaaraetticc uncertainty quantification for epidemic forecasting. Thirty- fifth Conference on Neural Information Processing Systems, 2021.
Harshavardhan Kamarthi, Lingkai Kong, Alexander Rodriguez, Chao Zhang, and B Aditya Prakash. Camul: Calibrated and accurate multi-view time-series forecasting. ACM The Web Conference (WWW), 2022. 9
Diederik P Kingma and Max Welling. arXiv:1312.61 14, 2013. 13
Auto-encoding variational bayes. arXiv preprint
Rahul Krishnan, Uri Shalit, and David Sontag. Structured inference networks for nonlinear state space models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
12
10

---

Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International conference on machine learning, pp. 2796-2890000 PMLR. 2018. 2
Longyuan Li, Junchi Yan, Xiaokang Yang, and Yaohui Jin. Learning interpretable deep state space model for probabilistic time series forecasting. arXiv preprint arXiv:2102.00397 2021.
12
Christos Louizos, Xiahan Shi, Klamer Schutte, and Max Welling. The functional neural process.
arXiv preprint arXiv: 08324, 2019. 4
Frank Nielsen. On the jensen-shannon shannon symmetrization of distances relying on abstract means. Entrpy, 21(5):485, 2019. 2019.
14
Julie Novak, Scott McGarvie, and Beatriz Etchegaray Garcia. A bayesian model for forecasting hierarchically structured time series, arXiv preprint arXiv:1711.04738, 2017. 12
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. Advances neural information processing systems, 31, 2018. 12
Syama Sundar Rangapuram, Lucien D Werner, Konstantinos Benidis, Pedro Mercado, Jan Gasthaus, and Tim Januschowski. End-to-end learning of coherent probabilistic forecasts for hierarchical time series. In International Conference on Machine Learning, pp.
PMLR, 2021. 2
12
Nicholas G Reich, Logan C Brooks, Spencer Fox, Sasikiran Kandula, Craig J McGowan, Evan Moore, Dave Osthus, Evan Ray, Abhinav Tushar, Teresa K Yamana, al. A collaborative multi- year, multimodel assessment of seasonal influenza forecasting in the united states. Proceedings of the National Academy of Sciences, 116(8)33146-315 2019.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):
1181-1191, 2020.
12
Souhaib Ben Taieb, James W Taylor, and Rob J Hyndman. Coherent probabilistic forecasts for hierarchical time series. In International Conference on Machine Learning, pp. 3348-3357. PMLR, 2017.
12
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Shanika L Wickramasuriya. Probabilistic forecast reconciliation under the gaussian framework. arXiv
preprint //3/ 2021. 12
Shanika L Wickramasuriya, George Athanasopoulos, and Rob J Hyndman. Optimal forecast rec- onciliation for hierarchical and grouped time series through trace minimization. Journal of the American Statistical Association, 814 819, 2019.
g 6 T2

---

Appendix for PROFHIT: Probabilistic Robust Forecasting for Hierarchical Time-series
A ADDITIONAL RELATED WORK
Probabilistic time-series forecasting Classical probabilistic time-series forecasting methods include exponential smoothing and ARIMA (Hyndman & Athanasopoulos, 2018). They are simple but focus on univariate time-series and model each time-series sequence independently. Recently, deep learning based methods have been successfully applied in this area. DeepVAR (Salinas et al.) (2020) trains an auto-regressive recurrent network model on large number of related time series to directly output the mean and variance parameters of the forecast distribution. Other works are inspired from the space-state models and explicitly model the transition and emission components with deep learning modules such as deep Markov models (Krishnan et al. (017)) and deep state space models (Li et al. 2021 Rangapuram et al. 2018) Recently, EpiFNP (Kamarthi et al. (02TT) has achieved state-of-art performance in epidemic forecasting. It learns the stochastic correlations between input data and datapoints to model a flexible non non-parametris distribution for univariate sequences.
Hierarchical time-series forecasting Classical works on hierarchical time-series forecasting used two-step approach (Hyndman et al. 2011 Hyndman & Athanasopoulos (018) and focus on point predictions. They first forecast for time-series only at single level of the hierarchy and then derive the forecasts for other nodes using the hierarchical relations.
Recent methods like MINT and ERM are post-processing steps applied on the set of forecasts at all levels of hierarchy. MINT (Wickramasuriya et al. 2019: Wickramasuriyal 2021) assumes that the base level forecasts are uncorrelated and unbiased and solves an optimization problem to minimize the variance of forecast errors of past predictions. The unbiased assumption is relaxed in ERM (Ben Taieb & Koo 2019). Corani al. (2020) and Novak al. (2017) uses a fully Bayesian bottom-up post-processing approach using raw forecasts from full hierarchy. Another line of works projects the raw forecasts of all time-series into subspace of coherent forecasts. (Erven Cugliari 2015) use iterative Game-theoretic approach of minimizing forecast error and projection error. Taieb et al. (2017) uses copula method to refine raw forecasts to be distributionally coherent as a post-processing step. Recent neural methods perform end-to-end learning that enables the model to leverage hierarchical relations while forecasting. Rangapuram et al. (202T) use a deep-learning based end-to-end approach to directly train the projected forecasts. SHARQ (Han et al. 2021) is another recent probabilistic deep-learning based method that uses quantile regression and regularizes for coherency at different quantiles of forecast distribution. However, unlike our approach, these end- to-end methods do not regularize for coherency over the entire distribution (Distributional Coherency) but only over fixed quantiles. Most of these methods also are not designed for cases where the hierarchical constraints are not always consistently followed.
B CODE AND DATASET
We evaluated all models on a system with Intel 64 core Xeon Processor with 128 GB memory and Nvidia Tesla V100 GPU with 32 GB VRAM. We provide anonymized repository of our implementation of PROFHIT along with the datasets used at We will release the code and data publicly after acceptance.
4open
sci rciiencer//PRRR http://ROFFHT16600000 6F2F
C HYPERPARAMETERS
C.I DATA PREPROCESSING
Most datasets used in our work assume the aggregation function to be simple summation (i.e, dıj = 1 for all weights). We first normalize the values leaf time-series training data to have 0 mean and variance of I. Since the aggregation of values at higher levels of the hierarchy can lead very large values in time-series, we instead divide each lon-leaf time-series by the number of children. Then the weights of hierarchical relations become oy
| where C2 is the set of all children nodes of
12

---

time-series i. For the remaining datasets (Flu-Symptoms, FB- S--mptotmm) the time-series values normalized by default and thus require no extra pre- precceseessingg.
C.2 MODEL ARCHITECTURE
The architecture of TSFNP used in PROFHIT is similar that used in the original implementation (Kamarthi et al.) 2021). The GRU unit contains 60 hidden units and is bi-directional. Thus the local latent variable also of dimension 60. NN and N N2 are both 2-layered neural networks with the first layer shared between both. Both layers have 60 hidden units. Finally. NN is three-layer neural network with the input layer having 180 units (for the concatenated input of three 60 dimensional vectors) and the last two layers having 60 hidden units. We found that the value of c in Equation 9 not very sensitive and usually set to
Note that we do not explicitly model covariance between every pair of time series (like MINT, ERM) and use a weighted combination raw forecast parameters to derive refined forecasts. Therefore the refinement module complexity (Section 3.2) is O(N2) which is par with previous methods like HIERE2E.
C.3 TRAINING AND EVALUATION
Given the training dataset D, extract training dataset for each node the set of prefix sequences f1 S 12 < t- and train the full model (TSFNP and refinement module). We tune the hyperparameter using backtesting by validating window to t. Finally train for entire training set with best hyperparameters.
[(y y/ ((2+1))
For each benchmark, used the validation set to mainly find the optimal batch size and learning rate. We searched over batch-size of {10, 50, 100, 200} and the optimal learning rate was usually around 0.001. We also found the optimal À be around 0.01 for strongly consistent datasets and 0.001 for weakly consistent datasets. We used early stopping with the patience of 150 epochs to prevent overfitting. For each independent run of model, initialized the random seeds from 0 for Py PyTooc and NumPy. We didn't 't observe large variations due to randomness for PROFHIT and all baselines.
During evaluation, we sampled 2000 Monte-Cari samples of the forecast distribution and used it to estimate the mean for MAPE, We also used the samples mean and variance to evaluate LS and CS whereas used ensemble scoring to evaluate CRPS directly from the samples using properscoring package 3l
D DERIVATION OF LIKELIHOOD ELBO LOSS
The full predictive distribution of PROFHIT from Equation
2 can be further expanded as:
( P(( Ji=1 ID)) / [I [II P(ully
N
II
((N)
Global Latent variable
Prohabilistic Fncoder
SDCG
|l
O(0,,,,|,,22 u:)
Plu, o, (t+T) (l
Raw forccasts
Refinement Module
(13)
To minimize the data likelihood (( ( ID.) requires intregration over latent variables ( and z, ) We instead perform amortized variational inference on the latent variables similar to VAE KKngmaa & Welling 2013).
13

---

We approximate the posterior of latent variables P([u:)=1, (2i) (N.) 21(1; (i=1) with
( )E expressed as: a variational distribution Quu ( Zuy
Q(uu, (z (N
2/K,
ri=1)
II P(u.ly((in)
)
[ (UN
IN
II (aa(z))! (11)))
P(zl(uiH(m))
(14)
where Yo feed-forward network over GRU embeddings of Probabilistic Neural Encoder that parameterizes to gaussain distribution of z,
The ELBO loss
E QcEu, M.N" log (((y (1+7) p 2, No z)
+ log P((u,) (2 ( (1) - log Q((u, (2) (N.) N z (u ( -
(15)
get simplified to Equation 10 by cancelling similar terms between variational and true distribution of latent variables.
E DERIVATION OF DISTRIBUTIONAL COHERENCY ERROR
The Distributional Coherency Error (Equation TI can be exactly expressed as:
o
Pi Ejec, )isps) ΣE,EC. 22
Ejeo, 0f + (ni Ejec, )WsH))
L2
(16)
20
To derive Equation 12 we use the following well-known result for JSD of Gaussian Distributions
NNillsnn 2019):
Lemma 1. Given two univariate Normal distributions Pi = NN(11, o,) and P2 = N2(U2, 02), the
JSD
1 o + (11 - 22)
of + (M) -22)2 1
JSD
(17)
203
20
Consider each JSD term
)) +1
JSD
1p. P
Σ
jEC;
of the summation in Equation 11
Note that
P(y, (attli,, b.) (
(18)
and P(E,ea, Therefore.
is weighted sum of Gaussian variables (N (N o;)lie,,
P
Σ
b dsbjec. =N
Σ
Σ
(19)
V
JEC.
IEC.
JEC.
Using Lemma I along with Equations 18
19 we get the desired result in Equation 16
14

---

Table Average deviation of observed values in time-series from hierarchical relations.
Data
Flu
FB 13-Survey
Tourism-
Labour
Wiki
Level 1
0.043
1.27
0
0
Level 2
3.41
2.83
0
0
Average across hierarchy
3.37
2.44
0
0
0
F CONSISTENCY OF DATASETS
We noted in Section 4.2 Q4 that Flu Stn-Syymtom and FB-Survey weakly consistent datasets since they do not strictly follow the aggregation relations Hy unlike strongly consistent datasets Tourism-L, Labour, Wiki.
We empirically observe this by measuring Consistency errors of all datasets (Definition I for entire hierarchy and each level of the hierarchy. The results are Table A As expected there is no deviations for strongly consistent datasets where as there is significant deviation in weakly consistent data.
15

---

G PERFORMANCE ACROSS EACH LEVEL OF HIERARCHY
Table Average CRPS scores at each level of hierarchy. PROFHIT significantly outperforms best baselines across all benchmarks. Note that P-Finetune's performance decreases higher levels of hierarchy compared to other variants whereas P-Global's performance is worse at lower levels.
Models/Data
Toari sm-1
Labour
Hierarchy Levels
2(Travel)
3(Trave))
Travvll
S1 Travel
2(Geo)
3(Gco)
4(Geo)
2
HIEREZE
0,081
0.103
0.141
0.205
0.272
0.103
0.136
0.175
0.031
0,034
0.034
0.038
SHARQ
0.093
0.131
0.163
0.218
0.295
0.131
0.138
0.152
0.097
0.124
0.133
0.149
PEMBU-MINT
0.112
0.121
0.139
0,203
0.185
0.116
0.128
0.167
0.063
0.033
0.042
0.085
PROFHIT (Ours)
0.051
0.095
0,12
0.264
0.083
0.106
0.142
0.023
0.019
0.023
0.029
P-FINETUNE
0.072
0.136
0.083
0.278
0.124
0.124
0.024
0.022
0.026
0.035
P-GLOBAL
0.093
0.113
0.122
0.13
0.261
0.093
0.113
0.147
0.021
0.027
0.028
0,027
P-DEEPAR
0.075
0.097
0.136
0.183
0.281
0.095
0.122
0.159
0.025
0.027
0.031
0.033
0.138
0.031
Models/Data
WIX1
Plu- Symp 2
FB Surrvy
Hierarchy Levels
2
5
2
HIEREZF
0,042
0.105
0.229
0.272
0.372
0.272
0.421
0.458
4.14
4.04
4.1.
SHARQ
0.039
0.136
0.235
0.291
0.378
0.258
0.376
0.381
3.08
3.21
3.13
PEMBU-MINT
0.031
0.171
0,241
0.385
0.433
0.337
0.567
0.773
4.82
5.53
6.15
PROFHIT (Ours)
0.031
0.074
0.133
0.216
0,252
0.216
0.133
0.338
0.32
0.43
1.89
P.FINETUNE
0,275
0.222
P-GLOBAL
0.048
0.103
0.187
0.265
0.186
0.269
0.213
0,376
0.37
0.37
2.11
P-DEEPAR
0.035
0.094
0.193
0.251
0,285
0.242
0.217
0.328
0.44
0.61
2.01
P-NOCOHERENT
0.49
0.227
0.36
Table 6: Average CS scores at each level of hierarchy. PROFHIT significantly outperforms best baselines across all benchmarks.
Models/Data
Tourism
Labour
Hierarchy Levels
2
2(Geo)
3Geo)
4(Geo)
2
HIRREZE
0.15
0.18
0.17
0.21
0.24
0.119
0118
11.22
11.21
0.23
0.22
0.27
SHARQ
0.09
0.08
0.12
0.11
0.11
0.12
0.16
0.16
0.15
0.21
PEMBU-MINT
0.14
0.21
0.22
0.21
0.26
0.18
0.23
0.25
0.21
0.22
0.24
0.21
PROFHIT
0.05
0.06
0.04
0.06
0.11
0.06
0.06
0.1
0.17
0.11
0.15
0.16
P.FINE TUNE
0.09
0.12
0.13
0.17
0.13
0.11
0.13
0.15
0.24
0.21
0.24
0.22
0.05
P-DEEPAR
0,11
0.09
0.09
0.14
0.13
0.15
0.14
0.13
0.14
0.19
0.17
0.14
P-NOCOHIERENT
0.18
0.19
0.17
0.19
0.22
0.18
0.19
0.24
0.24
0.22
0.25
0.31
PR-SSrvvy
Hierarchy Levels
2
HIERE2E
0.15
0.21
0.26
0.22
0.24
0.11
0.13
0.11
0.21
0.19
0.18
SHARQ
0.13
0.14
0.14
0.17
0.15
0.58
0.052
0.085
0.16
0.14
0.15
PEMBU-MINT
0.12
0.11
0.12
0.13
0.14
0.17
0.22
0.2
0.19
0.16
PROFHIT
0.11
0.15
0.12
0.14
0.11
0,031
0.044
0.052
0.09
0,07
0.06
P-FINE TUNE
0.19
0.18
0.23
0.22
0.24
0.033
0.031
0.042
0.05
0.06
0.09
P-GLOBAL
0.16
0.15
0.065
0.072
0.096
0.11
0.13
0.17
P-DEEPAR
0.21
0.24
0.26
0.22
0.23
0.064
0.077
0.083
0.15
0.19
0.17
P-NOCOHERENT
0.29
0.28
0.35
0.33
0.37
0.22
0.18
0.14
0.22
0.25
0.21
16

---

H DETAILS ON DATA IMPUTATION EXPERIMENT
During real-time forecasting in real-world applications such as Epidemic Sales forecasting, we encounter situations where the past few values of time-series are missing or unreliable for some of the nodes. This is observed specifically lower levels, due to discrepancies or delays during reporting and other factors (Chakraborty et al., (2018). Therefore, one approach to performing forecasting in such a situation first by imputation of missing values based on past data and then using the predicted missing values as part of the input for forecasting.
Motivation:
Task: To simulate such scenarios of missing data and evaluate the robustness of PROFHIT and all baselines, we design task called Hierarchical Forecasting with Missing Values (HFMV). Formally, time-period t, we are given full data for up time t p. We show results here for p = 5 which is the average forecast horizon of all tasks. For sequence values in time period between - p and i, we randomly remove k% of these values across all time-series. The goal HFMV task is to use p to as input along with complete dataset for time-period before t t- to predict future values at + r. Therefore, success in HFMV implies that models are robust to missing data from recent past by effeetively leveraging hierarchical relations.
the given partial dataset from
We first train PROFHIT and baselines on complete dataset till time and then fill in the missing values of input sequence using the trained model. Using the predicted missing values, we again forecast the output distribution. For each baseline and PROFHIT, we perform multiple iterations of Monte-Carke sampling for missing values followed by forecasting future values to generate the forecast distribution. We estimate the evaluation scores using sample forecasts from all sampling iterations.
Setup:
We compared the performance of PROFHIT with best performing baselines HIERE2E and SHARQ for each level of hierarchy of all datasets. PROFHIT significantly outperforms the best baselines as well as the variants. At the leaf nodes, which contains most data, PROFHIT outperforms best baselines by 7% in Wiki to 100% in FB- Survey. For the top node of time-series the performance improvement is largest at 35% (Wiki) ( to 962% (FB-Survey). We show detailed results in Table
13.5
215
10.
22
170
Deaesung
(a) -
(b) Labour
(c) Wiki
(d) F1FFFl-Syyaa Symptoms (e) FDFFFB-Sur -BB-Survyy
Figure 4: % increase in CRPS for PROFHIT and variants with increase in proportion of missing data.
Robustness of PROFHIT variants: We compare relative performance decrease with increase in percentage of missing data for PROFHIT and its variants in Figure We observe that P- NoCOHERENT's performance deteriorates very rapidly in most benchmarks, showing the importance of SDCR for learning provides robust calibrated coherent forecasts. The second worse- porse-performing variant across all datasets is P-FINETUNE which also relies less on the hierarchical relations due to fine-tuning of parameters for specific time-series. Finally, we observe that PROFHIT and P-GLOBAL suffer the least degradation in performance since both these models prioritize integrating hierarchical coherency information which enables them to provide better estimates for imputed data for missing input and use them to generate more accurate and calibrated forecasts.
I ADAPTING TO VARYING DATASET CONSISTENCY
Observation 1. The average improvement in performance of PROFHIT over best HTSF baselines is 72% higher for weakly consistent datasets over its improvement for strongly consistent datasets.
17

---

Since most previous state-of-art models assume datasets to be strongly consistent, deviations to this assumptions can cause under cederr-erfomaacce when used with weakly consistent datasets. This is evidenced in Table where some of the baselines like MINT and ERM that explicitly optimize for hierarchical coherency perform worse than even TSFNP, which does not leverage hierarchical relations, in Fl Pllu-Smmptom and FB-Survey. Overall, we found that for weakly consistent datasets, PROFHIT provides a much larger 93% average improvement in CRPS scores over best HTSF baselines compared 54% average improvement for strongly consistent datasets. These improvements are more pronounced at non-leaf nodes of hierarchy where PROFHIT improves by 2.8 times for F Sympt and 9.2 times for FB-Survey. This is because HTSF baselines which assume strong consistency do not adapt to noise at leaf nodes that compound to errors at higher levels of hierarchy.
Observation 2. PROFHIT's approach parameter sharing and soft coherency regularization helps adapt to varying hierarchical consistency.
We observe that that best performing variant for strongly consistent datasets in P-GLOBAL which is trained with both likelihood loss and SDCR (Table 3). But its performance severely degrades for weakly consistent datasets since sharing all model parameters across all time-series makes inflexible to model patterns and deviations specific to individual nodes. In contrast, P-FINETUNE and P-NOCOHERENT performs the best among variants for weakly consistent datasets since they train separate sets of decoder parameters for each node. But they perform poorly for strongly consistent datasets since they don't 't leverage Distributional Coherency effectively. PROFHIT combines the flexible parameter learning of P-FINETUNE and leverage Distributional Coherency jointly optimize the parameters like P-GLOBAL providing comparable performance to best variants over all datasets.
Observation 3. PROFHIT's Refinement module automatically learns to adapt to varying hierarchi- cal consistency.
The design choices of the refinement module help PROFHIT to adapt to datasets of different levels of hierarchical consistency. Specifically, by optimizing for values of {r:) & of Equation 8. PROFHIT aims to learn a good trade-off be- tween leveraging prior forecasts for a time- series and hierarchical relations of forecasts from the entire hierarchy. We study the learned values of (x of Equation jused to derive refined mean. Note that higher values of % indicate larger dependence on raw forecasts of node and smaller dependence of forecasts of the entire hierarchy. We plot the average values of n for each of the datasets in Table 7 We observe that
Table 7: Average value of %/8 for all datasets. Note that weakly coherent datasets have higher 7. (de- pends mode on past data of same time-series) where as strongly-cohens coherett data have lower % (leverages the hierarchical relations).
Consistency Strong
Datasel
Averuge value of 7:
Tourism-L
0.420 + 0.096
Labour
0.348 + 0.091
Wiki
0.313 + 0.057
Weak
Symp
0.759 1 0.152
Fbsymp
0.789 + 0.180
strongly consistent datasets have lower values of 7/ indicating that PROFHIT's refinement module automatically learns to strongly leverage the hierarchy for these datasets compared to weakly coherent datasets.
Table 8: Std. dev of CRPS and LS (aceros 5 runs) across all levels for all baselines, PROFHIT and its variants. PROFHIT performs significantly better than all baselines as noted using I-test with 1%.
Models/Data
Tourism-L
Labour
Wiki
Plu-Sympt
F3-Survey
CRPS
LS
CRPS
1.S
CRPS
1.S
CRPS
L.S
CRPS
1.S
Baselines
DEEPAR
0.011
0.040
0.004
0.038
0.002
0.044
0,018
0.098
0.482
0.434
TSFNP
0.006
0.021
0,003
0.018
0.015
0.069
0.019
0.004
0.251
0.217
MINT
0.005
0.019
0,002
0.121
0.018
0.006
0.014
0.111
0.468
0.213
ERM
0.044
0.005
0,002
0.110
0.016
0.069
0.018
0.133
0.148
0.209
HIERE2E
0.001
0.038
0.003
0.049
0.019
0.018
0.005
0.051
0.325
0.109
SHARQ
0.000
0.011
0.001
0.046
0.017
0.007
0.002
0.116
0.133
0.048
PROFHIT (Ours)
0.001
0.017
0.001
0.003
0.001
0.030
0,005
0.009
0.040
0.008
Ablation
P-PINETUNE
0.016
0.031
0.003
0.003
0.016
0.014
0.001
0.006
0.090
0.004
P-GLOBAL
0.012
0.033
0.000
0.013
0.002
0.001
0.033
0.024
0.248
0.119
P-DEEPAR
0.006
0.026
0.001
0.028
0.005
0.043
0.035
0.030
0.103
0.065
P-NOCOHERENT
0.005
0.012
0.001
0.009
0.015
0.043
0.012
0.025
0.110
0.053
18


Reviews:
[{"review_id": "vxnLTUydHN", "reviewer": "Reviewer_B3vd", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new probabilistic forecasting method for hierarchical time series. Compared to existing methods, the proposed method is more robust and can deal with inconsistent hierarchies. Specifically, the method is based on a flexible Bayesian approach which includes distributional coherency regularization. Experiments on multiple datasets show that the proposed method improves accuracy compared to baselines and provides reliable forecasts even if up to 10% of input time-series data is missing.\n", "summary_of_the_review": "I have reviewed this paper multiple times (for other conferences), and have given many suggestions for improvement. Unfortunately, the authors have ignored important past comments. \n\nThe biggest weakness of the paper is the lack of rigor and the quality of writing (notations, definitions, etc). Understanding differences and similarities with existing methods is also very important. Calibration is also not well defined.\n\nUnless the writing is significantly improved, I do not see this paper being accepted at any top ML conference.\n\n", "strength_and_weaknesses": "Strength\n- More flexibility and robustness in probabilistic forecasting methods are important for many applications.\n- The method can deal with inconsistent hierarchies.\n- The proposed method provides good accuracy for various datasets.\n\nWeaknesses\n\n- The paper lacks rigor and clarity.\n\n- Important related work is missing:\n\t- https://www.sciencedirect.com/science/article/abs/pii/S0377221722006087\t\n\t- https://arxiv.org/pdf/2103.11128.pdf\n\t- Bayesian methods\n\t\t- https://link.springer.com/chapter/10.1007/978-3-030-67664-3_13\n\t\t- https://arxiv.org/abs/1711.04738\n\n- It is not clear how the proposed method relates to existing work, e.g. MINT. Given that the authors make a gaussianity assumption, it is not clear how their method compares with MINT, for which we have closed-form expression for the entire hierarchy (for means and covariance matrix). \n\n\n- The logscore is improper with respect to incoherent predictive distributions. As a result, you cannot compare coherent with incoherent distributions with LS. See https://www.sciencedirect.com/science/article/abs/pii/S0377221722006087 for more details.\n\n- The authors claim that their method provides well-calibrated forecasts. However, this concept it not clearly defined (See Definition 3). Please use your predictive model p_M in the definition.\n\n- The authors claim that they model the joint distribution of all series in the hierarchy. However, in the sentence before Section 3, the authors refer to the marginals. This is not clear. Expression (2) gives the joint distribution of y_i for i=1, .., N, but the first sentence in \"overview\" talks about the marginal distributions. \n\n- The authors should clearly explain the factorization in (1).\n\nComments:\n- \"We performed ERM and MINT on Monte Carlo samples of TSFNP predictive distribution\": Please be more precise.\n\n- The authors use both consistency and coherency. I would suggest sticking to coherency.\n\n- CRPS does not measure accuracy and calibration, but sharpness and calibration.\n\n- Various notions of \"distributional coherency\" have been proposed in the literature. A precise definition is needed here. I think you are using sample coherency\n\n- \"Samples of distribution\": do you mean samples from the predictive distribution?\n\n- In latex, please use \\citep{} instead of \\cite{}.\n\n- What do you mean by \"time-series of employment\"?\n\n- \"both so-called strong and weakly consistent datasets\": I am not sure \"so-called\" is the right term to use here.\n\n- \"They produce unreliable confidence intervals\": I think you meant \"prediction intervals\".\n\n- In section 2, please specify the range of phi_{i,j}. Does it belong to {0, 1}?\n\n- Notations: \\hat{p}(Y_t), \\hat{y}_t, etc.\n\n- Note that the CRPS has a closed-form expression for a Gaussian distribution.\n\n- The authors should discuss the limitations of their method (computational complexity, statistical assumptions, etc)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MIHWaxFZDux", "reviewer": "Reviewer_eMuv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a probabilistic hierarchical time series forecasting model that provides a coherent distributional forecast that achieves better performance in forecasting accuracy. The core idea is to use a deep probabilistic forecasting framework, i.e., the neural Gaussian process, to produce a probabilistic base forecast and then refine the base forecast using a separate module. This refinement procedure is based on a variational inference framework coupled with a distributional coherency loss. The proposed method has achieved improved performance over various baselines, particularly in datasets with missing values.", "summary_of_the_review": "This paper proposed a novel method to address an important problem. I would recommend accepting this paper upon my concerns are addressed.", "strength_and_weaknesses": "Strength:\n1. This paper proposed a fully parameterized and probabilistic framework for HTS forecasting, which flexibly incorporates hierarchical information into both point and distributional forecasts by enabling parameter sharing.\n2. The proposed method obtains better performance on both point and probabilistic forecasts for various datasets.\n3. The authors have distinguished strong and weak consistency, i.e., whether time series at a lower level should strictly add up to obtain time series at the upper level, which is a useful step for many real-world applications.\n\nWeakness:\n1. Since the authors have distinguished strong and weak consistency in the main paper, it is also better to separate (or at least highlight) them in the experiment since some methods are designed for strong consistency (e.g., MinT, ERM, and HIERE2E) and some method is better for weak consistency (e.g., SHARQ). It is not fair to compare these methods from different categories.\n2. It seems that there is no formal guarantee of the robustness of PROFHIT over other baselines. The authors are using missing value imputation whereas other methods can also do. This experiment seems heuristic.\n3. Normally in many applications, time series at different aggregation levels have distinct properties over sparsity, noise distribution or sampling rate etc., which will affect the distributional assumption at each level. Since the proposed method uses a general Gaussian distribution for each level, the authors could discuss how this general assumption can be adapted to datasets with different properties.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FY7zg9x_zRr", "reviewer": "Reviewer_cEHM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to address two issues with prior work on the task of hierarchical time series forecasting (HSTF): (i) the assumption of rigid coherency (i.e. forecasts enforce the time-series values of datasets to satisfy the underlying hierarchical constraints strictly) and (ii) the lack of calibrated forecasts.\n\nTo address these issues, the paper introduces soft-distributional-coherency regularization (SDCR) that enables end-to-end learning of the forecast distribution at all levels by leveraging information from the underlying hierarchy. The method involves a two-stage process where\na network is trained to output raw predicted distributions for all time series in the dataset. Then a refinement module jointly refines the distribution parameters for all predicted distributions by leveraging the hierarchical relationships between the time series through SDCR.\n\nEvaluation across multiple datasets and missing data indicate that the proposed approach massively outperforms state-of-the-art methods.", "summary_of_the_review": "The goals of this paper and the proposed method are likely to benefit the community working on hierarchical time series forecasting. The issues involving unclear writing, notational imprecision, and formatting abuse make it hard to read this paper and evaluate the details. ", "strength_and_weaknesses": "### Strengths\n\n1. The paper aims to address important limitations of past methods (that of forecasting single values rather than distributions, enforcing rigid coherency, and the lack of calibrated forecasts). Addressing these limitations of the previous work are likely to benefit the part of the community interested in these tasks. \n\n2. Empirical evaluation is comprehensive, including several datasets, methods, and metrics. \n\n### Weaknesses\n\n1. Clarity, Formatting Abuse: The paper is hard to read due to various issues in writing, notional imprecision, and formatting abuse (see notes on Clarity below). \n\n2. Reporting the results makes comparisons in a way that the improvement numbers seem large, but the comparisons aren't always reasonable. E.g. an improvement of 550% on log-likelihood is a comparison on an unbounded metric, as compared to, say, accuracy where an improvement in terms of a % improvement carries more meaning. The results would still be meaningful without such % improvement metrics in these cases. \n\n3. It's unclear if a t-test is a reasonable test to compare model performance, specifically that it's unclear why the assumptions required for a t-test hold here. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "3rlldNZThky", "reviewer": "Reviewer_UgsV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposed a deep probabilistic framework for modeling the distributions of each time-series together using a soft distributional coherency regularization (SDCR) over forecast distributions. It has the advantages of 1) adapting to both strong and weak hierarchical consistent datasets and 2) producing well calibrated forecasting at all levels. The empirical results demonstrated that it improves performance on forecasting accuracy and calibration on all levels and provides robust predictions with missing data.\n", "summary_of_the_review": "The paper is well motivated. It fills the gap of enforcing hierarchical constraints in the probabilistic space, allowing for soft matching and improved calibration. The chosen methodology makes sense to me and the empirical work is relatively strong and convincing. It has limitations in the Gaussian output, but as demonstrated in the results, it already performs strongly compared to the baselines. The paper is mostly clearly written and the code is also provided for reproducibility. ", "strength_and_weaknesses": "Strength:\n* The paper is well motivated. It’s an Interesting setting with weakly consistent datasets, where the data generation process may follow a hierarchical set of constraints but may contain some deviations. HIERE2E (Rangapuram et al., 2021) and SHARQ (Han et al., 2021) impose hierarchical constraints on the mean or fixed quantiles of the forecast distributions, but the distribution may not be well calibrated.\n\n* Convincing experiments. The work considered many relevant baselines and reported various metrics, characterizing different aspects. The 4 model variants provide data for the impact of the model choices. Replacing TSFNP with DeepAR and comparing to baselines with DeepAR separates out the effect of the base forecasting model. \n\nOn the weakness, the proposed method assumes Gaussian for the prediction, which is fair start but it limits the choices. Extending the framework to other distributions can be beneficial. \n\n=== After discussion with AC and other reviewers ===\n\nI think the idea is interesting and the empirical evaluation is quite comprehensive. But at the same time, the math behind the model especially on the ELBO, as pointed out by the AC, is alarming to me. There are steps missing and it is hard to verify the correctness of the derivations. Please see more detail in the above for this point. This negatively impacts the soundness of the work. Thus, I will lower my score accordingly.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject