Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to address two issues with prior work on the task of hierarchical time series forecasting (HSTF): (i) the assumption of rigid coherency (i.e. forecasts enforce the time-series values of datasets to satisfy the underlying hierarchical constraints strictly) and (ii) the lack of calibrated forecasts.

To address these issues, the paper introduces soft-distributional-coherency regularization (SDCR) that enables end-to-end learning of the forecast distribution at all levels by leveraging information from the underlying hierarchy. The method involves a two-stage process where
a network is trained to output raw predicted distributions for all time series in the dataset. Then a refinement module jointly refines the distribution parameters for all predicted distributions by leveraging the hierarchical relationships between the time series through SDCR.

Evaluation across multiple datasets and missing data indicate that the proposed approach massively outperforms state-of-the-art methods.
strength_and_weaknesses: ### Strengths

1. The paper aims to address important limitations of past methods (that of forecasting single values rather than distributions, enforcing rigid coherency, and the lack of calibrated forecasts). Addressing these limitations of the previous work are likely to benefit the part of the community interested in these tasks. 

2. Empirical evaluation is comprehensive, including several datasets, methods, and metrics. 

### Weaknesses

1. Clarity, Formatting Abuse: The paper is hard to read due to various issues in writing, notional imprecision, and formatting abuse (see notes on Clarity below). 

2. Reporting the results makes comparisons in a way that the improvement numbers seem large, but the comparisons aren't always reasonable. E.g. an improvement of 550% on log-likelihood is a comparison on an unbounded metric, as compared to, say, accuracy where an improvement in terms of a % improvement carries more meaning. The results would still be meaningful without such % improvement metrics in these cases. 

3. It's unclear if a t-test is a reasonable test to compare model performance, specifically that it's unclear why the assumptions required for a t-test hold here. 

clarity,_quality,_novelty_and_reproducibility: ### Clarity

The paper is hard to read. The paper frequently introduces terms and concepts it only later defines. I understand there are several concepts involved and this might sometimes be hard. At the very least abbreviations should be expanded when stated first (e.g. TSFNP). Nevertheless, combined with the widespread vspace abuse and several notational issues, this makes reading the paper somewhat challenging overall.

#### Notational issues:
- Definition of H_T has "for every i in 1...N". However, the definition only applies to non-leaf nodes and therefore cannot apply to all N time series in the dataset.
- If the time variable is in the superscript, I suggest being consistent and writing D_t as D^t.
- What is u?
- In Eq. 4, if u is a vector, what is mu(u)? Seems like here u oughht to be a random variable not a vector.
- In Eq. 4, why is the subscript i italics in some places and roman in others?
- When N_i is a term explicitly used, please do not denote networks as NN_1, which simply adds to the confusion.
- Is it LS or IS? When defining the metric the paper says both LS and IS. For Q1 results, the paper says that the proposed method improves IS by up to 550% which seems like a typo. 
 
#### Other writing improvement issues/suggestions:
- "rigid coherency on generated forecasts generate" -> remove last 'generate'
- Fix citation style, citations should be in parenthesis unless using \citet
- "Most state-of-art HTSF methods" -> "state-of-the-art"
- vspace abuse around the tables and figures.
- vspace abuse around equations (see top of Eq. 2)

### Quality

It is hard to evaluate the work rigorously given the writing and notational issues, and the lack of code and data in the supplementary material (the link is anonymized but could have been submitted as supplementary). The goals of the paper are reasonable and likely to be useful.

### Novelty

The paper addresses important limitations/assumptions of the previous methods and is novel to the best of my knowledge. 

### Reproducibility

Hard to evaluate reproducibility without seeing the code and data. The link implies this will be available should the paper be accepted, which is better than not having any code and data. 


summary_of_the_review: The goals of this paper and the proposed method are likely to benefit the community working on hierarchical time series forecasting. The issues involving unclear writing, notational imprecision, and formatting abuse make it hard to read this paper and evaluate the details.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper proposes a new probabilistic forecasting method for hierarchical time series. Compared to existing methods, the proposed method is more robust and can deal with inconsistent hierarchies. Specifically, the method is based on a flexible Bayesian approach which includes distributional coherency regularization. Experiments on multiple datasets show that the proposed method improves accuracy compared to baselines and provides reliable forecasts even if up to 10% of input time-series data is missing.

strength_and_weaknesses: Strength
- More flexibility and robustness in probabilistic forecasting methods are important for many applications.
- The method can deal with inconsistent hierarchies.
- The proposed method provides good accuracy for various datasets.

Weaknesses

- The paper lacks rigor and clarity.

- Important related work is missing:
	- https://www.sciencedirect.com/science/article/abs/pii/S0377221722006087	
	- https://arxiv.org/pdf/2103.11128.pdf
	- Bayesian methods
		- https://link.springer.com/chapter/10.1007/978-3-030-67664-3_13
		- https://arxiv.org/abs/1711.04738

- It is not clear how the proposed method relates to existing work, e.g. MINT. Given that the authors make a gaussianity assumption, it is not clear how their method compares with MINT, for which we have closed-form expression for the entire hierarchy (for means and covariance matrix). 


- The logscore is improper with respect to incoherent predictive distributions. As a result, you cannot compare coherent with incoherent distributions with LS. See https://www.sciencedirect.com/science/article/abs/pii/S0377221722006087 for more details.

- The authors claim that their method provides well-calibrated forecasts. However, this concept it not clearly defined (See Definition 3). Please use your predictive model p_M in the definition.

- The authors claim that they model the joint distribution of all series in the hierarchy. However, in the sentence before Section 3, the authors refer to the marginals. This is not clear. Expression (2) gives the joint distribution of y_i for i=1, .., N, but the first sentence in "overview" talks about the marginal distributions. 

- The authors should clearly explain the factorization in (1).

Comments:
- "We performed ERM and MINT on Monte Carlo samples of TSFNP predictive distribution": Please be more precise.

- The authors use both consistency and coherency. I would suggest sticking to coherency.

- CRPS does not measure accuracy and calibration, but sharpness and calibration.

- Various notions of "distributional coherency" have been proposed in the literature. A precise definition is needed here. I think you are using sample coherency

- "Samples of distribution": do you mean samples from the predictive distribution?

- In latex, please use \citep{} instead of \cite{}.

- What do you mean by "time-series of employment"?

- "both so-called strong and weakly consistent datasets": I am not sure "so-called" is the right term to use here.

- "They produce unreliable confidence intervals": I think you meant "prediction intervals".

- In section 2, please specify the range of phi_{i,j}. Does it belong to {0, 1}?

- Notations: \hat{p}(Y_t), \hat{y}_t, etc.

- Note that the CRPS has a closed-form expression for a Gaussian distribution.

- The authors should discuss the limitations of their method (computational complexity, statistical assumptions, etc)

clarity,_quality,_novelty_and_reproducibility: Unfortunately, the paper lacks rigor and clarity. 

The proposed method is novel, but it is not clear how it relates to existing methods, such as MINT, which makes similar assumptions. 

Regarding reproducibility, the authors provide code to reproduce their experiments.
summary_of_the_review: I have reviewed this paper multiple times (for other conferences), and have given many suggestions for improvement. Unfortunately, the authors have ignored important past comments. 

The biggest weakness of the paper is the lack of rigor and the quality of writing (notations, definitions, etc). Understanding differences and similarities with existing methods is also very important. Calibration is also not well defined.

Unless the writing is significantly improved, I do not see this paper being accepted at any top ML conference.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a probabilistic hierarchical time series forecasting model that provides a coherent distributional forecast that achieves better performance in forecasting accuracy. The core idea is to use a deep probabilistic forecasting framework, i.e., the neural Gaussian process, to produce a probabilistic base forecast and then refine the base forecast using a separate module. This refinement procedure is based on a variational inference framework coupled with a distributional coherency loss. The proposed method has achieved improved performance over various baselines, particularly in datasets with missing values.
strength_and_weaknesses: Strength:
1. This paper proposed a fully parameterized and probabilistic framework for HTS forecasting, which flexibly incorporates hierarchical information into both point and distributional forecasts by enabling parameter sharing.
2. The proposed method obtains better performance on both point and probabilistic forecasts for various datasets.
3. The authors have distinguished strong and weak consistency, i.e., whether time series at a lower level should strictly add up to obtain time series at the upper level, which is a useful step for many real-world applications.

Weakness:
1. Since the authors have distinguished strong and weak consistency in the main paper, it is also better to separate (or at least highlight) them in the experiment since some methods are designed for strong consistency (e.g., MinT, ERM, and HIERE2E) and some method is better for weak consistency (e.g., SHARQ). It is not fair to compare these methods from different categories.
2. It seems that there is no formal guarantee of the robustness of PROFHIT over other baselines. The authors are using missing value imputation whereas other methods can also do. This experiment seems heuristic.
3. Normally in many applications, time series at different aggregation levels have distinct properties over sparsity, noise distribution or sampling rate etc., which will affect the distributional assumption at each level. Since the proposed method uses a general Gaussian distribution for each level, the authors could discuss how this general assumption can be adapted to datasets with different properties.
clarity,_quality,_novelty_and_reproducibility: This paper is technically sound and well-written. The authors have clearly distinguished the novelty over prior works and compared it with the baselines. They have also provided the code to reproduce the experiments.
summary_of_the_review: This paper proposed a novel method to address an important problem. I would recommend accepting this paper upon my concerns are addressed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposed a deep probabilistic framework for modeling the distributions of each time-series together using a soft distributional coherency regularization (SDCR) over forecast distributions. It has the advantages of 1) adapting to both strong and weak hierarchical consistent datasets and 2) producing well calibrated forecasting at all levels. The empirical results demonstrated that it improves performance on forecasting accuracy and calibration on all levels and provides robust predictions with missing data.

strength_and_weaknesses: Strength:
* The paper is well motivated. It’s an Interesting setting with weakly consistent datasets, where the data generation process may follow a hierarchical set of constraints but may contain some deviations. HIERE2E (Rangapuram et al., 2021) and SHARQ (Han et al., 2021) impose hierarchical constraints on the mean or fixed quantiles of the forecast distributions, but the distribution may not be well calibrated.

* Convincing experiments. The work considered many relevant baselines and reported various metrics, characterizing different aspects. The 4 model variants provide data for the impact of the model choices. Replacing TSFNP with DeepAR and comparing to baselines with DeepAR separates out the effect of the base forecasting model. 

On the weakness, the proposed method assumes Gaussian for the prediction, which is fair start but it limits the choices. Extending the framework to other distributions can be beneficial. 

=== After discussion with AC and other reviewers ===

I think the idea is interesting and the empirical evaluation is quite comprehensive. But at the same time, the math behind the model especially on the ELBO, as pointed out by the AC, is alarming to me. There are steps missing and it is hard to verify the correctness of the derivations. Please see more detail in the above for this point. This negatively impacts the soundness of the work. Thus, I will lower my score accordingly.
clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear written, I only have some minor comments:
* The author mentioned TSFNP is modified fromKamarthi et al., 2021, what is the difference and why? It’s provided in the appendix but it will help to also summarize it briefly in the main text. 
* In eq(11), the loss is computed on the refined $\mu, \sigma$ or the raw $\hat{\mu}, \hat{\sigma}$?
* Typo in the “IS measures the negative log likelihood…” IS->LS.
* Why not make the hyperparameter $c$ in eq(9) learnable?
* What are the hyperparameters and training details for the baselines?

The overall writing and technical qualities are good. To the best of my knowledge, enforcing hierarchical constraints in the probabilistic space for forecasting is novel. Good reproducibility is backed up with implementation detail and code.

summary_of_the_review: The paper is well motivated. It fills the gap of enforcing hierarchical constraints in the probabilistic space, allowing for soft matching and improved calibration. The chosen methodology makes sense to me and the empirical work is relatively strong and convincing. It has limitations in the Gaussian output, but as demonstrated in the results, it already performs strongly compared to the baselines. The paper is mostly clearly written and the code is also provided for reproducibility.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper deals with hierarchical forecasting, where observations of real-valued time series are available at different levels of aggregation. The paper stresses the importance of weak consistency, where observed values do not exactly form sums according to the hierarchy, but only approximately so. The authors propose what they call a Bayesian probabilistic model along with a variational inference learning procedure (closely related to prior work), and then extend this optimization problem by constraints coming from the hierarchy. Since these are applied as a form of regularization, the constraints are not enforced as equalities, which allows them to enforce weak consistency. The authors claim that due to their probabilistic model approach, their forecasts are better calibrated than for previous methods.

On the positive side, reviewers state that there could be some useful ideas in this work, which would extend prior work (which is numerous in statistics).

The major downside of this submission is lack of clarity in the presentation, to an extent that makes it hard to judge its value. Just some examples:
- The supposedly probabilistic model is introduced in a long equation with lots of variables that are not defined (u_i, z_i). It is not an equation for a joint probability, because most targets are conditioned on, but some form of predictive. This begs the question how all the posterior terms are being approximated, and what likelihood and model structure linking the latent variables they are based upon
- The authors do not properly explain the relation between u_i and z_i (supposedly both latent variables)
- The variational distributions are not defined, and the paper only talks about distributions over u_i. These equations contain several typos
- The calibration metric used is stated as integral, with undefined functions. It is not clear what is really being evaluated
- There is some talk about sampling something with probability exp(-gamma |u_i - u_j|^2) "into a set N_i", it is totally unclear what is meant here, but this seems an important step when using this model

These are just some examples. One of the reviewers spent several days to try and decode the writing and finally concluded that with a lot of extra, work, the gaps can maybe be filled.

ICLR has a certain bar at clarity and writing quality, and this paper is clearly below this bar. The authors should spend serious efforts on clarifying what they intend to do. Instead of changing the style file to squeeze out every bit of space (as was done in the submission), they should cite earlier closely related work instead of repeating things. If they intend to claim their work represents a proper probabilistic model and a valid variational approximation to it, they should state clearly and precisely what the model is (the joint generative one), and how its latent variables relate to each other, then describe the variational distributions and their structure. They should also state how their constraints or regularization comes in, and what type of dependencies among the variational distributions this implies. Finally, given the high complexity of what is proposed here, compared to previous work, the paper lacks a strong motivation why all this added complexity is needed, in terms of ablations. It also lacks statements on how this complexity is managed, say how the optimization problem is solved reliably, or how long it takes.

Finally, the reviewers also express valid concern about the evaluations.

justification_for_why_not_higher_score: The lack of clarity in the exposition simply make it too hard even for a somewhat informed reader to take much value out of this paper. Clarity is especially important in a field like forecasting, where predictions need to be interpreted, and need to be trustworthy. In fact, it is not that this paper comes in the context of an application where all that matters are good results, and we can just ignore a large set of mostly unmotivated technicalities.

justification_for_why_not_lower_score: N/A
Decision: Reject