Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this manuscript, the authors proposed a method for hierarchical time series forecasting, consisting of two
components, the TVAR model, and the BD model. Further, the performance of the proposed algorithm is empirically evaluated on three benchmark datasets and showed that the proposed model consistently improved over state of the art baselines.
main_review: 
- The assumption $|| \theta_n - \theta_0 ||_2 \leq \beta$ seems to be restrictive. The embedding parameters could satisfy the hierarchical structure while the parent embedding parameter $\theta_0$ would be different that the children embedding $\theta_n$. Authors mention at the top of page 7 that as $\beta \rightarrow 0$, the hierarchically regularized estimator approaches an error $L$ times smaller when compared to the un-regularized estimator. That seems to be more related to the assumption than the developed algorithm. For very small $\beta$, essentially, the sample size is increased from $T$ to $LT$ since $L$ additional time series are observed, but thanks to the assumption that $\beta$ is very small, the number of parameters to be estimated did not increase. As a result, the consistency rate for estimating the embedding parameter will be of order $O(1/(LT))$ instead of $O(1/(T))$. Thus, the theoretical justification of the proposed methodology is unclear.

- Results of Theorem 1 are related to prediction error consistency rather than parameter estimation consistency. It is appropriate to consider verifying the rate for $|| \tilde{\theta}_n - \theta_n ||_2$ instead of  $|| \tilde{\theta}_n - \theta_n ||_\Sigma$.

- It is not clear why the Non-Negative Matrix Factorization (NMF) algorithm is used to select a small set
of representative time series that encode the global state, please elaborate further.

- Typo in page 7: "Tables 1 and 2 shows ..."

- Typo in page 9: "... state of the art baselines for most levels of the hierarchically."
summary_of_the_review: 
- The paper is well written and easy to read.
- Theoretical results need clarification, while some assumptions need to be relaxed.
- Code for reproducing the results is provided together with links to publicly available real data sets used in the manuscript.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper considers point forecasting of hierarchical time series, i.e. multivariate time series with hierarchical aggregation constraints. The authors propose a new approach based on decomposing the series along a global set of basis time series where (approximate) hierarchical constraints are applied on the coefficients of the basis decomposition. Forecasts are produced using a dynamic linear autoregressive model. Compared to existing state-of-the-art hierarchical models, the proposed approach improved overall performance on forecasts at different levels of the hierarchy on several public datasets.

main_review: Strengths:
- Dimension reduction is an important topic that has been overlooked in the literature on hierarchical forecasting.
- Experiments with several publicly-available real-world datasets.

Weaknesses:
- Paper clarity should be improved. Reading the paper raised a lot of (unanswered) questions. See comments below.
- About the proposed method:
* The proposed method produces roughly/approximate coherent forecasts. It is hard to compare methods that produce coherent and incoherent forecasts.
* The paper only considers point forecasting. Probabilistic forecasting is a more challenging and important problem.
* It is not clear whether the basis decomposition should be applied on all series. A hierarchical time series is a multivariate time series with hierarchical aggregation constraints. As a result, some series are linear combinations of other series. Isn't it enough to just apply it to the bottom-level series? 
- I am wondering if the comparison between methods is fair. In fact, the proposed method minimizes (regularized) MAE while other baselines minimize MSE. Your method is more aligned with the evaluation metrics, WAPE/SMAPE, which are essentially scaled absolute errors.


Other comments:
- Related work: literature on hierarchical probabilistic forecasting should be discussed, as well as dimension reduction methods for multivariate time series.
- "Hierarchical coherency" has nothing to do with coherence as discussed in Thomson et al., 2019.
- Section 2 should be split in two different paragraphs. One on mean forecasting and the other on probabilistic/quantile forecasting. You cannot really compare these two types of methods.
- Figure 4: it is not clear why Tourism has a non-zero L3 colum.
- What does "asymptotically better" means? Please be more rigorous or avoid this term.
- "under the reasonable assumption of the parent embedding being close to all the children embeddings" -> Why is it a reasonable assumption?
- Why imposing the hierarchical constraints on the embedding is equivalent to constraining the forecasts? This should be clearly explained.
- The motivation for NMF should be clearly stated in the paper.
- More details on the baselines should be given, at least in the appendix.
- What does minibatch means with hierarchical time series data? This should be clearly explained.
- Given the simplified assumption, I am not sure the theoretical analysis really gives additional insight. Also, it only takes one paragraph in the whole paper. 
- While simplified assumption are often required for a theoretical analysis, these assumptions shoul be clearly stated. 
- "A small beta implies that the children time-series have structural similarities which is common in hierarchical datasets." -> I am not convinced by this sentence. Please give more details on Beta.
- Is Lemma 1 really useful? Replacing values in the Theorem should be enough.
- Simulation experiments could be used to confirm the usefulness of the method when indeed the data is low-dimensional. This would complement the theoretical analysis.

summary_of_the_review: While dimension reduction in hierarchical forecasting is an important problem, the paper lacks clarity and justifications for all the design choices. Major references are missing (dimension reduction for multivariate time series, probabilistic hierarchical forecasting, etc). The experimental setup is also questionable (loss function, etc). As a result, I do not think the paper is ready for publication.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a method for hierarchical time series forecasting.
The problem setting is: given historical hierarchical univariate time series 
data and given historical and future features (like holidays, etc..), try to 
predict future values for all time series, while keeping the coherence 
constraints of the hierarchy.
The forecasting is broken down into an autoregressive part with shared 
parameters for all time series, and a basis decomposition with different 
weights for each time series (regularized by the hierarchy structure).
The method is then being compared to different baseline methods on three 
datasets.
main_review: The paper’s subject, forecasting, is being studied regularly in academia and 
research groups, and hierarchical forecasting specifically, has many use 
cases in industry.
The paper is well written and organized overall and conveys a good “reader 
journey” experience. The figures and tables describing and emphasizing 
relevant parts and have good explanations.
The ‘Introduction’ describing the problem, gives relevant use cases and 
serves as a great entrance gate for the paper. Figure1 gives an important 
visual intuition. The ‘Problem Statement’ paragraph is a must have in every 
paper and helps to prepare the reader for the ‘Problem Setting’ section and 
the suggested model section. For a weakness, I would like to point out that 
the paper is dealing with ‘additive coherence constraints’ only, while one of 
the use cases described in the first paragraph is financial predictions, that 
might not have the additive constraint (for example in financial markets), but 
a different form of aggregation. It’s fine that the paper doesn’t deal with it, 
but it would be worth if the paper touched on this point.
In section 4, the ‘HiReD’ model is being very well explained and Figure2 
completes the picture of the model perfectly. For the model itself, the 
strength of it is the fact that it uses simple building blocks that can be 
implemented easily with state-of-the-art gradient based libraries, also it can 
deal with hierarchical structured data and scales linearly with more time 
series. Moreover, it uses information from the entire hierarchy in training but 
not in the forecasting phase, that should result in faster forecasts. Its 
weakness is the fact that you must know the hierarchy structure in advance 
(it’s OK for the use cases described in the paper, but it might not always be 
the case). The predictions are point forecasts and not probabilistic, while in 
many domains confidence level or intervals are very important for decision 
making (like in retail). Another issue that wasn’t discussed is the scalability of
the model to new time steps – do you have to train the model all over again 
with all history plus the new time step to produce an improved prediction (for
t+2)?

In the ‘Experiments’ section, the method is compared to a variety of 
methods, that can be divided into: (i) neural network (NN) models for time 
series - that are not necessarily optimized for hierarchical problems; (ii) graph
NN models; (iii) NN for hierarchical models; (iv) reconciliation models. The 
explanations, metrics and result tables are well written, but I want to point 
out that two methods that were introduced in the ‘Related Work on Deep 
Hierarchical Models’ part, Han et al. (2021) and Yanchenko et al. (2021), were
not  used as baseline. As actionable feedback, I recommend including in
the baseline more NN for hierarchical models methods than “regular” NN 
time series methods, as the main subject of the paper is hierarchical time 
series forecasting.
summary_of_the_review: The paper is well written and gives great introduction for the subject. The 
suggested method’s strength is in its easy to implement building blocks, the 
ability to take as into account information from all hierarchy and to scale well 
for larger hierarchies, though it could be better if the issue of dealing with 
new time steps would be addressed. The datasets chosen for the 
experiments are showing good results against relevant baseline, although 
there are few more relevant methods that were not included in the baseline.
review_title: null

Meta-review: The paper proposes a method for time series forecasting based on a hierarchical deep learning approach. Three reviewers submitted reviews, with two marginally accept and one marginally reject. The paper was therefore borderline, but the issues raised by the marginal reject reviewer on the justification for the design choice of a deep latent model and the experimental setup appear worth addressing in a revision resubmitted to another conference.
Decision: Reject