Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces a mechanism, C2FAR, to parametrize a conditional distribution via successive hierarchical binnings parametrized by an autoregressive function (here, a neural network). This parametrization can reportedly maintain expressibility in high dimensional binned distributions with logarithmically many parameters required. The authors present a thorough empirical study of their method in the backdrop of time series forecasting, where C2FAR-RNN (their hierarchical--coarse-to-fine--binned distribution on top of DeepAR) performs favorably both in forecast accuracy and calibration. 
strengths_and_weaknesses: The paper is well-written, well-organized and gives a good review of comparable methods in forecasting. To the best of my knowledge, the introduced techniques are novel, although I couldn't help comparing them to hierarchical softmax/categorical distributions used in NLP and wondered if any analogies could be drawn from there. In the forecasting domain, I believe the results posted by the authors are significant and advance the state of the art. Despite the simplicity of the ideas proposed, I believe the paper is insightfully written and benefits from a well-thought-out experimental set-up. Moreover, I believe C2F can have impact in other related areas of probabilistic deep learning.
questions: - The authors allude to limited precision in most forecasting data. I am not sure if this is really a justification for the method? Note that this is different than significant "atoms" of probability observed on 0% or 100%--reminiscent of zero-inflated distributions which are more similar to the authors' construction.
- In Eq (1), should the lhs be $p(\mathbf{z} | \mathbf{x})$ instead?
- On L129: "Inputs may be represented with 1-hot-encodings or embedding layers" Which inputs are these? $\mathbf{x}$? If not, what is the difference implied between one-hot encoding and an embedding layer?
- L192.  During training, and when evaluating NLL of test sequences, all values are known and all distributions can be computed in parallel. I am not sure of this, as the backprop would have to be sequential in any case with an RNN? 
- Why restrict the total number of hyperparameter optimization runs instead of total time? Are the training times on the same order? If so, please mention. Seeing as the sequential evaluation is the bottleneck in GPU implementations, then it may be that computation really takes B-times more time than DeepAR in training, which is unfair. I would urge the authors to report wall-clock training times of their method against DeepAR in the same number of bins. 

limitations: Other than the last question mentioned above, I do not believe there are significant limitations of the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes C2FAR, a new method for probabilistic univariate forecasting. C2FAR sequentially discretizes the time-series in bins, from coarse to fine resolution, using an RNN to learn temporal relations similar to the DeepAR model. At each layer, a shared RNN model further divides each bin into finer intervals based on covariates and values of different layers. Forecasts are produced by drawing samples conditional on the learned parameters. The paper compares the proposed method against other RNN probabilistic architectures, such as DeepAR, IQN-RNN, and SQF-RNN, on several benchmark datasets.
strengths_and_weaknesses: Strengths:
- Novel method for probabilistic univariate forecasting, based on sequential hierarchical binning.
- SoTA performance on 4 benchmark datasets, compared to other RNN-based models.
- The shared model between bins of each layer allows for exponentially expressiveness with linear cost.
- The last layer can handle extreme values thanks to the Pareto-distributed tails.
- Comprehensive ablation and stability experiments. The experimental setting is very clear and thoroughly explained.

Weaknesses:
- Only RNN architectures are used as baselines. While RNN is widely used, other architectures have shown superior performance in multiple forecasting tasks, especially for multi-step forecasting. Authors should include other models in the experiments, including simple baselines such as seasonal-naive, ETS, or ARIMA.
- C2FAR requires more training time and resources than baselines. Training time should be also compared against other baselines.
questions: - Inference (prediction) times are not reported or mentioned in the paper. Can you provide average inference times for different datasets?
- Have you evaluated C2FAR on longer horizons? Since forecasts are produced recursively, errors will accumulate, degrading the performance considerably.
limitations: Limitations are discussed, and I do not identify additional potential negative social impacts.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a new parametrize of the predictive distribution of probabilistic forecasting models. The build on prior work proposing binning or mixtures of uniform distributions (with parametric tails). They propose to avoid the pitfall of flat binning, where increasing the precision of the distribution requires to drastically increase the number of bins and so the number of output parameters of the neural network. They compare their method to existing work and show improvements on standard datasets.


strengths_and_weaknesses: Strengths:
- The core idea of the paper is very good. It is a simple and efficient mechanism to solve the niche but real problem of the increase in the bins requiring an increase in the number of output parameters. Having a finer grid of bins through auto-regressive additions, is very good idea.
- I want to call out the honesty of the authors in the presentation of their work, their are openly vocal about the limitations of the work, which is something that should be praised.
- The author make a very valid point on the discretization of time series in general in the section 2.2.


Weaknesses:
A few things are not explained or laid out very clearly in the current version of the paper, and I think that the paper would benefit a lot from having these written out more clearly:
- I am not very clear on the parametrization of the neural network for p(z_i|x,z_{<i}). What kind of neural network is being used?
- How is the NLL computed at training time? (I wonder if one could optimize the code since only one value is required, that of the bin where the data point falls) I think that a pseudo code would be helpful to understand how this is being done.
- In the evaluation, please make clear what C2FAR-RNN1 is, and what the others are. I only saw in the appendix that if corresponds to K = 1.

- I would like to point out that I could verify all these points by looking at the code, but the authors do not provide the code. I see this as a major issue and would want the code to be open sourced if the paper is to be accepted.

I have difficulties understanding how the Pareto tails work exactly. In general, and compared to [17].
- alpha_1 and alpha_2 are outputs of the last layer, are they values on the observed space or quantiles?
- If they are values on the observed space, how is the density corresponding to the Pareto tail estimated?
- Or, does the maximum bin value move with alpha_1? If so, how does the network cope with the changing bin positions?
- Does having these tails force to compute the full densities?
- Are any of the experiments done with the tails? Have you evaluated the calibration of the tails?

Weaknesses in the evaluation:
- I would like to point out that DeepAR-Binned is not uniform bins but simply discretized values. And with that, the paper does not compare to the main comparison partner [17] which has uniform bins (like the flat tiling of C2FAR) and Pareto tails.
- It would be interesting to see the evaluation on a wider range of probabilistic forecasting datasets, and potentially regression datasets.
- The is a natural trade-off between the number of bins K in each level and the number of levels B. It would be interesting to lay it out clearly and to evaluate what the optimal trade-off is.
- It would be interesting to see the comparison to more flexible distributions like normalizing flows?
- It would be good to show the results with standard deviation.
- Overall, given the small methodological contribution of the paper it would have been valuable to provide a more exhaustive evaluation.



[17] Elena Ehrlich, Laurent Callot, and François-Xavier Aubet. 2021. Spliced Binned-Pareto Distribution for Robust Modeling of Heavy-tailed Time Series
questions: - How does one obtain a wQL% with the naive baseline, as the naive baseline does not provide a predictive distribution?
- You mention using your implementation of DeepAR. Why not use the open source implementation of DeepAR in GluonTS?
limitations: The author discuss exhaustively the limitations of their work, which is to be called out. They discuss the complexity and the efficiency of the method, I can see indeed that the efficiency depends a lot on the implementation and I would have liked to get to see their implementation.
review_title: null

Meta-review: metareview: The reviewers highlight the clarity of the writing, the effectiveness of the proposed approach resulting in SOTA performance, and in particular the comprehensive and clear empirical evaluation with ablation studies and stability experiments, as well as a laudable discussion of the limitations of the proposed approach. The authors were able to address reviewers' concerns around baselines and comparison partners during the discussion period.
confidence: Certain
Decision: Accept