Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper provides a new view of the probabilistic forecasting problem. The authors have shown some drawbacks of some current state-of-the-art probabilistic forecasting methods and propose that a simple model coupled with quantile loss or heteroscedastic Gaussian regression can often outperform more complex and expensive alternatives, particularly in the long horizon setting. The authors support their claim on multiple time series datasets from different domains and with distinct properties. They have also studied how the prediction horizon and time series dimension affect the inference time. Finally, they reach the conclusion that the architecture design of forecasting models should be prioritized over uncertainty quantification techniques.
strength_and_weaknesses: Strength:
The authors have conducted extensive empirical evaluations and conducted a comprehensive background review. The most inspiring part is the discussion on the impact of intrinsic stochasticity and the structure of data on forecasting models. Based on this property, the authors have divided the analysis into time series from different domains. This can serve as an interesting reference for the characteristics of time series datasets and provide insights on which model to choose.

Weakness:
This paper is driven by pure empirical studies without theoretical insights or methodological innovation. The employed forecasting models and loss functions are all well-studied. The major work done by the authors is to combine them together and make a comparison on different datasets. Moreover, the authors don't provide a clear explanation of why the NHiTS is chosen as the base model and what is the intuition of its better performance than other complex methods. Lastly, I am also confused about how the authors reached the conclusion that architecture design should be prioritized over uncertainty quantification since the whole paper is comparing CRPS results across different methods. The authors apparently need a transition on this.
clarity,_quality,_novelty_and_reproducibility: This paper is ambiguous on how its conclusion is reached. There is also not too much technical novelty. The authors have provided code to reproduce their experiments.
summary_of_the_review: This paper is an empirical study of probabilistic forecasting models, while it lacks solid technical novelty and theoretical insights. There is not much innovation in experiments either. I think this paper will benefit from another round of revision.

**Update after rebuttal**

The authors' responses have addressed some of my concerns, but I believe they are still not sufficient to significantly change my view of this paper. I would encourage authors to incorporate answers to the abovementioned questions in the next revision and discuss the underlying theoretical insights on why the methodology is performed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work considers long-term time series forecasting based on on deep probabilistic techniques. It shows that intrinsic stochasticity in data is often so remarkable that it poses notable challenges for the interpretation of many current deterministic methods. Importantly, the authors argue that uncertainty quantification is less important than neural network architecture design in many practical scenarios. The authors suggest a probabilistic evaluation framework that solves this challenge and improves prediction on common benchmarks and practical problems in climatology and economy.
strength_and_weaknesses: Strength

+ intuitive representation of uncertainty through probabilistic treatment
+ scalability of the method
+ complements the current state of understanding in an innovative way
+ relevance to a broad variety real application cases

Weaknesses
- Limitations and weaknesses of the method could be addressed in the manuscript in more detail as these are relevant for applications
clarity,_quality,_novelty_and_reproducibility: The reporting is clear and good quality, the approach is novel and innovative, and the relevant literature has been properly cited. Reproducibility has been described properly, and links to public repositories are provided. I did not manage to access these (due to anonymization of the submission?) and could not investigate the code itself. Some data sets may pose challenges for reproducibility: it is said that "the hourly NASDAQ-100 dataset can also be easily reproduced using public logs"; a code for doing this could improve the reproducibility further. It is not clear whether the code is provided with an open license.




summary_of_the_review: Overall this is an interesting and timely contribution of good technical quality.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes an approach to provide a probabilistic forecast on top of any point forecast. Two post-processing approaches in particular are investigated, one fits a Gaussian noise by learning a condition heteroskedastic variance on top of an existing point forecast, the other fits a quantile-regression model. Experiments are conducted by applying this post-processing to an existing method (NHits) on 7 datasets against several baselines with an emphasis on long-term forecasting performance.
strength_and_weaknesses: The main strength of the paper are that it is clearly written and has clear motivation since providing accurate probabilistic forecast on top of point forecast has many relevant applications.

The main weaknesses are as follow.

The approach is very related to conformal prediction, see [Romano 2019] for instance who proposes to fit a quantile regression model that can be guaranteed to be calibrated or [Hasson2021] who already showed the benefit of the approach by demonstrating how the top point forecast of M5 could be made a top competitor to the uncertainty track with a post-processing. Sadly, those approaches are not mentioned nor compared while being directly similar to the method proposed.

In addition, the method proposed does not distinguish itself clearly from just fitting a Gaussian/Quantile regression on top of an existing architecture. Indeed, the main improvement reported in the experiments seems to come from using an architecture better suited for long term predictions (NHiTS). However, one could also fit the same model with a Gaussian likelihood or a quantile regression as the loss function of NHiTS and likely obtain similar results in term of CRPS and calibration. The main difference of the approach proposed is the two step estimation but it is not compared to the simpler approach consisting on just fitting a distribution on top of an existing model and also have drawbacks that are not mentioned (two training must be done and latency will likely be twice higher).

To clearly justify their contribution, the authors should probably show that their methods outperforms both non conformal prediction approaches and fitting Gaussian/quantile regression approaches (the first one because it is state-of-the-art and the standard approach, the second one because it is arguably simpler and faster than the proposed method by the authors). 

[Hasson 2021] Probabilistic Forecasting: A Level-Set Approach. Hasson et al. Neurips 2021.
[Romano 2019] Conformalized Quantile Regression. Romano et al. Neurips 2019.


clarity,_quality,_novelty_and_reproducibility: The paper is very clearly written. The novelty is poor given the similarity to quantile conformal regression which is not mentioned in this work or even just fitting simple likelihood on top of existing models. The reproducibility is good, the author provided code that should allow one to rerun the benchmarks (which I did not check nor try to run).

summary_of_the_review: Given the lack of novelty and lack of comparison with relevant approaches (conformal quantile prediction, likelihood directly fitted on top of previous models, [Hasson2021]), I recommend rejecting the paper in its current state.

Additional details:
* p3: Salinas et al. (2020) does not consider only Gaussian noise model but also other likelihood (neg-binomial for instance)
* p4: "To train this model, we use maximum likelihood in a two-step manner, first training the mean model ... using MSE loss, and then ... maximizing the likelihood of the Gaussian observation." This seems to be the claimed novelty but a clear baseline would to fit both mean/variance or have a quantile regression on top of an existing architecture. This would have twice lower training runtime and latency and should probably be added in the comparisons (in addition to conformal prediction).
* Figure 7: this figure is hard to read since you are plotting samples for some methods and confidence intervals for others. It would be much better to report calibration to allow to quantify the calibration of the probabilistic predictions as you are doing for wind/nasdaq
* why reporting calibration only on wind/nasdaq on not on other datasets (electricity/traffic/etc)? I would recommend reporting calibration plots on all datasets, calibration errors would also be important to provide quantitative metrics
* Figure 9: small comment calibration are typically plotted with equal axis as it is then easier to assess the quality of the fit
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presented a simple approach and multiple experiments in order to support their main claim in the paper: with simple adaptations high performing deterministic models can be made into state of the art probabilistic forecasters.

The main issue pointed out by the reviewers is the technological novelty of the proposed simple adaptation approach. In reply, the authors did not object to this criticism, but instead they emphasised on the need of careful evaluation of the methodologies.

To be clear, most of the reviewers welcomed this type of study of testing whether we really need complex methods to make long-term time series forecasting work, for which I completely agree with them. Rock-solid works on empirical evaluations with good insights are welcomed for presentations at ICLR.

However, even by considering the contribution on large-scale empirical evaluation side, reviewers found the results to be promising but not complete. Feedbacks for improvements include: adding suitable baselines and making the comparison fairer, having a better narrative to connect the many different experiments, and show a coherent insight about why the simpler method can perform better.

I would encourage the authors to improve their presentation and organisation of the many experiments, and make suitable amendments regarding experimental comparisons following reviewers' recommendations.
justification_for_why_not_higher_score: While I think large-scale empirical studies should be welcomed at ICLR, these type of papers need to provide rock-solid results and good insights to be considered for acceptance. From reviews, unfortunately this paper, while promising, is still not quite there yet in its current form.
Decision: Reject