Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the tradeoff between primary model performance metrics such CRPS and CRPS rank vs other criteria such as inference latency. A surrogate model is proposed which ranks the various forecasting approaches and is used to predict the performance metrics and the prediction latency of the corresponding models. The surrogate model is used to sequentially evaluate the considered models and optimize the hyper-volume of the Pareto front. An interesting observation made in the paper is that classical models do not always beat deep methods even for relatively smaller datasets consisting of few thousand data-points. It is also shown that, as expected, there is a tradeoff between latency and performance.
main_review: **Strengths**
- This is an excellent work comparing a large and diverse set of time series forecasting approach wrt to model performance and prediction latency.
- The main contribution of this paper is empirical, presenting the latency and performance tradeoffs for various models. The study presented in this paper while not absolutely exhaustive, is still quite valuable to the community. Some ways to improve the study even further are suggested in the weaknesses section.
- The main observation made in the paper is that deep learning models are competitive or even better than classical models, and the performance gap increases only slightly with exponentially more data. This is an important observation as it breaks a common myth in the time series community. Although this statement may be a little too strong since classical models may still outperform deep models when the number of data points is less than a thousand. The datasets considered in the paper have at least a 1000 points.
- The experimental results seem convincing, and the datasets used in the paper are quite diverse and vary significantly in size, implying the robustness of the evaluation.

**Weaknesses**
- The choice of hyper-parameters (number of hidden layers, width, etc) for the various forecasting models has not been clearly mentioned. Were the models tuned in any way? or were default hyper-parameters used? This is the most significant weakness of this paper.
- Classical models may still outperform deep models when the number of data points is less than a thousand. The datasets considered in the paper have at least a 1000 points.
- It is unclear why a ranking model is used in place of a regression model. A more concrete justification should be provided for why ranking is equivalent to regression under quantile normalization.
- The argument behind using a parametric model over a non-parametric model is not completely clear. Non-parametric models such as Gaussian processes can also make use of dataset features with the right choice of a kernel. Although, GPs are not necessarily the best choice for ranking.
- The latency values in Table 1, do not include confidence intervals / stddevs (only required for the unconstrained models).


**Update after rebuttal:**
Thanks for the response. Some of my concerns have been addressed. However, I do also agree with the other reviewers that the organization of the paper and the presentation of the results can be greatly improved and hence I am keeping the same score. Details below:
- Hyper-parameters: Thanks for pointing to the appendix. Column 2 shows that different search sets for the context length have been used for different deep learning approaches. For a fair comparison, the same set of context lengths should be used since a larger context may provide an extra advantage  to the models.
- Number of data-points: This point needs to be properly addressed in the paper as well and the claims need to be re-evaluated.
- Ranking for model selection: My concerns have been fully addressed. It seems sufficiently well motivated to use parametric ranking models since the scales of the accuracy metrics can vary widely across problems. It is only feasible to rank the performance rather than predict the exact accuracy.
- Contributions in the paper: I do agree with the other reviewers about the presentation of the contributions and the organization of the paper. The fact that this paper also claims to choose default hyper-parameters for various problems, missed my attention when I was reading the paper for the first time. While the results in the paper are quite interesting, the organization can be greatly improved.
summary_of_the_review: Overall this a great piece of work comparing tradeoffs between multiple performance objectives for time series forecasting models, including classical and deep models. Some interesting observations have been shown in the paper which are mostly well justified. The main weakness is that the choice of hyper-parameters has not been explained, raising the concern of whether better hyper-parameter choices can improve the baseline results. I am willing to increase my score once the concerns have been addressed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a large-scale benchmark comparison of univariate distributional time series forecasting algorithms (7 local methods; 6 global methods, and for the latter, ensembles thereof) against 44 time series datasets that have been previously introduced in the literature. It also proposes an algorithm, ParetoSelect, to choose hyperparameter defaults that lie close to the Pareto frontier in a multi-objective evaluation (e.g. model accuracy and inference latency).
main_review: The paper addresses the important question of a large-scale assessment of distributional time series forecasting algorithms, across a wide range of benchmark datasets, comparing both local methods (classical statistical techniques) with global ones (mostly deep learning approaches). Such a large-scale comparison is missing from the literature, and represents a gap in our current understanding of which method best apply to which context. The paper restricts its focus to univariate assessment in discrete time without covariates, which is a reasonable starting point. 

In addition, the paper introduces a method to select hyperparameters and ensembles based on approximation of the Pareto front in a multi-objective setting. This approach is interesting and to the best of my knowledge is novel in a time series context.

The paper is reasonably well written and easy to follow, although the mathematical notation and writing style could be made more precise, as a few points are listed below. The authors are to be commended for providing full source code to reproduce experiments, and a cursory look suggests that the code is well written and easy to understand.

The main concern with the paper is that its focus is unclear as it tries to do both too much and too little:

- If viewed as a contribution to the large-scale benchmarking of time series forecasting algorithms, it offers a starting point but falls far short in offering substantive analysis of the presented results, e.g. to convey strong conclusions as to the suitability of particular methods for particular datasets, e.g. regarding time series properties such as variability, intermittency, seasonalities, etc., and possibly grouping benchmark datasets by broad area of study (e.g. retail, economics, energy, ...).
- If viewed as contributing the `ParetoSelect` algorithm, then this should come much earlier in the paper (in its current position towards the end, it feels like an afterthought), and deeper analysis of the results should be presented, along with a theoretical analysis if possible.

As written, the paper attempts to cover both objectives, but falls short of offering strong guidance or conclusions. It is therefore difficult to appreciate the paper's takeaways for the ICLR audience, and as such, it would require more work before recommending acceptance to the conference.

Detailed comments:
- Typographic remark: please put table captions above the table, and figure captions below the figure. (https://tex.stackexchange.com/questions/3243/why-should-a-table-caption-be-placed-above-the-table)
- p. 3 "three different context lengths that governs" ==> govern
- Section 3.4: since the forecast horizons and history length differ widely, isn't it better to use a *relative* latency measure compared to some simple benchmark (e.g. seasonal naïve)? Otherwise, the latency results are dominated by the datasets with the longest time series, regardless of the model performance per time step.
- Section 3.5: The results reported in Table 1 include rank information across an unknown set of models: that's obviously more than 7+6=13 introduced in Section 3.2, and seems to include some form of ensembling. However, how the ensembles are constructed has not yet been introduced by that point in the paper. 
- p.5 [Comparison of classical & deep learning methods]: the paper should explain how the test statistic is constructed exactly. Does the test include only the raw deep learning methods, or it includes ensembles as well? Can anything be learned from the datasets for which local methods outperform global methods apart from the number of examples in the dataset?
- p. 5: "only a few thousands observations seem sufficient to outperform the classical models considered": in Figure 1, there are datasets with >1e6 observations where local methods are indistinguishable from global methods, this is way more than "a few thousands". 
- p. 6 [top] it seems that $\mathcal{Q}_\mathrm{deep}$ and $\mathcal{Q}_\mathrm{class}$ represent different model results (best models of each class, respectively) than those given a few paragraphs above (set of all models within each class). It would help readability to have a more precise and coherent notation.
- Section 4.1: it is not clear if the $M$ different tasks represent different datasets or different objectives. And why does eq. (5) evaluates on the $M$ related tasks only, but not on the main task? The notation should be tightened up.
- Eq (12): it's not clear how sort is defined for vectors, and does not easily relate to the numbering of datapoints shown in Figure 3.
- The bibliographic entries should be checked for completeness. For instance Pfisterer et al. (2021) and Winkelmolen et al. (2020) are incomplete.
- p. 20: "we set each quantile to the forecasted value" ==> do you mean that $q_{0.5}$ receives all the probability mass and the others receive zero mass? 
summary_of_the_review: The paper attempts a valuable contribution to the univariate discrete-time forecasting literature, but by attempting to cover too many things quite superficially, does not appear ready to recommend acceptance at ICLR in current form.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper provides comprehensive evaluations of various time series forecasting models on 44 heterogeneous and public datasets. The obtained results, which are evaluated in multiple metrics, can serve as guidance for future researchers on choosing appropriate time series models for a given task. In addition, the authors have also studied how to obtain (or search) a set of default models that can provide good performance under single and multiple objective scenarios. For multi-objective optimization, a method called ParetoSelect is proposed to find the set of models that are in the Pareto front. Results have shown that ParetoSelect has outperformed other model selection methods.
main_review: **Strengths**: 

1. Thorough empirical studies on both classical and state-of-the-art forecasting models on different datasets provide insights to researchers and practitioners on choosing appropriate models. The result itself is valuable: e.g., comparison between classical and NN-based methods, global and local models, etc.

2. Learning models in a multi-objective setting is a common problem in many applications. Constructing the model configuration set based on the Pareto front is a natural solution and is proven to be effective over other baselines.

**Weakness**: 

1. The overall structure of this paper can be improved. For example, the paper title is ''Multi-Objective Model Selection ...'', but the authors start discussing this topic in the second half of this paper with limited context. This can easily cause misunderstanding to readers what is the major focus of this work.

2. Although the empirical study is an important component, there are still too many experimental details in the paper. It is better to elaborate on some employed methodologies. For example, instead of a plain citation, the authors could add brief introductions on the hypervolume error and $\epsilon$-net.

3. Since the property of each dataset varies a lot, the authors should also consider providing insights on which type of model is more suitable to datasets with certain properties, while the current paper aggregate the data information too much, and just provide general results on all datasets.

**Other Questions**: 

1. Section 4.2, what is the reason that the parameters of MLP are not tuned?

2. What are the layers in figure 3? Are these layers of MLP or others?

3. For models in the Pareto front, how to finally trade off each objective in the experiments?

4. Section 4.4, why are non-uniform weights not helpful in ensemble? Consider an extreme case where the model is just a naive average, then it is reasonable to assign less weight to this model compared with others. If the set of model candidates has already been very good, then the weight distribution should be close to uniform in the first place. Authors could add more clarifications on this matter.
summary_of_the_review: This paper has conducted a thorough empirical study on forecasting models, but it still needs some major modifications on paper structures and clarifications on methodologies before being accepted.

## Update after Rebuttal

We thank the authors for their detailed response in addressing questions and updating their manuscripts. The analysis of dataset characteristics in Appendix F is interesting but it does not appear to provide useful information that can serve as an insight in the future: this is fine on my side since it is not easy to discover patterns on massive datasets. 
Although the authors have also addressed the minor questions, we still think the structure of the paper can be further improved to make the paper more readable. The authors need some non-trivial modifications before it can be accepted. We wish the authors good luck in future revisions.
review_title: null

Meta-review: The premise of this paper is that the development of time series forecasting methods has traditionally focused on accuracy rather than other criteria such as training time or latency. This work presents a new benchmark, evaluating classical and deep learning-based methods on a number of public datasets. They also propose a technique, ParetoSelect that is able to select models from the Pareto front that can efficiently select models in a multi-objective setting. 

Reviewer XSBL liked the observation that classical methods do not always beat deep learning methods even for very small datasets. They thought that the empirical contribution was valuable and myth-breaking. They also commented that the evaluation was robust. Their main concerns were: inadequate description of hyperparameters, lack of evaluations on *really* small datasets, missing confidence measures for latency results. They also made some suggestions for improving clarity. The reviewers responded, pointing to a description of the hyperparameters in Table 2 of the appendix. They also responded to the reviewer’s comment about very small datasets and explained the advantage of the ranking loss. They made some small adjustments to the paper based on the clarity comments. 

Reviewer PZ2f also thought that the large scale comparison was valuable for the community. Overall they thought it was well written though could be improved w.r.t. Notation and writing style. They even inspected the code. Their primary concern was that the paper lacked focus and “tries to do too much and too little”. Is this a benchmarking effort of previous methods, or is the main contribution the ParetoSelect algorithm? This reviewer thought that due to its superficial coverage of too many things, and it wasn’t ready yet for publication at ICLR. The authors provided quite a comprehensive response to reviewer PZ2f and pointed to some minor improvements in the manuscript.

Reviewer rQb3, like the others, viewed the benchmark analysis as valuable. They thought that the ParetoSelect approach was “natural” and that it was shown to be effective over baselines. Like PZ2f they had some structural criticisms and pressed for more insights. 

Reviewers XSBL and rQb3 continued to engage in discussion through the AC-reviewer discussion phase. XSBL said that the authors’ response addressed some concerns yet raised others w.r.t. hyper-parameter selection. rQb3 updated their review after considering the author's response, feeling that minor concerns were addressed but the paper could still use further development. Overall, after considering the discussion I think that it’s been difficult for the authors to provide any patterns regarding which model performs best for which datasets. To me, a benchmark paper should provide some deeper insight and the paper appears to be struggling to do that. On the other hand, the study is comprehensive. The authors have argued in their response to all reviews, that their evaluation is at quite a different scale compared to other published time series model evaluations. I think that this benchmark paper can provide value to the community yet it could use further work: specifically the authors need to focus the paper and communicate clearer insights from the study.
Decision: Reject