Venue: AutoML 2023 ABCD_Track

Reviews:

final_score: rating: 8: Accept: Technically sound paper with major impact, with perhaps some minor flaws.
review_content: summary_of_contributions: The paper presents an open-source AutoML library for probabilistic time series forecasting.  Only with three lines of code, the library is able to generate accurate point and quantile forecasts.
overall_review: Positive: Nice presented methodology, with a lot of arguments and clarity on how it works, arguments for which methods are implemented, and enough details of explaining the results achieved.

Negative: In the future, please also test them on more publicly available repositories of datasets.
review_summary: The paper is technically sound, explaining the concepts behind the pipeline and further providing arguments for all methods included in the modeling part. The paper is well-written and structured. It is easy to follow and the discussions are supporting the experimental results. The evaluation results are nicely presented.
actions_required_to_increase_overall_recommendation: Adress some above-mentioned issues.
review_title: null

final_score: rating: 9: Strong Accept: Technically flawless submission with major impact, with no obvious flaws. Should be highlighted in the program.
review_content: summary_of_contributions: 1. The paper introduces AutoGluon-TimeSeries, an open-source AutoML library for probabilistic time series forecasting. It can generate both point and probabilistic forecasts for collections of univariate time series. AutoGluon-TimeSeries can also work with static and time-varying covariates, making it applicable to most real-world forecasting tasks.
2. The framework focuses on ease of use and robustness. It empowers users with limited expertise in the target domain to generate accurate point and quantile forecasts with just 3 lines of Python code. The architecture can handle failures of individual models when necessary, producing a valid result as long as any single model is trained successfully.
3. AutoGluon-TimeSeries combines conventional statistical models, machine-learning-based forecasting approaches, and ensembling techniques to achieve high accuracy within a short training time. It incorporates local models (e.g., ARIMA, Theta, and ETS) and global models (e.g., DeepAR, Temporal Fusion Transformer, LightGBM, and CatBoost) from both conventional statistical methods and deep learning-based approaches.
4. The paper presents design considerations, including ensembles over hyperparameter optimization (HPO), presets and default hyperparameters, and model selection for time series forecasting.
5. The paper evaluates the performance of AutoGluon-TimeSeries against other established forecasting methods and AutoML systems using 29 publicly available benchmark datasets. The results demonstrate the framework's strong performance, outperforming various competing approaches in terms of both point and probabilistic forecast accuracy.
6. The authors discuss future work, including exploring advanced ensembling strategies, improving calibration, supporting new problem types, and enhancing the scalability of the framework to handle larger-scale forecasting tasks.
overall_review: 1. The paper presents a valuable contribution to the AutoML domain by introducing AutoGluon-TimeSeries, an open-source AutoML library for probabilistic time series forecasting.
2. It can generate both point and probabilistic forecasts for collections of univariate time series. AutoGluon-TimeSeries can also work with static and time-varying covariates, making it applicable to most real-world forecasting tasks.
3. AutoGluon-TimeSeries emphasizes simplicity and robustness. It is designed to be user-friendly, enabling users with limited domain expertise to generate accurate predictions with minimal coding effort.
4. AutoGluon-TimeSeries combines conventional statistical models, machine-learning-based forecasting approaches, and ensembling techniques to achieve high accuracy within a short training time.
5. The authors have demonstrated the effectiveness of AutoGluon-TimeSeries through extensive experiments on 29 benchmark datasets, showing that it outperforms other established methods and AutoML systems in terms of both point and probabilistic forecast accuracy. These results highlight the value of the proposed library as a powerful tool for practitioners and researchers in the field of AutoML.
6. The paper is well-written and organized, with clear explanations of the proposed methods, the experimental setup, and the results. This makes the paper accessible to a broad audience of researchers and practitioners.
7. The open-source nature of AutoGluon-TimeSeries can foster collaboration, inspire new ideas, and contribute to the growth of the AutoML community.
review_summary: The paper presents a comprehensive approach to probabilistic time series forecasting by introducing AutoGluon-TimeSeries, which leverages ensembles of diverse forecasting models to deliver high accuracy within a short training time. The paper demonstrates strong empirical results on 29 benchmark datasets, outperforming existing state-of-the-art approaches in terms of accuracy and computational efficiency. Furthermore, the paper is well-written and organized, making it accessible to a broad audience. The authors provide an open-source implementation of their method, which can promote further research and collaboration. Given the paper's positive aspects and potential for impact, I recommend its strong acceptance for publication.
actions_required_to_increase_overall_recommendation: One Minor Correction:
There is a typo in the first paragraph on Page 5. The word "first" is repeated twice in the sixth line.
review_title: null

final_score: rating: 8: Accept: Technically sound paper with major impact, with perhaps some minor flaws.
review_content: summary_of_contributions: The paper presents AutoGluon-TimeSeries, a library for probabilistic time series forecasting that supplements Amazon Web Services' AutoGluon ML framework. The library includes several statistical and machine-learning models for time series forecasting as well as other established methods. An experimental study and comparison with competing, state-of-the-art frameworks demonstrates very good and often top rankings for AutoGluon, both in terms of quality and runtime.
overall_review: This is a significant paper that has the potential to advance the field of AutoML in the specific domain of probabilistic time series forecasting. The proposed library is based on best principles, comes with carefully chosen presets for the unexperienced user, and is thoroughly compared to the state of the art. Overall, the experimental data place it usually at the top of the field in terms of quality and runtime, except for some simple tasks that are solved faster with simple statistical models. Since my knowledge in the specific field is limited, I am a bit in doubt about the future impact of the contribution.
review_summary: An essential flawless contribution adhering to all principles for an AutoML application paper. I expect the framework to have moderate to high impact on the field.
actions_required_to_increase_overall_recommendation: Elaborate on the impact of the software and describe its potential user base more clearly. Also, it would be interesting to discuss the role of purely quantitative time series forecasts, which is presently the main purpose of the software, and contrast it with qualitative predictions that might be achieved by modern ML techniques, including large language models.
review_title: null

final_score: rating: 9: Strong Accept: Technically flawless submission with major impact, with no obvious flaws. Should be highlighted in the program.
review_content: summary_of_contributions: The paper presents the part of the AutoGluon library dedicated for time series prediction. It targets the "A" part of the ABCD track and specifically highlights its ease of use. Its main working principles are ensembling of different models, ranging from statistical inference to neural networks, with an emphases on avoiding on-the-fly configuration during fitting. The paper features an experimental study of the library that includes several state-of-the-art time series prediction models, which it claims to outperform, and also an ablation study that tears off the components and sees what happens, which claims to show the necessity of most of the employed components.
overall_review: + The paper presents a nice part of their library that is designed to perform state-of-the-art time series predictions using three (not overly long) lines of Python code.
+ The experimental comparison shows that it outperforms the competitors more often than not, while it is arguably easier to use.
+ The paper is written quite nicely, with only a little interventions remaining to improve the overall quality.
+ The ablation study is performed to evaluate the importance of the main components of the architecture of the proposed approach.
- The results of that ablation study were not evaluated for statistical significance, which is highly desired given how similar some ablated configurations are to the main one.
- ~The preset derivation procedure was not disclosed, which makes it complicated to judge whether these presets are not overfitted to the benchmarks employed in the comparison.~
review_summary: I find the current state of the paper to fit the description of a "technically sound paper with major impact, with perhaps some minor flaws". Due to its shape, it is likely to have a major impact on the end-users, and it will probably alter some default design decisions of the research in the AutoML community. ~The minor flaws are listed elsewhere in the review, and I believe that they are easily fixable, with a possibility to improve the final score to Strong Accept.~

actions_required_to_increase_overall_recommendation: The most important thing to increase the overall recommendation is to address the two issues (statistical testing in the ablation study, and ~explanation of the preset derivation procedure~) that are detailed in "Technical Quality and Correctness".
review_title: null

final_score: rating: 9: Strong Accept: Technically flawless submission with major impact, with no obvious flaws. Should be highlighted in the program.
review_content: summary_of_contributions: This paper introduces a new algorithm for forecasting time series, called AutoGluon–TimeSeries (AG–TS). This algorithm is capable of predicting how a provided times series continues in two different ways: (1) by providing single points for each new time point of the forecast, and (2) by providing a probabilistic forecast, where the algorithm provides the user for each new time point of the forecast with the values for a specified range of quantiles (say, from 0.1 to 0.9 in steps of 0.1) that determine the likelihood of a new data point to lie within this range.

AG–TS provides a flexible interface with many reasonable default values such that it can be used with very few lines of code and without any expert knowledge about the domain of the time series it is being applied to.

The authors test this approach against various existing approaches, both in terms of solution quality and running time.
overall_review: This paper presents a very nice and potentially very impactful framework for forecasting time series. The framework is easy to use, and its performance against competing approaches is very good, as showcased by thoroughly executed experiments. My only concerns are that related work is not very well discussed and that some of the experiments are repeated a seemingly very small number of times. This makes it a bit harder to properly assess and appreciate the contribution of this paper.
review_summary: This paper is very well written and presents a very nice piece of code. The tests appear sound to me, and the performance of the new framework is very good (and better than the competition). Hence, I highly recommend to accept this paper.

EDIT: The authors responded to my suggestions satisfyingly. Hence, I do not decrease my score. However, I also do not increase it, as the current score is already very high and I do not think that the result is strong enough to receive a 10.
actions_required_to_increase_overall_recommendation: My points that I mention in “Clarity” should be addressed. Further, the authors should at the very least comment in the paper on why they believe that three iterations of the experiments for the approaches with random models is a sufficiently large number.
review_title: null

Decision: Accept