Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 8: Accept: Technically sound paper with major impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: In this work, the authors have introduced a new setting called "ordered transfer HPO", which is basically using or taking into consideration the "order" of tuning tasks during the transfer HPO process. The results obtained by the new approach OTHPO show to be promising and can outperform the standard transfer HPO methods. However, this new approach was not tested on a wide range of ML models and also needs to be tested on other datasets in the future.
overall_review: The contribution in this paper gives a new and promising view on the tuning of hyperparameters optimization problems. The results show a competitive/ outperforming performance for the proposed approach, however, it still needs to be tested on more datasets and more ML models to study better its performance for AutoML in a general view.
review_summary: I recommend this paper to be accepted for AutoMLconference because it proposes an interesting and high-quality work but also brings an interesting approach to the field of HPO.
actions_required_to_increase_overall_recommendation: Please address the recommendation mentioned previously.
review_title: null

final_score: rating: 5: Borderline Leaning Reject: Technically sound paper where reasons to reject nonetheless outweigh reasons to accept. Please use sparingly.
review_content: summary_of_contributions: This paper introduces Ordered Transfer Hyperparameter Optimization (OTHPO), a version of transfer learning for hyperparameter optimization (HPO) where tasks follow a sequential order. This paper proposes a formal definition, outlines the differences to related problems, and presents a basic OTHPO method that outperforms state-of-the-art transfer HPO. The authors investigate gradually accumulating data and empirically show the importance of taking order into account in ten benchmarks spanning XGBoost, random forest, approximate k-nearest neighbor, elastic net, support vector machines, and a separate real-world motivated optimization problem. The benchmarks are open-sourced to foster future research on ordered transfer HPO. This paper provides guidance for HPO practitioners in a deployed system.
overall_review: Positive aspects of the paper:

Novelty and practical relevance: The paper introduces the concept of ordered transfer hyperparameter optimization (OTHPO) that considers the sequential nature of hyperparameter tuning in practical scenarios where data is collected over time. This is a novel idea that has practical relevance in real-world machine learning systems.

Empirical evaluation: The authors provide empirical evidence to support their proposed OTHPO method by evaluating it on ten benchmarks spanning various machine learning methods such as XGBoost, support vector machines, random forest, approximate k-nearest neighbor, elastic net, and a real-world motivated optimization problem. They show that OTHPO outperforms state-of-the-art transfer HPO methods and highlight the importance of considering the order of tasks in hyperparameter optimization.

Negative aspects of the paper:

Limited Impacts: While the intuition is straightforward and reasonable, it doesn't provide many useful conclusions for practical usage. 

Lack of theoretical analysis: Although the paper provides a formal definition of OTHPO and highlights its differences from related problems, such as standard transfer HPO and continual learning, it does not provide any theoretical analysis of the method. A theoretical analysis could help provide a deeper understanding of why and how the proposed method works, and under what conditions it is expected to perform well or poorly.


review_summary: The empirical evaluation validates the methods. A deeper analysis from the theoretical aspect and exploration of the conclusion would further improve the paper.
actions_required_to_increase_overall_recommendation: See Overall Review
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak impact, and/or weak evaluation.
review_content: summary_of_contributions: In this paper, the authors tackle the issue of ordered hyperparameter optimization, which arises when there are changes in the underlying dataset or environmental variables, specifically data accumulation. 

To address this issue, the authors propose a straightforward solution that involves using the top-performing hyperparameters from previous tasks, where the dataset was smaller than the current one, to either meta-initialize Bayesian Optimization (BO) on the new task or to use these hyperparameters as a starting point for BO. The effectiveness of this approach is evaluated on three benchmarks with a total of 10 datasets and several model classes.

The experimental results demonstrate that selecting the best-performing configurations as the data accumulates is an efficient method to obtain good results. However, the authors acknowledge that there is no universal method that is preferred after five evaluations, as it depends on the set of tasks being considered.
overall_review: The paper has several notable strengths:

- The authors tackle a practical use case and propose a straightforward solution that is effective and efficient. The proposed solution addresses the problem of hyperparameter optimization of a pre-defined model when the dataset changes over time. 

- The paper is well-written and presented in an easily understandable manner.

- The results are well-visualized and straightforward to interpret. 

However, there are some negative aspects to the paper:

- The proposed idea is very simplistic and does not yield conclusive results. The authors merely document an empirical reference that aligns with practitioners' initial intuitions of reusing top-performing configurations from previous tasks.

- It is unclear how the hyperparameters of the first task are selected. If they are chosen randomly, then it is not clear why the performance of SimpleOrdered and SimpleOrderedShuffled are not the same at task 2 in Figure 5, especially since it reports the performance at the first iteration.

Additionally, there are some concerns:

- Based on the revised problem definition, it is concerning that TransferBO, which extends the domain of the surrogate model along an additional sequence dimension, does not show reliable results against SimpleOrdered and its variants.

- The authors mention that they reran the experiments for 50 seeds, which usually implies selecting new random configurations to initialize BO methods. However, the authors provide no further information about the source of randomness. A better way to validate the approach might be to introduce another source of randomness that affects the instances of the dataset at each task.

- The evaluation protocol is limited to only 25 hyperparameters, which is lower than the conventional value. 

- Additionally, the baselines used in the paper are not very recent, and there are more up-to-date approaches available in the literature, such as those presented in references [1-2], that have demonstrated good transfer learning results.

- Simpler baselines are also not evaluated, although addressed and referenced [3]. 

References: 

1. Wistuba, Martin, and Josif Grabocka. "Few-Shot Bayesian Optimization with Deep Kernel Surrogates." International Conference on Learning Representations. 2021.

2. Jomaa, Hadi Samer, et al. "Transfer learning for bayesian hpo with end-to-end landmark meta-features." Fifth Workshop on Meta-Learning at the Conference on Neural Information Processing Systems. 2021.

3. Feurer, M., Springenberg, J., and Hutter, F. (2015). Initializing Bayesian hyperparameter optimization 401
via meta-learning. In Proceedings of the AAAI Conference on Artifcial Intelligence, volume 29.

review_summary: The proposed problem setting is intriguing and casts transfer learning in a new light. However, the proposed approach is rather simplistic, and the authors do not provide conclusive recommendations. The idea of reusing the best hyperparameters, either from all previous tasks or only the immediate previous one, to initialize BO is aligned with the intuition of many practitioners.
actions_required_to_increase_overall_recommendation: The proposed solution is compared to transfer learning solutions designed for another context, which is interesting but puts the approach at a disadvantage in terms of empirical findings. It would be valuable to evaluate the recommended approach, SimpleOrdered, against other meta-initialization techniques, such as those that leverage meta-feature similarity, and simply random initialization for GP. Additionally, SimpleOrdered can be coupled with several other surrogate models aside from GP to concretely demonstrate its superior performance.
review_title: null

Decision: Accept(Workshop)