Venue: AutoML 2022 Track Main

Reviews:

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: The authors propose a new collection of benchmarks for HPO containing 14 scenarios (700 instances). The key focus is on multi-fidelity problems and hierarchical search spaces. Although it is not clear how many of these problems exhibit these features. The library also contains multi-objective problems (which is deemed important by the authors).

The benchmarks are surrogate-based, provided using the ONNX format, and have low memory requirements.
overall_review: While properties of the scenarios are hinted at in some places, there is no qualitative discussion of the different scenarios and instances. The benchmarks are listed mainly, e.g., in a table in a quantitative way. A large number of benchmark problems (and libraries) is not always better, as it is easy for important conclusions to be hidden among many results. For that reason, it's important for benchmark libraries to give more details about the problems it includes.
	
Readers would benefit from discussing (a subset of) the different benchmarks in terms of difficulty: complexity of the search space, non-stationarity, what type of algorithms are expected to do well, etc.

For surrogate-based benchmarks the ability to capture the fine detail and stochasticity of the problem is crucial. Figure 2 provides insights into how the surrogate performs, but it could use a more thorough validation of the surrogate itself. In the text, tests for Branin2D, Hartmann3D, and Currin2D are announced, but I could not find any results in the paper (in the figure or tables).
review_summary: I consider the efficient memory requirements and runtime execution of the benchmarks to be the biggest advantage. While there are many benchmark libraries, having a fast surrogate-based one is useful for the community. Although I'd prefer to extend existing frameworks I can see YAHPO-GYM being useful. However, I could not run the experiments of the paper (the link is not working). Together with the missing references (??) and benchmarks (Branin, Hartman3D,...), it is hard to recommend the paper for acceptance
review_title: null

final_score: rating: 2: Reject, not good enough
review_content: summary_of_contributions: The paper proposes YAHPO-Gym, a surrogate-based benchmark for (multi-objective) multi-fidelity hyperparameter optimization problems with 14 configuration spaces consisting of multiple instances to facilitate transfer learning. The authors empirically address the performance of several multi-fidelity approaches compared to random search, as well as the quality of advanced multi-fidelity methods. 
overall_review: Check review summary.
review_summary: The paper proposes YAHPO-Gym, a surrogate-based benchmark for (multi-objective) multi-fidelity hyperparameter optimization problems with 14 configuration spaces consisting of multiple instances to facilitate transfer learning. The authors empirically address the performance of several multi-fidelity approaches compared to random search, as well as the quality of advanced multi-fidelity methods. 

**Strengths**

- provides an easy-to-use GYM environment with 14 benchmarks for multi-objective multi-fidelity HPO.

- The authors seized the opportunity to benchmark some multi-objective Bayesian optimization solutions as well as multi-fidelity methods. 


**Weaknesses**

- The diversity of the instances for each benchmark collection is not studied. One suggestion would be to visualize the empirical cumulative distribution of the underlying tabular datasets.

- I would be interested to hear more about why the authors chose a deterministic surrogate benchmark instead of a surrogate that provides some uncertainty quantification.

- YAHPO-GYM does not contribute any new benchmarks, but rather provides a wrapper for existing ones.

I think the paper has slightly improved  over the previous submissions, especially with regards to benchmarking the baselines. However, the paper seems to be a not complete with several missing references that makes it difficult to follow through in a self-contained fashion.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: This paper proposes a benchmark tool for hyper-parameter Bayesian optimization. It includes many problems where one can test a Bayesian optimization method. Including multi-fidelity and multi-objective problems. It is based on using a surrogate model to predict the unknown target function. This model is a neural network that has been trained with a large number of instances of the original problem. The framework is fast and memory efficient.

overall_review: Overall I think that this is an interesting paper that proposes a tool that the community will most likely find useful.

review_summary: A nice paper providing a useful tool for benchmarking BO methods in the task of finding good hyper-parameters. The utility of the framework is illustrated with several experiments.
review_title: null

final_score: rating: 6: Strong accept, should be highlighted
review_content: summary_of_contributions: This paper presents the YAHPO gym system as a benchmarking tool for multi-objective and multi-fidelity hyperparameter optimization. This benchmark makes use of surrogate functions instead of real evaluations or tabular benchmarks as a compromise between the computational cost of real evaluations and the bias introduced by the tabular benchmarks that rely on explicit discretization. This benchmarking tool contains problems with a wide range of search spaces aand also provides the ability to evaluate transfer HPO algorithms. One significant feature of this benchmark (over existing benchmarks) is the efficiency in terms of time and memory overhead which is obtained via the use of ResNet based compressed neural network surrogates. This allows for a faithful evaluation of various forms of hyperparameter optimization algorithms with reduced overall computational cost, allowing for efficient evaluation between algorithms with multiple repetitions, making it easy to get reliable statistically significant comparisons.

overall_review: Positive aspects:
- The proposed system contains various scenarios that allow one to evaluate various aspects of any HPO algorithm efficiently with low memory overhead (compared to existing benchmarks).
- The paper is very well written and easy to follow, properly motivating and describing the different choices for the system and justifying them empirically and providing sufficient amount of details in the supplement.


Negative aspect:
- In my opinion, there are no negative aspects to this paper.

review_summary: This is a very strong HPO benchmarking system covering all relevant aspects of the HPO problem, and provides a more faithful yet more efficient (in terms of time and memory overhead) comparison between HPO algorithms. For this reason, I strongly recommend the acceptance of this paper.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: N/A for reproducibility reviewers
overall_review: YAHPO_GYM is clearly presented with every small detail is nicely covered. The reproducibility list is filled out fully, the answers are clearly justified. The results are adequately provided. Installation is a slightly challenging task as it requires cloning the second github repo. However, it is totally justifiable. Following the instructions provided by the authors, the user can easily reproduce the examples. Integrating YAHPO_GYM in other algorithms should be straightforward. For example, the example with BOHB provides clear instructions. In general, one needs to be familiar with ConfigSpace. Overall, it is easy to use YAHPO_GYM.
review_summary: N/A for reproducibility reviewers
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: This paper describes an efficient multi-objective multi-fidelity benchmark framework for hyperparameter optimization, which is referred to as YAHPO Gym.  In particular, YAHPO Gym is a surrogate-based benchmark for hyperparameter optimization.  Since a tabular benchmark may reduce a bias in performance estimation and ranking of hyperparameter optimization methods.  Moreover, this YAHPO Gym allows us to use multi-objective and multi-fidelity settings.  Finally, the authors demonstrate the comparisons between YAHPO Gym and other available implementations.
overall_review: I agree with the motivation of this paper.  It provides an interesting benchmark, in order to answer to the motivation.  It will be a potentially impactful project.

I have a question on the multi-fidelity setting.  How did you control the degree of fidelities?  It is not described in the paper.  I guess you did control the size of training data or any others?

Also, it is a minor issue, but could you provide how to pronounce YAHPO?
review_summary: Since this paper provides a solid benchmark for hyperparameter optimization including the settings with multiple objectives and multiple fidelities, it is novel and interesting.  The authors describe why we need this benchmark and how it will be used thoroughly.  Therefore, I would like to recommend the acceptance of this work.
review_title: null

Meta-review: metareview: This work proposed a new HPO surrogate benchmark for 14 scenarios and filled the gap of benchmarks in the multi-objective and multi-fidelity setting. The paper is clearly written and conducted experiments to demonstrate its usefulness. It also provided analysis on several interesting questions such as tabular vs surrogate; multi-fidelity vs full-fidelity. It is open sourced and being integrated into a larger benchmarking framework (HPOBench). I think it will make a good contribution to the HPO community.


confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept