Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes a benchmark for multi-fidelity HPO including a diverse set of tasks (different search space, surrogate, previous work, and new tasks). This benchmark alleviates the issue of reproducibility, open-source, consistency as in previous efforts. An empirical comparison of different HPO algorithms is conducted and authors ensure that advanced methods outperform random search and that multi-fidelity algorithms are beneficial.
strengths: •	Unified API for different tasks
•	Containerization for reproducibility
•	Collection of diverse tasks
•	Detailed experiments of different methods

weaknesses: •	The definition 1 of HPO benchmark should be more clarified.
•	Table 1 is a bit hard to follow. Kindly consider a better format
•	As by empirical experiments, it seems current algorithms from the community are good enough on this benchmark and there is limited space to improve, thus the value of the benchmark is doubted. Kindly consider the value of the benchmark itself.

correctness: Yes
clarity: The paper is mostly clear. Some points are mentioned in weaknesses.
relation to prior work: The authors discuss previous benchmark attempts for HPO.
additional feedback: No additional feedback
review_title: HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: &nbsp;

The paper introduces a hyper-parameter optimisation (HPO) benchmark with a particular focus on multi-fidelity HPO. The authors collate 7 existing benchmarks and additionally supplement these with 5 new multi-fidelity benchmarks. Importantly, the benchmarks are containerised and as such, are particularly attractive to practitioners seeking to compare HPO algorithms in the same programming environment.

&nbsp;

I believe the benchmark and associated codebase should serve as a practicable platform to compare HPO algorithms and as such I am leaning towards acceptance. I have a few concerns, which if addressed in the rebuttal, may warrant an increased score.

&nbsp;
strengths: &nbsp;

1. There is a particularly strong effort to ensure both reproducibility and comparability of HPO algorithms in the associated codebase. Particularly, the containerisation paradigm should facilitate HPO algorithm comparison to be performed in the same programming framework.

&nbsp;

2. The codebase for the benchmark is well documented, appears to be actively maintained, and seems to already be garnering interest from the broader HPO community.

&nbsp;
weaknesses: &nbsp;

## __MAJOR POINTS__

&nbsp;

1. In terms of the codebase documentation, it may be beneficial to include a **representative example of how a user may define a custom algorithm** and benchmark its performance.

2. In the related work section, it may be worth mentioning the Sherpa library [1] as well as the Olympus benchmark for optimisation algorithms [2].

3. It may be worth adding a note to distinguish between multi-fidelity HPO and multi-fidelity BO [3, 4, 5] so that BO researchers don't get confused!

4. It is not abundantly clear to me how the ECDF plots in Figure 3 emphasise the diversity of the new benchmarks. As I understand each line corresponds to a different dataset and hence what the plots show is that a given model's performance varies across datasets. It would be good to state exactly **what kind of diversity is implied** e.g. is the implied conclusion that the black-box functions have differing forms for these tasks?

5. For research question 2 it would be nice if the **nature of the performance improvement could be defined more precisely**. Presumably improvement is meant in terms of wallclock time as opposed to sample efficiency? It would be nice if this was stated explicitly on introducing the question.

6. In answering research question 1 the authors state that, "BO improves over the evolutionary algorithm DE in the beginning but loses to it in the very long run on the existing community benchmarks". In order to support claims such as this, it would be more meaningful to run the BO algorithm that won the NeurIPS 2020 Black-Box optimisation [6](https://github.com/huawei-noah/noah-research/tree/master/BO/HEBO) in place of the standard BO scheme. Otherwise, I would recommend **amending the claim to emphasise that the BO algorithm used is not state-of-the-art and additionally referencing the fact that more performant BO algorithms exist for this purpose [6].**

&nbsp;

## __MINOR POINTS__

&nbsp;

1. It would be nice to have the references in numbered order.

2. Abstract: "To demonstrate the broad compatibility of HPOBench", compatibility with what?

3. The figure axes and font size are quite small rendering them difficult to read. No doubt this can be amended in the camera-ready version for which an extra page is permitted.

4. Reference 51 has no journal. Reference 55, has no website.

5. line 247-249: It may be worth revising this sentence.

6. For Table 2 of the main paper as well as Tables 5 and 6 in the appendix, some row entries only have a single underlined figure. According to the paper the underlined results are not statistically different according to a two-sided Wilcoxon signed-rank test. As such, I would expect either no entries to be underlined or at least two entries per row to be underlined?

7. In Section G of the appendix figures are referenced in non-ascending order.

8. It may be worth supplying the GitHub link in the abstract directly.

9. It is not clear to me what rep. is an abbreviation of in Table 1.

&nbsp;

## __REFERENCES__

&nbsp;

[1] Hertel et al. Sherpa: Robust hyperparameter optimization for machine learning. SoftwareX. 2020.

[2] Hase et al., Olympus: A benchmarking framework for noisy optimization and experiment planning. Machine Learning: Science and Technology. 2021. 

[3] Wu et al., Practical multi-fidelity Bayesian optimization for hyperparameter tuning. UAI. 2020.

[4] Song et al., A general framework for multi-fidelity Bayesian optimization with Gaussian processes. AISTATS. 2019.

[5] Kandasamy et al., Multi-fidelity Bayesian optimisation with continuous approximations. ICML. 2017.

[6] Cowen-Rivers et al. An Empirical Study of Assumptions in Bayesian Optimisation, arXiv.  2021.

&nbsp;
correctness: &nbsp;

To the best of my knowledge the claims made by the authors are correct with the exception of a single claim that may warrant amendment as discussed above.

&nbsp;
clarity: &nbsp;

The paper is relatively easy to follow.

&nbsp;
relation to prior work: &nbsp;

The suggested references to be included are given above.

&nbsp;
additional feedback: &nbsp;

All points covered in the main response.

&nbsp;

----------

Update post-rebuttal: The authors have gone to great lengths to address all concerns raised by the reviewers and I have upgraded my score to 7 to reflect the improvements made to the submission!
review_title: Practicable Benchmark for Hyper-parameter Optimisation

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper proposes HPOBench, a collection of 100+ multi-fidelity hyperparameter optimization problems (HPO) from 12 different benchmark families. The authors highlight the importance of low-fidelity approximations compared to the standard black-box optimization problems, with the latter being a specific use-case of the former, and detail some of the existing benchmarks for HPO.  All HPOBench benchmarks are containerized for single task evaluation. In addition to 7 pre-established benchmark families, the authors create 5 new benchmarks and evaluate the performance of 13 optimization methods in a large-scale study, proving once again that advanced methods are better than random search, whereas multi-fidelity algorithms are better than black-box methods with a representative low-fidelity representation of the black-box function.
strengths: - presents a collection of diverse (multi-fidelity) hyperparameter optimization problems that span traditional HPO problems as well as neural architecture search, in raw/tabular/surrogate form.
- benchmarks are provided as easy-to-use containers
- large-scale evaluation of optimization algorithms
weaknesses: -  The benchmarks are presented in raw and tabular form, where hyperparameters are discretized on a grid. It would be important to define the range of the grid. An additional column to Table 1 with "# configuration" could be helpful instead of being the appendix.

- It would seem that there is missed opportunity on comparing more advanced black-box methods, e.g. [1],[2], with multi-fidelity methods. 

- Another important aspect for HPO benchmarks is transfer learning, which is overlooked in this paper. With increasing research in the community in that direction, the authors can shed some light on the problem and how HPOBench be leveraged towards that end.

References;
[1] Snoek, Jasper, et al. "Scalable bayesian optimization using deep neural networks." International conference on machine learning. PMLR, 2015.
[2] Springenberg, Jost Tobias, et al. "Bayesian optimization with robust Bayesian neural networks." Advances in neural information processing systems 29 (2016): 4134-4142.
correctness: There are no details offered about the datasets on the 5 new benchmark families:
- What was the training split used on the data
- What is the reported metric, e.g. cross-validation accuracy, mean squared error , etc. ? 
- L227: what exactly is 100x average runtime ? 

How many points where used to fit the initial surrogate for single-fidelity BO methods ?
clarity: Clear
relation to prior work: Authors overlook relevant literature for black-box HPO [3]

[3] Arango, Sebastian Pineda, et al. "HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenML." arXiv preprint arXiv:2106.06257 (2021).
additional feedback: No
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper proposes a set of APIs and new benchmarks for Hyper Parameter Optimization (HPO). Running two different classes of HPO algorithms on the dataset they are able to confirm results previously reported in the literature. 
strengths: * The paper organizes many existing benchmarks into a single framework for testing/evaluation. 
weaknesses: * The work does not appear to answer any questions that were previously unknown, or to enable future work to answer those questions. 
* The many figures are small, and pixelated, making them hard to read. 
* Contributed new benchmarks seem to have no consideration as to why they were selected. 
correctness: The Wilcoxon signed-rank test appears to have been applied incorrectly. It appears the authors in Table 2 are using the test to compare the results of multiple runs as the "rows" in the test, and appear to be performing multiple Wilcoxon tests, one for each row and (winning) column of the table. As both [a,b] note, the proper approach would be that the ranks are obtained per dataset, not per run (one could in this case select the mean score to determine rank) and then apply a non-parameter ANOVA such as the Friedman test to determine significance. 

It is also unfortunate that the paper does not take into account it's citation of [13] that the HPO algorithm itself introduces significant noise into the process, as well as the dataset selection. A more robust improvement / tool for benchmarking that accounts for this and looks at the variance of results would have been desirable. 

The matter of "reproducibility" in the paper has been a bit too oversimplified. Unfortunately there is inconsistent behaviors on which replication/reproduction means what[k], but many prior works have looked at such "re run the same code" style replication and are not cited/credited [c,d,e,f], and that this is not a panacea and can still lead to replication/reproduction issues is also not discussed/accounted for [g,h]. 

a. Demšar, J. (2006). Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research, 7, 1–30. Retrieved from http://dl.acm.org/citation.cfm?id=1248547.1248548

b. Benavoli, A., Corani, G., & Mangili, F. (2016). Should We Really Use Post-Hoc Tests Based on Mean-Ranks? Journal of Machine Learning Research, 17(5), 1–10. Retrieved from http://jmlr.org/papers/v17/benavoli16a.html

c. Forde, J., Head, T., Holdgraf, C., Panda, Y., Perez, F., Nalvarte, G., … Sundell, E. (2018). Reproducible Research Environments with repo2docker. In Reproducibility in ML Workshop, ICML’18.

d. Kluyver, T., Ragan-Kelley, B., Pérez, F., Granger, B., Bussonnier, M., Frederic, J., … development team, J. (2016). Jupyter Notebooks - a publishing format for reproducible computational workflows. In F. Loizides & B. Scmidt (Eds.), Positioning and Power in Academic Publishing: Players, Agents and Agendas (pp. 87–90). IOS Press. Retrieved from https://eprints.soton.ac.uk/403913/

e. Callahan, B., Proctor, D., Relman, D., Fukuyama, J., & Holmes, S. (2016). REPRODUCIBLE RESEARCH WORKFLOW IN R FOR THE ANALYSIS OF PERSONALIZED HUMAN MICROBIOME DATA. Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing, 21, 183–194. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/26776185

f. Claerbout, J. F., & Karrenbach, M. (1992). Electronic documents give reproducible research a new meaning. In SEG Technical Program Expanded Abstracts 1992 (pp. 601–604). Society of Exploration Geophysicists. https://doi.org/10.1190/1.1822162

g. Raff, E. (2019). A Step Toward Quantifying Independently Reproducible Machine Learning Research. In NeurIPS. Retrieved from http://arxiv.org/abs/1909.06674

h. Bouthillier, X., Laurent, C., & Vincent, P. (2019). Unreproducible Research is Reproducible. In K. Chaudhuri & R. Salakhutdinov (Eds.), Proceedings of the 36th International  Conference on Machine Learning (Vol. 97, pp. 725–734). Long Beach, California, USA: PMLR. Retrieved from http://proceedings.mlr.press/v97/bouthillier19a.html

i. Drummond, C. (2009). Replicability is not reproducibility: nor is it good science. In Proceedings of the Evaluation Methods for Machine Learning Workshop at the 26th ICML, Montreal, Canada,2009.

k. Barba, L. A. (2018). Terminologies for Reproducible Research. ArXiv, (January). Retrieved from http://arxiv.org/abs/1802.03311


clarity: The paper is "well written" on a paragraph by paragraph basis, but as a whole is not. I found great difficult understanding what the point/contribution of the paper itself was, and it felt like each paragraph/section had been written independent of any larger theme. I had to get all the way to the first paragraph of the conclusion to feel like I understood what the paper was presenting as it's contribution! 

As an example of the writing confusion, I re-read Section 2 several times when first encountered. Each paragraph read as if it was going to explain why one method/type of HPO was better/superior to another, or how some problem was to be fixed, constantly leaving me in expectation of a follow up paragraph that never occurred - instead repeating the process for a new HPO method. 
relation to prior work: See prior Correctness concerns for related work on the methodology side, on the HPO side specifically I see no special issues. 
additional feedback: Overall I do not see the paper as having a significant contribution. The largest contribution is organizing existing code/benchmarks into a consistent API. I see no discussion of maintenance plan for this framework, and I think it's design of many docker containers and the constantly evolving nature of deep learning frameworks puts it at risk of needing continuous effort to avoid bitrot. 

Beyond packaging existing benchmarks, a few basic ML algorithms are included, yet with no more than a paragraph of discussion. Not even an explanation of the kernel being used for the SVM is provided! The results seem to themselves provide no evidence that these new benchmarks provide some information not available from the existing benchmarks, with very similar mean-rank curves in plots. I think a much more thorough and thoughtful discussion of what existing benchmarks do not cover, why that is a problem, and how a expanded set of benchmarks will rectify those issues, would bring this paper to a much greater level of contribution and significance. In it's current form it feels like a bunch of popular ML tools were thrown in "just because" and left at that. 


_____

After revision the paper is _much_ improved, uses a proper hypothesis testing approach, and is much easier to read for both new and knowledgable researchers of HPO methods. IMO this work now passes the bar for a NeurIPS paper and am comfortable with acceptance. I do still think the references I provided in initial review would be valuable to fully incorporate, particularly ones like repo2docker that discuss reproducibility with respect to characteristics of this system.
review_title: Unclear Contribution 

Meta-review: All reviewers agree on acceptance.
Decision: Accept