Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The benchmark in this paper is obtained by collecting and cleaning data from openML. This cleaned up data is then presented in a way that allows for the comparison  of different HPO methods. 
strengths: Aggregating and cleaning up the data from openML is a large and time consuming work. 
The resulting dataset seems promising.
weaknesses: I think the biggest weakness of the work is that the search spaces are not covered exhaustively (like for example nasbench 101).
If I understand it correctly this means that an algorithm can only suggest samples from the actual datapoints in the dataset and it is not possible to  evaluate any point in the search space.  This is mentioned in section 6. 

I think that section 4 could be largely removed and used to expand the analysis of the results.

correctness: Most work seems correct.
clarity: In section 5, the name meta-datasets is used. I am confused whether this refers to results with multiple hyper parameters on a dataset. Or whether this is something else.

I think that section 4 can be shortened. GP's and EI are not needed for the rest of the paper.

I struggle to understand what the meta features used for transfer learning look like.

I am struggling with the critical difference plots and their interpretation. A better explanation would be helpful.
relation to prior work: The work appears to provide a nice extension compared to previous datasets for HPO evaluation.
additional feedback: See above
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes a large-scale and reproducible benchmark for evaluating black-box HPO methods on both traditional non-transfer HPO methods as well as transfer HPO techniques. To enable this benchmark, the author's have also released a large collection of meta-datasets for the purposes of benchmarking HPO algorithms. The work conducts the benchmark on a variety of methods and details an evaluation method that leverages average rank on a per-trial basis as an addition to normalized regret.
strengths: Strengths:

- The paper introduces a large public collection of pre-processed and sanitized meta-datasets which will be directly applicable to future research in efficient HPO methods.
- The benchmark was specifically designed to address concerns of reproducibility found in other benchmarks.
- The per-trial analysis of normalized regret and average rank is an interesting approach, and is more informative than either in isolation.
weaknesses: Weaknesses:

- It seems like there was an opportunity to compare non-transfer methods with transfer methods that was missed. It would be useful to know how BOHAMIANN and GP compare to TST-R and TAF-R, especially with ablation that limits the amount of training and validation data available to the transfer methods to measure the impact of the size of the meta-training set on the performance of transfer HPO methods. This ablation would help strengthen the proposed benchmark by highlighting the utility of having a larger corpus of metadata results compared to prior works.
- Average rank seems like a slightly strange metric to use when the search spaces are highly correlated. For example, 6 of the 16 meta-datasets in v2 are using ranger as the model. This seems to indicate that for the purposes of average rank, performing well with ranger is 2x as important as with XGBoost and 3x as important as with SVM. In reality, XGBoost is arguably more important to perform well on than ranger, at least in terms of absolute model performance. I'm curious to see how the authors address this apparent imbalance and search-space correlation.

Edit: The authors have addressed my main concern regarding the lack of comparison between transfer methods and non-transfer methods. Because of this, I have increased my initial score.
correctness: While I haven't confirmed the individual implementations of the methods benchmarked, the overall approach seems valid.
clarity: The paper is well written and flows nicely.
relation to prior work: Yes, relevant work appears to be cited and it is clarified that this work claims to be the largest meta-dataset collection in the field while also addressing reproducibility issues of prior work.
additional feedback: - I would be interested to hear the author's thoughts on making the prediction probabilities for each model-hyperparameter-dataset config available in this benchmark. While it may be prohibitively large to store / work with, it would enable further experiments such as simulated ensembling of different models trained for a given dataset which could be of great interest in the field of AutoML.
- It would be nice if the full set of meta-features used were listed, perhaps in the appendix.
review_title: Fast HPO evaluation via a large-scale meta-dataset collection

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper introduces HPO-B, a Python API for a tabular benchmark for black-box HPO, relying on publicly available OpenML data using different runs of flows (particular machine learning algorithms run on different datasets).
HPO-B is a tabular benchmark in the sense that for a limited pre-defined set of configurations of a search space (hyperparameters of a particular machine learning algorithm), the (pre-evaluated and as stored on OpenML) performance of these configurations on a specific dataset can be easily obtained via a simple look-up.
HPO-B comes in three different versions HPO-B-v1 (raw benchmark of all 176 meta-datasets), HPO-B-v2 (the subset of 16 meta-datasets with the most frequent search spaces) and HPO-B-v3 (split of HPO-B-v2 into training, validation and testing), with HPO-B-v3 claiming to being suited for benchmarking transfer-learning HPO methods.

Edit:
Final decision: While I am happy with how the authors responded to my concerns and suggestions, my final rating remains unchanged.
I do agree that the idea of HPO-B is a step forward with regards to existing benchmarks of black-box HPO methods, however, my main concerns regarding the construction and design of HPO-B (large scale, decentralized and sparse evaluated; limited, classical search spaces) remain and in my opinion limit the relevance to the community. 
strengths: - The topic of reproducible benchmarking is an extremely relevant one and the community surely would profit from clinging to a single gold standard benchmark
- The paper and documentation profits from establishing a clear overview of terminology, as provided in Table 1
- The Python API is easily accessible
weaknesses: - The significance of the contribution is limited by the nature of search spaces being used. That is, mostly traditional machine learning algorithms (SVM, random forest, ...) with the number of hyperparameters ranging from 2 to 18 (although it appears that the reported dimensionality is stated after already having applied one-hot encoding to categorical hyperparameters). Going in line with modern AutoML, search spaces that include different preprocessing pipelines and also allow for varying the actual learning algorithm (i.e., introducing a hyperparameter for the final learner, SVM, random forest, ... and conditionally on that tune the hyperparameters of the selected learner - so-called branching) could be of broader interest - although the nature of search spaces being included in HPO-B is of course limited by the nature of search spaces that can be found on OpenML
- Furthermore, HPO-B as a tabular benchmark has been constructed based on configurations of search spaces being evaluated relatively sparsely on multiple datasets. This results in that potentially only few different configurations can actually be queried for a single dataset, essentially resulting in using discretized versions of the original search spaces (with the resolutions of the discrete grids varying strongly depending on the diversity of the original evaluations as found on OpenML)
- The fact that essentially discretized versions of the original search spaces are used is a significant drawback, especially in the context of comparing Bayesian optimization methods (which appears to be the experimental focus of the accompanying paper): There is no longer the need for actually optimizing the acquisition function but simply calculating the acquisition function value for all remaining configurations (that have not been evaluated so far) and choosing the arg max is / needs to be done; this limits the potential for comparing fully modular Bayesian optimization methods. Moreover, this also limits the total number of "trials" (i.e., iterations) HPO methods can be run for, due to only a limited number of remaining configurations being available for evaluation (even in the scenario of purely continuous search spaces)
- While HPO-B can be used for benchmarking transfer-learning HPO methods, multifidelity evaluations are not possible which would have also been needed to "become the de facto standard for experimentation in the realm of black-box HPO", as a significant number of black-box HPO methods leverage the concept of multifidelity evaluations
- Although only evaluations (referred to as "runs" in OpenML) tagged with "Verified_Supervised_Classification" were included, the individual quality of the runs and therefore the whole quality of the benchmark cannot be assessed easily (as OpenML is open to submissions from anybody)
- The limitations of the work have not been discussed thoroughly
correctness: - The construction of the benchmark is described well and can be followed along easily
- However, potential errors in / and the individual quality of the OpenML runs cannot be assessed easily
- Proposing normalized regret and ranking measures as evaluation metrics is reasonable and in line with the current standard
clarity: - The paper is clearly structured and well written. The final experimental section could be improved in clarity, i.e., mainly different sequential Bayesian optimization methods are compared, varying different surrogate models and acquisition functions, however this is initially not stated
relation to prior work: - HPO-B is compared to a selection of tabular benchmarks previously being used in the literature (Table 2), however, only limited information such as the number of search spaces, datasets, hyperparameters and evaluations are discussed
- Especially the concepts of using explicit experimental protocols, splits, and evaluation measures are not discussed in detail
additional feedback: It may could have been interesting to mention the concept of surrogate benchmarks (i.e., benchmarks based on surrogate models themselves) and outline pros and cons compared to the tabular benchmark approach of HPO-B.
review_title: Honorable goal of establishing a new standard for black-box HPO benchmarking, but there are some limitations

Meta-review: Dear authors,

first of all thank you very much for your submission to the Datasets and Benchmarks Round 1.

We appreciate your hard work and fruitful discussions with the reviewers.

The final scores by the reviewers are 5/6/7 (with confidences 3/3/4 respectively), which is a borderline paper and resulted in some further discussions between area chairs.

Ultimately we decided to reject your submission but encourage a re-submission for round 2.

The main reasons for rejection are the following:

(1) Quality of the data: It needs to be assured that benchmarking on the discrete search space is equivalent to / representative of benchmarking on the real underlying problems (without discretization of the search space).

(1.5) The claim "*Furthermore, the response function is typically a piecewise constant function and small changes in a continuous hyperparameter have no impact. Therefore, it is oftentimes not necessary to query in close proximity to other points and it is unlikely that a better hyperparameter could be found.*" from your response should be (empirically) shown to be true for the used discretization of the search space.

(2) The discretization does not necessitate any acquisition function optimization, as all values can be tried exhaustively, it is unclear how large the impact of this actually is.

(3) The effect of preprocessing the search space (handling of categorical and dependent HPs) cannot be evaluated jointly with HPO strategies due to the the hardcoded preprocessing.

(4) Since grey-box HPO becomes more and more important it reduces the significance/impact of the banchmark that it only allows black-box evaluations.

(5) Since only single (relatively simple) algorithms without any preprocessing pipelines are used is a further limitation to properly evaluate the performance of different HPO strategies in real world settings.


Lastly I want to stress that the direction of this work is very promising and the HPO / AutoML / Meta-Learning community is in need of better benchmark platforms. But since there already exist quite a few benchmarks that are lacking in one or multiple areas, we need a single (or very few) systems for surrogate benchmarks that are very solid not more that are (quite) limited in what they allow.
 



Decision: Reject