Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper contributes a tabular benchmark called JAHS-Bench-201, which measures the performance configurations in a joint hyperparameter and NAS search space over three image classification tasks: CIFAR-10, Colorectal-Histology, and Fashion-MNIST. There are 140 million reported configurations per dataset, most of which are derived from surrogate predictions learned from an initial set of 270,000 evaluated configurations on each task. Furthermore, of the 270,000 configurations evaluated on each dataset, 20 different performance metrics are captured. 
strengths: - This work evaluates many dimensions of the JAHS problem, with plenty of metrics, support for multi-fidelity optimization
- Reasonable choices of hyperparameters
- Builds on NAS-Bench-201, a widely adopted tabular NAS search space. 
- Additional hyperparameters controlling network capactity -- these are often neglected in tabular NAS-Benches. 
- The idea to allow the image scaling to be a hyperparameter is interesting. 
- The authors use their benchmark to evaluate specific scientific hypotheses about the joint optimization of architectures and hyperparameters as well as a comparison of different types of fidelities. Both of which are understudied in the literature. 
- The multi-multi-fidelity setting is interesting and is a promising direction for future work promoted by this benchmark. 
weaknesses: - The use of surrogate models to generate the performance predictions is potentially risky. As far as I'm aware, using surrogates to produce benchmarks via performance prediction using an XGBoost regressor has mainly been used (somewhat) in NAS benchmarks, and has not been thoroughly vetted for hyperparameter benchmarks. The authors report some reasonable numbers in the main text that support the usage of surrogates, but this is a point that should have more evidence/support provided in the main text. 
- While the multi-fidelity evaluation in Figure 2 is interesting in that many of the fidelity types perform similarly, a separate evaluation combining multiple fidelities at once seems to be missing. This setting is described as future work, but this experiment would also help to strengthen the current claims in the "Are some fidelities more useful than others?" section. 
- Given that surrogate performance predictors were employed to generate the benchmark, it would have been nice to include more datasets beyond the three currently included in the benchmark. 
- Additional experiments, scientific hypothesis enabled by the benchmark, or any additional content would help to improve the paper. The current version of the paper has 8 pages of content, though the page limit is 9 pages. 
correctness: - Everything reported in the paper seems correct, although the surrogate modeling concerns me. 
clarity: - The paper is written clearly enough. 
relation to prior work: - Prior work is described in detail, and the relationship between JAHS-Bench-201 and other Benchmarks is discussed. 
additional feedback: - The page limit for this track is 9 pages -- though the paper has 8 pages of content. 
- (Line 73) typo "hyperparametwe"
review_title: Solid contribution but slight concerns about the lack of details w.r.t. surrogates  

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: &nbsp;

The authors extend the set of NAS benchmarks by introducing a new benchmark focussed on joint architecture and hyperparameter search. Currently, I lean towards acceptance and will consider raising my score following the authors' response to the points raised below.

&nbsp;
strengths: &nbsp;

1. The idea that hyperparameters and architectures should be considered jointly when evaluating hyperparameter optimisation algorithms is a very logical one and the proposed benchmark saves HPO researchers a great deal of time in terms of constructing a suitable evaluation framework.

&nbsp;
weaknesses: &nbsp;

## **MAJOR POINTS** ##

&nbsp;

1. The statement in the introduction,

&nbsp;

> Despite recent work showing that good hyperparameter settings are even more important than using the best architecture.

&nbsp;

seems to contradict the results of the experiment provided in Figure 1 i.e. NAS-only optimisation achieves lower error relative to HPO-only optimisation. It would be great if the authors could clarify this?

&nbsp;

2. As the reviewer understands, the difficulty in extending tabular NAS benchmarks to consider continuous hyperparameter domains jointly is that the hyperparameter domains are continuous and hence every possible configuration cannot be computed. As such, the authors interpolate a (presumably dense) grid of the continuous hyperparameter space using a surrogate model. I'm slightly confused about the methodology used to assess the model fit on this grid using the train/validation/test split in Appendix D. Presumably the test set is structured such that it contains points such that the train set becomes a finer grid relative to the entire dataset i.e. the test points lie inbetween the data points constituting the grid of the training points? Is this what the authors mean when they describe the construction of the splits in terms of fidelities? Presumably the surrogate model is then retrained on the entire set of collected data?

&nbsp;

3. Why is the R^2 for the surrogate on the Colorectal-histology task so low?

&nbsp;

## **MINOR POINTS** ##

&nbsp;

1. It would be great if the references appeared in numbered order e.g. via the LaTeX command:

&nbsp;

> \bibliographystyle{unsrtnat}

> \setcitestyle{numbers,open={[},close={]},citesep={,}}

&nbsp;

2. Typo line 29, "fueled"

&nbsp;

3. Line 42, the nested (multiple) is confusing.

&nbsp;

4. It may be worth making the distinction between an architecture and a hyperparameter (e.g. activation function type) in the introducton to make the paper more accessible to readers who are unfamiliar with the NAS literature.

&nbsp;

5. Line 53, typo, full stop in place of comma in 270,000.

&nbsp;

6. The attempt to run the benchmark using the minimal example provided in the repository resulted in an error related to XGBoost.

&nbsp;

7. Line 56, typo, capitalisation of "The" after the colon.

&nbsp;

8. Line 63, typo, superfluous colon.

&nbsp;

9. In Table 4 of the SI perhaps it would be an idea to give time in a more intuitive unit such as days.

&nbsp;

10. Line 73, typo in hyperparameter.

&nbsp;

11. Line 105, typo, missing comma in 10,000.

&nbsp;

12. R^2 instead of R2 for the coefficient of determination.

&nbsp;

13. The NAS and HPO acronyms are defined in the introduction. It would be great if they could be used consistently in the rest of the paper.

&nbsp;

14. It might be worth moving some of the details from section D of the appendix into the main paper. The page limit for the submission is actually 9 pages so this should be achievable.

&nbsp;

15. Line 133, typo, "settings".

&nbsp;

16. Line 135, typo, capitalisation following colon.

&nbsp;

17. Line 137, typo, leaderboards.

&nbsp;

18. Line 155, Might be worth using JAHS as the acronym since it has already been defined.

&nbsp;

19. The number of trials over which the standard error is computed could be given directly in the captions of Figure 1 and Figure 2.

&nbsp;

20. In terms of HPO benchmarks it may also be worth referencing Bayesmark and the NeurIPS black-box optimisation challenge benchmarks [1,2].

&nbsp;

21. Line 189, typo "hardware".

&nbsp;

22. The 6\dot\mathbb{R}^2 is somewhat confusing in Table 6.

&nbsp;

23. Line 197, typo, "700 times larger".

&nbsp;

24. Line 217, typo, "multi" instead of "multy".

&nbsp;

25. Line 41, Appendix D, missing "so".

&nbsp;

26. Typos in capitalisation in the references, e.g. "Bayesian".

&nbsp;

27. Line 82, Appendix G, missing space when mentioning "Table 4".

&nbsp;

28. Table 4 caption, missing "the".

&nbsp;

## **REFERENCES** ##

&nbsp;

[1] Turner et al. Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-box Optimization Challenge 2020. In NeurIPS 2020 Competition and Demonstration Track, 2021.

&nbsp;

[2] Cowen-Rivers et al. HEBO: Pushing The Limits of Sample-Efficient Hyper-parameter Optimisation. Journal of Artificial Intelligence Research, 2022.

&nbsp;
correctness: &nbsp;
cf. the main response.
&nbsp;
clarity: &nbsp;
cf. the main response.
&nbsp;
relation to prior work: &nbsp;
Adequately addressed.
&nbsp;
additional feedback: &nbsp;
All feedback provided in the main response.
&nbsp;
review_title: Practical and Useful Benchmark for the NAS Community

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper argues that hyperparameter optimization plays an equally important role as neural architecture search in getting well-performed models. Hence, the research community should focus more on developing methods that jointly optimize the architecture and training hyperparameters. To call for more attention to joint architecture and hyperparameter search (JAHS), the paper proposes a new benchmark, called JAHS-Bench-201, that evaluates 15k architectures across 3 image classification tasks with 4 different hyperparameters and 4 fidelity parameters. In addition to having an extensive search space, JAHS-Bench-201 also supports benchmarking in 6 optimization tasks and 4 algorithmic settings. Lastly, the paper provides insights into why JAHS matters and studies the efficacy of different fidelity parameters.

strengths: - The benchmark focuses on joint architecture and hyperparameter search which is required by many real-world AutoML applications nowadays. Hence, the topic studied is well-motivated and has practical significance. 
 
- The benchmark proposes a much more comprehensive search space comprised of an architecture space, a hyperparameter space, and a fidelity parameter space. In addition to the novel design of the search space, the benchmark considers a wide range of optimization tasks and algorithmic settings including multi-fidelity optimization, which distinguishes it from previous similar work. The considered data and tasks are also diverse, thus the benchmark can reflect better on the actual performance of different network models. 
 
- This paper answers interesting research questions such as how important it is to jointly optimize in the space of architectures and hyperparameters. It provides valuable insights into possible research directions in AutoML.
 
- The paper is well-organized and well-written. The code, datasets, and experiment scripts are provided for reproducibility. Trails are over 10 random seeds, and both mean and standard error are provided, so the results seem credible.
weaknesses: - Jointly optimizing neural network architectures and training hyperparameters is not a new topic for benchmark work. As mentioned in the paper, previous work such as NAS-HPO-Bench and NAS-HPO-Bench-II studies the same problem, so the novelty of this work mainly lies in the extended search space and more diverse evaluation settings.
 
- The hyperparameter search space is somehow restricted as some commonly considered hyperparameters are not considered, such as optimizer type, momentum, batch size, etc. How easy it is to extend the set of considered hyperparameters?
 
- It may be beneficial to include instructions on testing a custom NAS/JAHS algorithm on the search space in the code repo.
 
- To answer the first question in section 4, the paper compares the performance of randomly sampled architectures and hyperparameter configurations. However, rather than using only the random search baseline, it can be more informative to include comparisons between more sophisticated NAS algorithms and HPO algorithms.
correctness: The claims made in the paper appear to be correct and the experimental protocol seems reasonable.

clarity: The paper is mostly well-written and also tries to be accessible to researchers working outside of the NAS domain.

relation to prior work: The authors discuss related work in Section 5 and make it clear why previous NAS-HPO benchmarks are limited and insufficient.

additional feedback: N/A
review_title: A more comprehensive benchmark for joint architecture and hyperparameter search

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper presents JAHS-Bench-201, an extension of the popular neural architecture search benchmark NAS-Bench-201. JAHS-Bench-201 is a benchmark for testing joint architecture and hyperparameter search algorithms. In addition to presenting the dataset, this paper justifies its contributions by presenting promising joint architecture and hyperparameter search results.
strengths: The paper is extremely clear with a strong justification for the need of the dataset and strong ties to previous work. The dataset itself is extremely well documented with examples, easily accessible data, and built-in leaderboards. In addition to presenting the dataset, the authors conduct a thorough investigation into the usefulness of the dataset and JAHS more broadly. Their analysis seems sound and provides a strong justification for the novelty of this contribution.

weaknesses: This is a very strong submission. The authors clearly outline the significance of the contribution, relevance to the broader research community, accessibility and accountability, and ethical and social implications. Minor revisions are needed for clarity and some questions are presented in the additional feedback section for consideration.
correctness: Yes, to the best of my knowledge.

clarity: Yes. Some typos listed below:

Line 73: misspelling: "hyperparametwe"
Line 146: Inconsistent use of “(multi) multi-fidelity,” “\textit{multi} multi-fidelity,” and “multi multi-fidelity.”
Line 197: “​700 larger” should be “700x larger”

relation to prior work: Yes, the paper provides an extensive outline of the differences between JAHS-Bench-201 and prior NAS and HPO benchmarks.

additional feedback: Why was batch size not used as a hyperparameter as in NAS-HPO-Bench-II? 
Why was data after divergence excluded? 
And why weren’t divergent configurations used in surrogate models?
review_title: Extremely thorough and clear paper presenting a novel and useful dataset

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper provides the information of the construction of a new benchmark set, JAHS-bench-201. This benchmark includes the results of Neural Architecture search combined with hyper-parameter optimisation. These two processes are both hard and time-consuming. This benchmark will result in making research in the combination of the two optimisations less time and energy consuming for the testing of future methods in this field. This is the main contribution. 
strengths: The dataset is very extensive and structured. Research in NAS is a developing field and this benchmark will provide the community with an easy access point. 
weaknesses: A weakness is the limited search space for architectures to be considered. 
correctness: The dataset is constructed in a sound way. The authors rely on highly populair previous work in the field. 
clarity: This paper is very well written. The necessary information is provided in a structured and clear way. Tables that are provided are very helpful l and support the reader. 
relation to prior work: The authors clearly discuss the connection and differences compared to the NAS-Bench-201 benchmark, which they use as a starting point. 
additional feedback: There was a small typo in line 73: hyperparametwe
review_title: The next step in NAS 

Meta-review: metareview: The JAHS-Bench-201 provides a collection of surrogate benchmarks for JAHS that will enable research on multi-objective, cost-aware and multi-fidelity optimization methods. There is consensus among the reviewers that the contribution is significant and the discussion/rebuttal reinforces the strength of the paper.
confidence: 5: The area chair is absolutely certain
Decision: Accept