Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides a surrogate model-based benchmark dataset for neural architecture search, which includes search energy consumption and architecture/hyperparameter joint search space. The proposed dataset, called EA-HAS-Bench, enables us to compare the NAS method in energy-aware settings. A Bezier curve-based predictive model is used to provide the surrogate learning curves. The authors exhibit the benchmarking results of existing HPO/NAS algorithms on the energy-aware setting by using the proposed EA-HAS-Bench.
strength_and_weaknesses: [Strength]
* A novel benchmark dataset for AutoML/NAS is provided, which includes search energy consumption and architecture/hyperparameter joint search space. As the common benchmark datasets are useful to develop and compare the HPO/NAS algorithms and EA-HAS-Bench considers a novel aspect of benchmarking, it will be valuable in the community.
* The demonstration and benchmarking of existing algorithms using EA-HAS-Bench are sound.

[Weaknesses]
* The surrogate model-based benchmark datasets might have a gap with a really collected dataset. It might be better to provide not only the large-scale surrogate dataset but also the small "real" dataset.
* There exist several unclear points. Please see the following comments.

[Comments for Authors]
* The reviewer is not sure of the definition of energy consumption in this paper. How did the authors measure the search energy consumption? The detail of the data collection method, including machine spec, should be reported.
* The reviewer could not find the number of actually evaluated samples to build the surrogate model. Such information should be reported to ensure the reliability of datasets.
* The authors use the Bezier curve and train the model that predicts control points for learning curve prediction. The reviewer thinks that the Bezier curves of order $n$ are equivalent to polynomial functions of order $n$. Therefore, we can use the model that predicts the coefficients of polynomial functions of order $n$ instead of predicting the control points of Bezier curves. The advantage of predicting the control points of Bezier curves should be clarified.
* It would be better to report the total energy consumption to build EA-HAS-Bench.
* What does "GT" mean in Table 3?
* What does "LC" mean in Figure 6?
* Why is the result of BOHB for CIFAR-10 in Figure 6 omitted?
* It would be nice if the authors mentioned the following recent paper. The following paper seems to relate to this work, although it might be published after this paper's submission.

Archit Bansal, Danny Stoll, Maciej Janowski, Arber Zela, Frank Hutter, "JAHS-Bench-201: A Foundation For Research On Joint Architecture And Hyperparameter Search," NeurIPS 2022 Track Datasets and Benchmarks.
https://openreview.net/forum?id=_HLcjaVlqJ

clarity,_quality,_novelty_and_reproducibility: * This paper is well-written. The contribution is clearly explained.
* Although several NAS and AutoML benchmark datasets already exist, the difference and novelty compared to existing ones are clearly described. EA-HAS-Bench is the first benchmark dataset that considers energy consumption and is a large-scale surrogate-based benchmark of architecture/hyperparameter joint search space. 
* The dataset and code are not provided in the current phase, while the authors state that the dataset of EA-HAS-Bench will be released after the paper publication. The reviewer encourages the authors to release the code of the dataset collection and the code of experiments using the dataset.
summary_of_the_review: This paper provides a novel HPO/NAS benchmark dataset that is valuable for the community. However, several unclear points should be addressed.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a new benchmark dataset for joint neural architecture search (NAS) & hyperparmeter optimization (HPO). It includes information about the energy cost of training a model so that people studying NAS-HPO algorithms can incorporate energy cost. They use a large search space for joint NAS-HPO search and develop a new surrogate model for this search space. In particular, this surrogate model can predict learning curves of arbitrary length. They also demonstrate how to use this dataset.
strength_and_weaknesses: I agree that this is an important problem, and I think that this dataset could be very helpful for saving energy during future research. Also, including HPO makes a lot of sense to me, since a lot of energy could be saved by a model that converges faster. 

I have various questions and comments below to improve the writing of the paper. I am also concerned about the reproducibility, as discussed below.

clarity,_quality,_novelty_and_reproducibility: Quality:

Important point: "provides the full training information including training and test accuracy learning curves" What about validation curves? Validation curves are typically used for early stopping & choosing between models. Test error should only be used at the end of a search, not during the search (NAS-Bench-101, Ying et al., 2019). Hopefully this is just a typo.

Novelty:

This paper seems novel to me. It is of course building on a series of previous papers, but adds several contributions. To me, the main one is having a dataset for NAS-HPO reporting energy cost. However, other contributions, like providing a large search space for NAS-HPO joint search and using a new surrogate method that enables longer learning curve prediction, are also valuable, even for researchers not trying to balance energy cost. (However, I hope that balancing accuracy & energy cost becomes more common.)

Reproducibility:

The authors plan to share the dataset they created, which will enable reproducible HAS-NAS studies. However, it would be helpful to also share the code. If many people use the dataset, it will be important to fully understand how it was created. Also, the BSC surrogate model is fairly complicated, and future research would greatly benefit from being able to use the code instead of trying to recreate it. It is helpful that the hyperparameters of the surrogate models (both for predicting the learning curves & for predicting the other metrics) are listed in the appendix, though. 

"For different datasets (i.e., CIFAR-10 and TinyImageNet), we adjust this space accordingly via constraining parameters and FLOPs." Is this ever explained? 

Figure 6 contains acronyms for a variety of NAS algorithms that I don't believe are ever explained. For example, what is REA? Using Appendix D I can guess what the acronyms mean, but I think the acronyms should be explicitly explained, along with citations, in the main paper. Also, there is not much detail for this figure (Section 5.1 & Appendix D), so it would be difficult to reproduce. 

Clarity:

"LGB models" are introduced on page 4 but the acronym is never explained. From Googling, I'm guessing this is Light Gradient Boosting? 

"The energy cost surrogate models achieve 0.787 for R2 and 0.686 for KT on CIFAR-10 and 0.959 for R2 and 0.872 for KT on TinyImageNet." How do these numbers compare to previous papers that use surrogate models to fit a search space, such as NAS-Bench-301, NAS-Bench-X11, and NAS-HPO-Bench-II? I don't have a sense for if these metrics are good enough to make the surrogate useful for NAS/HPO research.  

What is GT in Table 3 & Figure 8? In Figure 8, it seems to mean "ground truth," but I'm not sure what "ground truth (1 seed)" would mean in Table 3. 

The fonts in the figures are often hard to read. 

There's something missing here: "we new energy-aware AutoML method that arms"

"Our dataset needs to provide the total search energy cost of running a specific AutoML method, which is the sum of the training energy cost of all the hyperparameter and architecture configurations the search method traverses." Technically, the dataset provides the energy cost of particular configurations, which then enables someone trying an AutoML method to calculate the total search energy cost, correct? 

"The size of the maximum number of epochs is almost proportional to the quality of the search space..." I don't understand this sentence, but perhaps I don't fully understand how to associate the ECDF with search space quality. For the other hyperparameters, I can clearly see in Figure 3 that changing the hyperparameters changes the distribution. 

The paper writing is fairly verbose, often repeating the same points. If you need to fit in more material to address reviews, I think you could condense some of the existing writing. I especially noticed the repetitiveness once I got to page 4. 

On page 2, "EE-HAS-Bench" is used instead of "EA-HAS-Bench."

"Tree-of-Parzen-Estimators (TPE) (Bergstra et al., 2011) is adopted for BCS to search for the best hyperparameter configuration." and "For the optimal three surrogate models in Table 3 and the LGB energy cost model (LGB-E), the optimal parameters found using TPE and the search space are listed in Table 5." So only the best surrogate models get their hyperparameters tuned? Perhaps the other surrogate models appear inaccurate by bad luck? 

"Since the prediction results may have anomalies that make the prediction performance much higher than the actual results, we introduce spike anomalies (Yan et al., 2021) to evaluate these anomalies." This is vague, and I don't know what it means. Spike anomalies are mentioned again in the next paragraph, where I also don't understand them. 

"On the other hand, the importance of the different parameters varies considerably, especially exponentially." I'm not sure what this means, especially the "especially exponentially" part. 

In the middle plot of Figure 4, I don't know what the orange means. I also thought it was unclear how we can tell that models in the Pareto Frontier on the right are not always on the Pareto Frontier in the middle figure, so perhaps that's what orange is for? 

Figure 5 would be easier to read with a diverging color palette with 0 being white. 

The description of Figure 5 in its caption and in the text would suggest that you're only showing correlation coefficients for accuracy & TEC. However, there are 4 columns. 

"Energy-aware BANANAS and vs. origin BANANAS and LS" I'm guessing this is supposed to be "Energy-aware BANANAS and Energy-aware LS vs. origin BANANAS and LS"

In Figure 6 (right), I'm guessing that the y-axis should be accuracy? If so, this figure and its description Section 5.2 tell a clear story that modifying these HPO algorithms to consider energy cost results in being able to find good solutions with less energy. However, the story is less clear for Figure 6 (left) and Figure 6 (middle). I see that with a limited energy budget, LS and bananas reach the lowest regret for CIFAR10, and BOHB and HB reach the lowest regret for TinyImageNet. Are these different conclusions than if energy was not considered? 
summary_of_the_review: Overall, I like this paper, although the writing & reproducibility could be improved. I'll tentatively mark this paper as "accept," with the assumption that the authors can make at least some of the points more clear. Also, I'm hoping to confirm that the validation curves are provided in the dataset, not just training & test. If validation curves are not provided, I'll lower the score. 

**Update post-rebuttal**

Thanks for improving the clarity and confirming that the validation curves are included. I'll keep the score at "8: accept, good paper."
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes EA-HAS-Bench, which is a surrogate neural architecture search and hyperparameter benchmark that can be used to query the energy cost (in kWh) of training and inference of neural networks in the search space. This can be used as another metric when benchmarking AutoML algorithm to compare them with the energy consumption, which according to the empirical results in the paper, is not well aligned with the training costs. The authors create the benchmark on 2 popular image classification datasets, namely CIFAR-10 and Tiny-ImageNet.
strength_and_weaknesses: Below I mention some pros and cons of this submission:

(+) Provides novelty in the field of NAS benchmarking by creating a new surrogate benchmark that includes the AutoML search energy consumption.

(+) Well-written and mainly easy to read.

(+) The surrogate model proposed seems to outperform most of previous surrogate models used in the literature.

(+) Claims supported by empirical evidence and analysis.


(-) I wasn't able to find the codebase for the benchmark, which is one of the most important features when proposing a new NAS benchmark

(-) No description of the API on how to use the benchmark.

(-) Does not support one-shot NAS algorithms.

(-) Section 5 could benefit of more empirical evaluations, as for instance evaluating multi-objective NAS algorithms on the benchmark.

**Questions and other comments**
- I guess the models used to encode the architecture and hyperparameters are MLPs? I could not find more details on them. Did the authors provide these in the text? Also, did they try other surrogate models such as GINs for the architecture?
- It would be interesting to compare the learning curves in Fig 6 to the same counterparts ran on the real benchmark, i.e. without the surrogate predictions, as done in the Surr-NAS-Bench paper.

Minor:
- Wrong reference to NAS-Bench-301 in Section 3.

clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow and well-written. The authors provide a valid motivation for proposing the new benchmark in the introduction and cover the related work properly. The main novelty aspect of this submission lie in (1) the energy-consumption benchmark that also includes the training cost (differently from HW-NAS-Bench which only has for inference). (2) new surrogate model for learning curve prediction. (3) Joint NAS and HPO search space where the architecture space is the one from RegNet.

In terms of reproducibility the authors do not provide the necessary code together with its API for the benchmark.
summary_of_the_review: In general the paper provides a useful contribution to the NAS community, by proposing a new benchmark that provides the energy consumption utilized to train and evaluate a configuration. Even though suited mainly for black-box algorithms the search space is useful and the surrogate model used to predict the learning curves is novel in the NAS literature. The claims are backed with empirical evidence. I lean towards acceptance for this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work presents a surrogate tabular benchmark for neural architecture search (NAS) -- EA-HAS-Bench -- with two primary contributions: 1. Joint space comprising architectures and hyperparameter configurations, and 2. Additional metrics to measure efficiency that include training and inference energy costs. This work then uses the EA-HAS-Bench dataset to demonstrate the use of different NAS strategies when constraining the overall budget to a fixed energy/power consumption criterion instead of max. epochs. The work also demonstrates the use of energy consumption per model cost as a factor in the model selection criterion to obtain more energy efficient architectures. Comprehensive experiments are carried out on a large joint space of architectures showing the usefulness of the proposed tabular benchmark dataset.

strength_and_weaknesses: **Strengths:**
* The main contribution of this work is the EA-HAS-Bench tabular benchmarking dataset which not only has conventional performance measures, but in addition tabulates training- and inference energy costs. This can become an important contribution as most existing tabular benchmarks mainly report number of parameters or training time as the proxies for efficiency.

* The scale of the new surrogate dataset presented in this work is massive. $6\times 10^7$ unique CNN architectures and 540 hyperparameter configurations are explored yielding about $3\times 10^{10}$ datapoints.

* A novel surrogate model that approximates the learning characteristics using Bezier Curve-based surrogate (BCS) model is presented and validated. Energy and training times are predicted using Surrogate NAS Benchmark approach in [3].

**Weaknesses:**

* **Energy measurements**: The main contribution in this work is on updating the energy consumption as an additional metric. However, the details on how this energy was measured is missing. Which of the components were monitored and on which hardware? It is mentioned that GPU costs are measured; while this is one of the large contributors the CPU, DRAM costs are not small [1]. Factors such as Power usage effectiveness further have impact on the energy efficiency of the models.

* **Surrogate Energy measurements**: As with other metrics, I assume the energy consumption is also estimated using some surrogate methods. In Sec. 2.2 the authors describe that energy costs are measured for one epoch and aggregated over total number of epochs. Authors do mention that energy costs were predicted according to [3] but in [3] there was no energy costs reported. Is this a linear scaling of the energy cost of one epoch? Does this extrapolation extend simply linearly?

* **Updated objective**: The objective that integrates energy consumption with performance is highly heuristic, and presented without any motivation or justification. Given that the key contribution in this work is to use energy consumption as an additional constraint when performing NAS, the specific formulation of the composite objective is quite vague. It is mentioned to be $ACC\times {\frac{TEC}{T}}^w$ with fixed values for T and w. Are there any other ways of incorporating these measurements that were tried? Would the choice of scaling influence the solutions obtained. Do the gains achieved still hold? Why is this specific formulation the preferred choice?

* **Efficiency in terms of FLOPS**: In Sec.4, the correlation (or the lack thereof) between training energy and training times is clearly captured. This is an important result to show the additional resource costs that are missed when only using training time. However, the authors do not extend this analysis or at least discuss how FLOPs as a measure of efficiency correlates (or not) to measuring energy. For instance, works such as [2] use FLOPs as a measure of efficiency and obtain the now famous efficient net architectures.

* **Missing details**: Although the proposed benchmark is a surrogate one, there are no clear reportings of the total cost of obtaining this dataset. For instance, how many architectures/hyperparameter configurations were sampled in the first place to train the two neural networks. This information was either missing or difficult to obtain from the main paper. In Sec 2.4 there's a mention of using random sampling of the search space -- but how many architectures from the $3\times 10^{10}$ architectures were sampled to be approximated using the BCS model? Further, what was the total energy cost/ GPU costs for obtaining this new dataset? 

[1] Henderson, Peter, et al. "Towards the systematic reporting of the energy and carbon footprints of machine learning." Journal of Machine Learning Research 21.248 (2020): 1-43.

[2] Tan, Mingxing, and Quoc Le. "Efficientnet: Rethinking model scaling for convolutional neural networks." International conference on machine learning. PMLR, 2019.

[3] Zela, Arber, et al. "Surrogate NAS benchmarks: Going beyond the limited search spaces of tabular NAS benchmarks." Tenth International Conference on Learning Representations. OpenReview. net, 2022.

clarity,_quality,_novelty_and_reproducibility: The paper is largely clearly written, with comprehensive experiments for validating the new benchmark dataset. The second objective of demonstrating the usefulness of energy as a conflicting objective to obtain efficient architectures is brief and lacking several details as listed in my main review.

summary_of_the_review: A new surrogate tabular benchmark which also includes training and inference energy costs, along with other standard performance measures. The dataset itself is large, comprehensive and well designed. The experiments pertaining to surrogate model predictions are reasonably done. The main contribution -- which is the use of energy consumption to design energy efficient architectures is not evaluated rigorously. For instance, the joint objective proposed to integrate energy costs with accuracy is a heuristic with fixed parameters presented without any justification; the impact of using such an objective on the class of obtained architectures is also not clearly investigated. For a tabular benchmark aiming to demonstrate energy efficiency the experiments about this aspect are not strong enough.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces a novel joint NAS+HPO benchmark that also includes measurements of energy. All reviewers judged this to be very helpful and gave acceptance scores. Joint NAS + HPO is very important, as also recently addressed in the NeurIPS datasets & benchmarks paper on JAHS-Bench-201. I agree with the reviewers and recommend acceptance.
justification_for_why_not_higher_score: Given the existence of the JAHS-Bench paper, this now "only" adds the energy measurements.
justification_for_why_not_lower_score: The benchmark will be very useful for the NAS communitiy.
Decision: Accept: notable-top-25%