Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper present a new benchmark that extends NASBench-101 with the training energy footprints. The author also presents a MOO algorithm to perform the NAS.
strength_and_weaknesses: Strength:
1) curate a new dataset and perform NAS on it. 
2) the authors also evaluate their algorithms on NAS benchmark.
3) the paper is quite easy to follow, it is more like a tool paper that proposes a new benchmark for others to use.

Weakness:
1) I don't think it is necessary for such paper to introduce a new optimizer then spent a huge amount of content on the optimizer. Instead, I think it is more reasonable for authors to reuse the latest MOO algorithm such as [1] and [2] to perform NAS on their datasets, then release them as a baseline.

[1] Zhao, Yiyang, et al. "Multi-objective Optimization by Learning Space Partitions." arXiv preprint arXiv:2110.03173 (2021).

[2] Daulton, Samuel, Maximilian Balandat, and Eytan Bakshy. "Parallel bayesian optimization of multiple noisy objectives with expected hypervolume improvement." Advances in Neural Information Processing Systems 34 (2021): 2187-2200.

2) The experiments spent too much on the optimizer but not on the dataset, but I'd like to see more analysis on the dataset since I assume the focus of this paper is on introducing a new dataset.


clarity,_quality,_novelty_and_reproducibility: 3) I'd like author clarify the following question? Currently there are training based evaluation (most accurate but most expensive), one-shot evaluation (least accurate but least expensive), and few-shot evaluations (between the one-shot and training) [3]. In one-shot or few-shot, the users only train one or few supernets, therefore it should be the most energy efficient. How do we quantify the energy efficiency in few-shot scenarios? [3] Thank you.

[3] Zhao, Yiyang, et al. "Few-shot neural architecture search." International Conference on Machine Learning. PMLR, 2021.

summary_of_the_review: It is a nice paper, but the authors seem to strike the reviews with some novelty. Frankly, it is okay for such tool paper to be not that novel as long as it solves practical problems. So I'd hope authors focus more on analyzing the dataset itself and establish a set of canonical baselines for using the dataset. Thank you.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes an energy consumption-aware tabular benchmark for NAS based on NAS-Bench-101. For each architecture, it adds the training energy consumption, power consumption, and carbon footprint. This work also demonstrates the usefulness of multi-objective architecture exploration for finding energy-efficient architectures without sacrificing much accuracy.
The MOO algorithm used is based on existing work from Krause et al. 2016.


strength_and_weaknesses: **Strengths**: Energy consumption is indeed an important factor to consider in NAS. This benchmark can help facilitate further research and development on energy-efficient NAS. This work presents an interesting showcase of using MOO to find energy-efficient NAS.

**Questions and concerns**: This benchmark changed the search space of NAS-Bench-101. For example, instead of evaluating a 7V space, this work tests 5V and 4V spaces. Can the authors justify why this change is made? Clarity of the results: The results presented in Figure 2 are not very clear to me. Figure 2a shows the Pareto front obtained from one of the MOO runs. But what do the different markers and colors mean? It is my understanding that the blue, green, and yellow points correspond to the three types of architecture, including two extrema and a knee point. Why the red one, i.e., the optimal solution in SOO is, not shown in the figure? It would be great if the authors could establish benchmarked baseline performance on this new benchmark.
clarity,_quality,_novelty_and_reproducibility: 1. Clarity: The paper is clear overall. But there are some places in the empirical evaluation that are not very clear.   
2. Quality: The quality of the presentation can be improved. 
3. Reproducibility: The authors mentioned that the proposed benchmark is open-sourced. This makes me believe the reproducibility is not an issue. 
summary_of_the_review: The established energy-aware NAS benchmark is a very good contribution. However, the presentation of results could be improved. And benchmarked baselines should be added.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed an energy consumption-aware tabular benchmark for NAS based on NAS-101, called EC-NAS-Bench. EC-NAS-Bench contains the training energy consumption, power consumption and carbon footprint of each architecture in the benchmark. The author performs single-objective optimization and multi-object optimization (based on the algorithm proposed in the paper "Multi-objective optimization with unbounded solution sets") on EC-NAS-Bench, and noticed that multi-objective optimization is able to figure out architectures with about 70% energy reduction and <1% performance degradation

strength_and_weaknesses: Strength

- Incorporating energy consumption data in NAS is an important problem. The author took the first step in this direction. Experiments show that it is able to find architecture that consumes much less energy but with comparable performance.

Weakness

- Different from the performance of the model, energy consumption of neural networks is a moving target. This is related to factors like global economy and the computational hardware. The author has not discussed how they will incorporate potential changes in the energy/power consumption and carbon footprint of the models.
- The multi-objective optimization algorithm is largely based on the cited paper "Multi-objective optimization with unbounded solution sets". Are there novel components proposed in the paper?
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is overall well-written. The author can improve the description of the MOO algorithm (e.g., give more details in the Appendix)

Quality + Novelty: The reviewer is concerned that it may be difficult for other researchers to continue their experiments on EC-NAS-Bench. The reason is that new hardwares appear every year and EC-NAS-Bench needs to be updated accordingly. The reviewer agrees that incorporting energy consumption in NAS is an important topic but EC-NAS-Bench won't be very useful when new hardwares appear.

Reproducibility: The author attached the source code. The reviewer has not ran the source code and only reviewed the content. It looks reproducible.
summary_of_the_review: I vote for weak rejection because EC-NAS-Bench may soon outdated when new hardwares appear. Thus, this benchmark may not be useful to the research community.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a tabular benchmark that includes various energy consumption metrics for a subset of networks in the NAS-Bench-101 search space. The authors provide simple multi-objective and single-objective baselines and run those on the benchmark.
strength_and_weaknesses: I find the proposed benchmark and the motivation for using such benchmark beneficial to the community as the awareness for resource efficient NAS increases. The paper is in general easy to follow and well-structured. The authors also release their codebase together with the API. However, I find the paper could improve with the following:

- **A larger and novel search space**. The smaller versions of the NAS-Bench-101 search space do seem way far from being realistic in my opinion. With 91 and 2532 architectures, respectively, they do not provide a realistic testbed and even the simplest search algorithms would lead to optimal solutions pretty quickly. Moreover, these spaces do not provide support for one-shot NAS methods.

- **More multi-objective algorithms to evaluate**. The authors only evaluate a single multi-objective algorithm in Section 4. It would be beneficial if they would add more methods to this section. Check [1] for some simple methods.

- **More on the benchmark than the algorithms used**. The authors spend most of Section 3 by describing the multi-objective optimization algorithm they use to evaluate on their benchmark. While such a detailed description is appreciated, it does not serve the main purpose of the paper and therefore I do not find it necessary to be included in the main paper. Rather, more statistics and analysis of the benchmark would be more useful.

-- References --

[1] Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization. Guerrero-Viu et al. 2021
clarity,_quality,_novelty_and_reproducibility: The paper is in general easy to follow. The authors provide the code with the supplementary material. The main issue with this submission is the novelty. When proposing a new benchmark, there should be some novel component in terms of the search space, benchmark construction (surrogate model used for instance in surrogate benchmarks), or empirical evaluations that provide major unforeseen insights into the field. Unfortunately, none of these criteria is fulfilled.
summary_of_the_review: The motivation for having a benchmark that provides energy consumption metrics is valid and would be beneficial for the NAS community. However, I think this submission is not ready for acceptance due to the reasons I mention above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: While the reviewers appreciated the paper’s motivation, the idea of using MOO for energy-efficient NAS, and the clarity of the paper, their main concerns were with (a) the realism of the benchmark, and (b) the divided focus of the paper. Specifically, for (a) they argued that smaller versions of the NAS-Bench-101 search space seem too small (91, 2532 architectures) to provide a realistic testbed for NAS algorithms. They argue that simple search algorithms which would work well here, may not work well in a more realistic setting with tens or hundreds of thousands of possible architectures. The authors responded by agreeing that those datasets are not sufficiently representative. They point out the computational complexity of obtaining results for their larger proposed benchmark, but then say they take a tip from the reviewer about using a surrogate model to predict energy consumption for this larger benchmark. To train this surrogate they subsample 4300 architectures (out of 423,000) and learn an MLP to predict energy consumption. They report that this model is accurate (0.9977 Pearson correlation). This speeds up training significantly and they present new results with this surrogate model. However, as far as I can tell, surrogate accuracy is computed with respect to the same subsampled architectures that were used to train the surrogate model. So it’s not at all surprising that they were able to achieve such a high correlation with an MLP. There is no indication that this surrogate is accurate for architectures outside of the training set. For this reason, the issue of a useable realistic energy-aware NAS benchmark is still open. For (b), the reviewers were confused that the paper seemed to spend so much space presenting a novel MOO method when the main goal of the paper seemed to be presenting a benchmark. The authors responded by shrinking this section, using the space to describe the surrogate above. I think this was an improvement, but there is still 3/4th of a page free in the paper, it’s unclear why say, Appendix C is not moved to the main paper given the additional space. For these reasons I vote to reject at this time. Once these things are fixed this paper will be improved.
justification_for_why_not_higher_score: See above metareview.
Decision: Reject