Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Overview: This work proposes a new NAS benchmark based on the results of surrogate models prediction. This surrogate model is able to predict all architectures in DARTS search space, which is about 10^18 possible architectures. The author compared the predict performance among different type of surrogate models and also leveraged surrogate models to investigate different NAS methods.
main_review: Pros. 
1. the authors have made several significant improvement over the past submission and I'm happy to see that. Compared to the prior submission, the authors have included results on CIFAR10, CIFAR100, and ImageNet. I thank authors for doing that, however, there is still some concerns for me to see this work get published.

Cons:
1. Apparently this work is using black box optimization to perform NAS. However the selected benchmark baselines are pretty old. I'd suggest authors taking a look at the recent public benchmark results from here: https://bbochallenge.com/ (the black box optimization challenges held at NeurIPS 2021). And update the paper accordingly to track the recent advance in the field. Essentially NAS is no different from those test functions.
2. All the results are plotted by the wall time, which is a bit tricky in NAS. The costs of querying a surrogate model can be cheap, but sample-efficiency (#samples over accuracy) is far more important to NAS as the cost of evaluations are the main bottleneck, not the search. Please correct me if I'm wrong, but I don't think you consider the cost of evaluations in the wall time.
3. Please diversify the tasks. Here I mean different tasks, e.g. detection or segmentation, rather than image recognition with different datasets and models. 




summary_of_the_review: I have been reviewing this paper for a few times, and I appreciate the effort that the authors have put to update the paper. However, the current draft can not truly reflect the recent advancement in the field, and it may mislead the future. Once the author have address my concerns 1, 2, I'm open to accept this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work explored how to use surrogate models to expand the existing (and limited) neural architecture search -- NAS -- benchmark. The new expanded benchmark is named as surrogate NAS benchmark. All codes are open-sourced, which demonstrated the good reproducibility of this work. The authors have conducted extensive experiments to demonstrate the usability of this new surrogate NAS benchmark and showed much analysis of existing NAS methods on this new benchmark.
main_review: In general, this is a good NAS paper that explored a new direction -- surrogate NAS benchmarks. I believe this work stand between 8 and 10. Please see my detailed comments below.

Strengths:
- In this work, the authors proposed a novel family of NAS benchmark -- surrogate NAS benchmark. This can break the limitations of existing NAS benchmarks -- the search space is unrealistically small.
- The effectiveness and usability of the proposed new surrogate NAS benchmarks have been systemically evaluated and analyzed.
- This work brings some fresh ideas and artifacts to the NAS community. Given various bottlenecks, the development of the NAS research is significantly slowed down. This work can potentially help bring some new insights.
- Section 2 is interesting that the surrogate model can yield strong/better predictive performance than standalone training.
- All codes have been released.
- We can see many NAS benchmarks were accepted in the top venues this year. The contribution, novelty, analysis of this work is above the most recent published NAS benchmarks.

Weakness:
- As surrogate HPO benchmarks have been proposed, it would be good to have a deep analysis on the comparison with that.
- I do not find other clear weaknesses.

Minor issues:
- In the first paragraph, the authors cited (Hao, 2019) for carbon emissions. A more proper reference for the carbon emissions of NAS algorithms would be "Carbon Emissions and Large Neural Network Training" and "Full-Cycle Energy Consumption Benchmark for Low-Carbon Computer Vision", which particularly discussed the carbon emissions of NAS methods.
- Some works have been officially accepted and then it is suggested to cite their official version instead of arxiv version, such as Dong et al 2020 NATS-Bench arxiv 2020 -> TPAMI 2021
summary_of_the_review: A good paper for the NAS community; hits a sweet spot for the current bottleneck of the reproducible NAS and the scientific research of NAS. I would strongly recommend accepting this work for ICLR.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a method to create surrogate benchmark for NAS. By modelling and predicting the performance of neural architecture in the search space, a much larget search space can be covered without expensive training.
main_review: Pros:
- **Clear motivation**: This work addresses 1) the limitations of small tabular benchmarks, 2) the expensive time and compute cost of large non-tabular search space. For instance, 60k models are sampled from the DARTS search space and their performance are predicted using the surrogate model. This creates a tabular benchmark which is much bigger than NAS-Bench-2.
- **Comprehensive analysis**: The paper has considered potential issues of using surrogate models and has proposed ways to work around them. For example, to yield good coverage of the search space and to train a strong surrogate model, a sampling scheme is used to collect models by random search as well as high performing models by 10 NAS methods.
- **Guidline for future research**: The paper summarizes several best practices to create surrogate NAS benchmarks for new search spaces. It is particular important as there are many promising architectures that are not currently covered in DARTS, or NAS-Bench-1/2. For example, research in transformer-like architectures are increasingly popular. This paper provides a feasible method to fast evaluate new search spaces.

Cons:
- **Search trajectories**: The paper has done lots of experiments to show that the search trajectories running on the surrogate benchmark closely resemble the ground truth. However, it is unclear if the searched models (via surrogate and via ground truth) are also similar. For example, a surrogate may underestimate the error of certain region of the search space, and the NAS method may be guided to favour models in that region. Even though the search trajectory may look right, the models explored could be different from the ground truth. If you take the models searched by different NAS methods on SNB-DARTS and train them, do they still preserve the same ranking?
- **Performance on unseen models**: In SNB-DARTS, 60k architectures are sampled and 0.8 of them are used to train the surrogate model. Since the search is performed on the same 60k search space, it is not suprising for the surrogate model to perform well. How about the unseen models in the DARTS search space? Similarly, NAS-Bench-1 has 423k architectures, which is bigger than SNB-DARTS. Any insight on how SNB-NAS-Bench-1 compare to the tabular NAS-Bench-1?
summary_of_the_review: Overall, the paper is very well written. It is clearly motivated and supported by lots of ablation studies. The dataset and search space might not be SOTA, but it does provide a promising practice for future NAS research. There are a few questions that I would like the authors to clarify, otherwise, I would recommend acceptance of this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors present another benchmark tabular benchmark for neural architecture search. In contrast to most other works, only few architectures are actually trained while the remaining ones are imputed using a regression model.
main_review: Overall, the paper is well-written but it does not contain significant contributions.
The two main contributions are the use of a surrogate to create a surrogate benchmark and the creation of an interesting new search space for NAS.

I see little contribution of demonstrating how to create a surrogate benchmark for NAS. This has been done earlier for hyperparameter optimization as pointed out by the authors themselves and NAS is nothing but a specific type of hyperparameter optimization. Additionally, Yan et al. presented with their paper "NAS-Bench-x11 and the Power of Learning Curves" already one way to create surrogate benchmarks for NAS.
While the authors claim that a larger search space might be of huge interest for the community, no evidence for this claim was provided. There is no reason to believe that a larger search space is more interesting than a smaller one if not chosen correctly. Furthermore, the search space considered (among others) is the DARTS search space which is well-known and well-explored by the community. Datasets such as CIFAR-10 and CIFAR-100 are used in many NAS papers and the community is already overfitting to these tasks such that they started looking for more challenging problems. Concluding, I do not see this benchmark to be particularly useful.

The authors don't discuss all the shortcomings of their benchmark. It is important to point out that the most efficient NAS methods do not evaluate an architecture completely. Methods such as DARTS or Hyperband search differently and it seems like that these methods cannot be evaluated on this benchmark. While methods exist that evaluate architectures completely, they have no practical purpose since they are simply too expensive to run.

A comment on societal impact: The authors claim that millions of GPU hours will be saved and carbon emissions will be reduced. This is a bold statement and not supported by any evidence. What we have learned from the very first NAS benchmark is that it led to more and more NAS benchmarks, only spending more GPU hours. If the authors want to take the societal impact section seriously, they should consider the risk that they have potentially wasted energy and hardware for no reason.
summary_of_the_review: Overall a well-written paper with no significant novelties. The presented benchmark does not seem to be a useful addition to the vast amount of already existing benchmarks.
review_title: null

Meta-review: This paper proposes a methodology to create cheap NAS surrogate benchmarks for arbitrary search spaces. Certainly, the work is interesting and useful, with comprehensive studies to validate such approach. It should be credited as belonging to the first efforts of introducing and comprehensively studying the concept of surrogate NAS benchmarks. In AC's opinion, it is a solid paper that will (or has already) inspire many follow up works. The paper is well written. 

This paper received highly mixed ratings. Although the authors might not see, all reviewers actually participated in the private discussions. Reviewer 1eb8 indicated hesitation in her/his support. Reviewer yTPb stated that if not considering the arXiv complicacy, she/he "would certainly raise score by one level".  AC also reached out to Reviewer yTPb about her/his mentioned possibility of updating scores, and got confirmed that her/his original opinions wasn't changing after rebuttals. Besides, AC agrees the arXiv/NeurIPS complicacy shouldn't brought into the current discussion, and ignored that factor during decision making. 

The main sticking (and considered-as-valid) critique is on the relatively outdated and incomplete selection of baselines. As a benchmark paper, it should capture and diversify the recent methods. For example, the authors might consider adding: https://botorch.org/docs/papers (latest methods in Bayesian Optimization) https://github.com/facebookresearch/LaMCTS (latest methods in Monte Carlo Tree Search) https://facebookresearch.github.io/nevergrad/ (latest methods in Evolutionary algorithms)

Given the above concerns, AC considers this paper to sit on the borderline, and perhaps with pros outweighing the cons. Hence, a weak accept decision is recommended at this moment.
Decision: Accept (Poster)