Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents the analysis of 13 zero-cost proxies on 28 tasks, and releases the pre-computed scores of them. The authors also studied a way to combine all zero cost proxies to allow better NAS performance of predictior-based algorithms.
strengths: - A good summary and comprehensive comparison of zero-cost proxies published.
- Emprical analysis of the performance of many zero-cost proxies. A deep dive into the generalization abilities of them on different search spaces and tasks. This kind of analysis is not done before.
- Understanding if the zero-cost proxies can explain the ground truth accuracy, and how they complement each other (in terms of information gain).
- A novel way to combine all zero-cost proxies studied as features into the NAS predictor.
weaknesses: - Most results are empirical which provide some valuable insight into so many ZC proxies. However, there is no theorethical analysis or explaination on such observations.

correctness: To the best of my knowledge the claims made are correct. The dataset is constructed property with detailed documentation and code repository. The evaluation methods are well designed.
clarity: The paper is well structured and written.
relation to prior work: Section 2 mentioned predictor-based NAS methods but only [46] is cited. There are more related work:
* BRP-NAS: Prediction-based NAS using GCNs. https://arxiv.org/abs/2007.08668
* Bridging the gap between sample-based and one-shot neural architecture search with bonas. https://arxiv.org/abs/1911.09336
* Bananas: Bayesian optimization with neural architectures for neural architecture search. https://arxiv.org/abs/1910.11858
additional feedback: - The greedy approach is not visible in Figure 4?
review_title: Comprehensive evaluation on an under-studied area

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In this work, NAS-Bench-Suite-Zero evaluates 13 zero-cost proxies for architecture performance prediction across 28 tasks. It creates by far the largest dataset, enabling orders-of-magnitude faster experiments on ZC proxies. The provided codebase is accessible through a unified interface, created with the aim to facilitate reproducible, generalizable, and rapid NAS research.
strengths: 1. The collection of benchmarks and ZC proxies that unifies and accelerates research on ZC proxies – a promising new sub-field of NAS – by enabling orders-of-magnitude faster evaluations on a large suite of diverse benchmarks.
2. The large-scale analysis of 13 ZC proxies across 28 different combinations of search spaces and tasks reveals the generalizability, bias, and mutual information among ZC proxies.
3.  As a kind of complementary information,  ZC proxies can significantly improve the predictive power of surrogate models commonly used for NAS
4. The paper is easy to read, and in terms of the open source code, it provides an easy-use interface and well-organized documentation. 
weaknesses: 1. It is largely based on NAS-Bench-Suite, and the main difference comes from more datasets, publicly releasing ZC proxy values, combining ZC proxies in a nontrivial way, and exploiting the complementary information of 13 ZC proxies simultaneously.
2. The in-depth analysis from Section4 could have been more as the basic information largely overlapped with NAS-Bench-Suite. The bias part in Section 4.3 is interesting, perhaps more analysis in the relation between the design principle of zero-cost proxy and results.
3. The definition and explanation in the table content can be more detailed such as in Table3.


correctness: The claims made in the submission look correct. The evaluation methods and experiment design are appropriate and performed correctly.
clarity: The paper is well written and easy to follow.
relation to prior work: It mainly extends the zero-cost part of NAS-Bench-Suite. The difference from the previous contributions is clearly discussed. 
additional feedback: Please check the Weakness part.
review_title: Official Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Recent prior work introduced zero-cost proxies (ZC proxies) as a means to predict architecture performance to significantly speed up neural architecture search algorithms. 
This work introduces a novel benchmark dataset "NAS-Bench-Suite-Zero" to study 13 ZC proxies across 28 tasks.  
Main contributions and findings are:
* Introduced largest (to date) benchmark dataset for ZC proxies, streamlining and speeding up experimentation
* Information-theoretic analysis of ZC proxy behavior yielding finding that they capture complementary information
* Evaluation showing that incorporating all 13 ZC proxies into NAS surrogate models improves their predictive performance significantly
strengths: * Very detailed documentation of creation of the benchmark suite, research questions as well as experimental setup. Following best practices throughout (clearly stating infrastructure detals, hyperparameter choices, filled-out NAS best practice checklist, etc) 
* Benchmark dataset access is straightforward and well documented, all code and supporting materials (e.g., figures) are included in the git repository
* Appendix contains a very detailed data sheet addressing many common questions users may have and extending even to ethical and social implications - this is outstanding
* Writing and elaboration of research questions and corresponding answers is very clear and accessible
* research findings are very thought-provoking and frequently point out clear directions for future research,  specifically:
    * Using ZC proxies as task features for meta-learning
    * Degree to which ZC information is complementary is task-dependent (motivating experimentation in this work to combine them by adding corresponding features to the XGBoost inputs)
weaknesses: * A potentially missed opportunity may consist in performing an analysis of feature importance of the ZC proxy features on the trained surrogate model. From the information-theoretic analysis we can argue that combining multiple ZC proxies is beneficial. However, it is not obvious to me from these results that we can exclude a merely marginal contribution to the speedup by individual ZC proxies. Feature importance values may indicate which (combinations of) proxies actually contribute to observed speedups.
* RQ3 answers seem a bit inconclusive, i.e. not immediately actionable. It may be helpful to outline how removing these biases would work, in a concrete case.
correctness: Construction of the dataset and decisions to include the selected 13 ZC proxies and 28 benchmarks are valid and explained well. Evaluation methods and experiment design for the benchmark experiments are appropriate and conclusions drawn from the results are valid.
clarity: The paper is written in a very accessible way and all research questions and associated findings are very thoroughly documented (including further details in the appendix). 
relation to prior work: The authors do a great job situating their contributions within the prior research work in this domain. 
It is clear, in what ways the new dataset differs from prior benchmarks. The new results that highlight complementary information in different ZC proxy methods as well as variability across benchmarks can confirms and explains some earlier findings referenced in this section (e.g., work cited in this one that had found a strong dependence of ZC proxy effectiveness on search space used).
additional feedback: Thank you for this insightful and inspiring work.  

I have two questions (these could also just be relevant as future directions):

* The work suggests that ZC proxies may be used to assess similarities across benchmarks and serve as task features for meta-learning (see "Answer to RQ 1"). Could one turn this around and instead use a separate set of task features to learn which ZC proxies are likely to provide benefits for a given benchmark?
* What is an optimal number of ZC proxies to ensemble (this ties into my point around feature importance analysis above)? Are all 13 proxies needed? I believe this question is likely to grow in importance if the benchmark grows and additional ZC proxies get developed and integrated.
review_title: Very well written and a great example of reproducible research

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper targets the ZC-NAS, which is an interesting and promising subfield of NAS, and aims to answer the important questions about the performance, generalization, and bias of ZC proxies and their combinations. To achieve this, the authors create the `NAS-Bench-Suite-Zero` codebase, precompute ZC scores on the architectures in existing benchmarks, and run a comprehensive analysis based on them.  
 
strengths: 1. The authors create the `NAS-Bench-Suite-Zero`, an extensible collection of 13 ZC proxies on 28 NAS benchmark tasks. Such proxies are precomputed and accessible through a unified interface. 
2. The authors run a large-scale analysis of the above dataset to study the generalizability, bias, and mutual information among ZC proxies.
weaknesses: 1. About the search space selection. As [3, 45] and other works including this one observe, the performance of ZC proxies highly depends on the search space and task. However, the search space used in this benchmark is far from the modern search space, such as ResNet and MobileNet [3]. This limits the significance of this work and makes observations in it less guiding to the real-world NAS applications. 

2. The authors use Spearman ranking correlation to evaluate the effectiveness of ZC proxies. However, ranking correlations cannot naturally reflect the true ability in the scenarios of (constrained) NAS [3,25]. That's why other metrics such as the Precision@K, BestRanking@K [25].  

3. The authors use conditional-entropy based information gain to prove the improvement of predictive power of a tuple of ZC proxies. The idea is interesting, but the IG is considering the whole distribution, which also deviates from the goal of NAS to select the top-`k` architectures. 
correctness: The dataset is constructed in a sound way. 
clarity: 1. Some citations have already been officially published in peer-reviewed conferences, but the authors still cite the pre-print version. 
relation to prior work: yes
additional feedback: a few additional questions

1. the necessity of introducing IG in Sec. 4.2 other than using xgboost with different combination of ZC proxies (like Section 5)
2. The discretizing (D.2) to group the ground truth accuracy into `round(1 + 3.322 ∗ log(N )))` bins. The paper doesn't give detailed bin size (how many samples are in each bin), but it could be estimated (through Table 2) that the number of bins is not so large.  Could this rough estimation be accurate enough for NAS? For example, even the top bin may have more samples than the final selected. Thus, a smaller conditional entropy may not help to predict whether a sample is in the top bin.
review_title: review of "NAS-Bench-Suite-Zero"

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work replicates some of the typical Zero NAS on multiple NAS benchmarks.This work gives Zero NAS fair comparison data and gives some new research challenges such as on trans datasets.However, I still think the contribution of this work is not enough for the top conferences.
strengths: 1. This work is very well organized and written.
2. The authors have done more experiments and have done a good job of open source work.
weaknesses: 1. This work seems to me to be just a review result, and it is more appropriate to appear in a new NAS method rather than a new benchmark, since almost all methods and datasets exist, and much of the data has already appeared in previous work.
2. Some of the data from this work is questionable, such as why it is so bad on Trans NAS.
correctness: Yes
clarity: Yes
relation to prior work: Yes
additional feedback: No
review_title: Solid work but lacking some highlights

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper creates a NAS benchmark NAS-Bench-Suite-Zero including 13 zero-cost(ZC) proxies, 28 tasks and 1.5 million ZC proxy scores in total. The dataset can be used to speed up ZC proxy-based NAS experiments and provide convenience for ZC proxy-based NAS research. The authors also conduct generalizability, bias and information-theoretic analysis of ZC proxies, showing that combining several ZC proxies can improve the performance of NAS surrogate models and NAS algorithms.
strengths: [+] The ZC proxy dataset can significantly speed up ZC proxy-based NAS experiments.

[+] The codes are accessible in github. Details of reproducibility are included.

[+] Extensive analysis of 13 ZC proxies across 28 different combinations of search spaces and tasks by studying the generalizability, bias, and mutual information among ZC proxies.

[+] The paper gives several valuable findings, e.g. combining several ZC proxies can improve the performance of NAS surrogate models and NAS algorithms.
weaknesses: [-] Although there are docs about how to reproduce the results in the paper, the interfaces using the datasets are missing. For further research, the authors are expected to provide detailed documentation and quickstart about how to use the codebase. 

[-] Conducting more theoretical analysis would be better.

correctness: Yes, all the claims are verified.

clarity: Yes, the paper is well-written and easy to follow.

（Minor typos）
1. Figure 4 caption, "top right/left"
2. Line 292
relation to prior work: Yes. Related works are discussed in Main paper Sec. 2 and Appendix Sec. C. 
additional feedback: None
review_title: A Good NAS Benchmark on Zero-cost Proxies.

Meta-review: metareview: This paper presents a systematic review of zero-cost proxies for neural architecture search. By design, these proxies are cheap and easy-to-evaluate but results reported in the literature have been mixed. In this light, a benchmark seems relevant and important for further progress in this area. The reviewers uniformly appreciated the extent of tasks and proxies considered. Moreover, the reviewers also appreciated efforts made to explore the complementary strengths of these proxies. There were some concerns related to the search space and metrics used, but those were addressed by the authors during the rebuttal period. Finally, the availability of open-sourced and well-documented code is a plus that was appreciated by everybody. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept