Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper present a wrapper for 25 NAS benchmarks and provide insightful analysis on NAS and predictor performances across the benchmarks. They show, among other things, that no NAS method is best across all tasks, that hyperparameters of NAS are not robust and that small tabular benchmarks are not representative of NAS method performances on larger benchmarks. 
main_review: Strengths:
- Very important topic. Benchmarks are critical to assess performance of NAS algorithms.
- Provide statistics across a wide array of NAS benchmarks, such as the distribution of accuracy across the search space.
- They show that no NAS method clearly outperform all other method across all benchmarks. Some do better with HPO, but this is typically to expensive to computer for each benchmark.
- Tests 2 relevant questions. 1) Whether NAS results generalizes from the small 101 and 202 benchmarks to larger ones and 2) NAS have robust hyperparameters. The third question seems to be to closely related to the second one: 3) We can cheaply optimize hyperparameters of NAS on tabular benchmarks and reuse them on another benchmark.. 
    - They show that 1) NAS algorithms do not generalize well from small 101 and 202 benchmarks to larger ones,
    - 2) Hyperparameters are not robust for NAS algorithms.
    - 3) Is not true obviously since hyperparameters of NAS algorithms are not robust.
- They contribute a valuable wrapper for 25 benchmarks, although it is not clear to me how the search space representation with NetworkX allows for a dynamic search space definition. 

Weaknesses:
- Out of the 3 questions, it seems to me the third one is to closely related to the first one. And the experimental answer to this question is not clearly outlined in the paper neither.
- I am quite concerned about the maintainability of the library. The benchmarks are bash scripts that will be difficult to maintain. Some even contain specific slurm configurations for the cluster of the researchers. 

Random walk plot is interesting. I would be great to have a confidence interval or at least a standard deviation for each curve.
summary_of_the_review: The wrapper of benchmark is a valuable contribution by itself, but this paper illustrates well the benefit of such benchmarks by providing insightful analysis of NAS algorithms.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors collected most of the existing NAS benchmarks to construct a new benchmark.
A unified API is provided to use these existing search spaces and architecture datasets. Based on this the authors re-analyze some NAS algorithms on this new large and comprehensive benchmark and have some interesting observations.
main_review: strengths:
- The paper is well motivated and written.
- This new benchmark can be used to do a more deep analysis for the NAS methods.
- Some interesting observations.

weaknesses:
- The technical part is somewhat weak.
- The table and figure captions are too short to explain themselves. Please at least explain the abbreviations used in the table or figure.
summary_of_the_review: The authors did much engineering efforts and emprical analysis, whereas the technical novelty is a little bit weak.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates the drawbacks of current existing NAS benchmarks and the works that evaluate on them. Then it presents NAS-Bench-Suite, an extensible collection of NAS bench-marks which is easy-to-use to facilitate reproducible, generalizable and rapid NAS research.
main_review: There exist many NAS benchmarks currently across diverse tasks. Many NAS works conduct experiments on several benchmarks to demonstrate the effectiveness and believe. 
However, researcher find that NAS algorithms may perform differently across these benchmarks. Sometimes even reverse conclusions are drawn.

This paper investigates this problem and points out that the differences may come from the different search space design, training pipeline and hyperparameters, via comprehensive experiments across many existing benchmarks. It finds that conclusions from a few benchmarks do not generalize well to other benchmarks

Further, the paper proposes NAS-Bench-Suite, a collection of NAS benchmarks, with a unified interface, which is easy-to-use, to facilitate reproducible, generalizable and rapid NAS research.

The motivation of the work is clear. The experimental analysis is convincing. The paper is well organized. The claim is sufficiently addressed by the method.

Overall, this is a good work for the NAS community.
summary_of_the_review: I think the paper is of good quality and will contribute to the NAS community.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a unified interface for access to a collection of Neural Architecture Search (NAS) benchmarks. With experiments performed on across multiple search spaces and datasets, the authors show that some conclusions drawn from a small subset of benchmarks do not generalize across diverse datasets and tasks.
main_review: The results are interesting and very useful for the community as it is - and the authors should be commended for their effort.
However the work lacks deeper insights from the experiments performed that could guide the community towards better methods. For example: [Fig.2] Are there identifiable components of architectures that lead to the interquartile range differences between the CIFAR10 benchmarks (NB101, NB201, DARTS)? Or [Table 2] What makes XGBoost generalize well where others don't?

Sec.4.2: Transfer is always expected to lead to a performance drop. Is there evidence that NAS is especially bad in this respect, compared to HP transfer in other settings? (say, transferring ResNets HPs from one dataset to another)

Fig.5: Since the figure contains a lot of information, could the authors provide some useful summary statistics in addition?

Minor: clarify whether $i$ and $j$ are columns or rows in Fig.5. 
summary_of_the_review: In my view the two main contributions of this work are (1) showcasing the limitations of drawing conclusions from single benchmarks; (2) providing the community with a larger analysis tool.

I expect that the community will welcome these resources and thereby **recommend the paper is accepted** (provisional score: 8)
review_title: null

Meta-review: This paper proposes a novel benchmark for neural architecture search methods, which consists of 25 different combinations of search spaces and datasets. The main motivation is that existing NAS benchmarks, such as NAS-Bench-201, consider very small search space and few datasets, such that conclusions drawn with them do not generalize to unseen settings with different search spaces and datasets. The authors first describe the 25 different combinations of the search space and tasks for the given benchmark, and then conduct an extensive empirical study of existing NAS methods and performance predictors with the proposed benchmark, to show that architectures and hyperparameters found with the popular benchmarks do not generalize to other settings, which is consistent with their assumption.

—

All reviewers were initially positive about the paper, and remained positive throughout the discussion period. The reviewers found the paper well-motivated, and the proposed benchmark useful, as they agree with the need of introducing a single, unified framework that can validate a NAS method under diverse settings, since existing benchmarks only consider specific datasets and search spaces. However, the reviewers were also concerned with the weak technical novelty (Reviewer 2xvD), and that the work lacks deeper insights that could guide the community towards better methods (Reviewer Gku7). 

I also agree with the authors and the reviewers on the necessity of having a unified benchmark that incorporates all different settings considered in the previous benchmarks, and find the extensive empirical study of existing NAS methods useful. 

However, I find the work as rather technically weak as mentioned by R2xvD, since the authors spent too much time describing and showing the limitations of existing benchmark methods, while what is more important for benchmarks, is to justify how the proposed benchmark can evaluate the performance of different methods in a fair manner, while being representative of the practical settings. In short, the authors need to justify their design choices. Yet, the 25 settings proposed in the paper seem to have been arbitrarily chosen, and it is not clear if having a good performance on this benchmark is indeed a fair evaluation, or well-reflects how the NAS method will perform in practice. The proposed benchmark also does not really consider a novel search space or setting that have been overlooked in the past either, and does not provide much insights on the problem, as mentioned by Reviewer Gku7. 

Thus, although I recommend an acceptance for its practical value acknowledged by the reviewers, the authors need to put a considerable amount of effort in revising the paper, and If this were a journal submission, the paper may need to undergo a major revision. Most importantly, as described, the authors should justify their design choices as well as whether evaluating a model on the benchmark yields “fair” and “representative” results, focusing more on describing the proposed benchmark itself.
Decision: Accept (Poster)