Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper introduces a benchmarking framework to evaluate a class of one-shot NAS methods by exploiting the NAS-Bench-101 database. In order to do so, the authors apply techniques from Bender et al. (2018) to match the search spaces in NAS-Bench to one-shot scenarios and weight edges in the search graph allowing for a variable number of edges per cell. These weights are then used to query a larger, discrete architecture from NAS-Bench for the corresponding evaluation errors. The authors motivate the soundness of a unique framework by showing minimal differences between three NAS optimizers. 
In their analysis, they also corroborate the findings of Sciuto et al. (2019) on a larger scale, showing little generalization of the ranking from one-shot validation to the final architecture. Finally, the authors also use this framework to show that a fine-tuned baseline compares favorably to SOTA methods, and that large regularization factors lead to more robust configurations.

The paper is well-written and introduces a framework to cheaply compare one-shot NAS optimizers based on expensive computations behind the NAS-Bench-101 benchmark. In addition, the authors provide empirical analyses that reflect generalization and regularization issues with current methods and that could lead towards designing more robust algorithms. 

In my opinion, the the background section might be improved by presenting higher level notions that would lead to an easier understanding by a wider audience. Also, although comprehensible due to the nature of this study, readers not familiar with a variety of previous work might find difficult to parse the paper as many concepts are just referenced.
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: 
This paper proposes a benchmark dataset for evaluating One-Short Neural Architecture Search models. It extends on the idea of the NASBench-101 (Ying et. al., 2019) into One-Shot models. 

I think this is important work. With all the development of neural archictecture search methods, reproducible research is paramount. This is even more timely considering the fact that these problems are computationally expensive, so it is even more computationally difficult than usual to run exhaustive comparisons among proposed methods. 

The paper is well-written and the design decisions are clearly explained. The comparison of NAS methods is also interesting to read. In general, I think this is a worthy publication. 

review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: In this submission, the authors present a benchmark NAS-Bench-1Shot1 for one-shot Network Architecture Search. The presented benchmark reuses the existing NAS-Bench-101 that only contains discrete architectures, and the authors give a method to discretize architectures to match NAS-Bench-101. Also, the authors claim that they introduce a general framework for one-shot NAS methods.

Overall speaking, the technique contribution and novelty of this submission requires further improvement, and I suggest to rejecting this submission. The reasons are detailed as follows:

1) The main contribution of this submission is the benchmark NAS-Bench-1Shot1, which can benefit future research for one-shot NAS. However, this benchmark is based on NAS-Bench-101, and the difference is that NAS-Bench-101 only contains discrete architecture, while the presented NAS-Bench-1Shot1 has a component to discretize architectures and then match to NAS-Bench-101. The novelty may not be enough.

2) The authors claim that they introduce a general framework for one-shot NAS methods. Personally, I think this is over-claimed. There are several variants of DARTS, and the authors just implement these variants and DARTS itself in a unified code base. It would be a true general framework if it can also work with other one-shot NAS methods such as ENAS and the rest ones.

Given these, this submission requires further improvement, especially in terms of technique contribution and novelty. 

One interesting thing is that the authors try to use HPO method such as BOHB to tune the hyperparameter of NAS methods.
review_title: Official Blind Review #1

Meta-review: The authors present a new benchmark for architecture search. Reviews were somewhat mixed, but also with mixed confidence scores. I recommend acceptance as poster - and encourage the authors to also cite https://openreview.net/forum?id=HJxyZkBKDr
Decision: Accept (Poster)