Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This works studies the evaluation of search strategies for neural architecture search. It points out existing problems of the current evaluation scheme: (1) only compares the final result without testing the robustness under different random seeds; (2) lacking fair comparison with random baseline under different random seeds. The authors analyzed three popular NAS methods with weight sharing (ENAS, DARTS, NAO), and showed that they don't significantly improve upon random baseline on PTB and CIFAR-10. On a reduced search space of RNN and CNN (NASBench), they showed that the three methods fail to find the best performing architecture. Then they compared search with and without weight sharing and showed the correlation between architecture performance under the two conditions in a reduced search space, which indicates the weight sharing is a potential cause for the suboptimal performance.

I recommend acceptance of the paper for the reasons below.

(1) It pointed out some important issues in the evaluation of NAS methods: evaluating under different random seeds and fair comparison with random baseline.
(2) The analysis is supported by experiments in the original search space and a reduced search space, which makes the result more convincing.
(3) It proposed the weight sharing as a potential cause and supported the hypothesis with experiments in the reduced search space, although more experiments in a realistic search space are needed to make the conclusion more solid.

Weakness:

(1) The problem that the search space is over-optimized and constrained is not unnoticed before. For example, table 1 in (Liu et al, 2018) showed that the random search baseline performs not much worse than the DARTS (~0.53% difference), which is similar to the conclusions on CIFAR-10 presented in this work. 
(2) More recent works in NAS is already evaluating under multiple random seeds and performing fair comparison with random search baselines, for example, (So et al, 2019). There should be more discussions about such improvements in the rigorous evaluation of NAS. 
(3) The comparison between with and without weight sharing in section 4.3 is interesting, but there should be more support in a realistic search space, because the landscape could be very different. Otherwise, it is better to make clear the scope of the conclusion, for example, instead of "in CNN space, the ranking disorder ...", it is better to use "in a reduced CNN space, ...". 

"Darts: Differentiable architecture search." Liu, Hanxiao, Karen Simonyan, and Yiming Yang.  ICLR, 2019
"The Evolved Transformer." David R. So, Chen Liang, and Quoc V. Le., International Conference on Machine Learning. 2019.

Typos:
"based one their results on the downstream task." -> "based on"
"obtained an an accuracy" -> "obtained an accuracy"

====================================

I have read the author response and would keep the same rating. The paper pointed out an important issue, but it has also been noticed before. The insight on weight sharing is interesting, although more experiments are needed to testify the claim over state-of-the-art NAS search space. 
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper studies an important problem, evaluating the performance of existing neural architecture search algorithms against a random sampling algorithm fairly. 

Neural architecture search usually involves two phases: model search and model tuning. In the search phase, best architectures after limited training are selected. In model tuning, the selected architectures are trained fully. However, it has been noticed that best architectures after limited training may not translate to globally best architectures. Although previous research has tried comparing to random sampling, such as Liu et al. 2019b, but the random architectures were not trained fully. The authors train random architectures fully before selecting the best one, which turns out to perform as well or better than the sophisticated neural architecture search methods. The paper also identifies that parameter sharing turns out to be a major reason why the sophisticated NAS methods do not really work well. 

The insights are obviously important and valuable. The insight on parameter sharing is even a bit disheartening. Parameter sharing is the main reason why NAS can scale to very large domains. Without it, is NAS still practical or useful? On the other hand, it is a bit unsatisfactory that the paper does not provide or even suggest solutions to remedy the identified issues.

Another comment is it is a stretch to consider the evaluation done in the paper a new framework. It is simply a new baseline plus a new experiment design.

About Equation (1) in Appendix A.2, it seems to simplify to p=(r/r_max)^n. Is the formula correct?
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper studies the effectiveness of several Neural Architecture Search (NAS) methods comparing it with that of random policy search. The paper concludes that none of these methods for a CNN (trained using CIFAR-10) and RNN model (trained using PTB) are statistically significantly better than the random search. The authors suggest that this is due to the weight sharing used by the NAS algorithms to accelerate the network training. 

This paper is written well with a good discussion of the problem. The problem considered is important and authors have raised the effectiveness of NAS methods correctly. Before this paper, Li and Talwalkar, “Random Search and Reproducibility for Neural Architecture Search” have also compared some of the NAS methods with random search and reported similar concerns. 
In this sense, the paper is not novel although I agree this paper has added an additional insight that “weight sharing” is the culprit.

I have two concerns about the methodology used in this paper:

(1)	The search space has been greatly, just 32 possible architectures. It is well known that in a small search space, difference between the performance of random search and any other systematic search algorithm is quite small. Only when the space gets larger, the power of systematic search starts to show up. Although, I completely understand the authors’ limitation of not having a ground truth for a large search space (infeasible due to a huge computational requirement), but without this, the claim of this paper is weak.

(2)	Secondly, among the NAS methods considered, I missed the whole class of methods based on Bayesian optimization. There are many such work, but I am listing just two of them here: Jin et al. (2018), “AUTO-KERAS: EFFICIENT NEURAL ARCHITECTURE SEARCH WITH NETWORK MORPHISM” and Kandasamy et al. (2018), “Neural Architecture Search with Bayesian Optimisation and Optimal Transport”. It would be useful to have them in the list of NAS methods considered here. 

Post Rebuttal:
I have read the rebuttal. I appreciate the authors 'prompt comparison of their method with Bayesian NAS. However, I still think that using a reduced search space, it is not appropriate to compare NAS methods with random search. Moreover, all the claims are only empirical and more experimental evidence needs to be provided to reject the current NAS methods.

review_title: Official Blind Review #2

Meta-review: This is one of several recent parallel papers that pointed out issues with neural architecture search (NAS). It shows that several NAS algorithms do not perform better than random search and finds that their weight sharing mechanism leads to low correlations of the search performance and final evaluation performance. Code is available to ensure reproducibility of the work.

After the discussion period, all reviewers are mildly in favour of accepting the paper. 

My recommendation is therefore to accept the paper. The paper's results may in part appear to be old news by now, but they were not when the paper first appeared on arXiv (in parallel to Li & Talwalkar, so similarities to that work should not be held against this paper).
Decision: Accept (Poster)