Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper studies the causes of why DARTS often results in models that do not generalize to the test set. The paper finds that DARTS models do not generalize due to sharp minima, partially caused by the discretizing step in DARTS. The paper presents many experiments and studies on multiple search spaces, showing that this problem is general. To address this problem, the paper proposes several different ways to address this, e.g., an early stopping criteria and regularization methods.

Overall, the paper is well written, thorough experiments (various tasks and search spaces) show the benefit of the approach.  The final experiments using the full DARTS space also show an improvement over standard DARTS.

The final method is fairly simple, running the search with different regularization parameters and keeping the best model, which suggests it could be widely used for DARTS-based approaches. 

 Two (very) minor comments:
- There's a missing space before "Similarly" in section 2.1
- Extra ")" in last paragraph of section 2.2 (after the argmin eq).
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: This paper seeks to understand why Differential Architecture Search (DAS) might fail to find neural net architectures that perform well. The authors perform a series of experiments using different kinds of search spaces and datasets, and concluded that a major culprit is the discretization/pruning step at the end of DARTS. To avoid this, the authors propose early stopping based on measuring the eigenvalue of the Hessian of the validation loss. The results look promising (though as someone who is not familiar with the datasets, I don't have a sense of the significance of improvements.)

In general, this is a strong paper. I enjoyed reading it. It describes the problem clearly and performs a set of convincing experiments to support the claims. I especially like how different constrained search spaces are investigated, as this makes the results easier to interpret. I think the analysis in this paper will benefit researchers who work on similar problems. 


review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: ----- Updated after rebuttal period ---

The author's detailed response effectively addressed my concerns. I am moving my score to Accept. This paper proposes an interesting systematic study of differentiable approach in NAS.

------ Original Review ----

Summary 

This paper presents a systematic evaluation on top of differentiable architecture search (DARTS) algorithm and shows it usually searched an architecture with all skip-connection. It empirically reveals that the largest eigenvalue of the Hessian matrix (\lambda) of loss w.r.t. architecture parameters has a strong correlation with the generalization ability (via loss of test dataset), and shows this \lambda will first decrease but then drastically increase after a certain epoch number on 4 different search spaces. It then proposes an early-stop scheme (DARTS-ES) to stop the search before this phenomenon occurs. In addition, it proposes to use data-augmentation, path-dropping and tuning L2 regularization during the search, namely Robust-DARTS(R-DARTS), and yield constantly better results over original DARTS on 3 datasets. 


Overall, the observation that the largest eigenvalue of the Hessian matrix is novel and intriguing, and the experiments are extensive and meaningful. The idea to use more search spaces for comparison is fair and performance increase demonstrates the proposed R-DARTS and DARTS-ES  are effective. Although I still have some questions regarding the detail settings, I think this paper provides a novel angle to understand the search phase of DARTS, and proposed simple but effective regularization can be beneficial to the research community using DARTS. 


Main concerns:

- Problem of DARTS as a motivation 
The claims of local smoothness/sharpness and generalization are related to network generalization is quite intriguing, however, only using largest eigenvalue of Hessian matrix as an indicator of this local shape does not seem to be enough. A recent paper on loss-landscape visualization [1] provides means to examine this hypothesis directly, and could the author try to provide additional visualization to support their claim? Otherwise, the paper's claim does not generalize to the local shape of the loss function, and should stays with the largest eigenvalue. It is totally okay in my perspective, but just indicates some revision to the main text and analysis of Section 4.2.

- Questions about Figure 3 experiments
How is test error computed? Is it on a batch of test-split, or the entire one? Also, which architecture is used to compute this test error? Paper mentioned, in Section 4.1, the word "final architecture", but does this refer to the super-net (the one-shot model in paper's definition), or the stand-alone model obtained via binarized architecture alphas? If latter, is this generalization error obtained from training from scratch? Or simply using the super-net parameters during the search? Since the conclusion of this plot serves as the foundation of designing R-DARTS and DARTS-ES, if the experiments are only conducted over a small set of images or the binarized model with super-net parameters, it undercuts the credibility of the conclusion, largest architectural eigenvalue, and generalization ability.


- More independent runs of experiments.
In Figure 3, validation of original DARTS, and Table 1, DARTS vs DARTS-ES, paper runs the experiment for 3 times and take the average, but for the proposed R-DARTS, it is not. Is there a reason why not scaling the experiments? I suggest the author provide results over 5 runs, like the one in Table 4 for PTB and show if the R-DARTS truly surpasses DARTS constantly. This question also applies to Figure 1, when paper claims the original DARTS found poor cell type, could this be repetitive over multiple runs?

I guess for the experiments in Table 3, it is already done since paper mentioned the reported results are the best model of searching 4 times. 

- R-DARTS failed to out-perform DARTS in the original space on CIFAR-10
This is confusing, will this suggest, if tuning well, DARTS will surpass R-DARTS(L2) in other cases as well? Since this is the only setting that DARTS is built upon. 


Minor comments and questions

- Using test data during search
After showing the strong correlation between the largest eigenvalue of Hessian and the network generalization error, the early-stop is natural, however, does this mean the model selection is using the test data? Or the actual test-data is never seen during the search phase of Section 4.3.

- L2 stabilizes max eigenvalue
Paper uses L2 coefficient up to 0.0243, showing constant improvement of test error while validation error drops in CIFAR-10 of Figure 11. Could the author try larger coefficients to determine when this trend will stop? 

- Question about section 4.2
Performance drop due to the binarized operation (pruning step) in DARTS analysis is very interesting, I am curious how many architectures does the paper evaluate in Figure 5, when the dominant eigenvalue is smaller than 0.25? Since the conclusion is "low curvature never led to large performance drops" if the number of points is too few, it is not that convincing, especially from the plot, we see at eigenvalue = 0.5, there exists 2 architecture with >20% drop. In addition, what does each point in Figure 5 refer to? The best model (and the binarized one according to the argmax of \alpha) of one independent DARTS run or some binarized models sampled from a distribution on top of the same DARTS run (meaning only one super-net)? Is this experiment follows the setting in Figure 4? 

- Figure 6, C10 S2, DARTS-ES is worse than DARTS when Drop probability = 0.6, whereas all other cases, DARTS-ES outperforms DARTS, why does this happen? Could the author comment on it?

- ScheduledDropPath in section 5.1
Does Drop-path belongs to data-augmentation techniques? It is more like a regularization in my perspective and should be grouped with 5.2.

- S1 S2... in Table 3
Does this refer to the search space? Or different random seed (mentioned )

- one-shot v.s. weight sharing model
One-shot in NAS domain is firstly introduced by Bender et al., while Pham et al. use parameter sharing. The reason to use one-shot is that all the sub-paths will have a fair chance to be trained. 

- Typos
1. In section 2.1, line 4 "better.Similarly" should have space.

--- Reference ---
[1] Li et al., Visualizing the Loss Landscape of Neural Nets, arxiv'17.
review_title: Official Blind Review #3

Meta-review: This paper studies the properties of Differentiable Architecture Search, and in particular when it fails, and then proposes modifications that improve its performance for several tasks. The reviews were all very supportive with three Accept opinions, and authors have addressed their comments and suggestions. Given the unanimous reviews, this appears to be a clear Accept. 
Decision: Accept (Talk)