Understanding and Robustifying Differentiable Architecture Search

1 INTRODUCTION . Neural Architecture Search ( NAS ) , the process of automatically designing neural network architectures , has recently attracted attention by achieving state-of-the-art performance on a variety of tasks ( Zoph & Le , 2017 ; Real et al. , 2019 ) . Differentiable architecture search ( DARTS ) ( Liu et al. , 2019 ) significantly improved the efficiency of NAS over prior work , reducing its costs to the same order of magnitude as training a single neural network . This expanded the scope of NAS substantially , allowing it to also be applied on more expensive problems , such as semantic segmentation ( Chenxi et al. , 2019 ) or disparity estimation ( Saikia et al. , 2019 ) . However , several researchers have also reported DARTS to not work well , in some cases even no better than random search ( Li & Talwalkar , 2019 ; Sciuto et al. , 2019 ) . Why is this ? How can these seemingly contradicting results be explained ? The overall goal of this paper is to understand and overcome such failure modes of DARTS . To this end , we make the following contributions : 1 . We identify 12 NAS benchmarks based on four search spaces in which standard DARTS yields degenerate architectures with poor test performance across several datasets ( Section 3 ) . 2 . By computing the eigenspectrum of the Hessian of the validation loss with respect to the architectural parameters , we show that there is a strong correlation between its dominant eigenvalue and the architecture ’ s generalization error . Based on this finding , we propose a simple variation of DARTS with early stopping that performs substantially more robustly ( Section 4 ) . 3 . We show that , related to previous work on sharp/flat local minima , regularizing the inner objective of DARTS more strongly allows it to find solutions with smaller Hessian spectrum and better generalization properties . Based on these insights , we propose two practical robustifications of DARTS that overcome its failure modes in all our 12 NAS benchmarks ( Section 5 ) . Our findings are robust across a wide range of NAS benchmarks based on image recognition and also hold for the very different domains of language modelling ( PTB ) and disparity estimation . They consolidate the findings of the various results in the literature and lead to a substantially more robust version of DARTS . We provide our implementation and scripts to facilitate reproducibility1 . 2 BACKGROUND AND RELATED WORK . 2.1 RELATION BETWEEN FLAT/SHARP MINIMA AND GENERALIZATION PERFORMANCE . Already Hochreiter & Schmidhuber ( 1997 ) observed that flat minima of the training loss yield better generalization performance than sharp minima . Recent work ( Keskar et al. , 2016 ; Yao et al. , 2018 ) focuses more on the settings of large/small batch size training , where observations show that small batch training tends to get attracted to flatter minima and generalizes better . Similarly , Nguyen et al . ( 2018 ) observed that this phenomenon manifests also in the hyperparameter space . They showed that whenever the hyperparameters overfit the validation data , the minima lie in a sharper region of the space . This motivated us to conduct a similar analysis in the context of differentiable architecture search later in Section 4.1 , where we see the same effect in the space of neural network architectures . 2.2 BI-LEVEL OPTIMIZATION . We start by a short introduction of the bi-level optimization problem ( Colson et al. , 2007 ) . These are problems which contain two optimization tasks , nested within each other . Definition 2.1 . Given the outer objective function F : RP × RN → R and the inner objective function f : RP × RN → R , the bi-level optimization problem is given by min y∈RP F ( y , θ∗ ( y ) ) ( 1 ) s.t . θ∗ ( y ) ∈ argmin θ∈RN f ( y , θ ) , ( 2 ) where y ∈ RP and θ ∈ RN are the outer and inner variables , respectively . One may also see the bi-level problem as a constrained optimization problem , with the inner problem as a constraint . In general , even in the case when the inner objective ( 2 ) is strongly convex and has an unique minimizer θ∗ ( y ) = argminθ∈RN f ( y , θ ) , it is not possible to directly optimize the outer objective ( 1 ) . A possible method around this issue is to use the implicit function theorem to retrieve the derivative of the solution map ( or response map ) θ∗ ( y ) ∈ F ⊆ RN w.r.t . y ( Bengio , 2000 ; Pedregosa , 2016 ; Beirami et al. , 2017 ) . Another strategy is to approximate the inner problem with a dynamical system ( Domke , 2012 ; Maclaurin et al. , 2015 ; Franceschi et al. , 2017 ; 2018 ) , where the optimization dynamics could , e.g. , describe gradient descent . In the case that the minimizer of the inner problem is unique , under some conditions the set of minimizers of this approximate problem will indeed converge to the minimizers of the bilevel problem ( 1 ) ( see Franceschi et al . ( 2018 ) ) . 2.3 NEURAL ARCHITECTURE SEARCH . Neural Architecture Search ( NAS ) denotes the process of automatically designing neural network architectures in order to overcome the cumbersome trial-and-error process when designing architectures manually . We briefly review NAS here and refer to the recent survey by Elsken et al . ( 2019b ) for a more thorough overview . Prior work mostly employs either reinforcement learning techniques ( Baker et al. , 2017a ; Zoph & Le , 2017 ; Zhong et al. , 2018 ; Zoph et al. , 2018 ) or evolutionary algorithms ( Stanley & Miikkulainen , 2002 ; Liu et al. , 2018b ; Miikkulainen et al. , 2017 ; Real et al. , 2017 ; 2019 ) to optimize the discrete architecture space . As these methods are often very expensive , various works focus on reducing the search costs by , e.g. , employing network morphisms ( Cai et al. , 2018a ; b ; Elsken et al. , 2017 ; 2019a ) , weight sharing within search models ( Saxena & Verbeek , 2016 ; Bender et al. , 2018 ; Pham et al. , 2018 ) or multi-fidelity optimization ( Baker et al. , 2017b ; Falkner et al. , 2018 ; Li et al. , 2017 ; Zela et al. , 2018 ) , but their applicability still often remains restricted to rather simple tasks and small datasets . 1 https : //github.com/automl/RobustDARTS 2.4 DIFFERENTIABLE ARCHITECTURE SEARCH ( DARTS ) . A recent line of work focuses on relaxing the discrete neural architecture search problem to a continuous one that can be solved by gradient descent ( Liu et al. , 2019 ; Xie et al. , 2019 ; Casale et al. , 2019 ; Cai et al. , 2019 ) . In DARTS ( Liu et al. , 2019 ) , this is achieved by simply using a weighted sum of possible candidate operations for each layer , whereas the real-valued weights then effectively parametrize the network ’ s architecture . We will now review DARTS in more detail , as our work builds directly upon it . Continuous relaxation of the search space . In agreement with prior work ( Zoph et al. , 2018 ; Real et al. , 2019 ) , DARTS optimizes only substructures called cells that are stacked to define the full network architecture . Each cell contains N nodes organized in a directed acyclic graph . The graph contains two inputs nodes ( given by the outputs of the previous two cells ) , a set of intermediate nodes , and one output node ( given by concatenating all intermediate nodes ) . Each intermediate node x ( j ) represents a feature map . See Figure 1 for an illustration of such a cell . Instead of applying a single operation to a specific node during architecture search , Liu et al . ( 2019 ) relax the decision which operation to choose by computing the intermediate node as a mixture of candidate operations , applied to predecessor nodes x ( i ) , i < j , x ( j ) = ∑ i < j ∑ o∈O exp ( αi , jo ) ∑ o′∈O exp ( α i , j o′ ) o ( x ( i ) ) , where O denotes the set of all candidate operations ( e.g. , 3 × 3 convolution , skip connection , 3 × 3 max pooling , etc . ) and α = ( αi , jo ) i , j , o serves as a real-valued parameterization of the architecture . Gradient-based optimization of the search space . DARTS then optimizes both the weights of the search network ( often called the weight-sharing or one-shot model , since the weights of all individual subgraphs/architectures are shared ) and architectural parameters by alternating gradient descent . The network weights and the architecture parameters are optimized on the training and validation set , respectively . This can be interpreted as solving the bi-level optimization problem ( 1 ) , ( 2 ) , where F and f are the validation and training loss , Lvalid and Ltrain , respectively , while y and θ denote the architectural parameters α and network weights w , respectively . Note that DARTS only approximates the lower-level solution by a single gradient step ( see Appendix A for more details ) . At the end of the search phase , a discrete cell is obtained by choosing the k most important incoming operations for each intermediate node while all others are pruned . Importance is measured by the operation weighting factor exp ( α i , j o ) ∑ o′∈O exp ( α i , j o′ ) . 3 WHEN DARTS FAILS . We now describe various search spaces and demonstrate that standard DARTS fails on them . We start with four search spaces similar to the original CIFAR-10 search space but simpler , and evaluate across three different datasets ( CIFAR-10 , CIFAR-100 and SVHN ) . They are quite standard in that they use the same macro architecture as the original DARTS paper ( Liu et al. , 2018a ) , consisting of normal and reduction cells ; however , they only allow a subset of operators for the cell search space : S1 : This search space uses a different set of only two operators per edge , which we identified using an offline process that iteratively dropped the operations from the original DARTS search space with the least importance . This pre-optimized space has the advantage of being quite small while still including many strong architectures . We refer to Appendix B for details on its construction and an illustration ( Figure 9 ) . S2 : In this space , the set of candidate operations per edge is { 3 × 3 SepConv , SkipConnect } . We choose these operations since they are the most frequent ones in the discovered cells reported by Liu et al . ( 2019 ) . S3 : In this space , the set of candidate operations per edge is { 3 × 3 SepConv , SkipConnect , Zero } , where the Zero operation simply replaces every value in the input feature map by zeros . S4 : In this space , the set of candidate operations per edge is { 3 × 3 SepConv , Noise } , where the Noise operation simply replaces every value from the input feature map by noise ∼ N ( 0 , 1 ) . This is the only space out of S1-S4 that is not a strict subspace of the original DARTS space ; we intentionally added the Noise operation , which actively harms performance and should therefore not be selected by DARTS . We ran DARTS on each of these spaces , using exactly the same setup as Liu et al . ( 2019 ) . Figure 1 shows the poor cells DARTS selected on these search spaces for CIFAR-10 ( see Appendix G for analogous results on the other datasets ) . Already visually , one might suspect that the found cells are suboptimal : the parameter-less skip connections dominate in almost all the edges for spaces S1-S3 , and for S4 even the harmful Noise operation was selected for five out of eight operations . Table 1 ( first column ) confirms the very poor performance standard DARTS yields on all of these search spaces and on different datasets . We note that Liu et al . ( 2019 ) and Xie et al . ( 2019 ) argue that the Zero operation can help to search for the architecture topology and choice of operators jointly , but in our experiments it did not help to reduce the importance weight of the skip connection ( compare Figure 1b vs . Figure 1c ) . We emphasize that search spaces S1-S3 are very natural , and , as strict subspaces of the original space , should merely be easier to search than that . They are in no way special or constructed in an adversarial manner . Only S4 was constructed specifically to show-case the failure mode of DARTS selecting the obviously suboptimal Noise operator . S5 : Very small search space with known global optimum . Knowing the global minimum has the advantage that one can benchmark the performance of algorithms by measuring the regret of chosen points with respect to the known global minimum . Therefore , we created another search space with only one intermediate node for both normal and reduction cells , and 3 operation choices in each edge , namely 3× 3 SepConv , SkipConnection , and 3× 3 MaxPooling . The total number of possible architectures in this space is 81 , all of which we evaluated a-priori . We dub this space S5 . We ran DARTS on this search space three times for each dataset and compared its result to the baseline of Random Search with weight sharing ( RS-ws ) by Li & Talwalkar ( 2019 ) . Figure 2 shows the test regret of the architectures selected by DARTS ( blue ) and RS-ws ( green ) throughout the search . DARTS manages to find an architecture close to the global minimum , but around epoch 40 the test performance deteriorated . Note that the search model validation error ( dashed red line ) did not deteriorate but rather converged , indicating that the architectural parameters are overfitting to the validation set . In contrast , RS-ws stays relatively constant throughout the search ; when evaluating only the final architecture found , RS-ws indeed outperformed DARTS . S6 : encoder-decoder architecture for disparity estimation . To study whether our findings generalize beyond image recognition , we also analyzed a search space for a very different problem : finding encoder-decoder architectures for the dense regression task of disparity estimation ; please refer to Appendix E for details . We base this search space on AutoDispNet ( Saikia et al. , 2019 ) , which used DARTS for a space containing normal , downsampling and upsampling cells . We again constructed a reduced space . Similarly to the image classification search spaces , we found the normal cell to be mainly composed of parameter-less operations ( see Figure 25 in Appendix G ) . As expected , this causes a large generalization error ( see first row in Table 2 of our later experiments ) . 4 THE ROLE OF DOMINANT EIGENVALUES OF ∇2αLvalid We now analyze why DARTS fails in all these cases . Motivated by Section 2.1 , we will have a closer look at the largest eigenvalue λαmax of the Hessian matrix of validation loss ∇2αLvalid w.r.t . the architectural parameters α .

Summary: This paper studies the causes of why DARTS often results in models that do not generalize to the test set. The paper finds that DARTS models do not generalize due to sharp minima, partially caused by the discretizing step in DARTS. The paper presents many experiments and studies on multiple search spaces, showing that this problem is general. To address this problem, the paper proposes several different ways to address this, e.g., an early stopping criteria and regularization methods.