Rethinking Architecture Selection in Differentiable NAS

1 INTRODUCTION . Neural Architecture Search ( NAS ) has been drawing increasing attention in both academia and industry for its potential to automatize the process of discovering high-performance architectures , which have long been handcrafted . Early works on NAS deploy Evolutionary Algorithm ( Stanley & Miikkulainen , 2002 ; Real et al. , 2017 ; Liu et al. , 2017 ) and Reinforcement Learning ( Zoph & Le , 2017 ; Pham et al. , 2018 ; Zhong et al. , 2018 ) to guide the architecture discovery process . Recently , several one-shot methods have been proposed that significantly improve the search efficiency ( Brock et al. , 2018 ; Guo et al. , 2019 ; Bender et al. , 2018 ) . As a particularly popular instance of one-shot methods , DARTS ( Liu et al. , 2019 ) enables the search process to be performed with a gradient-based optimizer in an end-to-end manner . It applies continuous relaxation that transforms the categorical choice of architectures into continuous architecture parameters α . The resulting supernet can be optimized via gradient-based methods , and the operations associated with the largest architecture parameters are selected to form the final architecture . Despite its simplicity , several works cast doubt on the effectiveness of DARTS . For example , a simple randomized search ( Li & Talwalkar , 2019 ) outperforms the original DARTS ; Zela et al . ( 2020 ) observes that DARTS degenerates to networks filled with parametric-free operations such as the skip connection or even random noise , leading to the poor performance of the selected architecture . While the majority of previous research attributes the failure of DARTS to its supernet optimization ( Zela et al. , 2020 ; Chen & Hsieh , 2020 ; Chen et al. , 2021 ) , little has been discussed about the validity of another important assumption : the value of α reflects the strength of the underlying operations . In this paper , we conduct an in-depth analysis of this problem . Surprisingly , we find that in many cases , α does not really indicate the operation importance in a supernet . Firstly , the operation associated with larger α does not necessarily result in higher validation accuracy after discretization . Secondly , as an important example , we show mathematically that the domination of skip connection observed in DARTS ( i.e . αskip becomes larger than other operations . ) is in fact a reasonable outcome of the supernet ’ s optimization but becomes problematic when we rely on α to select the best operation . If α is not a good indicator of operation strength , how should we select the final architecture from a pretrained supernet ? Our analysis indicates that the strength of each operation should be evaluated based on its contribution to the supernet performance instead . To this end , we propose an alternative perturbation-based architecture selection method . Given a pretrained supernet , the best operation on an edge is selected and discretized based on how much it perturbs the supernet accuracy ; The final architecture is derived edge by edge , with fine-tuning in between so that the supernet remains converged for every operation decision . We re-evaluate several differentiable NAS methods ( DARTS ( Liu et al. , 2019 ) , SDARTS ( Chen & Hsieh , 2020 ) , SGAS ( Li et al. , 2020 ) ) and show that the proposed selection method is able to consistently extract significantly improved architectures from the supernets than magnitude-based counterparts . Furthermore , we find that the robustness issues of DARTS can be greatly alleviated by replacing the magnitude-based selection with the proposed perturbation-based selection method . 2 BACKGROUND AND RELATED WORK . Preliminaries of Differentiable Architecture Search ( DARTS ) We start by reviewing the formulation of DARTS . DARTS ’ search space consists of repetitions of cell-based microstructures . Every cell can be viewed as a DAG with N nodes and E edges , where each node represents a latent feature map xi , and each edge is associated with an operation o ( e.g . skip connect , sep conv 3x3 ) from the search space O . Continuous relaxation is then applied to this search space . Concretely , every operation on an edge is activated during the search phase , with their outputs mixed by the architecture parameter α to form the final mixed output of that edge m̄ ( xi ) = ∑ o∈O expαo∑ o′ expαo′ o ( xi ) . This particular formulation allows the architecture search to be performed in a differentiable manner : DARTS jointly optimizes α and model weight w with the following bilevel objective via alternative gradient updates : min α Lval ( w∗ , α ) s.t . w∗ = arg min w Ltrain ( w , α ) . ( 1 ) We refer to the continuous relaxed network used in the search phase as the supernet of DARTS . At the end of the search phase , the operation associated with the largest αo on each edge will be selected from the supernet to form the final architecture . Failure mode analysis of DARTS Several works cast doubt on the robustness of DARTS . Zela et al . ( 2020 ) tests DARTS on four different search spaces and observes significantly degenerated performance . They empirically find that the selected architectures perform poorly when DARTS ’ supernet falls into high curvature areas of validation loss ( captured by large dominant eigenvalues of the Hessian∇2α , αLval ( w , α ) ) . While Zela et al . ( 2020 ) relates this problem to the failure of supernet training in DARTS , we examine it from the architecture selection aspects of DARTS , and show that much of DARTS ’ robustness issue can be alleviated by a better architecture selection method . Progressive search space shrinking There is a line of research on NAS that focuses on reducing the search cost and aligning the model sizes of the search and evaluation phases via progressive search space shrinking ( Liu et al. , 2018 ; Li et al. , 2019 ; Chen et al. , 2021 ; Li et al. , 2020 ) . The general scheme of these methods is to prune out weak operations and edges sequentially during the search phase , based on the magnitude of α following DARTS . Our method is orthogonal to them in this respect , since we select operations based on how much it contributes to the supernet ’ s performance rather than the α value . Although we also discretize edges greedily and fine-tune the network in between , the purpose is to let the supernet recover from the loss of accuracy after discretization to accurately evaluate operation strength on the next edge , rather than to reduce the search cost . 3 THE PITFALL OF MAGNITUDE-BASED ARCHITECTURE SELECTION IN . DARTS In this section , we put forward the opinion that the architecture parameter α does not necessarily represent the strength of the underlying operation in general , backed by both empirical and theoretical evidence . As an important example , we mathematically justify that the skip connection domination phenomenon observed in DARTS is reasonable by itself , and becomes problematic when combined with the magnitude-based architecture selection . 3.1 α MAY NOT REPRESENT THE OPERATION STRENGTH Following DARTS , existing differentiable NAS methods use the value of architecture parameters α to select the final architecture from the supernet , with the implicit assumption that α represents the strength of the underlying operations . In this section , we study the validity of this assumption in detail . Consider one edge on a pretrained supernet ; the strength of an operation on the edge can be naturally defined as the supernet accuracy after we discretize to this operation and fine-tune the remaining network until it converges again ; we refer to this as ” discretization accuracy at convergence ” for short . The operation that achieves the best discretization accuracy at convergence can be considered as the best operation for the given edge . Figure 1 shows the comparison of α ( blue ) and operation strength ( orange ) of randomly select edges on DARTS supernet . As we can see , the magnitude of α for each operation does not necessarily agree with their relative strength measured by discretization accuracy at convergence . Moreover , operations assigned with small αs are sometimes strong ones that lead to high discretization accuracy at convergence . To further verify the mismatch , we investigate the operation strength on search space S2 , where DARTS fails dramatically due to excessive skip connections ( Zela et al. , 2020 ) . S2 is a variant of DARTS search space that only contains two operations per edge ( skip connect , sep conv 3x3 ) . Figure 2 shows the selected operations based on α ( left ) and operation strength ( right ) on all edges on S2 . From Figure 2a , we can see that αskip connect > αsep conv 3x3 on 12 of 14 edges . Consequently , the derived child architecture will lack representation ability and perform poorly due to too many skip connections . However , as shown in Figure 2b , the supernet benefits more from discretizing to sep conv 3x3 than skip connect on half of the edges . 3.2 A CASE STUDY : SKIP CONNECTION . Several works point out that DARTS tends to assign large α to skip connections , resulting in shallow architectures with poor generability ( Zela et al. , 2020 ; Liang et al. , 2019 ; Bi et al. , 2019 ) . This ” skip connection domination ” issue is generally attributed to the failure of DARTS ’ supernet optimization . In contrast , we draw inspiration from research on ResNet ( He et al. , 2016 ) and show that this phenomenon by itself is a reasonable outcome while DARTS refines its estimation of the optimal feature map , rendering αskip ineffective in the architecture selection . In vanilla networks ( e.g. , VGG ) , each layer computes a new level of feature map from the output feature map of the predecessor layer ; thus , reordering layers at test time would dramatically hurt the performance ( Veit et al. , 2016 ) . Unlike vanilla networks , Greff et al . ( 2017 ) and Veit et al . ( 2016 ) discover that successive layers in ResNet with compatible channel sizes are in fact estimating the same optimal feature map so that the outputs of these layers stay relatively close to each other at convergence ; As a result , ResNet ’ s test accuracy remains robust under layer reordering . Greff et al . ( 2017 ) refers to this unique way of feature map estimation in ResNet as the ” unrolled estimation. ” DARTS ’ supernet resembles ResNet , rather than vanilla networks like VGG , in both appearance and behavior . Appearance-wise , within a cell of DARTS ’ supernet , edges with skip connection are in direct correspondence with the successive residual layers in ResNet . Behavior-wise , DARTS ’ supernet also exhibits a high degree of robustness under edge shuffling . As shown in Table 1 , randomly reordering edges on a pretrained DARTS ’ supernet at test time also has little effect on its performance . This evidence indicates that DARTS performs unrolled estimation like ResNet as well , i.e. , edges within a cell share the same optimal feature map that they try to estimate . In the following proposition , we apply this finding and provide the optimal solution of α in the sense of minimizing the variance of feature map estimation . Proposition 1 . 1 Without loss of generality , consider one cell from a simplified search space consists of two operations : ( skip , conv ) . Let m∗ denotes the optimal feature map , which is shared across all edges according to the unrolled estimation view ( Greff et al. , 2017 ) . Let oe ( xe ) be the output of convolution operation , and let xe be the skip connection ( i.e. , the input feature map of edge e ) . Assume m∗ , oe ( xe ) and xe are normalized to the same scale . The current estimation of m∗ can then be written as : me ( xe ) = exp ( αconv ) exp ( αconv ) + exp ( αskip ) oe ( xe ) + exp ( αskip ) exp ( αconv ) + exp ( αskip ) xe , ( 2 ) where αconv and αskip are the architecture parameters defined in DARTS . The optimal α∗conv and α∗skip minimizing var ( me ( xe ) − m∗ ) , the variance of the difference between the optimal feature map m∗ and its current estimation me ( xe ) , are given by : α∗conv ∝ var ( xe −m∗ ) ( 3 ) α∗skip ∝ var ( oe ( xe ) −m∗ ) . ( 4 ) We refer the reader to Appendix A.4 for detailed proof . From eq . ( 3 ) and eq . ( 4 ) , we can see that the relative magnitudes of αskip and αconv come down to which one of xe or oe ( xe ) is closer to m∗ in variance : • xe ( input of edge e ) comes from the mixed output of the previous edge . Since the goal of every edge is to estimate m∗ ( unrolled estimation ) , xe is also directly estimating m∗ . 1Proposition 1 unfolds the optimal α in principle and does not constraint the particular optimization method ( i.e. , bilevel , single-level , or blockwise update ) to achieve it . Moreover , this proposition can be readily extended to various other search spaces since we can group all non-skip operations into a single oe ( · ) . • oe ( xe ) is the output of a single convolution operation instead of the complete mixed output of edge e , so it will deviate from m∗ even at convergence . Therefore , in a well-optimized supernet , xe will naturally be closer tom∗ than oe ( xe ) , causing αskip to be greater than αconv . Our analysis above indicates that the better the supernet , the larger the ( αskip − αconv ) gap ( softmaxed ) will become since xe gets closer and closer tom∗ as the supernet is optimized . This result is evidenced in Figure 3 , where mean ( αskip − αconv ) continues to grow as the supernet gets better . In this case , although αskip > αconv is reasonable by itself , it becomes an inductive bias to NAS if we were to select the final architecture based on α .

Summary: In one-shot differentiable NAS, a supergraph is usually trained (via bilevel optimization as in DARTS, or other approximations to bilevel such as gumbel softmax, etc). After supergraph training, a final architecture is obtained by taking the operator at each edge which has the highest architecture weight magnitude. This step is usually termed as the 'finalization' step. (In DARTS the finalization step actually orders incoming edges by the max of the architecture weight magnitudes at each edge and selects the top two edges and the corresponding maximum architecture weight in them as the final operators.). This paper examines this quite ad hoc step very closely. It finds that the magnitude of architecture weights (alphas commonly in this niche literature) are misleading. It shows by careful ablation experiments that alpha magnitudes are very much not useful in selecting good operators.