DARTS-: Robustly Stepping out of Performance Collapse Without Indicators

1 INTRODUCTION . Recent studies ( Zela et al. , 2020 ; Liang et al. , 2019 ; Chu et al. , 2020b ) have shown that one critical issue for differentiable architecture search ( Liu et al. , 2019b ) regarding the performance collapse due to superfluous skip connections . Accordingly , some empirical indicators for detecting the occurrence of collapse have been produced . R-DARTS ( Zela et al. , 2020 ) shows that the loss landscape has more curvatures ( characterized by higher Hessian eigenvalues w.r.t . architectural weights ) when the derived architecture generalizes poorly . By regularizing for a lower Hessian eigenvalue , Zela et al . ( 2020 ) ; Chen & Hsieh ( 2020 ) attempt to stabilize the search process . Meanwhile , by directly constraining the number of skip connections to a fixed number ( typically 2 ) , the collapse issue becomes less pronounced ( Chen et al. , 2019b ; Liang et al. , 2019 ) . These indicator-based approaches have several main drawbacks . Firstly , robustness relies heavily on the quality of the indicator . An imprecise indicator either inevitably accepts poor models or mistakenly reject good ones . Secondly , indicators impose strong priors by directly manipulating the inferred model , which is somewhat suspicious , akin to touching the test set . Thirdly , extra computing cost ( Zela et al. , 2020 ) or careful tuning of hyper-parameters ( Chen et al. , 2019b ; Liang et al. , 2019 ) are required . Therefore , it ’ s natural to ask the following questions : • Can we resolve the collapse without handcrafted indicators and restrictions to interfere with the searching and/or discretization procedure ? • Is it possible to achieve robustness in DARTS without tuning extra hyper-parameters ? ∗Work done as an intern at Meituan Inc. †Correspondent Author . To answer the above questions , we propose an effective and efficient approach to stabilize DARTS . Our contributions can be summarized as follows : New Paradigm to Stabilize DARTS . While empirically observing that current indicators ( Zela et al. , 2020 ; Chen & Hsieh , 2020 ) can avoid performance collapse at a cost of reduced exploration coverage in the search space , we propose a novel indicator-free approach to stabilize DARTS , referred to as DARTS-1 , which involves an auxiliary skip connection ( see Figure 1 ) to remove the unfair advantage ( Chu et al. , 2020b ) in the searching phase . Strong Robustness and Stabilization . We conduct thorough experiments across seven search spaces and three datasets to demonstrate the effectiveness of our method . Specifically , our approach robustly obtains state-of-the-art results on 4 search space with 3× fewer search cost than R-DARTS ( Zela et al. , 2020 ) , which requires four independent runs to report the final performance . Seamless Plug-in Combination with DARTS Variants . We conduct experiments to demonstrate that our approach can work together seamlessly with other orthogonal DARTS variants by removing their handcrafted indicators without extra overhead . In particular , our approach is able to improve 0.8 % accuracy for P-DARTS , and 0.25 % accuracy for PC-DARTS on CIFAR-10 dataset . 2 RELATED WORK . Neural architecture search and DARTS variants . Over the years , researchers have sought to automatically discover neural architectures for various deep learning tasks to relieve human from the tedious effort , ranging from image classification ( Zoph et al. , 2018 ) , objection detection ( Ghiasi et al. , 2019 ) , image segmentation ( Liu et al. , 2019a ) to machine translation ( So et al. , 2019 ) etc . Among many proposed approaches , Differentiable Architecture Search ( Liu et al. , 2019b ) features weight-sharing and resolves the searching problem via gradient descent , which is very efficient and easy to generalize . A short description of DARTS can be found in A.1 . Since then , many subsequent works have been dedicated to accelerating the process ( Dong & Yang , 2019b ) , reducing memory cost ( Xu et al. , 2020 ) , or fostering its ability such as hardware-awareness ( Cai et al. , 2019 ; Wu et al. , 2019 ) , finer granularity ( Mei et al. , 2020 ) and so on . However , regardless of these endeavors , a fundamental issue of DARTS over its searching performance collapse remains not properly solved , which extremely prohibits its application . Robustifying DARTS . As DARTS ( Liu et al. , 2019b ) is known to be unstable as a result of performance collapse ( Chu et al. , 2020b ) , some recent works have devoted to resolving it by either designing indicators like Hessian eigenvalues for the collapse ( Zela et al. , 2020 ) or adding perturbations to regularize such an indicator ( Chen & Hsieh , 2020 ) . Both methods rely heavily on the indicator ’ s accurateness , i.e. , to what extent does the indicator correlate with the performance collapse ? Other methods like Progressive DARTS ( Chen et al. , 2019b ) and DARTS+ ( Liang et al. , 2019 ) employ a strong human prior , i.e. , limiting the number of skip-connections to be a fixed value . Fair DARTS ( Chu et al. , 2020b ) argues that the collapse results from the unfair advantage in an exclusive competitive environment , from which skip connections overly benefit to cause an abundant aggregation . To suppress such an advantage from overshooting , they convert the competition into collaboration where each operation is independent of others . It is however an indirect approach . SGAS ( Li et al. , 2020 ) , instead , circumvents the problem with a greedy strategy where the unfair advantage can be prevented from taking effect . Nevertheless , potentially good operations might be pruned out too early because of greedy underestimation . 1We name it so as we undertake an inward way , as opposed to those outward ones who design new indicators , add extra cost and introduce new hyper-parameters . 3 DARTS- . 3.1 MOTIVATION . We start from the detailed analysis of the role of skip connections . Skip connections were proposed to construct a residual block in ResNet ( He et al. , 2016 ) , which significantly improves training stability . It is even possible to deepen the network up to hundreds of layers without accuracy degradation by simply stacking them up . In contrast , stacking the plain blocks of VGG has degenerated performance when the network gets deeper . Besides , Ren et al . ( 2015 ) ; Wei et al . ( 2017 ) ; Tai et al . ( 2017 ) ; Li et al . ( 2018b ) also empirically demonstrate that deep residual network can achieve better performance on various tasks . From the gradient flow ’ s perspective , skip connection is able to alleviate the gradient vanishing problem . Given a stack of n residual blocks , the output of the ( i + 1 ) th residual block Xi+1 can be computed as Xi+1 = fi+1 ( Xi , Wi+1 ) + Xi , where fi+1 denotes the operations of the ( i + 1 ) th residual block with weights Wi+1 . Suppose the loss function of the model is L , and the gradient of Xi can be obtained as follows ( 1 denotes a tensor whose items are all ones ) : ∂L ∂Xi = ∂L ∂Xi+1 · ( ∂fi+1 ∂Xi + 1 ) = ∂L ∂Xi+j · j∏ k=1 ( ∂fi+k ∂Xi+k−1 + 1 ) ( 1 ) We observe that the gradient of shallow layers always includes the gradient of deep layers , which mitigates the gradient vanishing of Wi . Formally we have , ∂L ∂Wi = ∂L ∂Xi+j · j∏ k=1 ( ∂fi+k ∂Xi+k−1 + 1 ) · ∂fi ∂Wi ( 2 ) To analyze how skip connect affects the performance of residual networks , we introduce a trainable coefficient β on all skip connections in ResNet . Therefore , the gradient of Xi is converted to : ∂L ∂Xi = ∂L ∂Xi+1 · ( ∂fi+1 ∂Xi + β ) ( 3 ) Once β < 1 , gradients of deep layers gradually vanish during the back-propagation ( BP ) towards shallow layers . Here β controls the memory of gradients in BP to stabilize the training procedure . We conduct a confirmatory experiment on ResNet50 and show the result in Fig 2 . By initializing β with { 0 , 0.5 , 1.0 } , we can visualize the tendency of β along with training epochs . We observe that β converges towards 1 after 40 epochs no matter the initialization , which demonstrates that the residual structure learns to push β to a rather large value to alleviate gradient vanishing . Similarly , DARTS ( Liu et al. , 2019b ) utilizes a trainable parameter βskip to denote the importance of skip connection . However , In the search stage , βskip can generally increase and dominate the architecture parameters , and finally leads to performance collapse . We analyze that a large βskip in DARTS could result from two aspects : On the one hand , as the supernet automatically learns to alleviate gradient vanishing , it pushes βskip to a proper large value ; On the other hand , the skip connection is indeed an important connection for the target network , which should be selected in the discretization stage . As a consequence , the skip connection in DARTS plays two-fold roles : as an auxiliary connection to stabilize the supernet training , and as a candidate operation to build the final network . Inspired by the above observation and analysis , we propose to stabilize the search process by distinguishing the two roles of skip connection and handling the issue of gradient flow . 3.2 STEPPING OUT OF THE PERFORMANCE COLLAPSE . To distinguish the two roles , we introduce an auxiliary skip connection between every two nodes in a cell , see Fig . 1 ( b ) . On the one hand , the fixed auxiliary skip connection carries the function of stabilizing the supernet training , even when βskip is rather small . On the other hand , it also breaks the unfair advantage ( Chu et al. , 2020b ) as the advantageous contribution from the residual block is factored out . Consequently , the learned architectural parameter βskip can be freed from the role of controlling the memory of gradients , and is more precisely aimed to represent the relative importance of skip connection as a candidate operation . In contrast to Eq . 7 , the output feature map of edge e ( i , j ) can now be obtained by Eq . 4 , where βi , jo = exp ( α ( i , j ) o ) ∑ o′∈O exp ( α ( i , j ) o′ ) denotes the normalized importance , and β is a coefficient independent from the architecture parameters . Moreover , to eliminate the impact of auxiliary connection on the discretization procedure , we propose to decrease β to 0 in the search phase , and our method can be degenerated to standard DARTS at the end of the search . Note that our method is insensitive to the type of decay strategy , so we choose linear decay by default for simplicity . ō ( i , j ) ( x ) = βx+ ∑ o∈O β ( i , j ) o o ( x ) = ( β + β ( i , j ) skip ) x+ ∑ o 6=skip β ( i , j ) o o ( x ) ( 4 ) We then analyze how the auxiliary skip connection handles the issue of gradient flow . Referring to the theorem of a recent work by Zhou et al . ( 2020 ) , the convergence of network weight W in the supernet can heavily depend on βskip . Specifically , suppose only three operations ( none , skip connection , and convolution ) are included in the search space and MSE loss is utilized as training loss , when architecture parameters βoi , j are fixed to optimize W via gradient descent , training loss can decrease by ratio ( 1− ηλ/4 ) at one step with probability at least 1− δ , where η is the learning rate that should be bounded by δ , and λ follows Eq . 5. λ ∝ h−2∑ i=0 [ ( β ( i , h−1 ) conv ) 2 i−1∏ t=0 ( β ( t , i ) skip ) 2 ] ( 5 ) where h is the number of layers of the supernet . From Eq . 5 , we observe that λ relies much on βskip than βconv , which indicates that the network weights W can converge faster with a large βskip . However , by involving an auxiliary skip connection weighted as β , Eq . 5 can be refined as follows : λ ∝ h−2∑ i=0 [ ( β ( i , h−1 ) conv ) 2 i−1∏ t=0 ( β ( t , i ) skip + β ) 2 ] ( 6 ) Algorithm 1 DARTSRequire : Network weightsw ; Architecture parameters α ; Number of search epochs E ; Decay strategy for βe , e ∈ { 1 , 2 , ... , E } . Ensure : Searched architecture parameters α . 1 : Construct a super-network by stacking cells in which there is an auxiliary skip connection between every two nodes of choice 2 : for each e ∈ [ 1 , E ] do 3 : Update weights w by∇wLtrain ( w , α , βe ) 4 : Update parameters α by∇αLval ( w , α , βe ) 5 : end for 6 : Derive the final architecture based on learned α from the best validation supernet . where β βskip making λ insensitive to βskip , so that the convergence of network weights W depends more on βconv . In the beginning of the search , the common value for βskip is 0.15 while β is 1.0 . From the view of convergence theorem ( Zhou et al. , 2020 ) , the auxiliary skip connection alleviates the privilege of βskip and equalize the competition among architecture parameters . Even when β gradually decays , the fair competition still holds since network weights W have been converged to an optimal point . Consequently , DARTS- is able to stabilize the search stage of DARTS . Extensive experiments are performed to demonstrate the efficiency of the proposed auxiliary skip connection , and we emphasize that our method is flexible to combine with other methods to further improve the stabilization and searching performance . The overall algorithm is given in Alg . 1 .

Summary: The paper reveals two roles of skip connection to prevent model collapse: stabilize the supernet training, and as a candidate operation to build the final network. Intuitively, the skip connection playing the first role should only be there during the training phase. Therefore, the authors propose to add an auxiliary skip connection to play that role. This auxiliary skip connection is decayed gradually during training. The paper provides an interesting theoretical analysis that this can help prevent the gradient vanishing problem.