DARTS-: Robustly Stepping out of Performance Collapse Without Indicators

1 INTRODUCTION . Recent studies ( Zela et al. , 2020 ; Liang et al. , 2019 ; Chu et al. , 2020b ) have shown that one critical issue for differentiable architecture search ( Liu et al. , 2019b ) regarding the performance collapse due to superfluous skip connections . Accordingly , some empirical indicators for detecting the occurrence of collapse have been produced . R-DARTS ( Zela et al. , 2020 ) shows that the loss landscape has more curvatures ( characterized by higher Hessian eigenvalues w.r.t . architectural weights ) when the derived architecture generalizes poorly . By regularizing for a lower Hessian eigenvalue , Zela et al . ( 2020 ) ; Chen & Hsieh ( 2020 ) attempt to stabilize the search process . Meanwhile , by directly constraining the number of skip connections to a fixed number ( typically 2 ) , the collapse issue becomes less pronounced ( Chen et al. , 2019b ; Liang et al. , 2019 ) . These indicator-based approaches have several main drawbacks . Firstly , robustness relies heavily on the quality of the indicator . An imprecise indicator either inevitably accepts poor models or mistakenly reject good ones . Secondly , indicators impose strong priors by directly manipulating the inferred model , which is somewhat suspicious , akin to touching the test set . Thirdly , extra computing cost ( Zela et al. , 2020 ) or careful tuning of hyper-parameters ( Chen et al. , 2019b ; Liang et al. , 2019 ) are required . Therefore , it ’ s natural to ask the following questions : • Can we resolve the collapse without handcrafted indicators and restrictions to interfere with the searching and/or discretization procedure ? • Is it possible to achieve robustness in DARTS without tuning extra hyper-parameters ? ∗Work done as an intern at Meituan Inc. †Correspondent Author . To answer the above questions , we propose an effective and efficient approach to stabilize DARTS . Our contributions can be summarized as follows : New Paradigm to Stabilize DARTS . While empirically observing that current indicators ( Zela et al. , 2020 ; Chen & Hsieh , 2020 ) can avoid performance collapse at a cost of reduced exploration coverage in the search space , we propose a novel indicator-free approach to stabilize DARTS , referred to as DARTS-1 , which involves an auxiliary skip connection ( see Figure 1 ) to remove the unfair advantage ( Chu et al. , 2020b ) in the searching phase . Strong Robustness and Stabilization . We conduct thorough experiments across seven search spaces and three datasets to demonstrate the effectiveness of our method . Specifically , our approach robustly obtains state-of-the-art results on 4 search space with 3× fewer search cost than R-DARTS ( Zela et al. , 2020 ) , which requires four independent runs to report the final performance . Seamless Plug-in Combination with DARTS Variants . We conduct experiments to demonstrate that our approach can work together seamlessly with other orthogonal DARTS variants by removing their handcrafted indicators without extra overhead . In particular , our approach is able to improve 0.8 % accuracy for P-DARTS , and 0.25 % accuracy for PC-DARTS on CIFAR-10 dataset . 2 RELATED WORK . Neural architecture search and DARTS variants . Over the years , researchers have sought to automatically discover neural architectures for various deep learning tasks to relieve human from the tedious effort , ranging from image classification ( Zoph et al. , 2018 ) , objection detection ( Ghiasi et al. , 2019 ) , image segmentation ( Liu et al. , 2019a ) to machine translation ( So et al. , 2019 ) etc . Among many proposed approaches , Differentiable Architecture Search ( Liu et al. , 2019b ) features weight-sharing and resolves the searching problem via gradient descent , which is very efficient and easy to generalize . A short description of DARTS can be found in A.1 . Since then , many subsequent works have been dedicated to accelerating the process ( Dong & Yang , 2019b ) , reducing memory cost ( Xu et al. , 2020 ) , or fostering its ability such as hardware-awareness ( Cai et al. , 2019 ; Wu et al. , 2019 ) , finer granularity ( Mei et al. , 2020 ) and so on . However , regardless of these endeavors , a fundamental issue of DARTS over its searching performance collapse remains not properly solved , which extremely prohibits its application . Robustifying DARTS . As DARTS ( Liu et al. , 2019b ) is known to be unstable as a result of performance collapse ( Chu et al. , 2020b ) , some recent works have devoted to resolving it by either designing indicators like Hessian eigenvalues for the collapse ( Zela et al. , 2020 ) or adding perturbations to regularize such an indicator ( Chen & Hsieh , 2020 ) . Both methods rely heavily on the indicator ’ s accurateness , i.e. , to what extent does the indicator correlate with the performance collapse ? Other methods like Progressive DARTS ( Chen et al. , 2019b ) and DARTS+ ( Liang et al. , 2019 ) employ a strong human prior , i.e. , limiting the number of skip-connections to be a fixed value . Fair DARTS ( Chu et al. , 2020b ) argues that the collapse results from the unfair advantage in an exclusive competitive environment , from which skip connections overly benefit to cause an abundant aggregation . To suppress such an advantage from overshooting , they convert the competition into collaboration where each operation is independent of others . It is however an indirect approach . SGAS ( Li et al. , 2020 ) , instead , circumvents the problem with a greedy strategy where the unfair advantage can be prevented from taking effect . Nevertheless , potentially good operations might be pruned out too early because of greedy underestimation . 1We name it so as we undertake an inward way , as opposed to those outward ones who design new indicators , add extra cost and introduce new hyper-parameters . 3 DARTS- . 3.1 MOTIVATION . We start from the detailed analysis of the role of skip connections . Skip connections were proposed to construct a residual block in ResNet ( He et al. , 2016 ) , which significantly improves training stability . It is even possible to deepen the network up to hundreds of layers without accuracy degradation by simply stacking them up . In contrast , stacking the plain blocks of VGG has degenerated performance when the network gets deeper . Besides , Ren et al . ( 2015 ) ; Wei et al . ( 2017 ) ; Tai et al . ( 2017 ) ; Li et al . ( 2018b ) also empirically demonstrate that deep residual network can achieve better performance on various tasks . From the gradient flow ’ s perspective , skip connection is able to alleviate the gradient vanishing problem . Given a stack of n residual blocks , the output of the ( i + 1 ) th residual block Xi+1 can be computed as Xi+1 = fi+1 ( Xi , Wi+1 ) + Xi , where fi+1 denotes the operations of the ( i + 1 ) th residual block with weights Wi+1 . Suppose the loss function of the model is L , and the gradient of Xi can be obtained as follows ( 1 denotes a tensor whose items are all ones ) : ∂L ∂Xi = ∂L ∂Xi+1 · ( ∂fi+1 ∂Xi + 1 ) = ∂L ∂Xi+j · j∏ k=1 ( ∂fi+k ∂Xi+k−1 + 1 ) ( 1 ) We observe that the gradient of shallow layers always includes the gradient of deep layers , which mitigates the gradient vanishing of Wi . Formally we have , ∂L ∂Wi = ∂L ∂Xi+j · j∏ k=1 ( ∂fi+k ∂Xi+k−1 + 1 ) · ∂fi ∂Wi ( 2 ) To analyze how skip connect affects the performance of residual networks , we introduce a trainable coefficient β on all skip connections in ResNet . Therefore , the gradient of Xi is converted to : ∂L ∂Xi = ∂L ∂Xi+1 · ( ∂fi+1 ∂Xi + β ) ( 3 ) Once β < 1 , gradients of deep layers gradually vanish during the back-propagation ( BP ) towards shallow layers . Here β controls the memory of gradients in BP to stabilize the training procedure . We conduct a confirmatory experiment on ResNet50 and show the result in Fig 2 . By initializing β with { 0 , 0.5 , 1.0 } , we can visualize the tendency of β along with training epochs . We observe that β converges towards 1 after 40 epochs no matter the initialization , which demonstrates that the residual structure learns to push β to a rather large value to alleviate gradient vanishing . Similarly , DARTS ( Liu et al. , 2019b ) utilizes a trainable parameter βskip to denote the importance of skip connection . However , In the search stage , βskip can generally increase and dominate the architecture parameters , and finally leads to performance collapse . We analyze that a large βskip in DARTS could result from two aspects : On the one hand , as the supernet automatically learns to alleviate gradient vanishing , it pushes βskip to a proper large value ; On the other hand , the skip connection is indeed an important connection for the target network , which should be selected in the discretization stage . As a consequence , the skip connection in DARTS plays two-fold roles : as an auxiliary connection to stabilize the supernet training , and as a candidate operation to build the final network . Inspired by the above observation and analysis , we propose to stabilize the search process by distinguishing the two roles of skip connection and handling the issue of gradient flow . 3.2 STEPPING OUT OF THE PERFORMANCE COLLAPSE . To distinguish the two roles , we introduce an auxiliary skip connection between every two nodes in a cell , see Fig . 1 ( b ) . On the one hand , the fixed auxiliary skip connection carries the function of stabilizing the supernet training , even when βskip is rather small . On the other hand , it also breaks the unfair advantage ( Chu et al. , 2020b ) as the advantageous contribution from the residual block is factored out . Consequently , the learned architectural parameter βskip can be freed from the role of controlling the memory of gradients , and is more precisely aimed to represent the relative importance of skip connection as a candidate operation . In contrast to Eq . 7 , the output feature map of edge e ( i , j ) can now be obtained by Eq . 4 , where βi , jo = exp ( α ( i , j ) o ) ∑ o′∈O exp ( α ( i , j ) o′ ) denotes the normalized importance , and β is a coefficient independent from the architecture parameters . Moreover , to eliminate the impact of auxiliary connection on the discretization procedure , we propose to decrease β to 0 in the search phase , and our method can be degenerated to standard DARTS at the end of the search . Note that our method is insensitive to the type of decay strategy , so we choose linear decay by default for simplicity . ō ( i , j ) ( x ) = βx+ ∑ o∈O β ( i , j ) o o ( x ) = ( β + β ( i , j ) skip ) x+ ∑ o 6=skip β ( i , j ) o o ( x ) ( 4 ) We then analyze how the auxiliary skip connection handles the issue of gradient flow . Referring to the theorem of a recent work by Zhou et al . ( 2020 ) , the convergence of network weight W in the supernet can heavily depend on βskip . Specifically , suppose only three operations ( none , skip connection , and convolution ) are included in the search space and MSE loss is utilized as training loss , when architecture parameters βoi , j are fixed to optimize W via gradient descent , training loss can decrease by ratio ( 1− ηλ/4 ) at one step with probability at least 1− δ , where η is the learning rate that should be bounded by δ , and λ follows Eq . 5. λ ∝ h−2∑ i=0 [ ( β ( i , h−1 ) conv ) 2 i−1∏ t=0 ( β ( t , i ) skip ) 2 ] ( 5 ) where h is the number of layers of the supernet . From Eq . 5 , we observe that λ relies much on βskip than βconv , which indicates that the network weights W can converge faster with a large βskip . However , by involving an auxiliary skip connection weighted as β , Eq . 5 can be refined as follows : λ ∝ h−2∑ i=0 [ ( β ( i , h−1 ) conv ) 2 i−1∏ t=0 ( β ( t , i ) skip + β ) 2 ] ( 6 ) Algorithm 1 DARTSRequire : Network weightsw ; Architecture parameters α ; Number of search epochs E ; Decay strategy for βe , e ∈ { 1 , 2 , ... , E } . Ensure : Searched architecture parameters α . 1 : Construct a super-network by stacking cells in which there is an auxiliary skip connection between every two nodes of choice 2 : for each e ∈ [ 1 , E ] do 3 : Update weights w by∇wLtrain ( w , α , βe ) 4 : Update parameters α by∇αLval ( w , α , βe ) 5 : end for 6 : Derive the final architecture based on learned α from the best validation supernet . where β βskip making λ insensitive to βskip , so that the convergence of network weights W depends more on βconv . In the beginning of the search , the common value for βskip is 0.15 while β is 1.0 . From the view of convergence theorem ( Zhou et al. , 2020 ) , the auxiliary skip connection alleviates the privilege of βskip and equalize the competition among architecture parameters . Even when β gradually decays , the fair competition still holds since network weights W have been converged to an optimal point . Consequently , DARTS- is able to stabilize the search stage of DARTS . Extensive experiments are performed to demonstrate the efficiency of the proposed auxiliary skip connection , and we emphasize that our method is flexible to combine with other methods to further improve the stabilization and searching performance . The overall algorithm is given in Alg . 1 .

Summary: This paper presents an interesting method to alleviate the mode collapse of DARTS (all operations degenerate to skip-connect). This is done by simply adding a skip-connect operation to complement the output of the cell function and making the coefficient of the auxiliary operation decay with time. The method is tested on a few benchmarks and settings.