Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters

1 INTRODUCTION . Neural architecture search ( NAS ) has been an important topic in the research area of automated machine learning ( AutoML ) . The idea is to replace the manual way of designing neural network architectures with an automatic algorithm , by which deep learning methods become more flexible in fitting complex data distributions , e.g. , large-scale image datasets . Early efforts of NAS involved using heuristic search methods such as reinforcement learning ( Zoph & Le , 2017 ; Zoph et al. , 2018 ) and evolutionary algorithms ( Real et al. , 2017 ; Xie & Yuille , 2017 ) to sample networks from a large search space , and optimizing each sampled network individually to evaluate its quality . Despite notable successes obtained by this methodology , it often requires a vast amount of computation , which obstacles its applications in the scenarios of limited resources . Inspired by the idea of reusing and sharing parameters among trained networks , DARTS ( Liu et al. , 2019b ) was designed as a ‘ one-shot ’ solution of NAS . The major difference from the aforementioned methods is a differentiable formulation of architecture search and an end-to-end mechanism which optimizes model weights ( such as convolution ) and architectural weights simultaneously . Recently , improvements upon DARTS were made in various aspects ( Chen et al. , 2019 ; Xu et al. , 2019 ; Liang et al. , 2019 ) , making it a reasonable tradeoff between search cost and performance . Despite its broad applications , the current pipeline of DARTS ( or , generally speaking , differentiable NAS approaches ) suffers a critical weakness known as instability . Researchers reported ( Liang et al. , 2019 ) that DARTS-based algorithms can sometimes generate weird architectures that produce considerably worse accuracy than those generated in other individual runs , or even significantly worse than randomly generated architectures . There indeed exist tricks designed by human expertise ( Chen et al. , 2019 ; Nayman et al. , 2019 ; Liang et al. , 2019 ) to alleviate this issue , but we point out that these approaches violated the ideology of NAS , which is to maximally prevent human interventions . Moreover , even with such add-ons , a dramatic property of DARTS persists and has not been studied carefully in prior work . When DARTS , as well as its variants , gets trained for a longer time , e.g. , from the default number of 50 epochs to 200 epochs , we surprisingly observe that all these approaches converge to very similar architectures , in which almost all edges are occupied by skip-connect ( a.k.a. , identity ) . These architectures , with fewer trainable parameters , are often far from producing high accuracy in particular on large datasets like ImageNet , but they somehow produce sufficiently high validation accuracy in the search stage . In other words , convergence in search often leads to bad performance in re-training . This is why some previous DARTS-based approaches advocated for early termination ( Liang et al. , 2019 ) , a practical but non-essential solution . Also , we conjecture that early termination also contributes to the lack of stability and , more importantly , trustfulness , of DARTS-based approaches . This paper delves deep into the inconsistency between convergence and performance . We show that the devil lies in optimizing the loss function of the super-network , Lval ( ω ? ( α ) , α ) ( ω andα denote network and architectural parameters , respectively , and ω ? ( α ) is the global optimum of ω givenα ) , in which ω and α get updated alternately . Following the chain rule , ∇αLval ( ω ? ( α ) , α ) |α=αt equals to ∇αLval ( ω , α ) |ω=ω ? ( αt ) , α=αt + ∇αω ? ( α ) |α=αt · ∇ωLval ( ω , α ) |ω=ω ? ( αt ) , α=αt , in which the first term is easy to compute while the second term is not , mainly because ω ? ( α ) is difficult to estimate , and so is the term of ∇αω ? ( α ) |α=αt . DARTS-based approaches performed inaccurate approximation for this purpose , in which the first-order version of DARTS directly discarded the second term – but this term is often numerically significant , and the second-order version of DARTS applied an approximation to this term which is not mathematically guaranteed ( see Section 3.4 ) . Consequently , the accuracy of ∇αLval ( ω ? ( α ) , α ) |α=αt can not be guaranteed , and hence the update of α can be problematic . To the best of our knowledge , this issue is not studied by existing DARTS-based approaches . To deal with this problem , we propose an alternative way of computing ∇αω ? ( α ) |α=αt . We make use of an important property , i.e. , ∇ωLtrain ( ω , α ) |ω=ω ? ( α† ) , α=α† ≡ 0 holds for any α† , which directly comes from the optimality of ω ? ( α ) . Differentiating both sides with respect to α† , we obtain a new equality which enables computing derives ∇αω ? ( α ) |α=αt with the inverse of the Hesse matrix , ∇2ωLtrain ( ω , α ) ∣∣ ω=ω ? ( α ) , α=αt . This idea enables us to achieve a more accurate approximation on ∇αLval ( ω ? ( α ) , α ) |α=αt when ω ? ( α ) is not available . Mathematically , we prove that when we have ωest ≈ ω ? ( αt ) , the inner angle between the second term and our approximate term is smaller than 90 degrees . Note that this property does not hold in existing DARTS-based algorithms . Our final solution involves using the amended second term of ∇αLval ( ω ? ( α ) , α ) |α=αt meanwhile keeping the first term unchanged , which goes one step further in optimizing the supernetwork , which reflects in a higher validation accuracy in the search stage . Our approach is very easily implemented . The overall computational overhead is comparable to the second-order version of DARTS . Experiments are performed on image classification , with popular datasets including CIFAR and ImageNet being used . In all experiments , we allow the search stage to come to a complete convergence and report competitive accuracy among current state-of-thearts . The stability of our approach also enables us to close the gap between hyper-parameters of search and evaluation , as well as explore more complex search spaces , which are believed to be correct directions of NAS but existing DARTS-based approaches would mostly fail . Therefore , we believe our algorithm can expand the application scenario of differentiable NAS methods in particular DARTS-based approaches . The remainder of this paper is organized as follows . We briefly review related work in Section 2 , and illustrate our approach of amending architectural gradients in Section 3 . After experiments are shown in Section 4 , we conclude this work in Section 5 . 2 RELATED WORK . With the era of big data and powerful computational resources , deep learning ( LeCun et al. , 2015 ) , in particular , deep neural networks ( Krizhevsky et al. , 2012 ) , have rapidly grown up to be the standard tool for learning representations in a complicated feature space . Recent years have witnessed the trend of using deeper ( He et al. , 2016 ) and denser ( Huang et al. , 2017 ) networks to boost recognition performance , while there is no justification that whether these manually designed architectures are best for each specific task , e.g. , image classification . To advance , researchers started considering the possibility of learning network architectures automatically from data , which led to the appearance of neural architecture search ( NAS ) ( Zoph & Le , 2017 ) , which is now popular and known as a sub research field in automated machine learning ( AutoML ) . The common pipeline of NAS starts with a pre-defined space of network operators . Since the search space is often large ( e.g. , containing 1010 or even more possible architectures ) , it is unlikely that exhaustive search is tractable , and thus heuristic search methods are widely applied for speedup . Typical examples include reinforcement learning ( Zoph & Le , 2017 ; Zoph et al. , 2018 ; Liu et al. , 2018a ) and evolutionary algorithms ( Real et al. , 2017 ; Xie & Yuille , 2017 ; Real et al. , 2019 ) . These approaches followed a general pipeline that samples a set of architectures from a learnable distribution , evaluates them and learns from rewards by updating the distribution . In an early age , each sampled architecture underwent an individual training process from scratch and thus the overall computational overhead is large , e.g. , hundreds of even thousands of GPU-days . To alleviate the burden , researchers started to share computation among training sampled architectures , with the key lying in reusing network weights trained previously ( Cai et al. , 2018 ) or starting from a well-trained super-network ( Pham et al. , 2018 ) . These efforts shed light on the so-called one-shot architecture search methods , which required training the super-network only once and thus ran more efficiently , e.g. , two or three orders of magnitude faster than conventional approaches . Within the scope of one-shot architecture search , an elegant solution lies in jointly formulating architecture search and approximation , so that it is possible to apply end-to-end optimization for training network and architectural parameters simultaneously . This methodology is known today as differentiable NAS , and a typical example is DARTS ( Liu et al. , 2019b ) , which constructed a super-network with all possible operators contained and decoupled , and the goal is to determine the weights of these architectural parameters , followed by pruning and re-training stages . This kind of approach allowed more flexible search space to be constructed , unlike conventional approaches with either reinforcement or evolutionary learning , which suffer from the computational burden and thus must constrain search within a relatively small search space ( Tan & Le , 2019 ) . Despite the inspirations brought by differentiable NAS , these approaches still suffer a few critical issues that narrow down their applications in practice . One significant drawback lies in the lack of stability , which reflects in the way that results of differentiable search can be impacted by very small perturbations , e.g. , initialization of architectural weights , training hyper-parameters , and even randomness in the training process . Existing solutions include running search for several individual times and choosing the best one in validation ( Liu et al. , 2019b ) , or using other kinds of techniques such as decoupling modules ( Cai et al. , 2019 ; Guo et al. , 2019 ) , adjusting search space during optimization ( Noy et al. , 2019 ; Chen et al. , 2019 ; Nayman et al. , 2019 ) , regularization ( Xu et al. , 2019 ) , early termination ( Liang et al. , 2019 ) , etc. , however , these approaches seemed to develop heuristic remedies rather than analyze it from the mathematical fundamentals , e.g. , how instability happens in mathematics . In this paper , we investigate the stability issue in mathematics and show that the results produced by the current approaches are much less reliable than people used to think . Then , we fix this issue by amending optimization of the architectural parameters , so that each step of the update gets closer to the correct direction . We show great improvement on stability in a fundamental task , image classification , while we believe our approach can be applied to a wide range of tasks including object detection ( Ghiasi et al. , 2019 ) , semantic segmentation ( Liu et al. , 2019a ) , hyper-parameter learning ( Cubuk et al. , 2019 ) , etc .

Summary: The paper proposed to amend the 2nd order formulation of DARTS for improved stability. The main idea is to leverage the stationarity condition at a local optimum w* (that the derivative of the training loss equals zero). This is an interesting method, though the technique itself is not new outside the sub-field of architecture search. For instance, the same trick has been used in [1] for gradient-based hyperparameter optimization. Note the continuous architecture \alpha is mathematically equivalent to a high-dimensional hyperparameter.