ZARTS: On Zero-order Optimization for Neural Architecture Search

Differentiable architecture search ( DARTS ) has been a popular one-shot paradigm for NAS due to its high efficiency . It introduces trainable architecture parameters to represent the importance of candidate operations and proposes first/secondorder approximation to estimate their gradients , making it possible to solve NAS by gradient descent algorithm . However , our in-depth empirical results show that the approximation will often distort the loss landscape , leading to the biased objective to optimize and in turn inaccurate gradient estimation for architecture parameters . This work turns to zero-order optimization and proposes a novel NAS scheme , called ZARTS , to search without enforcing the above approximation . Specifically , three representative zero-order optimization methods are introduced : RS , MGS , and GLD , among which MGS performs best by balancing the accuracy and speed . Moreover , we explore the connections between RS/MGS and gradient descent algorithm and show that our ZARTS can be seen as a robust gradient-free counterpart to DARTS . Extensive experiments on multiple datasets and search spaces show the remarkable performance of our method . In particular , results on 12 benchmarks verify the outstanding robustness of ZARTS , where the performance of DARTS collapses due to its known instability issue . Also , we search on the search space of DARTS to compare with peer methods , and our discovered architecture achieves 97.54 % accuracy on CIFAR-10 and 75.7 % top-1 accuracy on ImageNet , which are state-of-the-art performance . 1 INTRODUCTION . Despite their success , neural networks are still designed mainly by humans ( Simonyan & Zisserman , 2014 ; He et al. , 2016 ; Howard et al. , 2017 ) . It remains open to automatically discover effective and efficient architectures . The problem of neural architecture search ( NAS ) has attracted wide attention , which can be modeled as bi-level optimization for network architectures and operation weights . One-shot NAS ( Bender et al. , 2018 ) is a popular search framework that regards neural architectures as directed acyclic graphs ( DAG ) and constructs a supernet with all possible connections and operations in the search space . DARTS ( Liu et al. , 2019 ) further introduces trainable architecture parameters to represent the importance of candidate operations , which are alternately trained by SGD optimizer along with network weights . It proposes a first-order approximation to estimate the gradients of architecture parameters , which is biased and may lead to the severe instability issue shown by ( Bi et al. , 2019 ) . Other works ( Zela et al. , 2020b ; Chen & Hsieh , 2020 ) point out that architecture parameters will converge to a sharp local minimum resulting in the instability issue and introduces extra regularization items so that architecture parameters converge to a flat local minimum . In this paper , we empirically show that the first-order approximation of optimal network weights sharpens the loss landscape and results in the instability issue of DARTS . It also shifts the global minimum , misleading the training of architecture parameters . To this end , we discard such approximation and turn to zero-order optimization algorithms , which can run without the requirement that the search loss is differentiable w.r.t . architecture parameters . Specifically , we introduce a novel NAS scheme named ZARTS , which outperforms DARTS by a large margin and can discover efficient architectures stably on multiple public benchmarks . In a nutshell , this paper sheds light on the frontier of NAS in the following aspects : 1 ) Establishing zero-order based robust paradigm to solve bi-level optimization for NAS . Differentiable architecture search has been a well-developed area ( Liu et al. , 2019 ; Xu et al. , 2020b ; Wang et al. , 2020b ) which solves the bi-level optimization of NAS by gradient descent algorithms . However , this paradigm suffers from the instability issue during search since biased approximation for optimal network weights distorts the loss landscape , as shown in Fig . 1 ( a ) and ( b ) . To this end , we propose a flexible zero-order optimization NAS framework to solve the bi-level optimization problem , which is compatible with multiple potential gradient-free algorithms in the literature . 2 ) Uncovering the connection between zero-order architecture search and DARTS . This work introduces three representative zero-order optimization algorithms without enforcing the unverified differentiability assumption for search loss w.r.t . architecture parameters . We reveal the connections between the zero-order algorithms and gradient descent algorithm , showing that two implementations of ZARTS can be seen as gradient-free counterparts to DARTS , being more stable and robust . 3 ) Strong empirical performance and robustness . Experiments on four datasets and five search spaces have been conducted to evaluate the performance of our method . Unlike DARTS that suffers the severe instability issue shown by ( Zela et al. , 2020b ; Bi et al. , 2019 ) , ZARTS can stably discover effective architectures on various benchmarks . In particular , the searched architecture achieves 75.7 % top-1 accuracy on ImageNet , outperforming DARTS and most of its variants . 2 RELATED WORK . One-shot Neural Architecture Search . Bender et al . ( 2018 ) construct a supernet so that all candidate architectures can be seen as its sub-graph . DARTS ( Liu et al. , 2019 ) introduces architecture parameters to represent the importance of operations in the supernet and update them by gradient descent algorithm . Some works ( Xu et al. , 2020b ; Wang et al. , 2020b ; Dong & Yang , 2019 ) reduce the memory requirement of DARTS in the search process . Other works ( Zela et al. , 2020b ; Chen & Hsieh , 2020 ) point out the instability issue of DARTS , i.e. , skip-connection gradually dominates the normal cells , leading to performance collapse during the search stage . Bi-level Optimization for NAS . NAS can be modeled as a bi-level optimization for architecture parameters and network weights . DARTS ( Liu et al. , 2019 ) proposes first/second-order approximations to estimate gradients of architecture parameters so that they can be trained by gradient descent algorithms . However , we show that such approximation will distort the loss landscape and mislead the training of architecture parameters . Amended-DARTS ( Bi et al. , 2019 ) derives an analytic formula of the gradient w.r.t . architecture parameters that includes the inverse of Hessian matrix of network weights , which is even unfeasible to compute . In contrast , this work discards the approximation in DARTS and attempts to solve the bi-level optimization by gradient-free algorithms . Zero-order Optimization . Unlike gradient-based optimization methods that require the objective differentiable w.r.t . the parameters , zero-order optimization can train parameters when the gradient of objective is unavailable or difficult to obtain , which has been widely used in adversarial robustness for neural networks ( Chen et al. , 2017 ; Ilyas et al. , 2018 ) , meta learning ( Song et al. , 2020 ) , and transfer learning ( Tsai et al. , 2020 ) . Liu et al . ( 2020b ) aim at AutoML and utilize zero-order optimization to discover optimal configurations for ML pipelines . In this work , ( to our best knowledge ) we make the first attempt to apply zero-order optimization to NAS and experiment with multiple algorithms , from vanilla random search ( Flaxman et al. , 2004 ) to more advanced and effective direct search ( Golovin et al. , 2020 ) , showing its great superiority against gradient-based methods . 3 BI-LEVEL OPTIMIZATION IN DARTS . Following one-shot NAS ( Bender et al. , 2018 ) , DARTS constructs a supernet stacked by normal cells and reduction cells . Cells in the supernet are represented by directed acyclic graphs ( DAG ) with N nodes { xi } Ni=1 , which represents latent feature maps . Each edge ei , j contains multiple operations { oi , j , o ∈ O } , whose importance is represented by architecture parameters αoi , j . Therefore , NAS can be modeled as a bi-level optimization problem by alternately updating the operation weights ω ( parameters within candidate operations on each edge ) and the architecture parameters α : min α Lval ( ω∗ ( α ) , α ) ; s.t . ω∗ ( α ) = arg min ω Ltrain ( ω , α ) . ( 1 ) 3.1 FUNDAMENTAL LIMITATIONS IN THE DARTS FRAMEWORK . By enforcing an unverified ( and in fact difficult to verify ) assumption that the search loss Lval ( ω∗ ( α ) , α ) is differentiable w.r.t . α , DARTS ( Liu et al. , 2019 ) proposes a second-order approximation for the optimal weights ω∗ ( α ) by applying one-step gradient descent : ω∗ ( α ) ≈ ω∗2nd ( α ) = ω − ξ∇ωLtrain ( ω , α ) = ω′ , ( 2 ) where ξ is the learning rate to update network weights . Thus the gradient of the loss function w.r.t . α , ∇αLval ( ω∗ ( α ) , α ) , can be computed by the chain rule : ∇αLval ( ω∗ ( α ) , α ) ≈ ∇αLval ( ω′ , α ) − ξ∇2α , ωLtrain ( ω , α ) ∇ω′Lval ( ω′ , α ) . Nevertheless , the second-order partial derivative is hard to compute , so the authors adopt the difference method , which is proved in Appendix A.1 . To further reduce the computational cost , first-order approximation is introduced by assuming ω∗ ( α ) being independent of α , as shown in Eq . 3 , which is much faster and widely used in many variants of DARTS ( Chen et al. , 2019 ; Wang et al. , 2020b ; Zela et al. , 2020b ) . ω∗ ( α ) ≈ ω∗1st ( α ) = w. ( 3 ) The gradient is then simplified as : ∇αLval ( ω∗ ( α ) , α ) ≈ ∇αLval ( ω , α ) , which , however , further exacerbates the estimation bias . Reexamining the definition of ω∗ ( α ) in Eq . 1 , one would note that it is intractable to derive a mathematical expression for ω∗ ( α ) , making Lval ( ω∗ ( α ) , α ) even non-differentiable w.r.t . α . Yet DARTS has to compromise with such approximations as Eq . 2 and Eq . 3 so that differentiability is established and SGD can be applied . However , such sketchy estimation of optimal operation weights can distort the loss landscape w.r.t . architecture parameters and thus mislead the search procedure , which is shown in Fig . 1 and analyzed in the next section . 3.2 DISTORTED LANDSCAPE AND INCORRECT OPTIMIZATION PROCESS IN DARTS . Fig . 1 illustrates the loss landscape with perturbations on architecture parameters α , showing how different approximations of ω∗ affect the search process . We train a supernet for 50 epochs and randomly select two orthonormal vectors as the directions to perturbα . The same group of perturbation directions is used to draw landscapes in Fig . 1 ( a ) and ( b ) for a fair comparison . Fig . 1 ( a ) shows the loss landscape with the first-order approximation in DARTS , ω∗1st ( α ) = ω , while Fig . 1 ( b ) shows the loss landscape with more accurate ω∗ ( α ) , which is obtained by fine-tuning the network weights ω for 10 iterations for each α. Landscapes ( contours ) are plotted by evaluating L at grid points ranged from -1 to 1 at an interval of 0.02 in both directions . Global minima are marked with stars on the landscapes , from which we have two observations : 1 ) The approximation ω∗1st ( α ) = ω shifts the global minimum and sharpens the landscape 1 , which is the representative characteristic of instability issue as pointed out by ( Zela et al. , 2020b ) . 2 ) Accurate estimation for ω∗ leads to a flatter landscape , indicating that the instability issue can be alleviated . Moreover , we display the landscape with second-order approximation ω∗2nd in Appendix A.2 , which is also sharp but slightly flatter than 1A “ sharp ” landscape has denser contours than a “ flat ” one . Algorithm 1 : ZARTS : Zero-order Optimization Framework for Architecture Search

Summary: This paper presents ZARTS, a zero-order optimization method for DARTS, to search without enforcing the approximation of the network weights. It conducts in-depth analysis on the first/second order approximation in DARTS, and points out that such approximation leads to bias and instability. Then the work proposes three zero-order optimization methods to solve the issue.