ZARTS: On Zero-order Optimization for Neural Architecture Search

Differentiable architecture search ( DARTS ) has been a popular one-shot paradigm for NAS due to its high efficiency . It introduces trainable architecture parameters to represent the importance of candidate operations and proposes first/secondorder approximation to estimate their gradients , making it possible to solve NAS by gradient descent algorithm . However , our in-depth empirical results show that the approximation will often distort the loss landscape , leading to the biased objective to optimize and in turn inaccurate gradient estimation for architecture parameters . This work turns to zero-order optimization and proposes a novel NAS scheme , called ZARTS , to search without enforcing the above approximation . Specifically , three representative zero-order optimization methods are introduced : RS , MGS , and GLD , among which MGS performs best by balancing the accuracy and speed . Moreover , we explore the connections between RS/MGS and gradient descent algorithm and show that our ZARTS can be seen as a robust gradient-free counterpart to DARTS . Extensive experiments on multiple datasets and search spaces show the remarkable performance of our method . In particular , results on 12 benchmarks verify the outstanding robustness of ZARTS , where the performance of DARTS collapses due to its known instability issue . Also , we search on the search space of DARTS to compare with peer methods , and our discovered architecture achieves 97.54 % accuracy on CIFAR-10 and 75.7 % top-1 accuracy on ImageNet , which are state-of-the-art performance . 1 INTRODUCTION . Despite their success , neural networks are still designed mainly by humans ( Simonyan & Zisserman , 2014 ; He et al. , 2016 ; Howard et al. , 2017 ) . It remains open to automatically discover effective and efficient architectures . The problem of neural architecture search ( NAS ) has attracted wide attention , which can be modeled as bi-level optimization for network architectures and operation weights . One-shot NAS ( Bender et al. , 2018 ) is a popular search framework that regards neural architectures as directed acyclic graphs ( DAG ) and constructs a supernet with all possible connections and operations in the search space . DARTS ( Liu et al. , 2019 ) further introduces trainable architecture parameters to represent the importance of candidate operations , which are alternately trained by SGD optimizer along with network weights . It proposes a first-order approximation to estimate the gradients of architecture parameters , which is biased and may lead to the severe instability issue shown by ( Bi et al. , 2019 ) . Other works ( Zela et al. , 2020b ; Chen & Hsieh , 2020 ) point out that architecture parameters will converge to a sharp local minimum resulting in the instability issue and introduces extra regularization items so that architecture parameters converge to a flat local minimum . In this paper , we empirically show that the first-order approximation of optimal network weights sharpens the loss landscape and results in the instability issue of DARTS . It also shifts the global minimum , misleading the training of architecture parameters . To this end , we discard such approximation and turn to zero-order optimization algorithms , which can run without the requirement that the search loss is differentiable w.r.t . architecture parameters . Specifically , we introduce a novel NAS scheme named ZARTS , which outperforms DARTS by a large margin and can discover efficient architectures stably on multiple public benchmarks . In a nutshell , this paper sheds light on the frontier of NAS in the following aspects : 1 ) Establishing zero-order based robust paradigm to solve bi-level optimization for NAS . Differentiable architecture search has been a well-developed area ( Liu et al. , 2019 ; Xu et al. , 2020b ; Wang et al. , 2020b ) which solves the bi-level optimization of NAS by gradient descent algorithms . However , this paradigm suffers from the instability issue during search since biased approximation for optimal network weights distorts the loss landscape , as shown in Fig . 1 ( a ) and ( b ) . To this end , we propose a flexible zero-order optimization NAS framework to solve the bi-level optimization problem , which is compatible with multiple potential gradient-free algorithms in the literature . 2 ) Uncovering the connection between zero-order architecture search and DARTS . This work introduces three representative zero-order optimization algorithms without enforcing the unverified differentiability assumption for search loss w.r.t . architecture parameters . We reveal the connections between the zero-order algorithms and gradient descent algorithm , showing that two implementations of ZARTS can be seen as gradient-free counterparts to DARTS , being more stable and robust . 3 ) Strong empirical performance and robustness . Experiments on four datasets and five search spaces have been conducted to evaluate the performance of our method . Unlike DARTS that suffers the severe instability issue shown by ( Zela et al. , 2020b ; Bi et al. , 2019 ) , ZARTS can stably discover effective architectures on various benchmarks . In particular , the searched architecture achieves 75.7 % top-1 accuracy on ImageNet , outperforming DARTS and most of its variants . 2 RELATED WORK . One-shot Neural Architecture Search . Bender et al . ( 2018 ) construct a supernet so that all candidate architectures can be seen as its sub-graph . DARTS ( Liu et al. , 2019 ) introduces architecture parameters to represent the importance of operations in the supernet and update them by gradient descent algorithm . Some works ( Xu et al. , 2020b ; Wang et al. , 2020b ; Dong & Yang , 2019 ) reduce the memory requirement of DARTS in the search process . Other works ( Zela et al. , 2020b ; Chen & Hsieh , 2020 ) point out the instability issue of DARTS , i.e. , skip-connection gradually dominates the normal cells , leading to performance collapse during the search stage . Bi-level Optimization for NAS . NAS can be modeled as a bi-level optimization for architecture parameters and network weights . DARTS ( Liu et al. , 2019 ) proposes first/second-order approximations to estimate gradients of architecture parameters so that they can be trained by gradient descent algorithms . However , we show that such approximation will distort the loss landscape and mislead the training of architecture parameters . Amended-DARTS ( Bi et al. , 2019 ) derives an analytic formula of the gradient w.r.t . architecture parameters that includes the inverse of Hessian matrix of network weights , which is even unfeasible to compute . In contrast , this work discards the approximation in DARTS and attempts to solve the bi-level optimization by gradient-free algorithms . Zero-order Optimization . Unlike gradient-based optimization methods that require the objective differentiable w.r.t . the parameters , zero-order optimization can train parameters when the gradient of objective is unavailable or difficult to obtain , which has been widely used in adversarial robustness for neural networks ( Chen et al. , 2017 ; Ilyas et al. , 2018 ) , meta learning ( Song et al. , 2020 ) , and transfer learning ( Tsai et al. , 2020 ) . Liu et al . ( 2020b ) aim at AutoML and utilize zero-order optimization to discover optimal configurations for ML pipelines . In this work , ( to our best knowledge ) we make the first attempt to apply zero-order optimization to NAS and experiment with multiple algorithms , from vanilla random search ( Flaxman et al. , 2004 ) to more advanced and effective direct search ( Golovin et al. , 2020 ) , showing its great superiority against gradient-based methods . 3 BI-LEVEL OPTIMIZATION IN DARTS . Following one-shot NAS ( Bender et al. , 2018 ) , DARTS constructs a supernet stacked by normal cells and reduction cells . Cells in the supernet are represented by directed acyclic graphs ( DAG ) with N nodes { xi } Ni=1 , which represents latent feature maps . Each edge ei , j contains multiple operations { oi , j , o ∈ O } , whose importance is represented by architecture parameters αoi , j . Therefore , NAS can be modeled as a bi-level optimization problem by alternately updating the operation weights ω ( parameters within candidate operations on each edge ) and the architecture parameters α : min α Lval ( ω∗ ( α ) , α ) ; s.t . ω∗ ( α ) = arg min ω Ltrain ( ω , α ) . ( 1 ) 3.1 FUNDAMENTAL LIMITATIONS IN THE DARTS FRAMEWORK . By enforcing an unverified ( and in fact difficult to verify ) assumption that the search loss Lval ( ω∗ ( α ) , α ) is differentiable w.r.t . α , DARTS ( Liu et al. , 2019 ) proposes a second-order approximation for the optimal weights ω∗ ( α ) by applying one-step gradient descent : ω∗ ( α ) ≈ ω∗2nd ( α ) = ω − ξ∇ωLtrain ( ω , α ) = ω′ , ( 2 ) where ξ is the learning rate to update network weights . Thus the gradient of the loss function w.r.t . α , ∇αLval ( ω∗ ( α ) , α ) , can be computed by the chain rule : ∇αLval ( ω∗ ( α ) , α ) ≈ ∇αLval ( ω′ , α ) − ξ∇2α , ωLtrain ( ω , α ) ∇ω′Lval ( ω′ , α ) . Nevertheless , the second-order partial derivative is hard to compute , so the authors adopt the difference method , which is proved in Appendix A.1 . To further reduce the computational cost , first-order approximation is introduced by assuming ω∗ ( α ) being independent of α , as shown in Eq . 3 , which is much faster and widely used in many variants of DARTS ( Chen et al. , 2019 ; Wang et al. , 2020b ; Zela et al. , 2020b ) . ω∗ ( α ) ≈ ω∗1st ( α ) = w. ( 3 ) The gradient is then simplified as : ∇αLval ( ω∗ ( α ) , α ) ≈ ∇αLval ( ω , α ) , which , however , further exacerbates the estimation bias . Reexamining the definition of ω∗ ( α ) in Eq . 1 , one would note that it is intractable to derive a mathematical expression for ω∗ ( α ) , making Lval ( ω∗ ( α ) , α ) even non-differentiable w.r.t . α . Yet DARTS has to compromise with such approximations as Eq . 2 and Eq . 3 so that differentiability is established and SGD can be applied . However , such sketchy estimation of optimal operation weights can distort the loss landscape w.r.t . architecture parameters and thus mislead the search procedure , which is shown in Fig . 1 and analyzed in the next section . 3.2 DISTORTED LANDSCAPE AND INCORRECT OPTIMIZATION PROCESS IN DARTS . Fig . 1 illustrates the loss landscape with perturbations on architecture parameters α , showing how different approximations of ω∗ affect the search process . We train a supernet for 50 epochs and randomly select two orthonormal vectors as the directions to perturbα . The same group of perturbation directions is used to draw landscapes in Fig . 1 ( a ) and ( b ) for a fair comparison . Fig . 1 ( a ) shows the loss landscape with the first-order approximation in DARTS , ω∗1st ( α ) = ω , while Fig . 1 ( b ) shows the loss landscape with more accurate ω∗ ( α ) , which is obtained by fine-tuning the network weights ω for 10 iterations for each α. Landscapes ( contours ) are plotted by evaluating L at grid points ranged from -1 to 1 at an interval of 0.02 in both directions . Global minima are marked with stars on the landscapes , from which we have two observations : 1 ) The approximation ω∗1st ( α ) = ω shifts the global minimum and sharpens the landscape 1 , which is the representative characteristic of instability issue as pointed out by ( Zela et al. , 2020b ) . 2 ) Accurate estimation for ω∗ leads to a flatter landscape , indicating that the instability issue can be alleviated . Moreover , we display the landscape with second-order approximation ω∗2nd in Appendix A.2 , which is also sharp but slightly flatter than 1A “ sharp ” landscape has denser contours than a “ flat ” one . Algorithm 1 : ZARTS : Zero-order Optimization Framework for Architecture Search

Summary: This paper (ZARTS) proposes to apply gradient-estimation-based zero-order optimization methods to tackle neural architecture search (NAS). Two major contributions are made by this paper: (1) it is the first to borrow the methods of zero-order optimization to solve NAS  problem; (2) it shows that zero-order optimization methods can greatly avoid the training instability of first- or second-order optimization NAS methods like DARTS (which sharpens the loss landscape). Moreover, this paper also provides an explanation on how ZARTS is connected with DARTS. Extensive experiments demonstrate the efficacy of ZARTS.