VAENAS: Sampling Matters in Neural Architecture Search

1 INTRODUCTION . Deep neural networks have greatly pushed the frontier of various influential applications by designing novel neural architectures ( Krizhevsky et al . ( 2012 ) ; Goodfellow et al . ( 2014 ) ; He et al . ( 2016 ) ) . Automatic model design of neural network architectures without human intervention , known as neural architecture search ( NAS ) , has drawn much attention of the community recently . It has resulted in state-of-the-art performance in the domain of image recognition ( Zoph et al . ( 2018 ) ; Real et al . ( 2019 ) ) , object detection ( Ghiasi et al . ( 2019 ) ; Chen et al . ( 2019 ) ) and semantic segmentation ( Liu et al . ( 2019 ) ) . Generally , the magnitude of search space for NAS tasks is enormous . For example , NasNet ( Zoph et al. , 2018 ) presents a search space with 6 × 109 possible cells . Searching on such huge designed space cost 2400 GPU days . Weight sharing mechanism has shown to be a promising avenue for efficient NAS . Latest algorithms on efficient NAS fall into two categories : one-shot approaches ( Bender et al . ( 2018 ) ) and gradient-based approaches ( Liu et al . ( 2018b ) ) . In one-shot approaches , prior works focus on adopting a fixed sampling strategy ( Guo et al . ( 2019 ) ; Bender et al . ( 2018 ) ; Chu et al . ( 2019 ) ) . In gradient-based approaches , search is typically performed without sampling procedure ( Liu et al . ( 2018b ) ; Xie et al . ( 2018 ) ) or hand-crafted sampling ( Liu et al . ( 2018a ) ) . Despite the success of these NAS methods on various benchmarks , however , these sampling approaches do not interactively learn the architecture distribution as the search process goes along , which makes the sampling procedure ineffective . In this paper , we propose a learnable and interactive architecture sampling module based on VAE for NAS , named as VAENAS . There are two advantages of VAENAS : 1 ) it learns the good-performing architecture distribution which could be used to reduce search space . 2 ) it can be embedded into existing NAS framework and improve the performance of current NAS methods . After developing the generic VAENAS approach , we study in detail the application of VAENAS module via two mainstream NAS approaches : one-shot approach ( Brock et al . ( 2017 ) ; Guo et al . ( 2019 ) ) and gradient-based approach ( Liu et al . ( 2018b ) ; Cai et al . ( 2018 ) ) , both of which have obtained state-of-the-art performance on neural architecture search tasks . We show how VAENAS module can be embedded in both approaches and make performance improvement consistently . We validate VAENAS with various search space on two benchmark datasets ( CIFAR-10 and ImageNet ) for image recognition . On NASNET-like search space , we apply VAENAS on both gradientbased methods and one-shot methods . Specifically , combined with one-shot approach , VAENAS achieves 2.26 % test error on CIFAR-10 and 75.8 % accuracy on ImageNet , outperforming state-ofthe-art NAS methods . On a Shufflenet-like search space , VAENAS combined with one-shot achieves 77.4 % top-1 accuracy with 365M FLOPs on ImageNet classification , outperforming state-of-the-art Efficient-B0 by 1.1 % with 6.5 % less computational complexity . Finally , we perform a thorough analysis of VAENAS on NAS-101 benchmark , to show the effectiveness of our proposed architecture sampling methods . 2 RELATED WORK . Neural Architecture Search . Recently the design of efficient neural networks has largely shifted from leveraging human knowledge to automatic methods , which is known as neural architecture search ( NAS ) . Early NAS methods adopt reinforcement learning ( RL ) ( Zoph et al. , 2018 ) or evolutionary strategy ( Real et al. , 2019 ) to search on thousands of individually evaluated networks , which is computational-consuming . Recent works focus on efficient search methods , which falls into two categories : one-shot approach ( Guo et al . ( 2019 ) ; Chu et al . ( 2019 ) ) and gradient-based approach ( Liu et al . ( 2018b ) ; Cai et al . ( 2018 ) ) , both achieve state-of-the-art results on a series of benchmark dataset ( Chen et al . ( 2019 ) ; Ghiasi et al . ( 2019 ) ; Liu et al . ( 2019 ) ) with various search space . We argue that sampling method matters in NAS and a learnable sampling algorithm can significantly improve the NAS performance . Sampling . Some prior works have applied sampling methods to NAS framework . For one-shot approach , Bender et al . ( 2018 ) randomly zero out a subset of the operations during super-network training . Guo et al . ( 2019 ) ; Stamoulis et al . ( 2019 ) adopt uniform sampling and Chu et al . ( 2019 ) employ fair uniform sampling to reduce training bias in one-shot models . For gradient-based approach , Cai et al . ( 2018 ) propose a multinomial distribution sampling to alleviate large memory consumption issue in conventional gradient NAS methods ( Liu et al . ( 2018b ) ) . Our approach presents a new perspective of sampling methods of NAS application . Algorithm 1 VAE Training Pipeline Input : Architectures and corresponding performance set S = { ( αi , ACCi ) } Ni , # of training itera- tions niters , parameters of VAE encoder φ , parameters of VAE deocder θ , ranking ratio K. Output : Updated VAE parameters φ , θ 1 : Rank S by the descending order of ACCi ; 2 : Select top K % architectures as VAE training dataset SK , SK ⊂ S ; 3 : for j = 1 , ... , niters do 4 : Optimize φ and θ with SK . 5 : end for 3 VAENAS . 3.1 MOTIVATION . Many prior works in NAS have discussed modeling the architecture distribution Γ ( α ) . The typical Γ ( α ) is to adopt either fixed architecture distribution ( e.g. , uniform distribution Bender et al . ( 2018 ) ; Guo et al . ( 2019 ) ) , or hand-crafted architecture distribution ( e.g. , Gumbel softmax distribution in Liu et al . ( 2018b ) ; Xie et al . ( 2018 ) ) . Conceptually , it is expected that the architecture distribution Γ ( α ) can interactively learn from the weight sharing super-network as the searching process goes along . To this end , we propose a learnable architecture distribution , where the distribution Γt ( α ) at time t is learned from the new incoming architecture α and prior architecture distribution at t− 1 time is Γt−1 ( α ) , we can formalize the distribution learning process as : Γt ( α ) ← T ( Γt−1 ( α ) , Metrict−1 ( α ) ) ( 1 ) where Metric denotes the predefined metric , e.g. , accuracy or loss , and T denotes the stochastic process . In equation 1 , the distribution of architectures at time t is learned by both the prior knowledge on architecture distribution Γ ( α ) and new architectures data α at t− 1 time , as in Figure 1 . As the architecture search proceeds , we randomly sample multiple architectures , rank these architectures according to the predefined metric . The T incrementally refines the distribution of architectures by continually learning the good-performing architectures from the weight sharing network . Alternatively , the weight sharing network updates the architectures weights , where the architectures is sampled from T . There are many methods that can be used to model this distribution , and we choose VAE due to its training stability as well as the convenience to be embedded into the existing NAS frameworks . 3.2 THE USAGE OF VAE . VAE approximates the marginal likelihood of architecture distribution log pθ ( α ) by maximizing variational lower bound : −DKL ( qφ ( z|α ) ||pθ ( z ) ) + Ez∼qφ ( z|α ) [ log pθ ( α|z ) ] ( 2 ) Specifically , the left term is the Kullback-Leibler divergence between the approximated posterior ( recognition model ) qφ ( z|α ) and the prior p ( z ) . And the right term is the expectation of reconstruction loss where pθ ( α|z ) denotes variational inference . As a result , to sample α from the marginal distribution pθ ( α ) , VAE samples latent variable z according to the prior distribution pθ ( z ) and feed z into the inference model pθ ( α|z ) . Commonly , the prior distribution pθ ( z ) can be set as standard Gaussian distribution N ( 0 , I ) . Let α = { oi } ni be a sequence of operation oi representing an architecture string , where n is the total number of nodes in a network or cell . We use an encoder E to take α as input and maps it into parameters µα and σα corresponding to a normal distribution N ( µα , σα ) . Symmetrically , a decoderD is used to map ze to a reconstructed architecture ᾱ , where ze ∼ N ( µe , σe ) . The encoder E corresponds qφ ( z|α ) and the decoder D corresponds pθ ( α|z ) . Specifically , the encoder E and the decoder D are implemented by two LSTM networks . Algorithm 2 VAENAS - One-Shot Input : Super-network training iterations nsupiters , VAE training iterations nvaeiters , interval to train VAE nvae , # of sampled architectures ( rank ) Ns , # of architectures seed Nseeds , super-network G , search space A , network weightsW , exploitation increase parameter inc , ranking ratio K , parameters of VAE module φ and θ . Output : The final architecture α∗ 1 : Define Γ ( α ) = ( 1− ) ∗ U ( α ) + ∗ pθ ( α|z ) ; Set = 0 ; 2 : for i = 1 , ... , nsupiters do 3 : Sample α ∼ Γ ( α ) ; 4 : Update network weightsWα ; 5 : if i mod nvae == 0 then 6 : Sample Ns number of architectures { α } from Γ ( α ) , α ∼ Γ ( α ) ; 7 : Evaluate { α } , obtain accuracy { ACC } ; 8 : Set S = { ( αi , ACCi ) } Nsi ; 9 : Call algorithm 1 VAE ( S , φ , θ , nvaeiters , K ) to update VAE module ; 10 : end if 11 : ← + inc ; 12 : end for 13 : Sample Nseeds number of architectures { α } from pθ ( α|z ) , α ∼ pθ ( α|z ) ; 14 : Set Sseeds = { α } . 15 : Derive the final architecture α∗ from Search ( G , A , Sseeds ) ; 4 VAENAS FOR NEURAL ARCHITECTURE SEARCH . In this section , we demonstrate how VAENAS is integrated into existing NAS frameworks . Section 4.1 gives the formal definition of weight sharing mechanism . Section 4.2 gives detailed description of VAENAS combined with one-shot framework . Section 4.3 shows VAENAS combined with gradient-based framework . 4.1 WEIGHT-SHARING MECHANISM . Weight sharing approaches construct a super-network G ( α , W ) that contains all the valid architectures α with shared weights W . Any architecture α is a sub-graph in the super-network and inherits corresponding weights Wα . Thus we can formalize the training of weight sharing mechanism as solving a bi-level optimization problem : W ∗α = arg min Wα Eα∼Γ ( α ) [ Ltrain ( G ( α , Wα ) ) ] ( 3 ) α∗ = arg min α∈α Lval ( G ( α , W ∗ α ) ) ( 4 ) where Γ ( α ) is an architecture distribution . There are two main frameworks which utilize weightsharing mechanism : one-shot approach and gradient-based approach .

Summary: This paper proposed to use VAE to learn a sampling strategy in neural architecture search. The main idea is to use the currently high-performing networks to train a VAE from which the sampled architectures for the next iteration will likely supply both high-performing networks and better diversity coverage. The experiments are extensive, including results under various settings.