Evo-NAS: Evolutionary-Neural Hybrid Agent for Architecture Search

1 INTRODUCTION . Neural Networks ( NN ) have yielded success in many supervised and unsupervised learning problems . However , the design of state-of-the-art deep learning algorithms requires many decisions , normally involving human time and expertise . As an alternative , Auto-ML approaches aim to automate manual design with meta-learning agents . Many different approaches have been proposed for architecture optimization , including random search , evolutionary algorithms , Bayesian optimization and an approach based on a NN trained with Reinforcement Learning ( RL ) . RL-based neural architecture search yielded success in automatic design of state-of-the-art RNN cells ( Zoph and Le , 2017 ) , convolutional blocks ( Zoph et al. , 2017 ) , activation functions ( Prajit Ramachandran , 2018 ) , optimizers ( Bello et al. , 2017 ; Wichrowska et al. , 2017 ) and data augmentation strategies ( Cubuk et al. , 2018 ) . Under this paradigm , the NN architecture is sampled from the agent ’ s policy , which in turn is optimized to maximize the performance of the generated models on the downstream task . Recent work has shown that evolutionary agents for architecture search can match or outperform deep RL methods ( Real et al. , 2018 ; So et al. , 2019 ) . Evolutionary agents can efficiently leverage a single good model by generating similar models via a mutation process . Deep RL methods generate new models by sampling from a learned distribution , and can not easily latch on to patterns of a single model , unless it has been promoted multiple times through the learning process . However , evolution has the disadvantage of relying on heuristics or random sampling when choosing mutations . Unlike approaches based on a Neural Network , Evolutionary approaches are unable to learn patterns to drive the search . The main contribution of this paper is to introduce a class of Evolutionary-Neural hybrid agents ( Evo-NAS ) . We propose an evolutionary agent whose mutations are guided by a NN trained with RL . This combines the sample efficiency of evolutionary agents with the ability to learn complex patterns of NNs . In Section 3 , we give a brief description of state-of-the-art Neural and Evolutionary agents , and introduce the Evo-NAS agent in Section 4 . In Section 5 , we discuss the different training algorithms used for training the NN-based agents . Then , in Section 6 , we present and discuss the properties of the proposed Evolutionary-Neural agent by applying it to a synthetic task . Finally , we apply Evo-NAS to architecture search benchmarks , showing that it outperforms both RL-based and Evolution-based algorithms on architecture search for a variety of text and image classification datasets . 2 RELATED WORK . In recent years , progress has been made in automating the design process required to produce stateof-the-art neural networks . Recent methods have shown that learning-based approaches can achieve state-of-the-art results on ImageNet ( Zoph and Le , 2017 ; Liu et al. , 2017a ) . These results have been subsequently scaled by transferring architectural building blocks between datasets ( Zoph et al. , 2017 ) . Some works explicitly address resource efficiency ( Zhong et al. , 2018 ; Pham et al. , 2018 ; Liu et al. , 2018 ; Xie et al. , 2018 ) , which is crucial , as architecture search is known to require a large amount of resources ( Zoph and Le , 2017 ) . In particular , ENAS ( Pham et al. , 2018 ) and DARTS ( Liu et al. , 2018 ) are computationally efficient techniques that have been employed to learn what segments of an architecture can be pruned . These methods can be applied on a narrow subset of the architecture search applications that satisfy the scalability and expressiveness constraints ( Sciuto et al. , 2019 ; Li and Talwalkar , 2019 ) . Furthermore , gradient-based approaches like DARTS can be applied to search over parameters for which gradients from the task loss can be computed . These approaches can not be applied to a broader set of architecture search applications such as those presented in this work . Prior works have used evolutionary methods to evolve neural networks ( Floreano et al. , 2008 ; Stanley et al. , 2009 ; Real et al. , 2017 ; Conti et al. , 2017 ; Xie and Yuille , 2017 ) . Real et al . ( 2018 ) have shown that evolutionary agents applied to architecture search can match or outperform the standard deep-RL based agents . Such et al . ( 2017 ) have applied genetic algorithms to evolve the weights of the model . RENAS ( Chen et al. , 2018 ) is an alternative RL-evolutionary hybrid that has been developed in parallel to Evo-NAS . RENAS also provides evidence that learning mutation patterns increases the efficiency of evolution . A key difference is that Evo-NAS learns a policy directly on the entire architecture search-space while RENAS learns a policy over the space of possible mutations applicable to a given parent model . Thus , the application of RENAS requires to define an additional mutation grammar and a more complex agent , e.g . composed of 4 distinct mutation generation sub-steps . The parent-to-child parameter sharing approach is orthogonal to the definition of the hybrid controller , and can be applied to Evo-NAS in future work . Other than deep RL and evolution , different approaches have been applied to architecture search and hyperparameter optimization : bayesian optimization ( Falkner et al. , 2018 ) , cascadecorrelation ( Fahlman and Lebiere , 1990 ) , boosting ( Cortes et al. , 2016 ) , deep-learning based tree search ( Negrinho and Gordon , 2017 ) , hill-climbing ( Elsken et al. , 2017 ) , model-based approaches ( Hutter et al. , 2011 ) , and random search ( Bergstra and Bengio , 2012 ) . 3 BASELINES . We compare the proposed Evo-NAS agent with alternative commonly used agents : Random Search ( RS ) generates a new model by sampling every architectural choice from a uniform distribution over the available values . The performance of the RS agent gives a sense of the complexity of the task , and allows to estimate the quality gains to attribute to the use of a more complex agent . Neural Architecture Search ( NAS ) ( Zoph and Le , 2017 ) uses an RNN to perform a sequence of architectural choices that define a generated model . The resulting model is then trained on a downstream task , and its quality on the validation set serves as the reward for training the agent using policy gradient . In the following sections , we will refer to the standard NAS agent as the Neural agent . Aging Evolution Architecture Search ( Real et al. , 2018 ) is a variant of the tournament selection method ( Goldberg and Deb , 1991 ) . A population of P generated models is improved in iterations . At each iteration , a sample of S models is selected at random . The best model of the sample , parent model , is randomly mutated to produce a new model , which is trained and added to the population . Each time a new model is added to the population , the oldest model in the population is discarded . In the following sections , we will refer to the Aging Evolution agent as the Evolutionary agent . 4 EVOLUTIONARY-NEURAL ARCHITECTURE SEARCH . The proposed Evo-NAS agent generates new models by mutating a parent model . At each iteration , the parent model is chosen by picking the model with highest reward among a random sample of S models drawn from the population of the most recent P models , similarly to the Evolutionary agent . However , unlike the Evolutionary agent , the mutations are not sampled at random among the possible architectural choices , but are sampled from distributions inferred by a recurrent neural network ( RNN ) . When mutating a parent model , for each architectural choice in the sequence defining the model , the Evo-NAS agent can either reuse the parent ’ s value or sample a new one from the corresponding learned distribution . The decision whether to mutate a value is performed for each architectural choice independently . The probability of mutating a parent value is a hyperparameter of the agent , which we refer to as the mutation probability : p. The values for the architectural choices that need to be mutated are re-sampled from the distributions inferred by the underlying RNN . The RNN is designed to condition each distribution on all the prior architectural choices values , independently of whether the values have been ported from the parent ’ s sequence or re-sampled . Formally , consider the parent sequence ( a1 , ... , an ) , and denote its prefix with Ai = ( a1 , ... , ai ) for i ∈ { 1 , ... , n } . Fix a possible child sequence ( a′1 , ... , a ′ n ) , and similarly denote A ′ i = ( a ′ 1 , ... , a ′ i ) . Denoting the parameters of the RNN by θ , the probability of sampling the child sequence A′n under the policy πθ defined by the RNN is p ( A′n|An , θ ) = n∏ i=1 p ( a′i|Ai−1 , A′i−1 , θ ) . Let M1 , ... , Mn be Bernoulli ( p ) -distributed random variables , where Mi defines whether a mutation occurs at position i . Then , under the law of total probability , we can formalize the probability of generating a sequence A′ as : p ( A′n|An , θ ) = n∏ i=1 ∑ mi∈ { 0,1 } p ( a′i , mi|Ai−1 , A′i−1 , θ ) = n∏ i=1 ( ( 1− p ) · 1 [ a′i = ai ] + p · πθ ( a′i|A′i−1 ) ) . Evo-NAS sampling algorithm is represented in Figure 1 . Refer to Appendix A for the pseudocode of the algorithm , and Appendix B for a comparison with the baseline approaches . The Evo-NAS agent is initialized so that the distributions over the architectural choices are uniform . Thus , an initialized Evo-NAS agent produces random mutations as an Evolutionary agent . During training , the Evo-NAS ’ s NN is updated by using policy gradient to maximize the expected quality metric achieved by the generated models on the downstream task . Therefore , the distributions over architectural choices will become more skewed with time to promote the architectural patterns of the good models . In contrast , an Evolutionary agent is unable to learn mutation patterns , since the distributions from which the mutations are sampled are constantly uniform . We conjecture that Evo-NAS can retain the best qualities of both baselines : the sample-efficiency of Evolutionary agents , and the ability to learn complex patterns of Neural agents . We base this on the following observations . First , we note that the Evolutionary agent essentially biases the search toward the area around the best samples found so far ; therefore , it acts with a prior belief that high quality samples are often close to each other . As shown by ( Real et al. , 2018 ) , this approach gives very strong performance on architecture search , suggesting that the clustering assumption is true for commonly used architecture search spaces . On the other hand , in high complexity search spaces , good models are likely to span only a low dimensional manifold . Under the assumption that the NN can learn the structure of this manifold , Evo-NAS can learn to restrict its local search to the manifold learned by the underlying NN . In Section 6 we empirically validate our conjecture , and show that Evo-NAS indeed outperforms both baseline algorithms .

Summary: In this paper, the authors proposed to combine both NN-based NAS and Aging EVO to get the benefit of both world: good global and local sample efficiency. The main idea is to use Aging EVO algorithm to guide the overall search process, and a NN predicting the final performance is used to guide the mutation process. The combined EVO-NAS has showed consistent good performance over a range of tasks.