Evolutionary-Neural Hybrid Agents for Architecture Search

Neural Architecture Search has recently shown potential to automate the design of Neural Networks . The use of Neural Network agents trained with Reinforcement Learning can offer the possibility to learn complex patterns , as well as the ability to explore a vast and compositional search space . On the other hand , evolutionary algorithms offer the greediness and sample efficiency needed for such an application , as each sample requires a considerable amount of resources . We propose a class of Evolutionary-Neural hybrid agents ( Evo-NAS ) , that retain the best qualities of the two approaches . We show that the Evo-NAS agent can outperform both Neural and Evolutionary agents , both on a synthetic task , and on architecture search for a suite of text classification datasets . 1 INTRODUCTION . Deep Neural Networks ( DNN ) have yielded success in many supervised learning problems . However , the design of state-of-the-art deep learning algorithms requires many decisions , normally involving human time and expertise . As an alternative , Auto-ML approaches aim to automate manual design with meta-learning agents . Many different approaches have been proposed for architecture optimization , including random search , evolutionary algorithms , Bayesian optimization and DNN trained with Reinforcement Learning . Deep Reinforcement Learning ( deep RL ) is one of the most common approaches . It samples architectures from a distribution , which is modeled by a deep neural network . The parameters of the controller model are trained using RL to maximize the performance of the downstream learning algorithm . Architecture search based on deep RL has shown many recent successes in automatic design of state-of-the-art RNN cells ( Zoph & Le , 2017 ) , convolutional blocks ( Zoph et al. , 2017 ) , activation functions ( Prajit Ramachandran , 2018 ) , optimizers ( Bello et al. , 2017 ; Wichrowska et al. , 2017 ) and data augmentation strategies ( Cubuk et al. , 2018 ) . Recently , Real et al . ( 2018 ) has shown that evolutionary approaches , with appropriate regularization , can match or surpass the RL-DNN meta learner on architecture search task where sample efficiency is critical . Indeed , those methods can efficiently exploit a single observation of a good trial , when learning based methods require more observations . Classical evolutionary methods have the disadvantage of relying on heuristics or random sampling when choosing mutations to be applied to the parent trial . Unlike approaches based on a Neural Network ( NN ) controller , they are unable to learn patterns to drive the search . The scientific contribution of this paper is to introduce a class of Evolutionary-Neural hybrid agents ( Evo-NAS ) . Concretely , we propose an evolutionary agent , where the mutations are guided by a NN trained with RL . This combines both the sample efficiency of Evolutionary agents , and the ability to learn complex patterns , retaining the best of both worlds . In Section 3 we give a brief description of state-of-the-art Neural and Evolutionary agents , and introduce by comparison the proposed Evo-NAS agent in Section 4 . Then , in Section 5 , we present and discuss the properties of the proposed Evolutionary-Neural agent , applying it to a synthetic task . Finally , we apply Evo-NAS to architecture search benchmarks , showing that it outperforms both RL-based and Evolution-based algorithms on architecture search for a variety of text classification datasets . 2 RELATED WORK . In recent years , progress has been made in automating the design process required to produce stateof-the-art neural networks . Recent methods have shown that learning-based approaches can achieve state-of-the-art results on ImageNet ( Zoph & Le , 2017 ; Liu et al. , 2017 ) . These results have been subsequently scaled by transferring architectural building blocks between datasets ( Zoph et al. , 2017 ) . Some works explicitly address resource efficiency ( Zhong et al. , 2018 ; Pham et al. , 2018 ) , which is crucial , as architecture search is known to require vast amount of resources ( Zoph & Le , 2017 ) . Another important approach to architecture search is neuro-evolution ( Floreano et al. , 2008 ; Stanley et al. , 2009 ; Real et al. , 2017 ; Conti et al. , 2017 ) . Recent work has highlighted the importance of regularizing the evolutionary process , showing that evolution can match or outperform a learningbased baseline ( Real et al. , 2018 ) . Others have applied genetic algorithms to evolve the weights of the model ( Such et al. , 2017 ) . Other than deep RL and evolution , different approaches have been applied to architecture search : cascade-correlation ( Fahlman & Lebiere , 1990 ) , boosting ( Cortes et al. , 2016 ) , deep-learning based tree search ( Negrinho & Gordon , 2017 ) , hill-climbing ( Elsken et al. , 2017 ) and random search ( Bergstra & Bengio , 2012 ) . 3 BASELINES . 3.1 NEURAL ARCHITECTURE SEARCH . Neural Architecture Search ( NAS ) ( Zoph & Le , 2017 ) uses a RNN controller to perform architectural choices that define a child model . These choices can also include hyperparameters such as learning rate . The resulting child model is then trained on the downstream task , and its accuracy on a validation set is computed . This accuracy serves as the reward for training the RNN controller using a policy gradient approach . In the following sections , we will refer to the standard NAS agent as the Neural agent . 3.2 REGULARIZED EVOLUTION ARCHITECTURE SEARCH . Regularized Evolution ( Real et al. , 2018 ) is a variant of the tournament selection method ( Goldberg & Deb , 1991 ) . A population of P trained models is improved in iterations . At each iteration , a sample of S models is selected at random . The best model of the sample is mutated to produce a child model with an altered architecture , which is trained and added to the population . The regularization consists of discarding the oldest model of the population instead of the one with the lowest reward . This avoids over-weighting of models that may have reached their high accuracy by chance during the noisy training process . In the following sections , we will refer to the standard Regularized Evolution agent as the Evolutionary agent . 4 EVOLUTIONARY-NEURAL ARCHITECTURE SEARCH . We propose a hybrid agent , which combines an Evolutionary approach with the use of a RL trained Neural controller . The population update and parent sampling is equivalent to Regularized Evolution . The main difference is in how the mutations are generated . At mutation sampling time , the parent sequence is injected in a standard NAS-RNN controller . The Evolutionary-Neural controller has only one modification , that allows it either to reuse the parent ’ s action or to sample a new one from the learned distributions over the action space . This decision to keep or re-sample an action is taken independently for any action in the sequence . If an action is re-sampled , it will be conditioned on all the prior actions as in the standard NAS-RNN controller . Concretely , the parent ’ s action is reused with probability p , and a new action is sampled otherwise , where p is a hyperparameter . This process is summarized in Figure 1 . At training time , the controller parameters are updated in the same way as in the standard NAS-RNN controller . Note that , compared to Regularized Evolution , this design just affects the distributions from which the mutations are sampled . Regularized Evolution is unable to learn good mutation patterns , since the distributions from which the mutations are sampled are constantly uniform ( random sampling ) . For Evo-NAS , these distributions are initialized as uniform , but with time are updated to promote the patterns of the good models . Evo-NAS is designed to retain the sample-efficiency of Evolution , which is able to efficiently exploit good trials , while the underlining Neural Network is able to learn biases and complex patterns , as the standard Neural agent . 4.1 TRAINING . We consider two alternative training algorithms : Reinforce ( Williams , 1992 ) . Reinforce is the standard on-policy policy-gradient training algorithm . It is often considered the default choice for applications where a NN needs to be trained to learn a task that has no supervision , but requires to explore a complex space to find the optimal solution , as is in the case of architecture search . This approach has the disadvantage of not being as sample efficient as its off-policy alternatives , which are able to reuse samples . Reinforce is the training algorithm chosen in the original NAS paper ( Zoph & Le , 2017 ) . Priority Queue Training ( PQT ) ( Abolafia et al. , 2018 ) . In PQT , samples are generated by sampling from the learned distributions as in Reinforce . However , at training time , the gradients are generated to directly maximize the log likelihood of the best samples produced so far . This training algorithm has the sample efficiency of RL off-policy training approaches , since the best models generate multiple updates . PQT has the simplicity of supervised learning , since the best models are directly promoted as if they constitute the supervised training set , with no need of reward scaling as in Reinforce , or sample probability scaling as in off-policy training . In the experiment section we will compare these two training approaches and conclude that PQT leads to higher sample-efficiency and better results . 5 EXPERIMENTS . To properly evaluate the advantages and disadvantages of different approaches , we propose to consider two characteristics : • whether the agent is capable of learning patterns • whether the agent is capable of efficiently exploiting good past experiences The two characteristics above are independent , and for a fixed architecture search algorithm , both of them may or may not be present . In Table 1 , we summarize the characteristics of the methods we aim to compare . 5.1 LEARN TO COUNT . To highlight the properties of the proposed Evo-NAS agent , we evaluated it on a synthetic task . In this task , the agent is asked to choose a sequence a = 〈a1 , a2 , · · · , an〉 of n integer numbers . Each number is selected from the set [ 1 , n ] ∩ Z . Here n is a hyperparameter that controls the difficulty of the task . Given a sequence , a , its imbalance , i ( a ) , is defined as follows : i ( a ) = n∑ k=0 ( ak+1 − ak ) 2 ( 1 ) Where we set the initial and last terms a0 = 0 and an+1 = n+ 1 . The reward observed by the agent after choosing a sequence a is then : r ( a ) = n+ 1 i ( a ) ( 2 ) Notice that : • in order to maximize reward , the agent must minimize imbalance . • r ( a ) ∈ ( 0 , 1 ] . • there is a single optimal sequence that achieves the reward of 1 , namely a = 〈1 , 2 , · · · , n〉 . The proposed toy task has multiple key properties : • The size of the search space is nn , which even for small n is already too big for any exhaustive search algorithm to succeed . • If U is uniform distribution over { 1 , · · · , n } n then : E a∼U [ r ( a ) ] = O ( 1 n2 ) . Moreover , Random Search performs very poorly , as shown by our experiments reported below . This allows to attribute the discovery of good sequences to properties of the algorithm , rather than to accidental discovery over time . • The search space exhibits a sequential structure , with a notion of locally optimal decisions , making it a good task both for learning patterns , and for mutating past trials by local modifications . We perform a preliminary tuning of the hyperparameters of the agents to ensure a fair comparison . For the following experiments , we set the decision exploitation probability p = 0.7 , the population size P = 500 and the sample size S = 50 , for both the Evolutionary and the Evo-NAS agent . We set the learning rate to be 0.0005 for the Neural agent and 0.0001 for the Evo-NAS agent . We set the entropy penalty of 0.1 for the Neural agent and 0.2 for the Evo-NAS agent . PQT maximizes the log likelihood of the top 5 % trials for the Neural agent and top 20 % trials for the Evo-NAS agent . We start by comparing Reinforce and PQT as alternative training algorithms for the Neural and EvoNAS agents . The results of the experiments are shown in Figure 2 . We found that PQT outperformed Reinforce for both the Neural and the Evo-NAS agent . For the Evo-NAS agent , the gain is especially pronounced at the beginning of the experiment . Thus , we conclude that PQT can provide a stronger training signal than Reinforce . We will use PQT for the following comparison between agents . The results of the comparison between agents on the “ Learn to count ” task are shown in Figure 3 . We find that PQT provides a stronger training signal , allowing the Neural agent to outperform the Evolutionary agent given enough training . However , the Evolutionary agent is learning faster during the initial 1000 samples , showing higher sample efficiency . Evo-NAS agent initial quick improvement shows its ability to take advantage of the sample efficiency of Evolution . We also observe that learning the proper mutation patterns allows the Evo-NAS agent to find good sequences faster than the Evolutionary agent . Hence , in sample efficiency the Evo-NAS agent strongly outperforms both the Evolutionary and the Neural agent .

Summary: The paper proposes a hybrid approach which combines evolution and RL. The key idea is to conduct tournament selection over a population of architectures with learned mutations. The mutations are defined as the output of an RNN controller which either reuses or alters the sequence descriptor of the parent at each step. The proposed hybrid architect is evaluated on both synthetic and text classification tasks and then compared against pure evolutionary and RL-based agents.