Evo-NAS: Evolutionary-Neural Hybrid Agent for Architecture Search
Authors: Krzysztof Maziarz
Mingxing Tan
Andrey Khorlin
Kuang-Yu Samuel Chang
Andrea Gesmundo
Venue:  (2020)

Abstract: Neural Architecture Search has shown potential to automate the design of neural networks. Deep Reinforcement Learning based agents can learn complex architectural patterns, as well as explore a vast and compositional search space. On the other hand, evolutionary algorithms offer higher sample efficiency, which is critical for such a resource intensive application. In order to capture the best of both worlds, we propose a class of Evolutionary-Neural hybrid agents (Evo-NAS). We show that the Evo-NAS agent outperforms both neural and evolutionary agents when applied to architecture search for a suite of text and image classification benchmarks. On a high-complexity architecture search space for image classification, the Evo-NAS agent surpasses the accuracy achieved by commonly used agents with only 1/3 of the search cost.

Under review as a conference paper at ICLR 2020
Evo-NAS: EVOLUTIONARY-NEURA HYBRID AGENT FOR ARCHITECTURE SEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
Neural Architecture Search has shown potential to automate the design of neural networks. Deep Reinforcement Learning based agents can learn complex archi- tectural patterns, well as explore vast and compositional search space. On the other hand, evolutionary algorithms offer higher sample efficiency, which is critical for such resource intensive application. In order to capture the hest of both worlds, we propose class of Evolutionary-Neut the Evo-NAS agent outperforms both neural and evolutionary agents when applied to architecture search for a suite of text and image classification benchmarks. On a high-complexity architecture search space for image classification, the Evo-NAS agent surpasses the accuracy achieved by commonly used agents with only 1/3 the search cost.
hybrid agents (Evo-NAS). We show that
INTRODUCTION
Neural Networks (NN) have yielded success in many supervised and unsupervised learning problems. However, the design of state-of-the-art deep learning algorithms requires many decisions, normally involving human time and expertise. As an alternative, Auto-ML approaches aim to automate manual design with meta-learning agents. Many different approaches have been proposed for architecture optimization, including random search, evolutionary algorithms, Bayesian optimization and an approach based on a NN trained with Reinforcement Learning (RL).
RL-based neural architecture search yielded success in automatic design of state-of-the-art RNN cells (Zoph and Le| 2017), convolutional blocks (Zoph et al. (017), activation functions (Prajit Ra- machandran] 2018), , optimizers (Bello / Wichrowsaa et al.) 2017) and data augmentation strategies (Cubuk et al. 2018). Under this paradigm, the NN architecture is sampled from the agent's policy, which in turn optimized to maximize the performance of the generated models the downstream task.
Recent work has shown that evolutionary agents for architecture search match outperform deep RL methods (Real et al. 20188) So et al. 2019). Evolutionary agents can efficiently leverage a single good model by generating similar models via mutation process. Deep RI methods generate new models by sampling from a learned distribution, and cannot easily latch on to patterns of a single model, unless has been promoted multiple times through the learning process. However, evolution has the disadvantage of relying on heuristics or random sampling when choosing mutations. Unlike approaches based on a Neural Network, Evolutionary approaches are unable to learn patterns to drive the search.
The main contribution of this paper is to introduce class of Evolutionary-Neut hybrid agents (Evo-NAS). We propose an evolutionary agent whose mutations are guided by NN trained with RL. This combines the sample efficiency of evolutionary agents with the ability learn complex patterns of NNs.
In Section 3 we give a brief description of state-of-the-art Neural and Evolutionary agents, and introduce the Evo-NAS agent in Section 4 In Section we discuss the different training algorithms used for training the NN-based agents. Then, in Section we present and discuss the properties the proposed Evolutionary- Neural agent by applying it to synthetic task. Finally, apply Evo-NAS to architecture search benchmarks, showing that it outperforms both RL- RL-based and Evolution-basee algorithms architecture search for variety of text and image classification datasets.

---

Under review as a conference paper at ICLR 2020
2
RELATED WORK
In recent years, progress has been made in automating the design process required to produce state- of-the-art neural networks. Recent methods have shown that learning-based approaches can achieve state-of-the-ar results on ImageNet (Zoph and Le 2017 Liu et al., 2017a). These results have been subsequently scaled by transferring architectural building blocks between datasets (Zoph et al. 2017).
Some works explicitly address resource efficiency (Zhong et al. 2018 Pham et al. 2018 Liu et al. 2018 Xie al. (2018), which is crucial, architecture search is known to require large amount of resources (Zoph and Le. 2017). In particular, ENAS (Pham et al.
2018 and DARTS (Liu et al.
2018 are computationally efficient techniques that have been employed to learn what segments of an architecture can be pruned. These methods can be applied on a narrow subset of the architecture search applications that satisfy the scalability and expressiveness constraints (Sciuto et al. 2019: Li and Talwalkar 2019). Furthermore, gradient-based approaches like DARTS can be applied to search over parameters for which gradients from the task loss can be computed. These approaches cannot be applied to a broader set of architecture search applications such as those presented in this work.
Prior works have used evolutionary methods evolve neural networks (Floreano al. 2008 Stanley et al., 2009, Real et al. 2017, Conti et al. 2017, Xie and Yuille, 2017). Real et al. (2018) have shown that evolutionary agents applied to architecture search can match or outperform the standard deep-RL. RI. based agents. (Such et al. (2017) have applied genetic algorithms to evolve the weights of the model.
RENAS (Chen et al. 2018) is an alternative RL-evolutionary hybrid that has been developed in parallel to Evo-NAS. RENAS also provides evidence that learning mutation patterns increases the efficiency of evolution. A key difference is that Evo-NAS learns a policy directly on the entire architecture search-space while RENAS learns policy over the space of possible mutations applicable to a given parent model. Thus, the application of RENAS requires to define an additional mutation grammar and more complex agent, e.g. composed of 4 distinct mutation generation sub-steps. The parent-to-child parameter sharing approach is orthogonal to the definition of the hybrid controller, and can be applied to Evo-NAS in future work.
Other than deep RL and evolution, different approaches have been applied to architecture search and hyperparameter optimization: bayesian optimization (Falkner al. 2018). cascade- correlation (Fahlman and Lebiere. 1990). boosting (Cortes et al. 2016), deep-lcarning based tree search (Negrinho and Gordon 2017). hill-climbing (Elsken et al. 2017). model-based ap-
proaches (Hutter et 201 and random search (Bergstra and Bengio 2012).
3 BASELINES
We compare the proposed Evo-NAS agent with alternative commonly used agents:
Random Search (RS) generates a new model by sampling every architectural choice from a uniform distribution over the available values. The performance of the RS agent gives a sense of the complexity of the task, and allows to estimate the quality gains to attribute to the use of a more complex agent.
Neural Architecture Search (NAS) (Zoph and Le 2017 uses an RNN perform sequence of architectural choices that define generated model. The resulting model is then trained on downstream task, and its quality on the validation set serves as the reward for training the agent using policy gradient. In the following sections, we will refer to the standard NAS agent as the Neural agent.
Aging Evolution Architecture Search (Real al. 2018) is a variant of the tournament selection method (Goldberg and Deb, 1991). A population of P generated models is improved in iterations. At each iteration, a sample of S models selected at random. The best model of the sample, parent model, is randomly mutated to produce new model, which is trained and added to the population. Each time a new model is added to the population. the oldest model in the population is discarded. In the following sections, will refer to the Aging Evolution agent as the Evolutionary agent.
4 EVOLUTIONARY-NEURRA ARCHITECTURE SEARCH
The proposed Evo-NAS agent generates new models by mutating a parent model. At each iteration, the parent model is chosen by picking the model with highest reward among random sample of S
2

---

Under review as a conference paper at ICLR 2020
models drawn from the population of the most recent P models, similarly to the Evolutionary agent. However, unlike the Evolutionary agent, the mutations are not sampled at random among the possible architectural choices, but sampled from distributions inferred by recurrent neural network (RNN).
When mutating a parent model, for each architectural choice in the sequence defining the model, the Evo-NAS agent either reuse the parent's value sample new one from the corresponding learned distribution. The decision whether to mutate value is performed for each architectural choice independently. The probability of mutating parent value is hyperparameter of the agent, which we refer to as the mutation probability: p. The values for the architectural choices that need be mutated are re-sampled from the distributions inferred by the underlying RNN. The RNN is designed to condition each distribution on all the prior architectural choices values, independently of whether the values have been ported from the parent's s sequence or re-sampled. Formally, consider the parent sequence (#1...., an). and denote its prefix with A; (01, .... as) for E (1,..., n). Fix possible child sequence (a', d, and similarly denote A (a1, ..... d(). Denoting the parameters of the RNN by 0, the probability of sampling the child sequence A', under the policy defined by the RNN is
22 pl ((4 AA,, = P
Let MI,..., Mn. be dernoulliy)-ddistiiien random variables, where M, defines whether mutation occurs at position i. Then, under the law of total probability, we can formalize the probability of generating sequence A'
n
p(A(,A,, = II Σ 1=1 (0.1
play m. A,4 A1
II ((1 ((1-p) p) 1[a; a,l +p =o(d((A/1))
Evo-NAS sampling algorithm is represented in Figure Refer to Appendix A for the pseudocode of the algorithm, and Appendix B for comparison with the baseline approaches.
Parent sequence
reuse
rouse
Generated
sample
sample
RNN
Evo-NAS agent
Figure 1: Evo-NAS agent sampling a sequence of architectural choices. Each of the colored blocks represents an architectural choice being set to a specific value. Each value of the generated sequence is sampled from the policy learned by the agent's 's neural network with probability or reused from the parent sequence with probability p.
The Evo-NAS agent is initialized so that the distributions over the architectural choices are uniform. Thus, an initialized Evo-NAS agent produces random mutations as an Evolutionary agent. During training, the Evo-NAS's S NN is updated by using policy gradient to maximize the expected quality metric achieved by the generated models on the downstream task. Therefore, the distributions over architectural choices will become more skewed with time promote the architectural patterns of the good models. In contrast, an Evolutionary agent is unable to learn mutation patterns, since the distributions from which the mutations sampled constantly uniform.
We conjecture that Evo-NAS can retain the best qualities of both baselines: the sample-efficieng of Evolutionary agents, and the ability to learn complex patterns of Neural agents. We base this the following observations. First, we note that the Evolutionary agent essentially biases the search toward the around the best samples found so far; therefore, it acts with a prior belief that high

---

Under review as a conference paper at ICLR 2020
quality samples often close to each other. As shown by (Real et al., 2018), this approach gives very strong performance on architecture search, suggesting that the clustering assumption is true for commonly used architecture search spaces. On the other hand, in high complexity search spaces, good models are likely to span only a low dimensional manifold. Under the assumption that the NN learn the structure of this manifold, Evo-NAS learn to restrict its local search the manifold learned by the underlying NN. In Section O we empirically validate our conjecture, and show that Evo-NAS indeed outperforms both baseline algorithms.
TRAINING ALGORITHMS
We consider two alternative training algorithms for NN based agents such Neural and Evo-NAS:
Reinforce (Williams) 1992). Reinforce a standard policy-gradient training algorithm. It is often considered the default choice for applications where an agent needs to be trained to explore complex search space to find the optimal solution, as is in the case of architecture search. This approach has the disadvantage of not being sample efficient, as it is not able to reuse samples. Reinforce is the training algorithm chosen in the original NAS paper (Zoph and Le 2017).
Priority Queue Training (PQT) (Abolañia et al. 2018). With PQT, the NN gradients are generated to directly maximize the log likelihood of the best samples produced so far. This training algorithm has higher sample efficiency than Reinforce, as good models generate multiple updates. PQT has the simplicity of supervised learning, since the best models are directly promoted as if they constitute the supervised training set, with no need of reward scaling as in Reinforce, or sample probability scaling as in off- off-poolic training.
6 EXPERIMENTS
6.1 SYNTHETIC TASK: LEARN TO COUNT
We compare the agents a synthetic toy task designed be complex enough such that random search would not succeed. Similarly to common architecture search spaces, it requires learning a sequential pattern. This task can be described learning count. The agent can sample sequence a = (01, 02 ... an) of n integer numbers, where each number is selected from the set [1, n] n Z. The reward of a sequence a is defined as:
raa) Ei (2 (an (an- (n++))
This reward is designed to encourage every two adjacent numbers to be close to each other, but also, keep the first number small, and the last number large. The maximum reward of is achieved by a" (1, .... n).
We compare Random, Evolutionary, Neural and Evo-NAS agents; additionally, for Neural and Evo-NAS agents we consider two alternative training algorithms: Reinforce and PQT.
We started by tuning the hyperparameters of all the agents independently, which we describe in Appendix C The results of the comparison are shown in Figure 2
We have found that PQT outperforms Reinforce for both the Neural and the Evo-NAS agent. For the Evo-NAS agent, the gain is especially pronounced at the beginning of the experiment. Thus, we conclude that training with PQT is more sample efficient than with Reinforce.
Now we turn to a comparison between: Random, Evolutionary. Neural (PQT) and Evo-NAS (PQT). The Evolutionary agent finds better models than the Neural agent during the initial 1000 samples, while in the second half of the experiment, the Neural agent outperforms the Evolutionary agent. Our interpretation is that Evolutionary agent's s efficient exploitation allows to have a better start by mutating good trials. While the Neural agent needs 1000+ samples to learn the required patterns, after this is achieved it generates better samples than those generated by Evolutionary agent's 's random mutations. The results show that Evo-NAS agent achieves both the sample efficiency of Evolutionary approaches and the learning capability of Neural approaches. Evo-NAS initial fast improvement shows the ability to take advantage of the sample efficiency of evolution. Learning the proper
4

---

Under review as a conference paper at ICLR 2020
0.8
0.8
0.7
0.7
Boward movre
0.3
0.2
02
0.1
0.1
Number
Number trials
(POT)
(PQT) Neural (Reinforce)
Evalutiorary
Evo-NAS (Reinforce)
Rardom search
Figure 2: Reward moving average (left) and best reward (right) on "Learn to count" task averaged over 20 replicas. Shaded area represents 70% confidence interval.
mutation patterns allows to keep outperforming the Evolutionary agent. The poor performance of the Random Search agent shows that the "Learn to count" task is non-trivial. We also see that the Neural agent would not have been able catch up with the Evolutionary agent within the 5000 trials of this experiment if it was trained with Reinforce instead of PQT. For more details about the properties this task see Appendix
6.2 NASBENCH
We perform the same comparison using NASBench (Ying et al. 2019), where the agents search for network architectures for CIFAR-10 classification task a confined search space, which consists of approximately 423k unique architectures. We compare the agents as in Section 66. Random, Evolutionary, Neural and Evo-NAS, and for Neural and Evo-NAS agents we consider training them with either Reinforce or PQT. We again tuned the hyperparameters for all the agents, and report the final values in Appendix D
We show the results of the comparison in Figure
0.00
0.007
0.005
0.005
Exe-NAS (PQT)
0.003
Evo NAS (Ramsorro)
Neuni (PQT)
Neuni (Renrforoa)
10' Estimated wall cook time (scconds)
10'
Figure 3: Reward moving average (left) and best reward (right) on NASBench task averaged over 64 replicas. Shaded area represents 70% confidence interval.
There are many similarities between Figures and 3] but a key feature that sets them apart is that the quality achieved by Random Search is significantly better for NASBench than for "Learn to count". This is because large percentage of architectures give relatively low test regret (less than 0.01).
Despite the difference mentioned above, we can draw consistent conclusion from both lines of experiments. Evo-NAS and Evolutionary agents perform well the beginning. Neural agent (trained with PQT) outperforms the Evolutionary agent when it collects sufficient amount of samples to learn the pattern. On the other hand, Evo-NAS combines the qualities of the Neural and Evolutionary agents to deliver the best result.
6.3 TEXT CLASSIFICATION TASKS
We now compare the different agents on a real architecture search task. The Neural, Evolutionary and Evo-NAS agents are applied the task of finding architectures for 7 text classification datasets.
5

---

Under review as a conference paper at ICLR 2020
Similarly to (Wong et al.) 2018), designed medium complexity search space of common architectural and hyperparameter choices that define two-tower "wide and deep" models (Cheng et al. 2016). This search space is not designed to discover original architectures that set new state-of-the-art for this type of tasks, but it allows to analyze the properties of the agents.
All the experiments in this section hours with CPUs each. Choosing a small budget allows to run a higher number of replicas for each experiment to increase the significance of the results. It also makes the budget accessible most of the scientific community, thus increasing the reproducibility of the results.
with fixed budget: 30 trials trained in parallel for
During the experiments, the models sampled by the agent are trained the training set of the current text classification task, and the area under the ROC curve (ROC-AUC) computed on the validation set is used as the reward for the agent. To compare the generated models that achieved the best reward, we compute the ROC-AUC on the holdout testset as the final unbiased evaluation metric. For the datasets that do not come with a pre-defined vrain/validationneest split, we split randomly 80% 10%0000% respectively for training, validation and test set.
We validate the results of the comparison between PQT and Reinforce presented in the previous lines of experiments by running 5 experiment replicas for each of the 7 tasks using the Neural agent with both training algorithms. We measure an average relative gain of +1.13% over the final test ROC-AUC achieved by using PQT instead of Reinforce.
We use PQT for the following experiments to train the Evo-NAS and Neural agents to maximize the log likelihood of the top 5 trials. For the Evo-NAS and Evolutionary agents, we have set the mutation probability to 0.5. Evo-NAS and Neural agents use PQT with learning rate 0.0001 and entropy penalty 0.1. These parameters were selected with preliminary tuning to ensure a fair comparison.
To measure the quality of the models generated by the three agents, we run 10 architecture search experiment replicas for each of the 7 tasks, and measure the test ROC-AUC obtained by the best model generated by each experiment replica. The results are summarized in Table The Evo-NAS agent achieves the best performance on all tasks. On 3 out of 7 tasks it significantly outperforms both the Neural and the Evolutionary agents.
Table I:
Best ROC-AUC(%) on the testset for each algorithm and dataset. We report the average over 10 distinct architecture search runs, as well as + 2 -oredard-rrro--o-tthee indicates the best performing algorithm or those within 2 s.c.m. of the best.
(s.e.m.). Bolding
Dataset
Neural
Evolutionary
Evo-NAS
20Newsgruups
95.45 0.17
95.31 +: 0.39
95.67 : 0.19
Brown
66.29 1.44
66.79 1.31
66.82 = 1.25
CompunerComplaants
55.08 0.97
54.43 1.41
56.63 1 0.71
MeDonalds
71.14 + 1.19
71.00
+ 1.43
71.90 + 1.03
NewsAggregator
99.03 = 0.04
99.01 0.04
99.03 = 0.04
Reuters
92.36 1 0.36
92.68
1 0.36
92.89 1 0.21
SmsSpanCollection
99.76 + 0.10
99.75 + 0.08
99.82 + 0.05
We also report the number of trials each of the agents performed during the 2h long experiments, and summarize the results in Appendix EE.5 We find that the Evolutionary and the Evo-NAS agents strongly outperform the Neural agent in terms of number of trials performed. The Evo-NAS agent achieves the largest number of trials on 6 out of 7 datasets, while the Evolutionary agent on 5 out of 7 datasets. On datasets the Evolutionary and Evo-NAS agents perform joint best. This shows that the evolutionary algorithms are biased towards faster models, as shown in (Real et al. 2018).
An in-depth analysis of the architectures that achieved the best performance beyond the scope of this paper. However, we describe some noticeable patterns, along with more information about the datasets, scarch space, experiments and setup, in Appendix
6.4
IMAGE CLASSIFICATION TASK
We also compare the agents different architecture search domain: image classification. This is higher complexity task and the most common benchmark for architecture search (Zoph and Le 2017
Real al. 2018 Liu al. 2017b 2018)

---

Under review as a conference paper at ICLR 2020
As shown in recent studies (Zoph al. 2017 Liu al. 2017b), the definition of the architecture search space is critical to be able to achieve state-of tftte-of-hh-arr performance, In this line of experiments, reuse the Factorized Hierarchical Search Space defined in (Tan al. 2018). This is recently proposed search space that has shown to be able to reach state-of-the-ard performance. Note that we use the variant of the search space that does not contain a squeeze-ande aaaeeee--and-eeiiattioon block (Hu et al. 2018), so it is more comparable with other works, most of which do not use this component. We abstain from proposing an improved search space that could allow the main objective of this work is to analyze and compare the properties of the agents.
new state-of-the-an since
As the target image classification task we use ImageNet (Russakovsky al., 2015). As is common in the urchitecture search literature, we create a validation set by randomly selecting 50K images from the training set. The accuracy computed on this validation set is used as the reward for the agents, while the original ImageNet test set is used only for the final evaluation.
Following common practice previous architecture search work (Zoph and Le| 2017 Real et 2018; Tan et al., 2018), we conduct architecture search experiments then transfer the top-performing discovered model to the target task. As a simple proxy task we use ImageNet itself, but with fewer training steps. During architecture search, we train each generated model for epochs using aggressive schedules for learning rate and weight decay, and then evaluate the model on the 50K validation images.
smaller proxy task, and
During a single architecture search experiment, each agent trains thousands models. However, only the model achieving the best reward is transferred to the full ImageNet. As (Tan al. 2018). for full ImageNet training, we train for 400 epochs using RMSProp optimizer with decay 0.9 and momentum 0.9, batch norm momentum 0.9997, and weight decay 0.00001. The learning rate linearly increases from 0 to 0.256 in the first 5-epoch warmup training stage, and then decays by 0.97 every 2.4 epochs. We use standard Inception preprocessing and resize input images to 224 x 224.
Every architecture search experiment trains 60 generated models in parallel. Each model is trained on Cloud TPUv2, and takes approximately 3 hours complete the training on the proxy task. Because of the high cost of experiment, we limit the agents' hyperparameters tuning, and we set them to the values that have worked well in the previous experiments. Evo-NAS and Neural agents use PQT with learning rate 0.0001 and entropy penalty 0.1. PQT maximizes the log likelihood of the top 5% trials. Population size P = 500 and the sample size S 50, for both the Evolutionary and the Evo-NAS agent. The only parameter we do a preliminary tuning for is the mutation probability p, since in our experience this parameter is the most sensitive to the complexity of the search space, and the Factorized Hierarchical Search Space used for this experiments orders of magnitude more complex: it contains 1.6 10 different architectures. To tune p. we run 4 experiments using the Evolutionary agent with values: p E {0.03, 0.05, 0.1, 0.2). and choose the best p 0.1 be used for both Evo-NAS and Evolutionary agents. Due to the high cost of the experiments, we do not run experiment replicas. Notice that this is common practice for architecture search on image domain.
58.
58.
Rewerd moving average
56-
56-
52.
52:
so.
50
48.
46.
45
44-
44
42
2000
4000
6000
0
2000
4000
6000
Number of trials
Number of trials
Eve-NAS agent
Evolutionary agent
Neural agent
Figure 4: Reward moving average (left) and best reward (right) on the ImageNet proxy task.
In Figure we show the plots of the metrics tracked during the architecture search experiments. Each architecture search experiment required -304 hours to produce 6000 trials. The plot of the moving average of the reward confirms the properties that observed in the previous lines of experiments. The Neural agent has a slower start, while Evo-NAS retains the initial sample efficiency of the Evolutionary agent, and is able to improve the quality of the samples generated in the longer
7

---

Under review as a conference paper at ICLR 2020
term by leveraging the learning ability. The discussed properties also visible the plot of the best reward in Figure [ (Right). The Neural agent has a slower start, but is able to close the gap with the Evolutionary agent in the longer term. While Evo-NAS shows initial rate of improvement comparable to the Evolutionary agent, it is able to outperform the other agents in the later stages.
As an additional baseline, we Random Search based agent up to 3000 trials. Its reward moving average does not show improvements over time as expected. The max reward achieved is 52.54, while Evo-NAS achieves reward of 57.68 with the same number of trials. These results confirm the com- plexity of the task. Refer to Appendix for more details about the comparison with Random Search.
The best rewards achieved by each agent are respectively: 56.73 for the Neural agent, 57.23 for the Evolutionary agent, and 57.99 .999 for Evo-NAS. Trial 2003 of Evo-NAS is the first one that outperforms all models generated by the other agents. Thus, Evo-NAS surpasses the performance of the other agents with only 1/3 of the trials. Furthermore, during the course of the entire experiment, Evo-NAS generates 1063 models achieving higher reward than any model generated by the other agents.
Finally, for each of the agents we select the generated model that achieved the best reward, train them on the full ImageNet task, and evaluate on the held-out test set. This allows to measure the extent to which the reward gains on the proxy task translate to the full task, and also compare with other results published on this benchmark. The results are reported in Table which shows, that the reward gains translate to the final task. Also, the achieved test errors are comparable to the best results published on this benchmark. Notice that this comparison is influenced by factors unrelated to the choice of the agent. For example, some of the methods presented in Table CIFAR-10 as the proxy task, while others used a subset of ImageNet. The definition of architecture search space is an other important factor in determining the quality of the generated models on the downstream task. NetsNett-2 is the only published result of a network that was generated by exploring the same Factorized Hierarchical Search Space (Tan et al., 2018). It achieves slightly lower results even compared to our Neural agent baseline. Our hypothesis is that this delta can be justified by considering that MNasNet was generated by maximizing hybrid reward accounting for model latency. Note that although Table presents a variety of different methods, all listed networks have a similar number of parameters of around 5M.
Table 2: Comparison of mobile-sized satte-of-the-nsd image classifiers on ImageNet.
Architecture
Test error (top-1)
Search cost (gpu days)
Search method
Inception-vl
Szegedy et all
2015
30.2
manual manual manual gradient rl smbo
MobileNet-vi
Howand et 2017
29.4
ShuffleNet-v2
Zhang et 2017
26.3
DARTS Lia al. 2018
26.9
4
NASNet-A Zopa 2017)
26.0
1800
PNAS Liu
2617bb
25.8
225
AmoebaNet
Real al. 2018)
24.3
3150
evo.
MnasNet-92 92
Tan al. 2018 2018
25.2
988
rl
Neural agent best model
24.78
740
r
Evolutionary best model
24.70
740
evo.
Evo-NAS best model
24.57
740
evo. + rl
The architectures of the best models generated by the 3 agents show noticeable common patterns, which we discuss in Appendix F.2
7 CONCLUSION
We introduce a class of Evo-NAS hybrid agents, which are designed to retain both the sample efficiency of evolutionary approaches, and the ability to learn good architectural patterns of Neural agents. We experiment on synthetic, text and image classification tasks, analyze the properties of the proposed Evo-NAS agent, and show that it outperforms both Neural and Evolutionary agents. Additionally, show that Priority Queue Training outperforms Reinforce also architecture search applications. Notice that Evo-NAS is not specific to architecture search. As future work, it would be interesting to apply it to other reinforcement learning tasks.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V. Le. Learning transferable architectures for scalable image recognition. CoRR, 0bbs/1777.07012. 2017.
Quoc V. Le Prajit Ramachandran, Barret Zoph. Searching for activation functions, 2018.
Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc V. Le. Neural optimizer search with reinforce- ment learning. In ICML, 2017.
Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. arXiv preprint arXiv: 00777003.041.. 2017.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:/805.0950), 2018.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularizeed evolutioo for image classifier architecture search. 2018. URL http:::ttt///tt /tt.///ts////aotoooo///////////. pdd
David R. So, Chen Liang, and Quoc V. Le. The evolved transformer. CoRR, ab111901.11117,201 URL
httpg//bb/19900/ 11117
Chenxi Liu, Barret Zoph, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. arXiv preprini arXiv:1712.00559, 2017a. 2017a.
Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Practical block-wise neural network architecture generation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2423-2432, 2018.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv: 8802.03268, 2018.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. CoRR, abs/1 806.09055, 2018. URL http::/// /aaxxiivvvv.. wwwp://g//////oo/./////..//eaae/ abs/180.. 09055
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. arXiv preprint arXiv: 1812.9996, 2018
Christian Sciuto, Kaicheng Yu, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. CoRR, abs/1902.08142. 2019. URL http:
Liam Li and Ameet Tarwalkar. Random search and reproducibility for neural architecture search. CoRR, abs/1902.007638, 2019.
http:///pggaaaaaaaoooaa/e 07638
Dario Floreano, Peter Dürr, and Claudio Mattiussi. Neuroevolution: from architectures to learning. Evolutionary Intelligence, 1(1):47-62, 2008
Kenneth O Stanley, David B D' Ambrosio, and Jason Gauci. A hypercube-based encoding for evolving large-scale neural networks. Artificial life, 15(2):185-212. 2009.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Quoe Le, and Alex Kurakin. Large-scale evolution of image classifiers. In ICML, 2017.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via population of novelty-seeking agents. arXiv preprint / 2017.
Lingxi Xie and Alan Yuille. Genetic enn. In Proceedings of the IEEE International Conference on Computer Vision, pages 1379-1388, 2017.

---

Under review as a conference paper at ICLR 2020
Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv preprint arXiv: 2017.
Yukang Chen, Gaofeng Meng, Qian Zhang, Shiming Xiang, Chang Huang, Lisen Mu, and Xinggang Wang. Reinforced evolutionary neural architecture search. arXiv preprint vrXi::/808.00/93 2018.
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimiza- tion at scale. In International Conference on Machine Learning, pages 1436-1445, 2018.
Scott E Fahlman and Christian Lebiere. The cascade-correlation learning architecture. In Advances in neural information processing systems, pages 524-532, 1990.
Corinna Cortes, Xavi Gonzalvo, Vitaly Kuznetsov, Mehryar Mohri, and Scott Yang. Adanet: Adaptive structural learning of artificial neural networks. arXiv preprint arXiv:1607.01097, 2016.
Renato Negrinho and Geoff Gordon. Deeparchitect: Automatically designing and training deep architectures. arXiv preprint arXiv:1704.08775 2017.
Thomas Elsken, Jan-Hendrik Metzen, and Frank Hutter. Simple and efficient architecture search for convolutional neural networks. arXiv preprint arXiv:/71 /11111111.,,, 2017.
Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In International conference learning and intelligent optimization, pages 507-523. Springer, 2011.
James Bergstra and Yoshua Bengio. Random search for hyper- parameter optimization. JMLR, 13 (Feb), 2012.
David E. Goldberg and Kalyanmoy Deb. A comparative analysis of selection schemes used in genetic algorithms. In Foundations of Genetic Algorithms, pages 69-93. Morgan Kaufmann, 1991.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Reinforcement Learning. Springer, 1992.
Daniel A Abolafia, Mohammad Norouzi, and Quoe V Le. Neural program synthesis with priority queue training. arXiv preprint 100i:v1801.033. 2018.
Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, and Frank Hutter. Nas- bench-101: Towards reproducible neural architecture search. CoRR, abs/1902.00635, 2019. URL
09635
Catherine Wong, Neil Houlsby, Yifeng Lu, and Andrea Gesmundo. Transfer automatic machine learning. arXiv preprint 1rXiv:1803.0278/ 2018.
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. Wide & deep learning for recommender systems. CoRR, (bb//600.00792. 2016.
Chenxi Liu, Barret Zoph, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan L. Yuille, Jonathan Huang, and Kevin Murphy, Progressive neural architecture search. CoRR, abs/1712.00559, 2017b. URL httt:p//aaxxiv.rrrrtt/
http//xxivv.o. org htsp///bs/1113 00559
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V. Le. Mnasnet: Platform-awar neural architecture search for mobile. CORR, abs/1807. 1162,, 2018. URL http::/arxiv
org/ h/rp/ab//nae 1807 61626
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-exciiation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vision, 115(3):211222)) 1-252, December 2015. ISSN 0920-5691. doi: 10. 1007// 1233-015-0801111 URL http:/ dx doi. wwt.dol..o//...111...7.7 s 111263-015-0811
10

---

Under review as a conference paper at ICLR 2020
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, abs/1704.04861 2017. URL URL httt::/arrxii org/ abs/
1784.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. CoRR, abs/1707.01083, 2017. URL http:
ptrg/abb/1700 01083
Ken Lang. Newsweeder: Learning to filter netnews. In ICML, 1995.
W. Nelson Francis and Henry Kuera. Frequency analysis of english usage. lexicon and grammar. In Houghton Mifflin, 1982.
M. Lichman. UCI machine learning repository, 2013.
Franca Debole and Fabrizio Sebastiani. An analysis of the relative hardness of Reuters-21578 subsets. Journal of the American Society for Information Science and, 56(6):384-566, 596, 2004. URL wwt.stii......tmmmmmmmmmmmmeeeee enr
http:
Tiago Almeida, José María G. Hidalgo, and Akebo Yamakami. Contributions the study of sms spam filtering: New collection and results. In Proceedings of the 1 11th ACM Symposium on Document Engineering. DocEng '11, New York, NY, USA, 2011. ACM.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespeder Steve Yuan, Chris Tar, et al. Universal sentence encoder. arXiv preprint arXiv: 11175, 2018
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(lu))2121-2155) 2011.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Brendan McMahan.
and mirror descent: Equivalence theorems and 11
regularization. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 525-533, 201

---

Under review as a conference paper at ICLR 2020
A EVO-NAS PSEUDOCODE
In Algorithm
|
we formalize our approach in the form of pseudocode.
Algorithm 1: Sampling algorithm for the Evo-NAS agent
mutation probability [0, 11, parent sequence A = (a,,..,,n)) Initialize the state the RNN h to O; Initialize input the next RNN time-step Zin 0; Initialize the output child sequence A' to Initialize togorobabyl to 0; Initialize lolal lolal Entropy to 0;
Parameters:
for i 1,2,...7 do
Feed Iin to the RNN and run it for one time-step to update à and obtain the output logits;; Sample ~ Bernouli(pp;;
m is 0 then
df = a;i
else
Sample af according to the distribution: softmar(logits));
end if
Append af to A';
Set Vin to the embedding of d/:
logProbability += log(p pl logits:));
Entalpn += entropy( so (lggitt))
end for
return A' defining the child model;
return TogProbability to compute the loss for this child model;
return total Entropy to compute the entropy regularization factor to add to the loss;
B DETAILS ON BASELINE ARCHITECTURE SEARCH ALGORITHMS
To highlight the properties of the different approaches, we propose to consider two characteristics: whether the agent has learnable parameters, enabling learn patterns; 2) whether the agent is capable of efficiently leveraging good past experiences by using mutations. These two characteristics independent, and for a fixed architecture search algorithm, both of them may or may not be present. In Table we summarize the characteristics of the methods aim to compare.
Table 3: Properties of the compared architecture search algorithms.
Algorithm
Learning
Mutation
Random Search
No
No
Neural agent
Yes
No
Evolutionary agent
No
Yes
Evo-NAS agent
Yes
Yes
To make comparing the Evo-NAS agent with the baseline agents easier, we provide equivalents of Figure See Figure the Evolutionary agent, and Figure for the Neural agent.
12

---

Under review as a conference paper at ICLR 2020
Parent sequence
reuse
reuse
Generated sequence
random
Figure Overview of the how the Evolutionary agent creates sequence of values for each architectural choice specifying generated model, given parent trial. Fach of the colored blocks represents an architectural choice set to specific value. Each action is re-sampled randomly with probability or reused from the parent with probability p.
Generaled sequence
sample
sample
sample
sample:
sample
RNN
RNN
RNN
RNN
RNN
NAS agent
Figure 6: Overview of how the Neural agent samples a trial. Each of the colored blocks represents an architectural choice a specific value. Each action is sampled from a distribution defined by an RNN.
13

---

Under review as a conference paper at ICLR 2020
C
DETAILS ON "LEARN TO COUNT" TOY TASK
The proposed toy task has multiple key properties:
The size of the search space is n", which even for small is already too big for any exhaustive search algorithm to succeed.
As shown by our experiments, Random Search performs very poorly. This allows to attribute the discovery of good sequences to properties of the algorithm, rather than to accidental discovery over time.
The experimental observation that Random Search performs badly can be intuitively plained as follows. Let be a uniform distribution over {1,-.. ,n)", then:
E [r(a)= E (n²)
ant
This suggests that, for random sequence r(a) can be expected to be much smaller than 1, which is again confirmed by the experiments.
The search space exhibits sequential structure, with notion of locally optimal decisions, making it a good task both for learning patterns, and for mutating past trials by local modifications.
Before conducting the experiments on the toy task, tuned hyperparameters of all the agents to ensure a fair comparison. We set the mutation probability p = 0.3, the population size P = 500, and the sample size S 50, for both the Evolutionary and the Evo-NAS agent. We set the learning rate to be 0.0005 for the Neural agent and 0.0001 for the Evo-NAS agent. We set the entropy penalty to 0.1 for the Neural agent and 0.2 for the Evo-NAS agent. PQT maximizes the log likelihood of the top 5% trials for the Neural agent and top 20% trials for the Evo-NAS agent.
D DETAILS ON NASBENCH
We used the hyperparameters from the synthetic task experiment as starting point, and optimized for mutation probability, learning rate and entropy penalty if applicable. In our experiment, PQT maximizes the log likelihood of top 20% trials for both Neural and Evo-NAS agents.
The hyperparameters were optimized using the following procedure:
Start with the hyperparameters used in Section 6.1 and scan learning rate: [0.0001, O. 0.0002,000 0. 0.0005, 0.001] and entropy penalty: [0.1, 0.2, 0.5, 1.0] for the best moving average reward with window size 50.
2 Use the best learning rate and entropy penalty from step (1) and scan mutation probability:
[0.4, 0.3, 0.2, 0.1] for the best moving average reward with window size 50.
Repeat step (1) but scan learning rate: [0.00001, 0.00005, 0.0001, 0.0002] and entropy penalty: [0.01, 0.05, 0.1, 0.2].
The resulting hyperparameters are collected in Table
Table 4:
Optimized hyperparameters for Random, Evolutionary, Neural (PQT/Reinforee) and Evo-NAS (PQT/Reinforce) agents in NASBench.
Agent
Mutation probability
Learning
Entropy penalty
Evo-NAS (PQT)
0.2
0.00005
0.01
Evo-NAS (Reinforce)
0.2
0.0002
0.01
Neural (PQT)
0.001
0.1
Neural (Reinforce)
0.001
0.1
Evolutionary
0.3
Random search
14

---

Under review as a conference paper at ICLR 2020
E DETAILS ON TEXT CLASSIFICATION EXPERIMENTS
E.1 DATASETS
Table 5: References for the datasets used in the text experiments.
Dataset
Reference
20Newsgroups
Lang 1995
Brown ConsumerComplaint
Francis and Kucra 1982
catalog. caroggg.dtgggoo... gov
McDonalds
crowdflower.com
NewsAggregator
(Lichman, 20133)
Reuters
Deeoole and Sebastiani 2004)
SmsSpamCCllecttion
(Almeida et al. 201
Table 6: Statistics of the text classification tasks.
Datasel
Train samples
Valid samples
Test samples
Classes
Lang
Mean lext length
20 Newsgroups
15,076
1,885
1,885
20
En
2,000
Brown Corpus
400
50
50
15
En
20,000
Consumer Complaints
146,667
18,333
18,334
157
En
1,000)
McDonalds
1,176
147
148
En
516
News Aggregator
338,349
42,294
42,294
4
Reuters
8,630
1,079
1,079
90
820
SMS Spam Collection
4,459
557
557
2
E.2
SEARCH SPACE
For our text classification experiments, we designed a search space of two-tower models, similar to the one used by (Wong et al.) (018)) One tower is deep FFNN, built by stacking: a pre-trained text-embedding module, a stack of fully connected layers, and a softmax classification layer. The other tower is wide-shallow layer that directly connects the one-hot token encodings to the softmax classification layer with a linear projection. The wide tower allows the model to learn task-specific biases for each token directly, such as trigger words, while the deep tower allows to learn complex patterns. The wide tower regularized with L1 loss promote sparsity, and the deep tower is regularized with L2 loss. We provide additional details in Table
The agent defines the generated model architecture by selecting value for every available archi- tectural or hyperparameter choice. The first action selects pre-trained text-embedding module. The details of the text-embedding modules are reported in Table These modules available via the TensorFlow Hub service" Using pre-trained text-embedding modules has two benefits: first, improves the quality of the generated models trained smaller datasets, and second, decreases convergence time of the generated models.
The optimizer for the deep column can be either Adagrad (Duchi et al. 2011 or Lazy Adam "Lazy Adam" refers to a commonly used version of Adam (Kingma and Ba, (0114) that computes the moving averages only on the current batch. These are efficient optimizers, that halve the back- paooppropngntion time, compared to more expensive optimizers such as Adam. The optimizer used for the wide column FTRL (MeMahan) 2011).
https: htttps://ww..cr http://wwwtttt.oteee//eeee www.eeewwswwrooooooooooooooo
https: /tt::///wwwww/mm www.orelw..orgroooooooo
a.7yAdam0ptimis
15

---

Under review as a conference paper at ICLR 2020
Table 7: The search space defined for text classification experiments.
Parameters
Search space
1) Input embedding modules
Refer to Table 8
2) Fine-tune input embedding module
(True, False)
3) Use convolution
(True, False]
4) Convolution activation
(relu, rclu6, Icaky relu, swish, sigmoid, tanh)
5) Convolution batch norm
(True, False)
6) Convolution max ngram length
(2.3)
7) Convolution dropout rate
[0.0, 0.1, 0.2, 0.3, 0.4)
8) Convolution number of filters
[32, 64, 128)
9) Number of hidden layers
(0, 1, 5.
10) Hidden layers size
(64, 128, 256)
11) Hidden layers activation
(relu, relu6, leaky relu, swish, sigmoid, tanh]
12) Hidden layers normalization
(none, batch norm, layer norm]
13) Hidden layers dropout rate
(0.0, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5)
14) Deep optimizer name
(adagrad. lazy adam
15) Lazy adam batch size
(128, 256]
16) Deep tower learning rate
(0.001, 0.005, 0.01, 0.05, 0.1, 0.5)
17) Deep tower regularization weight
(0.0, 0.0001, 0.001, 0.01)
18) Wide tower learning rate
(0.001, 0.005, 0.01, 0.05, 0.1, 0.5)
19) Wide tower regularization weight
(0.0, 0.0001, 0.001, 0.001,
0.01)
20) Number of training samples
(1e5, 2e5, 5e5, 1e6, 2e6, 5e6)
Table 8: Options for text input embedding modules. These are pre-trained text embedding tables, trained on datasets with different languages and size. The text input to these modules is tokenized according the module dictionary and normalized by lower-casing and stripping rare characters. We provide the handles for the modules that are publicly distributed via the TensorFlow Hub service (ht thttps:/
//www//wwwwwww tensorf foo. org/hub)
LD
Datasct
Embed dim.
Vocab. size
Training algoritimm
TensorFlow Handles
Prelix: https:// dww/geagle///
(tokens)
English small
Lang. model
English-hig
200B
128
999k
1. Lang model
Pn Pnim-en-diml 128 -o-ntrnnn 1ont
English-wiki-smalll
250
Skipgram
WIK1 worda 250 with hozmallzgtionn
Universal
512
Cer et al. 2018
Un1 ververse sa -sent Bhice ahooder77
E.3 ARCHITECTURAL PATTERNS
Here, we mention few relevant patterns of the best architectures generated across tasks. The FFNNs for the deep part of the network are often shallow and wide. The learning rate for both wide and deep parts is in the bottom of the range (0.001). The and L.2 regularization are often disabled. Our interpretation of this observation is that reducing the number of parameters is simpler and more effective regularization, which preferred over adding LI and L2 factors to the loss.
16

---

Under review as a conference paper at ICLR 2020
E.4 CHOICE OF METRIC
For the text classification experiments, we used ROC-AUC instead of the more commonly used accuracy, since it provides less noisy reward signal. In preliminary experiment, this hypothesis by running experiments on the ConsumerConplaints task. Then, for a sample of 30 models, we have computed 4 metrics: ROC-AUC on validation and test set, accuracy on validation and test set. The Pearson correlation between the validation ROC-AUC and the test ROC-AUC was 99. 99,96% while between the validation accuracy and the test accuracy was 99.70%. The scatter plot of these two sets is shown in Figure 7
validated
0.6
Accaracy
0.5
ROC-AUC
04
0.4
Figure 7: Correlation of validation accuracy with test accuracy (Left) and validation ROC-AUC with test ROC-AUC (Right). The correlation is higher for ROC-AUC. For plotting the correlations, used the ConsumerComplaint dataset.
E.5 COMPARISON OF NUMBER OF TRIALS PERFORMED
Table 9: The number of trials performed for the experiments from Figure 9 We report the average over 10 runs, as well as + standard-error--0 serror-of-thh-meee -e-thhe-een (s.e.m.). Bolding indicates the algorithm with the highest number of trials those that have performed within 2 s.e.m. of the largest number of trials.
Dataset
Neural
Evolutionary
Evo-NAS
20Newsgroups
136 6
175
174
Brown
109 = 4
119 -
116 1
ConsumerComplaints
122 5
150 9
142 - 12
McDonalds
213 - 21
258 27
359 65
NewsAggregator
304
342
291 26
Reuters
160+8
194 + 13
201 19
SmsSpamCollection
390 +:
410
617 + 150
E.6 ADDITIONAL DETAILS
In Figure we visually present the results of the experiments that we aggregated in Table
E.7 TRENDS OVER TIME
To verify that the learning patterns highlighted in Section 6.1 the reward moving average for two tasks: 20Newsgroups and ConsumerComplaints For these experiments, we have extended the time budget from 2h to 5h. This time budget extension is needed to be able to capture long-term trends exhibited by the Neural and Evo-NAS agents. We run
generalize, we plot in Figure 10
17

---

Under review as a conference paper at ICLR 2020
349
Brown
ConsumerCampaiinn
IcDionlds
Namcar bad
100
3 400
324
203
259
Renders
3092
SmaipanCattecinn
Narbo 21013
ON 000000
Neural
Evolutionary Evo-NAS
233
285
Figure Number trials performed for the experiments from Figure The empty circles represent the number of trials performed in each of the 10 experiment replicas. The filled circles represent the means of the empty circles. We superpose =1 standard deviation bars.
ConormeCoophaees
McDonn
G1 a00a
Tee ROC-MMC
0000
2,048
0.50
0.092
NowsAggi spotirr
0344
Rourars
1.893
Smo3panno lection
0 -0080O
THe ROC-UN
Neural Evolutionary
Evo-NAS
Figure 9: Results of the experiments on text classification tasks. Each experiment was run 10 times. For each run, we have selected the model that obtained the best ROC-AUC on the validation set (the best reward). These best models were then evaluated by computing the ROC-AUC on the holdout testset. The empty circles in the plot represent the test RÓC-AUC achieved by each best models. The filled circles represent the means of the empty circles. We superpose +: standard deviation bars.
10
replicas for each task. In the early stages of the experiments, we notice that the quality of the samples generated by the Neural agent are on the same level as the randomly generated samples, while the quality of the samples generated by the Evo-NAS and Evolutionary agents grows steadily. In the second half of the experiments, the Neural agent starts applying the learned patterns to the generated samples. The quality the samples generated by the Evolutionary agent flattens, which we assume due to the fact that the quality of the samples in the population is close to optimum, and the quality of the samples cannot improve, since good mutations patterns cannot be learned. Finally, we observe that the Evo-NAS agent keeps generating better samples.
18

---

Under review as a conference paper at ICLR 2020
20 gowesggoupss
ConsmeerComllaing
0.95
Reward moving average
Reweard mrvving everage
0.48
0.46
0.44
0.85
0.42
Evo NAS agent
Evo NAS agent Evolutionary agent.
0.80
Evolutionary agent
Neural agent
Neural agent
0.
Figure 10: Reward moving average for the compared agents. The average is computed over a window of 50 consecutive trials. We 3 replicas for each experiment. The shaded area represents minimum and maximum value of the rolling average across the runs.
F DETAILS ON IMAGE EXPERIMENTS
F.1 COMPARSION WITH RANDOM SEARCH
Reward moving average
555
Best reward
50-
1000
2000
3000
1000
2000
3000
Number of trials
Number trials
Evo-NAS
Evolutionary
Neural agent
Randem
Figure 11: Quality metries of the different agents during the first 3k trials of architecture search the image classification proxy task. The reward is computed on the validation set, while the test set used only for the final evaluation of the selected network. We report the moving average of the reward over 50 trials (Left) and the best reward attained so far (Right).
F.2 ARCHITECTURAL PATTERNS
The core of all networks is mostly constructed with convolutions with kernel size 5 by 5, and have similar network depth of 22 or 23 blocks. The networks found b3y Evolutionary and Evo-NAS agents both have same first and last block, but Evo-NAS tends use more filters (such 192 and 384) in later stages to achieve higher accuracy than the Evolutionary agent. We show the best networks in Figure I2
19

---

Under review as a conference paper at ICLR 2020
logits
logits
logits
Pooting, FC
Pooling, FC
Pooling, FC
(K3x3)
MaxPool
MaxPool
x2
7x7x192
7x7x180
7x7x192
MBConv8 (K5x5)
MBConv6 (K5x5)
MBConv6 (k5x5)
14x14x86
14x14x112
16x14xt
MBConvS (K3x3)
x4
MBConvS (K5x5)
x4
MBConw6 (K3x3)
x2
14x14e80
54x14e80
14x14a60
MBConeli (kdsch
MBConeli (kdsch
MBConwii (idisc)
28x28x40
28x28e40
28s28x40
MBConv8 (K5x5)
MBConv8 (K5x5)
MBConw6 (K5x55
68w58x52
MBConv8 (K3x3)
x3
MBConvS (K5w5)
x2
MBConw6 (+5x5)
x3
112x1 12x16
112x112x24
112x1 12x24
MBConis (K3x3)
MB MBConv6 (K:333)
MBConv1 (K3x3)
x1
MaxPool
MaxPool
x2
12x32
112x1 12x32
112x1
(k3x33
(k3333
((3xa)
224x22063
224x220x3
220x22643
images
Images (b) Evolutionary
Images
Neural
Evo-NAS
Figure 12: Neural networks achieving the best reward for image classification generated by: (a) Neural agent, (b) Evolutionary agent, (c) Evo-NAS agent. For a detailed description of the Factorized Hierarchical Search Space and its modules refer to Tan et al. 2018).
20


Reviews:
[{"review_id": "HyemFkD7cH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed to combine both NN-based NAS and Aging EVO to get the benefit of both world: good global and local sample efficiency. The main idea is to use Aging EVO algorithm to guide the overall search process, and a NN predicting the final performance is used to guide the mutation process. The combined EVO-NAS has showed consistent good performance over a range of tasks.\n\nOverall, while the novelty of the paper is not exceptional, since it is a rather straightforward combination of two existing approaches, the end-results is promising. I would like to see more in-depth analysis on the combined algorithm to validate authors' hypothesis on why EVO-NAS works better. More detailed comments can be found below.\n\n1. The experiment on the synthetic task is not very helpful, since the domain can be far apart from the real NAS applications. One evidence is that the NN significantly outperform EVO in this task but not in the other tasks.\n\n2. In difficult tasks, the proposed EVO-NAS and the original Aging EVO are very close in the first few hundreds of trials, however, later the gap remains the same. It would be interesting to see if we can eliminate the gap by adding the NN component in the middle of the Aging EVO experiment (e.g., at the point of 2000 trials).\n\n3. I am also curious about the difference between the NN learned from Neural agent vs. those learned from EVO-NAS agent. Moreover, do we need a NN for the EVO-NAS? Or something simpler would be sufficient to guide the search.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Byg4gXUTKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "It is a nice paper that combines the deep reinforcement learning and evolutionary learning techniques to neural architecture search problem. Experimental results are promising. However, I still have some concerns on the current submission.\n1.In Fig 1,2 &3, it seems that the performances of Neural (PQT) keeps increasing. For better compassion, we recommend the authors reports the performances of compared algorithms until they are convergent.\n2.The different training algorithms (Reinforce and PQT) have difference performances whether because different training algorithms converge to difference local minima or stationary points.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkxX0Fu0YS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is well organized. The applied methods are introduced in detail. But it lacks some more detailed analysis.\n\nMy concerns are as follows.\n1. The performance differences between Evolutionary agent and EVO-NAS agent seem not significant. Please conduct additional statistical tests such as the Wilcoxon signed-rank test to verify the performance improvements are significant.\n2. Many studies have been conducted to automatically adjust control parameters such as crossover and mutation probabilities in evolutionary algorithm literature. It would be better to compare one of these approaches in the experiments.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks to the authors for the revision and discussion. This paper provides a neural architecture search (NAS) method, called Evolutionary-Neural hybrid agents (Evo-NAS), which combines NN-based NAS and Aging EVO. While the authors' response addressed some of the reviewers' comments, during discussion period there is a new concern that the idea proposed here highly overlaps with the method of RENAS, which stands for Reinforced Evolutionary Neural Architecture Search. Reviewers acknowledge that this might discount the novelty of the paper. Overall, there is not sufficient support for acceptance.
Decision: Reject