Title: Faster Discovery of Neural Architectures by Searching for Paths in a Large Model

Abstract: We propose Efficient Neural Architecture Search (ENAS), a faster and less expensive approach to automated model design than previous methods. In ENAS, a controller learns to discover neural network architectures by searching for an optimal path within a larger model. The controller is trained with policy gradient to select a path that maximizes the expected reward on the validation set. Meanwhile the model corresponding to the selected path is trained to minimize the cross entropy loss. On the Penn Treebank dataset, ENAS can discover a novel architecture thats achieves a test perplexity of 57.8, which is state-of-the-art among automatic model design methods on Penn Treebank. On the CIFAR-10 dataset, ENAS can design novel architectures that achieve a test error of 2.89%, close to the 2.65% achieved by standard NAS (Zoph et al., 2017). Most importantly, our experiments show that ENAS is more than 10x faster and 100x less resource-demanding than NAS.

1. Introduction:

Neural architecture search (NAS) has been applied successfully to design model architectures for image classification and language modeling (Zoph & Le, 2017; Baker et al., 2017a; Bello et al., 2017b; Zoph et al., 2017; Cai et al., 2017) . NAS however is computationally expensive and time consuming: for example, Zoph et al. (2017) use 450 GPUs and train for 3-4 days. Meanwhile, using less resources tends to produce less compelling results (Negrinho & Gordon, 2017; Baker et al., 2017a) .

The main computational bottleneck of NAS is the training of each child model to convergence to measure its accuracy. We believe that it is very inefficient and wasteful to train every child model and then throw away all the trained weights even though the child models have much in common.

Figure 1 : The graph represents the entire search space while the red arrows define a model in the search space, which is decided by a controller. Here we assume that node 1 is the input to the model whereas nodes 3, 5, and 6 are the outputs of the model.

The goal of this work is to remove this inefficiency by enabling more sharing between the child models. This idea is similar to the concept of weight inheritance in neuro-evolution (e.g., Real et al. (2017) ). To understand our method, we first need to understand the standard NAS. In standard NAS (Zoph & Le, 2017; Baker et al., 2017a) , an RNN controller is trained by policy gradient to search for a good architecture, which is basically a computational graph. Our observation is that all of the graphs, that NAS has iterated over, can be viewed as sub-graphs of a larger graph. In other words, we can represent the space of these graphs as a single directed acyclic graph (DAG). As illustrated in Figure 1 , a neural network architecture can be found by taking a subset of edges in this DAG. This design is advantageous because it enables sharing parameters among all architectures in the search space.

Our method requires the training of a path within a large model together with the controller. The controller is trained with policy gradient to select a path that maximizes the expected reward on the validation set. Meanwhile the model corresponding to the selected path is trained to minimize a cross-entropy loss. In view of its efficiency, we name our method Efficient Neural Architecture Search (ENAS).

ENAS is both theoretically sound and empirically efficient. Using Lyapunov functions (Bottou, 1991) , we can show that the training of ENAS converges almost surely. Empirically, on CIFAR-10, ENAS achieves the error rate of 2.89%, compared to 2.65% by NAS. On Penn Treebank, ENAS discovers an architecture that achieves the perplexity of 57.8, which outperforms NAS's perplexity of 62.4 (Zoph & Le, 2017) . In all of our experiments, ENAS takes less than 16 hours to train, whilst running on a single Nvidia GTX 1080Ti GPU. Compared to NAS, our method achieves a 10X reduction of search time and 100X reduction of computational resources.

2. Related Work:

There is growing interest in improving the efficiency of neural architecture search. Concurrent to our work are the promising ideas of using learning curve prediction to skip bad models (Baker et al., 2017b) , predicting the accuracies of models before training (Deng et al., 2017) , using iterative search method for architectures of growing complexity (Liu et al., 2017a; Elsken et al., 2017) , or using hierarchical representation of architectures (Liu et al., 2017b) . Our method is also inspired by the concept of weight inheritance in neuroevolution, which has been demonstrated to have positive effects at scale (Real et al., 2017) .

Closely related to our method are other recent approaches that avoid training each architecture from scratch, such as convolutional neural fabrics -ConvFabrics (Saxena & Verbeek, 2016) and SMASH (Brock et al., 2017) . These methods are more computationally efficient than standard NAS. However, the search space of ConvFabrics is not flexible enough to include novel architectures, e.g. architectures with arbitrary skip connections as in Zoph & Le (2017) . Meanwhile, SMASH can design interesting architectures but requires a hypernetwork (Ha et al., 2017) to generate the weights, conditional on the architectures. While a hypernetwork can efficiently rank different architectures, as shown in the paper, the real performance of each network is different from its performance with parameters generated by a hypernetwork. Such discrepancy in SMASH can cause misleading signals for reinforcement learning. Even more closely related to our method is PathNet (Fernando et al., 2017) , which uses evolution to search for a path inside a large model for transfer learning between Atari games.

3. Efficient Neural Architecture Search:

In the following, we will first present our search space to design recurrent cells for recurrent networks. We will then explain how to train, and infer with the controller. Finally, we will explain our search space to design convolutional architectures.

3.1. Designing Recurrent Cells:

To facilitate our discussion of ENAS, we first describe how we employ ENAS to design a recurrent cell. The search space of ENAS is a Directed Acyclic Graph as mentioned in Section 1. The DAG has N nodes where the edges represent the flow of information between these N nodes. Similar to NAS (Zoph & Le, 2017) , ENAS has a controller RNN, which decides which edges are activated and which computations are performed at each node.

To create a recurrent cell, the controller RNN samples N blocks of decisions. Here we illustrate the ENAS mechanism via a simple example for a recurrent cell with N = 4 computational nodes. Let x t be the input signal for a recurrent cell (e.g., word embedding), and h t-1 be the output from the previous time step. The example cell, which we visualize in Figure 2 , is sampled as follows.

1. At node 1: The controller first samples an activation function. In our example in Figure 2 , it chooses the tanh activation function, which means that node 1 of the recurrent cell should compute

h 1 = tanh (x t • W (x) + h t-1 • W (h)

1 ). 2. At node 2: The controller then samples a previous index and an activation function.

In our example, it chooses the previous index 1 and the activation function ReLU. Thus, node 2 of the recurrent cell should compute

h 2 = ReLU(h 1 • W (h)

2,1 ). 3. At node 3: The controller again samples a previous index and an activation function.

In our example, it chooses the previous index 2 and the activation function ReLU. Therefore,

h 3 = ReLU(h 2 • W (h)

3,2 ). 4. At node 4: The controller again samples a previous index and an activation function.

In our example, it chooses the previous index 1 and the activation function tanh, leading to

h 4 = tanh (h 1 • W (h)

4,1 ). 5. For the output, we simply average all the loose ends, which are the nodes that are not input to any other nodes. In our example, since the indices 3 and 4 were never sampled to be the input for any node, the recurrent cell uses their average (h 3 + h 4 )/2 as its output. The outputs of the controller RNN that result in the cell in the middle and the DAG on the left. Note that nodes 3 and 4 are never sampled by the RNN, so their results are averaged and are treated as the cell's output.

In the example above, we note that for each pair of nodes j < , there is an independent parameter matrix W (h) ,j . As shown in the example, the controller decides which parameter matrices are used, by choosing the previous indices. Therefore, in ENAS, all recurrent cells in a search space share the same set of parameters.

Our search space includes an exponential number of configurations. Specifically, if the recurrent cell has N nodes and we allow 4 activation functions (namely tanh, ReLU, identity, and sigmoid), then the search space has 4 N ×N ! configurations. In our experiments, N = 12, which means there are approximately 10 15 models in our search space.

3.2. Training and Inference with ENAS:

Our controller network is a two-layer LSTM (Hochreiter & Schmidhuber, 1997) which samples decisions via softmax classifiers. The controller network samples these decisions in an autoregressive fashion: the decision in the previous step is fed as input embedding into the next step. At the first step, the controller network receives an empty embedding as input.

In ENAS, there are two sets of learnable parameters: the parameters of the controller LSTM, denoted by θ, and the shared parameters of the child models, denoted by ω. The training procedure of ENAS consists of two alternating phases. The first phase trains ω, the shared parameters of the child models, on a whole pass through the training data set, and the second phase trains θ, the parameters of the controller LSTM, for a fixed number of steps. Training the shared parameters ω of the child models. In this step, we fix the policy π(m; θ) and perform stochastic gradient descent (SGD) updates on ω to minimize the expected loss function E m∼π(m;θ) [L(m; ω) ]. Here, L(m; ω) is the standard cross-entropy loss, computed on a minibatch of training data, with a model m sampled from π(m; θ). The gradient is computed via the Monte Carlo estimate

∇ ω E m∼π(m;θ) [L(m; ω)] ≈ 1 M M i=1 ∇ ω L(m i , ω).

(1)

Note that Eqn 1 provides an unbiased estimate of the gradient

∇ ω E m∼π(m;θ) [L(m; ω)].

Therefore, according to Chapter 3.3.2 of Bottou (1991) , with an appropriate learning schedule, the SGD updates of ω converge almost surely. While convergence is guaranteed, these updates on ω have an inherently larger variance than SGD performed on a fixed model m.

Nevertheless, we find that M = 1 works just fine, i.e. we can update ω using the gradient from any single model m sampled from π(m; θ). As mentioned, we train ω for a whole pass through the training data set.

Training the parameters of θ of the controller LSTM. In this step, we fix ω and update the policy parameters θ, aiming to maximize the expected reward

E m∼π(m;θ) [R(m, ω)].

We employ the Adam optimizer (Kingma & Ba, 2015) , for which the gradient is computed using REINFORCE (Williams, 1992) , with a moving average baseline to reduce variance.

We compute the reward R(m, ω) on the validation set, rather than the training set to encourage ENAS, to select the model that generalizes well rather than the model that overfits the training set well. In our experiments with language modeling, the reward function is c/valid ppl, where the perplexity is computed on a minibatch, also sampled from the validation set. Later, in our experiments with image classification, the reward function is the classification accuracy on a minibatch of images sampled from the validation set.

Inference. We discuss how to derive novel architectures from a trained ENAS model. Following Zoph & Le (2017) , we sample several models from the trained policy π(m, θ).

For each sampled model, we compute its reward on a single minibatch sampled from the validation set. We then take the model with the highest reward to re-train from scratch.

3.3. Designing Convolutional Models:

We now discuss the search space for convolutional architectures. Recall that in the search space of the recurrent cell, the controller RNN samples two decisions at each decision block:

1) what previous node to connect to and 2) what activation function to use. In the search space for convolutional models, the controller RNN also samples two sets of decisions at each decision block: 1) what previous node to connect to and 2) what computation operation to use. The decision of what previous node to connect to allows the model to form skip connections (He et al., 2016; Zoph & Le, 2017) ; whereas the decision of what computation operation to use sets a particular layer into convolution or average pooling or max pooing. These decisions help construct a layer in the convolutional model.

To be even more flexible, instead of deciding if a particular layer is convolution or average pooling or max pooling, we change the search space to blend all these choices together. To achieve this, we treat all operations such as convolution, average pooling, and max pooling as channels, and allow the controller to select a mask over these channels. For example, in our experiments, we allow the controller to choose a mask over how many conv1x1, conv3x3, conv5x5, conv7x7, average pooling, max pooling operations to use (see Figure 3 ). Each operation at each layer in our network has its own convolutional parameters. During each data pass, only the parameters corresponding to the active channels are used and updated. The network of N layers, where each layer has 6 channels as described. Bottom: A block of the controller network, which consists of 6 binary masks, followed by the steps that sample skip connections.

For the skip connections, at layer k, up to k -1 mutually distinct previous indices are sampled. For example, at layer k = 5, let's suppose the controller samples {2, 4}, then the outputs of layer 2 and layer 4 will be concatenated and sent to layer 5 via skip connections.

In our experiments with the CIFAR-10 dataset, the training of the controller LSTM and the child models are also alternating. For each phase of training θ, we train it for 2000 steps with the Adam optimizer and REINFORCE. Meanwhile, for each phase of training the shared parameter ω, we train it for 450 minibatches, each has 100 images, using Nesterov Momentum (Nesterov, 1983) .

3.4. Designing Convolutional Cells:

An alternative to designing the entire convolutional network is to design smaller modules and then fit repeats of them together (Zoph et al., 2017; Zhong et al., 2017) . Figure 4 illustrates this approach, where the convolutional cell and reduction cell architectures are to be designed. We now discuss how to use ENAS to search for the architecture of these cells. Following Zoph et al. (2017) , we sample both our convolutional cell and our reduction cell using an RNN controller. Specifically, at each decision block, the controller RNN samples for two sets of decisions: 1) two previous nodes to be used as inputs to the current block and 2) two operations to respectively apply to the two sampled nodes. We allow the following 5 operations: identity, separable convolution with kernel size 3x3 and 5x5, and average pooling and max pooling with kernel size 3x3. Note that each cell receives two input nodes, indexed by node 1 and node 2, corresponding to the outputs of the two previous cells in the entire network. Figure 5 depicts an example run of our controller in the search space with 4 nodes. As with our other search spaces, each operation in each cell has its own parameters. During each data pass, only the relevant parameters are used and trained with their corresponding gradients.

Figure 5 : An example run of the controller for our convolutional cell search space. Left: the controller's outputs. Note that in our search space for convolutional cells, node 1 and node 2 respectively correspond to the output of the two previous layers, so the controller does not have to make any decisions for them. Right: the convolutional cell according to the controller's sample.

4. Experiments:

In the following, we will show our experimental results with ENAS to design recurrent cells on the Penn Treebank dataset and to design convolutional architectures on the CIFAR-10 dataset. We then present an ablation study which shows the role of ENAS in discovering novel architectures, as well as details regarding the efficiency of ENAS.

4.1. Language Model with Penn Treebank:

Dataset. We first apply ENAS to the task of language modeling, whose goal is predict next words in a text given a history of previous words, by fitting a probabilistic model over sentences. We use the Penn Treebank dataset (PTB) (Marcus et al., 1994) , a well-studied language modeling benchmark. In this experiment, we show that ENAS discovers a novel recurrent cell which, without extensive hyper-parameters tuning, can outperform models with the same or more parameters. Training details. Our controller is trained using Adam (Kingma & Ba, 2015), with a learning rate of 0.001. We use a tanh constant of 2.5 and a temperature of 5.0 for the sampling logits (Bello et al., 2017a) .

We also add the controller's sample entropy to the reward, with a weight of 10 -4 .

Additionally, we augment the simple transformations between the constructed recurrent cell's nodes with highway connections (Zilly et al., 2017) . For instance, instead of having h 2 = ReLU(h 1 • W (h) 2,1 ) as shown in the example from Section 3.1, we have

h 2 = c 2 ⊗ ReLU(h 1 • W (h) 2,1 ) + (1 -c 2 ) ⊗ h 1 , where c 2 = sigmoid(h 1 • W (c)

2,1 ). More details can be found in Appendix A. A novel RNN cell found in our search space is shown in Figure 6 .

The shared parameters of the child models ω are trained using SGD with a learning rate of 0.2, decayed by a factor of 0.9 after every 3 epochs starting at epoch 15, for a total of 150 epochs. During the architecture search process, following Melis et al. (2017) , we randomly reset the starting state with probability of 0.001. We also tie the model's word embeddings matrix with its softmax matrix (Inan et al., 2017) . When retraining the architecture recommended by the controller, however, we use variational dropout (Gal & Ghahramani, 2016) , an 2 regularization with weight decay of 10 -7 , and a state slowness regularization of 0.0005 (Merity et al., 2017) .

Results. Running on a single Nvidia GTX 1080Ti GPU, ENAS finds the recurrent cell in less than 10 hours. This cell is depicted in Figure 6 . Table 1 presents our results in comparison with other methods. The ENAS cell, with 24M parameters, outperforms the NAS cell and has a similar performance to the LSTM model that uses extensive hyperparameters tuning (Melis et al., 2017) , which we did not do.

Our ENAS cell has a few interesting properties. First, all non-linearities in the cell are either ReLU or tanh, even though the search space also has two other functions: identity and sigmoid. We suspect this cell is a local optimum, similar to the observations by Zoph & Le (2017) . When we randomly pick some nodes and switch the non-linearity into identity or sigmoid, the perplexity increases up to 8 points. When we randomly, swith some ReLU nodes into tanh or vice versa, the perplexity also increases, but only up to 3 points. 

Method:



4.2. Image Classification on CIFAR-10:

Dataset. The CIFAR-10 dataset (Krizhevsky, 2009) consists of 50,000 training images and 10,000 test images. We use the standard data pre-processing and augmentation techniques, i.e., subtracting the mean and dividing the standard deviation from each channel computed on the training images, centrally padding the training images to 40 × 40 and randomly cropping them back to 32 × 32, and randomly flipping them horizontally.

Search spaces. In Section 3.3, we present a search space for convolutional architectures in which the controller can make decisions over skip connections and the mask over the channels. To improve our results, we additionally explore two restricted versions of this search space: one where the controller only needs to make decisions of the mask over the channels and one where the controller only needs to make decisions of the skip connections. More details are available in Appendix B.

1. Searching for the masks over channels. We fix a pattern of skip connections and search for the masks at each branch and each layer in a 12-layer network. The pattern that we use is the dense pattern (Huang et al., 2016) . 2. Searching for skip connections. We force all the convolutions to have the filter size of 3 × 3, and only search for the skip connections.

3. Searching for convolutional and reduction cells. We search for both cells as discussed in Section 3.4.

Training details. The shared parameters ω are trained with Nesterov momentum (Nesterov, 1983) , where the learning rate follows the cosine schedule with l max = 0.05, l min = 0.001, T 0 = 10 and T mul = 2 (Loshchilov & Hutter, 2017) . Each architecture search is run for 10 + 20 + 40 + 80 + 160 = 310 epochs. The parameter ω is initialized from a scaled Gaussian as described in He et al. (2015) . We also apply an 2 weight decay of 10 -4 . The same settings are employed to train the architectures recommended by the controller.

The policy parameters θ are initialized uniformly in [-0.1, 0.1], and trained with the Adam optimizer at a learning rate of 10 -3 . We additionally utilize three techniques to prevent the premature convergence of REINFORCE. First, we apply a temperature τ = 5.0 and a tanh constant c = 2.5 to the controller's logits. Second, we add to the reward the entropy term of the controller's samples weighted by λ ent = 0.1, which discourages convergence (Williams & Peng, 1991) . Lastly, we enforce the sparsity in the skip connections by adding to the reward the Kullback-Leibler divergence between: 1) the skip connection probability between any two layers and 2) our chosen probability ρ = 0.4, which represents the prior belief of a skip connection being formed. The KL divergence term is weighted by λ kl = 0.5.

Tricks to stabilize and improve training. We find the following tricks crucial for achieving good performance with ENAS.

• Structure of Convolutional Layers. Each convolutional operation in our method is followed by a batch normalization (Ioffe & Szegedy, 2015) and then a ReLU layer. We find the alternate setting of batch norm-conv-ReLU (Zoph et al., 2017) to have worse results. • Stabilizing Stochastic Skip Connections. If a layer receives skip connections from multiple layers before it, then these layers' outputs are concatenated in their depth dimension, and then a convolution of filter size 1 × 1 (followed by a batch normalization layer and a ReLU layer) is performed to ensure the number of output channels is still equal to C. • Global Average Pooling. After the final convolutional layer, we average all the activations of each channel and then pass them to the Softmax layer. This trick was introduced by Lin et al. (2013) , with the purpose of reducing the number of parameters in the dense connection to the Softmax layer to avoid overfitting.

The last two tricks are extremely important, since the gradient updates of the shared parameters ω, as described in Eqn 1, have a very high variance. In fact, we find that without these last two tricks, the training of ENAS is very unstable. 2017), while taking 30x less time and using 800x less computing resource to discover.

Results.:

In the restricted search space over the masks, ENAS takes 11.6 hours to find a model that achieves the test error of 4.35%. The resulting model, depicted in Figure 7 -Top Left, almost always has 64 or 96 channels at each branch and each layer, indicating that the controller does not choose to activate all blocks. This is the desired behavior, as activating all channels would over-parametrize the model and result in overfitting. Moreover, the fact that the model found in a restricted search space has similar performance to the model found in the general search space indicates that ENAS can discover skip connection patterns that are comparable to the dense pattern, which is the state-of-the-art human-designed architecture on CIFAR-10 (Huang et al., 2016) .

In the restricted search space over skip connections, ENAS takes 12.4 hours to discover the pattern of skip connections depicted in Figure 7 that skip connections are formed much more densely at higher layers than at lower layers, where most connections are only between consecutive layers. The model has the test error of 5.04%, which is slightly worse than the one found in the restricted search space over masks. However, if we increase the number of output channels from 256 to 512, then the network achieves the test error of 3.87%.

In the search space over cells, ENAS takes 11.5 hours to discover the convolution cell and the reduction cell, which are shown in Figure 7 -Bottom. With the convolutional cell replicated for 6 times, ENAS achieves 3.54% test error, which is on par with the 3.41% error of NASNet-A (Zoph et al., 2017) . With CutOut (DeVries & Taylor, 2017), ENAS's error decreases to 2.89%, compared to 2.65% by NASNet-A. However, as discussed in Appendix B, our ENAS cells are trained for only 310 epochs, compared to 600 epochs by NAS.

4.3. The Importance and Efficiency of ENAS:

Sanity Check and Ablation Study. To understand the role of ENAS, we carry out two control experiments. In the first study, we uniformly at random pick a configuration of channels and skip connections and just train a model. As a result, about half of the channels and skip connections are selected, resulting in a model with 47.1M parameters and an error rate of 5.86%. This error rate is significantly worse than the models designed by ENASand has many more parameters. In the second study, we only train ω and do not update the controller. The effect is similar to dropout with a rate of 0.5 on both the channels and the skip connections. At convergence, the model has the error rate of 11.92%. On the validation set, the ensemble of 250 Monte Carlo configurations of the trained model could only reach 8.99% test error rate. We therefore conclude that the appropriate training of the ENAS controller is crucial for good performance. Efficiency of ENAS. Table 3 further compares of the time and resources needed for ENAS to discover good architectures on CIFAR-10 with other methods. In all of our experiments, ENAS takes less than 15 hours to train, whilst running on a single Nvidia GTX 1080Ti GPU. Compared to NAS by Zoph & Le (2017) ; Zoph et al. (2017) , our method achieves a 10X reduction of search time and 100X reduction of computational resources.

Method GPUs Time (days):

Macro NAS with Q-Learning (Baker et al., 2017a ) 10 8-10 Net Transformation (Cai et al., 2017) 5 3 SMASH (Brock et al., 2017) 1 1.5 1 Micro NAS with Q-Learning (Zhong et al., 2017) 32 3 NAS (Zoph & Le, 2017) 800 21-28 Micro NAS (Zoph et al., 2017 

5. Conclusion:

Neural Architecture Search (NAS) is an important advance that allows faster architecture design for neural networks. However, the computational expense of NAS prevents it from being widely adopted. In this paper, we presented ENAS, an alternative method to NAS, that requires three orders of magnitude less resources×time. The key insight of our method is to share parameters across child models during architecture search. This insight is implemented by having NAS search for a path within a larger model. We demonstrate empirically that the method works well on both CIFAR-10 and Penn Treebank datasets.

:

We took the authors' published code, ran it for a few epochs and interpolated the running time.

Figures and Tables:

Figure 2: An example of a recurrent cell in our search space with 4 computational nodes. Left: The computational DAG that corresponds to the recurrent cell. The red edges represent the activated path. Middle: The recurrent cell. Right: The outputs of the controller RNN that result in the cell in the middle and the DAG on the left. Note that nodes 3 and 4 are never sampled by the RNN, so their results are averaged and are treated as the cell's output.

These two phases are alternated during the training of ENAS. In our experiments with the Penn Treebank dataset, for each phase of training ω, we train ω for 450 steps with SGD, each on a batch of 64 examples, where gradients are computed using back-propagation through time, truncated to 35 time steps. Meanwhile, for each phase of training θ, we train it for 2000 steps with the Adam optimizer and REINFORCE. More details of their training are as follows.

Figure3: Our parameter sharing scheme between convolutional models. Top: The network of N layers, where each layer has 6 channels as described. Bottom: A block of the controller network, which consists of 6 binary masks, followed by the steps that sample skip connections.

Figure 4: The organization of the convolution cell and reduction cell to form a network.

Figure 6: The best RNN cell discovered by ENAS on the Penn Treebank dataset.

Figure 7: Top left: An architecture found by ENAS for CIFAR-10 for the search space over channels. Top right: An architecture found by ENAS for CIFAR-10 in the search space over skip connections. Bottom: the normal cell and the reduction cell found in the search space over cells.

Test perplexity on Penn Treebank of ENAS and other approaches. VD = Variational Dropout; WT = Weight Tying; MC = Monte Carlo sampling.

Table content: <table><tr><td>Parameters</td><td>Test</td></tr><tr><td>(million)</td><td>Perplexity</td></tr></table>

Table 2 summarizes the test errors of ENAS and other approaches. As can be seen from the table, ENAS successfully found several architectures that outperform other automatic model designing approaches with the same usage of computing resource. In particular, in the general search space, ENAS takes 15.6 hours to find a model that achieves 4.23% error rate on CIFAR-10. This model outperforms all but one model reported by Zoph & Le (

Table content: <table/>

-Top Right. This pattern has the property

Table content: <table><tr><td>Method</td><td>Depth</td><td colspan="2">Parameters Error (million) (%)</td></tr><tr><td>DenseNet-BC (Huang et al., 2016)</td><td>190</td><td>25.6</td><td>3.46</td></tr><tr><td>DenseNet + Shake-Shake 26 2x96d (Gastaldi, 2016)</td><td>26</td><td>26.2</td><td>2.86</td></tr><tr><td>DenseNet + CutOut (DeVries &amp; Taylor, 2017)</td><td>26</td><td>26.2</td><td>2.56</td></tr><tr><td>Budgeted Super Nets (Veniat &amp; Denoyer, 2017)</td><td>16</td><td>-</td><td>9.21</td></tr><tr><td>ConvFabrics Dense (Saxena &amp; Verbeek, 2016)</td><td>16</td><td>21.2</td><td>7.43</td></tr><tr><td>Macro NAS with Q-Learning (Baker et al., 2017a)</td><td>11</td><td>11.2</td><td>6.92</td></tr><tr><td>Net Transformation (Cai et al., 2017)</td><td>17</td><td>19.7</td><td>5.70</td></tr><tr><td>FractalNet (Larsson et al., 2017)</td><td>21</td><td>38.6</td><td>4.60</td></tr><tr><td>SMASH (Brock et al., 2017)</td><td>211</td><td>16.0</td><td>4.03</td></tr><tr><td>NAS (Zoph &amp; Le, 2017)</td><td>39</td><td>7.1</td><td>4.47</td></tr><tr><td>NAS + more filters (Zoph &amp; Le, 2017)</td><td>39</td><td>37.4</td><td>3.65</td></tr><tr><td>ENAS, general search space</td><td>18</td><td>34.9</td><td>4.23</td></tr><tr><td>ENAS, search for masks</td><td>12</td><td>12.6</td><td>4.35</td></tr><tr><td>ENAS, search for skip connections</td><td>15</td><td>14.1</td><td>5.04</td></tr><tr><td>ENAS, search for skip connections + 512 channels</td><td>15</td><td>38.0</td><td>3.87</td></tr><tr><td>MicroNAS with Q-Learning (Zhong et al., 2017)</td><td>24</td><td>-</td><td>3.60</td></tr><tr><td>NASNet-A (Zoph et al., 2017)</td><td>20</td><td>3.3</td><td>3.41</td></tr><tr><td>NASNet-A (Zoph et al., 2017) + CutOut</td><td>20</td><td>3.3</td><td>2.65</td></tr><tr><td>ENAS, search for cells</td><td>20</td><td>4.6</td><td>3.54</td></tr><tr><td>ENAS, search for cells + CutOut</td><td>20</td><td>4.6</td><td>2.89</td></tr></table>

Classification error rates of ENAS and other methods on CIFAR-10. In this table, the first block presents the state-of-the-art models, all of which are designed by human experts. The second block presents various approaches that design the entire network. ENAS outperforms all these methods but NAS, which requires much more computing resource and time. The last block presents techniques that design modular cells which are used to build a large model. ENAS outperforms MicroNAS, which uses 32 GPUs to search, and achieves similar performance with NASNet-A.

Table content: <table/>

Time and resources needed for different architecture search methods to find the good architectures for CIFAR-10.

Table content: <table/>

Peer review (iclr2018):
 Summary: The paper presents a method for learning certain aspects of a neural network architecture, specifically the number of output maps in certain connections and the existence of skip connections. The method is relatively efficient, since it searches in a space of similar architectures, and uses weights sharing between the tested models to avoid optimization of each model from scratch. Results are presented for image classification on Cifar 10 and for language modeling. Page 3: “for each channel, we only predict C/S binary masks” -> this seems to be a mistake. Probably “for each operation, we only predict C/S binary masks” is the right wording Page 4: Stabilizing Stochastic Skip Connections: it seems that the suggested configuration does not enable an identity path, which was found very beneficial in (He. et al., 2016). Identity path does not exist since layers are concatenated and go through 1*1 conv, which does not enable plain identity unless learned by the 1*1 conv. Page 5: - The last paragraph in section 4.2 is not clear to me. What does a compilation failure mean in this context and why does it occur? And: if each layer is connected to all its previous layers by skip connections, what remains to be learned w.r.t the model structure? Isn’t the pattern of skip connection the thing we would like to learn? - Some details of the policy LSTM network are also not clear to me: o How is the integer mask (output of the B channel steps) encoded? Using 1-hot encoding over 2^{C/S} output neurons? Or maybe C/S output neurons, used for sampling the C/S bits of the mask? this should be reported in some detail. o How is the mask converted to an input embedding for the next step? Is it by linear multiplication with a matrix? Something more complicated? And are there different matrices used/trained for each mask embedding (one for 1*1 conv, one for 3*3 conv, etc..)? o What is the motivation for using equation 5 for the sampling of skip connection flags? What is the motivation for averaging the winning anchors as the average embedding for the next stage (to let it ‘know’ who is connected to the previous?). Is anchor j also added to the average? o How is equation 5 normalized? That is: the probability is stated to be proportional to an exponent of an inner product, but it is not clear what the constant is and how sampling is done. Page 6: - Section 4.4: what is the fixed policy used for generating models in the stage of training the shared W parameters? (this is answered at page 7 Experiments: - The accuracy figures obtained are impressive, but I’m not convinced the ENAS learning is the important ingredient in obtaining them (rather than a very good baseline) - Specifically, in the Cifar -10 example it does not seem that the networks chooses the number of maps in a way which is diverse or different from layer to layer. Therefore we do not have any evidence that the LSTM controller has learnt any interesting rule regarding block type, or relation between block type width and layer index. All we see is that the model does not chose too many maps, thus avoid significant overfit. The relevant baseline here is a model with 64 or 96 maps on each block, each layer.Such a model is likely to do as well as the ENAS model, and can be obtained easily with slight parameter tuning of a single parameter. - Similarly, I’m not convinced the skip connection pattern found for Cifar-10 is superior to standard denseNet or Resnet pattern. The found configuration was not compared to these baselines. So again we do not have evidence showing the merit of keeping and tuning many parameters with the RINFORCE - The experiments with Penn Treebank are described with too less details: for example, what exactly is the task considered (in terms on input-output mapping), what is the dataset size, etc.. - Also, for the Penn treebank experiments no baseline is given, so it is not possible to understand if the structure learning here is beneficial. Comparison of the results to an architecture with all skip connections, and with a single skip connection per layer is required to estimate if useful structure is being learnt. Overall: - Pro: the method gives high accuracy results - Cons: o It is not clear if the ENAS search is responsible to the results, or just the strong baseline. The advantage of ENAS over plain hyper parameter choosing was not sufficiently established. o The controller was not presented in a clear enough manner. Many of its details stay obscure. o The method does not seem to be general. It seems to be limited to choosing a specific set of parameters in very specific scenario (scenario which enable parameter sharing between model. The conditions for this to happen seems to be rather strict, and where not elaborated). After revision: The controller is now better presented. However, the main points were not changed: - ENAS seems to be limited to a specific architecture and search space, in which probably the search is already exhausted. For example for the image processing network, it is determining the number of skip connections and structure of a single layer as a combination of several function types. We already know the answers to these search problems (denser skip connection pattern works better, more functions types in a layer in parallel do better, the number of maps should be adjusted to the complexity and data size to avoid overfit). ENAS does not reveal a new surprising architectures, and it seems that instead of searching in the large space it suggests, one can just tune a 1-2 parameters (for the image network, it is the number of maps in a layer). - Results comparing ENAS results to the simple baseline of just tuning 1-2 hyper parameters were not shown. I hence believe the strong empirical results of ENAS are a property of the search space (the architecture used) and not of the search algorithm.