Title: DARTS: DIFFERENTIABLE ARCHITECTURE SEARCH

Abstract: This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques. Our implementation has been made publicly available to facilitate further research on efficient architecture search algorithms.
Discovering state-of-the-art neural network architectures requires substantial effort of human experts. Recently, there has been a growing interest in developing algorithmic solutions to automate the manual process of architecture design. The automatically searched architectures have achieved highly competitive performance in tasks such as image classification (

:

The best existing architecture search algorithms are computationally demanding despite their remarkable performance. For example, obtaining a state-of-the-art architecture for CIFAR-10 and ImageNet required 2000 GPU days of reinforcement learning (RL) (Zoph et al., 2018) or 3150 GPU days of evolution (Real et al., 2018) . Several approaches for speeding up have been proposed, such as imposing a particular structure of the search space (Liu et al., 2018b; a) , weights or performance prediction for each individual architecture (Brock et al., 2018; Baker et al., 2018) and weight sharing/inheritance across multiple architectures (Elsken et al., 2017; Pham et al., 2018b; Cai et al., 2018; Bender et al., 2018) , but the fundamental challenge of scalability remains. An inherent cause of inefficiency for the dominant approaches, e.g. based on RL, evolution, MCTS (Negrinho & Gordon, 2017) , SMBO (Liu et al., 2018a) or Bayesian optimization (Kandasamy et al., 2018) , is the fact that architecture search is treated as a black-box optimization problem over a discrete domain, which leads to a large number of architecture evaluations required.

In this work, we approach the problem from a different angle, and propose a method for efficient architecture search called DARTS (Differentiable ARchiTecture Search). Instead of searching over a discrete set of candidate architectures, we relax the search space to be continuous, so that the architecture can be optimized with respect to its validation set performance by gradient descent. The data efficiency of gradient-based optimization, as opposed to inefficient black-box search, allows DARTS to achieve competitive performance with the state of the art using orders of magnitude less computation resources. It also outperforms another recent efficient architecture search method, ENAS (Pham et al., 2018b) . Notably, DARTS is simpler than many existing approaches as it does not involve controllers (Zoph & Le, 2017; Baker et al., 2017; Zoph et al., 2018; Pham et al., 2018b; Zhong et al., 2018) , hypernetworks (Brock et al., 2018) or performance predictors (Liu et al., 2018a ), yet it is generic enough handle both convolutional and recurrent architectures.

The idea of searching architectures within a continuous domain is not new (Saxena & Verbeek, 2016; Ahmed & Torresani, 2017; Veniat & Denoyer, 2017; Shin et al., 2018) , but there are several major distinctions. While prior works seek to fine-tune a specific aspect of an architecture, such as filter shapes or branching patterns in a convolutional network, DARTS is able to learn high-performance architecture building blocks with complex graph topologies within a rich search space. Moreover, DARTS is not restricted to any specific architecture family, and is applicable to both convolutional and recurrent networks.

In our experiments (Sect. 3) we show that DARTS is able to design a convolutional cell that achieves 2.76 ± 0.09% test error on CIFAR-10 for image classification using 3.3M parameters, which is competitive with the state-of-the-art result by regularized evolution (Real et al., 2018) obtained using three orders of magnitude more computation resources. The same convolutional cell also achieves 26.7% top-1 error when transferred to ImageNet (mobile setting), which is comparable to the best RL method (Zoph et al., 2018) . On the language modeling task, DARTS efficiently discovers a recurrent cell that achieves 55.7 test perplexity on Penn Treebank (PTB), outperforming both extensively tuned LSTM (Melis et al., 2018) and all the existing automatically searched cells based on NAS (Zoph & Le, 2017) and ENAS (Pham et al., 2018b) .

Our contributions can be summarized as follows:

• We introduce a novel algorithm for differentiable network architecture search based on bilevel optimization, which is applicable to both convolutional and recurrent architectures.

• Through extensive experiments on image classification and language modeling tasks we show that gradient-based architecture search achieves highly competitive results on CIFAR-10 and outperforms the state of the art on PTB. This is a very interesting result, considering that so far the best architecture search methods used non-differentiable search techniques, e.g. based on RL (Zoph et al., 2018) or evolution (Real et al., 2018; Liu et al., 2018b) .

• We achieve remarkable efficiency improvement (reducing the cost of architecture discovery to a few GPU days), which we attribute to the use of gradient-based optimization as opposed to non-differentiable search techniques.

• We show that the architectures learned by DARTS on CIFAR-10 and PTB are transferable to ImageNet and WikiText-2, respectively.

The implementation of DARTS is available at https://github.com/quark0/darts

2. DIFFERENTIABLE ARCHITECTURE SEARCH:

We describe our search space in general form in Sect. 2.1, where the computation procedure for an architecture (or a cell in it) is represented as a directed acyclic graph. We then introduce a simple continuous relaxation scheme for our search space which leads to a differentiable learning objective for the joint optimization of the architecture and its weights (Sect. 2.2). Finally, we propose an approximation technique to make the algorithm computationally feasible and efficient (Sect. 2.3).

2.1. SEARCH SPACE:

Following Zoph et al. (2018) ; Real et al. (2018) ; Liu et al. (2018a; b) , we search for a computation cell as the building block of the final architecture. The learned cell could either be stacked to form a convolutional network or recursively connected to form a recurrent network.

A cell is a directed acyclic graph consisting of an ordered sequence of N nodes. Each node x (i) is a latent representation (e.g. a feature map in convolutional networks) and each directed edge (i, j) is associated with some operation o (i,j) that transforms x (i) . We assume the cell to have two input nodes and a single output node. For convolutional cells, the input nodes are defined as the cell outputs in the previous two layers (Zoph et al., 2018) . For recurrent cells, these are defined as the input at the current step and the state carried from the previous step. The output of the cell is obtained by applying a reduction operation (e.g. concatenation) to all the intermediate nodes.

Each intermediate node is computed based on all of its predecessors: A special zero operation is also included to indicate a lack of connection between two nodes. The task of learning the cell therefore reduces to learning the operations on its edges.

x (j) = i<j o (i,j) (x (i) ) (1)

2.2. CONTINUOUS RELAXATION AND OPTIMIZATION:

Let O be a set of candidate operations (e.g., convolution, max pooling, zero) where each operation represents some function o(•) to be applied to x (i) . To make the search space continuous, we relax the categorical choice of a particular operation to a softmax over all possible operations:

EQUATION

where the operation mixing weights for a pair of nodes (i, j) are parameterized by a vector α (i,j) of dimension |O|. The task of architecture search then reduces to learning a set of continuous variables α = α (i,j) , as illustrated in Fig. 1 . At the end of search, a discrete architecture can be obtained by replacing each mixed operation ō(i,j) with the most likely operation, i.e., o (i,j) = argmax o∈O α (i,j) o . In the following, we refer to α as the (encoding of the) architecture.

After relaxation, our goal is to jointly learn the architecture α and the weights w within all the mixed operations (e.g. weights of the convolution filters). Analogous to architecture search using RL (Zoph & Le, 2017; Zoph et al., 2018; Pham et al., 2018b) or evolution (Liu et al., 2018b; Real et al., 2018) where the validation set performance is treated as the reward or fitness, DARTS aims to optimize the validation loss, but using gradient descent.

Denote by L train and L val the training and the validation loss, respectively. Both losses are determined not only by the architecture α, but also the weights w in the network. The goal for architecture search is to find α * that minimizes the validation loss L val (w * , α * ), where the weights w * associated with the architecture are obtained by minimizing the training loss w * = argmin w L train (w, α * ).

This implies a bilevel optimization problem (Anandalingam & Friesz, 1992; Colson et al., 2007) with α as the upper-level variable and w as the lower-level variable:

EQUATION

The nested formulation also arises in gradient-based hyperparameter optimization (Maclaurin et al., 2015; Pedregosa, 2016; Franceschi et al., 2018) , which is related in a sense that the architecture α could be viewed as a special type of hyperparameter, although its dimension is substantially higher than scalar-valued hyperparameters such as the learning rate, and it is harder to optimize.

Algorithm 1: DARTS -Differentiable Architecture Search Create a mixed operation ō(i,j) parametrized by α (i,j) for each edge (i, j) while not converged do 1. Update architecture α by descending ∇ α L val (w -ξ∇ w L train (w, α), α) (ξ = 0 if using first-order approximation) 2. Update weights w by descending ∇ w L train (w, α) Derive the final architecture based on the learned α.

2.3. APPROXIMATE ARCHITECTURE GRADIENT:

Evaluating the architecture gradient exactly can be prohibitive due to the expensive inner optimization. We therefore propose a simple approximation scheme as follows:

EQUATION

where w denotes the current weights maintained by the algorithm, and ξ is the learning rate for a step of inner optimization. The idea is to approximate w * (α) by adapting w using only a single training step, without solving the inner optimization (equation 4) completely by training until convergence. Related techniques have been used in meta-learning for model transfer (Finn et al., 2017) , gradientbased hyperparameter tuning (Luketina et al., 2016) and unrolled generative adversarial networks (Metz et al., 2017) . Note equation 6 will reduce to ∇ α L val (w, α) if w is already a local optimum for the inner optimization and thus ∇ w L train (w, α) = 0.

The iterative procedure is outlined in Alg. 1. While we are not currently aware of the convergence guarantees for our optimization algorithm, in practice it is able to reach a fixed point with a suitable choice of ξ1 . We also note that when momentum is enabled for weight optimisation, the one-step unrolled learning objective in equation 6 is modified accordingly and all of our analysis still applies.

Applying chain rule to the approximate architecture gradient (equation 6) yields

EQUATION

where w = w -ξ∇ w L train (w, α) denotes the weights for a one-step forward model. The expression above contains an expensive matrix-vector product in its second term. Fortunately, the complexity can be substantially reduced using the finite difference approximation. Let be a small scalar2 and w ± = w ± ∇ w L val (w , α). Then:

∇ 2 α,w L train (w, α)∇ w L val (w , α) ≈ ∇ α L train (w + , α) -∇ α L train (w -, α) 2 (8)

Evaluating the finite difference requires only two forward passes for the weights and two backward passes for α, and the complexity is reduced from

O(|α||w|) to O(|α| + |w|).

First-order Approximation When ξ = 0, the second-order derivative in equation 7 will disappear. In this case, the architecture gradient is given by ∇ α L val (w, α), corresponding to the simple heuristic of optimizing the validation loss by assuming the current w is the same as w * (α). This leads to some speed-up but empirically worse performance, according to our experimental results in Table 1 and Table 2 . In the following, we refer to the case of ξ = 0 as the first-order approximation, and refer to the gradient formulation with ξ > 0 as the second-order approximation.

2.4. DERIVING DISCRETE ARCHITECTURES:

To form each node in the discrete architecture, we retain the top-k strongest operations (from distinct nodes) among all non-zero candidate operations collected from all the previous nodes. The strength of an operation is defined as

exp(α (i,j) o ) o ∈O exp(α (i,j) o )

. To make our derived architecture comparable with ) , w (0) ) = (2, -2). The analytical solution for the corresponding bilevel optimization problem is (α * , w * ) = (1, 1), which is highlighted in the red circle. The dashed red line indicates the feasible set where constraint equation 4 is satisfied exactly (namely, weights in w are optimal for the given architecture α). The example shows that a suitable choice of ξ helps to converge to a better local optimum.

those in the existing works, we use k = 2 for convolutional cells (Zoph et al., 2018; Liu et al., 2018a; Real et al., 2018) and k = 1 for recurrent cells (Pham et al., 2018b) .

The zero operations are excluded in the above for two reasons. First, we need exactly k non-zero incoming edges per node for fair comparison with the existing models. Second, the strength of the zero operations is underdetermined, as increasing the logits of zero operations only affects the scale of the resulting node representations, and does not affect the final classification outcome due to the presence of batch normalization (Ioffe & Szegedy, 2015) .

3. EXPERIMENTS AND RESULTS:

Our experiments on CIFAR-10 and PTB consist of two stages, architecture search (Sect. 3.1) and architecture evaluation (Sect. 3.2). In the first stage, we search for the cell architectures using DARTS, and determine the best cells based on their validation performance. In the second stage, we use these cells to construct larger architectures, which we train from scratch and report their performance on the test set. We also investigate the transferability of the best cells learned on CIFAR-10 and PTB by evaluating them on ImageNet and WikiText-2 (WT2) respectively.

3.1. ARCHITECTURE SEARCH:



3.1.1. SEARCHING FOR CONVOLUTIONAL CELLS ON CIFAR-10:

We include the following operations in O: 3 × 3 and 5 × 5 separable convolutions, 3 × 3 and 5 × 5 dilated separable convolutions, 3 × 3 max pooling, 3 × 3 average pooling, identity, and zero. All operations are of stride one (if applicable) and the convolved feature maps are padded to preserve their spatial resolution. We use the ReLU-Conv-BN order for convolutional operations, and each separable convolution is always applied twice (Zoph et al., 2018; Real et al., 2018; Liu et al., 2018a) .

Our convolutional cell consists of N = 7 nodes, among which the output node is defined as the depthwise concatenation of all the intermediate nodes (input nodes excluded). The rest of the setup follows Zoph et al. (2018) ; Liu et al. (2018a) ; Real et al. (2018) , where a network is then formed by stacking multiple cells together. The first and second nodes of cell k are set equal to the outputs of cell k -2 and cell k -1, respectively, and 1 × 1 convolutions are inserted as necessary. Cells located at the 1/3 and 2/3 of the total depth of the network are reduction cells, in which all the operations adjacent to the input nodes are of stride two. The architecture encoding therefore is (α normal , α reduce ), where α normal is shared by all the normal cells and α reduce is shared by all the reduction cells.

Detailed experimental setup for this section can be found in Sect. A.1.1.

3.1.2. SEARCHING FOR RECURRENT CELLS ON PENN TREEBANK:

Our set of available operations includes linear transformations followed by one of tanh, relu, sigmoid activations, as well as the identity mapping and the zero operation. The choice of these candidate operations follows Zoph & Le (2017); Pham et al. (2018b) .

Our recurrent cell consists of N = 12 nodes. The very first intermediate node is obtained by linearly transforming the two input nodes, adding up the results and then passing through a tanh activation Figure 3 : Search progress of DARTS for convolutional cells on CIFAR-10 and recurrent cells on Penn Treebank. We keep track of the most recent architectures over time. Each architecture snapshot is re-trained from scratch using the training set (for 100 epochs on CIFAR-10 and for 300 epochs on PTB) and then evaluated on the validation set. For each task, we repeat the experiments for 4 times with different random seeds, and report the median and the best (per run) validation performance of the architectures over time. As references, we also report the results (under the same evaluation setup; with comparable number of parameters) of the best existing cells discovered using RL or evolution, including NASNet-A (Zoph et al., 2018 ) (2000 GPU days) , AmoebaNet-A (3150 GPU days) (Real et al., 2018) and ENAS (0.5 GPU day) (Pham et al., 2018b) .

function, as done in the ENAS cell (Pham et al., 2018b) . The rest of the cell is learned. Other settings are similar to ENAS, where each operation is enhanced with a highway bypass (Zilly et al., 2016) and the cell output is defined as the average of all the intermediate nodes. As in ENAS, we enable batch normalization in each node to prevent gradient explosion during architecture search, and disable it during architecture evaluation. Our recurrent network consists of only a single cell, i.e. we do not assume any repetitive patterns within the recurrent architecture.

Detailed experimental setup for this section can be found in Sect. A.1.2. 

3.2. ARCHITECTURE EVALUATION:

To determine the architecture for final evaluation, we run DARTS four times with different random seeds and pick the best cell based on its validation performance obtained by training from scratch for a short period (100 epochs on CIFAR-10 and 300 epochs on PTB). This is particularly important for recurrent cells, as the optimization outcomes can be initialization-sensitive (Fig. 3 ).

To evaluate the selected architecture, we randomly initialize its weights (weights learned during the search process are discarded), train it from scratch, and report its performance on the test set. We note the test set is never used for architecture search or architecture selection.

Detailed experimental setup for architecture evaluation on CIFAR-10 and PTB can be found in Sect. A.2.1 and Sect. A.2.2, respectively. Besides CIFAR-10 and PTB, we further investigated the transferability of our best convolutional cell (searched on CIFAR-10) and recurrent cell (searched on PTB) by evaluating them on ImageNet (mobile setting) and WikiText-2, respectively. More details of the transfer learning experiments can be found in Sect. A.2.3 and Sect. A.2.4. (Zhong et al., 2018) 3.54 39.8 96 8 RL AmoebaNet-A (Real et al., 2018) 3.34 ± 0.06 3.2 3150 19 evolution AmoebaNet-A + cutout (Real et al., 2018) † 3.12 3.1 3150 19 evolution AmoebaNet-B + cutout (Real et al., 2018) 2.55 ± 0.05 2.8 3150 19 evolution Hierarchical evolution (Liu et al., 2018b) 3.75 ± 0.12 15.7 300 6 evolution PNAS (Liu et al., 2018a) 3.41 ± 0.09 3.2 225 8 SMBO ENAS + cutout (Pham et al., 2018b) 2.89 4.6 0.5 6 RL ENAS + cutout (Pham et al., * Obtained by repeating ENAS for 8 times using the code publicly released by the authors. The cell for final evaluation is chosen according to the same selection protocol as for DARTS. † Obtained by training the corresponding architectures using our setup. ‡ Best architecture among 24 samples according to the validation error after 100 training epochs. (Merity et al., 2018) 60.7 58.8 24 --manual LSTM + skip connections (Melis et al., 2018) 60.9 58.3 24 --manual LSTM + 15 softmax experts (Yang et al., 2018) (Zoph et al., 2018) 27.5 9.0 4.9 558 2000 RL AmoebaNet-A (Real et al., 2018) 25.5 8.0 5.1 555 3150 evolution AmoebaNet-B (Real et al., 2018) 26.0 8.5 5.3 555 3150 evolution AmoebaNet-C (Real et al., 2018) 24.3 7.6 6.4 570 3150 evolution PNAS (Liu et al., 2018a) 25 

3.3. RESULTS ANALYSIS:

The CIFAR-10 results for convolutional architectures are presented in Table 1 . Notably, DARTS achieved comparable results with the state of the art (Zoph et al., 2018; Real et al., 2018) while using three orders of magnitude less computation resources (i.e. 1.5 or 4 GPU days vs 2000 GPU days for NASNet and 3150 GPU days for AmoebaNet). Moreover, with slightly longer search time, DARTS outperformed ENAS (Pham et al., 2018b ) by discovering cells with comparable error rates but less parameters. The longer search time is due to the fact that we have repeated the search process four times for cell selection. This practice is less important for convolutional cells however, because the performance of discovered architectures does not strongly depend on initialization (Fig. 3 ).

Alternative Optimization Strategies:

To better understand the necessity of bilevel optimization, we investigated a simplistic search strategy, where α and w are jointly optimized over the union of the training and validation sets using coordinate descent. The resulting best convolutional cell (out of 4 runs) yielded 4.16 ± 0.16% test error using 3.1M parameters, which is worse than random search.

In the second experiment, we optimized α simultaneously with w (without alteration) using SGD, again over all the data available (training + validation). The resulting best cell yielded 3.56 ± 0.10% test error using 3.0M parameters. We hypothesize that these heuristics would cause α (analogous to hyperparameters) to overfit the training data, leading to poor generalization. Note that α is not directly optimized on the training set in DARTS.

Table 2 presents the results for recurrent architectures on PTB, where a cell discovered by DARTS achieved the test perplexity of 55.7. This is on par with the state-of-the-art model enhanced by a mixture of softmaxes (Yang et al., 2018) , and better than all the rest of the architectures that are either manually or automatically discovered. Note that our automatically searched cell outperforms the extensively tuned LSTM (Melis et al., 2018) , demonstrating the importance of architecture search in addition to hyperparameter search. In terms of efficiency, the overall cost (4 runs in total) is within 1 GPU day, which is comparable to ENAS and significantly faster than NAS (Zoph & Le, 2017) .

It is also interesting to note that random search is competitive for both convolutional and recurrent models, which reflects the importance of the search space design. Nevertheless, with comparable or less search cost, DARTS is able to significantly improve upon random search in both cases (2.76 ± 0.09 vs 3.29 ± 0.15 on CIFAR-10; 55.7 vs 59.4 on PTB).

Results in Table 3 show that the cell learned on CIFAR-10 is indeed transferable to ImageNet. It is worth noticing that DARTS achieves competitive performance with the state-of-the-art RL method (Zoph et al., 2018) while using three orders of magnitude less computation resources.

Table 4 shows that the cell identified by DARTS transfers to WT2 better than ENAS, although the overall results are less strong than those presented in Table 2 for PTB. The weaker transferability between PTB and WT2 (as compared to that between CIFAR-10 and ImageNet) could be explained by the relatively small size of the source dataset (PTB) for architecture search. The issue of transferability could potentially be circumvented by directly optimizing the architecture on the task of interest. LSTM + augmented loss (Inan et al., 2017) 91.5 87.0 28 -manual LSTM + continuous cache pointer (Grave et al., 2016) -68.9 --manual LSTM (Merity et al., 2018) 69.1 66.0 33 -manual LSTM + skip connections (Melis et al., 2018) 69.1 65.9 24 -manual LSTM + 15 softmax experts (Yang et al., 2018) 66 

4. CONCLUSION:

We presented DARTS, a simple yet efficient architecture search algorithm for both convolutional and recurrent networks. By searching in a continuous space, DARTS is able to match or outperform the state-of-the-art non-differentiable architecture search methods on image classification and language modeling tasks with remarkable efficiency improvement by several orders of magnitude.

There are many interesting directions to improve DARTS further. For example, the current method may suffer from discrepancies between the continuous architecture encoding and the derived discrete architecture. This could be alleviated, e.g., by annealing the softmax temperature (with a suitable schedule) to enforce one-hot selection. It would also be interesting to explore performance-aware architecture derivation schemes based on the one-shot model learned during the search process.

A simple working strategy is to set ξ equal to the learning rate for w's optimizer.

We found = 0.01/ ∇ w L val (w , α) 2 to be sufficiently accurate in all of our experiments.

All of our experiments were performed using NVIDIA GTX 1080Ti GPUs.

We did not conduct extensive hyperparameter tuning.

Figures and Tables:

Figure 1: An overview of DARTS: (a) Operations on the edges are initially unknown. (b) Continuous relaxation of the search space by placing a mixture of candidate operations on each edge. (c) Joint optimization of the mixing probabilities and the network weights by solving a bilevel optimization problem. (d) Inducing the final architecture from the learned mixing probabilities.

Figure2: Learning dynamics of our iterative algorithm when L val (w, α) = αw -2α + 1 and L train (w, α) = w 2 -2αw+α 2 , starting from (α (0) , w (0) ) = (2, -2). The analytical solution for the corresponding bilevel optimization problem is (α * , w * ) = (1, 1), which is highlighted in the red circle. The dashed red line indicates the feasible set where constraint equation 4 is satisfied exactly (namely, weights in w are optimal for the given architecture α). The example shows that a suitable choice of ξ helps to converge to a better local optimum.

Figure 4: Normal cell learned on CIFAR-10.

Comparison with state-of-the-art image classifiers on CIFAR-10 (lower error rate is better). Note the search cost for DARTS does not include the selection cost (1 GPU day) or the final evaluation cost by training the selected architecture from scratch (1.5 GPU days).

Table content: <table><tr><td>Architecture</td><td colspan="4">Test Error Params Search Cost #ops (%) (M) (GPU days)</td><td>Search Method</td></tr><tr><td>DenseNet-BC (Huang et al., 2017)</td><td>3.46</td><td>25.6</td><td>-</td><td>-</td><td>manual</td></tr><tr><td>NASNet-A + cutout (Zoph et al., 2018)</td><td>2.65</td><td>3.3</td><td>2000</td><td>13</td><td>RL</td></tr><tr><td>NASNet-A + cutout (Zoph et al., 2018)  †</td><td>2.83</td><td>3.1</td><td>2000</td><td>13</td><td>RL</td></tr><tr><td>BlockQNN</td><td/><td/><td/><td/><td/></tr></table>

Comparison with state-of-the-art language models on PTB (lower perplexity is better). Note the search cost for DARTS does not include the selection cost (1 GPU day) or the final evaluation cost by training the selected architecture from scratch (3 GPU days).

Table content: <table><tr><td>Architecture</td><td colspan="2">Perplexity Params valid test (M)</td><td>Search Cost (GPU days)</td><td>#ops</td><td>Search Method</td></tr><tr><td>Variational RHN (Zilly et al., 2016)</td><td>67.9 65.4</td><td>23</td><td>-</td><td>-</td><td>manual</td></tr><tr><td>LSTM</td><td/><td/><td/><td/><td/></tr></table>

Comparison with state-of-the-art image classifiers on ImageNet in the mobile setting.

Table content: <table><tr><td>Architecture</td><td colspan="5">Test Error (%) Params +× Search Cost top-1 top-5 (M) (M) (GPU days)</td><td>Search Method</td></tr><tr><td>Inception-v1 (Szegedy et al., 2015)</td><td>30.2</td><td>10.1</td><td>6.6</td><td>1448</td><td>-</td><td>manual</td></tr><tr><td>MobileNet (Howard et al., 2017)</td><td>29.4</td><td>10.5</td><td>4.2</td><td>569</td><td>-</td><td>manual</td></tr><tr><td colspan="2">ShuffleNet 2× (g = 3) (Zhang et al., 2017) 26.3</td><td>-</td><td>∼5</td><td>524</td><td>-</td><td>manual</td></tr><tr><td>NASNet-A (Zoph et al., 2018)</td><td>26.0</td><td>8.4</td><td>5.3</td><td>564</td><td>2000</td><td>RL</td></tr><tr><td>NASNet-B (Zoph et al., 2018)</td><td>27.2</td><td>8.7</td><td>5.3</td><td>488</td><td>2000</td><td>RL</td></tr><tr><td>NASNet-C</td><td/><td/><td/><td/><td/><td/></tr></table>

Comparison with state-of-the-art language models on WT2.

Table content: <table><tr><td>Architecture</td><td>Perplexity Params Search Cost valid test (M) (GPU days)</td><td>Search Method</td></tr></table>

Obtained by training the corresponding architecture using our setup.

Table content: <table><tr><td/><td>.0 63.3</td><td>33</td><td>-</td><td>manual</td></tr><tr><td>ENAS (Pham et al., 2018b)  † (searched on PTB)</td><td>72.4 70.4</td><td>33</td><td>0.5</td><td>RL</td></tr><tr><td>DARTS (searched on PTB)</td><td>71.2 69.6</td><td>33</td><td>1</td><td>gradient-based</td></tr><tr><td>†</td><td/><td/><td/><td/></tr></table>

Peer review (iclr2019):
 This paper proposes a novel way to formulate neural architecture search as a differentiable problem. It uses the idea of weight sharing introduced in previous papers (convolutional neural fabrics, ENAS, and Bender et al*s one shot model) and combines this with a relaxation of discrete choices between k operators into k continuous weights. Then, it uses methods based on hyperparameter gradient search methods to optimize in this space and in the end removes the relaxation by dropping weak connections and selecting the single choice of the k options with the highest weight. This leads to an efficient solution for architecture search. Overall, this is a very interesting paper that has already created quite a buzz due to the simplicity of the methods and the strong results. It is a huge plus that there is code with the paper! This will dramatically increase the paper*s impact. In my first read through, I thought this might be a candidate for an award paper, but the more time I spent with it the more issues I found. I still think the paper should be accepted, but I do have several points of criticism / questions I detail below, and to which I would appreciate a response. Some criticisms / questions: 1. The last step of how to move from the one-shot model to a single model is in a sense the most interesting aspect of this work, but also the one that leaves the most questions open: Why does this work? Are there cases where we lose arbitrarily badly by rounding the solution to the closest discrete value or is the performance loss bounded? How would other ways of moving from the relaxation to a discrete choice work? I don*t expect the paper to answer all of these questions, but it would be useful if the authors acknowledge that this is a critical part of the work that deserves further study. Any insights from other approaches the authors may have tried before the mechanism in Section 2.4 would also be useful. 2. The related work is missing several papers, namely the entire category of work on using network morphisms to speed up the optimization process, Bender et al*s one shot model, and several early papers on neural architecture search (work on NAS did not only start in 2017 but goes back to work in the 1990s on neuroevolution that is very similar to the evolution approach by Real). This is a useful survey useful for further references: https://arxiv.org/abs/1808.05377 3. I find a few of the claims to be a bit too strong. In the introduction, the paper claims to outperform ENAS, but really the paper doesn*t give a head-to-head comparison. In the experiments, ENAS is faster and gives slightly worse results. The authors state explicitly that their method is slower because they run it 4 times and pick the best result. One could obviously also do that with ENAS, and since ENAS is 8 times faster one could even run it 8 times! This is unfair and should be fixed. I don*t really care even if it turns out that ENAS performs a bit better with the same budget, but comparisons should be fair and on even ground in order to help our science advance -- something that is far too often ignored in the ML literature in order to obtain a table with bold numbers in one*s own row. Likewise, why is ENAS missing in the Figure 3 plots for CIFAR, and why is its performance not plotted over time like that of DARTS? 4. The paper is not really forthcoming about clearly stating the time required to obtain the results: - On CIFAR, there are 4 DARTS run of 1 day each - Then, the result of each of these is evaluated for 100 epochs (which is only stated in the caption of Figure 3) to pick the best. Each of these validation runs takes 4 hours (which, again, one has to be inferred from the fact that random search can do 24 such evaluations in 4 GPU days), so this step takes another 16 GPU hours. - Then, one needs to train the final network for 600 epochs; this is a larger network, so this should take another 2-3 GPU days. So, overall, to obtain the result on CIFAR-10 requires about one GPU week. That*s still cheap, but it*s a different story than 1 day. Likewise, DARTS is *not* able to obtain 55.7 perplexity on PTB in 6 hours with 4 GPUs; again, there is the selection step (probably another 4*6 hours?) and I think training the final model takes about 2 GPU days. These numbers should be stated prominently next to the stated "search times" to not mislead the reader. 5. One big question I have is where the hyperparameters come from, for both the training pipeline and the final evaluation pipeline (which actually differ a lot!). For example, here are the hyperparameters for CIFAR, in this format: training pipeline value -> final evaluation pipeline value: #cells: 8 -> 20 batch size: 64 -> 96 initial channels: 16 -> 36 #epochs: 50 -> 600 droppath: no -> yes (with probability 0.2) auxiliary head: no -> yes (with weight 0.4) BatchNorm: enabled (no learnable parameters) -> enabled The situation is similar for PTB: embedding size: 300 -> 850 hidden units per RNN layer: 300 -> 850 #epochs: 500 -> 8000 batch size: 256 (SGD) -> 64 (ASGD), sped up by starting with SGD weight decay: 5e-7 -> 8e-7 BatchNorm: enabled (no learnable parameters) -> disabled The fact that there are so many differences in the pipelines is disconcerting, since it looks like a lot of manual work is required to get these right. Now you need to tune hyperparameters for both the training and the final evaluation pipeline? If you have to tune them for the final evaluation pipeline, then you can*t capitalize at all on the fact that DARTS is fast, since hyperparameter optimization on the full final evaluation pipeline will be order of magnitudes more expensive than running DARTS. 6. How was the final evaluation pipeline chosen? Before running DARTS the first time, or was it chosen to be tuned for architectures found by DARTS? 7. A question about how the best of 4 DARTS runs is selected, and how the best of the 24 random samples in random search is evaluated: is this based on 100 epochs using the *training* procedure or the *final evaluation* procedure? Seeing how different the hyperparameters are above, this should be stated. 8. A few questions to the authors related to the above: how did you choose the hyperparameters of DARTS? The DARTS learning rate for PTB is 10 times higher than for CIFAR-10, and the momentum also differs a lot (0.9 vs. 0.5). Did you ever consider different hyperparameters for DARTS? If so, how did you decide on the ones used? Is it sensitive to the choice of hyperparameters? In the author response period, could you please report the (1) result of running DARTS on PTB using the same DARTS hyperparameters as used for CIFAR-10 (learning rate 3*e-4 and momentum (0.5,0.999)) and (2) result of running DARTS on CIFAR-10 using the same DARTS hyperparameters as used for PTB (learning rate 3*e-3 and momentum (0.9,0.999))? 9. DARTS is being critizized in https://openreview.net/pdf?id=rylqooRqK7#page=10&zoom=180,-16,84 I am wondering whether the authors have a reply to this. The algorithm for solving the relaxed problem is also not mathematically derived from the optimization problem to be solved (equations 3,4), but it is more a heuristic. A derivation, or at least a clearer motivation for the algorithm would be useful. 10. Further comments: - Equation 1: This looks like a typo, shouldn*t this be x(j) = .sum_{i<j} o(i,j) x(i) ? Even if the authors wanted to use the non-intuitive way of edges going from j to i, then o(i,j) should still be o(j,i). - Just above Equation 5: "the the" - Equation 5: I would have found it more intuitive had .alpha_{k-1} already just been a generic .alpha here. - It would be nice if the authors gave the explicit equations for the extension with momentum in the appendix for completeness. - The authors should include citations for techniques such as batch normalization, Adam, and cosine annealing. Despite these issues (which I hope the authors will address in the author response and the final version), as stated above, I*m arguing for accepting the paper, due to the simplicity of the method combined with its very promising results and the direct availability of code.