Understanding and Robustifying Differentiable Architecture Search
Authors: Arber Zela
Thomas Elsken
Tonmoy Saikia
Yassine Marrakchi
Thomas Brox
Frank Hutter
Venue:  (2020)
Keywords: Neural Architecture Search
AutoML
AutoDL
Deep Learning
Computer Vision

Abstract: Differentiable Architecture Search (DARTS) has attracted a lot of attention due to its simplicity and small search costs achieved by a continuous relaxation and an approximation of the resulting bi-level optimization problem.  However, DARTS does not work robustly for new problems: we identify a wide range of search spaces for which DARTS yields degenerate architectures with very poor test performance. We study this failure mode and show that, while DARTS successfully minimizes validation loss, the found solutions generalize poorly when they coincide with high validation loss curvature in the  architecture space. We show that by adding one of various types of regularization we can robustify DARTS to find solutions with less curvature and better generalization properties. Based on these observations, we propose several simple variations of DARTS that perform substantially more robustly in practice.  Our observations are robust across five search spaces on three image classification tasks and also hold for the very different domains of disparity estimation (a dense regression task) and language modelling.

Published as a conference paper at ICLR 2020
UNDERSTANDING AND ROBUSTIFYING DIFFERENTIABLE ARCHITECTURE SEARCH
Arber Zela', Thomas Elsken Tonmoy Saikia', Yassine Marrakchi',
Thomas Brox & Frank Hutter
"Department of Computer Science, University of Freiburg [zelaa, saikiat, marrakch, brox, fh]ecs. uni- burg...ur.... .aurr.dee de "Bosch Center for Artificial Intelligence
Thomas Elsken@de bosch. com
ABSTRACT
Differentiable Architecture Search (DARTS) has attracted a lot of attention due to its simplicity and small search costs achieved by continuous relaxation and an approximation of the resulting bi-level optimization problem. However, DARTS does not work robustly for problems: we identify a wide range of search spaces for which DARTS yields degenerate architectures with very poor test per- formance. We study this failure mode and show that, while DARTS successfully minimizes validation loss, the found solutions generalize poorly when they coin- cide with high validation loss curvature in the architecture space. We show that by adding one of various types of regularization we can robustify DARTS to find solutions with less curvature and better generalization properties. Based on these observations, we propose several simple variations of DARTS that perform sub- stantially more robustly practice. Our observations are robust across five search spaces on three image classification tasks and also hold for the very different do- mains of disparity estimation (a dense regression task) and language modelling.
INTRODUCTION
Neural Architecture Search (NAS), the process of automatically designing neural network archi- tectures. has recently attracted attention by achieving oftee-of-the-att performance on a variety of tasks (Zoph & Le] [017]] Real et al., 2019). Differentiable architecture search (DARTS) (Liu et al.) 2019) significatyy improved the efficieccy of NAS over prior work, reducing costs to the same or- der of magnitude training single neural network. This expanded the scope of NAS substantially, allowing it to also be applied on more expensive problems, such as semantic segmentation (Chenxi al. 2019) or disparity estimation (Saikia al. 2019).
However, several researchers have also reported DARTS to not work well, in some cases even no better than random search (Li & Talwalkar 2019 Sciuto et al. 2019). Why is this? How can these seemingly contradicting results be explained? The overall goal of this paper is to understand and overcome such failure modes of DARTS. To this end, we make the following contributions:
We identify 12 NAS benchmarks based on four search spaces in which standard DARTS yields degenerate architectures with poor test performance across several datasets (Section 3
By computing the eigenspectrum of the Hessian of the validation loss with respect to the archi- tectural parameters, we show that there is strong correlation between its dominant eigenvalue and the architecture's 's generalization error. Based on this finding, we propose a simple variation of DARTS with early stopping that performs substantially more robustly (Section 4p.
3. We show that, related to previous work on sharp/flat local minima, regularizing the inner objective of DARTS more strongly allows it find solutions with smaller Hessian spectrum and better generalization propertics. Based on these insights, we propose two practical robustifications of DARTS that overcome its failure modes in all our 12 NAS benchmarks (Section
Our findings are robust across a wide range of NAS benchmarks based on image recognition and also hold for the very different domains of language modelling (PTB) and disparity estimation. They

---

Published as a conference paper at ICLR 2020
consolidate the findings of the various results in the literature and lead to substantially more robust version of DARTS. We provide our implementation and scripts to facilitate reproducibility
BACKGROUND AND RELATED WORK
2.1 RELATION BETWEEN FLAT/SHARP MINIMA AND GENERALIZATION PERFORMANCE
Already Hochreiter & Schmidhuber ( (997) observed that flat minima of the training loss yield better generalization performance than sharp minima. Recent work (Keskar et al. 2016 Yao et 2018) focuses more on the settings of large/small batch size training, where observations show that small batch training tends to get attracted flatter minima and generalizes better. Similarly, Nguueen et al. 2018) observed that this phenomenon manifests also in the hyperparameter space. They showed that whenever the hyperparameters overfit the validation data, the minima lie in a sharper region of the space. This motivated us conduct similar analysis in the context of differentiable architecture search later in Section 4.1 where we see the same effect in the space of neural network architectures.
2.2 BI-LEVEL OPTIMIZATION
We start by a short introduction of the bi-level optimization problem (Colson et al. 2007). These problems which contain two optimization tasks, nested within each other.
Definition 2.1. Given the outer objective function F R R + R and the inner objective function f R x IRN - I, the bi-level optimization problem is given by
min Fly, 0" (ÿ)) yERR
s.t. 0" ()) arg min f(y,
(2)
BERN
where y R.P and R. are the outer and inner variables, respectively. One may also see the bi-level problem as constrained optimization problem, with the inner problem as constraint.
In general, even the case when the inner objective (2) is strongly convex and has an unique
minimizer 0" (v) arg mingER [1, 0). it is not possible to directly optimize the outer objective [T]. A possible method around this issue is to use the implicit function theorem to retrieve the derivative of the solution map (or response map) 0 (y) F C R. w.r.t. y (Bengio 2000 Pedregosa 2016 Beirami et al., 2017). Another strategy is to approximate the inner problem with dynamical et al., 2015 Franceschi al. 2017 (018), where the optimization dynamics could, e.g., describe gradient descent. In the case that the minimizer of the inner problem unique, under some conditions the of minimizers of this approximate problem will indeed converge to the minimizers of the bilevel problem (I) (see Franceschi et al. (2018).
system (Domke (
2.3 NEURAL ARCHITECTURE SEARCH
Neural Architecture Search (NAS) denotes the process of automatically designing neural network architectures in order overcome the cumbersome trial-and-error process when designing archi- tectures manually. We briefly review NAS here and refer to the recent survey by Elsken et al. (2019b for more thorough overview. Prior work mostly employs either reinforcement learning techniques (Baker el al.] 2017a] Zoph & Le| 2017, Zhong et al. 2018 Zoph et al. 2018) or evo- lutionary algorithms (Stanley & Miikkulainen [2002] et al., 2018b Mikkuuliinen et al. 2017] Real et al. 2017: 2019) to optimize the discrete architecture space. As these methods are often very expensive, various works focus on reducing the search costs by, e.g., employing network mor- phisms (Cai et al., 2018a b: Elsken et al. 2017] 201 weight sharing within search models (Sax-) cna & Verbeek 2016; Bender et al. 2018 Pham ct al. (0018) or multi-fidelity optimization (Baker al. 2017b] Falkner et al. 22018 8 Li et al. 2017 Zela et al. 2018), but their applicability still often remains restricted to rather simple tasks and small datasets.
hhttp://ggtchb// homm/automl D
2

---

Published as a conference paper at ICLR 2020
2.4 DIFFERENTIABLE ARCHITECTURE SEARCH (DARTS)
A recent line of work focuses on relaxing the discrete neural architecture search problem to a con- tinuous one that can be solved by gradient descent (Liu al. 2019 Xie al. 2019 Casale al. 2019 Cai et al. 2019). In DARTS (Liu et al., 2019). this is achieved by simply using a weighted sum of possible candidate operations for each layer, whereas the real-valued weights then effee- tively parametrize the network's 's architecture. We will now review DARTS in more detail, as our work builds directly upon
Continuous relaxation of the search space. In agreement with prior work (Zoph al. 2018 Real et al. 2019). DARTS optimizes only substructures called cells that are stacked to define the full network architecture. Each cell contains N nodes organized in directed acyclic graph. The graph contains two inputs nodes (given by the outputs of the previous two cells), a set of intermediate nodes, and one output node (given by concatenating all intermediate nodes). Each intermediate node x(()) represents feature map. See Figure | for an illustration of such cell. Instead of applying a single operation to a specific node during architecture search, [Lu et al. (2019) relax the decision which operation to choose by computing the intermediate node as a mixture of candidate operations, applied to predecessor nodes 20(1), < j. x(1) Eiej
(z((1), where O denotes the set of all candidate operations (e.g., 3 x 3 convolution, skip connection, 3 x 3 max pooling, etc.) and a = (a.) ).j.o serves as a real-valued parameterization of the architecture.
exp(a)))
Ev'co exp(a"))
Gradient-based optimization of the search space. DARTS then optimizes both the weights of the search network (often called the weight-sharing or one-shot model, since the weights of all individual subgraphs/architectrrss are shared) and architectural parameters by alternating gradient descent. The network weights and the architecture parameters are optimized on the training and validation set, respectively. This can be interpreted as solving the bi-level optimization problem ( (2, where F and f are the validation and training loss, Lvalid and Ltrain. respectively, while y and 0 denote the architectural parameters a and network weights w, respectively. Note that DARTS only approximates the lower-level solution by single gradient step (see Appendix [A for more details).
At the end of the search phase, a discrete cell is obtained by choosing the most important incoming operations for each intermediate node while all others are pruned. Importance is measured by the
explo)")
operation weighting factor Eneoo exp(aa"))
3 WHEN DARTS FAILS
We now describe various search spaces and demonstrate that standard DARTS fails on them. We start with four search spaces similar to the original CIFAR-10 10 search space but simpler, and evaluate across three different datasets (CIFAR- 10, CIFAR- 100 and SVHN). They quite standard in that they use the same macro architecture as the original DARTS paper (Liu et al. 2018a), consisting of normal and reduction cells; however, they only allow a subset of operators for the cell search space:
S1:
This search space uses a different set of only two operators per edge, which identified using an offline process that iteratively dropped the operations from the original DARTS search space with the least importance. This pre-optimized space has the advantage of being quite small while still including many strong architectures. We refer to Appendix B for details its construction and illustration (Figure ().
S2: In this space, the set of candidate operations per edge is {3 x 3 SepComv, SkipConnect). We choose these operations since they are the most frequent ones in the discovered cells reported by Liu et al. (2019).
S3: In this space, the set of candidate operations per edge is {3 x 3 SepConv, SkipConnect, Zero), where the Zero operation simply replaces every value in the input feature map by zeros.
S4: In this space, the set of candidate operations per edge is {3 x 3 SepConv, Noise), where the Noise operation simply replaces every value from the input feature map by noise NN 1). This is the only space out of S1-S4 that is not strict subspace of the
3

---

Published as a conference paper at ICLR 2020
(a) Space! 1
(b) Space 2
(c) Space 3
(d) Space
Figure 1: The poor cells standard DARTS finds on spaces S1-S4. For all spaces, DARTS chooses mostly parameter-less operations (skip connection) or even the harmful N Noise operation. Shown are the normal cells on CIFAR-IO; see Appendix G for reduction cells and other datasets.
original DARTS space; we intentionally added the Noise operation, which actively harms performance and should therefore not be selected by DARTS.
We ran DARTS on each of these spaces, using exactly the same setup as Liu et al. (2019). Figure T shows the poor cells DARTS selected these search spaces for CIFAR-TO (see Appendix G for analogous results on the other datasets). Already visually, one might suspect that the found cells are suboptimal: the parameter-less skip connections dominate in almost all the edges for spaces S1-S3, and for S4 even the harmful Noise operation was selected for five out of eight operations. Table
T (first column) confirms the very poor performance standard DARTS yields on all of these search spaces and on different datasets. We note that Liu et al.] (2199) and Xie et al. (2019) argue that the Zero operation can help to search for the architecture topology and choice of operators jointly, but in our experiments it did not help to reduce the importance weight of the skip connection (compare Figure
Tb vs. Figure Ic].
We emphasize that search spaces S1-S3 very natural, and, strict subspaces of the original space, should merely be easier to search than that. They are in no way special or constructed in an adversarial manner. Only S4 was constructed specifically to show-case the failure mode of DARTS selecting the obviously suboptimal Noise operator.
S5: Very small search space with known global optimum. Knowing the global minimum has the advantage that one can benchmark the performance of algorithms by measuring the regret of chosen points with respect to the known global minimum. Therefore, we created another search space with only one intermediate node for both normal and reduction cells, and 3 operation choices in each edge, namely x 3 SepConv SkipConnection, and 3 x MaxPooling. The total number of possible architectures in this space is 81, all of which evaluated a-priori. We dub this space S5.
We ran DARTS on this search space three times for each dataset and compared its result to the baseline of Random Search with weight shar- ing (RS-ws) by Li & Talwalkar (2019). Fig- ure shows the test regret of the architectures selected by DARTS (blue) and RS-ws (green) throughout the search. DARTS manages to find an architecture close to the global minimum, but around epoch 40 the test performance de- teriorated. Note that the search model valida- tion error (dashed red line) did not deteriorate but rather converged, indicating that the archi- tectural parameters overfitting to the vali- dation set. In contrast, RS-ws stays relatively constant throughout the search; when evaluat- ing only the final architecture found, RS-ws in- deed outperformed DARTS.
La factor: 0.0003
tust rigrut
DARTS onw-shot vol. emtrr
RSSts lend. regiet
Validation orfer (%)
Test regret,
330
1.0
Search opoch
Figure 2:
Test regret of found architectures and validation error of the search model when running DARTS on S5 and CIFAR-10. DARTS finds the global minimum but starts overfitting the architec- tural parameters to the validation in the end.
S6: encoder-decoder architecture for disparity estimation. To study whether findings gen- eralize beyond image recognition, we also analyzed a search space for a very different problem: finding encoder-decoder architectures for the dense regression task of disparity estimation; please
4

---

Published as a conference paper at ICLR 2020
One-shol va adon errer (%)
30
Dominant Eigenvalun
S2
53
0.6
54
20
0.4
10
30 Search epoch
20
40
10
30 Search epoch
20
40
50
20
300
40
Search epoch
Figure 3: (left) validation error of search model; (middle) test error of the architectures deemed by DARTS optimal (right) dominant eigenvalue of V Cnatid throughout DARTS search. Solid line and shaded areas show mean and standard deviation of 3 independent runs. All experiments conducted on CIFAR-10.
refer to Appendix B for details. We base this search space AutoDispNet (Saikia al. 2019 which used DARTS for space containing normal, downsampling and upsampling cells. We again constructed a reduced space. Similarly to the image classification search spaces, we found the nor- mal cell to be mainly composed of parameter-less operations (see Figure 25 in Appendix G). As expected, this causes large generalization error (see first row in Table 2 of our later experiments).
4 THE ROLE OF DOMINANT EIGENVALUES OF V Cwalid
We now analyze why DARTS fails in all these cases. Motivated by Section 2. we will have
closer look at the largest eigenvalue Aa Amaz the Hessian matrix validation loss V Luatid w.r.. the architectural parameters
4.1 LARGE ARCHITECTURAL EIGENVALUES AND GENERALIZATION PERFORMANCE
One may hypothesize that DARTS performs poorly because its approximate solution of the bi-level optimization problem by iterative optimization fails, but we actually observe validation errors to progress nicely: Figure 3 (left) shows that the search model validation error converges all cases, even though the cell structures selected here are the ones in Figure
Rather, the architectures DARTS finds do not gen- eralize well. This can be seen in Figure 3 (middle). There, every 5 epochs, we evaluated the architecture deemed by DARTS to be optimal according to the values. Note that whenever evaluating on test set, we retrain from scratch the architecture obtained af- ter applying the argmax to the architectural weights As one notice, the architectures start to de- generate after a certain number of search epochs, similarly to the results shown in Figure? pothesized that this might be related to sharp local minima discussed in Section 2.1 To test this hy- pothesis, we computed the full Hessian V Cralid of the validation loss w.r.t. the architectural parameters on randomly sampled shows that the dominant eigenvalue Xmar (which serves proxy for the sharpness) indeed increases in standard DARTS, along with the test error (middle) of the final architectures, while the validation error still decreases (left). We also studied the correlation between Ama and test error more directly, by measuring these two quantities for 24 different architectures (obtained via standard DARTS and the regularized versions discuss in Section For the example of space SI CIFAR-10. 100 Figure 4 shows that A indeed strongly correlates with test error (with a Pearson correlation coefficient of 0.867).
S1 C10 (Average over the EV trajectory) Pearson coef.: 0.867, p-value: 0.00000
5.5
5.0
Test error (96)
4.0
We hy-
3.0
0.15
0.20
0.25
0.30
0.35
0.40
Average Dominant Eigenvalue
Correlation between dominant eigenvalue of V2 Cralidd and test error of cor- responding architectures.
Figure 4:
Figure (right)
5

---

Published as a conference paper at ICLR 2020
ara710
citar100
Syhn
05
175
150
Eigenvalue
0.5
0.5
0.75
0.3
0.50
0.2
0.5
Figure 6: Local average (LA) of the dominant eigenvalue Ama throughout DARTS search. Markers denote the early stopping point based on the criterion Section 4.3 Each line also corresponds to one of the runs in Table
4.2 LARGE ARCHITECTURAL EIGENVALUES AND PERFORMANCE DROP AFTER PRUNING
One reason why DARTS performs poorly when the architectural eigenvalues are large (and thus the min- imum is sharp) might be the pruning step at the end of DARTS: the optimal, continuous a from the search is pruned to obtain discrete diss some- where the neighbourhood
Lostwald
a². the case of sharp minimum a", dise might have loss func- tion value significantly higher than the minimum a", while in the case of flat minimum, disc is pected to have a similar loss function value. This hypothetically illustrated in Figure 5a where the y-axis indicates the search model validation loss and the x-axis the values.
Eigenvalues vs. Accuracy Drop Spearman corr. coef.: 0.736
Validation accuracy drap (%))
To investigate this hypothesis, we measured the per- formance drop: (a dise, w²) - (o" w*)
w.r.t. to the search model weights incurred by this discretization step and correlated it with Aq The results in Figure 5b) show that, indeed, low curva- ture never to large performance drops (here we
20
10-
actually compute the accuracy drop rather than the loss function difference, relationship). Having identified this relationship. now move on to avoid high curvature.
we observed a similar
0.5
1.0
Dominant Eigenvalue
(b)
4.3 EARLY STOPPING
Figure 5: (a) Hypothetical illustration of the loss function change in the case of flat sharp minima. (b) Drop in accuracy after dis- cretizing the search model the sharpness of minima (by means A²
BASED LARGE EIGENVALUES Lvatid
We propose a simple early stopping methods avoid large curvature and thus poor generalization. We emphasize that simply stopping search based on validation performance (as one would do in the case of training a network) does not apply here as NAS directly optimizes validation performance, which as we have seen in Figure 2 keeps on improving.
Instead, we propose to track Aataz over the course of architecture search and stop whenever it in- creases too much. To implement this idea, use simple heuristic that worked off-the-shelf without any tuiigg. Let (in denote the value of Xmaax smoothed over k epochs around then, we stop if Xmaz (i k)/ Amax 0.75 and return the architecture from epoch k.

---

Published as a conference paper at ICLR 2020
C10
C100
SWHN
30
0.7
Drop probabiity
Drop probsbility
Max. probability
Figure 7: Effect of regularization strength via ScheduledDropPath (during the search phase) on the test performance of DARTS (solid lines) and DARTS-ES (dashed-lines). Results for each of the search spaces and datasets.
By this early stopping heuristic, we do not only avoid exploding eigenvalues, which correlated with poor generalization (see Fig- ure 4. but also shorten the time of the search.
Table 1: Performance of DARTS and DARTS-ES.
(mean + std for 3 runs each).
Table 1 shows the results for running DARTS with this early stop- ping criterion (DARTS-ES) across S1-S4 and all three image classi- fication datasets. Figure Souure/@shor shows the local average of the eigenvalue trajectory throughout the search and the point where the DARTS search early stops for each of the settings in Table Note that we never use the test data when applying the early stopping mecha- nism. Early stopping significantly improved DARTS for all settings without ever harming it.
Benchmark
DARTS
DARTS-ES
S1
0.77
3,05 0.07
0.40
C10
S3
4.121.0.88
3.71
1.14
0.18
4.17
0.21
29.33 1 0.41
28.90
0.81
C100
28.75 1: 0.52
24,68
1.43
S3
25.01 1: 1:
26.99
1.79
5.50
2.80
0.09
3.60 + 0.12
2.68 0.18
SVHN
1,00 1.01
2.78 2.20000..
0.02
5 REGULARIZATION OF INNER OBJECTIVE IMPROVES GENERALIZATION OF ARCHITECTURES
As we saw in Section 4.1 sharper minima (by means of large eigenvalues) of the validation loss lead to poor generalization performance. In bi-level optimization setting, the variables' trajectory depends on the inner optimization procedure. Therefore, we hypothesized that modifying the landscape of the inner objective Ltrain could redirect the outer variables a flatter areas of the architectural space. We study two ways of regularization (data augmentation in Section 5. and L regularization in Section b.2p and find that both, along with the early stopping criterion from Section make DARTS more robust in practice. We emphasize that we do not alter the regularization of final training and evaluation phase, but solely that of the search phase. The setting we use for all experiments in this paper to obtain the final test performance is described in Appendix C
4.3
5.1 REGULARIZATION VIA DATA AUGMENTATION
We first investigate the effect of regularizing via data augmentation, namely masking out parts of the input and intermediate feature maps via Cutout (CO, Deiess & Taylor (2017) and Scheduled- DropPath (DP, Zoph et al. (2018)) (ScheduledDropPath is a regularization technique, but we list here since we apply it together with Cutout), respectively, during architecture search. We ran DARTS with CO and DP (with and without our early stopping criterion, DARTS-ES) with different maximum DP probabilities on all three image classification datasets and search spaces S1-$4.
Figure 7 summarizes the results: regularization improves the test performance of DARTS and DARTS-ES all cases, sometimes very substantially, and the same time kept the dominant eigen- value relatively low (Figure 13). This also directly results in smaller drops in accuracy after pruning,
as discussed Section 4.2 indeed, the search runs plotted in Figure 5b are the same as this sec- tion. Figure 17 in the appendix explicitly shows how regularization relates to the accuracy drops. We also refer to further results in the appendix: Figure [T] (showing test vs. validation error) and Table 5 (showing that overfitting of the architectural parameters is reduced).

---

Published as a conference paper at ICLR 2020
C10
C1.00
SVNN
30
0.0005
0.0933
0,002
CO2E1
0.9245
0.0005
5,209
0
0.0922
5.9455
0.0022
5.003
5,0227
0.028
0.5233
1. factor
factor
Lg factor
Figure 8: Effect of Le regularization of the inner objective during architecture search for DARTS (solid lines) and DARTS-ES (dashed).
Similar observations hold for disparity estimation on S6, where we vary the strength standard data augmentation methods, such as shearing brightness change, rather then masking parts of features, which is unreasonable for this task. The augmentation strength is described by an "augmentation scaling factor" (Appendix Ep. Subble/iumarizze the results. We report the average end point error (EPE), which is the Euclidean distance between the predicted and ground truth disparity maps. Data augmentation avoided the degenerate architectures and substantially improved results.
5.2 INCREASED L2 REGULARIZATION
Table 2: Effect of regularization for dispar- ity estimation. Search was conducted on FlyingThingss (FT) and then evaluated on both FT and Sintel. Lower is better.
As a second type of regularization, we also tested different L2 regularization factors 3i 10-4 for (1.3,9. 9, 27,80 81). Standard DARTS in fact does al- ready include small amount of L2 regularization: i i= yields its default. Sisors that DARTS' test performance (solid lines) be significantly improved by higher L2 factors across all datasets and spaces, while keeping the dominant eigenvalue low (Figure 14. DARTS with early stopping (dashed
Ang. Scale
Scarch model valid
FT tcst
Sintel tesil
Parans
EPE
EPE
EPE
(M)
3.83
5.69
9.65
1.0
4,61
3.12
5.47
12:46
1.5
5,23
2.60
4.15
12.57
2.0
7,45
2.33
3.76
12:25
reg. factur
Search model valid
FT test
Sintel test
Params
lines) also benefits from additional regularization. Again, we observe the implicit regularization effect on the outer objective which reduces the overfitting of the architectural parameters. We again refer to Table 2 for disparity estimation; Appendix shows similar results for language modelling (Penn TreeBank).
EPE
EPE
EPE
(M)
3 10
3:23
0.13
11.00
x 10
4,25
2.72
4.3
10.29
4,61
2.34
3.85
12.16
5.3 PRACTICAL ROBUSTIFICATION OF DARTS BY REGULARIZING THE INNER OBJECTIVE
Based on the insights from the aforementioned analysis and empirical results, we now propose two alternative simple modifications to make DARTS more robust in practice without having to manually tune its regularization hyperparameters.
DARTS with adaptive regularization One option is adapt DARTS' regularization hyperpa- rameters in an automated way, in order to keep the architectural weights in areas of the validation loss objective with smaller curvature. The simplest off-the-shelf procedure towards this desiderata would be to increase the regularization strength whenever the dominant eigenvalue starts increasing rapidly. Algorithm [ DARTS-ADA, Appendix D.I] shows such a procedure. We use the same stopping criterion
DARTS ES (Section 4.3), roll back DARTS to the epoch when this criterion met, and continue the search with a larger regularization value R for the remaining epochs (larger by factor of 7)). This procedure is repeated whenever the criterion is met, unless the regularization value exceeds some maximum predefined value Rmaz-
Multiple DARTS runs with different regularization strength Liu et al. (019)) already sug- gested to the search phase of DARTS four times, resulting in four architectures, and return
8

---

Published as a conference paper at ICLR 2020
the best of these four architectures w.r.t. validation perfonmance when retrained from scratch for limited number of epochs. We propose to use the same procedure, with the only difference that the four runs use different amounts of regularization. The resulting RobustDARTS (R-DARTS) method is conceptually very simple, trivial to implement and likely to work well if any of the tried regularization strengths works well.
Table 3 evaluates the performance of practical robustifications of DARTS, DARTS-ADA and R-DARTS (based on either L2 or SchoduhedDrooPathh regularization), by comparing them to the original DARTS, DARTS-ES and Random Search with weight sharing (RS-ws). For each of these methods, pro- posed in the DARTS paper (Liu et al. 2019), we the search four inde- pendent times with different random seeds and selected the architecture used for the final evaluation based on validation run as deseribed above. As the table shows, in accordance with Li & Talwalkar (019),, RS-ws often outperformed the original DARTS; however, with our robustifications, DARTS typically performs substantially better than us- DARTS-ADA consistently improved over standard DARTS for all benchmarks, indicating that gradual increase of regularization during search prevents ending up in the bad regions of the architectural space. Finally, RobustDARTS yielded the best performance and since it also easier to implement than DARTS-ES and DARTS-ADA, is the method that we recommend to be used in practice.
Table 3: Empirical evaluation of practical robustified ver-
sions of DARTS. Each entry is the test error after retraining the selected architecture as usual. The best method for each setting is boldface and underlined, the second best boldface.
Benchmark
R-DARTSDPI
R-DARTSIL21
S1
5.33
3.54
3.11
1.78
3,01
3.10
52
3,56
4.85
3.48
3.31
3.26
3.35
C10
53
2,93
3334
2.93
2,51
2.74
2.59
84
8.07
7.20
3,58
3,56
3,71
4,84
S1
23,30
29.46
25.95
24.25
28.37
24.03
21.21
22.30
23.52
C100
83
23.75
25.90
22,36
25.59
25.73
23.37
84
28.19
22.85
22.18
21.94
21.26
23,20
51
2.59
4.58
2.55
4.79
2.72
2.53
2,72
SVHN
53
288
3.41
2.49
2.48
2.30
2.30
54
3,46
3.05
2.61
2.50
2.51
2.46
Finally, since the evaluations in this paper have so far focussed smaller subspaces of the original DARTS search space. the reader may wonder how well Robust- DARTS works the full search spaces. As Table 4 shows, RobustDARTS performed similarly to DARTS for the original benchmarks from the DARTS paper (PTB and CIFAR-10) on which DARTS was developed and is well tuned: however, even when only changing the dataset to CIFAR- 100 or SVHN, RobustDARTS already performed significantly better than DARTS, underlining its robustness.
Table 4: DARTS vs. RobustDARTS on the original DARTS search spaces. We show mean + stddev for 5 repetitions (based on 4 fresh subruns each in Ta- ble B); for the more expensive PTB we could only afford 1 such repetition.
Benchmark
DARTS
R-DARTS(L2)
C10
2.91 + 0.25
2.95
0.21
C100
20.58 + 0.44
18.01 + 0,26
SVHN
2.46 +: 0.09
2.17 +: 0.09
PIB
58.64
57.59
6 CONCLUSIONS
We showed that the generalization performance of architectures found by DARTS is related to the eigenvalues of the Hessian matrix of the validation loss w.r.t. the architectural parameters. Stan- dard DARTS often results in degenerate architectures with large eigenvalues and poor generaliza- tion. Based on this observation, we proposed a simple early stopping criterion for DARTS based on tracking the largest eigenvalue. Our empirical results also show that properly regularizing the inner objective helps controlling the eigenvalue and therefore improves generalization. Our findings substantially improve
understanding of DARTS' failure modes and lead to much more robust versions. They are consistent across many different search spaces on image recognition tasks and also for the very different domains of language modelling and disparity estimation. Our code is available for reproducibility.
ACKNOWLEDGMENTS
The authors acknowledge funding by the Robert Bosch GmbH, support by the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation programme through grant no. 716721, and by BMBF grant DeTol..

---

Published as a conference paper at ICLR 2020
REFERENCES
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network archi- tectures using reinforcement learning. In International Conference on Learning Representations, 2017a.
Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating Neural Architecture Search using Performance Prediction. In NIPS Workshop on Meta-Learning, 2017b.
Ahmad Beirami, Meisam Razaviyayn, Shahin Shahrampour, and Vahid Tarokh. On optimal gener- alizability in parametric learning. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, Curran Associates, Inc., 2017.
pp. 3455-3465.
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Understand- ing and simplifying one-shot architecture search. International Conference on Machine Learn- ing. 2018.
Y. Bengio. Gradient-based optimization of hyperparameters. Neural Computation, 12(8):1889 1900, Aug 2000. ISSN 0899-7667. doi: 10.1 00116220899770000001511111
D. Butler, J. Wulff, G. B. Stanley, and M. Black. A naturalistic open source movie for optical flow evaluation. In A. Fitzgibbon et al. (Eds.) (ed.), European Conf. on Computer Vision (ECCV), Part IV, LNCS 7577, pp. 611 1225. Springer-Verlag October 2012.
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In AAAI, 2018a.
Han Cai, Jiacheng Yang, Weinan Zhang, Song Han, and Yong Yu. Path-Level Network Transforma- tion for Efficient Architecture Search. In International Conference on Machine Learning, June 2018b.
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations, 2019.
Francesco Casale, Jonathan Gordon, and Nicolo Fusi. Probabilistic neural architecture search. arXiv preprint, 2019.
Liu Chenxi, Chen Liang Chieh, Schroff Florian, Adam Hartwig. Hua Wei, Yuille Alan L., and Fei Fei Auto-deeplab: Hierarchical neural architecture search for semantic image segmenta- tion. In Conference Computer Vision and Pattern Recognition, 2019.
Benot Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization, 2007.
Terrance Vriei and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 2017.
Justin Domke. Generic methods for optimization- based modeling. Neil D. Lawrence and Mark Girolami (eds.), Proceedings of the Fifteenth International Conference Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Research, pp. 318-326, La Palma, Canary Islands, 21 Apr 2012. PMLR.
A. Dosovitskiy, P. Fischer, E. Ilg, P. Häusser, C. Hazzrbas, V. Golkov, P. v.d. Smagt, D. Cremers, and T. Brox. Flownet: Learning optical flow with convolutional networks. IEEE International Conference Computer Vision (ICCV), 2015.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Simple And Efficient Architecture Search for Convolutional Neural Networks. In NIPS Workshop on Meta-Learning, 2017.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural architec- ture search via lamarckian evolution. In International Conference Learning Representations, 2019a.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey.
Journal of Machine Learning Research, 20(55):1-21, 2019b.
10

---

Published as a conference paper at ICLR 2020
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter op- timization at scale. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th Inter- national Conference Machine Learning. volume 80 of Proceedings of Machine Learning Re- search, pp. 1437-1446, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL. http::/prrocedinggs
www.prrsssssee /v0// fa kner htmml
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In Doina Precup and Yee Whye Teh (eds.), Pro- ceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 165--1173, International Convention Centre, Sydney, Aus- tralia, 06-11 Aug 2017. PMLR.
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In Jennifer Dy and Andreas the
Krause (eds.), Proceedings the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1568-1577, Stockholmsmssan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Compur., 9(1):1-42, January 1997.
ISSN 0899-7667. doi: 10.1 11116//cc.1997..99
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On batch training for deep learning: Generalization gap and sharp minima. arXiv 2016.
preprint
I.. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: Bandit-based configuration evaluation for hyperparameter optimization. In Proceedings of the International Conference on Learning Representations (ICLR') 17), 2017. Published online: iclr. cc
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. CoRR, abs/1902.07638, 2019.
H. Liu, K. Simonyan, O. Vinyals, C.Fernando, and K. Kavukcuoglu. Hierarchical representations for efficient architecture search. In International Conference on Learning Representations (ICLR) 2018 Conference Track, April 2018a
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hier- archical Representations for Efficient Architecture Search. International Conference on Learn- ing Representations, 20.
Hanxiao Liu, Karen Simonyan, and Yiming Yang, DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient- based hyperparameter optimiza- tion through reversible learning. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research. pp. 13-2222,, Lille, France, 07-09 Jul 2015. PMLR.
N. Mayer, F. Ilg, P. Häusser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2016. arXiv:1512.02134.
Risto Mikkullaneen, Jason Liang, Elliot Meyerson, Aditya Rawal, Dan Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, and Babak Hodjat. Evolving Deep Neural Networks. In 1703:0/703300545 March 2017.
Thanh Dai Nguyen, Sumil Gupta. Santu Rana, and Svetha Venkatesh. Stable bayesian optimization. International Journal of Data Science and Analytics, 6(4):327-339, Dec 2018. ISSN 2364-4168. doi: 10.1007/:41000-018.oo0 19-9.
11

---

Published as a conference paper at ICLR 2020
Fabian Pedregosa. Hyperparameter optimization with approximate gradient. In Maria Florina Bal- can and Kilian Q. Weinberger (eds.). Proceedings of The 33rd International Conference on Ma- chine Learning. volume 48 of Proceedings of Machine Learning Research, pp. 737-746, New York, New York, USA, 20-22 Jun 2016. PMLR.
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoe V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In International Conference on Machine Learning, 2018.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V. Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2002-2922222111 International Conven- tion Centre, Sydney, Australia, 06-11 Aug 2017. PMLR.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Aging Evolution for Image Classi- fier Architecture Search. In AAAI, 2019.
T. Saikia, Marrakchi, A. Zela, Hutter, and Brox. Autodispnet: Improving disparity estimation with automl. In IEEE International Conference on Computer Vision (ICCV), 2019. URL http:
7/lmb. infommaatafaatttooommmmm uni wwwpe//wwrg..eee.liieeaeaeeeeeee
221 9/ SMB19
Shreyas Saxena and Jakob Verbeck. Convolutional neural fabrics. D. D. Lee, M. Sugiyama, U. V. Luxburg. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 14553-4066 Curran Associates, Inc., 2016.
Christian Sciuto, Kaicheng Yu, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. arXiv preprint, 2019.
Kenneth Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topolo- gies. Evolutionary Computation, 10:99-127, 2002.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In International Conference on Learning Representations, 2019.
Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W Mahoney. Hessian-based analysis of large batch training and robustness adversaries. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Bisa-Biacchh,,,, and R. Garnett (eds.), Advances in Neural Infommtion Processing Systems 31, pp. 4949 4959. Curran Associates, Inc., 2018
Arber Zela, Aaron Klein, Stefan Falkner, and Frank Hutter. Towards automated deep learning: Ef- ficient joint neural architecture and hyperparameter search. In ICML 2018 Workshop AutoML (AutoML 2018), July 2018
Zhao Zhong, Jingchen Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Practical block-wise neural network architecture generation. In CVPR. IEEE Computer Society, 2018
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In Interna- tional Conference on Learning Representations, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe L. Learning transferable architectures for scalable image recognition. In Conference on Computer Vision and Pattern Recognition, 2018.
12

---

Published as a conference paper at ICLR 2020
A MORE DETAIL ON DARTS
Here we present a detailed description of DARTS architectural update steps. We firstly provide the general formalism which computes the gradient of the outer level problem in ①T) by means of the implicit function theorem. Afterwards, we present how DARTS computes the gradient used to update the architectural parameters O..
A.1 DERIVATIVE WITH SMOOTHED XON-QUADRATIC LOWER LEVEL PROBLEM
Consider the general definition of the bi-level optimization problem given by ① and ( Given that is twice continuously differentiable and that all stationary points are local minimas, one can make use of the implicit function theorem find the derivative of the solution map 0 (y) w.r.t. y/ (Bengio 2000). Under the smoothness assumption, the optimality condition of the lower level (2 Vot (3. 8) O, which defines implicit function 0* (y). With the assumption that o. Under the condition that Vof(y,0") 0 is continuously differentiable and that (y) is continuously differentiable implicitly differentiating the last equality from both sides w.r.t. and applying the chain rule, yields:
ming. Tw:0) has a solution, there exists a (y. 0*) such that Vyf
((vo))
00 O(Vos) By (v) +
(v,0*)
dy (v,0") = 0.
00
Assuming that the Hessian Vjl(U: 0*) is invertible, we can rewrite 3 as follows:
00
dy (v) - (V35(u.0)))
((Vof) (y.0*).
(4)
dy
Applying the chain rule to T for computing the total derivative of F with respect to yields:
dF
dF
00
oF
(5)
dy
00
Oy
Oy
where we have omitted the evaluation at (y. 0) Substituting
4 into 5 and reordering yields:
dF OF
OF
(vof)
(6)
dy
dy
00
000y
equation 6 computes the gradient of F, given the function 6" (y), which maps outer variables to the inner variables minimizing the inner problem. However, in most of the cases obtaining such map- ping is computationally expensive, therefore different heuristics have been proposed to approximate
d.F dFydy (Maclaurin al. 2015 Pedregosa 2016; Franceschi et al. 2017 2018
A.2 DARTS ARCHITECTURAL GRADIENT COMPUTATION
DARTS optimization procedure is defined as bi-level optimization problem where Loalid is the outer objective ① and Ltrain is the inner objective
min Loatiala, w" (a))
s.t. w" (a) arg min Lirain w),
(8)
where both losses are determined by both the architecture parameters a (outer variables) and the network weights (inner variables). Based on Appendix A.I under some conditions, the total derivative of Lvalid w.r.t. a evaluated on (a, w" (a))) would be:
d.valted V Lovatdd V Craatal (a Peraii)))) da
V2 Gawofriinn
0 and 2 p..a Bodw Computing the inverse of the Hessian is in general not possible considering the high dimensionality the model parameters w, therefore resolving to gradient-based iterative algorithms for finding w² is necessary. However, this would also require
where V.
3 V Do
0²
13

---

Published as a conference paper at ICLR 2020
optimize the model parameters w till convergence each time is updated. If our model is deep neural network it is clear that this computation is expensive, therefore [iu et al. (2019) propose to approximate w² (a) by updating the current model parameters using single gradient descent step:
w (a) = -VV (a, w);
(10)
where & is the learning rate for the virtual gradient step DARTS takes with respect to the model weights w. From equation 10 the gradient of w" (a) with respect
Ow (a) -VV? a, Ctrain (a, w),
(11)
da
By setting the evaluation point w" = w V Loraain (a, w) and following the same derivation as in Appendix[A... obtain the DARTS architectural gradient approximation:
d.volid (a) = V VoLoalia w²) - Evatiala, w" V Ctratón w*),
(12)
do
where the inverse Hessian V train in (9) is replaced by the learning rate &. This expression however contains again an expensive vector- vector-matrix product. [Lu et al. (2019) reduce the complexity by using the finite difference approximation around w + V i Lnatia(a, w²) for some small e = 0.01/ waattiaaa w") Ill to compute gradient of VaLtrain (ex, w*) with respect to as
22 0, Ltrain (ar, wt) R VaLrrain (a, wt) VaLirain(ar w
2 Ve Wrratiaaa,,,, w*)
V VaLtrain (a, wt) VaLtrain(a, w
V Lvatial w" V Ceraii (a, w*)
(13)
2e
In the end, combining equation 12 and equation 13 gives the gradient compute the architectural updates in DARTS:
d.valid
(a) V. Lonaiil w²)
Laraii w+) "
V Carain wT))
(14)
da
2e
In all our experiments we always use & (also called second order approximation in Liu et al. 2019)) ) where 17 is the learning rate used SGD for updating the parameters w.
B CONSTRUCTION OF SI FROM SECTION
We ran DARTS two times on the default search space find the two most important operations per mixed operation. Initially. every mixed operation consists of 8 operations. After the first DARTS run, drop the 4 (out of 8) least important ones. In the second DARTS run, we drop the (out of the remaining 4) least important ones. S1 is then defined to contain only the two remaining most important operations per mixed op. Refer to Figure an illustration of this pre-optimized space.
C FINAL ARCHITECTURE EVALUATION
Similar to the original DARTS paper (Luu 2019). the architecture found during the search are scaled up by increasing the number of filters and cells and retrained from scratch to obtain the final test performance. For CIFAR- 100 and SVHN we use 16 number initial filters and 8 cells when training architectures from scratch for all the experiments we conduct. The rest of the settings is the same as Liu et 2019).
On CIFAR-10, 10, when scaling the ScheduledDropPathh drop probability, we use the same settings for training from scratch the found architectures as the original DARTS paper, i.e. 36 initial filters and 20 stacked cells. However, for search space S2 and S4 we reduce the number of initial filters to 16 order to avoid memory issues, since the cells found with more regularization usually are composed only with separable convolutions. When scaling the Lg factor on CIFAR- 10 experiments we use 16 initial filters and 8 stacked cells, except the experiments on S1, where the settings are the same in Liu et al. (219). i.e. 36 initial filters and 20 stacked cells.
14

---

Published as a conference paper at ICLR 2020
skip connect
max pool 3x3
(kx2)
clipeconneet
dil_coov_aa _5.5
dil cony 363
cL
skip_comeett
skip_connee connect
dil cony 5x5
dil conv _33
dil_ cony
conv 3xd
skip_cunnet
stp conv 343
545
dil conv
coav__a5 5x5 26p_ 300.
skip connec
ckip_connect
skip connect
(k-11
skip connect
cony 3x3
max pool 3x3
skip
Normal cell space
max pxol_333 3x3
49g_
max pool avo__ooo__Me 3x
(.22
pool
C
max, _X3
skip_ connect
avg. poop_ooolss 3x3
dil _xs
skip connect
skip connect
mix pool 3x3
dil conyy 3x3
dil conv 5xx5
dil conv 5x5
3x3
pool 335
skip connect
pool_ 3x3
max pool_poo__33 _x3
stp sep_cony _33
pool_ 3x3
(b) Reduction cell space
Figure Search space S1.
Note that although altering the regularization factors during DARTS search, when training the final architectures from scratch we always use the same values for them as Liu et al. (2019), i.e. ScheduledDropPPath maximum drop probability linearly increases from 0 towards 0.2 throughout training, Cutout is always enabled with cutout probability 1.0, and the L2 regularization factor is set 14-4
15

---

Published as a conference paper at ICLR 2020
D ADDITIONAL EMPIRICAL RESULTS
E fistor: 0.0009
factor: 0.0038
Daaes tet my
filea reghin
Ve Udee on eeco
20
Ia
15
TAC PICTPOPP
Search spocs
Search spoch
Search opocs
Figure 10: Test regret and validation error of the search (one-shot) model when running DARTS on S5 and CIFAR-10 10 with different L2 regularization values. The architectural parameters' overfit reduces as we increase the L2 factor and successfully finds the global minimum. However, we notice that the architectural parameters start underfitting we increase to much the L2 factor, i.e. both validation and test error increase.
Table 5: Validation (train) and test accuracy CIFAR- 10 of the search and final evaluation models, respectively. The values the last column show the maximum eigenvalue X (computed on random sampled mini-batch of the Hessian, at the end of search for different maximum drop path probability). The four blocks in the table state results for the search spaces S1-S4, respectively.
Drop Prob.
Valid
Params
seed
seed
seed
seed
seed
seed
seed 2
seed
seed
seed
seed
0.0
87.22
87.01
86.98
96.16
94,43
95.43
2.24
1.93
2.03
1.023
0.835
0.698
0.2
84.24
84.32
84,22
96.39
96.66
96.20
2.63
2.84
2,48
0.148
0.264
0.228
S1
0.4
82.28
82.18
82.79
96.44
96.94
96.76
2.63
2.99
3.17
0.192
0.199
0.149
79.17
79.18
78.84
96.89
96.93
96.96
3.38
3.02
3.17
0.300
0.255
0.256
88.45
95.15
96.11
0.97
0.409
85.29
84.81
85.36
95.15
95.40
96.14
1.28
1.44
1.36
0.270
0.217
0.145
$2
0.4
82.03
82.66
83.20
96.34
96.50
96.44
1.28
1.28
1.36
0.304
0.411
0.282
0.6
79.86
80.19
79.70
96.52
96.35
96.29
1.21
1.28
1.36
0.292
0.295
0.281
0.0
88.78
89.15
88.67
94.70
96.27
96.66
2.21
2.43
2.85
0.496
0.535
0.446
0.2
85.61
85.60
85.50
96.78
96.84
96.74
3.62
4.04
2.99
0.179
0.185
0.202
S3
83.03
83.24
83.43
97.07
96.85
96.48
4.10
3.74
3.38
0.156
0.370
0.184
79.86
80.03
96.91
94.56
96.44
4.46
2.66
0.275
0.280
0.0
86.33
86.72
86.46
92.80
93.22
93.14
1.05
1.13
1.05
0.400
0.442
0.314
0.2
81.01
82.43
82.03
95.84
96.08
96.15
1.44
1.44
1.44
0.070
0.054
0.079
S4
0.4
79.49
79.67
78.96
96.11
96.30
96.28
1.44
1.44
1.44
0.064
0.057
0.049
0.6
74.54
74.74
74.37
96.42
96,36
96.64
1.44
1.44
1.44
0.057
0.060
0.066
D.1 ADAPTIVE DARTS DETAILS
We evaluated DARTS-ADA (Section 5.3 with R 3 10-1 (DARTS default), Rymax 3 10-2 and 7 on all the search spaces and datasets we use for image classification. The results are shown in Table 3 (DARTS-ADA) The function train_andLeval conducts the normal DARTS search for one epoch and returns the architecture at the end that epoch's updates and the st stop value if decision made stop the search and rollback stop.epoch.
16

---

Published as a conference paper at ICLR 2020
Algorithm 1: DARTS ADA
/+ E: epechs to search; R: initial regularization value; Rmax: maximal regularization value; stop.criter: stopping criterion; 7 regularization increase facter Input :E, R, Rmaz, stop.criter,
/+ start search for g epochs for epoch in do
/* run DARTS for one epoch and return stop=Iruu together with the stop epoche
/. and the architecture at stop stop_epoch if the criterion is met stop, step.epoch, arch train (stop.criter);
stop S Rmer then
/x start DARTS from stop_epoch with a larger R arch + DARTS (E stop.epoch, R, Rmozs stop.criter, m)); break
end
end
Output: arch
Teat 194)
12
3.5
0.1
32
0.5
6.5
0.6
3.2
0.3
0.6
0.6
0.0
6.2
0.3
5.5
0.5
0.0
0.1
0.5
Oh Val 1961
Text eroor 660
26
0.0
21
0.2
0.3
0.4
2.5
0.6
0.0
0.
0.2
0.4
0.5
0.5
2.0
0.2
0.2
0.3
24
0.5
0.5
0.0
0.2
0.2
0.3
0.6
0.3
0.5
Text Diroshit
Ore-dhot Val
0.0
2.1
02
0.3
0.4
2.5
0.5
0.0
0.1
0.2
0.4
0.5
0.5
2.0
0.2
0.2
0.3
2.4
0.5
0.0
0.2
0.2
03
0.6
0.3
DP max, prob.
DP ma2 072b
DP max. prob.
max prob.
Figure 11: Test errors of architectures along with the validation error of the search (one-shot) model for each dataset and space when scaling the ScheduledDropPath drop probability. Note that these results (blue lines) are the same as the ones in Figure g
17

---

Published as a conference paper at ICLR 2020
Ftot Mal Wroz 196
Text ertor (50
4.2
4.0
3.0
20025
0.0022
0,0077
28045
0.0022
0.002)
2,0027
0.0001
2.043
02002
0.0000
2,022T
0.0032
0.0043
1.0002
2,0000
20227
0.0023
0.0243
51 C1.0000
C100
(80)
Text emrr (5)
SVHN
SVHN
SVHN
SVMN
Test oner One shat amo
27.5
(56)
12.5
Text encer
20.0
0.0
factor
facter
factor
La fectorr
Figure 12: Test errors architectures along with the validation error the search (one-shot) model for each dataset and space when scaling the L2 factor. Note that these results (blue lines) are the same as ones in Figure 7
18

---

Published as a conference paper at ICLR 2020
cfark
cior22
0.0
3.6
022
Nex Etomoate 400
4p-3 1306
Wan Eiporeaba
Siperattes
8:28
8:05
28
30
53 cbbrico
ctario0
cote1100
No4 Eigeeaa 90
Nax. w.g.edddcc... Pd.
8.23
3.8
8:28
8.25
04
24
5poch
E3ech
5p0th
50001
51 Jdfns
52 avfn
53 sxtn
54 avin
0.24
1.250
3.27
0.23
0.220
0.46
5.55
8:04
0:1
8.62
Spoch
Epich
Spac1
Figure 13: Local average of the dominant EV Ao Amaz throughout DARTS search (for different drop
path prob. values). Markers denote the early stopping point based on the criterion in Section
L
1,-20009
4--0041
066
Max Dipomanie
0:3
0:2
6.1
Soech
Esoch
51 dPM-100
52 ctvr000
53 cth-100
Si chtrr0o
3.75
0.20
6.25
Mea. (gr.ane
0.22
10
56
5poch
C20ch
1p0ch
ofhn
shn
54 cytn
6.5
0.27
0.34
Mae Eiproeeie MA
044
0.20
0:32
Max Bjsmclucks
0.200
0:20
3.225
0:00
4.2
812
20
20
20
10
20
50
23
33
26
Figure 14: Efffect of Le regularization no the EV trajectory. The figure is analogous to Figure 13
19

---

Published as a conference paper at ICLR 2020
Openrpectrum dbvi0
Openrpectrum: 12
Fommsetrum: 53 diar10
Eigrrppetrum: 54 1r/10
dp-6.2036
05
4p-6,4526
5.
4p-6.5826
02-
0.10
2.0
-0.3
5:0
aigencatuon
6:30
0.25
0:20-
4.7
6:2
125
2.00
20
14h digonealic
Fthegge WAUR
F3h djenslue
1th digenwake
gersacctrum: $1.sfn
Bgonpectrumm s2hn
Bystopect 53
persazectuum: idfn
0.12
6.2
0.5
0.10
0.0
3.00
3.06
0.04
0.05
003-
3.00
-003
3.00
0.00
10
27
30
30
20
35
elpemm
2--3
dhar1000
duttbbatoo:: chtrt00
Egen didibbztio ota/200
busiam: S4 dlar200
2.
2.0
2.0
2.5
22
02
03
Rigen detdiuatin: saln
Epen distrbuttoo 52 sulnn
Fipen ditlibution 53 coles
Eigen debritutoo: 54 sattit
2
403
-32
02
0.05
Figure 15: Effect of ScheduledDropPath and Cutout on the full eigenspectrum of the Hessian the end of architecture search for each of the search spaces. Since most of the eigenvalues after the 30-th largest one are almost zero, we plot only the largest (based on magnitude) 30 eigenvalues here. We also provide the cigenvalue distribution for these 30 eigenvalues. Notice that not only the dominant eigenvalue is larger when dp but in general also the others.
20

---

Published as a conference paper at ICLR 2020
Openrpectium dbvi0
Openrpectrum: 12
Fomseetrum: 53 diar10
Eeeespertomm ctar3
0000
06
7.0
5.
12 1081
4,4 6067
033
011
0.1
2.0
-03
3.00
-2.00
aigencation
0.50
4.75
1.2
3.00
20
10h digovallic
Fthegge WAUe
F-1 dgennaiue
F:th eihhelgenooe MC
gersacttrum: $1.sfn
Bgonpectrumm sihn
tigurapccrum:
diperppccrann
0.3
0.25
0.325
0.2
011
6.10-
3.000
0.2
Bolpohu
3.005
0.6-
3.000
6:20
-0.55
-2.05
27
30
30
32
20
35
20.
27.5
15.
L--0000)
12.5
04.
20-
2.5
Epen. attibbtttn clar220
dttnibtion: citar100
Bpen diditbztion otar200
butlan: S4 dlar200
4.5
00
2.0
2.
22
22
034
Rigen detiiusion: SL saln
Esem distrbution 52 sulnn
Eipen ritlibution coles
Eigen. debritutoo: 54 saltn
35
34-
0
-03
-32
-0.00
Figure 16: Effect of La regularization on the full eigenspectrum of the Hessian at the end of archi- tecture search for each of the search spaces. Since most of the eigenvalues after the 30-th largest one are almost zero, we plot only the largest (based on magnitude) 30 eigenvalues here. We also provide the eigenvalue distribution for these 30 eigenvalues. Notice that not only the dominant eigenvalue is larger when L2 3 10 but in general also the others.
21

---

Published as a conference paper at ICLR 2020
C10-S1
Validation accuracy drop (%)
C10-52
C10-53
C10-54
C100-S1
SVHN-S1
4-
0.2 Max. Drop probability
Figure 17: Drop in accuracy after discretizing the search model for different spaces, datasets and drop path regularization strengths.. Example of some of the settings from Section
More L2 regul arization
More DropPath regularization
WWH--S
504451
Test error (%)
0.0303
0.0029
0.0027
20001
0.0043
0.0729
0.2187
0.0
02
0.
L2 value
Drop prob. value
Figure 18: Effect of more regularization on the performance of found architectures by DARTS.
Table Performance of architectures found by DARTS (-ES/ / -ADA) RandomNAS with weight sharing. For each of the settings we repeat the search 3 times and report the mean + std of the found architectures retrained from scratch.
Setting
RandomNAS
DARTS
DARTS-ES
DARTS-ADA
S1
3.17 1177001 0.15
4.66 + 0.71
3.05 + 0.07
3.03 0.08
S2
3.46 0.15
4.42 + 0.40
3.41 441++..14 0.14
3.59 + 0.31
C10
S3
2.92 + 0.04
4.12 = 0.85
3.71 - 1.14
2.99 = 0.34
89.39
0.84
6.95 + 0.18
4.17 0.21
3.89 0.67
SI
25.81 + 0.39
29.93 1 0.41
28.90 0.81
24.94 0.81
S2
22.88 + 0.16
28.75 + 0.92
24.68 + 1.43
26.88 + 1.11
C100
S3
24.58
0.61
29.01 0.24
26.99
1.79
24.55
0.63
S4
30.01 + 1.52
24.77 + 1.51
23.90 + 2.01
23.66
0.90
S1
2.64 + 0.09
9.88 + 5.50
2.80 + 0.09
2.59 + 0.07
52
2.57
+ 0.04
3.69 = 0.12
2.68 - 0.18
2.79 0.22
SVHN
S3
2.89
0.09
4.00 + 1.01
2.78 + 0.29
2.58 + 0.07
S4
3.42 - 0.04
2.90 = 0.02
2.55 - 0.15
2.52 1 0.06
22

---

Published as a conference paper at ICLR 2020
D.2 A CLOSER LOOK AT THE EIGENVALUES
Over the course of all experiments from the paper, we tracked the largest eigenvalue across all configuration and datasets to see how they evolve during the search. Figures [3 and [4 shows the results across all the settings for image classification. It can be clearly seen that increasing the inner objective regularization, both in terms of L2 or data augmentation, helps controlling the largest eigenvalue and keeping it to small value, which again helps explaining why the architectures found with stronger regularization generalize better. The markers on each line highlight the epochs where DARTS is early stopped. As can from Figure 4 there is indeed some correlation between the average dominant eigenvalue throughout the search and the test performance of the found architectures by DARTS.
Figures 15 and T6 (top 3 rows) show the full spectrum (sorted based on eigenvalue absolute values) the end of search, whilst bottom rows plot the distribution of eigenvalues in the eigenspectrum. As one can see, not only the dominant eigenvalue is larger compared to the cases when the regu- larization stronger and the generalization of architectures is better, but also the other eigenvalues in the spectrum have larger absolute value, indicating a sharper objective landscape towards many dimensions. Furthermore, from the distribution plots note the presence of more negative eigenvalues whenever the architectures are degenerate (lower regularization value) indicating that DARTS gets stuck in a point with larger positive and negative curvature of the validation loss objective, associated with more degenerate Hessian matrix.
E DISPARITY ESTIMATION
E. DATASETS
We use the FlyingThings3D dataset (Mayer et al. 2016 for training AutoDispNet. consists of rendered stereo image pairs and their ground truth disparity maps. The dataset provides training and testing split consisting of 818 and 4248 samples respectively with an image resolution of 960 x 540. We use the Sintel dataset (Butler et al. (2012) for testing our networks. Sintel is another synthetic dataset from derived from an animated movic which also provides ground truth disparity maps (1064 samples) with a resolution of 1024 x 436.
E.2 TRAINING
We use the AutoDispNet-C architecture as described in Saikia et al. (2019). However, we use the smaller search which consists of three operations: MarPool3 Skipponnect. For training the search network, images are downsampled by a factor of two and trained for 300k mini-batee iterations. During search, we use SGD and ADAM optimize the inner and outer objectives respectively. Differently from the original AutoDispNet we do not warm- start the search model weights before starting the architectural parameter updates. The extracted network is also trained for 300k mini-batch iterations but full resolution images are used. Here, ADAM is used for optimization and the learning rate is annealed to from le 4, using a cosine decay schedule.
SepConw3 3, and
E.3 EFFECT OF REGULARIZATION ON THE INNER OBJECTIVE
To study the effect of regularization on the inner objective for AutoDispNet-C we use experiment with two types of regularization: data augmentation and of L2 regularization network weights.
We note that we could not test the early stopping method on AutoDispNet since AutoDispNet relies on custom operations to compute feature map correlation (Dosovitskiy et al. 2015) and resampling, for which second order derivatives are currently not available (which are required to compute the Hessian).
Data augmentation. Inspite of fairly large number of training samples in FlyingThingg3D, data augmentation is crucial for good generalization performance. Disparity estimation networks employ spatial transformations such as translation, cropping. shearing and scaling. Additionally, appearance transformations such additive Gaussian noise, changes in brightness, contrast, gamma and color
23

---

Published as a conference paper at ICLR 2020
also applied. Parameters for such transformations sampled from uniform Gaussian distri- bution (parameterized by a mean and variance). In our experiments, we vary the data augmentation strength by multiplying the variance of these parameter distributions by fixed factor, which we dub the augmentation scaling factor. The extracted networks are evaluated with the same augmentation parameters. The results of increasing the augmentation strength of the inner objective be in Table 2 We observe that as augmentation strength increases DARTS finds networks with more number of parameters and better test performance. The best performance obtained for the network with maximum augmentation for the inner objective. At the same time the search model validation error increases when scaling up the augmentation factor, which again enforces the argu- ment that the overfitting of architectural parameters is reduced by this implicit regularizer. 1, regularization. We study the effect of increasing regularization strength on the weights of the network. The results are shown in Table 2 Also in this case best test performance is obtained with the maximum regularization strength.
F RESULTS ON PENN TREEBANK
Here we investigate the effect more Le regularization on the inner objective for searching re- current cells on Penn Treebank (PTB). We again used a reduced search space with only ReLU and identity mapping as possible operations. The rest of the settings is the same as in (Liu et al. 2019).
We run DARTS search four independent times with different random seeds, each with four L regularization factors, namely x 10 (DARTS default), 15 x 10-7, 45 x 10 and 135 x 10 Figure 19 shows the test perplexity of the architectures found by DARTS with the aforementioned L2 regularizzzaa laguzarization values. As we can see, a stronger regularization factor on the inner objective makes the search procedure more robust. The median perplexity of the discovered architectures gets better as we increase the L2 factor from 5 10-? to 45 10- while the search model (one-shot) validation mean perplexity increases. This observation similar the ones on image classification shown in Figure I0 showing again that properly regularizing the inner objective helps reduce overfitting the architectural parameters.
Effect of more L, regularization generalization
Valid ppt
742
70.0
One shot validatiion perplesn perplexity
Test perplexity
67.5
738
62.5
60.0
736
734
5e-07
1.5e-06
4.5e-06
1.35e-05
factor
Figure 19: Performance of recurrent cells found with different L2 regularization factors on the inner objective PTB. We run DARTS independent times with different random seeds, train each of them from scratch with the evaluation settings for 1600 epochs and report the median test perplexity. The blue dashed line denotes the validation perplexity of the search model.
24

---

Published as a conference paper at ICLR 2020
G DISCOVERED CELLS ON SEARCH SPACES S1-S4 FROM SECTION 3 ON OTHER DATASETS
(a) SI
(b) S2
(c) S3
(d) S4
Figure 20: Reduction cells found by DARTS when ran CIFAR-10 with its default hyperparame- ters on spaces S1-S4. These cells correspond with the normal ones in Figure [
(b) S2 (C100)
(d) S4 (C100)
(a) S1 (C100)
(c) S3 (C100)
(h) S4 (SVHN)
(f) S2 (SVHN)
(c) SI (SVHN)
(g) S3 (SVHN)
Figure 21: Normal cells found by DARTS on CIFAR-100 and SVHN when ran with its default hyperparameters on spaces S1-S4. Notice the dominance of parameter-less operations such as skip connection and pooling ops.
(a) S1 (C100)
(b) S2 (C100)
(d) S4 (C100)
(c) S3 (C100)
(e) SI (SVHN)
(g) S3 (SVHN)
(1) S2 (SVHN)
(h) S4 (SVHIN
Figure 22: Reduction cells found by DARTS on CIFAR-100 and SVHN when ran with its default hyperparameters on spaces S1-S4.
25

---

Published as a conference paper at ICLR 2020
(b) S2 (C10)
(a) SI (C10)
(d) S4 (C10)
(c) S3 (C10)
(c) S1 (C100)
(f) S2 (C100)
(g) S3 (C100)
(h) S4 (C100)
ii S2 (SVHN)
(i) SI (SVHN)
(k) S3 (SVHN)
()) S4 (SVHN)
Figure 23: Normal cells found by DARTS-ES when ran with DARTS default hyperparameters on spaces S1-S4.
MELOWE 763
(b) S2 (C10)
(a) S1 (C10)
(c) S3 (C10)
(d) S4 (C10)
S1 S1 (C100)
(f) S2 (C100)
(g) S3 (C100)
(h) S4 (C100)
( SI (SVHN)
(i) S2 (SVHN)
(k) S3 (SVHN)
(I) S4 (SVHN)
Figure 24: Reduction cells found by DARTS-ES when ran with DARTS default hyperparameters on spaces S1-S4.
26

---

Published as a conference paper at ICLR 2020
OPUAI
(b) Reduction cell
(a) Normal cell
(c) Upsampling cell
Figure 25: Cells found by AitoDispNet when ran on S6-d. These cells correspond to the results for
augmentation scale 0.0 of Table 2
(b) Reduction cell
(c) Upsampling cell
(a) Normal cell
Figure 26: Cells found by Auto AutoDispNet when ran on S6-d. These cells correspond the results for augmentation scale 2.0 of Table 2
(a) Normal cell
(b) Reduction cell
(c) Upsampling cell
Figure 27: Cells found by AutoDispNet when S6-d. These cells correspond the results for
Lg = 3 10 of Table
(a) Normal cell
(b) Reduction cell
Upsampling cell
Figure 28: Cells found by AutoDispNet when ran on S6-d. These cells correspond to the results for
L 81 10-4 of Table
27

---

Published as a conference paper at ICLR 2020
(a) SI
(d) S4
(b) S2
(c) S3
(f)
(g) S3
(c) SI
(h) S4
Figure 29: Normal (top row) and reduction (bottom) cells found by DARTS on CIFAR-10 when ran with its default hyperparameters on spaces S1-S4. Same as Figure I but with different random seed (seed 2).
(a) SI
(b) S2
(d) S4
S3
(f) S2
(g)
(c) SI
(h) S4
Figure 30: Normal (top row) and reduction (bottom) cells found by DARTS on CIFAR-10 when with its default hyperparameters on spaces $1-S4. Same as Figure I but with different random seed (seed 3).
28


Reviews:
[{"review_id": "SylDRJcpYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "----- Updated after rebuttal period ---\n\nThe author's detailed response effectively addressed my concerns. I am moving my score to Accept. This paper proposes an interesting systematic study of differentiable approach in NAS.\n\n------ Original Review ----\n\nSummary \n\nThis paper presents a systematic evaluation on top of differentiable architecture search (DARTS) algorithm and shows it usually searched an architecture with all skip-connection. It empirically reveals that the largest eigenvalue of the Hessian matrix (\\lambda) of loss w.r.t. architecture parameters has a strong correlation with the generalization ability (via loss of test dataset), and shows this \\lambda will first decrease but then drastically increase after a certain epoch number on 4 different search spaces. It then proposes an early-stop scheme (DARTS-ES) to stop the search before this phenomenon occurs. In addition, it proposes to use data-augmentation, path-dropping and tuning L2 regularization during the search, namely Robust-DARTS(R-DARTS), and yield constantly better results over original DARTS on 3 datasets. \n\n\nOverall, the observation that the largest eigenvalue of the Hessian matrix is novel and intriguing, and the experiments are extensive and meaningful. The idea to use more search spaces for comparison is fair and performance increase demonstrates the proposed R-DARTS and DARTS-ES  are effective. Although I still have some questions regarding the detail settings, I think this paper provides a novel angle to understand the search phase of DARTS, and proposed simple but effective regularization can be beneficial to the research community using DARTS. \n\n\nMain concerns:\n\n- Problem of DARTS as a motivation \nThe claims of local smoothness/sharpness and generalization are related to network generalization is quite intriguing, however, only using largest eigenvalue of Hessian matrix as an indicator of this local shape does not seem to be enough. A recent paper on loss-landscape visualization [1] provides means to examine this hypothesis directly, and could the author try to provide additional visualization to support their claim? Otherwise, the paper's claim does not generalize to the local shape of the loss function, and should stays with the largest eigenvalue. It is totally okay in my perspective, but just indicates some revision to the main text and analysis of Section 4.2.\n\n- Questions about Figure 3 experiments\nHow is test error computed? Is it on a batch of test-split, or the entire one? Also, which architecture is used to compute this test error? Paper mentioned, in Section 4.1, the word \"final architecture\", but does this refer to the super-net (the one-shot model in paper's definition), or the stand-alone model obtained via binarized architecture alphas? If latter, is this generalization error obtained from training from scratch? Or simply using the super-net parameters during the search? Since the conclusion of this plot serves as the foundation of designing R-DARTS and DARTS-ES, if the experiments are only conducted over a small set of images or the binarized model with super-net parameters, it undercuts the credibility of the conclusion, largest architectural eigenvalue, and generalization ability.\n\n\n- More independent runs of experiments.\nIn Figure 3, validation of original DARTS, and Table 1, DARTS vs DARTS-ES, paper runs the experiment for 3 times and take the average, but for the proposed R-DARTS, it is not. Is there a reason why not scaling the experiments? I suggest the author provide results over 5 runs, like the one in Table 4 for PTB and show if the R-DARTS truly surpasses DARTS constantly. This question also applies to Figure 1, when paper claims the original DARTS found poor cell type, could this be repetitive over multiple runs?\n\nI guess for the experiments in Table 3, it is already done since paper mentioned the reported results are the best model of searching 4 times. \n\n- R-DARTS failed to out-perform DARTS in the original space on CIFAR-10\nThis is confusing, will this suggest, if tuning well, DARTS will surpass R-DARTS(L2) in other cases as well? Since this is the only setting that DARTS is built upon. \n\n\nMinor comments and questions\n\n- Using test data during search\nAfter showing the strong correlation between the largest eigenvalue of Hessian and the network generalization error, the early-stop is natural, however, does this mean the model selection is using the test data? Or the actual test-data is never seen during the search phase of Section 4.3.\n\n- L2 stabilizes max eigenvalue\nPaper uses L2 coefficient up to 0.0243, showing constant improvement of test error while validation error drops in CIFAR-10 of Figure 11. Could the author try larger coefficients to determine when this trend will stop? \n\n- Question about section 4.2\nPerformance drop due to the binarized operation (pruning step) in DARTS analysis is very interesting, I am curious how many architectures does the paper evaluate in Figure 5, when the dominant eigenvalue is smaller than 0.25? Since the conclusion is \"low curvature never led to large performance drops\" if the number of points is too few, it is not that convincing, especially from the plot, we see at eigenvalue = 0.5, there exists 2 architecture with >20% drop. In addition, what does each point in Figure 5 refer to? The best model (and the binarized one according to the argmax of \\alpha) of one independent DARTS run or some binarized models sampled from a distribution on top of the same DARTS run (meaning only one super-net)? Is this experiment follows the setting in Figure 4? \n\n- Figure 6, C10 S2, DARTS-ES is worse than DARTS when Drop probability = 0.6, whereas all other cases, DARTS-ES outperforms DARTS, why does this happen? Could the author comment on it?\n\n- ScheduledDropPath in section 5.1\nDoes Drop-path belongs to data-augmentation techniques? It is more like a regularization in my perspective and should be grouped with 5.2.\n\n- S1 S2... in Table 3\nDoes this refer to the search space? Or different random seed (mentioned )\n\n- one-shot v.s. weight sharing model\nOne-shot in NAS domain is firstly introduced by Bender et al., while Pham et al. use parameter sharing. The reason to use one-shot is that all the sub-paths will have a fair chance to be trained. \n\n- Typos\n1. In section 2.1, line 4 \"better.Similarly\" should have space.\n\n--- Reference ---\n[1] Li et al., Visualizing the Loss Landscape of Neural Nets, arxiv'17.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HylLUgPU5S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the causes of why DARTS often results in models that do not generalize to the test set. The paper finds that DARTS models do not generalize due to sharp minima, partially caused by the discretizing step in DARTS. The paper presents many experiments and studies on multiple search spaces, showing that this problem is general. To address this problem, the paper proposes several different ways to address this, e.g., an early stopping criteria and regularization methods.\n\nOverall, the paper is well written, thorough experiments (various tasks and search spaces) show the benefit of the approach.  The final experiments using the full DARTS space also show an improvement over standard DARTS.\n\nThe final method is fairly simple, running the search with different regularization parameters and keeping the best model, which suggests it could be widely used for DARTS-based approaches. \n\n Two (very) minor comments:\n- There's a missing space before \"Similarly\" in section 2.1\n- Extra \")\" in last paragraph of section 2.2 (after the argmin eq).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJgGSAKk9H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper seeks to understand why Differential Architecture Search (DAS) might fail to find neural net architectures that perform well. The authors perform a series of experiments using different kinds of search spaces and datasets, and concluded that a major culprit is the discretization/pruning step at the end of DARTS. To avoid this, the authors propose early stopping based on measuring the eigenvalue of the Hessian of the validation loss. The results look promising (though as someone who is not familiar with the datasets, I don't have a sense of the significance of improvements.)\n\nIn general, this is a strong paper. I enjoyed reading it. It describes the problem clearly and performs a set of convincing experiments to support the claims. I especially like how different constrained search spaces are investigated, as this makes the results easier to interpret. I think the analysis in this paper will benefit researchers who work on similar problems. \n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the properties of Differentiable Architecture Search, and in particular when it fails, and then proposes modifications that improve its performance for several tasks. The reviews were all very supportive with three Accept opinions, and authors have addressed their comments and suggestions. Given the unanimous reviews, this appears to be a clear Accept. 
Decision: Accept (Talk)