NAS evaluation is frustratingly hard
Authors: Antoine Yang
Pedro M. Esperança
Fabio M. Carlucci
Venue:  (2020)
Keywords: neural architecture search
nas
benchmark
reproducibility
harking

Abstract: Neural Architecture Search (NAS) is an exciting new field which promises to be as much as a game-changer as Convolutional Neural Networks were in 2012. Despite many great works leading to substantial improvements on a variety of tasks, comparison between different methods is still very much an open issue. While most algorithms are tested on the same datasets, there is no shared experimental protocol followed by all. As such, and due to the under-use of ablation studies, there is a lack of clarity regarding why certain methods are more effective than others. Our first contribution is a benchmark of 8 NAS methods on 5 datasets. To overcome the hurdle of comparing methods with different search spaces, we propose using a method’s relative improvement over the randomly sampled average architecture, which effectively removes advantages arising from expertly engineered search spaces or training protocols. Surprisingly, we find that many NAS techniques struggle to significantly beat the average architecture baseline. We perform further experiments with the commonly used DARTS search space in order to understand the contribution of each component in the NAS pipeline. These experiments highlight that: (i) the use of tricks in the evaluation protocol has a predominant impact on the reported performance of architectures; (ii) the cell-based search space has a very narrow accuracy range, such that the seed has a considerable impact on architecture rankings; (iii) the hand-designed macrostructure (cells) is more important than the searched micro-structure (operations); and (iv) the depth-gap is a real phenomenon, evidenced by the change in rankings between 8 and 20 cell architectures. To conclude, we suggest best practices, that we hope will prove useful for the community and help mitigate current NAS pitfalls, e.g. difficulties in reproducibility and comparison of search methods. The
code used is available at https://github.com/antoyang/NAS-Benchmark.

Published as a conference paper at ICLR 2020
NAS EVALUATION IS FRUSTRATINGLY HARD
Antoine Yang*
Pedro M Esperança Huawei Noah's Ark Lab London, UK
Fabio Maria Carlucci Huawei Noah's Ark Lab London, UK
École Polytechnique France
ABSTRACT
Neural Architecture Search (NAS) is exciting new field which promises to be as much as a game-changer as Convolutional Neural Networks were in 2012. De- spite many great works leading substantial improvements comparison between different methods is still very much an open issue. While most algorithms are tested on the same datasets, there no shared experimental protocol followed by all. As such, and due to the under-use of ablation studies, there lack of clarity regarding why certain methods are more effective than others. Our first contribution is benchmark of 8 NAS methods on 5 datasets. To overcome the hurdle of comparing methods with different search spaces, we propose using a method's relative improvement over the randomly sampled av- erage architecture, which effectively removes advantages arising from expertly engincered search spaces training protocols. Surprisingly. we find that many NAS techniques struggle to significantly beat the average architecture baseline. We perform further experiments with the commonly used DARTS search space in order to understand the contribution of each component in the NAS pipeline. These experiments highlight that: (i) the of tricks in the evaluation protocol has a predominant impact on the reported performance of architectures; (ii) the cell-based search space has a very narrow accuracy range, such that the seed has a considerable impact on architecture rankings; (iii) the hand-designed macro- structure (cells) is more important than the searched micro-structure (operations); and (iv) the depth-gap gap is a real phenomenon, evidenced by the change in rankings between and 20 cell architectures. To conclude, we suggest best practices, that we hope will prove useful for the community and help mitigate current NAS pit- falls, e.g. difficulties reproducibility and comparison of search methods. The code used is available at http:///gituub..oo/aatttaaaaaap -en.------Bnha........
variety of tasks,
INTRODUCTION
As the deep learning revolution helped us move away from hand crafted features (Krizhevsky al. 2012) and reach new heights (He et al. 20166) Szegedy et al., (007)). so does Neural Architecture Search (NAS) hold the promise of freeing us from hand-crafted architectures, which requires te- dious and expensive tuning for each new task or dataset. Identifying the optimal architecture is indeed a key pillar of any Automated Machine Learning (AutoML) pipeline. Research in the last two years has proceeded at rapid pace and many search strategies have been proposed, from Rein- forcement Learning (Zoph & Le 2017 Pham et al.] 2018), to Evolutionary Algorithms (Real et al., 2017). to Gradient-baas based methods (Liu et al. 2019 Liang et al. 2019). Still, it remains unclear which approach and search algorithm is preferable. Typically, methods have been evaluated on accuracy alone, even though accuracy is influenced by many other factors besides the search algo- rithm. Comparison between published search algorithms for NAS is therefore either very difficult (complex training protocols with no code available) simply impossible (different search spaces), as previously pointed out (Li & Talwalkar 2019 Sciuto et al. 20019] Lindauer & Hutter 2019).
NAS methods have been typically decomposed into three components (Elsken et al. 2019 Li &
Talwalkar 2019): search space, search strategy and model evaluation strategy. This division is
com, esperanca,,,,,, fabi o.m.oo........m... maria.caaruuae carlucci Jhuuawei com com friis work was done when the first author was an intern at Huawei Noah's 8. Ark Lab, London, United Kingdom.
"antoi nentng3gyyag

---

Published as a conference paper at ICLR 2020
important to keep in mind, an improvement in any of these elements will lead to better final per- formance. But is a method with a more (manually) tuned search space a better AutoML algorithm? If the key idea behind NAS is to find the optimal architecture, without human intervention, why are we devoting so much energy to infuse expert knowledge into the pipeline? Furthermore, the lack of ablation studies in most works makes it harder pinpoint which components are instrumental the final performance, which can easily lead to Hypothesizing After the Results are Known (HARKing; Gencoglu et al. 2019).
Paradoxically, the huge effort invested in finding better search spaces and training protocols, has led to a situation in which any randomly sampled architecture performs almost as well as those obtained by the search strategies. Our findings suggest that most of the gains in accuracy in recent contributions to NAS have come from manual improvements in the training protocol, not in the search algorithms.
As a step towards understanding which methods are more effective, we have collected code for 8 reasonably fast (search time of less than 4 days) NAS algorithms, and benchmarked them on well known CV datasets. Using a simple metric-the relative improvement over the average architecture of the search space-we find that most NAS methods perform very similarly and rarely substantially above this baseline. The methods used are DARTS, StacNAS, PDARTS, MANAS, CNAS, NSGANET, ENAS and NAO. The datasets used are CIFARIO, CIFARIO0, SPORTS, MIT67 and FLOWERS102..
Through number of additional experiments the widely used DARTS search space (Liu al., 2019), we will show that: (a) how you train your model has a much bigger impact than the actual architecture chosen; (b) different architectures from the same search space perform very similarly, so much so that (c) hyperparameters, like the number of cells, or the seed itself have a very sig- nificant effect on the ranking: and (d) the specific operations themselves have less impact on the final accuracy than the hand-designed macro-structure of the network. Notably, we find that the 200+ architectures sampled from this search space (available from the link in the abstract) are all within range of one percentage point (top-1 accuracy) after standard full training on CIFAR 10. Finally, we include some observations on how to foster reproducibility and discussion on how to potentially avoid some of the encountered pitfalls.
2 RELATED WORK
As mentioned, NAS methods have the potential to truly revolutionize the field, but to do so it is crucial that future research avoids common mistakes. Some of these concerns have been recently raised by the community.
For example, Li & Talwalkar (2019) highlight that most NAS methods a) fail to compare against an adequate baseline, such as properly implemented random search strategy, b) are overly complex, with no ablation to properly assign credit to the important components, and c) fail to provide all details needed for successfully reproducing their results. In our paper we go step further and argue that the relative improvement over the average (randomly sampled) architecture an useful tool to quantify the effectiveness of a proposed solution and compare with competing methods. To partly answer their second point, and understand how much the final accuracy depends on the specific architecture, we implement an in-depth study of the widely employed DARTS (Liu al. 2019) search space and perform an ablation on the commonly used training techniques (e.g. Cutout, DropPath, AutoAugment).
In addition, Sciuto et al. (2019) also took the important step of systematically using fair baselines, and suggest random search with early stopping, averaged over multiple seeds, as an extremely com- petitive baseline. They find that the search spaces of three methods investigated (DARTS, ENAS, NAO) have been expertly engineered to the extent that any randomly selected architecture performs very well. In contrast, we show that even random sampling (without search) provides an incredibly competitive baseline. Our relative improvement metric allows us to isolate the contribution of the search strategy from the effects of the search space and training pipeline. Thus. we further confirm the authors' claim, showing that indeed the average architecture performs extremely well and that how you train model has more impact than any specific architecture.
2

---

Published as a conference paper at ICLR 2020
3 NAS BENCHMARK
In this section we present a systematic evaluation of 8 methods on 3 datasets using a strategy that designed to reveal the quality of each method's search strategy, removing the effect of the manually- engineered training protocol and search space. The goal is to find general trends and highlight common features rather than just pin-pointing the most accurate algorithm.
Understanding why methods are effective is not an easy task: most introduce variations to previous search spaces, search strategies, and training protocols- with ablations disentangling the contribu- tion of each component often incomplete or missing. state-of-te -frte-----e-tttttt method is not simply due to a better engineered search space training protocol? To address this issue we compare set of 8 methods with randomly sampled architectures from their respective search spaces, and trained with the same protocol as the searched architectures.
other words, how can we be sure that a new
The ultimate goal behind NAS should be to return the optimal model for any dataset given, at least within the limits of a certain task, and we feel that the current practices searching almost exclu- sively on CIFARIO 10 go against this principle. Indeed, to avoid the very concrete risk of overfitting this set of data, NAS methods should be tested on a variety of tasks. For this reason we run experiments on 5 different datasets.
3.1 METHODOLOGY
Criteria for dataset selection. We selected datasets to cover a variety of subtasks within image classification. In addition to the standard CIFAR10 we select CIFAR100 for a more challenging object classification problem (Krizhevsky 2209); SPORTS for action classification (Li & Fei- 2007]; MIT67 for scene classification (Quattoni & Torralba 2009); and FLOWERSTO2 for fine- grained object classification (Nilsback /isserman 2008). More details are given in the Appendix.
Criteria for method selection. We selected methods which (a) have open-source code, or pro- vided it upon reguest, and (b) have reasonable running time, specifically search time under 4 GPU-days on CIFARIO. The selected methods are: DARTS (Liu et al. 2019), StacNAS (Li et al.
2019) PDARTS (Xin Chen 2019), MANAS (Carlucci et al., 2019), CNAS (Weng et al. 2019). NSGANET (Lu et al. 2018 ENAS (Pham et al.) 2018), and NAO (Luo et al. 2018). With the exception of NAO and NSGANET, all methods are DARTS variants and use weight sharing.
Evaluation protocol. NAS algorithms usually consist of two phases: (i) search, producing the best architecture according to the search algorithm used; (ii) augmentation, consisting in training from scratch the best model found in the search phase. We evaluate methods as follows: 1. Sample 8 architectures from the search space, uniformly at random, and use the method's code to augment these architectures (same augment seed for all); 2. Use the method's code to search for 8 architec- tures and augment them (different search seed, same augment seed); 3. Report mean and standard deviation of the top-1 test accuracy, obtained the end of the augmentation, for both the randomly sampled and the searched architectures;
Since both learned and randomly sampled architectures share the same search space and training protocol, calculating a relative improvement over this random baseline as RI = 100 x (Acem Acc,)/Acc, can offer insights into the quality of the search strategy alone. Acc,, and Acc, represent the top-1 accuracy of the search method and random sampling strategies, respectively. A good, general-purpose NAS method is expected yield RI consistently over different searches and across different subtasks. We emphasize that the comparison is not against random search, but rather against random sampling, i.e., the average architecture the search space. For example, in the DARTS search space, for each edge in the graph that defines a cell we select one out of eight possible operations (e.g. pooling or convolutions) with uniform probability 1/8.
Hyperparameters are optimized on CIFARIO, according to the values reported by the corresponding authors. Since most methods do not include their optimization as part of the search routine, we assumed them to be robust and generalizable to other tasks. As such, aside from scaling down the architecture depending on dataset size, experiments on other datasets use the same hyperparameters. Other training details and references are given in the Appendix.
3

---

Published as a conference paper at ICLR 2020
CIFAR1O
CIFAR100
SPORTB
Search methed, top-1 accuracy (%)
DARTS
X 84
95
CNAS NOGARE1
83
94
82
93
92
NAO
91
79
90
78
Search methed
89
77
88
76
87
86
Seare
75
85
74
94
92
random sampling, top-1 accuracy (%)
random sampling, top-1 accuracy (%)
random sampling, top-1 accuracy (%)
FLOWERS102
72
97
search method, tap-] accuracy
71
96
70
95
69
94
68
93
67
search method,
92
66
91
65
90
64
66
68
70
94
96
random sampling, top-1 accuracy (%)
randem sampling, top-1 accuracy (%)
Figure 1: Comparison of search methods and random sampling from their respective search spaces. Methods lying the diagonal perform the same as the average architecture, while methods above the diagonal outperform it. See also Table T.
(ordered by search method accuracy)
Table 1:
Relative improvement metric, RI 100 x (Accm - Acc,,) -Acc,)/Acc, (in %), where Acoo and Acc, the accuracies of the search method and random sampling baseline, respectively.
random sampling
search method
search cost (right axis]
search cost (GPU days)
98
tog-1 test accuracy
96-
e
C10
C100
M67
F102
DARTS
0.32
0.23
-0.13
0.10
0.25
95
PDARTS
0.52
1.20
0.51
1.19
0.20
94
NSGANET
-0.48
1.37
0.43
2.00
1.47
ENAS
0.01
-3.44
0.67
0.13
0.47
93
CNAS
0.74
-0.89
-1.06
-0.66
-2.48
MANAS
0.18
-0.20
0.33
1.48
0.70
CNAS
ENAS
NSGANET
NAO
PDARTS
MANAS
DARTS
StacNAS
StacNAS
0.43
2.87
0.38
0.05
-0.16
NAO
0.44
-0.01
-2.05
-1.53
-0.13
Figure Performance and computational
cost of the search phase on CIFAR10.
3.2 RESULTS
Figure shows the evaluation results on the datasets, from which we draw two main conclusions. First, the improvements over random sampling tend to be small. In some cases the average perfor- mance of method even below the average randomly sampled architecture, which suggests that the search methods are not converging desirable architectures. Second, the small range of accura- cies obtained hints at narrow search spaces, where even the worst architectures perform reasonably well. See Section 5 for more experiments corroborating this conclusion.
We observe also that, on CIFARIO, the top half of best best-performing methods (PDARTS, MANAS, DARTS, StacNAS) all perform similarly and positively in relation to their respective search spaces, but more variance is seen on the other datasets. This could be explained by the fact that most methods' hyperparameters have been optimized CIFAR10 and might not generalize well on
4

---

Published as a conference paper at ICLR 2020
different datasets. As matter of fact, we found that all NAS methods neglect report the time needed to optimize hyperparameters. In addition, Table shows the relative improvement metric RI (see intro Section for each method and dataset.
The computational cost of searching for architectures is a limiting factor in their applicability and, therefore, important variable in the evaluation of NAS algorithms. Figure shows the perfor- mance as well and the computational cost of the search phase on CIFAR10.
COMPARISON OF TRAINING PROTOCOLS
In this section we attempt to shed some light on the sur- prising results of the previous section. We noticed that there was much larger differences between the random
98.5
98.0
97.5
top-1 1op-1 accuracy (%)
baselines of different methods than the actual increase in performance of each approach. We hypothesized that how
97.0
96.5
a network is trained (the training protocol) has larger impact on the final accuracy than which architecture is trained, for each search space. To test this, we performed sensitivity analysis using the most common performance- boosting training protocols.
96.0
95.5
95.0
94.5
91.0
AuxilaaryTTwersiy
DrapPatn(D)
4.1 METHODOLOGY
Cuioul(C)
C+D
C+O+A [DARTS]
C+ S.-DAA*SSCChamelssssee
We decided to evaluate architectures from the commonly used DARTS search space (Liu et al.) 2019) on the CI- FAR10 dataset. We use the following process: 1) sample random architectures, 2) train them with different train- ing protocols (details below) and 3) report mean, standard deviation and maximum of the top-1 test accuracy at the end of the training process.
C+DDAA-AArrAuummennlA
C+O+A+6A+50C+
DARTS'S
Figure 3: Comparison of different aug- mentation protocols for the DARTS search space CIFAR1O. Same col- ored dots represent minimum and max- imum accuracies in the runs.
Training protocols. The simplest training protocol, which we will call Base is similar to the one used in DARTS, but with all tricks disabled: the model simply trained for 600 epochs. On the other extreme, our full protocol uses several tricks which have been used in recent works (Xie et al. 2019b] Nayman et al., 2019): Auxiliary Towers (A), DropPath (D:Land Larsson & Shakhnarovich 2017). Cutout (C: Vries & Taylor 2017), AutoAug- ment (AA; Cubuk et al.) 2018), extended training for 1500 epochs (1500E), and increased number of channels (50C). In between these two extremes, by selectively enabling and disabling each com- ponent, we evaluated a further 8 intermediate training protocols. When active, DropPath probability 0.2, cutout length is 16, auxiliary tower weight is 0.4, and AutoAugment combined with Cutout are used after standard data pre-processeee proccessesing techniques previously described, in Popien (2019).
4.2 RESULTS
As shown in Figure a large difference of over percentage points (p.p.) exists between the sim- plest and the most advanced training protocols. Indeed, this is much higher than any improvement over random sampling observed in the previous section: for example, on CIFAR1O, 10, the best im- provement observed was 0.69 p.p. In other words, the training protocol is often far more important than the architecture used. Note that the best accuracy of the 8 random architectures training with the best protocol is 98. 15%, which is only 0.25 p.p. below state-of-the-oe (Nayman et al. 2019).
To summarize, it seems that most recent ofate-of -hee---f-fhhhaaeeee art results, though impressive, cannot always be attributed to superior search strategies. Rather, they are often the result of expert knowledge applied to the evaluation protocol. In Figure 10) (Appendix A.3.II we show similar results when training a ResNet-50 (He et al. 2016) with the same protocols.
5

---

Published as a conference paper at ICLR 2020
1.0
Kendall tau tau rank correlation
top-1 accuracy (96)
0.8
80
epech 600
density
epoch 450
0.6
epoch
60
0.4
40
94
96
98
0.2
non-smoothed accuracy
top-1 1 test accuracy (%)
smoothed accuracy (w=20)
0.0
0 100
200
300
400
500
600
0
100
200
300
400
500
600
epoch
epoch
Figure 4: Training curves for the 214 randomly sampled architectures. Inset plot shows the his- togram of accuracies at different epochs.
Figure 5: Correlation between accuracies at dif- ferent epochs and final accuracy, using raw and smoothed accuracies over window
5 STUDY OF DARTS' SEARCH SPACE
5.1 DISTRIBUTION OF THE RANDOM SAMPLING WITHIN DARTS SEARCH SPACE
To better understand the results from the previous section, we sampled a considerable number of architectures (214) from the most commonly used search space (Liu al. 2019) and fully trained them with the matching training protocol Buutouut-DropPaat---Axlliil Towers). This allows us to get sense of how much variance exists between the different models (training statistics made available at the link in the abstract).
As observe from Figure 4 architectures sampled from this search space all perform similarly, with a mean 97.03 + 0.23. The worst architecture we found had an accuracy of 16.18, while the best achieved 97.56.
To put this into perspective, many methods using the same training protocol, fall within (or very close to) the standard deviation of the average architecture. Furthermore, we can observe in Figure 7 the number of cells (a human-picked hyperparameter) has a much larger impact on the final accuracy.
In Figure 5 we used the training statistics of the 214 models to plot the correlation between test accuracies at different epochs: it grows slowly in an almost linear fashion. We note that using the moving average of the accuracies yields stronger correlation, which could be useful for methods using early stopping.
5.2 OPERATIONS
To test whether the results from the previous section were due the choice of available operations, we developed an intentionally sub-optimal search space containing 4 plain convolutions (1 x 1, 3 3,, x 7, 11 11), max pooling operators (3
plus the none and skip connect operations. This proposed search space is clearly more parameter inefficient compared to the commonly used DARTS one (which uses both dilated and separable ones), and we expect to perform worse.
We sampled 56 architectures from this new search space and trained them with the DARTS train- ing protocol CCuoutt-rroPatth-uuiiie
Towers), for fair comparison with the results from the previous section. Figure 6 shows the resulting histogram, together with the one obtained from the classical DARTS space of operations. The two distributions are only shifted by 0.18 accuracy points. Given the minor difference in performance, the specific operations not key ingredient in the success of this search space. Very likely, it's the well engineered cell structure that allows the model perform well as it does.

---

Published as a conference paper at ICLR 2020
2.0
original modifled
best result
97.5
top-1 test accuracy
1.5
97.0
density
96.5
1.0
96.0
0.5
95.5
0.0
95.0
96.0
96.5
97.0
97.5
8 12
16
20
24
top-1 test accuracy
number cells
Figure 6: Histograms of the final accuracies (600 epochs) for architectures sampled from the DARTS search space (214 models) and our modified version (56 models).
Figure 7: Performance of 16 randomly sampled architectures with different numbers of cells. Error bars represent standard deviation.
Kendall-tau correlation: 0.48 (32 samples)
Kendall-tau correlation: 0.54 (32 samples)
30
25
25
20
20
15
15
10
10
5
0
B
20
seed
number of cells
Changes in the ranking of differ- architectures when trained with differ- ent seeds (A and B).
Figure 8:
Figure 9: Changes in the ranking of different architectures when trained with different num- bers of cells (same seed).
5.3 DOES CHANGING SEED AND NUMBER OF CELLS AFFECT RANKING?
A necessary practice for many weight-sharing methods (Liu et al. 2019 Pham et al.) 2018) is to restart the training from scratch after the search phase, with a different number of cells. Recent works have warned that this procedure might negatively affect ranking: similarly, the role of the seed has been previously recognized as as a fundamental element in reproducibility (Li & Talwalkar
2019 Sciuto al. 2019).
To test the impact of seed, we randomly sampled 32 architectures and trained them with two different seeds (Figure 8). Ranking is heavily influenced, as the Kendall tau correlation between the two sets of training is 0.48. On average, the test accuracy changes by 0.13% + 0.08 (max change is 0.39%), which is substantial considering the small gap between random architectures and NAS methods.
To test the depth-gap we trained another 32 with different number of cells (Figure D.. The corre- lation between the two different depths is not very strong as measured by Kendall Tau (0.54), with architectures shifting up and down the rankings by up 18 positions (out of 32). Methods employ- ing weight sharing (WS) would see an even more pronounced effect as the architectures normally chosen at 8 cells would have been training sub-optimally due to the WS itself (Sciuto et al. 2019).
These findings point towards two issues. The first is that since the seed has such a large effect on ranking, it stands to reason that the final accuracy reported should be averaged over multiple seeds. The second that, the lottery ticket hypothesis holds so that specific sub-networks are better mainly due to their lucky initialization; Frankle & Carbin. 20188 -together with findings, this could be an additional reason why methods searching on a different number of cells than the final model, struggle significantly improve the average randomly sampled architecture.

---

Published as a conference paper at ICLR 2020
6 DISCUSSION AND BEST PRACTICES
In this section we offer some suggestions on how to mitigate the issues in NAS research.
Augmention tricks: while achieving higher accuracies is clearly desirable goal, we have shown in section 4 that using well engineered training protocols can hide the contribution of the search al- gorithm. We therefore suggest that both results, with and without training tricks, should be reported. An example of best practice is found in Hundt et al. (2019).
Search Space:
it is difficult to evaluate the effectiveness of any given proposed method without measure of how good randomly sampled architectures are. This is not the same thing as performing random search which a search strategy in itself; random sampling is simply used to establish how good the average model is. A simple approach to measure the variability of any new given search space could be to randomly sample k architectures and report mean and standard deviation. We hope that future works will attempt to develop more expressive search spaces, capable of producing both good and bad network designs. Restricted search spaces, while guaranteeing good performance and quick results, will inevitably be constrained by the bounds of expert knowledge (local optima) and will be incapable of reaching more truly innovative solutions (closer to the global optima). As our findings in section 5.2 suggest, the overall wiring (the macro-structure) is an extremely influential component in the final performance. As such, future research could investigate the optimal wiring global level: an interesting work in this direction is Xie et al. (2119)).
Multiple datasets: as the true goal of AutoML is to minimize the need for human experts, focusing single dataset will inevitably lead to algorithmic overfitting and/or methods heavily dependent on hyperparameter tuning. The best solution for this is likely to test NAS algo- rithms on battery of datasets, with different characteristics: image sizes, number of samples, class granularity and learning task.
the research efforts
our experiments in Sections 4 and 5.2 show, the DARTS search space is not only effective due to specific operations that are being chosen, but in greater part due the overall macro-structure and the training protocol used. We suggest that proper ablation studies can lead to better understanding of the contributions of each element of the pipeline.
Investigating hidden components:
The importance of reproducibility: reproducibility is of extreme relevance in all sciences. To this end, it is very important that authors release not only their best found architecture but also the corresponding seed (if they did not average over multiple ones), as well as the code and the detailed training protocol (including hyperparameters). To this end, NAS- MAS-Bench-109 (Ying et al., 2019), dataset mapping architectures to their accuracy, can be extremely useful, as it allows the quality of search strategies to be assessed in isolation from other NAS components (e.g. search space, training protocol) quick and reproducible fashion. The code for this paper is open-source (link in the abstract). We also open-source the 270 trained architectures used in Section
Hyperparameter tuning cost: tuning hyperparameters in NAS is an extremely costly component. Therefore, we argue that either (i) hyperparameters are general enough so that they do not require tuning for further tasks, or (2) the cost included in the search budget.
7 CONCLUSIONS
AutoML. and NAS in particular, have the potential to truly democratize the use of machine learning for all, and could bring forth very notable improvements on a variety of tasks. To truly step forward, a principled approach, with a focus on fairness and reproducibility is needed.
In this paper we have shown that, for many NAS methods, the search space has been engineered such that all architectures perform similarly well and that their relative ranking can easily shift. We have furthermore showed that the training protocol itself has a higher impact on the final accuracy than the actual network. Finally, we have provided some suggestions on how to make future research more robust to these issues.
We hope that our findings will help the community focus their efforts towards a more general approach to automated neural architecture design. Only then can we expect to learn from NAS- generated architectures as opposed to the current paradigm where search spaces are heavily influ- enced by our current (human) expert knowledge.
8

---

Published as a conference paper at ICLR 2020
REFERENCES
Fabio Carlucci, Pedro M. Esperança, Marco Singh, Antoine Yang, Victor Gabillon, Hang Xu, Zewei Chen, and Jun Wang. MANAS: Multi-Agent Neural Architecture Search. arxiv: (009:/99900011 2019.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe Le. Autoaugment: Learning augmentation policies from data. arXi::/805.0950/ 2018.
Terrance DeVries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv:1708.0444 2017.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21, 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In arX.v::803.00000 00633, 2018.
Oguzhan Gencoglu, Mark van Gils, Esin Guldogan, Chamin Morikawa, Mehmet Süzen, Mathias Gruber, Jussi Leinonen, and Heikki Huttunen. Hark side of deep learning-from grad student descent automated machine learning. arXiv:1904.07633 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Zhang Xiangyu Rein Shaoqing He, Kaiming and Jian Sun. Delving deep into rectifiers: Surpassing human level performance imagenet classification. In Computer Vision and Pattern Recognition (CVPR), 2015.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pp. 646 666-661. Springer, 2016.
Andrew Hundt, Varun Jain, and Gregory D. Hager. sharpDARTS: Faster and more accurate differ- entiable architecture search. arXiv:1903.0900 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, pp. 1097-1105, 2012.
Michael Maire Larsson, Gustav and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. In International Conference on Learning Representations (ICLR), 2017.
Guilin Li, Xing Zhang, Zitong Wang, Zhenguo Li, and Tong Zhang. StacNAS: Towards stable and
consistent optimization for differentiable Neural Architecture Search, arXiv: /909: /926, 2019.
Li-Jia Li and Li Fei-Fei. What, where and who? classifying events by scene and object recognition.
In International Conference on Computer Vision (ICCV), pp. 1-8, 2007.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. arXhv:1902.07638, 2019.
Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang. and Zhenguo Li. DARTS+: Improved differentiable architecture search with early stopping. 1909005909 2019.
Marius Lindauer and Frank Hutter. Best practices for scientific research on neural architecture search. 1r09vv/909.0244 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference Learning Representations (ICLR), 2019.

---

Published as a conference paper at ICLR 2020
Iya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In Interna- tional Conference on Learning Representations (ICLR), 2017.
Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, and Wolfgang Banzhaf. Nsga-net: multi-objective genetic algorithm for neural architecture search. In GECCO-2019, 2018.
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural architecture optimization. In Advances in Neural Information Processing Systems (NIPS), pp. 7816-7827, 2018.
Niv Nayman, Asaf Noy, Tal Ridnik, Itamar Friedman, Rong Jin, and Lihi Zelnik-Manor. Xnas: Neural architecture search with expert advice. arXiv: 1900:00000003,, 2019.
Yurii E Nesterov. A method for solving the convex programming problem with convergence rate (1/K" 2). In Dokl. akad. nauk Sssr, volume 269, pp. 543-547, 1983.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Sixth Indian Conference Computer Vision, Graphics & Image Processing, pp-
722-729, 2008.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In International Conference on Machine Learning (ICML), pp- 4092-4101, 2018.
Philip Popien. AutoAugment. GitHub repository, http::///gthhbb.ccm/me/eeevaaatt
Aut.oAugment. 2019.
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In Computer Vision and Patlern Recognition (CVPR), pp. 413-420, 2009.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In International Conference on Machine Learning (ICML), pp. 2902-291 2017.
Christian Sciuto, Kaicheng Yu, Martin Jaggi. Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. 1002:0990.00144 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, (5(1):1929-199) 2014.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, Inception-Res and the impact of residual connections on learning. In AAAI Conference on Artificial Intelligence, 2017.
Yu Weng, Tianbao Zhou, Lei Liu, and Chunlei Xia. Automatic convolutional neural architecture search for image classification under different scenes. IEEE Access, 7:38495-38506, 2019.
Saining Xie, Alexander Kirillov, Ross Girshick, and Kaiming He. Exploring randomly wired neural networks for image recognition. arXiv preprint 194.011904 2019a.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: Stochastic neural architecture search. In International Conference on Learning Representations (ICLR), 2019b.
Jun Wu Qi Tian Xin Chen, Lingxi Xie. Progressive differentiable architecture search. In Interna- tional Conference on Computer Vision (ICCV), 2019.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS- Bench- 101: Towards reproducible neural architecture search. In International Conference Machine Learning (ICMI. 2019), pp. 7105-7114, 2019.
Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations (ICLR), 2017.
10

---

Published as a conference paper at ICLR 2020
Table 2: Hyperparameters for different NAS methods. "S" denotes search stage and "A" denotes augmentation stage. For learning rates, + means cosine annealing from a to b.
Method PDARTS
DARIS
StacNAS
MANAS
CNAS
batch size S batch size A init channels
64
64
96
64
64
96
96
128
128
64
16
16
16
16
16
init channels A
36
36
36
36
36
epochs $
100+100
25+25+25
50
epochs A
600
600
600
600
200
optimizer S/A
SGD
SGD
SGD
SGD
Adam
learning rates S learning rates A weight decay S/A optimizer arch learning rates arch weight decay arch nb cells $ CIFAR nb cells CIFAR
.025 4 .001
.025 4 .001
.025 .
.025 4 .001
.025-0003
.025 L .0
.025 .0
.025 ..0
.025 L
.025-0001
3 10-4
3 10-4
10
10
x 10
Adam
Adam
Adam
Adam
3 x 10-1
10-4 10-3
10-4
10
10-3
10-3
10
14/20
5/11/17
8
20
20
20
20
20
nb cells S other datasets
8
8/8
8/8/8
8
6
nb cells other datasets
8
8
8
nb intermediate nodes
4
4
4
4
6
A APPENDIX
This section details the datasets and the hyperparameters used for each method on each dataset. Search spaces were naturally left unchanged. Hyperparameters were chosen as close as possible to the original paper and occasionally updated to more recent implementations. The network size was tuned similarly for all methods for SPORT8, MIT67 and FLOWERS102. All experiments were run on NVIDIA Tesla V100 GPUs.
A.1 METHODS AND HYPERPARAMETERS
During search, models trained/validated the training/validation subsets, respectively During the final evaluation, the model is trained on the training+validation subsets and tested on the test subset.
Common hyperparameters. All methods share a common number of hyperparameters precised here. When SGD optimizer is used, momentum .9 while when Adam is used, momentum is [ = (0.5, 0.999). Gradient clipping is set at 5.
DARTS, StacNAS, PDARTS, MANAS, CNAS common hyperparameters. These methods inspired by DARTS code-wise and consequently share common number of hyperparameters, which we precise in table
It notably up- dates the official implementation to a pytorch version posterior to 0.4. Additional enhancements include cutout of size 16 (DeVries & Taylor 2017). path dropout of probability 0.2 (Larsson & Shakhnarovich 2017), and auxiliary tower with weight 0.4.
DARTS.
We used the following repository https//githb..com//aann/t.t.tt
StacNAS. We used unofficial implementation provided by the authors. The search process sists of 2 stages, of which the details are given in table 1, Additional enhancements are the same as DARTS.
We used the official implementation https://github..e process consists of stages, of which general details given in table 1. At stage 2 and 3 respectively, the number of operations decreases from 8 to to 3, and the dropout probability on skip-connect increases from 0.0 0.4 to 0.7 for CIFARIO, SPORTS, MTT67 and FLOWERS102
PDARTS.
The search

---

Published as a conference paper at ICLR 2020
(0.1 to 0.2 to 0.3 for CIFARI00). Discovered cells are restricted to keep most 2 skip-connect operations. Additional enhancements include cutout of size 16 (DeVries & Taylor 2017). DropPath of probability 0.3 (Larsson & Shakhnarovich 2017) and auxiliary tower with weight 0.4.
MANAS. We used an unofficial implementation provided by the authors. The reward bascline is 5, gamma is 0.1 (0.07 for SPORT8, 0.05 for FLOWERS102, 0.01 for MTT67) and the Boltzmann temperature decays from 1000 to 200 (300 to 100 for SPORT8, 5000 to 2000 for MIT67). Additional enhancements are the same as DARTS.
Label Smoothing is used with epsilon 0.1. Other additional enhancements include cutout of size 16 (De- Vries & Taylor 2017). DropPath of probability 0.25 (Larsson& & Shakhnarovich 2017).
CNAS. We used the official implementation : https//github.com/iinbbbcccooooe
https://github.comm//i htt.u//b.ooo//aaaalan/naaaaaeate The search is done in the miero search space (with 2 cells to search, 9 operations in the search space, 5 blocks in each cell) on 8 layers networks. The population size is 40, the number of generations is 30 and the number of offsprings created by generation is 20. Networks are trained for 20 epochs, with batch size 128, and the initial number of channels is 16. For architecture evaluation, the network is composed of 20 cells for CIFAR 10 and CIFAR 100, and 8 cells for SPORT8, MIT67 and FI FLOWERSIO 122. The final selected models are trained for 600 epochs with batch size 96 and the initial number of channels 34. Momentum SGD is used with initial learning rate The 0.025 (annealed down to zero following a cosine schedule), and weight decay 3 x 10 The filter increment is set to 4, and squeeze and excitation is used. Additional enhancements include cutout of size 16 VDevs & Taylor 2017), DropPath of probability 0.2 (Larsson & Shakhnarovich) 2017) and auxiliary tower with weight 0.4.
NSGANET. We used the official implementation:
pytorch. The LSTM model used to encode architecture has a token embedding size of 48 and a hidden state size of 96. The LSTM model used to decode architecture has hidden state size of 96. The encoder and decoder are trained using Adam for 1000 epochs with a learning rate of 0.001. The trade-off parameters
NAO. We used the official Pytorch implementation:
http://gituub..om/rengiaaaaaii
= 0.9. The step size to perform continuous optimization is 17 10. The number of nodes is fixed to 5, normal cell is stacked 3 (2 for SPORT8, MIT67 and FLOWERS 102) times to form the CNN architecture, which corresponds to 11 (8 for SPORTS, MIT67 and FLOWERS102) cells network and the initial number of channels is 20. Networks are trained for 100 epochs with batch size 64 for training, and validated for 20 epochs with batch size 500. For architecture evalua- tion, the final CNN architecture is a 20 cells network (8 cells network). This network is trained for 600 epochs with batch size 128 for the training set (96 for both for SPORTS, MIT67 and FLOW- ERS102), 500 for the validation set (128 for both for SPORTS, MIT67 and FLOWERS 102) and the initial number of channels is 36. Momentum SGD is used with initial learning rate T) = 0.025 (an- nealed down to zero following cosine schedule) and weight decay 3 x 10-3 Additional enhance- ments include cutout of size 16 (DeVs & Taylor| 2017). DropPath of probability 0.2 (Larsson & Shakhnarovich 2017). dropout of probability 0.4 (Srivastava et al. 2014), and auxiliary tower with
weight 0.4.
ENAS. We used the official tensorflow implementation for experiments on CIFARIO and CIFARI00: FLOWERS102 are done using more recent version in Pytorch for the search aaagg/iiaaaeaas-yyttotto
Experiments on SPORTS, MIT67 and
Because only the search process is implemented there, the evaluation code used is the same as DARTS. During the search, networks are composed of 8 cells. The shared parameters are trained with Nesterov momentum (Nesterov) 1983). weight decay 10-4, gradient clipping 5. batch size 160, 20 output filters, and a cosine learning rate schedule with an 0.05, 4n in 0.001, To 10, T,, ul = 2 (Loshchilov & Hutter, 2017). Each archi- tecture search is run for 150 epochs. w are initialized with He initialization (He & Sun 2015). The policy parameters& 0 are initialized uniformly in 1- 0.1], and trained with Adam at learning rate of 0.00035. A tanh constant of 1.10 and a temperature of 2.5 is applied to the controllers logits, and the controller entropy is added to the reward with weight 0.1. For the evaluation, the architecture searched is extended to 17 cells (8 for SPORTS, MIT67 and FLOWERS 102), trained for 630 epochs, with batch size 144, and a cosine learning rate schedule with /max 0.05, mmin = 0.001, To = 10, Tmul = 2.
12

---

Published as a conference paper at ICLR 2020
A.2 DATASETS
We present here the datasets used and how they are pré-processed.
dataset of 10 classes and consists of 50, 000 training images and 10, 000 test images of size 32 x.32.
CIFARIO. The CIFAR10 dataset (Krizhevsky 2009)
CIFAR100. The CIFARTI 100 dataset (Krizhevsky 2009) is dataset of 100 classes and consists of 50, 000 training images and 10, 000 test images of size 32x32.
Each of these datasets is split into a training, validation and testing subsets of size 25, 000, 25, 000 and 10, 000 respectively. For both these datasets, we use standard data pre-processing and augmen- tation techniques, i.e. subtracting the channel mean and dividing by the channel standard deviation; centrally padding the training images to 40 x 40 and randomly cropping them back to x2; and randomly clipping them horizontally.
SPORTS. This is an action recognition dataset containing 8 sport event categories and a total of 1579 images (Li & Fei-Fei 2007). The tiny size of this dataset stresses the generalization capabilities of any NAS method applied to it.
MIT67. This dataset of 67 classes representing different indoor scenes and consists of 15, 620 images of different sizes (Quattoni & Torralba 2009).
FLOWERS102. This dataset of 102 classes representing different species of flowers and consists of 8, 189 images of different sizes (Nilsback & Zisserman 2008).
Each of these datasets split into training, validation and testing subsets with proportions 40/40/20 (%) For each one, we use use standard data pre-processing and augmentation tech- niques, i.e. subtracting the channel mean and dividing the channel standard deviation, cropping the training images random size and aspect ratio, resizing them to 224x224, and randomly changing their brightness, contrast, and saturation, while resizing test images to 256 and cropping them at the center.
A.3 ADDITIONAL RESULTS
A.3.1
DIFFERENT TRAINING PROTOCOLS FOR RESNET-50
98
Base
Auxiliary Towers(A) DrapPath(D) Cutout(C)
69 accuracy
C+D
C+D++A [DARTS]
D-A 50Charmeis5550)
++0+A++ C+D-AAAAUuAAAgggenntt
C+ +AD+++AA+50C
DARTS's best
Figure 10: Extension of Figure 3 including results obtained by training a ResNet-50 on CIFAR1O. Bars with darker shade are for ResNet-50 and bars with lighter shade are for DARTS (same as Figure 3 Result are for 8 runs of each training protocol. For ResNet-50, the auxiliary tower was added after layer 2. As DropPath (Larsson & Shakhnarovich 2017) would not have been straightforward to apply, we instead implemented Stocastic Depth (Huang et al. 2016). to a similar
effect.
13


Reviews:
[{"review_id": "HygAVxO3OH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this submission, the authors conduct a series of experiments on five image classification datasets to compare several existing NAS methods. Based on these experimental results, they point out: 1) how a network is trained (i.e., training protocols/tricks such as DropPath, Cutout) plays an important role for the final accuracy; 2) within the search space, the existing NAS methods perform close to or slightly better than a random sampling baseline; 3) hyperparameters of NAS methods also have significant effect on the performance. \n\nWith these interesting findings, I suggest rejecting this submission. The reasons are as follows:\n1) For the first finding of training protocol, several existing papers and books already discussed it, such as Li & Talwalkar (2019) and the book chapter {Neural Architecture Search} by Thomas, Jan Hendrik and Frank.\n\n2) For the second finding of the search space and the performance of a randomly sampled architecture, existing work from Facebook AI Research group has studied this. And the existing work gives more experiments and discussion than this submission (from my own perspective). https://arxiv.org/pdf/1904.01569.pdf\n\n3) The conducted experiments in this submission also have certain risks to support the claims/conclusions of it. For example, only datasets of image classification are adopted. Another factor is the hyper-parameter tuning (actually, the authors also mention this in the last paragraph on Page 4). All the compared methods, either NAS methods or random sample baseline, should receive the same training procedure to get a fair experimental comparison. \n\nThe above mentioned existing work makes the contributions of this submission less, and the experimental results may not be convincing enough. These lead to a reject.\n\nHowever, a great point is made by the authors in the last paragraph of Section 6: hyperparameter of NAS methods should be either stable enough or counted toward the cost.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Hyezhj16YS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a benchmark of 8 the-state-of-the-art NAS methods (DARTS, StacNAS, PDARTS, MANAS, CNAS, NSGANET, ENAS, NAO) on 5 datasets (CIFAR10, CIFAR100, SPORT8, MIT67, FLOWERS102). The paper first points out how fair comparisons of NAS methods is difficult, especially those with different search spaces. The paper proposes making relative comparisons to random \"samples\" of architectures in search space to remove advantages of expertly engineered search spaces or training protocols. Also, the paper further investigates the case of commonly used the DARTS search space through ablation studies. The results suggest that many sensitive factors such as tricks in evaluation protocols, random seeds, hand-designed macro structures, and depth gaps can have predominant impacts rather than primary factors of NAS such as search strategy. The paper concludes with best practices to mitigate these disturbing factors to design NAS with reproducibility.\n\nThis is a very nice paper with extensive empirical evaluations. The topic of empirical comparisons of NAS algorithms is already very difficult to tackle in a fair way, but it gives thought-provoking strategies to evaluate the target NAS algorithms. Also, the fact that even random sampling (without search) provides an incredibly competitive baseline is quite informative and gives a very important recognition on how to design and evaluate NAS. \n\nThe paper is well written and well organized, and I have no problems to report, but would like to make sure one thing. In section 4.1, the same 8 initial random architectures from DARTS search space are used for all of the variants? Since the non-negligible impact of random seeds is reported, I just wonder how seeds are controlled in individual experiments.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1xfPBr3FH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper scrutinizes commonly used evaluation strategies for neural architecture search.\nThe first contribution is to compare architectures found by 5 different search strategies from the literature against randomly sampled architectures from the same underlying search space.\nThe paper shows that across different image classification datasets, the most neural architecture search methods are not consistently finding solutions that achieve a better performance than random architectures.\nThe second major contribution of the paper is to show that the state-of-the-art performance achieved by the many neural architecture search methods can be largely attributed to advanced regularization tricks.\n\n\nIn general the paper is well written and easy to follow.\nThe paper sheds a rather grim light on the current state of neural architecture search, but I think it could raise awareness of common pitfalls and help to make future work more rigorous.\nWhile poorly designed search spaces is maybe a problem that many people in the community are aware of, this is, to the best of my knowledge, the first paper that systematically shows that for several commonly used search spaces there is not much to be optimized.\nBesides that, the paper shows that, maybe not surprisingly, the training protocol seems to be more important than the actual architecture search, which I also found rather worrisome.\nIt would be nice, if Figure 3 could include another bar that shows the performance of a manually designed architecture, for example a residual network, trained with the same regularization techniques.\n\nA caveat of the paper is that mostly local methods with weight-sharing are considered, instead of more global methods, such as evolutionary algorithms or Bayesian optimization, which also showed strong performance on neural architecture search problems in the past.\nFurthermore, the paper doesn't mention some recent work in neural architecture search that present more thoroughly designed search spaces, e.g Ying et al.\nIt would also be helpful if the paper could elaborate, on how better search spaces could be designed.\n\nNas-bench-101: Towards reproducible neural architecture search\nC Ying, A Klein, E Real, E Christiansen, K Murphy, F Hutter\nICML 2019\n\n\nMinor comments:\n\n- Section 4.1 How are the hyperparameters of drop path probability, cutout length and auxiliary tower weight chosen?\n\n\n\npost rebuttal\n------------------\n\nI thank the authors for taking the time to answer my questions and performing the additional experiments. The paper highlights two important issues in the current NAS literature: evaluating methods with different search spaces and non-standardised training protocols. I do think that the paper makes an important contribution which hopefully helps future work to over come these flaws.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary:
This paper provides comprehensive empirical evidence for some of the systemic issues in the NAS community, for example showing that several published NAS algorithms do not outperform random sampling on previously unseen data and that the training pipeline is more important in the DARTS space than the exact choice of neural architecture. I very much appreciate that code is available for reproducibility.

Reviewer scores and discussion: 
The reviewers' scores have very high variance: 2/3 reviewers gave clear acceptance scores (8,8), very much liking the paper, whereas one reviewer gave a clear rejection score (1). In the discussion between the reviewers and the AC, despite the positive comments of the other reviewers, AnonReviewer 2 defended his/her position, arguing that the novelty is too low given previous works. The other reviewers argued against this, emphasizing that it is an important contribution to show empirical evidence for the importance of the training protocol (note that the intended contribution is *not* to introduce these training protocols; they are taken from previous work).

Due to the high variance, I read the paper myself in detail. Here are my own two cents:
- It is not new to compare to a single random sample. Sciuto et al clearly proposed this first; see Figure 1 (c) in https://arxiv.org/abs/1902.08142 
- The systematic experiments showing the importance of the training pipeline are very useful, providing proper and much needed empirical evidence for the many existing suggestions that this might be the case. Figure 3 is utterly convincing.
- Throughout, it would be good to put the work into perspective a bit more. E.g., correlations have been studied by many authors before. Also, the paper cites the best practice checklist in the beginning, but does not mention it in the section on best practices (my view is that this paper is in line with that checklist and provides important evidence for several points in it; the checklist also contains other points not being discussed in this paper; it would be good to know whether this paper suggests any new points for the checklist).

Recommendation:
Overall, I firmly believe that this paper is an important contribution to the NAS community. It may be viewed by some as "just" running some experiments, but the experiments it shows are very informative and will impact the community and help guide it in the right direction. I therefore recommend acceptance (as a poster).
Decision: Accept (Poster)