user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: NAS evaluation is frustratingly hard

Paper content:
Neural Architecture Search (NAS) is an exciting new field which promises to be as much as a game-changer as Convolutional Neural Networks were in 2012.
Despite many great works leading to substantial improvements on a variety of tasks, comparison between different methods is still very much an open issue.
While most algorithms are tested on the same datasets, there is no shared experimental protocol followed by all.
As such, and due to the under-use of ablation studies, there is a lack of clarity regarding why certain methods are more effective than others.
Our first contribution is a benchmark of 8 NAS methods on 5 datasets.
To overcome the hurdle of comparing methods with different search spaces, we propose using a method’s relative improvement over the randomly sampled average architecture, which effectively removes advantages arising from expertly engineered search spaces or training protocols.
Surprisingly, we find that many NAS techniques struggle to significantly beat the average architecture baseline.
We perform further experiments with the commonly used DARTS search space in order to understand the contribution of each component in the NAS pipeline.
These experiments highlight that:
(i) the use of tricks in the evaluation protocol has a predominant impact on the reported performance of architectures;
(ii) the cell-based search space has a very narrow accuracy range, such that the seed has a considerable impact on architecture rankings;
(iii) the hand-designed macrostructure (cells) is more important than the searched micro-structure (operations); and
(iv) the depth-gap is a real phenomenon, evidenced by the change in rankings between 8 and 20 cell architectures.
To conclude, we suggest best practices, that we hope will prove useful for the community and help mitigate current NAS pitfalls, e.g. difficulties in reproducibility and comparison of search methods.
The
code used is available at https://github.com/antoyang/NAS-Benchmark.
As the deep learning revolution helped us move away from hand crafted features (Krizhevsky et al., 2012) and reach new heights (He et al., 2016; Szegedy et al., 2017) , so does Neural Architecture Search (NAS) hold the promise of freeing us from hand-crafted architectures, which requires tedious and expensive tuning for each new task or dataset.
Identifying the optimal architecture is indeed a key pillar of any Automated Machine Learning (AutoML) pipeline.
Research in the last two years has proceeded at a rapid pace and many search strategies have been proposed, from Reinforcement Learning (Zoph & Le, 2017; Pham et al., 2018) , to Evolutionary Algorithms (Real et al., 2017) , to Gradient-based methods Liang et al., 2019) .
Still, it remains unclear which approach and search algorithm is preferable.
Typically, methods have been evaluated on accuracy alone, even though accuracy is influenced by many other factors besides the search algorithm.
Comparison between published search algorithms for NAS is therefore either very difficult (complex training protocols with no code available) or simply impossible (different search spaces), as previously pointed out (Li & Talwalkar, 2019; Sciuto et al., 2019; Lindauer & Hutter, 2019) .
NAS methods have been typically decomposed into three components (Elsken et al., 2019; Li & Talwalkar, 2019) : search space, search strategy and model evaluation strategy.
This division is important to keep in mind, as an improvement in any of these elements will lead to a better final performance.
But is a method with a more (manually) tuned search space a better AutoML algorithm?
If the key idea behind NAS is to find the optimal architecture, without human intervention, why are we devoting so much energy to infuse expert knowledge into the pipeline?
Furthermore, the lack of ablation studies in most works makes it harder to pinpoint which components are instrumental to the final performance, which can easily lead to Hypothesizing After the Results are Known (HARKing; Gencoglu et al., 2019) .
Paradoxically, the huge effort invested in finding better search spaces and training protocols, has led to a situation in which any randomly sampled architecture performs almost as well as those obtained by the search strategies.
Our findings suggest that most of the gains in accuracy in recent contributions to NAS have come from manual improvements in the training protocol, not in the search algorithms.
As a step towards understanding which methods are more effective, we have collected code for 8 reasonably fast (search time of less than 4 days) NAS algorithms, and benchmarked them on 5 well known CV datasets.
Using a simple metric-the relative improvement over the average architecture of the search space-we find that most NAS methods perform very similarly and rarely substantially above this baseline.
The methods used are DARTS, StacNAS, PDARTS, MANAS, CNAS, NSGANET, ENAS and NAO.
The datasets used are CIFAR10, CIFAR100, SPORT8, MIT67 and FLOWERS102.
Through a number of additional experiments on the widely used DARTS search space , we will show that:
(a) how you train your model has a much bigger impact than the actual architecture chosen;
(b) different architectures from the same search space perform very similarly, so much so that
(c) hyperparameters, like the number of cells, or the seed itself have a very significant effect on the ranking; and
(d) the specific operations themselves have less impact on the final accuracy than the hand-designed macro-structure of the network.
Notably, we find that the 200+ architectures sampled from this search space (available from the link in the abstract) are all within a range of one percentage point (top-1 accuracy) after a standard full training on CIFAR10.
Finally, we include some observations on how to foster reproducibility and a discussion on how to potentially avoid some of the encountered pitfalls.
In this section we offer some suggestions on how to mitigate the issues in NAS research.
Augmention tricks: while achieving higher accuracies is clearly a desirable goal, we have shown in section 4, that using well engineered training protocols can hide the contribution of the search algorithm.
We therefore suggest that both results, with and without training tricks, should be reported.
An example of best practice is found in Hundt et al. (2019) .
Search Space: it is difficult to evaluate the effectiveness of any given proposed method without a measure of how good randomly sampled architectures are.
This is not the same thing as performing a random search which is a search strategy in itself; random sampling is simply used to establish how good the average model is.
A simple approach to measure the variability of any new given search space could be to randomly sample k architectures and report mean and standard deviation.
We hope that future works will attempt to develop more expressive search spaces, capable of producing both good and bad network designs.
Restricted search spaces, while guaranteeing good performance and quick results, will inevitably be constrained by the bounds of expert knowledge (local optima) and will be incapable of reaching more truly innovative solutions (closer to the global optima).
As our findings in section 5.2 suggest, the overall wiring (the macro-structure) is an extremely influential component in the final performance.
As such, future research could investigate the optimal wiring at a global level: an interesting work in this direction is Xie et al. (2019a) .
Multiple datasets: as the true goal of AutoML is to minimize the need for human experts, focusing the research efforts on a single dataset will inevitably lead to algorithmic overfitting and/or methods heavily dependent on hyperparameter tuning.
The best solution for this is likely to test NAS algorithms on a battery of datasets, with different characteristics: image sizes, number of samples, class granularity and learning task.
Investigating hidden components: as our experiments in Sections 4 and 5.2 show, the DARTS search space is not only effective due to specific operations that are being chosen, but in greater part due to the overall macro-structure and the training protocol used.
We suggest that proper ablation studies can lead to better understanding of the contributions of each element of the pipeline.
The importance of reproducibility: reproducibility is of extreme relevance in all sciences.
To this end, it is very important that authors release not only their best found architecture but also the corresponding seed (if they did not average over multiple ones), as well as the code and the detailed training protocol (including hyperparameters).
To this end, NAS-Bench-101 (Ying et al., 2019) , a dataset mapping architectures to their accuracy, can be extremely useful, as it allows the quality of search strategies to be assessed in isolation from other NAS components (e.g. search space, training protocol) in a quick and reproducible fashion.
The code for this paper is open-source (link in the abstract).
We also open-source the 200+ trained architectures used in Section 5.
Hyperparameter tuning cost: tuning hyperparameters in NAS is an extremely costly component.
Therefore, we argue that either (i) hyperparameters are general enough so that they do not require tuning for further tasks, or (2) the cost is included in the search budget.
AutoML, and NAS in particular, have the potential to truly democratize the use of machine learning for all, and could bring forth very notable improvements on a variety of tasks.
To truly step forward, a principled approach, with a focus on fairness and reproducibility is needed.
In this paper we have shown that, for many NAS methods, the search space has been engineered such that all architectures perform similarly well and that their relative ranking can easily shift.
We have furthermore showed that the training protocol itself has a higher impact on the final accuracy than the actual network.
Finally, we have provided some suggestions on how to make future research more robust to these issues.
We hope that our findings will help the community focus their efforts towards a more general approach to automated neural architecture design.
Only then can we expect to learn from NASgenerated architectures as opposed to the current paradigm where search spaces are heavily influenced by our current (human) expert knowledge.
A APPENDIX This section details the datasets and the hyperparameters used for each method on each dataset.
Search spaces were naturally left unchanged.
Hyperparameters were chosen as close as possible to the original paper and occasionally updated to more recent implementations.
The network size was tuned similarly for all methods for SPORT8, MIT67 and FLOWERS102.
All experiments were run on NVIDIA Tesla V100 GPUs.

assistant: text: A study of how different components in the NAS pipeline contribute to the final accuracy. Also, a benchmark of 8 methods on 5 datasets.