NAS evaluation is frustratingly hard

1 INTRODUCTION . As the deep learning revolution helped us move away from hand crafted features ( Krizhevsky et al. , 2012 ) and reach new heights ( He et al. , 2016 ; Szegedy et al. , 2017 ) , so does Neural Architecture Search ( NAS ) hold the promise of freeing us from hand-crafted architectures , which requires tedious and expensive tuning for each new task or dataset . Identifying the optimal architecture is indeed a key pillar of any Automated Machine Learning ( AutoML ) pipeline . Research in the last two years has proceeded at a rapid pace and many search strategies have been proposed , from Reinforcement Learning ( Zoph & Le , 2017 ; Pham et al. , 2018 ) , to Evolutionary Algorithms ( Real et al. , 2017 ) , to Gradient-based methods ( Liu et al. , 2019 ; Liang et al. , 2019 ) . Still , it remains unclear which approach and search algorithm is preferable . Typically , methods have been evaluated on accuracy alone , even though accuracy is influenced by many other factors besides the search algorithm . Comparison between published search algorithms for NAS is therefore either very difficult ( complex training protocols with no code available ) or simply impossible ( different search spaces ) , as previously pointed out ( Li & Talwalkar , 2019 ; Sciuto et al. , 2019 ; Lindauer & Hutter , 2019 ) . NAS methods have been typically decomposed into three components ( Elsken et al. , 2019 ; Li & Talwalkar , 2019 ) : search space , search strategy and model evaluation strategy . This division is ∗antoineyang3 @ gmail.com , { pedro.esperanca , fabio.maria.carlucci } @ huawei.com †This work was done when the first author was an intern at Huawei Noah ’ s Ark Lab , London , United Kingdom . important to keep in mind , as an improvement in any of these elements will lead to a better final performance . But is a method with a more ( manually ) tuned search space a better AutoML algorithm ? If the key idea behind NAS is to find the optimal architecture , without human intervention , why are we devoting so much energy to infuse expert knowledge into the pipeline ? Furthermore , the lack of ablation studies in most works makes it harder to pinpoint which components are instrumental to the final performance , which can easily lead to Hypothesizing After the Results are Known ( HARKing ; Gencoglu et al. , 2019 ) . Paradoxically , the huge effort invested in finding better search spaces and training protocols , has led to a situation in which any randomly sampled architecture performs almost as well as those obtained by the search strategies . Our findings suggest that most of the gains in accuracy in recent contributions to NAS have come from manual improvements in the training protocol , not in the search algorithms . As a step towards understanding which methods are more effective , we have collected code for 8 reasonably fast ( search time of less than 4 days ) NAS algorithms , and benchmarked them on 5 well known CV datasets . Using a simple metric—the relative improvement over the average architecture of the search space—we find that most NAS methods perform very similarly and rarely substantially above this baseline . The methods used are DARTS , StacNAS , PDARTS , MANAS , CNAS , NSGANET , ENAS and NAO . The datasets used are CIFAR10 , CIFAR100 , SPORT8 , MIT67 and FLOWERS102 . Through a number of additional experiments on the widely used DARTS search space ( Liu et al. , 2019 ) , we will show that : ( a ) how you train your model has a much bigger impact than the actual architecture chosen ; ( b ) different architectures from the same search space perform very similarly , so much so that ( c ) hyperparameters , like the number of cells , or the seed itself have a very significant effect on the ranking ; and ( d ) the specific operations themselves have less impact on the final accuracy than the hand-designed macro-structure of the network . Notably , we find that the 200+ architectures sampled from this search space ( available from the link in the abstract ) are all within a range of one percentage point ( top-1 accuracy ) after a standard full training on CIFAR10 . Finally , we include some observations on how to foster reproducibility and a discussion on how to potentially avoid some of the encountered pitfalls . 2 RELATED WORK . As mentioned , NAS methods have the potential to truly revolutionize the field , but to do so it is crucial that future research avoids common mistakes . Some of these concerns have been recently raised by the community . For example , Li & Talwalkar ( 2019 ) highlight that most NAS methods a ) fail to compare against an adequate baseline , such as a properly implemented random search strategy , b ) are overly complex , with no ablation to properly assign credit to the important components , and c ) fail to provide all details needed for successfully reproducing their results . In our paper we go one step further and argue that the relative improvement over the average ( randomly sampled ) architecture is an useful tool to quantify the effectiveness of a proposed solution and compare it with competing methods . To partly answer their second point , and understand how much the final accuracy depends on the specific architecture , we implement an in-depth study of the widely employed DARTS ( Liu et al. , 2019 ) search space and perform an ablation on the commonly used training techniques ( e.g . Cutout , DropPath , AutoAugment ) . In addition , Sciuto et al . ( 2019 ) also took the important step of systematically using fair baselines , and suggest random search with early stopping , averaged over multiple seeds , as an extremely competitive baseline . They find that the search spaces of three methods investigated ( DARTS , ENAS , NAO ) have been expertly engineered to the extent that any randomly selected architecture performs very well . In contrast , we show that even random sampling ( without search ) provides an incredibly competitive baseline . Our relative improvement metric allows us to isolate the contribution of the search strategy from the effects of the search space and training pipeline . Thus , we further confirm the authors ’ claim , showing that indeed the average architecture performs extremely well and that how you train a model has more impact than any specific architecture . 3 NAS BENCHMARK . In this section we present a systematic evaluation of 8 methods on 5 datasets using a strategy that is designed to reveal the quality of each method ’ s search strategy , removing the effect of the manuallyengineered training protocol and search space . The goal is to find general trends and highlight common features rather than just pin-pointing the most accurate algorithm . Understanding why methods are effective is not an easy task : most introduce variations to previous search spaces , search strategies , and training protocols—with ablations disentangling the contribution of each component often incomplete or missing . In other words , how can we be sure that a new state-of-the-art method is not so simply due to a better engineered search space or training protocol ? To address this issue we compare a set of 8 methods with randomly sampled architectures from their respective search spaces , and trained with the same protocol as the searched architectures . The ultimate goal behind NAS should be to return the optimal model for any dataset given , at least within the limits of a certain task , and we feel that the current practices of searching almost exclusively on CIFAR10 go against this principle . Indeed , to avoid the very concrete risk of overfitting to this set of data , NAS methods should be tested on a variety of tasks . For this reason we run experiments on 5 different datasets . 3.1 METHODOLOGY . Criteria for dataset selection . We selected datasets to cover a variety of subtasks within image classification . In addition to the standard CIFAR10 we select CIFAR100 for a more challenging object classification problem ( Krizhevsky , 2009 ) ; SPORT8 for action classification ( Li & Fei-Fei , 2007 ) ; MIT67 for scene classification ( Quattoni & Torralba , 2009 ) ; and FLOWERS102 for finegrained object classification ( Nilsback & Zisserman , 2008 ) . More details are given in the Appendix . Criteria for method selection . We selected methods which ( a ) have open-source code , or provided it upon request , and ( b ) have a reasonable running time , specifically a search time under 4 GPU-days on CIFAR10 . The selected methods are : DARTS ( Liu et al. , 2019 ) , StacNAS ( Li et al. , 2019 ) , PDARTS ( Xin Chen , 2019 ) , MANAS ( Carlucci et al. , 2019 ) , CNAS ( Weng et al. , 2019 ) , NSGANET ( Lu et al. , 2018 ) , ENAS ( Pham et al. , 2018 ) , and NAO ( Luo et al. , 2018 ) . With the exception of NAO and NSGANET , all methods are DARTS variants and use weight sharing . Evaluation protocol . NAS algorithms usually consist of two phases : ( i ) search , producing the best architecture according to the search algorithm used ; ( ii ) augmentation , consisting in training from scratch the best model found in the search phase . We evaluate methods as follows : 1 . Sample 8 architectures from the search space , uniformly at random , and use the method ’ s code to augment these architectures ( same augment seed for all ) ; 2 . Use the method ’ s code to search for 8 architectures and augment them ( different search seed , same augment seed ) ; 3 . Report mean and standard deviation of the top-1 test accuracy , obtained at the end of the augmentation , for both the randomly sampled and the searched architectures ; Since both learned and randomly sampled architectures share the same search space and training protocol , calculating a relative improvement over this random baseline as RI = 100 × ( Accm − Accr ) /Accr can offer insights into the quality of the search strategy alone . Accm andAccr represent the top-1 accuracy of the search method and random sampling strategies , respectively . A good , general-purpose NAS method is expected to yield RI > 0 consistently over different searches and across different subtasks . We emphasize that the comparison is not against random search , but rather against random sampling , i.e. , the average architecture of the search space . For example , in the DARTS search space , for each edge in the graph that defines a cell we select one out of eight possible operations ( e.g . pooling or convolutions ) with uniform probability 1/8 . Hyperparameters are optimized on CIFAR10 , according to the values reported by the corresponding authors . Since most methods do not include their optimization as part of the search routine , we assumed them to be robust and generalizable to other tasks . As such , aside from scaling down the architecture depending on dataset size , experiments on other datasets use the same hyperparameters . Other training details and references are given in the Appendix . 3.2 RESULTS . Figure 1 shows the evaluation results on the 5 datasets , from which we draw two main conclusions . First , the improvements over random sampling tend to be small . In some cases the average performance of a method is even below the average randomly sampled architecture , which suggests that the search methods are not converging to desirable architectures . Second , the small range of accuracies obtained hints at narrow search spaces , where even the worst architectures perform reasonably well . See Section 5 for more experiments corroborating this conclusion . We observe also that , on CIFAR10 , the top half of best-performing methods ( PDARTS , MANAS , DARTS , StacNAS ) all perform similarly and positively in relation to their respective search spaces , but more variance is seen on the other datasets . This could be explained by the fact that most methods ’ hyperparameters have been optimized on CIFAR10 and might not generalize as well on different datasets . As a matter of fact , we found that all NAS methods neglect to report the time needed to optimize hyperparameters . In addition , Table 1 shows the relative improvement metric RI ( see intro to Section 3 ) for each method and dataset . The computational cost of searching for architectures is a limiting factor in their applicability and , therefore , an important variable in the evaluation of NAS algorithms . Figure 2 shows the performance as well and the computational cost of the search phase on CIFAR10 . 4 COMPARISON OF TRAINING PROTOCOLS In this section we attempt to shed some light on the surprising results of the previous section . We noticed that there was a much larger differences between the random baselines of different methods than the actual increase in performance of each approach . We hypothesized that how a network is trained ( the training protocol ) has a larger impact on the final accuracy than which architecture is trained , for each search space . To test this , we performed sensitivity analysis using the most common performanceboosting training protocols .

Summary: This paper provides a benchmark of 8 the-state-of-the-art NAS methods (DARTS, StacNAS, PDARTS, MANAS, CNAS, NSGANET, ENAS, NAO) on 5 datasets (CIFAR10, CIFAR100, SPORT8, MIT67, FLOWERS102). The paper first points out how fair comparisons of NAS methods is difficult, especially those with different search spaces. The paper proposes making relative comparisons to random "samples" of architectures in search space to remove advantages of expertly engineered search spaces or training protocols. Also, the paper further investigates the case of commonly used the DARTS search space through ablation studies. The results suggest that many sensitive factors such as tricks in evaluation protocols, random seeds, hand-designed macro structures, and depth gaps can have predominant impacts rather than primary factors of NAS such as search strategy. The paper concludes with best practices to mitigate these disturbing factors to design NAS with reproducibility.