Evaluating The Search Phase of Neural Architecture Search

1 INTRODUCTION . By automating the design of a neural network for the task at hand , Neural Architecture Search ( NAS ) has tremendous potential to impact the practicality of deep learning ( Zoph & Le , 2017 ; Liu et al. , 2018b ; a ; Tan et al. , 2018 ; Baker et al. , 2016 ) , and has already obtained state-of-the-art performance on many tasks . A typical NAS technique ( Zoph & Le , 2017 ; Pham et al. , 2018 ; Liu et al. , 2018a ) has two stages : the search phase , which aims to find a good architecture , and the evaluation one , where the best architecture is trained from scratch and validated on the test data . In the literature , NAS algorithms are typically compared based on their results in the evaluation phase . While this may seem intuitive , the search phase of these algorithms often differ in several ways , such as their architecture sampling strategy and the search space they use , and the impact of these individual factors can not be identified by looking at the downstream task results only . Furthermore , the downstream task results are often reported for a single random seed , which leaves unanswered the question of robustness of the search strategies . ∗Equal contribution †Work done at Swisscom digital lab . In this paper , we therefore propose to investigate the search phase of existing NAS algorithms in a controlled manner . To this end , we compare the quality of the NAS solutions with a random search policy , which uniformly randomly samples an architecture from the same search space as the NAS algorithms , and then trains it using the same hyper-parameters as the NAS solutions . To reduce randomness , the search using each policy , i.e. , random and NAS ones , is repeated several times , with different random seeds . We perform a series of experiments on the Penn Tree Bank ( PTB ) ( Marcus et al. , 1994a ) and CIFAR10 ( Krizhevsky et al. , 2009 ) datasets , in which we compared the state-of-the-art NAS algorithms whose code is publicly available—DARTS ( Liu et al. , 2019b ) , NAO ( Luo et al. , 2018 ) and ENAS ( Pham et al. , 2018 ) —to our random policy . We reached the surprising conclusions that , as shown in Table 1 , none of them significantly outperforms random sampling . Since the mean performance for randomlysampled architectures converges to the mean performance over the entire search space , we further conducted Welch Student ’ s t-tests ( Welch , 1947 ) , which reveal that , in RNN space , ENAS and DARTS can not be differentiated from the mean of entire search space , while NAO yields worse performance than random sampling . While the situation is slightly better in CNN space , all three algorithms still perform similarly to random sampling . Note that this does not necessarily mean that these algorithms perform poorly , but rather that the search space has been sufficiently constrained so that even a random architecture in this space provides good results . To verify this , we experiment with search spaces where we can exhaustively evaluate all architectures , and observe that these algorithms truly can not discover top-performing architectures . In addition to this , we observed that the ranking by quality of candidate architectures produced by the NAS algorithms during the search does not reflect the true performance of these architectures in the evaluation phase . Investigating this further allowed us to identify that weight sharing ( Pham et al. , 2018 ) , widely adopted to reduce the amount of required resources from thousands of GPU days to a single one , harms the individual networks ’ performance . More precisely , using reduced search spaces , we make use of the Kendall Tau τ metric1 to show that the architecture rankings obtained with and without weight sharing are entirely uncorrelated in RNN space ( τ = -0.004 over 10 runs ) ; and have little correlation in the CNN space ( τ = 0.195 over 10 runs ) . Since such a ranking is usually treated as training data for the NAS sampler in the search phase , this further explains the small margin between random search and the NAS algorithms . We also show that training samplers without weight sharing in CNN space surpasses random sampling by a significant margin . In other words , we disprove the common belief that the quality of architectures trained with and without weight sharing is similar . We show that the difference in ranking negatively impacts the search phase of NAS algorithms , thus seriously impeding their robustness and performance . In short , evaluating the search phase of NAS , which is typically ignored , allowed us to identify two key characteristics of state-of-the-art NAS algorithms : The importance of the search space and the negative impact of weight sharing . We believe that our evaluation framework will be instrumental in designing NAS search strategies that are superior to the random one . Our code is publicly available at https : //github.com/kcyu2014/eval-nas . 2 RELATED WORK . Since its introduction in ( Zoph & Le , 2017 ) , NAS has demonstrated great potential to surpass the human design of deep networks for both visual recognition ( Liu et al. , 2018b ; Ahmed & Torresani , 2018 ; Chen et al. , 2018 ; Pérez-Rúa et al. , 2018 ; Liu et al. , 2019a ) and natural language processing ( Zoph & Le , 2017 ; Pham et al. , 2018 ; Luo et al. , 2018 ; Zoph et al. , 2018 ; Liu et al. , 2018b ; Cai et al. , 2018a ) . Existing search strategies include reinforcement learning ( RL ) samplers ( Zoph & Le , 2017 ; Zoph et al. , 2018 ; Pham et al. , 2018 ) , evolutionary algorithms ( Xie & Yuille , 2017 ; Real et al. , 2017 ; Miikkulainen et al. , 2019 ; Liu et al. , 2018b ; Lu et al. , 2018 ) , gradient-descent ( Liu et al. , 2019b ) , bayesian optimization ( Kandasamy et al. , 2018 ; Jin et al. , 2019 ; Zhou et al. , 2019 ) and performance predictors ( Liu et al. , 2018a ; Luo et al. , 2018 ) . Here , our goal is not to introduce a new search policy , but rather to provide the means to analyze existing ones . Below , we briefly discuss existing NAS methods and focus on how they are typically evaluated . Neural architecture search with weight sharing . The potential of vanilla NAS comes with the drawback of requiring thousands of GPU hours even for small datasets , such as PTB and CIFAR-10 . 1The Kendall Tau ( Kendall , 1938 ) metric measures the correlation of two ranking . Details in Appendix A.1 . Furthermore , even when using such heavy computational resources , vanilla NAS has to restrict the number of trained architectures from a total of 109 to 104 , and increasing the sampler accuracy can only be achieved by increasing the resources . ENAS ( Pham et al. , 2018 ) was the first to propose a training scheme with shared parameters , reducing the resources from thousands of GPU days to one . Instead of being trained from scratch each sampled model inherits the parameters from previously-trained ones . Since then , NAS research has mainly focused on two directions : 1 ) Replacing the RL sampler with a better search algorithm , such as gradient descent ( Liu et al. , 2019b ) , bayesian optimiziation ( Zhou et al. , 2019 ) and performance predictors ( Luo et al. , 2018 ) ; 2 ) Exploiting NAS for other applications , e.g. , object detection ( Ghiasi et al. , 2019 ; Chen et al. , 2019 ) , semantic segmentation ( Liu et al. , 2019a ) , and finding compact networks ( Cai et al. , 2018b ; Wu et al. , 2018 ; Chu et al. , 2019 ; Guo et al. , 2019 ) . Characterizing the search space . Ying et al . ( 2019 ) ; Dong & Yang ( 2020 ) introduced a dataset that contains the ground-truth performance of CNN cells , and Wang et al . ( 2019 ) evaluated some traditional search algorithms on it . Similarly , Radosavovic et al . ( 2019 ) characterizes many CNN search spaces by computing the statistics of a set of sampled architectures , revealing that , for datasets such as CIFAR-10 or ImageNet , these statistics are similar . While these works support our claim that evaluation of NAS algorithms is crucial , they do not directly evaluate the state-of-the-arts NAS algorithms as we do here . Evaluation of NAS algorithms . Typically , the quality of NAS algorithms is judged based on the results of the final architecture they produce on the downstream task . In other words , the search and robustness of these algorithms are generally not studied , with ( Liu et al. , 2019b ; So et al. , 2019 ) the only exception for robustness , where results obtained with different random seeds were reported . Here , we aim to further the understanding of the mechanisms behind the search phase of NAS algorithms . Specifically , we propose doing so by comparing them with a simple random search policy , which uniformly randomly samples one architecture per run in the same search space as the NAS techniques . While some works have provided partial comparisons to random search , these comparisons unfortunately did not give a fair chance to the random policy . Specifically , ( Pham et al. , 2018 ) reports the results of only a single random architecture , and ( Liu et al. , 2018b ) those of an architecture selected among 8 randomly sampled ones as the most promising one after training for 300 epochs only . Here , we show that a fair comparison to the random policy , obtained by training all architectures , i.e. , random and NAS ones , for 1000 epochs and averaging over multiple random seeds for robustness , yields a different picture ; the state-of-the-art search policies are no better than the random one . The motivation behind this comparison was our observation of only a weak correlation between the performance of the searched architectures and the ones trained from scratch during the evaluation phase . This phenomenon was already noticed by Zela et al . ( 2018 ) , and concurrently to our work by Li & Talwalkar ( 2019 ) ; Xie et al . ( 2019 ) ; Ying et al . ( 2019 ) , but the analysis of its impact or its causes went no further . Here , by contrast , we link this difference in performance between the search and evaluation phases to the use of weight sharing . While this may seem to contradict the findings of Bender et al . ( 2018 ) , which , on CIFAR-10 , observed a strong correlation between architectures trained with and without weight sharing when searching a CNN cell , our work differs from ( Bender et al. , 2018 ) in two fundamental ways : 1 ) The training scheme in ( Bender et al. , 2018 ) , in which the entire model with shared parameters is trained via random path dropping , is fundamentally different from those used by state-of-the-arts weight sharing NAS strategies ( Pham et al. , 2018 ; Liu et al. , 2019b ; Luo et al. , 2018 ) ; 2 ) While the correlation in ( Bender et al. , 2018 ) was approximated using a small subset of sampled architectures , we make use of a reduced search space where we can perform a complete evaluation of all architectures , thus providing an exact correlation measure in this space .

Summary: This works studies the evaluation of search strategies for neural architecture search. It points out existing problems of the current evaluation scheme: (1) only compares the final result without testing the robustness under different random seeds; (2) lacking fair comparison with random baseline under different random seeds. The authors analyzed three popular NAS methods with weight sharing (ENAS, DARTS, NAO), and showed that they don't significantly improve upon random baseline on PTB and CIFAR-10. On a reduced search space of RNN and CNN (NASBench), they showed that the three methods fail to find the best performing architecture. Then they compared search with and without weight sharing and showed the correlation between architecture performance under the two conditions in a reduced search space, which indicates the weight sharing is a potential cause for the suboptimal performance.