A Surgery of the Neural Architecture Evaluators

Neural architecture search ( NAS ) has recently received extensive attention due to its effectiveness in automatically designing effective neural architectures . A major challenge in NAS is to conduct a fast and accurate evaluation ( i.e. , performance estimation ) of neural architectures . Commonly used fast architecture evaluators include parameter-sharing ones and predictor-based ones . Despite their high evaluation efficiency , the evaluation correlation ( especially of the well-performing architectures ) is still questionable . In this paper , we conduct an extensive assessment of both the parameter-sharing and predictor-based evaluators on the NASBench-201 search space , and break up how and why different configurations and strategies influence the fitness of the evaluators . Specifically , we develop a set of NAS-oriented criteria to understand the behavior of fast architecture evaluators in different training stages . And based on the findings of our experiments , we give pieces of knowledge and suggestions to guide NAS application and motivate further research . 1 INTRODUCTION . Studies have shown that the automatically discovered architectures by NAS can outperform the hand-crafted architectures for various applications , such as classification ( Nayman et al. , 2019 ; Zoph & Le , 2017 ) , detection ( Ghiasi et al. , 2019 ; Chen et al. , 2019b ) , video understanding ( Ryoo et al. , 2019 ) , text modeling ( Zoph & Le , 2017 ) , etc . Early NAS algorithms ( Zoph & Le , 2017 ) suffer from the extremely heavy computational burden , since the evaluation of neural architectures is slow . Thus , how to estimate the performance of a neural architecture in a fast and accurate way is vital for addressing the computational challenge of NAS . A neural architecture evaluator outputs the evaluated score of an architecture that indicates its quality . The straightforward solution is to train an architecture from scratch to convergence and then test it on the validation dataset , which is extremely time-consuming . Instead of exactly evaluating architectures on the target task , researchers usually construct a proxy model with fewer layers or fewer channels ( Pham et al. , 2018 ; Real et al. , 2019 ; Wu et al. , 2019 ) , and train this model to solve a proxy task of smaller scales ( Cai et al. , 2018a ; Elsken et al. , 2018 ; Klein et al. , 2017 ; Wu et al. , 2019 ) , e.g. , smaller dataset or subsets of dataset , training or finetuning for fewer epochs . Traditional evaluators conduct separate training phases to acquire the weights that are suitable for each architecture . In contrast , one-shot evaluation amortized the training cost of different architectures through parameter sharing or a global hypernetwork , thus significantly reduce the evaluation cost . Pham et al . ( 2018 ) constructs an over-parametrized super network ( supernet ) such that all architectures in the search space are sub-architectures of the supernet . Throughout the search process , the shared parameters in the supernet are updated on the training dataset split , and each architecture is evaluated by directly using the corresponding subset of the weights . Afterwards , the parameter sharing technique is widely used for architecture search in different search spaces ( Wu et al. , 2019 ; Cai et al. , 2020 ) , or incorporated with different search strategies ( Liu et al. , 2018b ; Nayman et al. , 2019 ; Xie et al. , 2018 ; Yang et al. , 2019 ; Cai et al. , 2020 ) . Hypernetwork ( Brock et al. , 2018 ; Zhang et al. , 2018 ) based evaluation is another type of one-shot evaluation strategy , in which a hypernetwork is trained to generate proper weights for each architecture . Since hypernetwork solutions are not generic currently , this paper concentrates on the evaluation of parameter sharing evaluators . One-Shot ( Hypernetwork ) SuperNet Train Loss Valid Accuracy Gradient Update Generate weights Architecture ( with weights ) Data Predictor-based Architecture Performance Loss Predicted Score Ground-Truth Performance MLPEncoder Predictor Gradient Update [ Arch1 , 78.3 % ] … [ ArchN , 80.4 % ] Architecture Provided by another “ oracle ” evaluator HyperNet One-Shot ( Shared weights ) Gradient Update Training Inference ( Evaluator Output ) ( Evaluator Output ) ( Evaluator Input ) ( Evaluator Input ) Figure 1 : An overview of fast neural architecture evaluators ( i.e. , performance estimators ) . Whether or not one-shot strategies can provide highly-correlated architecture evaluation results is essential for the efficacy of the NAS process . Many recent studies have been focusing on assessing the evaluation correlation of one-shot architecture evaluators ( Bender et al. , 2018 ; Sciuto et al. , 2019 ; Zela et al. , 2020 ) . Besides one-shot evaluation strategies , predictor-based evaluation strategies ( Luo et al. , 2018 ; Liu et al. , 2018a ; Deng et al. , 2017 ; Sun et al. , 2019 ; Wang et al. , 2018 ; Xu et al. , 2019 ; Ning et al. , 2020 ) use a performance predictor that takes the architecture description as inputs and outputs a predicted performance score . The performance predictor should be trained using “ ground-truth ” architecture performances . This paper utilizes the same set of criteria to evaluate and compare different performance predictors . The fast neural architecture evaluators ( i.e. , performance estimators ) are summarized in Fig . 1 , including parameter sharing , hypernetworks , and predictor-based ones . And this paper aims at revealing the status of current architecture evaluation strategies systematically . Specifically , we develop a set of NAS-oriented criteria to understand the behavior of fast architecture evaluators in different training stages . And based on the findings of our experiments , we give pieces of knowledge and suggestions to guide NAS application and motivate further research . The knowledge revealed by this paper includes : 1 ) Layer proxy brings a larger evaluation gap than using channel proxy , thus channel proxy can be utilized to reduce the computational cost , while proxy-less search w.r.t the layer number is worth studying . 2 ) The convergence rate of different criteria varies during the one-shot supernet training , which shows that the good architectures are distinguished from bad architectures in the early stage . 3 ) As training goes on , the one-shot performances of isomorphic sub-architectures become closer . 4 ) De-isomorphic sampling or post de-isomorphism handling can help avoid the over-estimation of simple architectures . 5 ) Parameter sharing evaluator tends to over-estimate smaller architectures , and is better at comparing smaller models than larger models . 6 ) One should use ranking losses rather than regression losses to train predictors , since they are more stable . 7 ) Different predictors under- or over-estimate different architectures , and currently , the best predictor might still have trouble in comparing large architectures . 8 ) As expected , architecture predictors can distinguish good architectures better after multiple stages of training , as the training data are more and more concentrated on the good architectures . 2 RELATED WORK . 2.1 ONE-SHOT EVALUATORS . One-shot evaluation mainly consists of two types of strategies : 1 ) parameter sharing ( Pham et al. , 2018 ; Wu et al. , 2019 ; Liu et al. , 2018b ; Nayman et al. , 2019 ; Xie et al. , 2018 ; Yang et al. , 2019 ; Cai et al. , 2020 ) , 2 ) hypernetworks ( Brock et al. , 2018 ; Zhang et al. , 2018 ) . These two strategies both amortize the training cost of different architectures via the sharing of the network or hypernetwork parameters . The ranking correlation gaps of existing shared weights evaluators are brought by two factors : 1 ) proxy model and task : due to the memory constraint , a proxy supernet ( supernet ) ( Liu et al. , 2018b ; Pham et al. , 2018 ) with fewer channels or layers is usually used ; 2 ) parameter sharing . To alleviate the first factor , there are some studies ( Cai et al. , 2018b ; Chen et al. , 2019a ) that aim at making oneshot evaluation more memory efficient , thus the one-shot search could be conducted without using a proxy supernet . As for the second factor , there are a few studies that carried out correlation evaluation for one-shot evaluators . Zhang et al . ( 2018 ) conducted a correlation comparison between the GHN hypernetwork evaluator , shared weights evaluator , and several small proxy tasks . However , the correlation is evaluated using 100 architectures randomly sampled from a large search space , which is not a convincing and consistent benchmark metric . Luo et al . ( 2019 ) did a preliminary investigation into why parameter sharing evaluation fails to provide correlated evaluations , and proposed to increase the sample probabilities of the large models . Their evaluation is also conducted on dozens of architectures sampled from the search space . Zela et al . ( 2020 ) compare the evaluation correlation of different search strategies on NAS-Bench-101 . Sciuto et al . ( 2019 ) conduct parameter sharing NAS in a toy RNN search space with only 32 architectures in total , and discover that the parameter sharing rankings do not correlate with the true rankings of architectures . To improve the evaluation correlation , Chu et al . ( 2019 ) proposed a sampling strategy in a layer-wise search space . In this paper , we analyze the ranking correlation gaps brought by the model proxy ( difference in the number of channels and layers ) and the parameter sharing technique , as well as the behavior of one-shot evaluators during the training process . 2.2 PREDICTOR-BASED EVALUATORS . An architecture performance predictor takes the architecture descriptions as inputs , and outputs the predicted performance scores without training the architectures . Two factors are crucial to the fitness of the predictors : 1 ) embedding space ; 2 ) training technique . On the one hand , to embed neural architectures into a continuous space and get a meaningful embedding space , there are studies that propose different architecture encoders , e.g. , sequence-based ( Luo et al. , 2018 ; Liu et al. , 2018a ; Deng et al. , 2017 ; Sun et al. , 2019 ; Wang et al. , 2018 ) , graph-based ( Shi et al. , 2019 ; Ning et al. , 2020 ) . As for nonparametric predictors , Kandasamy et al . ( 2018 ) design a kernel function in the architecture space and exploits gaussian process to get the posterior of the architecture performances . Shi et al . ( 2019 ) combined a graph-based encoder and nonparametric gaussian process to construct the performance predictor . On the other hand , from the aspect of training techniques , Luo et al . ( 2018 ) employed an encoder-decoder structure and used an auxiliary reconstruction loss term . Xu et al . ( 2019 ) ; Ning et al . ( 2020 ) employed learning-to-rank techniques to train the predictors . Actually , in the overall NAS framework , the predictor-based evaluator plays a different role from the traditional or one-shot evaluators , since the predictor should be trained using “ ground-truth ” architecture performances . Usually , expensive traditional evaluators that can provide relatively accurate architecture performances are chosen as the “ oracle ” evaluators to output the “ ground-truth ” scores ( Kandasamy et al. , 2018 ; Liu et al. , 2018a ; Luo et al. , 2018 ) . 3 EVALUATION CRITERIA . In this section , we introduce the evaluation criteria used in this paper . We denote the search space size as M , the true performances and approximated evaluated scores of architectures { ai } i=1 , ··· , M as { yi } i=1 , ··· , M and { si } i=1 , ··· , M , respectively . And we denote the ranking of the true performance yi and the evaluated score si as ri ∈ { 1 , · · · , M } and ni ∈ { 1 , · · · , M } ( ri = 1 indicates that ai is the best architecture in the search space ) . The correlation criteria adopted in our paper are • Linear correlation : The pearson correlation coefficient corr ( y , s ) / √ corr ( y , y ) corr ( s , s ) . • Kendall ’ s Tau ranking correlation : The relative difference of concordant pairs and discor- dant pairs ∑ i < j sgn ( yi − yj ) sgn ( si − sj ) / ( M 2 ) . • Spearman ’ s ranking correlation : The pearson correlation coefficient between the rank variables corr ( r , n ) / √ corr ( r , r ) corr ( n , n ) . Besides these correlation criteria , criteria that emphasize more on the relative order of architectures with good performances are desired . Denoting AK = { ai|ni < KM } as the set of architectures whose evaluated scores s is among the top K portion of the search space , we use two criteira • Precision @ K ( P @ K ) ∈ ( 0 , 1 ] = # { i|ri < KM ∧ ni < KM } KM : The proportion of true top-K proportion architectures in the top-K architectures according to the scores . • BestRanking @ K ( BR @ K ) ∈ ( 0 , 1 ] = argmini∈AK ri : The best normalized ranking among the top K proportion of architectures according to the scores . The two criteria are similar to those used in Ning et al . ( 2020 ) , except that rankings and architecture numbers are all normalized with respect to the search space size M . The above criteria are used to compare the fitness of various architecture evaluators with different configurations and in different stages . Besides that , we ’ d also like to interpret their evaluation results . To identify which architectures are under- or over-estimated by various evaluators , and analyze the reasons accordingly , we investigate the relationship of the true-predicted ranking differences { ri − ni } i=1 , ··· , M and the architecture properties such as the FLOPs : { FLOPs ( ai ) } i=1 , ··· , M .

Summary: The paper assesses two different approaches to speed up the evaluations of neural network architectures for neural architecture search (NAS). The first one is weight sharing, which trains a supernetwork that contains all possible architecture of the search space. The performance of single architectures can be then approximated by simply using the shared parameters of the supernetwork. The second approach is to use different kind of predictors that are trained on offline evaluated architectures. Several methods following these two approaches from the literature are evaluated on the NASBench201 benchmark based on different rank-based evaluation scores.