Evaluating Efficient Performance Estimators of Neural Architectures

Conducting efficient performance estimations of neural architectures is a major challenge in neural architecture search ( NAS ) . To reduce the architecture training costs in NAS , one-shot estimators ( OSEs ) amortize the architecture training costs by sharing the parameters of one “ supernet ” between all architectures . Recently , zero-shot estimators ( ZSEs ) that involve no training are proposed to further reduce the architecture evaluation cost . Despite the high efficiency of these estimators , the quality of such estimations has not been thoroughly studied . In this paper , we conduct an extensive and organized assessment of OSEs and ZSEs on five NAS benchmarks : NAS-Bench-101/201/301 , and NDS ResNet/ResNeXt-A . Specifically , we employ a set of NAS-oriented criteria to study the behavior of OSEs and ZSEs , and reveal their biases and variances . After analyzing how and why the OSE estimations are unsatisfying , we explore how to mitigate the correlation gap of OSEs from three perspectives . Through our analysis , we give out suggestions for future application and development of efficient architecture performance estimators . Furthermore , the analysis framework proposed in our work could be utilized in future research to give a more comprehensive understanding of newly designed architecture performance estimators . The code is available at https : //github . com/walkerning/aw_nas [ 24 ] . 1 Introduction . Neural architecture search ( NAS ) can automatically discover architectures that outperform the handcrafted ones for various applications [ 48 , 10 , 11 ] . Early NAS methods [ 48 , 30 ] suffer from an extremely heavy computational burden , and can take tens of thousands of GPU hours to run . One of the major reasons for the computational challenge of NAS is that evaluating each candidate architecture is slow , which includes a full training and testing process . In the past years , studies [ 2 , 27 , 3 , 5 , 8 , 45 , 20 , 1 ] have been focusing on developing more efficient performance estimators of neural architectures . One-shot Estimator ( OSE ) Traditional NAS methods [ 48 , 30 , 2 ] conduct a costly separate training process to acquire the suitable parameters to evaluate each candidate architecture . To make NAS computationally tractable , ENAS [ 27 ] proposes the parameter-sharing technique to accelerate the architecture evaluation . Following this work , the parameter sharing technique is widely used for architecture search in different search spaces [ 39 , 16 ] or incorporated with different search strategies [ 19 , 16 , 23 , 40 ] . We refer to the parameter-sharing estimations as the “ one-shot ” estimations since it requires the training cost of one supernet . How well the one-shot estimations are correlated with the standalone architecture performances is essential for the efficacy of NAS methods . Despite the widespread use of OSEs , studies [ 43 ] have 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . revealed that the OSE estimations might fail to reflect the true ranking of architectures . However , their experiments are conducted in a toy search space with only 32 architectures . In this work , we conduct a more comprehensive study on OSEs in five search spaces with distinct properties , including three topological search spaces ( NAS-Bench-101 [ 41 ] , NAS-Bench-201 [ 9 ] , and NAS-Bench-301 [ 32 ] ) , and two non-topological search spaces [ 29 ] ( NDS ResNet , and NDS ResNeXt-A ) . We further analyze how and why OSE estimations have bias and variance , and explore how to improve OSEs . Zero-shot Estimator ( ZSE ) More recently , in order to further reduce the architecture evaluation cost , several studies [ 20 , 1 , 15 , 17 , 26 , 6 ] introduce “ zero-shot ” estimators that involve no training . In this work , we study various ZSEs on several benchmarks and reveal their properties and weakness . Knowledge Our work reveals pieces of knowledge on OSEs and ZSEs . First of all , some behaviors of OSEs and ZSEs vary across search spaces ( Appendix A.1.1 , Sec . 4.2 ) . Some of the common knowledge for OSEs revealed by our work include 1 ) OSEs bias towards architectures with lower complexity in the early training phase [ 18 ] . And this bias can be alleviated to various extents with sufficient training in different spaces ( Sec . 5.1 ) . 2 ) OSEs have variance and can be mitigated to some extent ( Sec . 5.3 , Sec . 6.1 ) . 3 ) Reducing the sharing extent of OSEs can potentially improve their ranking quality [ 47 ] ( Sec . 6.3 ) . As for ZSEs , we reveal that 1 ) Current ZSEs can not benefit from one-shot training . The ranking qualities of ZSEs utilizing high-order information ( i.e. , gradients ) even degrade a lot after one-shot training ( Sec . 4.2 ) . 2 ) Parameter-level ZSEs adapted from pruning literature are not suitable for ranking architectures , and their ranking qualities can not surpass those of parameter size ( # Param ) or # FLOPs ( Sec . 4.2 ) . 3 ) Existing ZSEs have improper biases , some overestimate linear architectures without skip connections , and some overestimate architectures with smaller kernel sizes and receptive fields ( Sec . 5.2 ) . 4 ) The relative effectiveness of ZSEs varies between search spaces , relu_logdet [ 21 ] is the best on the three topological search spaces , and synflow [ 33 ] is better on the two non-topological search spaces ( Sec . 4.2 ) . 5 ) Most ZSEs are not sensitive to the input data distribution : They get similar architecture rankings when using random noises as the input ( Appendix B.1 ) . Suggestions Based on our experiments and analyses , we give out suggestions for future OSE applications . For example : 1 ) Longer training makes one-shot estimations better ( Sec . 4.1 ) ; 2 ) Using one-shot loss instead of accuracy significantly improves the ranking qualities in the DARTS space [ 16 ] ( Sec . 4.1 ) ; 3 ) One should use enough validation data for OSEs , instead of merely several batches as in ZSEs ( Sec . 4.1 ) ; 4 ) Using temporal ensemble helps reduce the ranking instability , and brings non-negative improvements on the ranking quality in different search spaces ( Sec . 6.1 ) ; 5 ) In search space with isomorphic architectures , augmenting the sampling strategy to improve the sampling fairness is essential to avoid overestimating simple architectures ( Sec . 6.2 ) ; 6 ) Affine operation should not be used in batch normalization ( BN ) during supernet training ( Sec . 6.3 ) . As for ZSEs , we point out several open research problems : 1 ) Is there a general ZSE suitable for different types of search spaces ? 2 ) Do we need to make ZSEs utilize the input data information better , and how can we do that ? 3 ) Can we develop ZSEs that distinguish top architectures better ? We also list out some technical suggestions for improving ZSEs : 1 ) Future ZSEs should conduct architecture-level analysis instead of using parameter-level analysis ( Sec . 4.2 ) . 2 ) According to some prominent bias of existing ZSEs , we can add some structural knowledge into ZSE voting ensembles , e.g. , receptive field analysis seems promising for improving jacob_cov or relu_logdet ( Sec . 5.2 ) . 3 ) In future developments of ZSEs , researchers should add two simple comparison baselines , # Params , and # FLOPs , as they are actually very competitive baseline ZSEs ( Sec . 4.2 ) . Our work provides strong baselines and diagnosis tools for future research of architecture performance estimators , and we suggest future research to utilize these baselines and tools for a more comprehensive understanding of newly designed performance estimators . Analysis Framework Our analysis framework of efficient architecture performance estimators is organized as follows . We first introduce the evaluation criteria for estimator quality in Sec . 3 . And Sec . 4 presents the quality evaluation of multiple OSEs and ZSEs . Then , we conduct an organized analysis on how and why the OSE and ZSE estimations have biases and variances in Sec . 5 . Specifically , their complexity-level , operation-level , and architecture-level biases are demonstrated and analyzed . And the stability of OSE accuracy and ranking along the training process are analyzed . And in Sec . 6 , based on our analysis framework , we present several case studies on improving OSEs from three perspectives : i.e . reducing the variance , bias , and parameter sharing extent . 2 Related Work . 2.1 Efficient Performance Estimators of Neural Architectures . One-shot Estimators The vanilla NAS method [ 48 ] trains each architecture for 50 epochs to acquire its suitable parameters , which makes the NAS process prohibitively costly . As a remedy , ENAS [ 27 ] proposes to amortize the separate training costs by sharing parameters among architectures . Specifically , ENAS constructs an over-parametrized supernet such that all architectures can be evaluated using its parameter subsets . Throughout the search process , the shared supernet parameters are updated on the training set , and an RNN controller is updated alternatively on the validation set . There are two types of parameter-sharing methods : 1 ) One-shot NAS methods [ 3 , 13 ] that first train a supernet and then conduct architecture search without further supernet tuning . 2 ) Non-one-shot methods [ 27 , 16 , 40 ] that conduct supernet training and architecture search ( i.e . controller update ) jointly . And this work focuses on evaluating the estimations of the “ one-shot ” supernet , since it is the cleaner case without the complexity of varying controller settings and possible controller-supernet co-adaption . In each supernet training step , S architectures are randomly sampled to process a batch of training data . Here S denotes the number of Monte-Carlo architecture samples . Then , the gradients of these architectures are averaged to update the supernet . Correlation of One-shot Estimators There exist some studies that carry out correlation evaluation for one-shot estimators . Zhang et al . [ 45 ] compare the correlation of OSEs and their proposed hyper-network-based estimator . However , their work is not aiming for a large-scale evaluation of OSEs and ZSEs , thus they only evaluate the OSE correlations on one search space , and do not conduct further analysis . Yu et al . [ 43 ] conduct parameter sharing NAS in a toy RNN search space with only 32 architectures in total , and discover that the parameter sharing rankings do not correlate with the true rankings of architectures . Zela et al . [ 44 ] also report that the correlation of parameter-sharing estimations is not satisfying with a Spearman correlation coefficient between -0.25 and 0.3 on a larger search space with around 15k architectures . Pourchot et al . [ 28 ] evaluate the Spearman ’ s ranking correlation of OSEs on NAS-Bench-101 . Yu et al . [ 42 ] provide an analysis on how the heuristics and hyperparameters influence the supernet training on three benchmarks ( i.e . NAS-Bench-101 , NAS-Bench-201 , and DARTS-NDS ) . But they only use the variants of Kendall ’ s Tau as the evaluation criteria , and do not further explore the biases and failing reasons of OSEs . Zhang et al . [ 47 ] point out the instability and poor ranking correlation of OSEs , and claim that the high extent of parameter sharing causes the unsatisfying performance . However , they only conduct experiments on a small search space with about 200 architectures . In this paper , we conduct a more comprehensive study of OSE behaviors across five search spaces , and further investigate how and why the OSE estimations are not satisfying . We also propose and compare several techniques to mitigate the OSE correlation gap . Zero-shot Estimators More recently , in order to further reduce the architecture evaluation cost , several researches [ 20 , 1 ] propose “ zero-shot ” estimators that conduct no training and use random initialized models to estimate architecture performances . Based on the observation that good architectures have distinct local jacobian on different images , Mellor et al . [ 20 ] propose an indicator based on input jacobian correlation . Lopes et al . [ 17 ] improve the above indicator by calculating the jacobian correlation with respect to the class . Abdelfattah et al . [ 1 ] adapt several ZSEs from the pruning literature , and claim that these adapted ZSEs can perform well on NAS-Bench-201 . Lin et al . [ 15 ] define the expected Gaussian complexity to measure the network expressivity , and efficiently discover architectures with state-of-the-art accuracy on ImagetNet . With a small training overhead , Ru et al . [ 31 ] propose to evaluate an architecture ’ s performance by its training speed . Concurrent to our work , White et al . [ 38 ] also evaluate various performance estimators on multiple benchmarks .

Summary: This work analyzes several one-shot and zero-shot NAS methods on multiple NAS benchmark datasets. The authors inspect NAS algorithms from various aspects, including training bias, stability, variance, loss ranking v.s. accuracy ranking, etc. . The authors conclude that one-shot methods and zero-shot methods might fail to work due to multiple reasons and propose several solutions to alleviate these issues.