Evaluating Efficient Performance Estimators of Neural Architectures

Conducting efficient performance estimations of neural architectures is a major challenge in neural architecture search ( NAS ) . To reduce the architecture training costs in NAS , one-shot estimators ( OSEs ) amortize the architecture training costs by sharing the parameters of one “ supernet ” between all architectures . Recently , zero-shot estimators ( ZSEs ) that involve no training are proposed to further reduce the architecture evaluation cost . Despite the high efficiency of these estimators , the quality of such estimations has not been thoroughly studied . In this paper , we conduct an extensive and organized assessment of OSEs and ZSEs on five NAS benchmarks : NAS-Bench-101/201/301 , and NDS ResNet/ResNeXt-A . Specifically , we employ a set of NAS-oriented criteria to study the behavior of OSEs and ZSEs , and reveal their biases and variances . After analyzing how and why the OSE estimations are unsatisfying , we explore how to mitigate the correlation gap of OSEs from three perspectives . Through our analysis , we give out suggestions for future application and development of efficient architecture performance estimators . Furthermore , the analysis framework proposed in our work could be utilized in future research to give a more comprehensive understanding of newly designed architecture performance estimators . The code is available at https : //github . com/walkerning/aw_nas [ 24 ] . 1 Introduction . Neural architecture search ( NAS ) can automatically discover architectures that outperform the handcrafted ones for various applications [ 48 , 10 , 11 ] . Early NAS methods [ 48 , 30 ] suffer from an extremely heavy computational burden , and can take tens of thousands of GPU hours to run . One of the major reasons for the computational challenge of NAS is that evaluating each candidate architecture is slow , which includes a full training and testing process . In the past years , studies [ 2 , 27 , 3 , 5 , 8 , 45 , 20 , 1 ] have been focusing on developing more efficient performance estimators of neural architectures . One-shot Estimator ( OSE ) Traditional NAS methods [ 48 , 30 , 2 ] conduct a costly separate training process to acquire the suitable parameters to evaluate each candidate architecture . To make NAS computationally tractable , ENAS [ 27 ] proposes the parameter-sharing technique to accelerate the architecture evaluation . Following this work , the parameter sharing technique is widely used for architecture search in different search spaces [ 39 , 16 ] or incorporated with different search strategies [ 19 , 16 , 23 , 40 ] . We refer to the parameter-sharing estimations as the “ one-shot ” estimations since it requires the training cost of one supernet . How well the one-shot estimations are correlated with the standalone architecture performances is essential for the efficacy of NAS methods . Despite the widespread use of OSEs , studies [ 43 ] have 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . revealed that the OSE estimations might fail to reflect the true ranking of architectures . However , their experiments are conducted in a toy search space with only 32 architectures . In this work , we conduct a more comprehensive study on OSEs in five search spaces with distinct properties , including three topological search spaces ( NAS-Bench-101 [ 41 ] , NAS-Bench-201 [ 9 ] , and NAS-Bench-301 [ 32 ] ) , and two non-topological search spaces [ 29 ] ( NDS ResNet , and NDS ResNeXt-A ) . We further analyze how and why OSE estimations have bias and variance , and explore how to improve OSEs . Zero-shot Estimator ( ZSE ) More recently , in order to further reduce the architecture evaluation cost , several studies [ 20 , 1 , 15 , 17 , 26 , 6 ] introduce “ zero-shot ” estimators that involve no training . In this work , we study various ZSEs on several benchmarks and reveal their properties and weakness . Knowledge Our work reveals pieces of knowledge on OSEs and ZSEs . First of all , some behaviors of OSEs and ZSEs vary across search spaces ( Appendix A.1.1 , Sec . 4.2 ) . Some of the common knowledge for OSEs revealed by our work include 1 ) OSEs bias towards architectures with lower complexity in the early training phase [ 18 ] . And this bias can be alleviated to various extents with sufficient training in different spaces ( Sec . 5.1 ) . 2 ) OSEs have variance and can be mitigated to some extent ( Sec . 5.3 , Sec . 6.1 ) . 3 ) Reducing the sharing extent of OSEs can potentially improve their ranking quality [ 47 ] ( Sec . 6.3 ) . As for ZSEs , we reveal that 1 ) Current ZSEs can not benefit from one-shot training . The ranking qualities of ZSEs utilizing high-order information ( i.e. , gradients ) even degrade a lot after one-shot training ( Sec . 4.2 ) . 2 ) Parameter-level ZSEs adapted from pruning literature are not suitable for ranking architectures , and their ranking qualities can not surpass those of parameter size ( # Param ) or # FLOPs ( Sec . 4.2 ) . 3 ) Existing ZSEs have improper biases , some overestimate linear architectures without skip connections , and some overestimate architectures with smaller kernel sizes and receptive fields ( Sec . 5.2 ) . 4 ) The relative effectiveness of ZSEs varies between search spaces , relu_logdet [ 21 ] is the best on the three topological search spaces , and synflow [ 33 ] is better on the two non-topological search spaces ( Sec . 4.2 ) . 5 ) Most ZSEs are not sensitive to the input data distribution : They get similar architecture rankings when using random noises as the input ( Appendix B.1 ) . Suggestions Based on our experiments and analyses , we give out suggestions for future OSE applications . For example : 1 ) Longer training makes one-shot estimations better ( Sec . 4.1 ) ; 2 ) Using one-shot loss instead of accuracy significantly improves the ranking qualities in the DARTS space [ 16 ] ( Sec . 4.1 ) ; 3 ) One should use enough validation data for OSEs , instead of merely several batches as in ZSEs ( Sec . 4.1 ) ; 4 ) Using temporal ensemble helps reduce the ranking instability , and brings non-negative improvements on the ranking quality in different search spaces ( Sec . 6.1 ) ; 5 ) In search space with isomorphic architectures , augmenting the sampling strategy to improve the sampling fairness is essential to avoid overestimating simple architectures ( Sec . 6.2 ) ; 6 ) Affine operation should not be used in batch normalization ( BN ) during supernet training ( Sec . 6.3 ) . As for ZSEs , we point out several open research problems : 1 ) Is there a general ZSE suitable for different types of search spaces ? 2 ) Do we need to make ZSEs utilize the input data information better , and how can we do that ? 3 ) Can we develop ZSEs that distinguish top architectures better ? We also list out some technical suggestions for improving ZSEs : 1 ) Future ZSEs should conduct architecture-level analysis instead of using parameter-level analysis ( Sec . 4.2 ) . 2 ) According to some prominent bias of existing ZSEs , we can add some structural knowledge into ZSE voting ensembles , e.g. , receptive field analysis seems promising for improving jacob_cov or relu_logdet ( Sec . 5.2 ) . 3 ) In future developments of ZSEs , researchers should add two simple comparison baselines , # Params , and # FLOPs , as they are actually very competitive baseline ZSEs ( Sec . 4.2 ) . Our work provides strong baselines and diagnosis tools for future research of architecture performance estimators , and we suggest future research to utilize these baselines and tools for a more comprehensive understanding of newly designed performance estimators . Analysis Framework Our analysis framework of efficient architecture performance estimators is organized as follows . We first introduce the evaluation criteria for estimator quality in Sec . 3 . And Sec . 4 presents the quality evaluation of multiple OSEs and ZSEs . Then , we conduct an organized analysis on how and why the OSE and ZSE estimations have biases and variances in Sec . 5 . Specifically , their complexity-level , operation-level , and architecture-level biases are demonstrated and analyzed . And the stability of OSE accuracy and ranking along the training process are analyzed . And in Sec . 6 , based on our analysis framework , we present several case studies on improving OSEs from three perspectives : i.e . reducing the variance , bias , and parameter sharing extent . 2 Related Work . 2.1 Efficient Performance Estimators of Neural Architectures . One-shot Estimators The vanilla NAS method [ 48 ] trains each architecture for 50 epochs to acquire its suitable parameters , which makes the NAS process prohibitively costly . As a remedy , ENAS [ 27 ] proposes to amortize the separate training costs by sharing parameters among architectures . Specifically , ENAS constructs an over-parametrized supernet such that all architectures can be evaluated using its parameter subsets . Throughout the search process , the shared supernet parameters are updated on the training set , and an RNN controller is updated alternatively on the validation set . There are two types of parameter-sharing methods : 1 ) One-shot NAS methods [ 3 , 13 ] that first train a supernet and then conduct architecture search without further supernet tuning . 2 ) Non-one-shot methods [ 27 , 16 , 40 ] that conduct supernet training and architecture search ( i.e . controller update ) jointly . And this work focuses on evaluating the estimations of the “ one-shot ” supernet , since it is the cleaner case without the complexity of varying controller settings and possible controller-supernet co-adaption . In each supernet training step , S architectures are randomly sampled to process a batch of training data . Here S denotes the number of Monte-Carlo architecture samples . Then , the gradients of these architectures are averaged to update the supernet . Correlation of One-shot Estimators There exist some studies that carry out correlation evaluation for one-shot estimators . Zhang et al . [ 45 ] compare the correlation of OSEs and their proposed hyper-network-based estimator . However , their work is not aiming for a large-scale evaluation of OSEs and ZSEs , thus they only evaluate the OSE correlations on one search space , and do not conduct further analysis . Yu et al . [ 43 ] conduct parameter sharing NAS in a toy RNN search space with only 32 architectures in total , and discover that the parameter sharing rankings do not correlate with the true rankings of architectures . Zela et al . [ 44 ] also report that the correlation of parameter-sharing estimations is not satisfying with a Spearman correlation coefficient between -0.25 and 0.3 on a larger search space with around 15k architectures . Pourchot et al . [ 28 ] evaluate the Spearman ’ s ranking correlation of OSEs on NAS-Bench-101 . Yu et al . [ 42 ] provide an analysis on how the heuristics and hyperparameters influence the supernet training on three benchmarks ( i.e . NAS-Bench-101 , NAS-Bench-201 , and DARTS-NDS ) . But they only use the variants of Kendall ’ s Tau as the evaluation criteria , and do not further explore the biases and failing reasons of OSEs . Zhang et al . [ 47 ] point out the instability and poor ranking correlation of OSEs , and claim that the high extent of parameter sharing causes the unsatisfying performance . However , they only conduct experiments on a small search space with about 200 architectures . In this paper , we conduct a more comprehensive study of OSE behaviors across five search spaces , and further investigate how and why the OSE estimations are not satisfying . We also propose and compare several techniques to mitigate the OSE correlation gap . Zero-shot Estimators More recently , in order to further reduce the architecture evaluation cost , several researches [ 20 , 1 ] propose “ zero-shot ” estimators that conduct no training and use random initialized models to estimate architecture performances . Based on the observation that good architectures have distinct local jacobian on different images , Mellor et al . [ 20 ] propose an indicator based on input jacobian correlation . Lopes et al . [ 17 ] improve the above indicator by calculating the jacobian correlation with respect to the class . Abdelfattah et al . [ 1 ] adapt several ZSEs from the pruning literature , and claim that these adapted ZSEs can perform well on NAS-Bench-201 . Lin et al . [ 15 ] define the expected Gaussian complexity to measure the network expressivity , and efficiently discover architectures with state-of-the-art accuracy on ImagetNet . With a small training overhead , Ru et al . [ 31 ] propose to evaluate an architecture ’ s performance by its training speed . Concurrent to our work , White et al . [ 38 ] also evaluate various performance estimators on multiple benchmarks .

Summary: Weight sharing is a very important technique in NAS to speed up the search and reduce resources. However, such technique suffers from performance drop, and attracts a lot of interset from the community to investigate into it. This paper is one of the works that investigate the behavior and underline principles of such weight sharing technique (aka one-shot). Besides it also investigate current popular zero-shot estimators. The paper analyzes the one-shot and zero-shot estimators from multiple perspectives on three benchmark datasets, with sufficient experiments and further suggestions.  Though some of the points/findings are not novel and well-known, the sufficient experiments still make a convening verification on these. It will bring much to the community. The findings and suggestions will make using one-shot and zero-shot NAS easier for practitioners.