How Powerful are Performance Predictors in Neural Architecture Search?

1 Introduction . Neural architecture search ( NAS ) is a popular area of machine learning , which aims to automate the process of developing neural architectures for a given dataset . Since 2017 , a wide variety of NAS techniques have been proposed [ 78 , 45 , 32 , 49 ] . While the first NAS techniques trained thousands of architectures to completion and then evaluated the performance using the final validation accuracy [ 78 ] , modern algorithms use more efficient strategies to estimate the performance of partially-trained or even untrained neural networks [ 11 , 2 , 54 , 34 , 38 ] . Recently , many performance prediction methods have been proposed based on training a model to predict the final validation accuracy of an architecture just from an encoding of the architecture . Popular choices for these models include Gaussian processes [ 60 , 17 , 51 ] , neural networks [ 36 , 54 , 65 , 69 ] , tree-based methods [ 33 , 55 ] , and so on . However , these methods often require hundreds of fully-trained architectures to be used as training data , thus incurring high initialization time . In contrast , learning curve extrapolation methods [ 11 , 2 , 20 ] need little or no initialization time , but each individual prediction requires partially training the architecture , incurring high query time . Very recently , a few techniques have been introduced which are fast both in query time and initialization time [ 38 , 1 ] , computing predictions based on a single minibatch of data . Finally , using shared weights [ 45 , 4 , 32 ] is a popular paradigm for NAS [ 73 , 25 ] , although the effectiveness of these methods in ranking architectures is disputed [ 53 , 74 , 76 ] . Despite the widespread use of performance predictors , it is not known how methods from different families compare to one another . While there have been some analyses on the best predictors within ∗ { colin , yang } @ abacus.ai , { zelaa , fh } @ cs.uni-freiburg.de , robin @ robots.ox.ac.uk 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . each class [ 41 , 72 ] , for many predictors , the only evaluation is from the original work that proposed the method . Furthermore , no work has previously compared the predictors across different families of performance predictors . This leads to two natural questions : how do zero-cost methods , model-based methods , learning curve extrapolation methods , and weight sharing methods compare to one another across different constraints on initialization time and query time ? Furthermore , can predictors from different families be combined to achieve even better performance ? In this work , we answer the above questions by giving the first large-scale study of performance predictors for NAS . We study 31 predictors across four popular search spaces and four datasets : NAS-Bench-201 [ 13 ] with CIFAR-10 , CIFAR-100 , and ImageNet16-120 , NAS-Bench-101 [ 71 ] and DARTS [ 32 ] with CIFAR-10 , and NAS-Bench-NLP [ 21 ] with Penn TreeBank . In order to give a fair comparison among different classes of predictors , we run a full portfolio of experiments , measuring the Pearson correlation and rank correlation metrics ( Spearman , Kendall Tau , and sparse Kendall Tau ) , across a variety of initialization time and query time budgets . We run experiments using a training and test set of architectures generated both uniformly at random , as well as by mutating the highest-performing architectures ( the latter potentially more closely resembling distributions encountered during an actual NAS run ) . Finally , we test the ability of each predictor to speed up NAS algorithms , namely Bayesian optimization [ 36 , 54 , 69 , 51 ] and predictor-guided evolution [ 66 , 59 ] . Since many predictors so far had only been evaluated on one search space , our work shows which predictors have consistent performance across search spaces . Furthermore , by conducting a study with three axes of comparison ( see Figure 1 ) , and by comparing various types of predictors , we see a more complete view of the state of performance predictor techniques that leads to interesting insights . Notably , we show that the performance of predictors from different families are complementary and can be combined to achieve significantly higher performance . The success of these experiments opens up promising avenues for future work . Overall , our experiments bridge multiple areas of NAS research and act as recommendations for the best predictors to use under different runtime constraints . Our code , based on the NASLib library [ 52 ] , can be used as a testing ground for future performance prediction techniques . In order to ensure reproducibility of the original results , we created a table to clarify which of the 31 predictors had previously published results on a NAS-Bench search space , and how these published results compared to our results ( Table 7 ) . We also adhere to the NeurIPS 2021 checklist along with the specialized NAS best practices checklist [ 31 ] . Our contributions . We summarize our main contributions below . • We conduct the first large-scale study of performance predictors for neural architecture search by comparing model-based methods , learning curve extrapolation methods , zero-cost methods , and weight sharing methods across a variety of settings . • We release a comprehensive library of 31 performance predictors on four different search spaces . • We show that different families of performance predictors can be combined to achieve substantially better predictive power than any single predictor . 2 Related Work . NAS has been studied since at least the 1990s [ 19 , 58 ] , and has been revitalized in the last few years [ 78 ] . While initial techniques focused on reinforcement learning [ 78 , 45 ] and evolutionary search [ 37 , 49 ] , one-shot NAS algorithms [ 32 , 12 , 4 ] and predictor-based NAS algorithms [ 65 , 54 , 69 ] have recently become popular . We give a brief survey of performance prediction techniques in Section 3 . For a survey on NAS , see [ 15 ] . The most widely used type of search space in prior work is the cell-based search space [ 79 ] , where the architecture search is over a relatively small directed acyclic graph representing an architecture . A few recent works have compared different performance predictors on popular cell-based search spaces for NAS . Siems et al . [ 55 ] studied graph neural networks and tree-based methods , and found that gradient-boosted trees and graph isomorphism networks performed the best . However , the comparison was only on a single search space and dataset , and the explicit goal was to achieve maximum performance given a training set of around 60 000 architectures . Another recent paper [ 41 ] studied various aspects of supernetwork training , and separately compared four model-based methods : random forest , MLP , LSTM , and GATES [ 42 ] . However , the comparisons were again on a single search space and dataset and did not compare between multiple families of performance predictors . Other papers have proposed new model-based predictors and compared the new predictors to other model-based baselines [ 34 , 65 , 54 , 69 ] . Finally , a recent paper analyzed training heuristics to make weight-sharing more effective at ranking architectures [ 72 ] . To the best of our knowledge , no prior work has conducted comparisons across multiple families of performance predictors . 3 Performance Prediction Methods for NAS . In NAS , given a search space A , the goal is to find a∗ = argmina∈Af ( a ) , where f denotes the validation error of architecture a after training on a fixed dataset for a fixed number of epochs E. Since evaluating f ( a ) typically takes hours ( as it requires training a neural network from scratch ) , many NAS algorithms make use of performance predictors to speed up this process . A performance predictor f ′ is defined generally as any function which predicts the final accuracy or ranking of architectures , without fully training the architectures . That is , evaluating f ′ should take less time than evaluating f , and { f ′ ( a ) | a ∈ A } should ideally have high correlation or rank correlation with { f ( a ) | a ∈ A } . Each performance predictor is defined by two main routines : an initialization routine which performs general pre-computation , and a query routine which performs the final architecture-specific computation : it takes as input an architecture specification , and outputs its predicted accuracy . For example , one of the simplest performance predictors is early stopping : for any query ( a ) , train a for E/2 epochs instead of E [ 77 ] . In this case , there is no general pre-computation , so initialization time is zero . On the other hand , the query time for each input architecture is high because it involves training the architecture for E/2 epochs . In fact , the runtime of the initialization and query routines varies substantially based on the type of predictor . In the context of NAS algorithms , the initialization routine is typically performed once at the start of the algorithm , and the query routine is typically performed many times throughout the NAS algorithm . Some performance predictors also make use of an update routine , when part of the computation from initialization needs to be updated without running the full procedure again ( for example , in a NAS algorithm , a model may be updated periodically based on newly trained architectures ) . Now we give an overview of the main families of predictors . See Figure 1 ( left ) for a taxonomy of performance predictors . Model-based ( trainable ) methods . The most common type of predictor , the model-based predictor , is based on supervised learning . The initialization routine consists of fully training many architectures ( i.e. , evaluating f ( a ) for many architectures a ∈ A ) to build a training set of datapoints { a , f ( a ) } . Then a model f ′ is trained to predict f ( a ) given a . While the initialization time for model-based predictors is very high , the query time typically takes less than a second , which allows thousands of predictions to be made throughout a NAS algorithm . The model is also updated regularly based on the new datapoints . These predictors are typically used within BO frameworks [ 36 , 54 ] , evolutionary frameworks [ 66 ] , or by themselves [ 67 ] , to perform NAS . Popular choices for the model include tree-based methods ( where the features are the adjacency matrix representation of the architectures ) [ 33 , 55 ] , graph neural networks [ 36 , 54 ] , Gaussian processes [ 47 , 51 ] , and neural networks based on specialized encodings of the architecture [ 69 , 42 ] . Learning curve-based methods . Another family predicts the final performance of architectures using only a partially trained network , by extrapolating the learning curve . This is accomplished by fitting the partial learning curve to an ensemble of parametric models [ 11 ] , or by simply summing the training losses observed so far [ 50 ] . Early stopping as described earlier is also a learning curve-based method . Learning curve methods do not require any initialization time , yet the query time typically takes minutes or hours , which is orders of magnitude slower than the query time in model-based methods . Learning curve-based methods can be used in conjunction with multi-fidelty algorithms , such as Hyperband or BOHB [ 27 , 16 , 24 ] . Hybrid methods . Some predictors are hybrids between learning curve and model-based methods . These predictors train a model at initialization time to predict f ( a ) given both a and a partial learning curve of a as features . Models in prior work include an SVR [ 2 ] , or a Bayesian neural network [ 20 ] . Although the query time and initialization time are both high , hybrid predictors tend to have strong performance . Zero-cost methods . Another class of predictors have no initialization time and very short query times ( so-called “ zero-cost ” methods ) . These predictors compute statistics from just a single forward/backward propagation pass for a single minibatch of data , by computing the correlation of activations within a network [ 38 ] , or by adapting saliency metrics proposed in pruning-at-initialization literatures [ 23 , 1 ] . Similar to learning curve-based methods , since the only computation is specific to each architecture , the initialization time is zero . Zero-cost methods have recently been used to warm start NAS algorithms [ 1 ] . Weight sharing methods . Weight sharing [ 45 ] is a popular approach to substantially speed up NAS , especially in conjunction with a one-shot algorithm [ 32 , 12 ] . In this approach , all architectures in the search space are combined to form a single over-parameterized supernetwork . By training the weights of the supernetwork , all architectures in the search space can be evaluated quickly using this set of weights . To this end , the supernetwork can be used as a performance predictor . This results in NAS algorithms [ 32 , 28 ] which are significantly faster than sequential NAS algorithms , such as evolution or Bayesian optimization . Recent work has shown that although the shared weights are sometimes not effective at ranking architectures [ 53 , 74 , 76 ] , one-shot NAS techniques using shared weights still achieve strong performance [ 73 , 25 ] . Tradeoff between intialization and query time . The main families mentioned above all have different initialization and query times . The tradeoffs between initialization time , query time , and performance depend on a few factors such as the type of NAS algorithm and its total runtime budget , and different settings are needed in different situations . For example , if there are many architectures whose performance we want to estimate , then we should have a low query time , and if we have a high total runtime budget , then we can afford a high initialization time . We may also change our runtime budget throughout the run of a single NAS algorithm . For example , at the start of a NAS algorithm , we may want to have coarse estimates of a large number of architectures ( low initialization time , low query time such as zero-cost predictors ) . As the NAS algorithm progresses , it is more desirable to receive higher-fidelity predictions on a smaller set of architectures ( model-based or hybrid predictors ) . The exact budgets depend on the type of NAS algorithm . Choice of performance predictors . We analyze 31 performance predictors defined in prior work : BANANAS [ 69 ] , Bayesian Linear Regression [ 6 ] , BOHAMIANN [ 57 ] , BONAS [ 54 ] , DNGO [ 56 ] , Early Stopping with Val . Acc . ( e.g . [ 77 , 27 , 16 , 79 ] ) Early Stopping with Val . Loss . [ 50 ] , Fisher [ 1 ] , Gaussian Process ( GP ) [ 48 ] , GCN [ 75 ] , Grad Norm [ 1 ] , Grasp [ 64 ] , Jacobian Covariance [ 38 ] , LCE [ 11 ] , LCE-m [ 20 ] , LcSVR [ 2 ] , LGBoost/GBDT [ 33 ] , MLP [ 69 ] , NAO [ 35 ] , NGBoost [ 55 ] , OneShot [ 73 ] , Random Forest ( RF ) [ 55 ] , Random Search with Weight Sharing ( RSWS ) [ 26 ] , SemiNAS [ 34 ] , SNIP [ 23 ] , SoTL [ 50 ] , SoTL-E [ 50 ] , Sparse GP [ 3 ] , SynFlow [ 61 ] , Variational Sparse GP [ 63 ] , and XGBoost [ 55 ] . For any method that did not have an architecture encoding already defined ( such as the tree-based methods , GP-based methods , and Bayesian Linear Regression ) , we use the standard adjacency matrix encoding , which consists of the adjacency matrix of the architecture along with a one-hot list of the operations [ 71 , 68 ] . By open-sourcing our code , we encourage 104 105 106 Init . time ( seconds ) 101 102 103 Qu er y tim e ( s ec on ds ) .93 .93 .93 .93 .93 .93 .93 .93 .93 .93 .93.93 .4 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .6 .58 .62 .65 .6 .63 .66 .6 .64 .67 .7 .65 .67 .7 .69 .72 .74 .84 .73 .75 .77 .85 .63 .69 .74 .77 .8 .82 .83 .84 .84 .63 .69 .74 .77 .8 .82 .83 .84 .84 .63 .69 .74 .77 .8 .82 .83 .84 .84 .63 .69 .74 .77 .8 .82 .83 .84 .84 .63 .69 .74 .77 .8 .82 .83 .84 .84 .69 .74 .77 .8 .82 .83 .84 .84 .69 .74 .77 .8 .82 .83 .84 .84 .74 .77 .8 .82 .83 .84 .84 .74 .77 .8 .82 .83 .84 .84 .77 .8 .82 .83 .84 .8 .82 .83 .84 .63 .69 .74 .77 .8 .82 .83 .84 .84 .28 .42 .46 .5 .53 .57 .6 .6 .6 .6 .63 .63 .63 .63 .63 .68 .68 .68 .68 .68 .68 .73 .73 .73 .73 .73 .73 .73 Kendall Tau on NAS-Bench-201 CIFAR-10 105 106 107 Init . time ( seconds ) 102 103 104 Qu er y tim e ( s ec on ds ) NAS-Bench-NLP 105 106 107 Init . time ( seconds ) 102 103 104 Qu er y tim e ( s ec on ds ) NAS-Bench-201 ImageNet16-120 104 105 106 Init . time ( seconds ) 101 102 103 Qu er y tim e ( s ec on ds ) NAS-Bench-101 104 105 Init . time ( seconds ) 101 102 103 Qu er y tim e ( s ec on ds ) DARTS 5.00 5.25 5.50 5.75 6.00 5.0 5.2 5.4 5.6 5.8 6.0 BANANAS Bayes . Lin . Reg . Early Stop ( Acc . ) GCN Jacob . Cov . LGBoost LcSVR NGBoost SoTL-E SemiNAS SynFlow XGBoost Figure 2 : The performance predictors with the highest Kendall Tau values for all initialization time and query time budgets on NAS-Bench-201 , NAS-Bench-101 , NAS-Bench-NLP and DARTS . For example , on NAS-Bench-201 CIFAR-10 ( left ) with an initialization time of 106 seconds and query time of 10 seconds , XGBoost achieves a Kendall Tau value of .73 which is the highest value out of the 31 predictors that we tested at that budget . implementing more ( existing and future ) performance predictors which can then be compared to the 31 which we focus on in this work . In Section B.1 , we give descriptions and detailed implementation details for each performance predictor . In Section D , we give a table that describes for which predictors we were able to reproduce published results , and for which predictors it is not possible ( e.g. , since some predictors were released before the creation of NAS benchmarks ) .

Summary: This paper compares 3 kinds of performance predictors, model based predictors, learning curve based predictor and zero nas predictors. They benchmark the performance of all 31 predictors against 5 benchmarks: NAS-Bench101, DARTS, NASBench201 and NLP.  In addition to this, they also incorporate SOTL-E and jacobian variance as addition features to NGBOOST and SemiNAS predictors and demonstrate that it improves the performance over all 3 of them.