How Powerful are Performance Predictors in Neural Architecture Search?

1 Introduction . Neural architecture search ( NAS ) is a popular area of machine learning , which aims to automate the process of developing neural architectures for a given dataset . Since 2017 , a wide variety of NAS techniques have been proposed [ 78 , 45 , 32 , 49 ] . While the first NAS techniques trained thousands of architectures to completion and then evaluated the performance using the final validation accuracy [ 78 ] , modern algorithms use more efficient strategies to estimate the performance of partially-trained or even untrained neural networks [ 11 , 2 , 54 , 34 , 38 ] . Recently , many performance prediction methods have been proposed based on training a model to predict the final validation accuracy of an architecture just from an encoding of the architecture . Popular choices for these models include Gaussian processes [ 60 , 17 , 51 ] , neural networks [ 36 , 54 , 65 , 69 ] , tree-based methods [ 33 , 55 ] , and so on . However , these methods often require hundreds of fully-trained architectures to be used as training data , thus incurring high initialization time . In contrast , learning curve extrapolation methods [ 11 , 2 , 20 ] need little or no initialization time , but each individual prediction requires partially training the architecture , incurring high query time . Very recently , a few techniques have been introduced which are fast both in query time and initialization time [ 38 , 1 ] , computing predictions based on a single minibatch of data . Finally , using shared weights [ 45 , 4 , 32 ] is a popular paradigm for NAS [ 73 , 25 ] , although the effectiveness of these methods in ranking architectures is disputed [ 53 , 74 , 76 ] . Despite the widespread use of performance predictors , it is not known how methods from different families compare to one another . While there have been some analyses on the best predictors within ∗ { colin , yang } @ abacus.ai , { zelaa , fh } @ cs.uni-freiburg.de , robin @ robots.ox.ac.uk 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . each class [ 41 , 72 ] , for many predictors , the only evaluation is from the original work that proposed the method . Furthermore , no work has previously compared the predictors across different families of performance predictors . This leads to two natural questions : how do zero-cost methods , model-based methods , learning curve extrapolation methods , and weight sharing methods compare to one another across different constraints on initialization time and query time ? Furthermore , can predictors from different families be combined to achieve even better performance ? In this work , we answer the above questions by giving the first large-scale study of performance predictors for NAS . We study 31 predictors across four popular search spaces and four datasets : NAS-Bench-201 [ 13 ] with CIFAR-10 , CIFAR-100 , and ImageNet16-120 , NAS-Bench-101 [ 71 ] and DARTS [ 32 ] with CIFAR-10 , and NAS-Bench-NLP [ 21 ] with Penn TreeBank . In order to give a fair comparison among different classes of predictors , we run a full portfolio of experiments , measuring the Pearson correlation and rank correlation metrics ( Spearman , Kendall Tau , and sparse Kendall Tau ) , across a variety of initialization time and query time budgets . We run experiments using a training and test set of architectures generated both uniformly at random , as well as by mutating the highest-performing architectures ( the latter potentially more closely resembling distributions encountered during an actual NAS run ) . Finally , we test the ability of each predictor to speed up NAS algorithms , namely Bayesian optimization [ 36 , 54 , 69 , 51 ] and predictor-guided evolution [ 66 , 59 ] . Since many predictors so far had only been evaluated on one search space , our work shows which predictors have consistent performance across search spaces . Furthermore , by conducting a study with three axes of comparison ( see Figure 1 ) , and by comparing various types of predictors , we see a more complete view of the state of performance predictor techniques that leads to interesting insights . Notably , we show that the performance of predictors from different families are complementary and can be combined to achieve significantly higher performance . The success of these experiments opens up promising avenues for future work . Overall , our experiments bridge multiple areas of NAS research and act as recommendations for the best predictors to use under different runtime constraints . Our code , based on the NASLib library [ 52 ] , can be used as a testing ground for future performance prediction techniques . In order to ensure reproducibility of the original results , we created a table to clarify which of the 31 predictors had previously published results on a NAS-Bench search space , and how these published results compared to our results ( Table 7 ) . We also adhere to the NeurIPS 2021 checklist along with the specialized NAS best practices checklist [ 31 ] . Our contributions . We summarize our main contributions below . • We conduct the first large-scale study of performance predictors for neural architecture search by comparing model-based methods , learning curve extrapolation methods , zero-cost methods , and weight sharing methods across a variety of settings . • We release a comprehensive library of 31 performance predictors on four different search spaces . • We show that different families of performance predictors can be combined to achieve substantially better predictive power than any single predictor . 2 Related Work . NAS has been studied since at least the 1990s [ 19 , 58 ] , and has been revitalized in the last few years [ 78 ] . While initial techniques focused on reinforcement learning [ 78 , 45 ] and evolutionary search [ 37 , 49 ] , one-shot NAS algorithms [ 32 , 12 , 4 ] and predictor-based NAS algorithms [ 65 , 54 , 69 ] have recently become popular . We give a brief survey of performance prediction techniques in Section 3 . For a survey on NAS , see [ 15 ] . The most widely used type of search space in prior work is the cell-based search space [ 79 ] , where the architecture search is over a relatively small directed acyclic graph representing an architecture . A few recent works have compared different performance predictors on popular cell-based search spaces for NAS . Siems et al . [ 55 ] studied graph neural networks and tree-based methods , and found that gradient-boosted trees and graph isomorphism networks performed the best . However , the comparison was only on a single search space and dataset , and the explicit goal was to achieve maximum performance given a training set of around 60 000 architectures . Another recent paper [ 41 ] studied various aspects of supernetwork training , and separately compared four model-based methods : random forest , MLP , LSTM , and GATES [ 42 ] . However , the comparisons were again on a single search space and dataset and did not compare between multiple families of performance predictors . Other papers have proposed new model-based predictors and compared the new predictors to other model-based baselines [ 34 , 65 , 54 , 69 ] . Finally , a recent paper analyzed training heuristics to make weight-sharing more effective at ranking architectures [ 72 ] . To the best of our knowledge , no prior work has conducted comparisons across multiple families of performance predictors . 3 Performance Prediction Methods for NAS . In NAS , given a search space A , the goal is to find a∗ = argmina∈Af ( a ) , where f denotes the validation error of architecture a after training on a fixed dataset for a fixed number of epochs E. Since evaluating f ( a ) typically takes hours ( as it requires training a neural network from scratch ) , many NAS algorithms make use of performance predictors to speed up this process . A performance predictor f ′ is defined generally as any function which predicts the final accuracy or ranking of architectures , without fully training the architectures . That is , evaluating f ′ should take less time than evaluating f , and { f ′ ( a ) | a ∈ A } should ideally have high correlation or rank correlation with { f ( a ) | a ∈ A } . Each performance predictor is defined by two main routines : an initialization routine which performs general pre-computation , and a query routine which performs the final architecture-specific computation : it takes as input an architecture specification , and outputs its predicted accuracy . For example , one of the simplest performance predictors is early stopping : for any query ( a ) , train a for E/2 epochs instead of E [ 77 ] . In this case , there is no general pre-computation , so initialization time is zero . On the other hand , the query time for each input architecture is high because it involves training the architecture for E/2 epochs . In fact , the runtime of the initialization and query routines varies substantially based on the type of predictor . In the context of NAS algorithms , the initialization routine is typically performed once at the start of the algorithm , and the query routine is typically performed many times throughout the NAS algorithm . Some performance predictors also make use of an update routine , when part of the computation from initialization needs to be updated without running the full procedure again ( for example , in a NAS algorithm , a model may be updated periodically based on newly trained architectures ) . Now we give an overview of the main families of predictors . See Figure 1 ( left ) for a taxonomy of performance predictors . Model-based ( trainable ) methods . The most common type of predictor , the model-based predictor , is based on supervised learning . The initialization routine consists of fully training many architectures ( i.e. , evaluating f ( a ) for many architectures a ∈ A ) to build a training set of datapoints { a , f ( a ) } . Then a model f ′ is trained to predict f ( a ) given a . While the initialization time for model-based predictors is very high , the query time typically takes less than a second , which allows thousands of predictions to be made throughout a NAS algorithm . The model is also updated regularly based on the new datapoints . These predictors are typically used within BO frameworks [ 36 , 54 ] , evolutionary frameworks [ 66 ] , or by themselves [ 67 ] , to perform NAS . Popular choices for the model include tree-based methods ( where the features are the adjacency matrix representation of the architectures ) [ 33 , 55 ] , graph neural networks [ 36 , 54 ] , Gaussian processes [ 47 , 51 ] , and neural networks based on specialized encodings of the architecture [ 69 , 42 ] . Learning curve-based methods . Another family predicts the final performance of architectures using only a partially trained network , by extrapolating the learning curve . This is accomplished by fitting the partial learning curve to an ensemble of parametric models [ 11 ] , or by simply summing the training losses observed so far [ 50 ] . Early stopping as described earlier is also a learning curve-based method . Learning curve methods do not require any initialization time , yet the query time typically takes minutes or hours , which is orders of magnitude slower than the query time in model-based methods . Learning curve-based methods can be used in conjunction with multi-fidelty algorithms , such as Hyperband or BOHB [ 27 , 16 , 24 ] . Hybrid methods . Some predictors are hybrids between learning curve and model-based methods . These predictors train a model at initialization time to predict f ( a ) given both a and a partial learning curve of a as features . Models in prior work include an SVR [ 2 ] , or a Bayesian neural network [ 20 ] . Although the query time and initialization time are both high , hybrid predictors tend to have strong performance . Zero-cost methods . Another class of predictors have no initialization time and very short query times ( so-called “ zero-cost ” methods ) . These predictors compute statistics from just a single forward/backward propagation pass for a single minibatch of data , by computing the correlation of activations within a network [ 38 ] , or by adapting saliency metrics proposed in pruning-at-initialization literatures [ 23 , 1 ] . Similar to learning curve-based methods , since the only computation is specific to each architecture , the initialization time is zero . Zero-cost methods have recently been used to warm start NAS algorithms [ 1 ] . Weight sharing methods . Weight sharing [ 45 ] is a popular approach to substantially speed up NAS , especially in conjunction with a one-shot algorithm [ 32 , 12 ] . In this approach , all architectures in the search space are combined to form a single over-parameterized supernetwork . By training the weights of the supernetwork , all architectures in the search space can be evaluated quickly using this set of weights . To this end , the supernetwork can be used as a performance predictor . This results in NAS algorithms [ 32 , 28 ] which are significantly faster than sequential NAS algorithms , such as evolution or Bayesian optimization . Recent work has shown that although the shared weights are sometimes not effective at ranking architectures [ 53 , 74 , 76 ] , one-shot NAS techniques using shared weights still achieve strong performance [ 73 , 25 ] . Tradeoff between intialization and query time . The main families mentioned above all have different initialization and query times . The tradeoffs between initialization time , query time , and performance depend on a few factors such as the type of NAS algorithm and its total runtime budget , and different settings are needed in different situations . For example , if there are many architectures whose performance we want to estimate , then we should have a low query time , and if we have a high total runtime budget , then we can afford a high initialization time . We may also change our runtime budget throughout the run of a single NAS algorithm . For example , at the start of a NAS algorithm , we may want to have coarse estimates of a large number of architectures ( low initialization time , low query time such as zero-cost predictors ) . As the NAS algorithm progresses , it is more desirable to receive higher-fidelity predictions on a smaller set of architectures ( model-based or hybrid predictors ) . The exact budgets depend on the type of NAS algorithm . Choice of performance predictors . We analyze 31 performance predictors defined in prior work : BANANAS [ 69 ] , Bayesian Linear Regression [ 6 ] , BOHAMIANN [ 57 ] , BONAS [ 54 ] , DNGO [ 56 ] , Early Stopping with Val . Acc . ( e.g . [ 77 , 27 , 16 , 79 ] ) Early Stopping with Val . Loss . [ 50 ] , Fisher [ 1 ] , Gaussian Process ( GP ) [ 48 ] , GCN [ 75 ] , Grad Norm [ 1 ] , Grasp [ 64 ] , Jacobian Covariance [ 38 ] , LCE [ 11 ] , LCE-m [ 20 ] , LcSVR [ 2 ] , LGBoost/GBDT [ 33 ] , MLP [ 69 ] , NAO [ 35 ] , NGBoost [ 55 ] , OneShot [ 73 ] , Random Forest ( RF ) [ 55 ] , Random Search with Weight Sharing ( RSWS ) [ 26 ] , SemiNAS [ 34 ] , SNIP [ 23 ] , SoTL [ 50 ] , SoTL-E [ 50 ] , Sparse GP [ 3 ] , SynFlow [ 61 ] , Variational Sparse GP [ 63 ] , and XGBoost [ 55 ] . For any method that did not have an architecture encoding already defined ( such as the tree-based methods , GP-based methods , and Bayesian Linear Regression ) , we use the standard adjacency matrix encoding , which consists of the adjacency matrix of the architecture along with a one-hot list of the operations [ 71 , 68 ] . By open-sourcing our code , we encourage 104 105 106 Init . time ( seconds ) 101 102 103 Qu er y tim e ( s ec on ds ) .93 .93 .93 .93 .93 .93 .93 .93 .93 .93 .93.93 .4 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .57 .6 .58 .62 .65 .6 .63 .66 .6 .64 .67 .7 .65 .67 .7 .69 .72 .74 .84 .73 .75 .77 .85 .63 .69 .74 .77 .8 .82 .83 .84 .84 .63 .69 .74 .77 .8 .82 .83 .84 .84 .63 .69 .74 .77 .8 .82 .83 .84 .84 .63 .69 .74 .77 .8 .82 .83 .84 .84 .63 .69 .74 .77 .8 .82 .83 .84 .84 .69 .74 .77 .8 .82 .83 .84 .84 .69 .74 .77 .8 .82 .83 .84 .84 .74 .77 .8 .82 .83 .84 .84 .74 .77 .8 .82 .83 .84 .84 .77 .8 .82 .83 .84 .8 .82 .83 .84 .63 .69 .74 .77 .8 .82 .83 .84 .84 .28 .42 .46 .5 .53 .57 .6 .6 .6 .6 .63 .63 .63 .63 .63 .68 .68 .68 .68 .68 .68 .73 .73 .73 .73 .73 .73 .73 Kendall Tau on NAS-Bench-201 CIFAR-10 105 106 107 Init . time ( seconds ) 102 103 104 Qu er y tim e ( s ec on ds ) NAS-Bench-NLP 105 106 107 Init . time ( seconds ) 102 103 104 Qu er y tim e ( s ec on ds ) NAS-Bench-201 ImageNet16-120 104 105 106 Init . time ( seconds ) 101 102 103 Qu er y tim e ( s ec on ds ) NAS-Bench-101 104 105 Init . time ( seconds ) 101 102 103 Qu er y tim e ( s ec on ds ) DARTS 5.00 5.25 5.50 5.75 6.00 5.0 5.2 5.4 5.6 5.8 6.0 BANANAS Bayes . Lin . Reg . Early Stop ( Acc . ) GCN Jacob . Cov . LGBoost LcSVR NGBoost SoTL-E SemiNAS SynFlow XGBoost Figure 2 : The performance predictors with the highest Kendall Tau values for all initialization time and query time budgets on NAS-Bench-201 , NAS-Bench-101 , NAS-Bench-NLP and DARTS . For example , on NAS-Bench-201 CIFAR-10 ( left ) with an initialization time of 106 seconds and query time of 10 seconds , XGBoost achieves a Kendall Tau value of .73 which is the highest value out of the 31 predictors that we tested at that budget . implementing more ( existing and future ) performance predictors which can then be compared to the 31 which we focus on in this work . In Section B.1 , we give descriptions and detailed implementation details for each performance predictor . In Section D , we give a table that describes for which predictors we were able to reproduce published results , and for which predictors it is not possible ( e.g. , since some predictors were released before the creation of NAS benchmarks ) .

Summary: The paper give a large-scale comparison of performance predictors in NAS. The results act as recommendations for the best predictors to use in different settings. The author also show that certain families of predictors can be combined to achieve even better predictive power.