NAS-Bench-x11 and the Power of Learning Curves

1 Introduction . In the past few years , algorithms for neural architecture search ( NAS ) have been used to automatically find architectures that achieve state-of-the-art performance on various datasets [ 82 , 53 , 38 , 13 ] . In 2019 , there were calls for reproducible and fair comparisons within NAS research [ 34 , 57 , 75 , 36 ] due to both the lack of a consistent training pipeline between papers and experiments with not enough trials to reach statistically significant conclusions . These concerns spurred the release of tabular benchmarks , such as NAS-Bench-101 [ 76 ] and NAS-Bench-201 [ 11 ] , created by fully training all architectures in search spaces of size 423 624 and 6 466 , respectively . These benchmarks allow researchers to easily simulate NAS experiments , making it possible to run fair NAS comparisons and to run enough trials to reach statistical significance at very little computational cost or carbon emissions [ 17 ] . Recently , to extend the benefits of tabular NAS benchmarks to larger , more realistic NAS search spaces which can not be evaluated exhaustively , it was proposed to construct surrogate benchmarks [ 60 ] . The first such surrogate benchmark is NAS-Bench-301 [ 60 ] , which models the DARTS [ 38 ] search space of size 1018 architectures . It was created by fully training 60 000 architectures ( both drawn randomly and chosen by top NAS methods ) and then fitting a surrogate model that can estimate the performance of all of the remaining architectures . Since 2019 , dozens of papers have used these NAS benchmarks to develop new algorithms [ 67 , 55 , 73 , 64 , 59 ] . An unintended side-effect of the release of these benchmarks is that it became significantly easier to devise single fidelity NAS algorithms : when the NAS algorithm chooses to evaluate an architecture , the architecture is fully trained and only the validation accuracy at the final epoch of training is outputted . This is because NAS-Bench-301 only contains the architectures ’ accuracy at epoch 100 , ∗Equal contribution . Correspondence to yanshen6 @ msu.edu , colin @ abacus.ai , ysavani @ cs.cmu.edu , fh @ cs.uni-freiburg.de . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and NAS-Bench-101 only contains the accuracies at epochs 4 , 12 , 36 , and 108 ( allowing single fidelity or very limited multi-fidelity approaches ) . NAS-Bench-201 does allow queries on the entire learning curve ( every epoch ) , but it is smaller in size ( 6 466 ) than NAS-Bench-101 ( 423 624 ) or NAS-Bench-301 ( 1018 ) . In a real world experiment , since training architectures to convergence is computationally intensive , researchers will often run multi-fidelity algorithms : the NAS algorithm can train architectures to any desired epoch . Here , the algorithm can make use of speedup techniques such as learning curve extrapolation ( LCE ) [ 63 , 8 , 1 , 28 ] and successive halving [ 35 , 14 , 32 , 29 ] . Although multi-fidelity techniques are often used in the hyperparameter optimization community [ 20 , 21 , 35 , 14 , 63 ] , they have been under-utilized by the NAS community in the last few years . In this work , we fill in this gap by releasing NAS-Bench-111 , NAS-Bench-311 , and NAS-BenchNLP11 , surrogate benchmarks with full learning curve information for train , validation , and test loss and accuracy for all architectures , significantly extending NAS-Bench-101 , NAS-Bench-301 , and NAS-Bench-NLP [ 30 ] , respectively . With these benchmarks , researchers can easily incorporate multifidelity techniques , such as early stopping and LCE into their NAS algorithms . Our technique for creating these benchmarks can be summarized as follows . We use a training dataset of architectures ( drawn randomly and chosen by top NAS methods ) with good coverage over the search space , along with full learning curves , to fit a model that predicts the full learning curves of the remaining architectures . We employ three techniques to fit the model : ( 1 ) dimensionality reduction of the learning curves , ( 2 ) prediction of the top singular value coefficients , and ( 3 ) noise modeling . These techniques can be used in the future to create new NAS benchmarks as well . To ensure that our surrogate benchmarks are highly accurate , we report statistics such as Kendall Tau rank correlation and Kullback Leibler divergence between ground truth learning curves and predicted learning curves on separate test sets . See Figure 1 for examples of predicted learning curves on the test sets . To demonstrate the power of using the full learning curve information , we present a framework for converting single-fidelity NAS algorithms into multi-fidelity algorithms using LCE . We apply our framework to popular single-fidelity NAS algorithms , such as regularized evolution [ 53 ] , local search [ 68 ] , and BANANAS [ 67 ] , all of which claimed state-of-the-art upon release , showing that they can be further improved across four search spaces . Finally , we also benchmark multi-fidelity algorithms such as Hyperband [ 35 ] and BOHB [ 14 ] alongside single-fidelity algorithms . Overall , our work bridges the gap between different areas of AutoML and will allow researchers to easily develop effective multi-fidelity and LCE techniques in the future . To promote reproducibility , we release our code and we follow the NAS best practices checklist [ 36 ] , providing the details in Appendix A . Our contributions . We summarize our main contributions below . 2 Related Work . NAS has been studied since at least the late 1980s [ 43 , 26 , 62 ] and has recently seen a resurgence [ 82 , 44 , 52 , 22 , 53 , 18 ] . Weight sharing algorithms have become popular due to their computational efficiency [ 2 , 38 , 10 , 79 , 77 , 51 , 78 ] . Recent advances in performance prediction [ 65 , 46 , 59 , 74 , 39 , 69 , 55 , 67 ] and other iterative techniques [ 14 , 45 ] have reduced the runtime gap between iterative and weight sharing techniques . For detailed surveys on NAS , we suggest referring to [ 13 , 71 ] . Learning curve extrapolation . Several methods have been proposed to estimate the final validation accuracy of a neural network by extrapolating the learning curve of a partially trained neural network . Techniques include fitting the partial curve to an ensemble of various parametric functions [ 8 ] , predicting the performance based on the derivatives of the learning curves of partially trained neural network configurations [ 1 ] , summing the training losses [ 54 ] , using the basis functions as the output layer of a Bayesian neural network [ 28 ] , using previous learning curves as basis function extrapolators [ 4 ] , using the positive-definite covariance kernel to capture a variety of training curves [ 63 ] , or using a Bayesian recurrent neural network [ 15 ] . While in this work we focus on multi-fidelity optimization utilizing learning curve-based extrapolation , another main category of methods lie in bandit-based algorithm selection [ 35 , 14 , 29 , 19 , 40 ] , and the fidelities can be further adjusted according to the previous observations or a learning rate scheduler [ 20 , 21 , 27 ] . NAS benchmarks . NAS-Bench-101 [ 76 ] , a tabular NAS benchmark , was created by defining a search space of size 423 624 unique architectures and then training all architectures from the search space on CIFAR-10 until 108 epochs . However , the train , validation , and test accuracies are only reported for epochs 4 , 12 , 36 , and 108 , and the training , validation , and test losses are not reported . NAS-Bench-1shot1 [ 80 ] defines a subset of the NAS-Bench-101 search space that allows one-shot algorithms to be run . NAS-Bench-201 [ 11 ] contains 15 625 architectures , of which 6 466 are unique up to isomorphisms . It comes with full learning curve information on three datasets : CIFAR-10 [ 31 ] , CIFAR-100 [ 31 ] , and ImageNet16-120 [ 7 ] . Recently , NAS-Bench-201 was extended to NATS-Bench [ 9 ] which searches over architecture size as well as architecture topology . Virtually every published NAS method for image classification in the last three years evaluates on the DARTS search space with CIFAR-10 [ 61 ] . The DARTS search space [ 38 ] consists of 1018 neural architectures , making it computationally prohibitive to create a tabular benchmark . To overcome this fundamental limitation and query architectures in this much larger search space , NAS-Bench-301 [ 60 ] evaluates various regression models trained on a sample of 60 000 architectures that is carefully created to cover the whole search space . The surrogate models allow users to query the validation accuracy ( at epoch 100 ) and training time for any of the 1018 architectures in the DARTS search space . However , since the surrogates do not predict the entire learning curve , it is not possible to run multi-fidelity algorithms . NAS-Bench-NLP [ 30 ] is a search space for language modeling tasks . The search space consists of 1053 architectures , of which 14 322 are evaluated on Penn Tree Bank [ 42 ] , containing the training , validation , and test losses/accuracies from epochs 1 to 50 . Since only 14 322 of 1053 architectures can be queried , this dataset can not be directly used for NAS experiments . NAS-Bench-ASR [ 41 ] is a recent tabular NAS benchmark for speech recognition . The search space consists of 8 242 architectures with full learning curve information . For an overview of NAS benchmarks , see Table 1 . 3 Creating Surrogate Benchmarks with Learning Curves . In this section , we describe our technique to create a surrogate model that outputs realistic learning curves , and then we apply this technique to create NAS-Bench-111 , NAS-Bench-311 , and NASBench-NLP11 . Our technique applies to any type of learning curve , including train/test losses and accuracies . For simplicity , the following presentation assumes validation accuracy learning curves . 3.1 General Technique . Given a search space D , let ( xi , yi ) ∼ D denote one datapoint , where xi ∈ Rd is the architecture encoding ( e.g. , one-hot adjacency matrix [ 76 , 66 ] ) , and yi ∈ [ 0 , 1 ] Emax is a learning curve of validation accuracies drawn from a distribution Y ( xi ) based on training the architecture for Emax epochs on a fixed training pipeline with a random initial seed . Without loss of generality , each learning curve yi can be decomposed into two parts : one part that is deterministic and depends only on the architecture encoding , and another part that is based on the inherent noise in the architecture training pipeline . Formally , yi = E [ Y ( xi ) ] + i , where E [ Y ( xi ) ] ∈ [ 0 , 1 ] Emax is fixed and depends only on xi , and i ∈ [ 0 , 1 ] Emax comes from a noise distribution Zi with expectation 0 for all epochs . In practice , E [ Y ( xi ) ] can be estimated by averaging a large set of learning curves produced by training architecture xi with different initial seeds . We represent such an estimate as ȳi . Our goal is to create a surrogate model that takes as input any architecture encoding xi and outputs a distribution of learning curves that mimics the ground truth distribution . We assume that we are given two datasets , Dtrain and Dtest , of architecture and learning curve pairs . We use Dtrain ( often size > 10 000 ) to train the surrogate , and we use Dtest for evaluation . We describe the process of creating Dtrain and Dtest for specific search spaces in the next section . In order to predict a learning curve distribution for each architecture , we split up our approach into two separate processes : we train a model f : Rd → [ 0 , 1 ] Emax to predict the deterministic part of the learning curve , ȳi , and we train a noise model pφ ( | ȳ , x ) , parameterized by φ , to simulate the random draws from Zi . Surrogate model training . Training a model f to predict mean learning curves is a challenging task , since the training datapoints Dtrain consist only of a single ( or few ) noisy learning curve ( s ) yi for each xi . Furthermore , Emax is typically length 100 or larger , meaning that f must predict a high-dimensional output . We propose a technique to help with both of these challenges : we use the training data to learn compression and decompression functions ck : [ 0 , 1 ] Emax → [ 0 , 1 ] k and dk : [ 0 , 1 ] k → [ 0 , 1 ] Emax , respectively , for k Emax . The surrogate is trained to predict compressed learning curves ck ( yi ) of size k from the corresponding architecture encoding xi , and then each prediction can be reconstructed to a full learning curve using dk . A good compression model should not only cause the surrogate prediction to become significantly faster and simpler , but should also reduce the noise in the learning curves , since it would only save the most important information in the compressed representations . That is , ( dk ◦ ck ) ( yi ) should be a less noisy version of yi . Therefore , models trained on ck ( yi ) tend to have better generalization ability and do not overfit to the noise in individual learning curves . We test two compression techniques : singular value decomposition ( SVD ) [ 16 ] and variational autoencoders ( VAEs ) [ 25 ] , and we show later that SVD performs better . We give the details of SVD here and describe the VAE compression algorithm in Appendix C. Formally , we take the singular value decomposition of a matrix S of dimension ( |Dtrain| , Emax ) created by stacking together the learning curves from all architectures in Dtrain . Performing the truncated SVD on the learning curve matrix S allow us to create functions ck and dk that correspond to the optimal linear compression of S. In Figure 2 ( left ) , we see that for architectures in the NAS-Bench-101 search space , there is a steep dropoff of importance after the first six singular values , which intuitively means that most of the information for each learning curve is contained in its first six singular values . In Figure 2 ( middle ) , we compute the mean squared error ( MSE ) of the reconstructed learning curves ( dk ◦ ck ) ( yi ) compared to a test set of ground truth learning curves averaged over several initial seeds ( approximating E [ Y ( xi ) ] ) . The lowest MSE is achieved at k = 6 , which implies that k = 6 is the value where the compression function minimizes reconstruction error without overfitting to the noise of individual learning curves . We further validate this in Figure 2 ( right ) by plotting ( dk ◦ ck ) ( yi ) for different values of k. Now that we have compression and decompression functions , we train a surrogate model with xi as features and ck ( yi ) as the label , for architectures in Dtrain . We test LGBoost [ 23 ] , XGBoost [ 5 ] , and MLPs for the surrogate model . Noise modeling . The final step for creating a realistic surrogate benchmark is to add a noise model , so that the outputs are noisy learning curves . We first create a new dataset of predicted i values , which we call residuals , by subtracting the reconstructed mean learning curves from the real learning curves in Dtrain . That is , ̂i = yi − ( dk ◦ ck ) ( yi ) is the residual for the ith learning curve . Since the training data only contains one ( or few ) learning curve ( s ) per architecture xi , it is not possible to accurately estimate the distribution Zi for each architecture without making further assumptions . We assume that the noise comes from an isotropic Gaussian distribution , and we consider two other noise assumptions : ( 1 ) the noise distribution is the same for all architectures ( to create baselines , or for very homogeneous search spaces ) , and ( 2 ) for each architecture , the noise in a small window of epochs are iid . In Appendix C , we estimate the extent to which all of these assumptions hold true . Assumption ( 1 ) suggests two potential ( baseline ) noise models : ( i ) a simple sample standard deviation statistic , σ ∈ REmax+ , where σj = √√√√ 1 |Dtrain| − 1 |Dtrain|∑ i=1 ̂2i , j . To sample the noise using this model , we sample from N ( 0 , diag ( σ ) ) . ( ii ) The second model is a Gaussian kernel density estimation ( GKDE ) model [ 58 ] trained on the residuals to create a multivariate Gaussian kernel density estimate for the noise . Assumption ( 2 ) suggests one potential noise model : ( iii ) a model that is trained to estimate the distribution of the noise over a window of epochs of a specific architecture . For each architecture and each epoch , the model is trained to estimate the sample standard deviation of the residuals within a window centered around that epoch . Throughout this section , we described two compression methods ( SVD and VAE ) , three surrogate models ( LGB , XGB , and MLP ) , and three noise models ( stddev , GKDE , and sliding window ) , for a total of eighteen disctinct approaches . In the next section , we describe the methods we will use to evaluate these approaches .

Summary: The authors describe a mechanism to predict learning curves which they use to generate surrogate benchmarks from existing NAS-Bench versions without learning curves. The authors explore different ways to create learning curve embeddings, different ways to predict these embeddings using the architecture representation as an input and different ways to model the training noise. Furthermore, the authors discuss a short study on using learning curve extrapolation in context of speeding-up NAS.