NAS-Bench-x11 and the Power of Learning Curves

1 Introduction . In the past few years , algorithms for neural architecture search ( NAS ) have been used to automatically find architectures that achieve state-of-the-art performance on various datasets [ 82 , 53 , 38 , 13 ] . In 2019 , there were calls for reproducible and fair comparisons within NAS research [ 34 , 57 , 75 , 36 ] due to both the lack of a consistent training pipeline between papers and experiments with not enough trials to reach statistically significant conclusions . These concerns spurred the release of tabular benchmarks , such as NAS-Bench-101 [ 76 ] and NAS-Bench-201 [ 11 ] , created by fully training all architectures in search spaces of size 423 624 and 6 466 , respectively . These benchmarks allow researchers to easily simulate NAS experiments , making it possible to run fair NAS comparisons and to run enough trials to reach statistical significance at very little computational cost or carbon emissions [ 17 ] . Recently , to extend the benefits of tabular NAS benchmarks to larger , more realistic NAS search spaces which can not be evaluated exhaustively , it was proposed to construct surrogate benchmarks [ 60 ] . The first such surrogate benchmark is NAS-Bench-301 [ 60 ] , which models the DARTS [ 38 ] search space of size 1018 architectures . It was created by fully training 60 000 architectures ( both drawn randomly and chosen by top NAS methods ) and then fitting a surrogate model that can estimate the performance of all of the remaining architectures . Since 2019 , dozens of papers have used these NAS benchmarks to develop new algorithms [ 67 , 55 , 73 , 64 , 59 ] . An unintended side-effect of the release of these benchmarks is that it became significantly easier to devise single fidelity NAS algorithms : when the NAS algorithm chooses to evaluate an architecture , the architecture is fully trained and only the validation accuracy at the final epoch of training is outputted . This is because NAS-Bench-301 only contains the architectures ’ accuracy at epoch 100 , ∗Equal contribution . Correspondence to yanshen6 @ msu.edu , colin @ abacus.ai , ysavani @ cs.cmu.edu , fh @ cs.uni-freiburg.de . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and NAS-Bench-101 only contains the accuracies at epochs 4 , 12 , 36 , and 108 ( allowing single fidelity or very limited multi-fidelity approaches ) . NAS-Bench-201 does allow queries on the entire learning curve ( every epoch ) , but it is smaller in size ( 6 466 ) than NAS-Bench-101 ( 423 624 ) or NAS-Bench-301 ( 1018 ) . In a real world experiment , since training architectures to convergence is computationally intensive , researchers will often run multi-fidelity algorithms : the NAS algorithm can train architectures to any desired epoch . Here , the algorithm can make use of speedup techniques such as learning curve extrapolation ( LCE ) [ 63 , 8 , 1 , 28 ] and successive halving [ 35 , 14 , 32 , 29 ] . Although multi-fidelity techniques are often used in the hyperparameter optimization community [ 20 , 21 , 35 , 14 , 63 ] , they have been under-utilized by the NAS community in the last few years . In this work , we fill in this gap by releasing NAS-Bench-111 , NAS-Bench-311 , and NAS-BenchNLP11 , surrogate benchmarks with full learning curve information for train , validation , and test loss and accuracy for all architectures , significantly extending NAS-Bench-101 , NAS-Bench-301 , and NAS-Bench-NLP [ 30 ] , respectively . With these benchmarks , researchers can easily incorporate multifidelity techniques , such as early stopping and LCE into their NAS algorithms . Our technique for creating these benchmarks can be summarized as follows . We use a training dataset of architectures ( drawn randomly and chosen by top NAS methods ) with good coverage over the search space , along with full learning curves , to fit a model that predicts the full learning curves of the remaining architectures . We employ three techniques to fit the model : ( 1 ) dimensionality reduction of the learning curves , ( 2 ) prediction of the top singular value coefficients , and ( 3 ) noise modeling . These techniques can be used in the future to create new NAS benchmarks as well . To ensure that our surrogate benchmarks are highly accurate , we report statistics such as Kendall Tau rank correlation and Kullback Leibler divergence between ground truth learning curves and predicted learning curves on separate test sets . See Figure 1 for examples of predicted learning curves on the test sets . To demonstrate the power of using the full learning curve information , we present a framework for converting single-fidelity NAS algorithms into multi-fidelity algorithms using LCE . We apply our framework to popular single-fidelity NAS algorithms , such as regularized evolution [ 53 ] , local search [ 68 ] , and BANANAS [ 67 ] , all of which claimed state-of-the-art upon release , showing that they can be further improved across four search spaces . Finally , we also benchmark multi-fidelity algorithms such as Hyperband [ 35 ] and BOHB [ 14 ] alongside single-fidelity algorithms . Overall , our work bridges the gap between different areas of AutoML and will allow researchers to easily develop effective multi-fidelity and LCE techniques in the future . To promote reproducibility , we release our code and we follow the NAS best practices checklist [ 36 ] , providing the details in Appendix A . Our contributions . We summarize our main contributions below . 2 Related Work . NAS has been studied since at least the late 1980s [ 43 , 26 , 62 ] and has recently seen a resurgence [ 82 , 44 , 52 , 22 , 53 , 18 ] . Weight sharing algorithms have become popular due to their computational efficiency [ 2 , 38 , 10 , 79 , 77 , 51 , 78 ] . Recent advances in performance prediction [ 65 , 46 , 59 , 74 , 39 , 69 , 55 , 67 ] and other iterative techniques [ 14 , 45 ] have reduced the runtime gap between iterative and weight sharing techniques . For detailed surveys on NAS , we suggest referring to [ 13 , 71 ] . Learning curve extrapolation . Several methods have been proposed to estimate the final validation accuracy of a neural network by extrapolating the learning curve of a partially trained neural network . Techniques include fitting the partial curve to an ensemble of various parametric functions [ 8 ] , predicting the performance based on the derivatives of the learning curves of partially trained neural network configurations [ 1 ] , summing the training losses [ 54 ] , using the basis functions as the output layer of a Bayesian neural network [ 28 ] , using previous learning curves as basis function extrapolators [ 4 ] , using the positive-definite covariance kernel to capture a variety of training curves [ 63 ] , or using a Bayesian recurrent neural network [ 15 ] . While in this work we focus on multi-fidelity optimization utilizing learning curve-based extrapolation , another main category of methods lie in bandit-based algorithm selection [ 35 , 14 , 29 , 19 , 40 ] , and the fidelities can be further adjusted according to the previous observations or a learning rate scheduler [ 20 , 21 , 27 ] . NAS benchmarks . NAS-Bench-101 [ 76 ] , a tabular NAS benchmark , was created by defining a search space of size 423 624 unique architectures and then training all architectures from the search space on CIFAR-10 until 108 epochs . However , the train , validation , and test accuracies are only reported for epochs 4 , 12 , 36 , and 108 , and the training , validation , and test losses are not reported . NAS-Bench-1shot1 [ 80 ] defines a subset of the NAS-Bench-101 search space that allows one-shot algorithms to be run . NAS-Bench-201 [ 11 ] contains 15 625 architectures , of which 6 466 are unique up to isomorphisms . It comes with full learning curve information on three datasets : CIFAR-10 [ 31 ] , CIFAR-100 [ 31 ] , and ImageNet16-120 [ 7 ] . Recently , NAS-Bench-201 was extended to NATS-Bench [ 9 ] which searches over architecture size as well as architecture topology . Virtually every published NAS method for image classification in the last three years evaluates on the DARTS search space with CIFAR-10 [ 61 ] . The DARTS search space [ 38 ] consists of 1018 neural architectures , making it computationally prohibitive to create a tabular benchmark . To overcome this fundamental limitation and query architectures in this much larger search space , NAS-Bench-301 [ 60 ] evaluates various regression models trained on a sample of 60 000 architectures that is carefully created to cover the whole search space . The surrogate models allow users to query the validation accuracy ( at epoch 100 ) and training time for any of the 1018 architectures in the DARTS search space . However , since the surrogates do not predict the entire learning curve , it is not possible to run multi-fidelity algorithms . NAS-Bench-NLP [ 30 ] is a search space for language modeling tasks . The search space consists of 1053 architectures , of which 14 322 are evaluated on Penn Tree Bank [ 42 ] , containing the training , validation , and test losses/accuracies from epochs 1 to 50 . Since only 14 322 of 1053 architectures can be queried , this dataset can not be directly used for NAS experiments . NAS-Bench-ASR [ 41 ] is a recent tabular NAS benchmark for speech recognition . The search space consists of 8 242 architectures with full learning curve information . For an overview of NAS benchmarks , see Table 1 . 3 Creating Surrogate Benchmarks with Learning Curves . In this section , we describe our technique to create a surrogate model that outputs realistic learning curves , and then we apply this technique to create NAS-Bench-111 , NAS-Bench-311 , and NASBench-NLP11 . Our technique applies to any type of learning curve , including train/test losses and accuracies . For simplicity , the following presentation assumes validation accuracy learning curves . 3.1 General Technique . Given a search space D , let ( xi , yi ) ∼ D denote one datapoint , where xi ∈ Rd is the architecture encoding ( e.g. , one-hot adjacency matrix [ 76 , 66 ] ) , and yi ∈ [ 0 , 1 ] Emax is a learning curve of validation accuracies drawn from a distribution Y ( xi ) based on training the architecture for Emax epochs on a fixed training pipeline with a random initial seed . Without loss of generality , each learning curve yi can be decomposed into two parts : one part that is deterministic and depends only on the architecture encoding , and another part that is based on the inherent noise in the architecture training pipeline . Formally , yi = E [ Y ( xi ) ] + i , where E [ Y ( xi ) ] ∈ [ 0 , 1 ] Emax is fixed and depends only on xi , and i ∈ [ 0 , 1 ] Emax comes from a noise distribution Zi with expectation 0 for all epochs . In practice , E [ Y ( xi ) ] can be estimated by averaging a large set of learning curves produced by training architecture xi with different initial seeds . We represent such an estimate as ȳi . Our goal is to create a surrogate model that takes as input any architecture encoding xi and outputs a distribution of learning curves that mimics the ground truth distribution . We assume that we are given two datasets , Dtrain and Dtest , of architecture and learning curve pairs . We use Dtrain ( often size > 10 000 ) to train the surrogate , and we use Dtest for evaluation . We describe the process of creating Dtrain and Dtest for specific search spaces in the next section . In order to predict a learning curve distribution for each architecture , we split up our approach into two separate processes : we train a model f : Rd → [ 0 , 1 ] Emax to predict the deterministic part of the learning curve , ȳi , and we train a noise model pφ ( | ȳ , x ) , parameterized by φ , to simulate the random draws from Zi . Surrogate model training . Training a model f to predict mean learning curves is a challenging task , since the training datapoints Dtrain consist only of a single ( or few ) noisy learning curve ( s ) yi for each xi . Furthermore , Emax is typically length 100 or larger , meaning that f must predict a high-dimensional output . We propose a technique to help with both of these challenges : we use the training data to learn compression and decompression functions ck : [ 0 , 1 ] Emax → [ 0 , 1 ] k and dk : [ 0 , 1 ] k → [ 0 , 1 ] Emax , respectively , for k Emax . The surrogate is trained to predict compressed learning curves ck ( yi ) of size k from the corresponding architecture encoding xi , and then each prediction can be reconstructed to a full learning curve using dk . A good compression model should not only cause the surrogate prediction to become significantly faster and simpler , but should also reduce the noise in the learning curves , since it would only save the most important information in the compressed representations . That is , ( dk ◦ ck ) ( yi ) should be a less noisy version of yi . Therefore , models trained on ck ( yi ) tend to have better generalization ability and do not overfit to the noise in individual learning curves . We test two compression techniques : singular value decomposition ( SVD ) [ 16 ] and variational autoencoders ( VAEs ) [ 25 ] , and we show later that SVD performs better . We give the details of SVD here and describe the VAE compression algorithm in Appendix C. Formally , we take the singular value decomposition of a matrix S of dimension ( |Dtrain| , Emax ) created by stacking together the learning curves from all architectures in Dtrain . Performing the truncated SVD on the learning curve matrix S allow us to create functions ck and dk that correspond to the optimal linear compression of S. In Figure 2 ( left ) , we see that for architectures in the NAS-Bench-101 search space , there is a steep dropoff of importance after the first six singular values , which intuitively means that most of the information for each learning curve is contained in its first six singular values . In Figure 2 ( middle ) , we compute the mean squared error ( MSE ) of the reconstructed learning curves ( dk ◦ ck ) ( yi ) compared to a test set of ground truth learning curves averaged over several initial seeds ( approximating E [ Y ( xi ) ] ) . The lowest MSE is achieved at k = 6 , which implies that k = 6 is the value where the compression function minimizes reconstruction error without overfitting to the noise of individual learning curves . We further validate this in Figure 2 ( right ) by plotting ( dk ◦ ck ) ( yi ) for different values of k. Now that we have compression and decompression functions , we train a surrogate model with xi as features and ck ( yi ) as the label , for architectures in Dtrain . We test LGBoost [ 23 ] , XGBoost [ 5 ] , and MLPs for the surrogate model . Noise modeling . The final step for creating a realistic surrogate benchmark is to add a noise model , so that the outputs are noisy learning curves . We first create a new dataset of predicted i values , which we call residuals , by subtracting the reconstructed mean learning curves from the real learning curves in Dtrain . That is , ̂i = yi − ( dk ◦ ck ) ( yi ) is the residual for the ith learning curve . Since the training data only contains one ( or few ) learning curve ( s ) per architecture xi , it is not possible to accurately estimate the distribution Zi for each architecture without making further assumptions . We assume that the noise comes from an isotropic Gaussian distribution , and we consider two other noise assumptions : ( 1 ) the noise distribution is the same for all architectures ( to create baselines , or for very homogeneous search spaces ) , and ( 2 ) for each architecture , the noise in a small window of epochs are iid . In Appendix C , we estimate the extent to which all of these assumptions hold true . Assumption ( 1 ) suggests two potential ( baseline ) noise models : ( i ) a simple sample standard deviation statistic , σ ∈ REmax+ , where σj = √√√√ 1 |Dtrain| − 1 |Dtrain|∑ i=1 ̂2i , j . To sample the noise using this model , we sample from N ( 0 , diag ( σ ) ) . ( ii ) The second model is a Gaussian kernel density estimation ( GKDE ) model [ 58 ] trained on the residuals to create a multivariate Gaussian kernel density estimate for the noise . Assumption ( 2 ) suggests one potential noise model : ( iii ) a model that is trained to estimate the distribution of the noise over a window of epochs of a specific architecture . For each architecture and each epoch , the model is trained to estimate the sample standard deviation of the residuals within a window centered around that epoch . Throughout this section , we described two compression methods ( SVD and VAE ) , three surrogate models ( LGB , XGB , and MLP ) , and three noise models ( stddev , GKDE , and sliding window ) , for a total of eighteen disctinct approaches . In the next section , we describe the methods we will use to evaluate these approaches .

Summary: This paper proposes another benchmark dataset for NAS, NAS-Bench-x11. Compared to the original benchmarks NAS-Bench-101, NAS-Bench-301 and NAS-Bench-NLP, NAS-Bench-x11 contains the full training information for each candidate architecture especially the learning curves so that it allows multi-fidelity NAS algorithms  NAS-Bench-x11 is a surrogate benchmark, which trains a surrogate model to predict the learning curve of the vast amount of candidate architectures. Prediction of the learning curve can be accomplished by singular value decomposition and noise modeling.