NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search

1 INTRODUCTION . Neural Architecture Search ( NAS ) promises to advance representation learning by automatically finding architectures that facilitate the learning of strong representations for a given dataset . NAS has already achieved state-of-the-art performance on many tasks ( Real et al. , 2019 ; Liu et al. , 2019a ; Saikia et al. , 2019 ; Elsken et al. , 2020 ) and to create resource-aware architectures ( Tan et al. , 2018 ; Elsken et al. , 2019a ; Cai et al. , 2020 ) . For a review , we refer to Elsken et al . ( 2019b ) . Despite many advancements in terms of both efficiency and performance , empirical evaluations in NAS are still problematic . Different NAS papers often use different training pipelines , different search spaces and different hyperparameters , do not evaluate other methods under comparable settings , and can not afford enough runs for testing significance . This practice impedes assertions about the statistical significance of the reported results , recently brought into focus by several authors ( Yang et al. , 2019 ; Lindauer & Hutter , 2019 ; Shu et al. , 2020 ; Yu et al. , 2020 ) . To circumvent these issues and enable scientifically sound evaluations in NAS , several tabular benchmarks ( Ying et al. , 2019 ; Zela et al. , 2020b ; Dong & Yang , 2020 ; Klyuchnikov et al. , 2020 ) have been proposed recently ( see also Appendix A.1 for more details ) . However , all these benchmarks rely on an exhaustive evaluation of all architectures in a search space , which limits them to unrealistically small search spaces ( so far containing only between 6k and 423k architectures ) . This is a far shot from standard spaces used in the NAS literature , which contain more than 1018 architectures ( Zoph & Le , 2017 ; Liu et al. , 2019b ) . This discrepancy can cause results gained on existing tabular NAS benchmarks to not generalize to realistic search spaces ; e.g. , promising anytime results of local search on existing tabular NAS benchmarks were shown to not transfer to realistic search spaces ( White et al. , 2020b ) . To address these problems , we make the following contributions : 1 . We present NAS-Bench-301 , a surrogate NAS benchmark that is first to cover a realistically-sized search space ( namely the cell-based search space of DARTS ( Liu et al. , 2019b ) ) , containing more than 1018 possible architectures . This is made possible by estimating their performance via a surrogate model , removing the constraint to exhaustively evaluate the entire search space . 2 . We empirically demonstrate that a surrogate fitted on a subset of architectures can in fact model the true performance of architectures better than a tabular benchmark ( Section 2 ) . 3 . We analyze and release the NAS-Bench-301 training dataset consisting of∼60k fully trained and evaluated architectures , which will also be publicly available in the Open Graph Benchmark ( Hu et al. , 2020 ) ( Section 3 ) . 4 . Using this dataset , we thoroughly evaluate a variety of regression models as surrogate candidates , showing that strong generalization performance is possible even in large spaces ( Section 4 ) . 5 . We utilize NAS-Bench-301 as a benchmark for running various NAS optimizers and show that the resulting search trajectories closely resemble the ground truth trajectories . This enables sound simulations of thousands of GPU hours in a few seconds on a single CPU machine ( Section 5 ) . 6 . We demonstrate that NAS-Bench-301 can help in generating new scientific insights by studying a previous hypothesis on the performance of local search in the DARTS search space ( Section 6 ) . To foster reproducibility , we open-source all our code and data in a public repo : https : // anonymous.4open.science/r/3f99ef91-c472-4394-b666-5d464e099aca/ 2 MOTIVATION – CAN WE DO BETTER THAN A TABULAR BENCHMARK ? . We start by motivating the use of surrogate benchmarks by exposing an issue of tabular benchmarks that has largely gone unnoticed . Tabular benchmarks are built around a costly , exhaustive evaluation of all possible architectures in a search space , and when an architecture ’ s performance is queried , the tabular benchmark simply returns the respective table entry . The issue with this process is that the stochasticity of mini-batch training is also reflected in the performance of an architecture i , hence making it a random variable Yi . Therefore , the table only contains results of a few draws yi ∼ Yi ( existing NAS benchmarks feature up to 3 runs per architecture ) . Given the variance in these evaluations , a tabular benchmark acts as a simple estimator that assumes independent random variables , and thus estimates the performance of an architecture based only on previous evaluations of the same architecture . From a machine learning perspective , knowing that similar architectures tend to yield similar performance , and that the variance of individual evaluations can be high ( both shown to be the case by Ying et al . ( 2019 ) ) , it is natural to assume that better estimators may exist . In the remainder of this section , we empirically verify this hypothesis and show that surrogate benchmarks can provide better performance estimates than tabular benchmarks based on less data . Setup We choose NAS-Bench-101 ( Ying et al. , 2019 ) as a tabular benchmark for our analysis and a Graph Isomorphism Network ( GIN , Xu et al . ( 2019a ) ) as our surrogate model.1 Each architecture xi in NAS-Bench-101 contains 3 validation accuracies y1i , y 2 i , y 3 i from training xi with 3 different seeds . We excluded all diverged models with less than 50 % validation accuracy on any of the three evaluations in NAS-Bench-101 . We split this dataset to train the GIN surrogate model on one of the seeds , e.g. , Dtrain = { ( xi , y1i ) } i and evaluate on the other two , e.g. , Dtest = { ( xi , ȳ23i ) } i , where ȳ23i = ( y2i + y3i ) /2 . We emphasize that training a surrogate to model a search space is not a typical inductive regression task but rather a transductive one . By definition of the search space , the set of possible architectures is known ahead of time ( although it may be very large ) , hence a surrogate model does not have to generalize to out-ofdistribution data if the training data covers the space well . Results We compute the mean absolute error MAE = ∑ i |ŷi−ȳ 23 i | n of the surrogate model trained on Dtrain = { ( xi , y1i ) } i , where ŷi is predicted validation accuracy and n = |Dtest| . Table 1 shows that the surrogate model yields a lower MAE than the tabular benchmark , i.e . MAE = ∑ i |y 1 i−ȳ 23 i | n . We also report the mean squared error and Kendall tau correlation coefficient in Table 6 in the Appendix showing that the ranking between architectures is also predicted better by the surrogate . 1We used a GIN implementation by Errica et al . ( 2020 ) ; see Appendix B for details on training the GIN . We repeat the experiment in a cross-validation fashion w.r.t to the seeds and conclude : In contrast to a single tabular entry , the surrogate model learns to smooth out the noise.2 Next , we fit the GIN surrogate on subsets of Dtrain and plot how its performance scales with the amount of training data used in Figure 1 . The surrogate model performs better than the tabular benchmark when the training set has more than ∼21,500 architectures . Note that Dtest remains the same as in the previous experiment , i.e. , it includes all architectures in NAS-Bench-101 . As a result , we conclude that : A surrogate model can yield strong predictive performance when only a subset of the search space is available as training data . These empirical findings suggest that we can create reliable surrogate benchmarks for much larger and more realistic NAS spaces , which are infeasible to be exhaustively evaluated as done by tabular benchmarks . In the remainder of the paper , we focus on creating such a benchmark . 3 THE NAS-BENCH-301 DATASET We now describe the NAS-Bench-301 dataset which consists of ∼60k architectures and their performances on CIFAR-10 ( Krizhevsky , 2009 ) sampled from the most popular NAS cell search space : the one from DARTS ( Liu et al. , 2019b ) . We use this dataset not only to fit surrogate models but also to gain new insights , such as which regions of the architecture space are being explored by different NAS methods , or what the characteristics of architectures are that work well . 3.1 DATA COLLECTION . Since the DARTS search space ( detailed description in Appendix C.1 ) is far too large to be exhaustively evalu- ated , care has to be taken when sampling the architectures which will be used to train the surrogate models . Sampling should yield a good overall coverage of the architecture space while also providing a special focus on the well-performing regions that optimizers tend to exploit . Our principal methodology is inspired by Eggensperger et al . ( 2015 ) , who collected unbiased data about hyperparameter spaces by random search , as well as biased and dense samples in high-performance regions by running hyperparameter optimizers . This is desirable for a surrogate benchmark since we are interested in evaluating NAS methods that exploit such good regions of the space . Table 2 lists the NAS methods we used to collect such samples and the respective number of samples . Additionally , we evaluated ∼1k architectures in poorly-performing regions for better coverage and another ∼10k for the analysis conducted on the dataset and surrogates . We refer to Appendices C.2 and C.3 for details on the data collection and the optimizers , respectively . We would like to point out that in hindsight adding training data of well-performing regions may be less important for a surrogate NAS benchmark than for a surrogate HPO benchmark , which we demonstrated in Appendix E.3 . We argue that this is a result of HPO 2We do note that the average estimation error of tabular benchmarks could be reduced by a factor of √ k by performing k runs for each architecture . The error of a surrogate model would also shrink when the model is based on more data , but as k grows large tabular benchmarks would become competitive with surrogate models . search spaces containing many configurations which yield disfunctional models , which is less common for architectures in many NAS search spaces , hence allowing random search to give us good coverage of the space . In Figure 2 , we visualize the overall coverage of the search space as well as the similarity between sampled architectures using t-SNE ( van der Maaten & Hinton , 2008 ) . Besides showing a good overall coverage , some well-performing architectures in the search space form distinct clusters which are mostly located outside the main cloud of points . This clearly indicates that architectures with similar performance are close to each other in the architecture space . Additionally , we observe that different optimizers sample different types of architectures , see Figure 9 in the Appendix . 3.2 PERFORMANCE STATISTICS . Figure 3 shows the validation error on CIFAR-10 ( Krizhevsky , 2009 ) of all sampled architectures in relation to the model parameters and training runtime . Generally , as expected , models with more parameters are more costly to train but achieve lower validation errors . We also find that different NAS methods yield quite different performance distributions ( see Appendix C.4 for their individual performances ) . Validation and test errors are highly correlated with a Kendall tau rank correlation of τ = 0.852 ( Spearman rank corr . 0.969 ) , minimizing the risk of overfitting on the validation error . Furthermore , we find that cells of all depths can reach a good performance , but shallow topologies are slightly favored in our setting ( see Figure 10 in the Appendix ) . Also , a small number of parameter-free operations ( e.g. , skip connections ) can benefit the performance but featuring many of these significantly deteriorates performance . For the full analysis , see Appendix C.5 . Following standard practice in modern NAS papers ( e.g. , Liu et al . ( 2019b ) ) , we employ various data augmentation techniques during training for more reliable estimates of an architecture ’ s performance . For a description of our full training pipeline , please see Appendix C.6 .

Summary: This work filled an important gap in the NAS benchmarks. The previous benchmarks only contain small search space due to the expensive cost of evaluation of neural architecture. In this search space, random search often becomes competitive in the narrow search space. Thus, to provide meaningful comparison, this work provided a benchmark in a large NAS search space (same as in DARTS), and using  surrogate models to predict validation performance of untrained neural architecture. The empirical results suggested using the surrogate benchmarks resulted in similar optimization trajectory as real evaluations and the author also shows one can derive/validate research ideas quickly with the benchmarks.