Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks

The most significant barrier to the advancement of Neural Architecture Search ( NAS ) is its demand for large computational resources , which hinders scientifically sound empirical evaluations of NAS methods . Tabular NAS benchmarks have alleviated this problem substantially , making it possible to properly evaluate NAS methods in seconds on commodity machines . However , an unintended consequence of tabular NAS benchmarks has been a focus on extremely small architectural search spaces since their construction relies on exhaustive evaluations of the space . This leads to unrealistic results that do not transfer to larger spaces . To overcome this fundamental limitation , we propose a methodology to create cheap NAS surrogate benchmarks for arbitrary search spaces . We exemplify this approach by creating surrogate NAS benchmarks on the existing tabular NAS-Bench-101 and on two widely used NAS search spaces with up to 1021 architectures ( 1013 times larger than any previous tabular NAS benchmark ) . We show that surrogate NAS benchmarks can model the true performance of architectures better than tabular benchmarks ( at a small fraction of the cost ) , that they lead to faithful estimates of how well different NAS methods work on the original non-surrogate benchmark , and that they can generate new scientific insight . We open-source all our code and believe that surrogate NAS benchmarks are an indispensable tool to extend scientifically sound work on NAS to large and exciting search spaces . 1 INTRODUCTION . Neural Architecture Search ( NAS ) has seen huge advances in search efficiency , but the field has recently been criticized substantially for non-reproducible research , strong sensitivity of results to carefully-chosen training pipelines , hyperparameters and even random seeds ( Yang et al. , 2019 ; Li & Talwalkar , 2020 ; Lindauer & Hutter , 2020 ; Shu et al. , 2020 ; Yu et al. , 2020 ) . A leading cause that complicates reproducible research in NAS is the computational cost of even just single evaluations of NAS algorithms , not least in terms of carbon emissions ( Patterson et al. , 2021 ; Li et al. , 2021a ) . Tabular NAS benchmarks , such as NAS-Bench-101 ( Ying et al. , 2019 ) and NAS-Bench-201 ( Dong & Yang , 2020 ) , have been a game-changer for reproducible NAS research , for the first time allowing scientifically sound empirical evaluations of NAS methods with many seeds in minutes , while ruling out confounding factors , such as different search spaces , training pipelines or hardware/software versions . This success has recently led to the creation of many additional tabular NAS benchmarks , such as NAS-Bench-1shot1 ( Zela et al. , 2020b ) , NATS-Bench ( Dong et al. , 2021 ) , NAS-HPO-bench ( Klein & Hutter , 2019 ) , NAS-Bench-NLP ( Klyuchnikov et al. , 2020 ) , NAS-Bench-ASR ( Mehrotra et al. , 2021 ) , and HW-NAS-Bench ( Li et al. , 2021b ) ( see Appendix A.1 for more details on these previous NAS benchmarks ) . However , these tabular NAS benchmarks rely on an exhaustive evaluation of all architectures in a search space , limiting them to unrealistically small search spaces ( so far containing only between 6k and 423k architectures ) . This is a far shot from standard spaces used in the NAS literature , which contain more than 1018 architectures ( Zoph & Le , 2017 ; Liu et al. , 2019 ; Wu et al. , 2019a ) . This discrepancy can cause results gained on tabular NAS benchmarks to not generalize to realistic search spaces ; e.g. , promising anytime results of local search on tabular NAS benchmarks were indeed shown to not transfer to realistic search spaces ( White et al. , 2020b ) . Making things worse , as discussed in the panel of the most recent NAS workshop at ICLR 2021 , to succeed in automatically discovering qualitatively new types of architectures ( such as , e.g. , Transformers ( Vaswani et al. , 2017 ) ) the NAS community will have to focus on even more expressive search spaces in the future . To not give up the recent progress in terms of reproducibility that tabular NAS benchmarks have brought , we thus need to develop their equivalent for arbitrary search spaces . That is the goal of this paper . Our contributions . Our main contribution is to introduce the concept of surrogate NAS benchmarks that can be constructed for arbitrary NAS search spaces and allow for the same cheap query interface as tabular NAS benchmarks . We substantiate this contribution as follows : 1 . We demonstrate that a surrogate fitted on a subset of architectures can model the true performance of architectures better than a tabular benchmark ( Section 2 ) . 2 . We showcase our methodology by building surrogate benchmarks on a realistically-sized NAS search space ( up to 1021 possible architectures , i.e. , 1013 times more than any previous tabular NAS benchmark ) , thoroughly evaluating a range of regression models as surrogate candidates , and showing that strong generalization performance is possible even in large spaces ( Section 3 ) . 3 . We show that the search trajectories of various NAS optimizers running on the surrogate benchmarks closely resemble the ground truth trajectories . This enables sound simulations of runs usually requiring thousands of GPU hours in a few seconds on a single CPU machine ( Section 3 ) . 4 . We demonstrate that surrogate benchmarks can help in generating new scientific insights by rectifying a previous hypothesis on the performance of local search in large spaces ( Section 4 ) . To foster reproducibility , we open-source all our code , data , and surrogate NAS benchmarks.1 . 2 MOTIVATION – CAN WE DO BETTER THAN A TABULAR BENCHMARK ? . We start by motivating the use of surrogate benchmarks by exposing an issue of tabular benchmarks that has largely gone unnoticed . Tabular benchmarks are built around a costly , exhaustive evaluation of all possible architectures in a search space , and when an architecture ’ s performance is queried , the tabular benchmark simply returns the respective table entry . The issue with this process is that the stochasticity of mini-batch training is also reflected in the performance of an architecture i , hence making it a random variable Yi . Therefore , the table only contains results of a few draws yi ∼ Yi ( existing NAS benchmarks feature up to 3 runs per architecture ) . Given the variance in these evaluations , a tabular benchmark acts as a simple estimator that assumes independent random variables , and thus estimates the performance of an architecture based only on previous evaluations of the same architecture . From a machine learning perspective , knowing that similar architectures tend to yield similar performance and that the variance of individual evaluations can be high ( both shown to be the case by Ying et al . ( 2019 ) ) , it is natural to assume that better estimators may exist . In the remainder of this section , we empirically verify this hypothesis and show that surrogate benchmarks can provide better performance estimates than tabular benchmarks based on less data . Setup . For the analysis in this section , we choose NAS-Bench-101 ( Ying et al. , 2019 ) as a tabular benchmark and a Graph Isomorphism Network ( GIN , Xu et al . ( 2019a ) ) as our surrogate model . 2 Each architecture xi in NAS-Bench-101 contains 3 validation accuracies y1i , y 2 i , y 3 i from training xi with 3 different seeds . We excluded all diverged models with less than 50 % validation accuracy on any of the three evaluations in NAS-Bench-101 . We split this dataset to train the GIN surrogate model on 1https : //anonymous.4open.science/r/3f99ef91-c472-4394-b666-5d464e099aca 2We used a GIN implementation by Errica et al . ( 2020 ) ; see Appendix B for details on training the GIN . one of the seeds , e.g. , Dtrain = { ( xi , y1i ) } i and evaluate on the other two , e.g. , Dtest = { ( xi , ȳ23i ) } i , where ȳ23i = ( y 2 i + y 3 i ) /2 . Results . We compute the mean absolute error MAE = ∑ i |ŷi−ȳ 23 i | n of the surrogate model trained on Dtrain = { ( xi , y1i ) } i , where ŷi is the predicted validation accuracy and n = |Dtest| . Table 1 shows that the surrogate model yields a lower MAE than the tabular benchmark , i.e . MAE = ∑ i |y 1 i−ȳ 23 i | n . We also report the mean squared error and Kendall tau correlation coefficient in Table 4 in the appendix showing that the ranking between architectures is also predicted better by the surrogate . We repeat the experiment in a cross-validation fashion w.r.t to the seeds and conclude : In contrast to a single tabular entry , the surrogate model learns to smooth out the noise.3 Next , we fit the GIN surrogate on subsets of Dtrain and show in Figure 1 how its performance scales with the amount of training data used . The surrogate model performs better than the tabular benchmark when the training set has more than∼21,500 architectures . ( Note that Dtest remains the same as in the previous experiment , i.e. , it includes all 423k architectures in NAS-Bench-101 . ) As a result , we conclude that : A surrogate model can yield strong predictive performance when only a subset of the search space is available as training data . These empirical findings suggest that we can create reliable surrogate benchmarks for much larger and more realistic NAS spaces , which are infeasible to be exhaustively evaluated ( as would be required to construct tabular benchmarks ) . 3 GOING BEYOND SPACE SIZE LIMITS WITH SURROGATE BENCHMARKS . We now introduce the general methodology that we propose to effectively build realistic and reliable surrogate NAS benchmarks . We showcase this methodology by building surrogate benchmarks on two widely used search spaces and datasets , namely DARTS ( Liu et al. , 2019 ) + CIFAR-10 and FBNet ( Wu et al. , 2019a ) + CIFAR-100 . Having verified our surrogate NAS benchmark methodology on these well-known spaces , we strongly encourage the creation of additional future surrogate NAS benchmarks on a broad range of large and exciting search spaces , and the compute time saved by replacing expensive real experiments on the DARTS or FBNet space with cheap experiments on our surrogate versions of them might already be used for this purpose . We name our surrogate benchmarks depending on the space and surrogate model considered as follows : Surr-NAS-Bench- { space } - { surrogate } ( or , SNB- { space } - { surrogate } for short ) . For example , we introduce SNB-DARTS-GIN , which is a surrogate benchmark on the DARTS space and uses a GIN ( Xu et al. , 2019a ) as a surrogate model . 3.1 GENERAL METHODOLOGY AND SURR-NAS-BENCH-DARTS We first explain our methodology for the DARTS ( Liu et al. , 2019 ) search space , which consists of more than 1018 architectures . We selected this search space for two main reasons : ( 1 ) due to the huge number of papers building on DARTS and extending it , the DARTS search space , applied to CIFAR-10 is the most widely used non-tabular NAS benchmark in the literature , and as such it provides a convincing testbed for our surrogate benchmarks4 ; and ( 2 ) the surrogate benchmark we construct frees up the substantial compute resources that are currently being invested for experiments on this non-tabular benchmark ; we hope that these will instead be used to study additional novel and more exciting spaces and/or datasets . 3We note that the average estimation error of tabular benchmarks could be reduced by a factor of √ k by performing k runs per architecture . The error of surrogate models would also shrink when they are based on more data , but as k grows large tabular benchmarks would become competitive with surrogate models . 4In particular , the alternative of first constructing a new non-tabular benchmark and then building a surrogate benchmark for it would have been susceptible to many confounding factors .

Summary: This work explored how to use surrogate models to expand the existing (and limited) neural architecture search -- NAS -- benchmark. The new expanded benchmark is named as surrogate NAS benchmark. All codes are open-sourced, which demonstrated the good reproducibility of this work. The authors have conducted extensive experiments to demonstrate the usability of this new surrogate NAS benchmark and showed much analysis of existing NAS methods on this new benchmark.