NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy

1 INTRODUCTION . Automated methods for neural network design , referred to as neural architecture search ( NAS ) , have been used to find architectures that are more efficient and more accurate than the best manually designed architectures ( Zoph et al. , 2018 ; Real et al. , 2019 ; So et al. , 2019 ) . However , it is notoriously challenging to provide fair comparisons among NAS methods due to potentially high computational complexity ( Zoph & Le , 2017 ; Real et al. , 2019 ) and the use of different training pipelines and search spaces ( Li & Talwalkar , 2019 ; Lindauer & Hutter , 2020 ) , resulting in the conclusion that “ NAS evaluation is frustratingly hard ” ( Yang et al. , 2020 ) . To make fair , statistically sound comparisons of NAS methods more accessible , tabular NAS benchmarks have been released ; these exhaustively evaluate all architectures in a given search space , storing the relevant training metrics in a lookup table ( Ying et al. , 2019 ; Dong & Yang , 2020 ; Zela et al. , 2020b ; Mehrotra et al. , 2021 ) . This substantially lowers the computational overhead of NAS experiments , since the performance of an architecture can be found simply by querying these tables , hence allowing for a rigorous comparison of various NAS algorithms with minimal computation . While early tabular NAS benchmarks , such as NAS-Bench-101 ( Ying et al. , 2019 ) and NAS-Bench201 ( Dong & Yang , 2020 ) , have been widely adopted by the community , they are limited to small search spaces and focus solely on image classification . Recently , benchmarks have been introduced for natural language processing ( Klyuchnikov et al. , 2020 ) , speech recognition ( Mehrotra et al. , 2021 ) , object detection , and self-supervised tasks ( Duan et al. , 2021 ) . Furthermore , the release of surrogate NAS benchmarks ( Siems et al. , 2020 ; Yan et al. , 2021 ) , which estimate the performance of all architectures in a search space via a surrogate model , has removed the constraint of exhaustively evaluating the entire search space , expanding the scope of possible search space sizes to 1018 and beyond . However , substantial differences in the abstractions ( such as whether a node or an edge denotes an operation ) , capabilities ( such as whether all , or only some , of the architectures can be queried ) , and implementations ( such as incompatible deep learning libraries ) have so far prevented nearly all research in NAS from providing results on more than two families of benchmarks . Overall , the lack of consistency in “ NAS-Bench ” datasets has significantly slowed their collective adoption . In this work , we show that there is a need to adopt newer benchmarks because many conclusions drawn from a small subset of benchmarks do not generalize across diverse datasets and tasks . Specifically , we present an in-depth analysis of popular black-box ( Real et al. , 2019 ; White et al. , 2021a ; Ottelander et al. , 2021 ) , one-shot ( Liu et al. , 2019b ; Chen et al. , 2021 ; Dong & Yang , 2019 ) , and performance prediction methods ( White et al. , 2021c ) across 25 different combinations of search spaces and datasets . To the best of our knowledge , this is by far the largest set of NAS search spaces and datasets on which experiments have been conducted to date . We show that many implicit assumptions in the NAS community are wrong . First , if a NAS algorithm does well on NAS-Bench101 and NAS-Bench-201 , it does not necessarily perform well on other search spaces . Second , NAS algorithms may not have robust default hyperparameters and therefore require tuning . Finally , tuning the hyperparameters of a NAS method on one search space and transferring these hyperparameters to other search spaces often make the NAS method perform significantly worse . In order to help NAS researchers and practitioners avoid these pitfalls , we release the NAS Benchmark Suite ( NAS-Bench-Suite ) , a comprehensive and extensible collection of NAS benchmarks , accessible through a unified interface , created with the aim to facilitate reproducible , generalizable , and rapid NAS research . Our work eliminates the overhead for NAS research to evaluate on several different datasets and problem types , helping the community to develop NAS methods that generalize to new problem types and unseen datasets . See Figure 1 for an overview . To ensure reproducibility and other best practices , we release our code and adhere to the NAS best practices checklist ( Lindauer & Hutter , 2020 , see Section 8 for details ) . Our contributions . We summarize our main contributions below . • We conduct a comprehensive study of the generalizability of NAS algorithms and their hyperparameters across 25 settings , showing that it is often not sufficient to tune on just a few benchmarks , and showing that the best hyperparameters depend on the specific search space . • We introduce a unified benchmark suite , NAS-Bench-Suite , which implements 25 different combinations of search spaces and datasets . By making it easy to quickly and comprehensively evaluate new NAS algorithms on a broad range of problems , our benchmark suite can improve experimental rigor and generalizability in NAS research . 2 NAS BENCHMARKS OVERVIEW . Preliminaries . A search space in NAS is the set of all architectures that the NAS algorithm is allowed to select . Most recent search spaces are defined by a cell-based ( micro ) structure and a macro structure . A cell is a small set of neural network operations arranged in a directed acyclic graph ( DAG ) , with constraints on the number of nodes , edges , and incoming edges per node . The macro structure consists of the architecture skeleton and the arrangement of cells , such as how many times each cell is duplicated . For many popular search spaces , the macro structure is completely fixed , while for other search spaces , the macro structure can have variable length , width , and number of channels for different architectures in the search space . A NAS benchmark ( Lindauer & Hutter , 2020 ) consists of a dataset ( with a fixed train-test split ) , a search space , and a fixed evaluation pipeline with predefined hyperparameters for training the architectures . A tabular NAS benchmark is one that additionally provides precomputed evaluations with that training pipeline for all possible architectures in the search space . Finally , a surrogate NAS benchmark ( Siems et al. , 2020 ; Yan et al. , 2021 ) is a NAS benchmark that provides a surrogate model that can be used to predict the performance of any architecture in the search space . We say that a NAS benchmark is queryable if it is either a tabular or surrogate benchmark . Queryable NAS benchmarks can be used to simulate NAS experiments very cheaply by querying the performance of neural networks ( using a table or a surrogate ) instead of training the neural networks from scratch . NAS benchmarks . Now we describe the characteristics of many popular NAS benchmarks . For a summary , see Table 1 , and for a more comprehensive and detailed survey , see Appendix A . The first tabular NAS benchmark to be released was NAS-Bench-101 ( Ying et al. , 2019 ) . This benchmark consists of 423 624 architectures trained on CIFAR-10 . The cell-based search space consists of a directed acyclic graph ( DAG ) structure in which the nodes can take on operations . A follow-up work , NAS-Bench-1Shot1 ( Zela et al. , 2020b ) , defines three subsets of NAS-Bench-101 which allow one-shot algorithms to be run . The largest subset size in NAS-Bench-1Shot1 is 363 648 . NAS-Bench-201 Dong & Yang ( 2020 ) is another popular tabular NAS benchmark . The cell-based search space consists of a DAG where each edge can take on operations ( in contrast to NAS-Bench101 , in which the nodes are operations ) . The number of non-isomorphic architectures is 6 466 and all are trained on CIFAR-10 , CIFAR-100 , and ImageNet-16-120 . NATS-Bench ( Dong et al. , 2021 ) is an extension of NAS-Bench-201 which also varies the macro architecture . NAS-Bench-NLP ( Klyuchnikov et al. , 2020 ) is a NAS benchmark for natural language processing , which is size 1053 . However , only 14 322 of the architectures were trained on Penn TreeBank ( Mikolov et al. , 2010 ) , meaning NAS-Bench-NLP is not queryable . NAS-Bench-ASR ( Mehrotra et al. , 2021 ) is a tabular NAS benchmark for automatic speech recognition . The search space consists of 8 242 architectures trained on the TIMIT dataset . TransNAS-Bench ( Duan et al. , 2021 ) is a tabular NAS benchmark consisting of two separate search spaces ( cell-level and macro-level ) and seven tasks including pixel-level prediction , regression , and self-supervised tasks . The cell and macro search spaces are size 4 096 and 3 256 , respectively . NAS-Bench-MR ( Ding et al. , 2021 ) is a surrogate NAS benchmark which evaluates across four datasets : ImageNet50-1000 , Cityscapes , KITTI , and HMDB51 . NAS-Bench-MR consists of a single search space of size 1023 . The DARTS ( Liu et al. , 2019b ) search space with CIFAR-10 , consisting of 1018 architectures , is arguably the most widely-used NAS benchmark . Recently , 60 000 of the architectures were trained and used to create NASBench-301 ( Siems et al. , 2020 ) , the first surrogate NAS benchmark . More recently , NAS-Bench-111 , NAS-Bench-311 , and NAS-Bench-NLP11 ( Yan et al. , 2021 ) were released as surrogate benchmarks that extend NAS-Bench-101 , NAS-Bench-301 , and NAS-Bench-NLP by predicting the full learning curve information . 3 NAS BENCHMARK STATISTICS . In this section and in Appendix B , we use NAS-Bench-Suite to compute a set of aggregate statistics across a large set of NAS benchmarks . There is a high variance with respect to the distribution of accuracies and other statistics across benchmarks due to substantial differences in the tasks performed and layout of the search space . It is essential to keep this in mind to ensure a fair comparison of the performance of NAS algorithms across these benchmarks . To the best of our knowledge , this is the first large-scale aggregation of statistics computed on NAS benchmarks . Figure 2 shows box plots for the validation accuracy distribution for a representative set of the 25 NAS benchmarks . We find that TransNAS-Bench ( Sem . Segment ) and DARTS achieve the highest median and maximum accuracies , yet they also have among the smallest variance in validation accuracy across the search space . On the other hand , the search space with the highest interquartile range is TransNAS-Bench Jigsaw . In Figure 3 , we assess the level of locality in each search space , or the similarity of validation accuracy among neighboring architectures ( architectures which differ by a single operation or edge ) using the random walk autocorrelation ( RWA ) ( Weinberger , 1990 ; Ying et al. , 2019 ; White et al. , 2021b ) . RWA computes the autocorrelation of accuracies of architectures during a random walk , in which each step perturbs one operation or edge . We see that NAS-Bench-201 ImageNet16-120 has the highest autocorrelation , while NAS-Bench-101 has the lowest . In Appendix B , we also discuss plots describing the average runtime for training architectures and the average neighborhood size , for each NAS benchmark . Overall , we see substantial differences among the search spaces along the various axes that we tested . Overall , we find that the diversity is important to keep into context when comparing across many different NAS benchmarks . For example , it is more impressive if a NAS algorithm discovers an architecture within 0.1 % of the optimal on NAS-Bench-201 ImageNet16-120 , compared to DARTS , because the standard deviation of accuracies for DARTS is much lower . Additional factors , such as locality and neighborhood size , also affect the difficulty of NAS benchmarks for some NAS algorithms more than for others ; for example , locality has a large effect on the performance of regularized evolution but not for random search .

Summary: The authors collected most of the existing NAS benchmarks to construct a new benchmark. A unified API is provided to use these existing search spaces and architecture datasets. Based on this the authors re-analyze some NAS algorithms on this new large and comprehensive benchmark and have some interesting observations.