NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy
Authors: Yash Mehta
Colin White
Arber Zela
Arjun Krishnakumar
Guri Zabergja
Shakiba Moradian
Mahmoud Safari
Kaicheng Yu
Frank Hutter
Venue: ICLR 2022 Poster (2022)
Keywords: neural architecture search
AutoML

Abstract: The release of tabular benchmarks, such as NAS-Bench-101 and NAS-Bench-201, has significantly lowered the computational overhead for conducting scientific research in neural architecture search (NAS). Although they have been widely adopted and used to tune real-world NAS algorithms, these benchmarks are limited to small search spaces and focus solely on image classification. Recently, several new NAS benchmarks have been introduced that cover significantly larger search spaces over a wide range of tasks, including object detection, speech recognition, and natural language processing. However, substantial differences among these NAS benchmarks have so far prevented their widespread adoption, limiting researchers to using just a few benchmarks. In this work, we present an in-depth analysis of popular NAS algorithms and performance prediction methods across 25 different combinations of search spaces and datasets, finding that many conclusions drawn from a few NAS benchmarks do \emph{not} generalize to other benchmarks. To help remedy this problem, we introduce \nasbs, a comprehensive and extensible collection of NAS benchmarks, accessible through a unified interface, created with the aim to facilitate reproducible, generalizable, and rapid NAS research. Our code is available at https://github.com/automl/naslib.

Published as a conference paper at ICLR 2022
NAS-BENCH-SUITE:
NAS EVALUATION IS (Now) SURPRISINGLY EASY
Yash Mehta'; Colin ;hite?; Arber Zela', Arjun Krishnakumar',
Guri Zabergja', Shakiha Moradian', Mahmoud Safari', Kaicheng Yu², Frank Hutter University of Freiburg," Abacus.AI, Bosch Center for Al
ABSTRACT
The release of tabular benchmarks, such as NAS-Bench-101 and NAS-Bench-20) has significantly lowered the computational overhead for conducting scientific research in neural architecture search (NAS). Although they have been widely adopted and used to tune real-world NAS algorithms, these benchmarks are limited to small search spaces and focus solely on image classification. Recently, several new NAS benchmarks have been introduced that cover significantly larger search spaces over wide range of tasks, including object detection, speech recognition, and natural language processing. However, substantial differences among these NAS benchmarks have so far prevented their widespread adoption, limiting searchers to using just a few benchmarks. In this work, we present an in in-depth analysis of popular NAS algorithms and performance prediction methods across 25 different combinations of search spaces and datasets, finding that many conclusions drawn from few NAS benchmarks do not generalize other benchmarks. To help remedy this problem, we introduce NAS -ench Suite, a comprehensive and extensible collection of NAS benchmarks, accessible through a unified inter- face, created with the aim to facilitate reproducible, generalizable, and rapid NAS research. Our code is available at htttt::// com/ aut automm/nasll
INTRODUCTION
Automated methods for neural network design, referred to neural architecture search (NAS), have been used to find architectures that are more efficient and more accurate than the best manually designed architectures (Zoph et al. 2018 Real et al. 2019: So al. 2019). However, it is notoriously challenging to provide fair comparisons among NAS methods due to potentially high computational complexity (Zoph & Le 2017 Real al. 2019) and the use of different training pipelines and search spaces (Li & Talwalkar 2019: Lindauer & Hutter 2020), resulting in the conclusion that "NAS
evaluation is frustratingly hard" (Yang al.]| (020). To make fair, statistically sound comparisons of NAS methods more accessible, tabular NAS benchmarks have been released; these exhaustively evaluate all architectures in a given search space, storing the relevant training metrics in a lookup table (Ying et al. 2019, Dong & Yang. 2020: Zela et al.) 2020b Mehrotra et al. 2021) This substantially lowers the computational overhead NAS experiments, since the performance of an architecture can be found simply by querying these tables, hence allowing for rigorous comparison of various NAS algorithms with minimal computation.
While early tabular NAS benchmarks, such as NAS-Bench-101 (Ying et al. 2019) and NAS-Bench- 201 (Dong& & Yang 2020). have been widely adopted by the community, they are limited to small search spaces and focus solely on image classification. Recently, benchmarks have been introduced for natural language processing (Klyuchnikov et al.] 20220), speech recognition (Mehrotra et al. 2021). object detection, and self-supervised tasks (Duan al. 202T). Furthermore, the release of surrogate NAS benchmarks (Siems et al.) [2020] Yan et al. 2021) which estimate the performance of all architectures in a search space via surrogate model, has removed the constraint of exhaustively evaluating the entire search space, expanding the scope of possible search space sizes to 10 and beyond. However, substantial differences the abstractions (such as whether node or an edge
"Equal contribution. Email to: (mehtay, zel aa) (cs. uni - -frriburgggrr.e de, col in in@abacus.ar

---

Published as a conference paper at ICLR 2022
NAS Derichaarks
Img. Cls
R GP
RS REE
DARTS
NAS Methood
fo NASiFinchh))) 6 NoSSeerc20 c DARTS
Performance Predictors
Algorithm
Black-Box
One-Shot DARTS
Model-Based
Other CV tasks
Randomm Search
CP
Reg. Evalution
GDAS
RF
Local Search
BANANAS
NPENAS
DrNAS
& ROHAKIANN
WNAD
Pvbluate
Speech
NASDenchwAä
-NASBenchNLP
Figure 1: Overview of NAS-Bench-Suite.
denotes an operation), capabilities (such as whether all, or only some, of the architectures can be queried), and implementations (such as incompatible deep learning libraries) have so far prevented nearly all research NAS from providing results more than two families of benchmarks. Overall, the lack of consistency in "NAS-Bench" datasets has significantly slowed their collective adoption.
In this work, we show that there is need adopt newer benchmarks because many conclusions drawn from small subset of benchmarks do not generalize across diverse datasets and tasks. Specifically, we present an in-depth analysis of popular black-box (Real et al. 2019 White al.) 2021a] Ottelander et al. 202T), one-shot (I.iu et al. 2019b; Chen et al. 2021 Dong & Yang. 2019). and performance prediction methods (White et al. 2021c) across (nearly) every publicly available queryable NAS benchmark. This includes 25 different combinations of search spaces and datasets, which is, to the best of our knowledge, by far the largest set NAS search spaces and datasets on which experiments have been conducted to date. We show that many implicit assumptions in the NAS community are wrong. First, a NAS algorithm does well on NAS- Bench- 101 and NAS- Bencch-211, it does not necessarily perform well on other search spaces. Second, NAS algorithms may not have robust default hyperparameters and therefore require tuning. Finally, tuning the hyperparameters of a NAS method on one search space and transferring these hyperparameters other search spaces often make the NAS method perform significantly worse.
In order help NAS researchers and practitioners avoid these pittaals, we release the NAS Benchmark Suite huss-Beech-Suitee accessible through a unified interface, created with the aim facilitate reproducible, generalizable, and rapid NAS research. Our work eliminates the overhead for NAS research to evaluate on several different datasets and problem types, helping the community to develop NAS methods that generalize to new problem types and unseen datasets. See Figure for an overview. To ensure reproducibility and other best practices, we release our code and adhere to the NAS best practices checklist (Lindauer & Hutter 2020, see Section A for details).
comprehensive and extensible collection of NAS benchmarks,
Our contributions. We summarize our main contributions below.
We conduct comprehensive study of the generalizability of NAS algorithms and their hyperpa- rameters across 25 settings, showing that it often not sufficient to tune on just a few benchmarks, and showing that the best hyperparameters depend the specific search space.
We introduce unified benchmark suite, NAS Bench Suite, which implements nearly every publicly available queryable NAS benchmark - 25 different combinations of search spaces and datasets. By making it easy to quickly and comprehensively evaluate new NAS algorithms, our benchmark suite can improve experimental rigor and generalizability in NAS research.
NAS BENCHMARKS OVERVIEW
Preliminaries. A search space in NAS the set of all architectures that the NAS algorithm is allowed to select. Most recent search spaces are defined by a cell-based (micro) structure and a macro structure. cell is a small set of neural network operations arranged directed acyclic graph (DAG), with constraints the number of nodes, edges, and incoming edges per node. The macro structure consists of the architecture skeleton and the arrangement cells, such as how many times each cell is duplicated. For many popular search spaces, the macro structure is completely fixed,
2

---

Published as a conference paper at ICLR 2022
Table 1: Overview of NAS benchmarks in WAS-Bench-Suike
Queryable
Benchmark
Size
Tab.
Surr.
LCs
Macro
Type
#Tasks
NAS-Bench-Suite
NAS-Bench-100
423k
Image class.
NAS-Bench-201
6k
Image class.
3
NAS-Bench-NN.P
105a
NL.P
NAS-Bench- 1Shotl
364k
Image class.
NAS-Bench-301
1
Image class.
NAS-Bench-ASR
8k
ASR
NAS-Bench-MR
1023
/
V
Var. CV
TransNAS-Bendl
7k
/
Var. CV
14
NAS-Bench-111
423k
/
Image class.
NAS-Bench-311
10t8
/
Image class.
WAS-Bench-NTPPI
10
NI.P
while for other search spaces, the macro structure can have variable length, width, and number of channels for different architectures in the search space.
A NAS benchmark (Lindauer & Hutter, 2020) consists of dataset (with fixed train-test split), a search space, and a fixed evaluation pipeline with predefined hyperparameters for training the architectures. A tabular NAS benchmark is that additionally provides precomputed evaluations with that training pipeline for all possible architectures in the search space. Finally, a surrogate NAS benchmark (Siems al., 2020 Yan et al. (02II) is NAS benchmark that provides a surrogate model that can be used to predict the performance of any architecture in the search space. We say that a NAS benchmark queryable if it either tabular or surrogate benchmark. Queryable NAS benchmarks can be used to simulate NAS experiments very cheaply by querying the performance of neural networks (using table or surrogate) instead of training the neural networks from scratch.
NAS benchmarks. Now we describe the characteristics of many popular NAS benchmarks. For summary, see Table [ and for more comprehensive and detailed survey, see Appendix B The first tabular NAS benchmark to be released was NAS-Bench-Di (Ying et al., (019). This benchmark consists of 423 624 architectures trained on CIFAR-10. The cell-based search space consists of directed acyclic graph (DAG) structure in which the nodes can take operations. A follow-up work, NAS- Bench- 1Shot (Zela et al., 2020b), defines three subsets of NAS-Bench- 101 which allow one-shot algorithms be run. The largest subset size in MAAS-Bench-1S5 1Shotl is 363 648. NAS MAS-Bench- 201 [Dngg & Yang (2020) another popular tabular NAS benchmark. The cell-based search space consists of DAG where each edge take operations (in contrast NAS-Bench- 101, which the nodes are operations). The number of non-isomorphic architectures is 6 466 and all are trained on NATS-Bence (Dong et al. 2021 is an extension of
CIFAR-10. CIFAR- 100, and ImageNet-16-122 NAS-Bench-200 which also varies the macro architecture.
NAS-Bench-NLP (Klyuchnikov et al., 2020) is a NAS benchmark for natural language process- ing, which is size 10 However, only 14322 of the architectures were trained on Penn Tree- Bank (Mikolov et al. 2010). meaning NÁS Bench-NLP. is not queryable. MAS--ench--AR. (Mehro
tra et al.) 202 1) is tabular NAS benchmark for automatic speech recognition. The search space consists of 8 242 architectures trained on the TIMIT dataset. Trans (Duan et al. 2021) tabular NAS benchmark consisting of two separate search spaces (cell-level and macro-level and seven tasks including pixel-level prediction, regression, and self-supervised tasks. The cell and macro search spaces size 4 096 and 3 256, respectively. NAS-Bench-Mo (Ding et al. (021) is a surrogate NAS benchmark which evaluates across four datasets: ImageNet50-1000 Cityscapes, KITTI, and HMDB51. MAS--Bench-ME consists of single search space of size 20
The DARTS [Luu et al. 2019b) search space with CIFAR-10, 10, consisting of 10 architectures, is arguably the most widely- used NAS benchmark. Recently, 60 000 of the architectures trained
3

---

Published as a conference paper at ICLR 2022
Validation Accuracy
100
75
50
25
+
NB-101
CIFAR10
NB. NBB-001
CIFAR10
NB: NB-201
CIFAR100
NB-201
ImageNet
DARTS
CIFAR10
NB-ASR
NB NLP
TreeBank
NB-MR
KITTI
NB-MR
HMDB51
NB-MR
ImageNet
NB-MR
City
TNB RicrM
Jigsaw
TNB-Micro
Semantic
TNB. Micro
Surface
INC-Micro
Autoenc.
TNB- Macro
Jigsaw
TNB-Macro
Semantic
TNB Mie-Macro
Surface
TND TND-Macre Macro
Autoenc.
Figure Validation accuracy box plots for each NAS benchmark. The whiskers represent the minimum and maximum accuracies in each search space. For NAS-Bench-NLP and TransNAS- Bench, perplexity and SSIM are used instead of validation accuracy, respectively. the case of extremely large search spaces such as DARTS and of precomputed architectures.
the statistics are computed only
with respect to the
and used to create NASBench-301 (Siems al. 2020), the first surrogate NAS benchmark. More recently, NAS-Bench-I 1 I1 NAS-Bench-311, and NAS-Bench-NLPII (Yan et al. [021] were released as surrogate benchmarks that extend existing benchmarks with the full learning curve information.
3 NAS BENCHMARK STATISTICS
In this section and in Appendix C we use NAS-Bench Suite to compute a set of aggregate statistics across large set of NAS benchmarks. There is high variance with respect to the distribution of accuracies and other statistics across benchmarks due to substantial differences in the tasks performed and layout of the search space. It is essential keep this in mind ensure fair comparison of the performance of NAS algorithms across these benchmarks. To the best of our knowledge, this the first large-scale aggregation of statistics computed on NAS benchmarks.
Figure shows box plots for the validation accuracy dis- tribution for representative set of the 25 NAS bench- marks. We find that TransNAS-Bende (Sem. Segment) and DARTS achieve the highest median and maximum ac- curacies, yet they also have among the smallest variance in validation accuracy across the search space. On the other hand, the search space with the highest interquartile range TransNAS-Beni Bench Jigsaw. level of locality in each search space, or the similarity of validation accuracy among neighboring architectures (ar- chitectures which differ by single operation edge) ing the random walk autocorrelation (RWA) (Weinberger] 1990, Ying et al.) [2019] White et 2021b). RWA com- putes the autocorrelation of accuracies of architectures during a random walk, in which each step perturbs one op- eration or edge. We see that HAS-Bench-200 ImageNet166 120 has the highest autocorrelation. while
RWA for NAS Benchmarks
Random walk autocorr. (RWA)
1.0
NB-101
CIFAR10
NB-201
CIFAR10
NB-201
0.6
ImageNet
NB-ASR
Figure 3 we assess the
TIMIT
TNB-Micro
Object
TNB-Micro
0.2
Jigsaw
0.0
Sqrt. of random walk distance
1011 has the lowest. In Appendix C we also discuss plots de- scribing the average runtime for training architectures and the average neighborhood size, for each NAS benchmark. Overall, we see substantial differences among the search spaces along the various axes that tested.
Figure 3: RWA for NAS benchmarks. RWA computes the autocorrelation of accuracies of architectures during a ran- dom walk, in which each step perturbs one operation or edge.
Overall, we find that the diversity is important to keep into context when comparing across many different NAS benchmarks. For example, it is more impressive if NAS algorithm discovers architecture within 0.1% of the optimal on NAS-Benche Bench-201 ImageNet16- 120, compared to DARTS, because the standard deviation of accuracies for DARTS is much lower. Additional factors affect the
4

---

Published as a conference paper at ICLR 2022
difficulty of NAS benchmarks for some NAS algorithms more than for others; for example, locality has a large effect on the performance of regularized evolution but not for random search.
4 ON THE GENERALIZABILITY OF NAS ALGORIHTMS
In this section, we carry out large-scale empirical study on the generalizability of NAS algorithms across diverse search spaces and tasks, using five different black-bos box algorithmms, five different performance predictors, and three one-shot methods across the largest set of NAS benchmarks to date. Throughout, we empirically assess three assumptions we have witnessed in the NAS community about the generalizability of NAS algorithms across diverse search spaces and tasks:
1. "If a NAS algorithm does well on the popular NAS benchmarks MAS-Bench-100 datasets of MAS-BBech--20, it surely must generalize other NAS benchmarks."
101 and all three
2. "NAS algorithms tend to have robust default hyperparameters and do not require tuning."
3. "To improve NAS algorithm on new benchmark, we can cheaply optimize its hyperparameters tabular benchmark and then transfer the optimized hyperparameters."
Experimental details. A black-box NAS algorithm is an algorithm which iteratively chooses architectures to train, and then uses the final validation accuracies in the next iteration. We run experiments for five popular black-box NAS algorithms: random search (RS) (Li & Talwalkar 2019 Sciuto et al. 2020), regularized evolution (RE) (Real et al., 2019), local search (LS) (White et al. 2021b; Onderander et al. 2021), BANANAS (White et al. a021aa. and NPENAS (Wei et al. 2020). We run each black-box algorithm for 200 iterations.
Recently, model-based performance prediction methods have gained popularity subroutines to speed up NAS algorithms (Ning et al. 2020). These methods work by training a model using a set of already evaluated architectures, and then using the model to predict the performance of untrained architectures. We compare five popular performance predictors: BOHAMIANN (Springenberg
al. 2016), Gaussian process (GP) (Rasmussen) 2003), random forest (RF) (Breiman [2001), p. neural architecture optimization (NAO) (Luo et al. 2018). and XGBoost (Chen & Guestrin 2016). We evaluate the performance prediction methods by computing the Spearman rank correlation of the predictions versus ground truth validation accuracy on a held-out test set of 200 architectures. For each black-box method and each performance predictor, we evaluate the default hyperparameter configuration, as well as 300 randomly sampled hyperparameter configurations, with each reported performance averaged over 10 seeds. We give additional details of each method in Appendix C
4.1 THE BEST NAS METHODS
In Figure we plot the scaled (relative) performance of all five black-box algorithms and performance predictors across representative set of NAS benchmarks (with the full plot in Appendix C In Table we give summary by computing the average rank of each black-box algorithm or performance prediction method across all 25 NAS benchmarks.
Across black-box algorithms with their default hyperparameters, we find that no algorithm performs well across all search spaces: no algorithm achieves an average rank close to I across all search spaces. RE and LS perform the best on average across the search spaces, with average rankings of 2.36 and 2.66, respectively. We also find that although RE performed the best average, it performs worse than random search in three cases. Therefore, there is no "best" black- algorithm. When comparing black-box algorithms tuned on each individual benchmark, RE achieves ranking of 1.96, although we note that since box algorithms are expensive to evaluate, it is computationally prohibitive tune for each individual NAS benchmark.
Across performance predictors, we find that the best predictor with default parameters is RF, and the best predictor when tuned on each individual benchmark is XGBoost, with average rankings of 1.57 and 1.23, respectively. Note that since performance prediction subroutines are often not the bottleneck of NAS, is common to run hyperparameter tuning during the NAS search. Therefore. we conclude that XGBoost (when tuned) does generalize well across all 25 search spaces we tested.
Generalizing beyond NAS-Bench-101 and -201. Now we test how well NAS methods generalize from
and -201 to the rest of the NAS benchmarks. In Table 2 for both black-box
5

---

Published as a conference paper at ICLR 2022
Search
Scaled Accuracy of NAS Algorithms
Search-4
Reg. Evo.
Scaled Accurac)
Reg.
ex
Local Search
8
6x
Local Search+11P
x
OX
X
BANANAS
BANANAS+IPO
NPENAS
NPIO
Scaled Spearman Rank Correlation of Performance Predictors
Scalled Spearman Rank
1.00
BOHAMIANN
BOHAMIANN + HPO
0.75
GP
0.50
RF
0.25
0.00
NB-101
CIFAR1O
NB 201-201
CIFAR10
NB-201
CIFAR100
NB-201
ImageNet
DARTS
CIFAR10
NB-A5R
TIMIT
NB NLL
Trees Bank
NB-MR
KITTI
NB-MR
HMD851 l
NB-MR
ImageNet
NB-MR
TNB Micro City
Scene
TNB IN--Micro Semantic
TNB Macro
Scene
INB-Macro Semantic
XGBoost
XGBoost+HPO
NAG+HPO
Figure 4: Relative performance of black-box algorithms (top) and performance predictors (bottom) across NAS benchmarks. The solid circles show the performance of the algorithm with default hyperparameters, while the crosses show performance after hyperparameter optimization (HPO).
and predictor methods, we compare the average rank each method across two different subsets of benchmarks: NAS-Bench- 101 and the three different datasets of of the 21 settings excluding
versus the rest We find that for both black-bor
101 and method and performance predictor methods, the best method substantially changes between these two subsets. For example, NAO is the top-performing predictor across NAS-Bench-10P and NAS- Bench-201, yet it achieves very poor performance the rest of the benchmarks. This suggests that the insights derived from empirical results are highly dependent on the benchmarks used, and that in order make reliable claims, evaluating on more than a few benchmarks crucial.
Table 2: Average relative performance ranking among five NAS algorithms (left) or five performance predictors (right) across 25 settings. Results weighted by search space; e.g., each of the three AAS-Bench-201 benchmarks are weighted by 1/3, For abbreviations, see Table 3
NAS Algorithms
Performance Predictors
RS
RE
BANANAS
LS
NPENAS
GP
BOHAM.
RF
XGB
NAO
Avg. Rank Avg. Rank, HPO
3.47
2.36
3.02
2.66
3.48
4.25
3,00
1.57
2.95
3.25
3.97
1.96
3.17
2.41
3.49
4.37
3.36
2.41
1.23
3.62
Avg. Rank, 101&201
4.50
3.00
3.50
1.50
2.50
4.67
2.83
2.17
4.17
1.17
Avg. Rank, non- 101&
3.06
2.11
2.83
3.13
3.87
4.08
3.06
1.33
2.46
4.08
4.2 GENERALIZABILITE OF HYPERPARAMETERS
While the previous section assessed the generalizability of NAS methods, now we assess the general- izability of the hyperparameters within NAS methods. For given NAS method, we can tune on NAS benchmark A, and then evaluate the performance of the tuned method on NAS benchmark B, compared to the performance the best hyperparameters from NAS benchmark B. other words, compute the "regret" of tuning method
NAS benchmark and deploying on another.
In Figure 5 (left), we run this experiment for all pairs of search spaces, averaged over all perfor- mance predictors, to give a general estimate of the regret across all search spaces. Unsurprisingly, hyperparameters transfer well within a given search space (such as within the three datasets in NAS-Bench. or the seven datasets in TransNAA-Benchh Mench-Micro). However, find that search space achieves strong regret across most search spaces. NAS-Bench- 101, DARTS, and the four benchmarks in
have particularly bad regret compared the other benchmarks.

---

Published as a conference paper at ICLR 2022
Generalizability for Predictors
Generalizability for Black-Box Algorithms
1.0
NB-101
NB-201
DARTS NB-NLP
NB. MR
TNB-Micro
TNB-Macro
N8-1002
NO 201
BARTS
TNB-Micro
TNE-Macro
NB-101
NB 201
DARTS NB ASR NG NLP
NG MR
TNB-Micro
THE Macro
Figure 5: Transferability results for predictors (left) and black-box algorithms (right). Row i, column j denotes the scaled regret of algorithm tuned search space and evaluated search space For abbreviations, see Table 3 and for summary statistics, see Appendix D.2
Next, in Figure 5 (right), we run the same experiment for black-box algorithms. We find that the transferabilityy of hyperparameters across black-box algorithms is worse than across predictors: the hyperparameters do not always transfer well even within different tasks of fixed search space. We also that, interestingly, the matrix is less symmetric than for performance predictors. For example, is particularly hard for hyperparameters to transfer to NAS-Bench-MR, but easier to transfer from MAS-Bench-MR. Overall, on one NAS benchmark and deploy on other benchmarks, as this can often make the performance worse. In Appendix D we give additional experiments and summary statistics on the transferability of hyperparameters across search spaces, well as a guide to interpreting the experiments. We also present additional experiments that combine our algorithm and statistics experiments to give relationships between properties of the search space and the performance of different algorithms.
experiments show that it is not sufficient tune hyperparameters
Scaled Accuracy
1.00
0.75
DARTS
0.25
DrNAS
0.00
NB-101
CIFARIO
NB-201
CIFAR10
NB. NB-201
CIFAR100
NB-201
ImageNet
DARTS
CIFAR10
TNB Object
TNB Scene
TNB-Micro liasow
Autoenc.
TNB
Figure 6: Performance of one-shot algorithms across NAS benchmarks. The bars show the minimum,
maximum and average performance over five seeds. For abbreviations, see Table 3
4.3 ONE-SHOT ALGORITHMS
One-shot NAS algorithms, in which single supernetwork representing the entire search space is trained, are a popular choice for NAS due to their strong performance and fast runtimes. In this section, compare the performance of three one-shot algorithms: DARTS (Liu et al. 2019b), GDAS (Dong Yang 2019), and DrNAS (Chen et al., 2021). across several different NAS benchmarks. Note that since ohe-shot algorithms must be able to represent the entire search space in the form of supernetwork, the algorithms can effectively only be run on cell-based search spaces with a complete graph (Zela al. 2020b), precluding the use of all 25 NAS benchmarks in the previous section.
7

---

Published as a conference paper at ICLR 2022
In Figure 6 we plot the scaled (relative) performance of the three one-shot methods run for five seeds each, across nine NAS benchmarks. There is no clear best algorithm: DrNAS performed best on five benchmarks, and GDAS performed best the remaining four. DARTS did not perform well, which is consistent with prior work (Zela et al.
2020a Dong & Yang 2020).
Throughout this section, showed that many implicit assumptions in the NAS community regarding NAS algorithm generalizability are incorrect, and that it is important to consider a large set of NAS benchmarks in order avoid false conclusions. In the next section, in order to help NAS researchers and practitioners avoid these pitfalls, we describe our new NAS benchmark suite, which is designed with the aim help the community develop generalizable NAS methods.
5 NAS P BSS-BENCH-SUITE OVERVIEW
In this section, we give overview of the NAS NAS-Bench-Suice codebase, which allows researchers to easily compare NAS algorithms across many tasks, as shown in Section
5.1 KEY CHALLENGES IN CREATING A FLEXIBLE BENCIMARK SUITE
Before the introduction of tabular NAS benchmarks, it was common practice to release new NAS methods along with custom-designed search spaces to go with the methods (Lindauer & Hutter 2020, & Talwalkar 2019), leading to many early NAS methods being intertwined with and hard to separate from their original search space. The result was that many algorithms needed substantial code changes in order to run on other search spaces.
Even after the release of several queryable benchmarks, is still not common practice
bb..eeron searcn_apaces apacea Import NasBench1 Searen Space from naallb. oaalib.optiin imizers Impozt engularizedEvoll loa naslib.defaut from maslib.utile import wtils, get_dataseet_ap.
NAS algo- rithms on more than a few benchmarks due to the nontrivial differences among each benchmark. For example, as de- scribed in Section 2 operations on the nodes (Yin et al., 2019) versus on the edges (Liu et al.J 2019b) added complexity in adapting one-shot op- timizers to many search spaces, and for some search spaces one-shot opti- mizers could only be run on subsets of the full space (Zela et al. (2200b). Other differences, such as the presence al.
import Irainer
config utile.get_.onfii_ffee wwoniiieeooiniiooooo hyp.eiig_ttppeoooeee nes!)
seaceh_apace NasBencai Bew.echeppcececeeeeeecccc.e ep:imizes - RegulerizedEvoldtee (config
docaset_apt get_datosat__pg apt (config. search_
11
config.dateset)
12
optiniser.aden cedddap__earree wewsseerreaaaeeeeee space (search sarcchappcceppp
trainer Trainer (optimizee) config) trainer. search
16
trainer. leaa luate
Snippet A minimal example on how one can run NAS
of hidden nodes (Klyuchnikov 2020) or skip connections (Mehrotra al., 2021), cause NAS components to require different implementations. Creating a robust NAS benchmark suite is not as simple as "combining the individual codebases". because such a solution would require re-implementing each new NAS algorithm on several search spaces. In fact, the difficulty in creating a benchmark suite is likely key reason why there are so few papers that evaluate on more than few benchmarks.
algorithm in NAS-Bench-Suite Both the search space and the algorithm can be changed in one line of code.
5.2 THE NAS BENCH SUI TE CODEBASE
To overcome these difficulties, NAS-Bench- Sui te enacts two key principles: flexibility in defining the search space and modularity of individual NAS components. We first describe how we achieve flexible search space definitions, and then we detail the modular design of Be.s-Bench-SS
te.
A search space is defined with graph object using PyTorch and NetworkX (Hagberg et al.) 2008), package that allows for easy-to-use and flexible graph creation and manipulations. This functionality allows for dynamic search space definition, where candidate operations can be encoded both as part of a node (Ying et al., 2019: Klyuchnikov et al., 2020) as well as a edge (Dong & Yang 2020 Liu et al.) 2019b). It also allows the representation of multiple layers of graphs top of the computational graph, allowing the formation nested graph-structures that can be used to define
hierarchical spaces (Ru et al. 2020 Liu al. 2019a).
8

---

Published as a conference paper at ICLR 2022
S-S--Bnch-Suitee is modular in the sense that individual NAS components, such as the search space or NAS algorithm, are disentangled and defined separately from one another. In Snippet [ we showcase minimal example that runs a black-box NAS algorithm on tabular benchmark. Other optimizers and benchmarks can be imported and run similarly. Due to these design principles, NAS-Bench-Suf
allows researchers implement their NAS algorithm in isolation, and then evaluate on all the benchmarks integrated in NAS -enccB-ecc---- Suite e without writing any additional code. Since variety of NAS algorithms, search spaces, and performance predictors have already been integrated into the open-source framework, this allows for the user to build on top of predefined NAS components. With the entire pipeline in place, along with the possibility of quick evaluations across search spaces and tasks, we believe that NAS MAs-Bench-SSite will allow researchers to rapidly prototype and fairly evaluate NAS methods. All the scripts to run the evaluations conducted in this paper come together with the library codebase, For more details, see Appendix E
RELATED WORK
We describe work that provides experimental surveys, benchmark suites, or unified codebases within NAS. For detailed surveys on NAS, see (Elsken et al. 2019 Xie et al. 2020
NAS experimental surveys. Multiple papers have found that random search is competitive NAS baseline (Li & Talwalkar 2019 Sciuto et al. 2020), including a recent work that benchmarked eight
NAS methods with five datasets on the DARTS search space Yaang et al. 2020). Other recent works
have shown experimental surveys for NAS performance predictors (Ning et al. 2020 White et al.,
2021c), and experimental analyses on weight sharing (Yu et al.. 2020).
Benchmark suites. NAS-Bench-360 (Tu et al., 2021) is a very recent benchmark suite which presents NAS benchmarks for ten diverse datasets on three search spaces. However, drawback is that evaluating NAS algorithms requires to 100+ GPU-hours (Tu et al.. 202Tp. This is in contrast to where NAS algorithms take at most 5 minutes
CPU due to the use of queryable benchmarks. Outside of NAS, multiple hyperparameter tuning benchmark suites have been released (Eggensperger al. 2021 Arango et al. 2021).
the
NAS codebases. The DeepArchitect library (Negrinho & Gordon 2017) was the first to have a modular design for search spaces and NAS algorithms. PyGlove (Peng et al. 2021 is library for NAS featuring dynamically adapting components, however, it is not open-sourced. Neural Network Intelligence (NNI) (Microsofi. (017) is a platform for AutoML. that implements many algorithms as well as BASSBBenc-- 101 and NAS-Bench-201. Other NAS repositories are actively being built, such as archai (Shah & Dey 2020) and aw_nas (Ning al. 2020)
7 CONCLUSION AND FUTURE WORK
In large-scale study across 25 NAS benchmark settings, blackbox NAS methods, 5 NAS predictors, and one-shot methods, we showed that many implicit assumptions in the NAS community are wrong. Firstly, there is no single best NAS method: which method performs best very much depends on the benchmark. Along similar lines, if a NAS method performs well on the popular NAS benchmarks NAS-Bench- 101 and all three datasets of NAS-Bench-201 in contrast what one might have expected, this does not imply that it will also perform well on other NAS benchmarks. Finally, tuning NAS algorithm's 's hyperparameters can make it dramatically better, but transferring such hyperparameters across benchmarks often fails. This analysis strongly suggests adapting the empirical standard of the field, to stop focusing too much on smaller NAS benchmarks like AAS--Bnnch- 101 and MAS-Benchh-211, and rather also embrace larger and novel NAS benchmarks for non object classification tasks (Klyuchnikov et al., 2020. Mehrotra et al. 2021 Duan et al. 202T). While substantial differences across NAS search spaces has far made it very hard to many NAS benchmarks, we showed a way out of this dilemma by introducing an easy-to-use, unified benchmark suite that we hope will facilitate reproducible, generalizable, and rapid NAS research.
For future work, NAS-Bench-Suiee can benefit from additional options, such as distributed training. Furthermore, although practitioners using MAS-Bench-Suite have the option to choose their own hand winked subset of the 25 tasks based on their specific application, it would be useful to define representative subsets of the benchmarks in RAS-Bench-Suit based application type.

---

Published as a conference paper at ICLR 2022
8 ETHICS STATEMENT
Our work gives a large scale evaluation of generalizability in NAS and then proposes new benchmark suite for NAS. The goal of our work to make it faster and more accessible for researchers to run generalizable and reproducible NAS experiments. Specifically, the use of tabular and surrogate NAS benchmarks allow researchers to simulate NAS experiments cheaply on CPU, rather than requiring GPU cluster, reducing the carbon footprint of NAS research (Patterson et al. [202]] Hao 2019). This is especially important since the development stage of NAS research may be extremely computationally intensive without the use of NAS benchmarks (Zoph & Le 2017: Real et al. 2019). Our work is a tool for the NAS community, which facilitates NAS research that may be used for positive impacts on society (for example, algorithms that reduce CO2 emissions (Rolnick et al. 2019) or negative impacts on society (for example, models that discriminate or exclude groups of people).
Due to the increase in conversations about ethics and societal impacts in the Al community (Hecht et al. 2018), we are hopeful that the applications of our work will have a net positive impact on society.
ACKNOWLEDGMENT AND DISCLOSURE FUNDING
FH and his group acknowledge support by the German Federal Ministry of Education and Research (BMBF, grant RenormalizedFlow 011S 9077CC and grant DeToL), the Robert Bosch GmbH, the European Research Council (ERC) under the European Union Horizon 2020 research and innovation programme through grant no. 716721, and by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215. This research was funded by the Deutsche Torschungsgemeinschaaf (DFG, German Research Foundation) under grant number 417962828. We thank Danny Stoll and Falak Vora for their helpful contributions to this project.
REFERENCES
Sebastian Pineda Arango, Hadi S Jomaa, Martin Wistuba, and Josif Grabocka. Hpo-b: A large-scale reproducible benchmark for black-box hpo based on openml. arXiv preprint 10X.v:2006.06257, 2021.
Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. In ICLR Workshop, 2018
Eli Bingham, Jonathan P Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D Goodman. Pyro: Deep universal probabilistic programming. The Journal of Machine Learning Research, 20(1):933-778, 2019.
Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Xiangning Chen, Ruochen Wang, Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsieh. Drnas: Dirichlet neural architecture search. In ICLR, 2021.
Mingyu Ding, Yuqi Huo. Haoyu Lu, Linjie Yang, Zhe Wang, Zhiwu Lu, Jingdong Wang, and Ping Luo. Learning versatile neural architectures by propagating network codes. arXiv preprint arXiv:2103. 13253,, 2021.
Xuanyi Dong and Yi Yang. Searching for a robust neural architecture in four gpu hours. CVPR, 2019.
Xuanyi Dong and Yi Yang. Nas-bench- 2011: Extending the scope of reproducible neural architecture search. In ICLR, 2020.
Xuanyi Dong, Lu Liu, Katarzyna Musial, and Bogdan Gabrys. Nats-bench: Benchmarking nas algorithms for architecture topology and size. In PAMI, 2021.
10

---

Published as a conference paper at ICLR 2022
Yawen Duan, Xin Chen, Hang Xu, Zewei Chen, Xiaodan Liang, Tong Zhang, and Zhenguo Li. Transnas- Transnas-bench-100 we--bennhhhh1 Improving transferability and generalizability cross-task neural archi- tecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5251-5260, 2021.
Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik, Matthias Feurer, René Sass, Aaron Klein, Noor Awad, Marius Lindauer, and Frank Hutter. Hpobench: A collection of reproducible multi-fidelity benchmark problems for hpo. arXiv preprint arXiv:2109.067/6 2021.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey, In JMLR, 2019.
Jiemin Fang, Yuzhu Sun, Qian Zhang, Yuan Li, Wenyu Liu, and Xinggang Wang. Densely connected search space for more flexible neural architecture search. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 80628-10637, 2020.
Peter Frazier, A tutorial on bayesian optimization. arXiv preprint arXiv: 1807.0281, 2018.
Aric Hagberg, Pieter Swart, and Daniel S Chult. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National (LANLL,,)) Los Alamos, NM (United States), 2008.
Karen Hao. Training a single ai model can emit as much carbon as five cars in their lifetimes. MIT Technology Review, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Brent Hecht, Lauren Wilcox, Jeffrey P Bigham, Johannes Schöning, Ehsan Hoque, Jason Ernst, Yonatan Bisk, Luigi De Russis, Lana Yarosh, Bushra Anjum, Danish Contractor, and Cathy Wu. It's time do something: Mitigating the negative impacts of computing through change the peer review process. ACM Future of Computing Blog, 2018.
Aaron Klein, Stefan Falkner, Jost Tobias Springenberg, and Frank Hutter. Learning curve prediction with bayesian neural networks. In ICLR, 2017.
Nikita Klyuchnikov, Ilya Trofimov, Ekaterina Artemova, Mikhail Salnikov, Maxim Fedorov, and Evgeny Burnaev. Was-bench-nlpp neural architecture search benchmark for natural language processing. arXiv preprint arXiv:2006.07116 2020.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In UAI, 2019.
Marius Lindauer and Frank Hutter, Best practices for scientific research on neural architecture search. In JMLR, 2020.
Chenxi Liu, Liang-Chieh Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 82-92, 2019a.
Chich Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei-
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In ICLR, 2019b.
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture optimization. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2018.
Renqian Luo, Xu Tan, Rui Wang, Tao Qin, Enhong Chen, and Tie- Liu. Semi-supervised neural architecture search. In NeurIPS, 2020.
Abhinav Mehrotra, Alberto Gil C. P. Ramos, Sourav Bhattacharya, Lukasz Dudziak, Ravichander Vipperla, Thomas Chau, Mohamed Abdelfattah, Samin Ishtiaq, and Nicholas Donald Lane. Nas-bench-asr: Reproducible neural architecture search for speech recognition. In ICLR, 2021.

---

Published as a conference paper at ICLR 2022
Microsoft. Neural network intelligence. CitHub repository microsoft/mmi, 2017.
Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. Recur- rent neural network based language model. Annual conference of the international speech communication association, 2010.
Renato Negrinho and Geoff Gordon. Deeparchitect: Automatically designing and training deep architectures. arXiv preprint urXiv:/704.0879) 2017.
Xuefei Ning, Wenshuo Li, Zixuan Zhou, Tianchen Zhao, Yin Zheng, Shuang Liang, Huazhong Yang, and Yu Wang. A surgery of the neural architecture evaluators. arXiv preprint arXiv:2008. 03064, 2020.
Den Ottelander, Arkadiy Dushatskiy, Marco Virgolin, and Peter AN Bosman. Local search is remarkably strong baseline for neural architecture search. In International Conference Evolutionary Mulli-Criterion Optimization, 2021.
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104, 1000:0000000300, 2021.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2830, 2011.
Daiyi Peng, Xuanyi Dong, Esteban Real, Mingxing Tan, Yifeng Lu, Hanxiao Liu, Gabriel Bender, Adam Kraft, Chen Liang, and Quoc V Le. Pyglove: Symbolic programming for automated machine learning. arXiv preprint arXiv:2 (0t::2001.0000 008009, 2021.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer School Machine Learning, pp. 63-71. Springer, 2003.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In AAAI, 2019.
David Rolnick, Priya Donti, Lynn H Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Milojevic-Dupont Natasha Jaques, Anna Galdman-Brownn al. Tackling climate change with machine learning. arXiv preprini arXiv:1906.05433, 2019.
Binxin Ru, Pedro Esperanca, and Fabio Carlueci. Neural architecture generator optimization. arXiv preprint arXiv:2004.01339) 2020.
Christian Sciuto, Kaicheng Yu, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. ICLR, 2020.
Shital Shah and Debadcepta Dey. Archai. GitHub repository microsoft/archai, 2020.
Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. Nas- bench-301 and the case for surrogate benchmarks for neural architecture search. arXiv preprint 0977772000..07777 2020.
Jasper Snock, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2012.
David R. So, Chen Liang, and Quoe V. Le. The evolved transformer. In ICML, 2019.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. In NeurIPS, pp. 4134-4142, 2016.
Xiu Su, Tao Huang, Yanxi Li, Shan You, Fei Wang, Chen Qian, Changshui Zhang, and Chang Xu. Prioritized architecture sampling with monto-carke tree search. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19968-1097,, 2021.
12

---

Published as a conference paper at ICLR 2022
Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes, In Artificial Intelligence and Statistics, pp. 567-574, 2009.
Renbo Tu, Mikhail Khodak, Nicholas Carl Roberts, Nina Balcan, and Ameet Talwalkar. Nas- bench-360: Benchmarking diverse tasks for neural architecture search. Openreview submission, 2021.
Chen Wei, Chuang Niu, Yiping Tang, and Jimin Liang. Npenas: Neural predictor guided evolution for neural architecture search. arXiv preprint arXiv:2003. /2857, 2020.
Edward Weinberger. Correlated and uncorrelated fitness landscapes and how to tell the difference. Biological cyberneties, 63(5)332--336, 1990.
Colin White, Willie Neiswanger, Sam Nolen, and Yash Savani. A study encodings for neural architecture search. In NeurlPS, 2020.
Colin White, Willie Neiswanger, and Yash Savani. Bananas: Bayesian optimization with neural architectures for neural architecture search. In AAAI, 2021a.
Colin White, Sam Nolen, and Yash Savani. Exploring the loss landscape in neural architecture search. In UAI, 2021b.
Colin White, Arber Zela, Binxin Ru, Yang Liu, and Frank Hutter. How powerful are performance predictors in neural architecture search? In NeurIPS, 2021c.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10734-10742, 2019.
Lingxi Xie, Xin Chen, Kaifeng Bi, Longhui Wei, Yuhui Xu, Zhengsu Chen, Lanfei Wang, An Xiao, Jianlong Chang, Xiaopeng Zhang, et al. Weight-sharing neural architecture search: A battle to shrink the optimization gap. arXiv preprint arXiv:2008.01475. 2020.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? ICLR, 2019.
Shen Yan, Colin White, Yash Savani, and Frank Hutter. Nas-bench-x!1 and the power of learning curves. In NeurIPS, 2021.
Antoine Yang, Pedro M Esperança, and Fabio M Carlueci. Nas evaluation is frustratingly hard. In ICLR, 2020.
Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, and Frank Hutter, Nas- bench-101: Towards reproducible neural architecture search. In JCML, 2019.
Kaicheng Yu, Rene Ranftl, and Mathieu Salzmann. How to train your super-net: An analysis of training heuristics in weight-sharing nas. arXiv preprint trXiv:2003.042) 2020.
Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hutter. Understanding and robustifying differentiable architecture search. In ICLR, 2020a.
Arber Zela, Julien Siems, and Frank Hutter. Vas-bench-Ishotle Benchmarking and dissecting one-shot neural architecture search. In ICLR, 2020b.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018.
13

---

Published as a conference paper at ICLR 2022
Table 3: List of abbreviations used in the text.
Abbreviation
Full
RS
Random Search
RE
Regularized Evolution
LS
Local Search
NPENAS
Neural Predictor Guided Evolution for Neural Architecture Search
BANANAS
Bayesian Optimization with Neural Architectures for Neural Architecture Search
GP
Gaussian Process
NAO
Neural Architecture Optimization
RF
Random Forest
XGB
XGBoost (Extreme Gradient Boosting)
Bayesian Optimization with Hamiltonian Monte Carlo Artificial Neural Networks
BOHAMIANN
NB-101
NAS-Bench- 101
NB-201
NAS-Bench-201
NB-301
NAS-Bench-3B0
NB-ASR
NAS-Bench-ASR (Automated Speech Recognition)
NB-NLP
NAS-Bench-NLP (Natural Language Processing)
NB-MR
NAS-Bench-MR (Multi-Resolution)
TNB-Micro
TransNAS-Benchh Micro search space
TNB-Macro
TransNAS-Bench. Macro search space
DARTS
Differentiable Architecture Search
GDAS
Gradient-basdd search using Differentiable Architecture Sampler
DrNS
Dirichlet Neural Architecture Search
A BEST PRACTICES FOR NAS RESEARCH
There have been few recent works which have called for improving the reproducibility and fairness
in experimental comparisons in NAS research (Li Talwalkar 2019 Ying et al.) 2019: Yang et al.)
2020). This led to the release of NAS best practices checklist (Lindauer & Hutter 2020). We address each part of the checklist.
Best practices for releasing code. For each NAS benchmark that used, the code for the training
pipeline and the search space is already publicly available. Since we used NAS benchmarks for all of our experiments, we did evaluate the architectures ourselves. All of the code for the NAS methods, including the hyperparameters, are available at putttps://qithht com/ automm/nassll We discuss the choices of hyperparameters in Appendix C
Best practices for comparing NAS methods. Since we made use of NAS benchmarks, all of the details for training the architectures are fixed across NAS methods. We included baselines such as random search and local search in our experiments in Section 4 We averaged 20 trials and 100 or more hyperparameter configurations for each experiment, and the choice of seeds (0-19) and hyperparameter configurations is available at htttttt:///gitttaa com/ http//autommnaaaian.
Best practices for reporting important details. We reported how we tuned the hyperparameters of NAS methods in Section We included all details of our experimental setup in Section and
14

---

Published as a conference paper at ICLR 2022
B DETAILS FROM SECTION 2
This section is an extension of the "NAS benchmarks" part of Section 2 with additional details for each NAS benchmark, as well as more comprehensive list of NAS benchmarks. For an extension of Table which contains more comprehensive list, see Table 4
(Yigg NAS benchmarks. The first tabular NAS benchmark to be released was architectures with ct al. 2019). This benchmark consists a cell-based search space of 423 624 fixed macro structure. NAS-Bench-100 comes with precomputed validation and test accuracies at epochs 4, 12, 36, and 108 from training on CIFAR-10. The cell-based search space of NAS- ----Bench-- 101 consists of five nodes which can take on any DAG structure with at most seven edges. Each node can take on one of three operations. Since training with stochastic gradient descent is random, all architectures were trained three times with different seeds and therefore have three sets of accuracies.
Since NAS-Bench-Iue 101 architectures contain a variable amount of nodes, it is not possible to evaluate one-shot algorithms. Therefore, NAS-Bench-1Shoee (Zela et al. (020b) defines three subsets of NAS- Bench-101 which allow one-shot algorithms to be run. The largest subset size in NAS-Bench-ISS5 1 363 648.
NAS-Bench-201 is the second tabular NAS benchmark. It consists of a cell which is a complete directed acyclic graph over 4 nodes. Therefore, there are (3) 6 edges. Each edge can take one of five operations (note that this is in contrast NAS-Bench-1001 in which the nodes operations). The search space consists of 56 = 15, 625 neural architectures, although due to none and identity operations, the number of non-isomorphic architectures is 6 6666. Each architecture has precomputed train, validation, and test losses and accuracies for 200 epochs on CIFAR- 10, CIFAR- 100, and ImageNet- 1- As in NASBench-109 on each dataset, each architecture was trained three times using different random seeds.
NATS- (Dong et al. 2021) is an extension of NAS-Bench-201 which also varies the macro architecture. Specifically, a search space of 32 768 architectures with varying size were trained across three datasets for three seeds.
The first non computer vision NAS benchmark to be released was MMS-Bench-NLL. Klyuchnikov jet al., 2020). Its search space consists of DAG of up to 24 nodes, each of which can take on one of seven operations and can have at most three incoming edges. With size of at least 1033, NAS-Bench-NLPP is currently the largest NAS benchmark. 14 322 of the architectures were trained on Penn Tree Bank (Mikolov et al. 2010) for 50 epochs. Since only fraction of architectures were trained, NAS-Bench-NIPP is not queryable.
The DARTS (Liu et al. 2019b) search space with CIFAR-10 is arguably the most popular NAS benchmark. The search space contains 18 architectures, consisting of two cells, each of which has six nodes. Each node has exactly two incoming edges, and each edge can take one of eight operations. Recently, 000 of the architectures were trained for 100 epochs and used to create NASBench-301 (Siems et al. (2020), the first surrogate NAS benchmark. The authors released pretrained surrogates created using XGBoost Chen & Guestrin (016)) and graph isomorphism networks (Xu et al. 2019).
NAS-Bench-ASR (Mehrotra et al. 2021) is a tabular NAS benchmark for automatic speech recog- nition. The search space consists of 8 242 architectures trained on the TIMIT dataset. The search space consists of four nodes, with three main edges that can take on one of six operations, and six skip connection edges, which can be set to on or off.
NAS-Bench-11 11 NAS-Bench-311 and NAS-Bench-NN
Yan et al., 2021 were recently released as surrogate benchmarks that extend NAS-Bench- 101, NAS-Bench-300 301, and PAS-Bench-NI.P by predicting the full learning curve information. In particular, none of 301, and NAS-Bench-NLP allow the validation accuracies to be queried at arbitrary epochs, which is necessary for multi fideeiity NAS techniques such as learning curve extrapolation (Baker et al. 2018 Klein et al.) 2017). The surrogates used to create MAS-Bench- 111,
NAS, NAS-Bench-
and NAS BeS--Bench-LPP include singular value decomposition and noise modeling (Yan et al.) 2021).
TransNAS-Bent (Duan et al. (022)) is a tabular NAS benchmark consisting of two separate search spaces (cell-level and mecro-level) and seven tasks including pixel-level prediction, regression, and
15

---

Published as a conference paper at ICLR 2022
Table 4: A comprehensive overview of NAS benchmarks.
Queryable
Benchmark
Size
Tab.
Surr.
1.Cs
Macro
Type
#Tasks
NAS-Bench-Suite
NAS-Bench-101
423k
Image class.
NAS-Bench-201
6k
Image class.
NATS-Bench
6k
Image class.
NAS-Bench-NLP
1033
NLP
WAS-Bench-IShol
364k
Image class.
/
NAS-Bench-301
10ts
Image class.
NAS AAS-Bench-ASR
8k
ASR
TrnneNAS-Bench
7k
Var. CV
14
NAS- MAS-Bench-111
423k
Image class.
NAS-Bench-311
10as
Image class.
NAS-Bench-NLPII
10
NLP
NAS-Bench-MR
1028
Var. CV
NAS-Bench-360
Var.
Var.
30
MAS-Bench-Macro
6k
Image class.
HW-NAS-Bench (201)
6k
Image class.
HW-NAS-Bench (FBNet)
1021
Image class.
self-supervised tasks. The-cell level search space of TransNAS-Bench is similar that of NAS- Bench-201, but with choices operations per edge, hence 4996 architectures altogether. The macro-level search locations to down-sample feature maps, and locations to raise the channels, leading to a total of 3 256 architectures.
space instead has flexible macro skeleton with variable number of blocks,
MNS-Bench-MRR Ding et al. total, across four datasets: ImageNet50- 1000, Cityscapes, KITTI, and HMDB51. NAS-Bench-Ne consists of a single search space of size 1023 and for each of the nine settings, 2 500 architectures trained, create nine different surrogates for each of the nine settings.
2021 is surrogate NAS benchmark which evaluates nine settings
NAS-Bench-360 (Tu et al., (02I) is a very recent benchmark suite which gives NAS benchmarks for ten differrnt datasets, including tasks that are novel for NAS such as spherical projection, fluid dynamics, DNA sequencing. medical imaging, surface electromyography, and cosmic ray detection. The tasks are carried out on three different search spaces based on Wide ResNet (He al., 2016). DARTS (Liu et al. 2019b), and DenseNAS (Fang et al.) (020). However, drawback of 360 that none the NAS benchmarks are queryable. Therefore, evaluating NAS algorithms on these benchmarks requires I to 100+ GPU-hours of runtime (Tu et al. 2021).
NAS-Bench-Mand Maccr (Su et al. 2021) is NAS benchmark which focuses on macro search space. It consists of 6561 pretrained architectures CIFAR-10. The search space consists of 8 layers, each with choices of blocks.
HW-NAS-Bench is NAS benchmark focusing on hardware-aware neural architecture search. It gives the measured/estimaate hardware-cost for all architectures (Wu et al. 2019) on six hardware devices, including commercial edge. FPGA, and ASIC devices. HW-NAS Bennc-Beechhe can be used alongside NAS-Bench. for the full information on hardware cost and model accuracy for all architectures in NAS-Bench-201
PAS-Bench-201 and FBNet
16

---

Published as a conference paper at ICLR 2022
C DETAILS FROM SECTION
C.I NAS ALGORITHM IMPLEMENTATION DETAILS
Here, we give implementation details for all algorithms that we compared in Section effort to keep the implementations close possible to the original implementation. We start with the black-box optimizers. For list of the default hyperparameters and hyperparameter ranges, see https: / /ggttttthubbb com/ aut om //SASiibbbb
4 We made an
Random search. Random search is the simplest baseline for NAS & Talwalkar (2019);
Sciuto et al. (202)). It draws architectures at random and then returns the best architecture.
Local search. Another baseline, local search has been shown to perform well on multiple NAS benchmarks (White et al. 2021b; Ottelander al.. 2022 Siems et al., 2020). It works by evaluating all architectures in the neighborhood of the current best architecture found far. The neighborhood of an architecture the set of architectures which differ by one operation or edge. We used the implementation from White et al. (White et al. 2021b).
Regularized evolution. This algorithm (Real et al. 2019) consists of iteratively mutating the best architectures drawn from sample of the most recent architectures evaluated. A mutation is defined by randomly changing one operation or edge. We used the NAS-Bench- 101 (Ying et al., 2019) implementation.
BANANAS. This NAS algorithm (White al. 202 a uses Bayesian optimization with an ensemble of three predictors as the surrogate. We use the code from the original repository, but using PyTorch for the MLPs instead of Tensorflow. We use the adjacency matrix encoding instead of the path encoding, since the path encoding does not scale to large search spaces such as MAS-Bench-NN.P. We use variational sparse GPs (Titsias) 2009) in the ensemble of predictors, since this was shown in prior work to perform well and have low runtime (White et al., 2021c).
NPENAS. This algorithm (Wei et al., 2020) based on predictor-gus guided evolution. iteratively chooses the next architectures by mutating the most recent architectures in random sample of the population to create set of candidate architectures, and then using predictor to pick the architectures with the highest expected accuracy. Again, we use variational sparse GP (Tississ 2009) as the predictor.
Now we describe the performance predictors. For each method, we used the adjacency one-hot encoding (White et al. 2020).
BOHAMIANN. BOHAMIANN (Springenberg et al. 2016) is Bayesian inference predic- tion method which uses stochastic gradient Hamiltonian Monte Carlo (SGHMC) in order to sample from a Bayesian Neural Network. We use the original implementation from the pybnn package.
GP. Gaussian Process (GP) (Rasmussen 2003) is a popular surrogate often used with Bayesian optimization (Frazier 2018 Snoek et al. 2012). Every feature has a joint Gaussian distribution. We use the Pyro implementation (Bingham et al. 2019).
NAO. Neural Architecture Optimization is a NAS method that uses encoder-decoder (Luo al. 2018). The encoder is feedfoward neural network, and the decoder is an I.STM and attention mechanism. We used the implementation from SemiNAS (L.uo et al. 2020).
Random Forest. Random forests (Breiman 2001 are ensembles of decision trees. Random forests have been used as model-based predictors in NAS (Siems et al. 2020 White et al..
2021c). We use the Scikit-learn implementation (Pedregosa et al. 20111.
XGBoost. eXtreme Gradient Boosting (XGBoost) (Chen & Guestrin, 2016) is popular
gradient- decision tree which has been used in NAS (Siems et al. 2020 White et al.)
2021c). We used the original code (Chen & Guestrin 2016).
For one-shot methods, we mainly focus on the differentiable architecture search approaches, and select popular algorithms like DARTS (Liu et al. 2019b), GDAS (Dong & Yang 2019) and DrNAS (Chen et al. 2021) in our paper.
17

---

Published as a conference paper at ICLR 2022
Validation Accuracy
100
H
75
1
+
50
D+
H
H
25
+
NB-101 CIFARIO
NB-201 CIFARIO
NB-28 CIFAR100
NB. 201 ImageNet-
DARTS CIFAR10
NB-AS -3-ASR TIMIT
NB-NLP TreeBank
NB. NB-MR HMDB51
NB-MR ImageNet
NB-MR City
TNB-Micro Object
TNB- MicMi Scene
TNB. MicMicro Jigsaw
TNB Micro Room-
TNB-M LEe-Micro Semantic
TNB- Micro Surface
TNB Micro Autoenc.
TNB-Mace Object
TNB. Macro Scene
TNB-Macro JIgsaw
TNB- 1ee-Macro Room
TNB: Macre Semantic
TNB-Macro Surface
TNB TNB Mac-o Autoenc.
NB. MR
Figure 7: Validation accuracy box plots for each NAS benchmark. The whiskers represent the minimum and maximum accuracies in each search space. For NAS-Bench-NLP, perplexity used instead of validation accuracy, and three datasets of TransNAS-Benn Normal uses SSIM, Autoencoding uses SSIM, and Room Layout uses negative loss. These are in accordance with the metrics used in the original work. Finally, in the case of extremely large search spaces such as DARTS and NAS-Bench-NI.P the statisties are computed only with respect to tens-of-thousands of precomputed architectures.
do not use accuracy: Surface
DARTS. DARTS (Liu et al., 2019b) is the first work on differentiable architecture search. Compared to normal one-shot method that has one-hot encoding to select one architecture out of different choices, it uses vector, a.k.a. architecture parameters, where each element ranges from to as probability. During training, sums branches as weight summation. After the search, the final architecture is converted by selecting the branch that with highest weight.
GDAS. Since differentiable architecture search suffers from unstable training compared traditional one-shot methods, GDAS (Dong Yang 2019) bridges the gap. Instead of having weighted summation of all paths, it discretizes the paths during training to select the path with highest probability, while the rest of algorithms remains similar as original DARTS.
DrNAS. Dirichlet architecture search (DrNAS) (Chen et al. 2021) is another attempt to solve the instability issue of differentiable architecture search. This work treats the continuously relaxed architecture weights random variable, which is modeled by Dirichlet distribution. To this end, the Direchlet parameters can be updated by the traditional differentiable architecture search optimizer easily in the end-to-end manner.
D ADDITIONAL EXPERIMENTS
In this section, we give additional statistics, algorithm, and insight experiments augment Sections
and 4
D.1 ADDITIONAL STATISTICS EXPERIMENTS
We start by giving additional experiments on the statistics of NAS benchmarks, to supplement Section In Figure 7 we give the full box plots, extending Figure 2 We also add new statistics: in Figure plot the average time to train architecture for each NAS benchmark. In Figure 9 plot the average neighborhood size for each NAS benchmark. Note that for some NAS benchmarks, the neighborhood size is fixed, and for other NAS benchmarks, the neighborhood size varies.
18

---

Published as a conference paper at ICLR 2022
Avg. training time (sec)
40000
30000
20000
10000
NB-101 CIFAR10
MB-201 CIFAR10
NB- NB-201 CIFAR100
NB-201 ImageNet
DARTS CIFARIO-
NB-ASR TIMIT
NB IB-NLP TreeBank
TNB TNB-Micr RE--Micro Object
TNB-Micro Scene
TNB Mi-Miccro Jigsaw
TNB TNB- MiB-Micro Room
TNB- INB-Micro Semantic
TNB TNB- Pie-Micro Surface
INB-Micro Autoenc.
TNB-Macro Object
TNB-Macro Scene
TNB INB-Macro Jigsaw
TNB-Macrd Room
TNB-Macro Semantic
TNB TNB-Macre Macro Surface
TNB TNB-Mach Mac-Macro Autoenc.
Figure 8: Average time train an architecture for each NAS benchmark.
Neighborhood size
100
75
50
25
NB-101 CIFAR10
NB-201 CIFARIO
NB-22 NB-201 CIFAR100
NB- NB-201 ImageNet-
DARTS CIFAR10
NB-ASR TIMIT
NB- NB-NLP TreeBank
UB-MR KITTI
NB MR HMDB51
NB-MR ImageNet-
NB-MR City
TNB-Micro Object
INB-Micro Scene
TNB- Micro Jigsaw
INB-Micro Room-
TNB- INB-Micro Semantic
TNB- Mic-Micro Surface
TNB TNB-MicH Micro Autoenc.
TNB- INB-Macro Object-
TNB-Macro Scene
TNB-Macro Jigsaw
TNB-Macro Room
TNB INB-Macro Semantic
TNB-Macro Surface
TNB TNB-Macra Macro Autoenc.
Figure 9: Average neighborhood size for each NAS benchmark. Note that for some NAS benchmarks,
the neighborhood size is fixed, and for other NAS benchmarks, the neighhorhood size varies.
D.2 ADDITIONAL ALGORITHM EXPERIMENTS
Next, we give additional experiments from Section In Figure 10 we give the full performance predictor and box results, extending Figure
Recall that in Section 4 we assessed the transferability of hyperparameters by tuning algorithms on NAS benchmark A, and evaluating the performance of the tuned method NAS benchmark B, compared to the performance of the best hyperparameters from NAS benchmark B. The results were plotted in Figure In Tables and 6 to present the results in another format, we give the raw values from these experiments. All values are averaged over each search space. For example, the three rows for NAS-Bench-201 were averaged into one row, and similarly for the columns.
Furthermore, in Tables 7 and 8 we give summary statistics from this experiment. For each search hyperparameters on average or from other search space, we compute the transferability find that hyperparameters from TAS-Bench-MR transfer spaces. For performance predictors, the least well to or from other search spaces. NAS-Bende Bench-NLP transfers the best. For black- blx algorithms, MAS-Bench-MEF transfers the worst, and DARTS transfers the best. Therefore, it is safest for practitioners to tune their techniques on NAS-Benche Bench-NLP and DARTS before deploying them in new setting. It is not safe to tune techniques other benchmarks such NAS- VAS-Bench-MR.
19

---

Published as a conference paper at ICLR 2022
NAS Algorithms
Staled Accuracy
Rand, Search
Rand, Search HPO
Reg.
x
Reg. Evo +IIP
Local HPO
BANANAS
NB-101 CIFARIO
NB 2012002 CIFARIO
NB. 20-2011 CIFAR100
N8-201 ImageNet
DARTS CIFAR1O
NB- AB-ASR TIMIT
NB. AI--LPP TreeBank
NB- BR-M KITTI
NB BE-MMR HMD851
NO MR-MR ImaggeNe
NB. City
TNB-Micro Objject
TNB Mi--Mrrrro Scene
TNB Micro Jigsaw
TNB Miero Room
TNB-M Semantic
TNB-Micro Surface
TNS Autoenc.
TNB Macra Object
Mac-Macro Scone
TNB-Macro Jigsaw
TNB-M Mee-Macro Room-
TNB Macro Semantic
TNB: Macro Surfoce
TNB-Macro Autoenc.
NPENAS
HAO
NPENAS
Performance Predictors
Scaled Spearman Rank
1.00
BOHAMIANN BOHARINN-HHO
0.75
%
x
x
GP
0.50
Rx
RF
0.25
Ox
RF+IIO
x
XGBocst
0.00
NB 101 CIFAR10
N8- 20-201 CIFARIO
NB-201 CIFARI
NB 201 ImageNet
DARTS CIFRRIO
NB-ASR TIMIT
TreeBank
NB. MR KITTI
NB-MM HMDB51-
NB. MR ImageNet
NB- MR City
TNB-Micro Oboject
TNE Micro Scene
TNB Micro Jigsow
TNB-Micro Room-
TNB-Micro Semantic
TNB Micro Surface
TNB- Micro Autoonc.
TNO MacrMMcro Object-
TNB- Macro Scene
TNB Macro Jigsow
TNB-Macro Room
TNB. Macro Surfoce
TNB Mra-Macco Autoenc.
NGBeest+HIPO
TNB Mac-Macoo Semanti
NAO-HPO
Figure 10: Relative performance of black-box algorithms (top) and performance predictors (bottom)
across NAS benchmarks. The solid circle shows the performance of the algorithm with default hyperparameters, while the cross shows performance after hyperparameter optimization.
While all of the hyperparameter transfer experiments to this point have focused on the optimal hyperparameters, we now present two more experiments that focus on the hyperparameters on average. In Figure 11 for search spaces A and B, we compute the Kendall Tau rank correlation of the ranking lists for all hyperparameters on search spaces and
Finally, in Table 9 we present "leave one out"" experiments for all search spaces. For search space A, the best hyperparameter setting on average over all search spaces except is computed, and compared the performance of the best hyperparameters for A. This is similar to Table in that it provides a summary of the average transferability the hyperparameters for each search space. However, the leave out experiments focused more on the performance of hyperparameters average, while Table is focused on transferability of the optimal hyperparameters.
D.3 A GUIDE TO INTERPRETING HYPERPARAMETER TRANSFER EXPERIMENTS
Throughout Sections and D.2, we presented several different analyses and summaries on the extent to which hyperparameters transfer from one search space to another. In this section, we give guide for interpreting the results.
First, practitioners interested in the transfer of hyperparameters which are optimally trained on one search space, should focus on Figure and Tables [8) because these figures and tables express the regret of hyperparameters tuned one search space and evaluated on others. On the other hand, practitioners interested on how hyperparameters overall transfer from one search space to others (not just optimal), should focus Figure 11 and Table 9 because these represent the average transferability of all sets of hyperparameters that we tried.
Practitioners interested in the specific transfer from one search space (or setting) another, should focus on our matrix results, Figures and Practitioners interested in the general transferability on average from one search space others, should focus the summary tables, Tables and
20

---

Published as conference paper at ICLR 2022
Table Raw values from the performance predictor hyperparameter transferability experiment from Figure 5 (left). Each search space has 0-1 scaling done to fairly compare trends between search spaces. Results are weighted by search space. E.g., each of the three NAS-Bench-20 benchmarks are averaged into one row/column.
each
NB-101
NB-201
DARTS
NB-ASR
NB-NLP
NB-MR
TNB-101
NB-101
.00
.42
.28
.46
.48
.21
.41
NB-201
.43
.02
.29
.11
.11
.32
.09
DARTS
.08
.47
.00
.48
.34
.26
.41
NB-ASR
.28
.23
.31
.00
.07
.32
.12
NB-NLP
.25
.31
.27
.09
.00
.36
.15
NB-MR .19
.39
.28
.42
.47
.20
.40
TNB-101
.37
.13
.41
.16
.15
.40
.15
Table 6: Raw values from the black-box algorithm hyperparameter transferabilityy experiment from Figure (right). Each search space has 0-1 scaling done to fairly compare trends between search spaces. Results are weighted by search space. E.g., cach of the three TAS--eench-25 benchmarks are averaged into one row/column.
NB-101
NB-201
DARTS
NB-ASR
NB-NLP
NB-MR
TNB-101
NB-101
.00
.25
.25
.00
.25
.75
.27
NB-201
.25
.22
.33
.75
.50
.83
.28
DARTS
.33
.00
.75
.50
.50
.23
NB-ASR
.75
.50
.50
.00
.75
.25
.50
NB-NLP
.07
.28
.28
.73
.00
.64
.28
NB-MR
.86
.80
.70
.42
.84
.32
.76
TNB-101
.27
.28
.33
.73
.48
.70
.28
D.4 INSIGITT EXPERIMENTS
Now we present experiments that give new insights into NAS algorithms and search spaces.
First, we run experiments test the following hypothesis: larger search spaces have smaller interquar- tile ranges (IQR), because larger cell size gives most architectures chance to have performant operations. For example, for a search space of size three, some architectures will have two or three convolution operations, and some architectures will have two or three pooling operations, creating large IQR. But for search space size 10, the vast majority of architectures will have good mix of convolution and pooling operations, creating comparatively lower IQR. We this experiment on NAS-Benchh 101 and NAS-Bench-201. See Figure 12 We find that in all benchmarks, there is strict negative correlation between number of operations and IQR.
Finally, we compute correlations between the relative ranking of each NAS technique and properties of the search spaces, such as total size and neighhorhood size. See Figure 10 The largest correlations we find are as follows:
GP performs comparatively much better on search spaces with small neighborhood sizes.
When tuned, and XGBoost perform comparatively much better on large search spaces and also search spaces with large neighborhood sizes.
Surprisingly, BOHAMIANN performs comparatively better for large neighborhood sizes when not tuned, and comparatively better for small neighborhood sizes when tuned.
Default regularized evolution performs comparatively much better on search spaces with small neighborhood sizes.
21

---

Published as a conference paper at ICLR 2022
Table Summaries from the performance predictor transferability experiment from Figure 5 (left). Each value in "transfer to" is the average of the corresponding row from Figure 5 Each value in "transfer from" is the average of the corresponding column. Therefore, for each search space, we have a measure of the extent to which hyperparametters can transfer to from other search spaces, on average.
NB-101
NB-201
DARTS
NB-ASR
NB-NLP
NB-MR
TNB-101
Transfer to Transfer from
0.376
0.229
0.340
0.224
0.239
0.393
0.293
0.268
0.328
0.307
0.288
0.270
0.345
0.287
Table 8: Summaries from the black-box algorithm transfcrability experiment from Figure 5 (right). Each value in "transfer to" is the average of the corresponding from Figure Each value in "transfer from" is the average of the corresponding column. Therefore, for each search space, we have measure of the extent which hyperparameters transfer to from other search spaces, on average.
NB-101
NB-201
DARTS
NB-ASR
NB-NLP
NB-MR
TNB-101
Transfer to Transfer from
0.461
0.528
0.428
0.542
0.381
0.784
0.495
0.407
0.444
0.383
0.731
0.555
0.666
0.433
Local search performs comparatively better small search spaces and also search spaces with small neighborhood sizes.
E DETAILS FROM SECTION 5
In this section, we give more details on the API for NAS-Bench-Suitee
When designing
we strove for both minimalism and generalism. In order to add new benchmark in MASS-Bnch-SSiie te, one has to first define the computational graph using the NetworkX API. This graph encompasses high-level abstractions such as the add_node and add edges___...... _dges_frrmm from methods. If we are implementing tabular surrogate benchmark as the ones used in Section 4 get_dataset_ap function needs to be implemented, which is used as interface the original pre-computed benchmark data. The PyTorch computational graph generated via the adapt_ss adapt_search search _sacee method of the NAS algorithms or performance predictors. For instance, it can determine if operation choices in edges should be combined mixed operation (as done in DARTS (Liu et al. (019b)) or if they should be categorical choices from which the NAS algorithms sample. Afterwards, the graph instance is stored as an attribute of the optimizer instance. The Trainer, which runs the optimization loop. interacts only with the optimizer (see line 16 Snippet I.
22

---

Published as a conference paper at ICLR 2022
HParam. Predictors
HParam. Corr. Black-Box Algorithms
NB-101
DARTS NB NLP
0.8
NB-MR
0.6
TNB-Micro
0.4
0.2
TNB-Macro
0.0
NB-100
NB-201
DARTS
NB ASR
NB NIP
NB NB-MR
TNE Mil-ro
TNB TacrM
NB-101
NB 10-201
DARTS
NB-ASt
NBB. NLP
NO. Ma
TNB-Micra
TNB-Macre
Figure 11: Transferability results for predictors (left) and black-box algorithms (right). Row i, column
j denotes the Kendall Tau rank correlation of the performance of hyperparameters between search spaces and j. For abbreviations, see Table
Table 9: Leave one experiments for performance predictors. For search space A, the best hyperparameter setting on average over all search spaces except A is computed, and compared against the best hyperparameter setting of search space A when transferring to (or from) search space A.
NB-101
NB-201
DARTS
NB-ASR
NB-NLP
NB-MR
TNB-101
Transfer to Transfer from
0.372
0.37
0.48
0.195
0.185
0,445
0.252
0.226
0.16
0.223
0.058
0.029
0.241
0.056
Interquartile Range (logscale)
WAS-Bench-101
NAS-Bench-201
CIFAR-10
CIFAR-100
NAS Bench-201
ImageNet16 120
10a.
Number of Operations
Figure 12: Interquartile ranges of subsets of NAS-Bench-101 and NAS-Bench-201 as function of the number of operations. As the size of the search space increases, the interquartile range decreases.
23

---

Published as a conference paper at ICLR 2022
Table 10: Correlation insights for five NAS algorithms (left) or five performance predictors (right). Kendall Tau rank correlations are computed between properties the search spaces (search space size or neighborhood size) and the relative ranking list for predictors or NAS algorithms, with or without HPO. Since a ranking list used, negative correlation means positive correlation between the search space property and algorithmic performance.
Performance Predictors
NAS Algorithms
BOHAM.
GP
RF
XGB
NAO
RS
RE
BANANAS
LS
NPENAS
Default vs. SS size
-07
.25
-166
-39
.20
-29
-21
-.15
.48
.26
HPO vs. SS size
.15
.05
-51
-339
.20
-26
-21
-05
.35
Default Nbhd. size
-.33
.45
-26
.00
-10
.25
.37
.16
HPO vs. Nbhd. size
.35
.37
-.39
-.51
.00
-16
-31
.05
.31
.05
24


Reviews:
[{"review_id": "LOoYp48pN9Q", "reviewer": "Reviewer_2xvD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths:\n- The paper is well motivated and written.\n- This new benchmark can be used to do a more deep analysis for the NAS methods.\n- Some interesting observations.\n\nweaknesses:\n- The technical part is somewhat weak.\n- The table and figure captions are too short to explain themselves. Please at least explain the abbreviations used in the table or figure.", "summary_of_the_paper": "The authors collected most of the existing NAS benchmarks to construct a new benchmark.\nA unified API is provided to use these existing search spaces and architecture datasets. Based on this the authors re-analyze some NAS algorithms on this new large and comprehensive benchmark and have some interesting observations.", "summary_of_the_review": "The authors did much engineering efforts and emprical analysis, whereas the technical novelty is a little bit weak.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IwciH1vI5Ql", "reviewer": "Reviewer_Ensh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There exist many NAS benchmarks currently across diverse tasks. Many NAS works conduct experiments on several benchmarks to demonstrate the effectiveness and believe. \nHowever, researcher find that NAS algorithms may perform differently across these benchmarks. Sometimes even reverse conclusions are drawn.\n\nThis paper investigates this problem and points out that the differences may come from the different search space design, training pipeline and hyperparameters, via comprehensive experiments across many existing benchmarks. It finds that conclusions from a few benchmarks do not generalize well to other benchmarks\n\nFurther, the paper proposes NAS-Bench-Suite, a collection of NAS benchmarks, with a unified interface, which is easy-to-use, to facilitate reproducible, generalizable and rapid NAS research.\n\nThe motivation of the work is clear. The experimental analysis is convincing. The paper is well organized. The claim is sufficiently addressed by the method.\n\nOverall, this is a good work for the NAS community.", "summary_of_the_paper": "This paper investigates the drawbacks of current existing NAS benchmarks and the works that evaluate on them. Then it presents NAS-Bench-Suite, an extensible collection of NAS bench-marks which is easy-to-use to facilitate reproducible, generalizable and rapid NAS research.", "summary_of_the_review": "I think the paper is of good quality and will contribute to the NAS community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "98wRTtYhzJR", "reviewer": "Reviewer_Gku7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The results are interesting and very useful for the community as it is - and the authors should be commended for their effort.\nHowever the work lacks deeper insights from the experiments performed that could guide the community towards better methods. For example: [Fig.2] Are there identifiable components of architectures that lead to the interquartile range differences between the CIFAR10 benchmarks (NB101, NB201, DARTS)? Or [Table 2] What makes XGBoost generalize well where others don't?\n\nSec.4.2: Transfer is always expected to lead to a performance drop. Is there evidence that NAS is especially bad in this respect, compared to HP transfer in other settings? (say, transferring ResNets HPs from one dataset to another)\n\nFig.5: Since the figure contains a lot of information, could the authors provide some useful summary statistics in addition?\n\nMinor: clarify whether $i$ and $j$ are columns or rows in Fig.5. ", "summary_of_the_paper": "This paper proposes a unified interface for access to a collection of Neural Architecture Search (NAS) benchmarks. With experiments performed on across multiple search spaces and datasets, the authors show that some conclusions drawn from a small subset of benchmarks do not generalize across diverse datasets and tasks.", "summary_of_the_review": "In my view the two main contributions of this work are (1) showcasing the limitations of drawing conclusions from single benchmarks; (2) providing the community with a larger analysis tool.\n\nI expect that the community will welcome these resources and thereby **recommend the paper is accepted** (provisional score: 8)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "6z5A_nNakdb", "reviewer": "Reviewer_xtz2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Very important topic. Benchmarks are critical to assess performance of NAS algorithms.\n- Provide statistics across a wide array of NAS benchmarks, such as the distribution of accuracy across the search space.\n- They show that no NAS method clearly outperform all other method across all benchmarks. Some do better with HPO, but this is typically to expensive to computer for each benchmark.\n- Tests 2 relevant questions. 1) Whether NAS results generalizes from the small 101 and 202 benchmarks to larger ones and 2) NAS have robust hyperparameters. The third question seems to be to closely related to the second one: 3) We can cheaply optimize hyperparameters of NAS on tabular benchmarks and reuse them on another benchmark.. \n    - They show that 1) NAS algorithms do not generalize well from small 101 and 202 benchmarks to larger ones,\n    - 2) Hyperparameters are not robust for NAS algorithms.\n    - 3) Is not true obviously since hyperparameters of NAS algorithms are not robust.\n- They contribute a valuable wrapper for 25 benchmarks, although it is not clear to me how the search space representation with NetworkX allows for a dynamic search space definition. \n\nWeaknesses:\n- Out of the 3 questions, it seems to me the third one is to closely related to the first one. And the experimental answer to this question is not clearly outlined in the paper neither.\n- I am quite concerned about the maintainability of the library. The benchmarks are bash scripts that will be difficult to maintain. Some even contain specific slurm configurations for the cluster of the researchers. \n\nRandom walk plot is interesting. I would be great to have a confidence interval or at least a standard deviation for each curve.", "summary_of_the_paper": "This paper present a wrapper for 25 NAS benchmarks and provide insightful analysis on NAS and predictor performances across the benchmarks. They show, among other things, that no NAS method is best across all tasks, that hyperparameters of NAS are not robust and that small tabular benchmarks are not representative of NAS method performances on larger benchmarks. ", "summary_of_the_review": "The wrapper of benchmark is a valuable contribution by itself, but this paper illustrates well the benefit of such benchmarks by providing insightful analysis of NAS algorithms. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel benchmark for neural architecture search methods, which consists of 25 different combinations of search spaces and datasets. The main motivation is that existing NAS benchmarks, such as NAS-Bench-201, consider very small search space and few datasets, such that conclusions drawn with them do not generalize to unseen settings with different search spaces and datasets. The authors first describe the 25 different combinations of the search space and tasks for the given benchmark, and then conduct an extensive empirical study of existing NAS methods and performance predictors with the proposed benchmark, to show that architectures and hyperparameters found with the popular benchmarks do not generalize to other settings, which is consistent with their assumption.

—

All reviewers were initially positive about the paper, and remained positive throughout the discussion period. The reviewers found the paper well-motivated, and the proposed benchmark useful, as they agree with the need of introducing a single, unified framework that can validate a NAS method under diverse settings, since existing benchmarks only consider specific datasets and search spaces. However, the reviewers were also concerned with the weak technical novelty (Reviewer 2xvD), and that the work lacks deeper insights that could guide the community towards better methods (Reviewer Gku7). 

I also agree with the authors and the reviewers on the necessity of having a unified benchmark that incorporates all different settings considered in the previous benchmarks, and find the extensive empirical study of existing NAS methods useful. 

However, I find the work as rather technically weak as mentioned by R2xvD, since the authors spent too much time describing and showing the limitations of existing benchmark methods, while what is more important for benchmarks, is to justify how the proposed benchmark can evaluate the performance of different methods in a fair manner, while being representative of the practical settings. In short, the authors need to justify their design choices. Yet, the 25 settings proposed in the paper seem to have been arbitrarily chosen, and it is not clear if having a good performance on this benchmark is indeed a fair evaluation, or well-reflects how the NAS method will perform in practice. The proposed benchmark also does not really consider a novel search space or setting that have been overlooked in the past either, and does not provide much insights on the problem, as mentioned by Reviewer Gku7. 

Thus, although I recommend an acceptance for its practical value acknowledged by the reviewers, the authors need to put a considerable amount of effort in revising the paper, and If this were a journal submission, the paper may need to undergo a major revision. Most importantly, as described, the authors should justify their design choices as well as whether evaluating a model on the benchmark yields “fair” and “representative” results, focusing more on describing the proposed benchmark itself.
Decision: Accept (Poster)