NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search
Authors: Julien Niklas Siems
Lucas Zimmer
Arber Zela
Jovita Lukasik
Margret Keuper
Frank Hutter
Venue:  (2021)
Keywords: Neural Architecture Search
Benchmarking
Performance Prediction
Deep Learning

Abstract: The most significant barrier to the advancement of Neural Architecture Search (NAS) is its demand for large computational resources, which hinders scientifically sound empirical evaluations. As a remedy, several tabular NAS benchmarks were proposed to simulate runs of NAS methods in seconds.  However, all existing tabular NAS benchmarks are limited to extremely small architectural spaces since they rely on exhaustive evaluations of the space. This leads to unrealistic results that do not transfer to larger search spaces. To overcome this fundamental limitation, we propose NAS-Bench-301, the first surrogate NAS benchmark, using a search space containing $10^{18}$ architectures, many orders of magnitude larger than any previous tabular NAS benchmark. After motivating the benefits of a surrogate benchmark over a tabular one, we fit various regression models on our dataset, which consists of $\sim$60k architecture evaluations, and build surrogates via deep ensembles to model uncertainty. We benchmark a wide range of NAS algorithms using NAS-Bench-301 and obtain comparable results to the true benchmark at a fraction of the real cost. Finally, we show how NAS-Bench-301 can be used to generate new scientific insights.

Under review as a conference paper at ICLR 2021
NAS-BENCH-300 AND THE CASE FOR SURROGATE BENCHMARKS FOR NEURAL ARCHITECTURE SEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
The most significant barrier to the advancement of Neural Architecture Search (NAS) its demand for large computational resources, which hinders scientifi- cally sound empirical evaluations. As a remedy, several tabular NAS benchmarks were proposed to simulate runs of NAS methods in seconds. However, all exist- ing tabular NAS benchmarks are limited to extremely small architectural spaces since they rely on exhaustive evaluations of the space. This leads to unrealistic results that do not transfer to larger search spaces. To overcome this fundamen- the first surrogate NAS benchmark,
tal limitation, we propose WAS-Bench-300 using a search space containing 10 architectures, many orders of magnitude larger than any previous tabular NAS benchmark. After motivating the benefits of surrogate benchmark over tabular one, we fit various regression models on our dataset, which consists of 6600 architecture evaluations, and build surrogates via deep ensembles to also model uncertainty. We benchmark wide range of NAS algorithms using benchmark fraction of the real cost. Finally, we show how MAS-Bench-331 can be used to generate new scientific insights.
and obtain comparable results to the true
INTRODUCTION
Neural Architecture Search (NAS) promises to advance representation learning by automatically
finding architectures that facilitate the learning of strong representations for a given dataset. NAS
has already achieved sfate-of-the-at performance on many tasks (Real et al. 2019: Liu et al. 2019a
Saikia al. 2019 Elsken et al. 2020) and to create resource-aware architectures (Tan et 2018
Elsken et al.
2019a Cai et al. 2020). For a review, we refer to Elsken et al. (2019b).
Despite many advancements terms of both efficiency and performance, empirical evaluations in NAS are still problematic. Different NAS papers often use different training pipelines, differ- ent search spaces and different hyperparameters, do not evaluate other methods under comparable settings, and cannot afford enough runs for testing significance. This practice impedes assertions about the statistical significance the reported results, recently brought into focus by several au- thors (Yang et al. 2019: Lindauer & Hutter 2019 Shu et al.) 2020: Yu et al. 2020).
To circumvent these issues and enable scientifically sound evaluations in NAS, several tabular benchmarks (Ying al. 2019 Zela et al. 2020b Dong & Yang 2020 Klyuchnikov et al., 2020) have been proposed recently (see also Appendix A. for more details). However, all these bench- marks rely an exhaustive evaluation of all architectures in search space, which limits them to unrealistically small search spaces (so far containing only between 6k and 423k architectures). This far shot from standard spaces used in the NAS literature, which contain more than 1018 archi- tectures (Zoph & Le| 2017 [iu et al. 2019b). This discrepancy can cause results gained on existing tabular NAS benchmarks not generalize to realistic search spaces; e.g., promising anytime results of local search on existing tabular NAS benchmarks were shown to not transfer to realistic search spaces (White al. 2020b). To address these problems, we make the following contributions:
1.
We present NAS AAS--Bncc--01, a surrogate NAS benchmark that is first to cover a realistically sized search space (namely the cell-based search space of DARTS (Liu et al., 2019bp). containing more than 18 possible architectures. This is made possible by estimating their performance via surrogate model, removing the constraint to exhaustively evaluate the entire search space.

---

Under review as a conference paper at ICLR 2021
2. We empirically demonstrate that a surrogate fitted subset of architectures can in fact model the true performance of architectures better than a tabular benchmark (Section 2
3. We analyze and release the NAS-Bench-300 training dataset consisting of ~60k fully trained and evaluated architectures, which will also be publicly available in the Open Graph Benchmark (uu
al. 2020) (Section 3).
4. Using this dataset, we thoroughly evaluate a variety of regression models as surrogate candidates, showing that strong generalization performance is possible even in large spaces (Section 4
5. We utilize MAS-Bench-301
benchmark for running various NAS optimizers and show that the resulting search trajectories closely resemble the ground truth trajectories. This enables sound simulations of thousands of GPU hours in a few seconds
single CPU machine (Section 5 6. We demonstrate that NAS-Bench-301 can help in generating new scientific insights by studying a previous hypothesis on the performance of local search in the DARTS search space (Section
To foster reproducibilityy we open-source all our code and data in public repo: https: // anonymous www..........ccceeeeo
c4 72 4394 106---666------11 5d464e099aca/
MOTIVATION - CAN WE DO BETTER THAN A TABULAR BENCHMARK?
We start by motivating the use of surrogate benchmarks by exposing an issue of tabular benchmarks that has largely gone unnoticed. Tabular benchmarks are built around costly, exhaustive evaluation of all possible architectures in a search space, and when an architecture 's performance is queried, the tabular benchmark simply returns the respective table entry. The issue with this process is that the stochasticity of mini-batch training is also reflected in the performance of an architecture i, hence making it random variable Y, Therefore, the table only contains results of a few draws y; ~ Y, (existing NAS benchmarks feature up to 3 runs per architecture). Given the variance in these evalu- ations, tabular benchmark acts as simple estimator that assumes independent random variables, and thus estimates the performance of an architecture based only on previous evaluations of the same architecture. From a machine learning perspective, knowing that similar architectures tend to yield similar performance, and that the variance of individual evaluations can be high (both shown to be the case by Ying et al. (2019), it is natural to assume that better estimators may exist. In the remainder of this section, we empirically verify this hypothesis and show that surrogate benchmarks can provide better performance estimates than tabular benchmarks based on less data.
(Ying et al. 2019) as tab- ular benchmark for our analysis and a Graph Isomorphism Net- work (GIN, Xu et al. (019a)) as surrogate model Each ar- chitecture x; in NAS- Bench- 101 contains 3 validation accuracies v v²= v² from training x; with 3 different seeds. We excluded all diverged models with less than 50% validation accuracy on any of the three evaluations in NAS-Bench-101. We split this dataset to train the GIN surrogate model on one of the seeds, e.g., Dirain
Setup We choose
Model
Mean Absolute Error (MAE)
1, 3]
2. [1.
3.
Tab. Surr.
4.534e-3
4.546e-3
4.539e-3
3,446e-3
3,455e-3
3.441e-3
Table 1: MAE between perfor- mance predicted by a tab./sur.. benchmark fitted with one seed each, and the true performance of evaluations with the two other seeds. Test seeds in brackets.
(Iz; v! } and evaluate on the other two, e.g., D = (1,, U. ). where 07a = (v, + v²)/
We emphasize that training surrogate to model a search space is not a typical inductive regression task but rather a transductive one. By definition of the search space, the set of possible architectures is known ahead of time (although it may be very large), hence a surrogate model does not have to generalize to out-of- distribution data if the training data covers the space well.
17 of the surrogate model trained n
Results We compute the mean absolute error MAE
on Derain (z,, b/) where ÿJ is predicted validation accuracy and n (Diest] Table I shows
that the surrogate model yields a lower MAE than the tabular benchmark, i.e. MAE > 2.21
We also report the mean squared error and Kendall tau correlation coefficient in Table 6 in the Appendix showing that the ranking between architectures is also predicted better by the surrogate.
We used GIN implementation by Errica al. 2020 g see Appendix for details on training the GIN.
2

---

Under review as a conference paper at ICLR 2021
We repeat the experiment in cross- velidatain fashion w.r. to the seeds and conclude: In contrast to a single tabular entry, the surrogate model learns to smooth out the noise
Next, fit the GIN surrogate subsets of Dirain and plot how its performance scales with the amount of train- ing data used in Figure The surrogate model performs better than the tabular benchmark when the training set has more than ~21,500 architectures. Note that Diest
Mean Absolute Error
Surrogate Tabular
10-2
remains the same as in the previous experiment, i.e., it includes all architectures in
101. As re- sult, we conclude that: A surrogate model can yield strong predictive performance when only subset of the search space is available as training data.
10'
10a
10a
10"
10a
architectures NB-101
Figure I: Number of architectures used for training the GIN surrogate model vs MAE on the NAS-Benchi 101 dataset.
These empirical findings suggest that we can create reli- able surrogate benchmarks for much larger and more real- istic NAS spaces, which are infeasible to exhaustively evaluated as done by tabular benchmarks. In the remainder of the paper, we focus on creating such a benchmark.
THE NAS-BENCH-301 DATASET
We now describe the NAS- Be-cchnnh-0 dataset which con- sists of ~60k architectures and their performances CIFAR-10 10 (Krizhevsky) (009) sampled from the most popular NAS cell search space: the one from DARTS (Liu et aL. 2019b). We use this dataset not only to fit surro- gate models but also to gain new insights, such as which regions of the architecture space are being explored by different NAS methods, or what the characteristics of ar- chitectures are that work well.
NAS methods
eval
RS
Reratr Bengin 2012
23746
Evolution
DE
2020
7275
RE
4639
TPF
Rergrr et
6741
BO
BANANAS White
2243
COMBO
745
DARTS is all
2053
PC-DARTS
1588
One-Shot
DrNAS
Charet 2020
947
3.1 DATA COLLECTION
GDAS
Dong Ying 2019
234
Table 2: NAS methods used to cover the search space.
Since the DARTS search space (detailed description in Appendix C is far too large be exhaustively evalu- ated, care has to be taken when sampling the architectures which will be used to train the surrogate models. Sampling should yield a good overall coverage of the architecture space while also provid- ing a special focus on the well-performing regions that optimizers tend to exploit.
Our principal methodology is inspired by Eggensperger et al. (2015), who collected unbiased data about hyperpa- rameter spaces by random search, as well biased and dense samples in high-performance regions by running hyperparameter optimizers. This is desirable for surro- gate benchmark since are interested in evaluating NAS methods that exploit such good regions the space. Ta- ble lists the NAS methods we used to collect such sam- ples and the respective number of samples. Additionally, evaluated ~1 architectures in poorly-performing gions for better coverage and another ~ 10k for the anal- ysis conducted on the dataset and surrogates. We refer Appendices C.2 and C.3 for details on the data collection and the optimizers, respectively. We would like to point out that in hindsight adding training data of well-performing regions may be less important for a surrogate NAS benchmark than for a surrogate
Architecture Ranking
2nd component
50
40k
20k
-50
500
50
1st component
Figure 2: t-SNE visualization of the sampled architectures.
HPO benchmark, which we demonstrated in Appendix E.3 We argue that this result of HPO
We do note that the average estimation eиrоr of tabular benchmarks could be reduced by factor of v by performing & runs for each architecture. The error of surrogate model would also shrink when the model is based on more data, but l grows large tabular henchmarks would become competitive with surrogate models.
3

---

Under review as a conference paper at ICLR 2021
search spaces containing many configurations which yield disfunctional models, which is less com- mon for architectures in many NAS search spaces, hence allowing random search to give us good coverage of the space.
In Figure 2 we visualize the overall coverage of the search space as well as the similarity between sampled architectures using t-SNE (van der Maaten & Hinton 2008). Besides showing good over- all coverage, some well-perforning architectures in the search space form distinct clusters which mostly located outside the main cloud of points. This clearly indicates that architectures with similar performance are close to each other in the architecture space. Additionally, we observe that different optimizers sample different types of architectures, see Figure 9 in the Appendix.
3.2 PERFORMANCE STATISTICS
Figure shows the validation error on CIFAR-10 (Krihhessyy 2009) of all sampled architectures in relation the model parameters and training runtime. Generally, expected, models with more parameters are more costly to train but achieve lower validation errors. We also find that different NAS methods yield quite different performance distributions (see Appendix C.4 for their individual performances). Validation and test errors are highly correlated with a Kendall tau rank correlation of 0.852 (Spearman rank corr. 0.969), minimizing the risk of overfitting on the validation error.
Furthermore, we find that cells of all depths can reach a good performance, but shallow topolo- gies are slightly favored in our setting (see Figure 10 in the Appendix). Also, a small number of parameter-free operations (e.g.. skip connections) can benefit the performance but featuring many of these significantly deteriorates performance. For the full analysis, see Appendix
C.5
Following standard practice in modern NAS papers (e.g., Liu et al. (2019))), we employ various data augmentation techniques during training for more reliable estimates of an architecture' 's perfor- mance. For description of our full training pipeline, please see Appendix C.6
3.3 NOISE IN ARCHITECTURE EVALUATIONS
As discussed in Section the noise in architecture evaluations can be large enough for surrogate mod- els yield more realistic estimates of architecture performance than tabular benchmark based on a single evaluation per architecture. To study the mag- nitude of this noise NAS-Bench-301,
10
Runtime (x10 s)
Validation error
0.2
6
evalu-
ated 500 architectures randomly sampled from our Differential Evolution (DE) (Awad et al., 2020) with 5 different seeds each
0.1
4
R We find a mean stan- possi-
3
dard deviation of 1.6e-3 3 for the final validation
0.05
curacy which slightly less than the noise observed in NAS-Bence AAS-Bench-10) (Ying al. 2019); ble reason for this could be a more robust training pipeline. Figure 12 in the Appendix shows that, while the noise tends to be lower for the best archi- tectures, a correct ranking based on a single evalua- tion is still difficult. Finally, we compare the MAE when estimating the architecture performance from only one sample to the results from Table [] 1.38e-3 than for NAS-Bench-101.
1
2
Num. parameters [10e6]
Figure 3: Number of parameters against val. error with model training time colorbar.
Here, we also find a slightly lower MAE of
FITTING SURROGATE MODELS ON THE NAS-
NAS-BENCH- BEN-B-ENHHH 301 DATASET
We now focus on creating a surrogate model. To that end, we evaluated a wide range of regression models on the NAS-Bench-301 dataset. In principle, any such model can give rise surrogate NAS benchmark, but models that fit the true performance better yield surrogate NAS benchmarks whose characteristics are more similar the ones of the true benchmark. Therefore, we naturally
We chose DE because it both explored and exploited well, Figure in the Appendix.
4

---

Under review as a conference paper at ICLR 2021
strive for the best-fitting model. We emphasize that in this work do not attempt introduce new regression model but rather build on the shoulders of the architecture performance prediction community.
4.1
SURROGATE MODEL CANDIDATES
Deep
Graph Convolutional Neural Networks are frequently used as NAS predictors (Friede et al.
2019
Wen al. 2019; Ning et al. 2020p. In particular, we choose the GIN since several works have
found
it to perform well on many benchmark datasets (Errica et al. 2020 Hu et al. 2020 Dwivedi
al.
2020
We use the publicly available implementation from the Open Graph Benchmark (Hu
et al.
2020 and refer to Appendix D.2 for further details.
We compare the GIN to variety of common regression models. We evaluate Random Forests (RF) and Support Vector Regression (SVR) using implementations from scikit-learn (Pedregosa et al.,
2011). We also compare to the tree-based gradient boosting methods XGBoost (Chen & Guestrin,
2016 LGBoost (Ke al. 2017) and NGBoost (Duan et al., (020), recently used for predictor-
based NAS (Luo et al. 2020). We comprehensively review architecture performance prediction in Appendix A.2
4.2 EVALUATING THE DATA FIT
Similarly to Wen et al. (0119) and Baker et al. (017)) we assess the quality of the data via the coefficient determination (R²)
Test
Model
and the Kendall rank correlation coefficient (r). Since Kendall is sensitive to noisy evaluations that change the rank of an architec- ture, we follow the recent work by Yu et (2020) and use sparse Kendall Tau (sKT), which ignores rank changes 0.1% accuracy precision, by rounding the predicted validation accuracy prior computing T.
R²
sKT
LGBoost
0.892
0.816
XCBoost
0.832
0.817
GIN
0.832
0.778
NGBaost
0.810
0.759
n-SVR
0.709
0.677
MLP (Path enc.)
0.704
0.697
0.679
0.683
All hyperparameters of the surrogate models BOHB (Falkner et al. 2018) as a black-box optimizer; details on their respective hyperparameter search spaces are given in Table 7 in the appendix. We use train/val/test splits (0.8/0. 1/0.1) stratified across the NAS methods used for the data collection. This means that the ratio of architectures from particular optimizer is constant contains 50% of its architectures from RS since RS used to
tuned using
cSSV
0.675
0.660
Table Performance of dif-
ferent regression models fit-
ted on the NB-301 dataset.
the splits, e.g. the obtain 50% of the total urchitectures we trained and evaluated. We provide additional details on the preprocessing of the architectures for the surrogate models in Appendix [D.I] As Table Table shows, the three best- pest-performing models are LGBoost, XGBoost and GIN; we therefore focus our analysis on these in the following.
In addition to evaluating the data fit on our data splits, we investigate the impact of parameter- and and D.7 respectively. We find that all of LGBoost, XGBoost and GIN accurately predict the drop in performance when increasingly replac- ing operations with parameter-fire operations in a normal cell.
free operations and the cell topology Appendices [.6]
Model
No RE
No
No COMBO
No TPE:
No BANANAS
No DARTS
No PC-DARTS
No DrNAS
No GDAS
LGB
0.917
0.892
0.919
0.857
0,909
-0.093
0.826
0.699
0-429
112
0.907
0.672
GIN
0.356
0.864
0.775
0.789
0.881
0.115
0.661
0.790
0.572
0.780
0.595
sKT
XGB
0.831
0.780
0.817
0.762
0.596
0.775
0.710
0.709
0,638
Table 4: Leave One-Optimizer-OOt performance of the best surrogate models.
5

---

Under review as a conference paper at ICLR 2021
4.3 LEAVE THREE-OPIMIZER-OO OUT ANALYSIS
Since the aim of necessary to ensure that the surrogate model can deliver accurate performance estimation on data from trajectories by unseen NAS methods. Similarly to |ggennspergr et al. (013). we therefore perform form of vross-validation on the optimizers we used for data collection, i.e. we leave out all data collected by one of the NAS methods entirely during training (using a stratified 0.9/0.1 train/val split over the other NAS methods). Then, we predict the unseen results from the left-out NAS method to evaluate how well the models extrapolate to the region covered by the 'unseen' method. We refer to this as the save-oon-optimmizr-oo
is to allow efficient benchmarking of novel NAS algorithms,, it is
(LO0O) setting.
Results
The results in Table show that the rank correlation between the predicted and observed validation accuracy remains high even when a well-performing optimizer such as RE is left out. Predicting BANANAS in the LOO0 fashion yields lower rank correlation, because it focuses well-performing architectures that are harder to rank; however, the high R² shows that the fit is still good.
Conversely, leaving out DARTS causes low Ri but still high sKT; this is due to architectures with many skip connections in the DARTS data that are overpredicted (further discussed in Section (5.).. For full details, Figure [6 the appendix provides scatter plots of the predicted vs. true performance for each NAS method.
4.4 NOISE MODELLING
Ensemble methods are commonly used to improve predictive performance (Dietterich) 2000). More- over, ensembles of deep neural networks, so-called deep ensembles, have been proposed as a simple way to obtain predictive uncertainty (Lakshminarayanaa) semble of 10 base learners for each of our three best performing models (GIN, XGB, LGB) using a 10-fold cross-validation for our train and validation split, as well as different initializations. We use the architectures with multiple evaluations (see Section (3.3) to mirror the analysis in the motivation in Section 2 We train using only one evaluation per architecture (i.e., seed 1) and take the mean accuracy of the remaining ones as groundtruth (i.e., seeds 2-5). We then compare against a tabular model with just one evaluation (seed) 1).
al.) 2017). We therefore create en-
Table|3]shows that the GIN and LGB surrogate mod- els yield estimates closer groundtruth than the table lookup based on one evaluation. This con- firms our main finding from Section 2 but this time on much larger search space. We also compare predictive distribution of our ensembles to the groundtruth. To that end, we assume the noise in the architecture performance to be normally distributed (KL) divergence
Model
MAE 12,3,455
Mean
div.iv.
Tabular
1.38e-3
undef.
undef.
GIN
1.13e-3
0.6e-3
16.4
LGB
1.33e-3
0.3e-2 3
68.9
XGB
1.51e-3
0.3e-
134.4
Table 5: Metrics for the selected surrogate models on 500 architectures that were evalu- ated 5 times.
and compute the Kullback-Leibe between the groundtruth accuracy distribution and predicted distribution. We find the GIN ensemble quite clearly provide the best estimate.
To allow evaluations multi-objective NAS methods, and to allow using "simulated wallclock time" the axis of plots, also predict the runtime of architecture evaluations. For this, train an LGB model with the runtime as targets (see Appendix [D4 details). Runtime prediction is less challenging than performance prediction, resulting in excellent fit of our LGB runtime model on the test set (sKT: 0.936, R2: 0.987). Other metrics of architectures, such as the number of parameters and multiply-adde. do not require a surrogate model but can be queried exactly.
NAS-BENCH-300 AS SURROGATE NAS BENCHMARK
Having assessed the ability the surrogate models to model the search space, we now use NAS-Bench-30 to benchmark various NAS algorithms.

---

Under review as a conference paper at ICLR 2021
True Benchmark
GIN Surrogate Benchmark
XGB Surrogate Benchmark
valication achiever
9 10
8 10
TPE
10
BANANAS
RS
6 10
10
10a
10a
10
10'
10*
10a
10"
10'
10"
10*
10a Simulated Wallclock
10a
10"
Wallclock
Simulated Wallclock Time [s]
Figure 4: Anytime performance of different optimizers the real benchmark (left) and the surrogate benchmark (GIN (middle) and XGB (right)) when training ensembles on data collected from all optimizers. Trajectories the surrogate benchmark averaged over 5 optimizer runs.
True Benchmark
GIN Surrogate Benchmark
XGB Surrogate Benchmark
Best validation achieved
9 10
BANANAS
10-2
RE
DE
10-
6 10
5 10-2
10
10a
10
10'
10c
10a
10f
107
10
10*
10a
10
10'
Wallclock Time (s)
Simulated Wallclock Time [s]
Simulated Wallclock Time [s]
Figure 5: Anytime performance of blackbox optimizers, comparing performance achieved on the real benchmark and on surrogate benchmarks built with GIN and XGB in an LOOO fashion.
5.1 BLACKBOX OPTIMIZERS
We first compare the trajectories on the true benchmark and on the surrogate benchmark for blackbox optimizers when training the surrogate on all data. For the true benchmark, we show the trajectories contained in our dataset (based on single run, since we could not afford repetitions due the ex- treme compute requirements of 115 GPU days for single run). For the evaluations the surrogate, on the other hand, we can trivially afford to perform multiple runs. For the surrogate trajectories, identical initialization for the optimizers (e.g., initial population for RE) but evaluations of the surrogate benchmark are done by sampling from the surrogate model's predictive distribution for the architecture at hand, leading different trajectories.
Results (all data) As Figure 4 shows, both the XGB and the GIN surrogate capture behaviors present on the true benchmark. For instance, the strong improvements of BANANAS and RE are also present the surrogate benchmark the correct time. In general, the ranking of the optimizers towards convergence is accurately reflected on the surrogate benchmark. Also, the initial random exploration algorithms like TPE, RE and DE is captured the large initial variation in perfor- mance indicates. Notably, the XGB surrogate ensemble exhibits a high variation in well-performing regions as well and seems slightly underestimate the error of the best architectures. The GIN surrogate, on the other hand, shows less variance in these regions but slightly overpredicts for the best architectures.
Note, that due to the size of the search space, random search stagnates and cannot identify one of the best architectures even after tens of thousands of evaluations, with BANANAS finding better architectures orders of magnitude faster. This stands in contrast to previous NAS benchmarks. For instance, NAS- SuS--Becc-2000 (Dong & Yangl (2020) only contains 6466 unique architectures total, causing the median of random search runs to find the best architecture after only 3233 evaluations.
To simulate benchmarking of novel NAS methods, we expand on the leave-one-optime www.oonooiieeooooouotoo anal- ysis (LO00) from Section 4.3 and assess each optimizer with surrogate benchmarks based data excluding that gathered by said optimizer. We again compare the trajectories obtained from 5 runs on the surrogate benchmark the groundtruth.

---

Under review as a conference paper at ICLR 2021
True Benchmark
GIN Surrogate Benchmark
XGB Surrogate Benchmark
0.09
DARIS
Validation error
0.08
GDAS
PC DARTS
0.07
0.06
0.05
10
20
30
40
50
10
20
30
40
50
0
10
20
30
40
50
Epochs
Epochs
Epachs
Figure 6: Anytime performance of one-shot optimizers, comparing performance achieved on the real benchmark and on surrogate benchmarks built with GIN and XGB in LOO0 fashion.
the trajectories in the leave-one-optimiizr--uu setting. The XGB and GIN surrogates again capture the general behavior of different optimizers well, illustrating that characteristics of new optimization algorithms can be captured with the surrogate benchmark. Leav- ing DE appears to be bigger problem for XGB than GIN, pointing to advantages of the smooth embedding learned by the GIN compared to wradient-boosting
Results (LO00)
In additional experiment, (92% and above) yielded poor extrapolation to worse architectures, but that surrogate benchmarks based them still yielded realistic trajectories. We attribute this to NAS optimizers' focus on good architectures. For details, see Appendix E.2 We also investigate whether it is possible to create benchmarks only on random architectures obtain realistic trajectories but lose some predictive performance in the well-performing regions. Nevertheless, such benchmarks have the advantage of not possibly favouring any NAS optimizer used for the generation of training data, and we thus recommend their release in addition to the benchmarks based on the full training data.
found that surrogates built only well-performing architectures
Appendix E.3 and find that we can indeed
5.2 ONE-SHOT OPTIMIZERS
NAS-Bench-301 can also be used to monitor the behavior of one-shot NAS optimizers throughout their search phase, by querying the surrogate model with the currently most promising discrete architecture. This can be extremely useful in many scenarios since uncorrelated proxy and true objectives lead to potential failure modes, e.g., case where the found architectures contain only skip connections in the normal cell (Zela et al., 2020a b Dong & Yang 2020) (we study such failure case in Appendix too ensure robustness of the surrogates in said case). We demonstrate this use case in a similar LO00 analysis as for the black-box optimizers, using evaluations of the discrete architectures from each search epoch of multiple runs of DARTS, PC-DARTS and GDAS as ground-truth Figure shows that the surrogate trajectories closely resemble the true trajectories.
USING NAS-BENCH-301 TO DRIVE NAS RESEARCH
We finally our new benchmark perform case study that demonstrates how NAS-Bench- 301 can drive NAS research. Coming up with research hypotheses and drawing conclusions when prototyping or evaluating NAS algorithms less realistic benchmarks difficult, particularly when these evaluations require high computational budgets. NAS-Bench-301 alleviates this dilemma via its cheap and reliable estimates.
To showcase such scenario, we evaluate Local Search (LS) on our surrogate benchmark and the actual DARTS benchmark. |White al. (2020b) concluded that LS does not perform well on such a large space by running for 11.8 GPU days ( 10t seconds), and we are able to reproduce the same results via MAS-Bench-30 in a few seconds (see 7.. While White et al. (2020b) could not afford longer runs (nor repeats), qualitatively different behavior when run for an order of magnitude longer, transitioning from being
MAS-Bench-Ao 301 this is trivial. Doing suggests that LS shows
Wee the implementation and settings of Local Search provided by White et al.
2020h
8

---

Under review as a conference paper at ICLR 2021
the worst method to being one of the best. We verified this suggestion by running LS for longer on the actual DARTS benchmark (also see Fig This allows us to revise the initial conclusion of White et al. (2020b) to: LS is also state-of tatee---f-eeaaaeeeeeeeee the-trtteee for the DARTS search space, but only when given enough time.
Local Search
This case study shows how ready used to cheaply obtain hints on a research hy- pothesis that lead correcting a previous finding that only held for short runtimes. We look forward to additional uses along such lines.
was al-
Best vallidation error achieved
x 10
LT-GT
8 10
LS GIN
LS-XGB
BANANAS-GT
x 10
x 10
7 CONCLUSIONS & GUIDELINES FOR USING NAS-BENCH-301
x 10
11.8 GPU days;
We proposed MAS-Bench-301. the first surrogate NAS benchmark and first to cover a realistic search space which is orders of magnitude larger than all previous tabular NAS benchmarks. After motivating the benefits of surrogate benchmark over a tabu- lar one, we described the strategy used to collect the data which we used to fit our selected surrogate models and evaluated their predictive performance. Lastly, we demonstrated that our surrogate benchmark can accurately simulate real anytime per- formance trajectories of various NAS methods at a fraction of the true cost and can lead to new scientific findings. We hope that NAS-Bench-301 will allow the NAS practitioner to quickly pro- totype and benchmark NAS algorithms on the currently most used search space, without requiring large computational resources. We also argue that NAS-Bench-301 could also be used to monitor one-shot optimizers during their search phase, to detect failure cases early on. Finally, the ideas and methods discussed in our work trivially transfer other search spaces or datasets, allowing for the design of many interesting surrogate benchmarks in the future.
10 10a 10 10
(Simulated) Wallclock Time [s]
Figure 7:
Case study results for Local Search. GT is the ground truth, GIN and XGB are results on MAS-Bench-301.
Finally, want to mention the risk that prior knowledge about the surrogate model in NAS-Bench-301 could lead to the design of algorithms that may overfit to the surrogate benchmark. To this end, we recommend the following best practices ensure safe and fair benchmarking of NAS methods on NAS-Bench- and future surrogate benchmarks:
The surrogate model should be treated as black-box function, hence only be used for perfor- mance prediction and not exploited to extract, e.g., gradient information.
We discourage benchmarking methods that internally use the same model as the surrogate model picked in NAS-Bench-301 (e.g. GNN-based Bayesian optimization should not only be bench- marked using the GIN surrogate benchmark).
We encourage rumning experiments on versions of rogate benchmarks) that are based on (1) all available training architectures and (2) only archi- tectures collected with uninformed methods, such as random search or space-filling designs. As shown in Appendix [..3 (1) yields better predictive models, but (2) avoids any potential bias (in the sense of making more accurate predictions for architectures explored by particular type of NAS optimizer) and can still yield strong benchmarks.
(and other, future NAS sur-
In order to ensure comparability of results in different published papers, we ask users to state the benchmark's version number. We will continuously collect more training data and further improve the surrogate model predictions. So far, we release NB301-XGB- XGB-v1.0,, NB301-GIN- v1.0, 1-800------ann--ii and
Due to the flexibility of surrogate NAS benchmarks to cover arbitrary search spaces, we expect NAS-Bench-300 to be the first of many such benchmarks. We collect best practices for the creation of new surrogate benchmarks in Appendix F Having access to a variety of benchmarks is essential to the development and evaluation of new NAS methods. We therefore encourage the community to expand the scope of current NAS benchmarks to different search spaces, datasets, and problem domains utilizing surrogate benchmarks to cover large spaces.
Acknowledgements We thank the anonymous reviewers for suggesting very insightful experiments, in particular the experiments for NAS benchmarks based only random architectures.

---

Under review as a conference paper at ICLR 2021
REFERENCES
N. Awad, N. Mallik, and F. Hutter. Differential evolution for neural architecture search (dchb). In International Conference on Machine Learning (ICLR) Neural Architecture Search (NAS) Work- shop, 2020.
B. Baker, O. Gupta, R. Raskar, and N. Naik. Accelerating Neural Architecture Search using Perfor- mance Prediction. In NeurlPS Workshop on Meta- Letan-Lernnnn, 2017.
R. Baptista and M. Poloczek. Bayesian optimization of combinatorial structures. In International Conference on Machine Learning (ICLR). pp. 462-471, 2018.
Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13:281-305, 2012.
J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for hyper-parameter optimization. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Weinberger (eds.), Proceedings of the 25th International Conference Advances in Neural Information Processing Systems (NIPS') 11), pp. 2546-2554, 2011.
H. Cai, C. Gan, T. Wang, Z. Zhang, and S. Han. Once-for-all: Train one network and specialize it for efficient deployment. International Conference on Learning Representations (ICLR), 2020.
Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference knowledge discovery and data mining, pp. 785-794, 2016.
X. Chen, Wang, M. Cheng, X. Tang, and C. Hsich. Drnas: Dirichlet neural architecture search. arXiv preprint arXiv:2006. 10355, 2020.
P. Chrabaszez, I. Loshchilov, and F. Hutter. A downsampled variant of imagenet alternative to the cifar datasets. arXiv preprint arXiv:1707 080999 2017.
B. Deng, J. Yan, and D. Lin. Peephole: Predicting network performance before training. arXiv preprint
2017.
DeVries and G. W. Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint arXiv: 708.04552, 2017.
T. Dietterich. Ensemble Methods in Machine Learning, volume 1857 of Lecture Notes in Computer Science. Springer Berlin Heidelberg, 2000.
T. Domhan, J. T. Springenberg, and F. Hutter. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Q. Yang and M. Wooldridge (eds.), Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI' (U), pp- 3460-3468, 2015.
X. Dong and Y. Yang. Searching for a robust neural architecture four gpu hours. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 1761-1770, 2019.
X. Dong and Y. Yang. Nas-bench-102: Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations (ICLR), 2020.
Duan, A. Avati, D. Ding, Khanh K. Thai, S. Basu, A. Ng, and A. Schuler. Ngboost: Natural gradient boosting for probabilistic prediction. In Proceedings of Machine Learning and Systems 2020, pp. 6138-6148, 2020.
V. P. Dwivedi, C. K. Joshi, T. Laurent, Y. Bengio, and X. Bresson. Benchmarking graph neural networks. arXiv preprint arXiv:2003. 200:02003,,0 2020.
K. Eggensperger, F. Hutter, H.H. Hoos, and K. Leyton-Brown. Brown. Efficient benchmarking of hyperpa- rameter optimizers via surrogates. In B. Bonet and S. Koenig (eds.), Proceedings the Twenty- nineth National Conference on Artificial Intelligence (AAAI'1S) pp. 1114-1120, AAAI Press, 2015.
10

---

Under review as a conference paper at ICLR 2021
T. Elsken, J. H. Metzen, and F. Hutter. Efficient multi-objectivg neural architecture search via lamar- ckian evolution. In International Conference on Learning Representations, 2019a.
T. Elsken, H. Metzen, and F. Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-211 2019b.
T. Elsken, B. Staffler, J. Metzen, and F. Hutter. Meta-lcarning of neural architectures for few- shot learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12365-12373 2020.
F. Errica, M. Podda, D. Bacciu, and A. Micheli. A fair comparison of graph neural networks for graph classification. In International Conference on Learning Representations, 2020.
Falkner, Klein, and F. Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. In Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1437-1446, 2018.
M. Fey and E. Lenssen. Fast graph representation learning with Py PyTorch Geometric. In ICLR Workshop Representation Learning Graphs and Manifolds, 2019.
D. Friede, J. Lukasik, H. Stuckenschmidt, and M. Keuper. A variatiional-sequeniddl graph autoen- coder for neural architecture performance prediction. ArXiv, abs/1912.05317, 2019.
D. Golovin and A. Krause. Adaptive submodularity: Theory and applications in active learning and stochastic optimization. Journal Artificial Intelligence Research, 42:427-44 2011.
M. Gori, G. Monfardini, and F. Scarselli. A new model for learning in graph domains. In Pro- ceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pp. 729- 734. IEEE, 2005.
K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In Computer Vision and Pattern Recognition (CVPR), 2016.
W. Hu, M. Fey, M. Zitnik, Dong, H. Ren, B. Liu, M. Catasta, and Leskovec. Open graph benchmark: Datasets for machine learning graphs. arXiv preprint arXiv:2005.00687, 2020.
J. Hwang, S. Lay, and A. Lippman. Nonparametric multivariate density estimation: a comparative study, IEEE Transactions on Signal Processing, 42(10))2795-2810, 1994.
R. Istrate, Scheidegger, G. Mariani, D. Nikolopoulos, C. Bekas, and A. C. I. Malossi. Tapas: Train-less accuracy predictor for architecture search. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 3927-3934, 2019.
G. Ke, Q. Meng, Finley, T. Wang, Chen, W. Ma, Q. Ye, and Liu. Lightghm: A highly efficient gradient boosting decision tree. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 3146-3154. Curran Associates, Inc., 2017.
T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
A. Klein, S. Falkner, J. T. Springenberg, and F. Hutter. Learning curve prediction with Bayesian neural networks. In Proceedings the International Conference Learning Representations (ICLR' CBLS, 2017.
N. Klyuchnikov, Trofimov, E. Artemova, M. Salnikov, M. Fedorov, and E. Burnaev. Nas-bench- nlp: Neural architecture search benchmark for natural language processing. arXiv preprint arXiv:2006.07 2020.
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty esti- mation using deep ensembles. In Advances in neural information processing systems, pp. 6402- 6413, 2017.
11

---

Under review as a conference paper at ICLR 2021
G. Larsson, M. Maire, and G. Shakhnarovich. Fractalnet: Ultra-deep neural networks without resid- uals. In International Conference on Learning Representations (ICLR), 2017.
M. Lindauer and F. Hutter. Best practices for scientific research on neural architecture search. arXiv preprint urxiiv:9099 1900.0249,,9 2019.
M. Lindauer, K. Eggensperger, M. Feurer, A. Biedenkapp. J. Marben, P. Müller, and F. Hut- ter. Boah: A tool suite for multi-fidelity bayesian optimization & analysis of hyperparameters. arXiv:1908, 06756 [cs. 2019.
C. Liu, L. Chen, Schroff, H. Adam, W. Hua, A. Yuille, and L. Fei-Fei. Auto-deeplab: Hierar- chical neural architecture search for semantic image segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 82-92, 2019a.
H. Liu, K. Simonyan, and Y. Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019b.
I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference Learning Representations (ICLR) 2017 Conference Track, April 2017.
R. Luo, X. Tan, R. Wang. T. Qin, E. Chen, and T. Liu. Neural architecture search with gbdt. arXiv preprint ar arXiv:2007. 200700077744 2020.
M. D. McKay, R. J. Beckman, and W. J. Conover. A comparison of three methods for selecting values of input variables the analysis output from a computer code. Technometrics, 42(1); 55-61, 2000.
X. Ning, Y. Zheng, T. Zhao, Y. Wang, and H. Yang. A generic graph- neural architecture encoding scheme for predictor predietor-based nas. arXiv preprint (rXiv:2004.0189) 2020.
C. Oh, J. Tomezak, E. Gavves, and M. Welling, Combinatorial bayesian optimization using the graph cartesian product. In Advances Neural Information Processing Systems, pp. 2910-2920, 2019.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, et al. Scikit-learn: Machine learning in python. Journal of machine learning research, (((ee))2825-2830) 2011.
K. Price, R. M. Storn, and J. A. Lampinen. Differential evolution: a practical approach to global optimization. Springer Science & Business Media, 2006.
E. Real, A. Aggarwal, Y. Huang, and Q V. Le. Regularized evolution for image classifier archi- tecture search. In Proceedings of the uaai conference on artificial intelligence, volume 33, pp- 4780-4789, 2011.
T. Saikia, Y. Marrakchi, A. Zela, F. Hutter, and T. Brox. Autodispnet: Improving disparity estimation with automl. In IEEE International Conference on Computer Vision (ICCV), 2019.
H. Shi, R. Pi, H. Xu, Z. Li, J. T. Kwok, and T. Zhang. Multi-objective neural architecture search via predictive network performance optimization. arXiv preprint arXiv:1911.09336, 2019.
Y. Shu, W. Wang, and S. Cai. Understanding architectures learnt by cell-based neural architecture search. In International Conference Learning Representations (ICLR), 2020.
I. M. Sobol'. On the distribution of points in cube and the approximate evaluation of integrals. Zhurnal Vychislitel noi Matematiki Matematicheskoi Fiziki, 7(4):784-802, 1967.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Ra- binovich. Going deeper with convolutions, In Computer Vision and Pattern Recognition (CVPR), 2015.
M. Tan, B. Chen, R. Pang, V. Vasudevan, and Q Le Mnasnet: Platform-aware neural architecture search for mobile. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2815-2823, 2018.
12

---

Under review as a conference paper at ICLR 2021
Y. Tang. Y. Wang, Y. Xu, H. Chen, B. Shi, C. Xu, C. Xu, Q Tian, and C. Xu. A semi semi-supervised assessor of neural architectures. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
L. van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research (JMLR), (7N7)22577- 2008.
W. Wen, H. Liu, H. Li, Y. Chen, G. Bender, and P. Kindermans. Neural predictor for neural archi- tecture search. arXiv preprint arXiv: 1912.0088, 2019.
C. White, W. Neiswanger, and Y. Savani. Bananas: Bayesian optimization with neural architectures for neural architecture search. arXiv preprint arXiv: /ttp//911./1881/ 2019.
C. White, W. Neiswanger, S. Nolen, and Y. Savani. A study encodings for neural architecture search. arXiv preprint UrXiv:2007.04965 2020a.
C. White, S. Nolen, and Y. Savani. Local search is state of the art for nas benchmarks. arXiv preprint arXiv:2005.0296 2020b.
Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and Yu. A comprehensive survey on graph neural networks. arXiv preprint 10X1::1901000590 2019.
K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? In Interna- tional Conference on Learning Representations, 2019a.
Y. Xu, Y. Wang, K. Han, H. Chen, Y. Tang, S. Jui, C. Xu, Q. Tian, and C. Xu. Rnas: Architecture ranking for powerful networks. arXiv preprint 1.Ki::19000052:, 2019b.
Y. Xu, L. Xie, X. Zhang, X. Chen, G. Qi, Q. Tian, and H. Xiong. Pc-darts: Partial channel connec- tions for memory- architecture search. In International Conference Learning Repre- sentations, 2020.
A. Yang, P. M. Esperança, and F. M. Carlucci. Nas evaluation is frustratingly hard. arXiv preprint 19Xiv:111..12522 2019.
C. Ying, A. Klein, E. Christiansen, Real, K. Murphy, and Hutter. MAS-bench-100 101: Towards reproducible neural architecture search. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceed- ings of Machine Learning Research, pp. 7105-7114, 2019.
K. Yu, R. Ranftl, and M. Salzmann. How to train your super-net: An analysis of training heuristics in weight-sharing nas. arXiv preprint arXiv:2003.0422 04277, 2020.
A. Zela, T. Elsken, T. Saikia, Y. Marrakchi, T. Brox, and F. Hutter. Understanding and robustify- ing differentiable architecture search. In International Conference on Learning Representations, 2020a.
A. Zela, J. Siems, and F. Hutter. Nas-bench-1shoti. Benchmarking and dissecting one-shot neural architecture search. In International Conference on Learning Representations, 2020b.
H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference Learning Representations, 2018.
J. Zhou, G. Cui, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun. Graph neural networks: A review of methods and applications. arXiv preprint arXiv: 2018.
B. Zoph and Q V. Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations (ICLR) 2017 Conference Track, 2017.
13

---

Under review as a conference paper at ICLR 2021
A RELATED WORK
A.1 EXISTING NAS BENCHMARKS
Benchmarks for NAS were introduced only recently with MAS--Benc--10 101 (Ying et al. 2019) as the first among them. NAS-Bench-100 is tabular benchmark consisting of 4223 unique architectures in cell structured search space evaluated on CIFAR-IP 10 (Krizhevsky 2009). To restrict the number of architectures the search space, the number of nodes and edges was given an upper bound and only three operations are considered. One result of this limitation is that One-Shot NAS methods can only be applied to subspaces of NAS-Bench- 101 as demonstrated in NAS-Bench- Shott (Zela et al. 2020b).
20S-Bench-201 (Dong & Yang 2020). in contrast, uses a search space with a fixed number of nodes and edges, hence allowing for straight-forward application of one-shot NAS methods. How- ever, this limits the total number of unique architectures to as few as 6466. NAS-Bench-201 in- cludes evaluations of all these architectures on three different datasets, namely CIFAR-10, CIFAR- 100 (Krizhevsky] 2009) and Downsampled Imagenet 16x 16 (Chrabaszez et al. 2017). allowing for transfer learning experiments.
NAS-Bench-NLP Klyuchniikov et al, 2020) was recently proposed as a tabular benchmark for NAS in the Natural Language Processing domain. The search space resembles NAS-Bench-101 as limits the number of edges and nodes to constrain the search space size resulting 14k evaluated architectures.
A.2 NEURAL NETWORK PERFORMANCE PREDICTION
In the past, several works have attempted predict the performance of neural networks by ex- trapolating learning curves (Domhan et al. 2015 Klein et al., 2017 Baker et al.] (017). A more recent line of work in performance prediction focuses more on feature encoding of neural archi- tectures. Peephole (Deng al., 2017) and TAPAS (Istrate al., 2019) both use an LSTM to aggregate information about the operations chain-structured architectures. On the other hand, BANANAS (White al.) 2019) introduces path-based encoding of cells that automatically solves the computational equivalence of architectures.
Graph Neural Networks (GNNs) (Gori al., 2005; Kipf & Welling 2017 Zhou al.
2018
Wu
et al. 2019) with their capability of learning representations of graph-structured data appear to be a
natural choice to learning embeddings of NN architectures. [Shi et al. (2019) and Wen et al. (2019)
trained a Graph Convolutional Network (GCN) on a subset of NAS- Bench- 101 (Ying et al. 2019)
showing its effectiveness in predicting the performance of unseen architectures. Moreover, Friede
et al. (2019) propose a new variational-sequee sequential graph autoencoder (VS-GAE) which utilizes
GNN encoder-decoder model the space of architectures and generates valid graphs in the learned
latent space.
Several recent works further adapt the GNN message passing to embed architecture bias via extra weights to simulate the operations such as in GATES (Ning et al. 2020) integrate additional information on the operations (e-g. flop count) (Xu et al.] 2019b). Tang et al.) (2020) chose to operate GNNs on relation graphs based architecture embeddings in a metric learning setting,
allowing to pose NAS performance prediction as a semi-supervised setting.
B TRAINING DETAILS FOR THE GIN IN THE MOTIVATION
We set the GIN have hidden dimension of 64 with 4 hidden layers resulting in around ~40k parameters. We trained for 30 epochs with a batch size of 128. We chose the MSE loss function and add logarithmic transformation to emphasize the data fit well-performing architectures.
14

---

Under review as a conference paper at ICLR 2021
Model
Mean Squared Error (MSE)
Kendall tau
1, [2,
2, [1. 3]
3, [1, 2]
1, [2, 3]
2, [1, 3]
3. [1, 2]
Tab.
5,44e-5
5,43e-5
5.34e-5
0.83
0.83
0.83
Surr.
3.02e-5
3.07e-5
3.02e-5
0.87
0.87
0.87
Table 6: MSE and Kendall tau correlation between performance predicted by tab./sur.. benchmark fitted with one seed each, and the true performance of evaluations with the two other seeds (see Section 2
Test seeds in brackets.
C NAS-BENCH-301 DATASET
C.I SEARCH SPACE
We use the same architecture search space as in DARTS (Liu et al. 2019b). Specifically, the normal and reduction cell each consist of DAG with input nodes (receiving the output feature maps from the previous and previous-previous cell), 4 intermediate nodes (each adding element-wise feature maps from two previous nodes in the cell) and I output node (concatenating the outputs of all intermediate nodes). Input and intermediate nodes are connected by directed edges representing one of the following operations: Sep. conv 3 3, Sep. conv 5 x 5, Dil. conv 3, Dil. conv 5 x 5, Max pooling 3 x 3, Avg. pooling 3 x 3, Skip connection.
C.2 DATA COLLECTION
To achieve good global coverage, we use random search evaluate ~233 architectures. We note that space-filling designs such as quasi-randon sequences, e.g. Sobol sequences "Sobol"|[16) [1667) or Latin Hypercubes (McKay al. 2000) and Adaptive Submodularity (Golovin & Krause 2011 may also provide good initial coverage.
Random search is supplemented by data which we collect from running variety of optimizers, rep- resenting Bayesian Optimization (BO), evolutionary algorithms and One-Shot Optimizers. We used Tree-of-Parzen- Esstimatrss (TPE) (Bergstra et al. 2011) as implemented by Falkner et al. (2018) as bascline BO method. Since several recent works have proposed to apply BO over combinatorial spaces (Oh et al.) 2019 [Baptista & Poloczek 2018) we also used COMBO (Oh et al. 2019). We included BANANAS (White al.] 2019)
third BO method, which uses neural network with a path-based encoding as surrogate model and hence scales better with the number of func- tion evaluations. As representatives of evolutionary approaches to NAS, we chose Regularized Evolution (RE) (Real et al. 2019) as it is still one of the state-of-the art methods in discrete NAS and Differential Evolution (Price al. 2006) as implemented by Awad et al. (2020). Accounting for the surge in interest in One-Shot NAS, our collected data collection also entails evaluation of architectures from search trajectories of DARTS (Liu al. 2019b), GDAS (Dong Yang 2019), DrNAS (Chen et al. (020) and PC-DARTS (Xu et al. 2020 For details on the architecture training details, we refer to Section C.6
For each architecture E A, the dataset contains the following metrics: train/validationntes accu- racy, training time and number of model parameters.
C.3 DETAILS ON EACH OPTIMIZER
In this section we provide the hyperparameters used for the evaluations of NAS optimizers for the collection our dataset. Many of the optimizers require a specialized representation to function on architecture space because most of them are general HPO optimizers. As recently shown by White et al. 2020a). this representation can be critical for the performance of NAS optimizer. Whenever the representation used by the Optimizer did not act directly on the graph representation, such as in RE, we detail how we represented the architecture for the optimizer. All optimizers were set to optimize the validation error.
15

---

Under review as a conference paper at ICLR 2021
BANANAS, We initialized BANANAS with 100 random architectures and modified the opti- mization of the surrogate model neural network, by adding early stopping based on a 90%/10% train/validation split and lowering the number of ensemble models to be trained from 5 to 3. These changes to bananas avoided a computational bottleneck in the training of the neural network.
COMBO. COMBO only attempts to maximize the acquisition function after the entire initial de- sign (100 architectures) has completed. For workers which are done earlier, we sample random architecture, hence increasing the initial design by the number of workers (30) we used for running the experiments. The search space considered in our work is larger than all search spaces evaluated in COMBO (Oh et al. 2019) and we regard not simply binary architectural choices, as we have to make choices about pairs of edges. Hence, we increased the number of initial samples for ascent acquisition function optimization from 20 to 30. Unfortunately, the optimization of the GP already became the bottleneck of the BO after around 600 function evaluations, leading to many workers waiting for new jobs to be assigned.
Representation: contrast to the COMBO's original experimental setting, the DARTS search re- quires choices based on pairs of parents of intermediate nodes where the number of choices increase with the index the intermediate nodes. The COMBO representation therefore consists the graph cartesian product of the combinatorial choice graphs, increasing in size with each intermediate node. In addition, there exist 8 choices over the number of parameters for the operation in cell.
Differential Evolution. DE was started with generation size of 100. As we used a parallelized implementation, the workers would have to wait for one generation plus its mutations be com- pleted for selection to start. We decided to keep the workers busy by training randomly sampled architectures in this case, as random architectures provide us good coverage of the space. However, different methods using asynchronous DE selection would also be possible. Note, that the DE im- plementation by Awad et al. (2020), performs boundary checks and resamples components of any individual that exceeds 1.0. We use the rand1 mutation operation which generally favors exploration over exploitation.
Representation: DE uses vector representation for each individual in the population. Categorical choices are scaled to lie within the unit interval [O, 1] and are rounded to the nearest category when converting back to the discrete representation in the implementation by Awad et al. (2020). Similarly to COMBO, we represent the increasing number of parent pair choices for the intermediate nodes by interpreting the respective entries to have an increasing number of sub-intervals in [0. 1].
DARTS, GDAS, PC-DARTS and DrNAS. We collected the architectures found by all of the above one-shot optimizers with their default search hyperparameters. We performed multi- ple searches for each one-shot optimizer.
1.00
Cumulative prob.
0.75
BANANAS
COMBO
0.50
DE
DRNAS
GDAS
RE. To allow for good initial coverage be- fore mutations start, we decided to randomly
0.25
DARTS
RE
RS
sample 3000 architectures as initial population.
0.00
RE then proceeds with sample size of 100 to extract well performing architectures from the population and mutates them. During mu- tations RE first decides whether to mutate the normal or reduction cell and then proceeds to perform either a parent change, an operation change mutation.
0.05
0.1
0.2
Validation error
Figure Empirical Cumulative Density Func- tion (ECDF) plot comparing all optimizers in the dataset. Optimizers which cover good regions of the search space feature higher values in the low validation error region.
TPE. For TPE we the default settings also used by BOHB. We the Kernel-Densite Estimator surrogate model and build two models where the good configs are chosen as the top 15%. The acquisition function's expected improvement is optimized by sampling 64 points.
16

---

Under review as a conference paper at ICLR 2021
C.4 OPTIMIZER PERFORMANCE
The trajectories from the different NAS optimizers yield quite different performance distributions. This can be seen in Figure X which shows the ECDF of the validation errors of the architectures evaluated by each optimizer. As the computational budgets allocated to each optimizer vary widely, this data does not allow for a fair comparison between the optimizers. However, it is worth men- tioning that the evaluations of BANANAS feature the best distribution of architecture performances, followed by PC-DARTS, DrNAS, DE, GDAS, and RE, TPE only evaluated marginally better archi- tectures than RS, while COMBO and DARTS evaluated the worst architectures.
We also perform a t-SNE analysis on the data collected by the different optimizers in Figure We find that RE discovers well-performing architectures which form clusters distinct from the architec- tures found via RS. We observe that COMBO searched previously unexplored areas of the search space. BANANAS, which found some of the best architectures, explores clusters outside the main cluster. However, it heavily exploits regions at the cost of exploration. We argue that this is a re- sult of the optimization of the acquisition function via random mutations based on the previously found iterates, rather than on new random architectures. DE is the only optimizer which finds well performing architectures in the center of the embedding space.
C.5 CELL TOPOLOGY, OPERATIONS AND NOISE
In this section, we investigate the influence of the cell topology and the operations on the perfor- mance of the architectures in our setting. The discovered properties of the search space inform choice of metrics for the evaluation of different surrogate models.
First, we study how the validation error depends the depth of architectures. Figure 10 visualizes the performance distribution of normal and reduction cells of different depth by approximating empirical distributions with kernel density estimation used in violin plots (Hwang et al. 1994). We observe that the performance distributions are similar for the normal and reduction cells with the same cell depth. Although cells of all depths can reach high performances, shallower cells seem slightly favored. Note that these observations are subject to changes in the hyperparameter setting, e.g. training for more epochs may render deeper cells more competitive. The best-found architecture features a normal and reduction cell of depth 4. Color-coding the cell depth in our 1-SNE projection also confirms that the t-SNE analysis captures the cell depth well as structural property (c.f. Figure 13). It also reinforces that the search space is well-covered.
We also show the distribution of normal and reduction cell depths of each optimizer in Figure to get sense for the diversity between the discovered architectures. We observe that DARTS and BANANAS generally find architectures with a shallow reduction cell and a deeper normal cell, while the reverse is true for RE. DE, TPE, COMBO and RS appear to find normal and reduction cells with similar cell depth.
Aside from the cell topology, can also use dataset to study the influence of operations to the ar- chitecture performance. The DARTS search space contains operation choices without parameters such as Skip-Connection, Max Pooling 3 x 3 and Avg
2
Validation accuracy
0.95
Pooling 3 x 3. We visualize the influence of these
parameter-free operations on the validation error in the normal and reduction cell in Figure Sal respec- tively Figure [4 While pooling operations in the normal cell seem to have a negative impact on per- formance, a small number of skip connections im- proves the overall performance. This is somewhat expected, since the normal cell is dimension preserv- ing and skip connections help training by improving gradient flow like in ResNets (He et al. 2016)). In the reduction cell, the number of parameter free In contrast to the normal cell where 2-3 skip-
-1
0.9
0.002
0.004
Standard deviation
Figure 12: Standard deviation of the val. ac- curacy for multiple architecture evaluations.
operations has less effect as shown in Figure 14
We follow the definition of cell depth used by Shu al. 2020), i.c. the length of the longest simple path through the cell.
17

---

Under review as a conference paper at ICLR 2021
connections lead to generally better performance, the reduction cell shows similar trend. For both cells, however, featuring many parameter-Fe free operations significantly deteriorates performance. We therefore expect that good surrogate also models this case poorly performing region.
C.6 TRAINING DETAILS
Each architecture was evaluated on CIFAR-10 (Krizhevsky 2009) using the standard 40k, 10k, 10k split for train, validation and test set. The networks were trained using SGD with momentum 0.9, initial learning rate of 0.025 and cosine annealing schedule (Loshchilov & Hutter 2017). annealing towards 10-
We apply a variety of common data augmentation techniques which differs from previous NAS benchmarks where the training accuracy of many evaluated architectures reached 100% (Ying al.) 2019 Dong & Yang, 2020) indicating overfitting the training set. We used CutOut (DeVsies & Taylor (017)) with cutout length 16 and MixUp (Zhang et al. 2018) with alpha 0.2. For regulariza-
tion, we used auxiliary tower (Szegedy al. 2015) with weight of 0.4 and DropPath (Larsson et al. 2017) with drop probability 0.2. We trained each architecture for 100 epochs with a batch
BANANAS
COMBO
DARTS
50000
40
40000
DE
DRNAS
GDAS
-20
30000
Architecture Ranking
-40
PC DARTS
RE
RS
20000
-20
440
TPE
10000
-20
-40
Figure 9: Visualization of the exploration of different parts of the architectural t-SNE embedding space for all optimizers used for data collection. The architecture ranking by validation accuracy (lower is better) is global over the entire data collection of all optimizers.
18

---

Under review as a conference paper at ICLR 2021
Validation error
normal reduction
Cell Depth
0.2
4
0.1
DE
GDAS
DARTS
COMBO
PC PC_DARTS
TPE
DRNAS RE
BANANAS RS
0.05
3
4
6
Cell depth
Figure 10: Distribution of the validation error for different cell depth.
Figure 11: Comparison between the normal and reduction cell depth for the architectures found by each optimizer.
50
Normal Cell Depth
Validation error
0.2
Num avg pool
2nd component
Num max pool
25
Num skip conn.
0.1
-25
#
-50
0.05
-50
0
50
0
1
3 4 5 6 7 8 Num operations
2
1st component
Figure 13: t-SNE projection colored by the depth of the normal cell.
Figure 14: Distribution of validation error in de- pendence of the number of parameter-free opera- tions in the reduction cell. Violin plots are cut off at the respective observed minimum and maxi- mum value.
size of 96, using 32 initial channels and cell layers. We chose these values be close the proxy model used by DARTS while also achieving good performance.
D SURROGATE MODEL ANALYSIS
D.I PREPROCESSING OF THE GRAPHI TOPOLOGY
DGN preprocessing All DGN were implemented using PyTorch Geometrie (Fey & Lenssen 2019 which supports the aggregation of edge attributes. Hence, we can naturally represent the DARTS architecture cells, by assigning the embedded operations the edges. The nodes labeled as input, intermediate and output nodes. We represent the DARTS graph as shown in Figure [5 by connecting the output node of each cell type with the inputs of the other cell, allowing information from both cells to be aggregated per node during message passing. Note the self- on the output node of the normal cell, which found necessary to get the best performance.
Preprocessing for other surrogate models Since make use of the framework implemented by BOHB (Falkner et al. 2018) to easily parallelize the architecture search algorithms across many compute nodes, also represent More precisely, we encode each pair of incoming edges for cell as one choice of a categorical pa- rameter. For instance, for node 4 in the normal cell, add parameter i inputs node with the choices of edge pairs 0_1, 0_2, 0_3, 1_2, 1_3, 2_3. The edge operations are then imple- mented as categorical parameters for each edge and are only active if the corresponding edge was
search space using ConfigSpace LLndauerr et al. 2019)
1.4
httpps:/ http:///gtttubccoo wwwp//uutmmmmmmmmmm /Confmm/CCooffi gSpace
19

---

Under review as a conference paper at ICLR 2021
chosen. For instance, in the example above, if the incoming edge 0 is sampled, the parameter sociated with the edge from node to node 4 becomes activate and one operation is sampled. We provide the configuration space with code. For all non-DGN based surrogate models, the vector representation of a configuration given by ConfigSpace as input to the model. This vector representation contains value between and for each parameter in the configuration space.
D.2 DETAILS ON THE GIN
The GIN implementation on the Open Graph Benchmark (OGB) (Hu et al.) 2020) uses virtual nodes (additional nodes which are connected to all nodes the graph) to boost performance as well generalization and consis- tently achieves good performance on their public leader- boards. Other GNNs from Erricaa et al. (2220), such DGCNN and DifiPool, performed worse in our initial ex- periments and therefore not considered.
Normal
Following recent work in Predictor-basee based NAS (Ning per batch rank-
et al. 2020 Xu al., 2019b), ing loss because the ranking of an architecture is equally important an accurate prediction of the validation ac- curacy in NAS setting. We use the ranking loss for- mulation by GATES (Ning al., (2020) which hinge pair-wise ranking loss with margin m=0.1.
Reduction
Figure 15: Architecture with inputs in green, intermediate nodes in blue and outputs in red.
D.3 DETAILS ON HPO
All detailed table for the hyperparameter ranges for the HPO and the best values found by BOHB are listed in Table 7
D.4 HPO FOR RUNTIME PREDICTION MODEL
Our runtime prediction model is an LGB model trained on the runtimes of architecture evaluations of DE. This because partially evaluated the architectures utilizing different CPUs. Hence, we only choose to train on the evaluations carried out by the same optimizer on the same hardware to keep a consistent estimate of the runtime. DE is good choice in this case because it both explored and exploited the architecture space well. The HPO space used for the LGB runtime model is the same used the L.GB surrogate model.
D.5 LEAVE ONE-OPTIMIZER-OO OUT ANALYSIS
detailed scatter plot of the predicted performance against the true performance for each optimizer and surrogate model in LOO0 analysis is provided in Figure 16 and Figure 17
D.6 PARAMETER FREE OPERATIONS
Several works have found that methods based on DARTS (Liu et al. 2019b are prone to finding sub-optimal architectures that contain many, even only, parameter free operations (max. pooling, avg. pooling or skip connections) and perform poorly (Zela al. (020a). We therefore evaluated the surrogate models on such architectures by replacing a random selection of operations in a cell with one type parameter-free operations to match a certain ratio parameter-free operations in cell. This analysis is carried out over the test set of the surrogate models and hence contains architectures collected by all optimizers. For a more robust analysis, we repeated this experiment times for each ratio of operations to replace.
Results Figure 18 shows that both the GIN and the XGB model correctly predict that the accuracy drops with too many parameter-fie free operations, particularly for skip connections. The groundtruth of architectures with only parameter-free operations is displayed scatter plot. Out of the models,
20

---

Under review as a conference paper at ICLR 2021
GIN
LGB
XGB
95
DARTS True
90
85
95
BANANAS
True
90
85
95
COMBO
True
90
85
95
DE
True
90
85
95
RE
True
90
85
85
90
95
85
90
95
85
90
95
Predicted
Predicted
Predicted
Figure 16:
Scatter plots of the predicted performance against the true performance of different surrogate models on the test set in Leave-One- Leave-One-ppimizzr-- Out setting.
21

---

Under review as a conference paper at ICLR 2021
GIN
LGB
XGB
95
TPE
True
90
85
95
PC DARTS
True
90
85
95
GDAS
True
90
85
95
DRNAS
True
90
85
85
90
95
85
90 Predicted
95
85
90 Predicted
95
Predicted
Figure 17: (continued) Scatter plots of the predicted performance against the true performance of different surrogate models on the test set in Leave-Oee-Opimizer--0 setting.
22

---

Under review as a conference paper at ICLR 2021
Default
Model
Hyperparameter
Range
Log-transform
Value
Hidden dim.
[16.
true
24
Num. Layers
10]
false
Dropout Prob.
[0. I]
falss
0.035
Learning rute
[1e-3, 1e-2]
truc
Learnng min.
0.0
GIN
Batch size
const.
Undizected graph
[uue,
false
Pairwise ranking loss Self-Loops
true, [alse]
true
[iue,
Loss log transform
[tuee [alse]
true
Node degree one hot
const.
true
Num. Layers
10]
Layer width
[16, 256]
trie
31
Dropout Prob.
const.
BANANAS
I Learning Learning rate min.
[ee33 le-1]
0,0021
const.
0.0
Batch
128]
122
Loss log transform
true, [alse]
true
Pairwise ranking loss
[tree false]
false
Early Stopping
Rounds
const.
100
Booster
gbtree
Max. depth
[1. 15]
false
13
Min. cluld weight
[1. 100)
true
XGBoost
Col. sample bylevel
[0.0. 1.0]
false
0.6909
Col. sample bytree lambda
[0.0. 1.0]
false
0,2545
[0.001. 1000]
31.3933
alpha
[0.001, 1000]
tree
0,2417
Learning rute
[0.001.0.] 0.1]
true
0.00824
Early stop. rounds
100
Max. depth
[1.
false
Num. leaves
[10. 100]
false
Max. bin
[100.400]
false
336
Feature Fraction
[0.1. 1.0]
false
0.1532
LGBoost
Min. child weight
[0.001, 10]
0,5822
I Lambda 1.1 Lambda L2
[0.001. 1000]
trie
0.01
[0.001. 1000]
134.5075
Boosting type
const.
ghdr
Learning rate
[0.001, 0.1]
true
0.0218
Num. estimators
[16, 128]
116
Min. samples split.
[2, 20]
false
2
Random Forest
samples
[1, 20]
false
Max. features
[0.1. L.0]
false
0.1706
Bootstrap
[uree false]
false
[1.0. 20.0]
3.066
coef.
[-0.5, 0.5]
false
0.1627
degree
[1. 128]
true
c-VVR
epsilon
[0.01, 0.99] [scalle, auto]
true
0.0251
gamma
auto
kernel
[linear, rhf, poly, sigmoid]
sigmoid
shrinking
[tue, false]
false
[0.0001, 0.01]
[1.0, 20.0]
5,3131
coef.
[-0.5,0.5] 128]
false
-0.3316
degree
128
J-SVR
gamma
[scale, auto]
scale
kernel
[lincar, rbf, poly, sigmoid]
[0.01. 1.0]
false
0.1839
shrinking
[tree, false]
true
[0.0001, 0.01]
0.003
Table 7: Hyperparameters of the surrogate models and the default values found via HPO.
XGB captures the slight performance improvement of using few skip connections better. LGB failed to capture this trend but performed very similarly XGB for the high number parameter- free operations.
23

---

Under review as a conference paper at ICLR 2021
0.2
Num avg pool
Validation arror
0.2
0.2
pool
Num
Num. max pool
Num. max pool
Num skip conn.
slip
Num.
Validation
Validation
0.1
0.1
0.1
0.05
0.05
0.05
5
0
1
5 (c) XGB
3
Num operations
operations
Num. operations
NAS-Bench-301 dataset
(b) GIN
Figure 18: (Left) Distribution of validation error in dependence of the number of parameter free operations in the normal cell on the NAS-Bench-301 dataset. (Middle and Right) Predictions of the GIN and XGB surrogate model. The collected groundtruth data shown as seatter plot. Violin plots are cut off at the respective observed minimum and maximum value.
D.7 CELL TOPOLOGY ANALYSIS
Furthermore, we analyze how accurate changes in the cell topology (rather than in the operations) are modeled by the surrogates. We collected groundtruth data by eval-
SKendall Tau
0.80
(k++1jk = 180 different cell topologies (not accounting for isomorphisms) with fixed sets of op- erations. We assigned the same architecture to the normal
uating all IT 2
0.75
GIN
LGB
0.70
and reduction cell, focus the effect of the cell topol- ogy. We sampled 10 operation sets uniformly at random, leading to 1800 architectures groundtruth for this anal- ysis.
XGB
3
4
5
6
Cell Depth
We evaluated all architectures and group the results based on the cell depth. For each of the possible cell depths, we then computed the sparse Kendall rank correlation between the predicted and true validation accuracy.
Figure 19: Comparison between GIN,
XGB and LGB in the cell topology anal- ysis.
Results Results of the cell topology analysis are shown in Figure 19 We observe that LGB slightly outperforms XGB, both of which perform better deeper cells. The GIN performs best for the shallowest cells.
All data
LOTO
LODO
0.10
0.30
0.09
GIN
0.09
0.09
Valclelen e3t1
0.08
0.08
0.08
0.06
0.06
0.06
0.05
0.05
0.05
10
40
10
20
40
Epochs
Epochs
Epochs
Figure 20: Ground truth (GT) and surrogate trajectories constrained search space where the surrogates are trained with all data, leaving out the trajectories under consideration (LOTO), and leaving out all DARTS architectures (LO00).
24

---

Under review as a conference paper at ICLR 2021
BANANAS
RS
Best validation error accieved
9 x
True
x
GIN
XGB
x 10-
6 10
10
10
10a
10
107
10*
10
10
10'
Wallclock Time
Simulated Wallclock [s]
Figure 22: Comparison between the observed true trajectory of BANANAS and RS with the surro- gate benchmarks only trained on well performing regions of the space
E BENCHMARK ANALYSIS
E ONE-SHOT TRAIECTORIES
To obtain groundtruth trajectories for DARTS, PC-DARTS and GDAS, we performed runs for each optimizer with 50 search epochs and evaluated the architecture obtained by discretizing the one-shot model at each search epoch. For DARTS, in addition to the default search space, we collected trajectories on the constrained search spaces from Zela et al. (2020a) to cover failure case where DARTS diverges and finds architectures that only contain skip connections in the normal cell. To show that our benchmark able to predict this divergent behavior, we show surrogate trajectories when training on all data, when leaving out the trajectories under consideration from the training data, and when leaving out all DARTS data in Figure 20
While the surrogates model the divergence in all cases, they still overpredict the architectures with only skip connections the normal cell especially when leaving out data from DARTS. The bad performance of these architectures is predicted more accurately when including data from other DARTS runs. This can be attributed to the fact that the surrogate models have not seen any, re- spectively very few data, in this region of the search space. Nevertheless, it is modeled a bad- performing region and we expect that this could be further improved on by including additional training data uccordingly, since including all data training shows that the models are capable of capturing this behavior.
E.2 ABLATION STUDY: FITTING SURROGATE MODELS ONLY ON TELL-PERFORMING REGIONS OF THE SEARCH SPACE
To assess whether noorly-performin architectures are important for the surrogate benchmark, we fitted GIN ensemble and XGB ensemble model only architec- tures that achieved a validation accuracy above 92%. We then tested on all architectures that achieved validation below 92%.
95
90
True
85
Indeed, observe that the resulting surrogate model overpredicts accuracy in regions of the space with poor performance, resulting in a low R² of -0.142 and sparse Kendall tau of 0.293 for the GIN. The results for one member of the GIN ensemble are shown in Figure [1] The XGB model achieved similar results. Next, to study whether these weaker surrogate models can still used to benchmark NAS optimizers, we also studied optimiza- tion trajectories of NAS optimizers on surrogate bench- marks based on these surrogate models. Figure 2 shows that these surrogate models indeed suffice to accurately predict the performance achieved by Random Search and BANANAS as a function of time.
Predicted
Figure 21: Scatter plot of GIN predic- tions on architectures that achieved be- low 92% validation accuracy.
25

---

Under review as a conference paper at ICLR 2021
E.3 ABLATION STUDY: FITTING SURROGATE MODELS ONLY WITH RANDOM DATA
In this section, we would like to take the Laa--One-Opiizze---ee analysis from Section 5.1 one step further by leaving out all architectures that were collected from NAS optimizers other than ran- dom search. While the LOO0 analysis removes some "bias" from the benchmark ("bias" referring to its precision similar subspaces, and leaving out one of them still yields "bias" induced by architectures from similar optimizer used for generating training data. For instance, the t-SNE analysis from Figure suggests that some optimizers exploit very distinct regions (e.g.. BANANAS and DE) while others exploit regions somewhat similar to others (e.g., RE and PC-DARTS). The exploration behavior, on the other hand, is quite similar across optimizers since most of them perform random sampling in the beginning. Thus, in the following, we investigate whether we can create a benchmark that has no prior information about solutions any optimizer might find.
subspace), there still is the possibility that different optimizers we used explore
To that end, we studied surrogate models based i) only on the 23746 architectures explored by random search and ii) only on 23 746 (47.3%) architectures of the original training (sampled in a stratified manner, i.e., using 47.3% of the architectures from each of our sources of architectures).
First, we investigated the difference in the predictive performance of surrogates based these two different types of architectures. Specifically, we fitted our GNN and XGB surrogate models on dif- ferent subsets of the respective training sets and assess their predictions on unseen architectures from all optimziers as a test set. Figure 23 shows that including architectures from optimizer trajectories in the training set consistently yields significantly better generalization.
Next, we also studied the usefulness of surrogate benchmarks based on the 23 746 random architec- tures, compared surrogate benchmarks based on the 23 746 architectures sampled in stratified manner from the original set of architectures. Specifically, we used them to assess the best perfor- mance achieved by various NAS optimizers as a function of time. Comparing the trajectories in Figure [4 (based on purely random architectures for training) and Figure 25 (based on 23 746 archi- tectures sampled in a stratified manner), we find that the surrogates fitted only on random architec- tures work just as well for this task as the surrogates that use architectures from NAS optimizers in their training set.
Given this positive result for surrogates based purely random architectures, we conclude that is indeed possible to create surrogate NAS benchmarks that are by design free of bias towards any particular NAS optimizer (other than random search). While the inclusion of architectures generated with NAS optimizers in the training set substantially improves performance predictions of individual architectures, realistic trajectories of incumbent performance as a function of time can also be obtained with surrogate benchmarks based solely on random architectures. We note that the "unbiased" benchmark could possibly be further improved by utilizing more sophisticated space- filling sampling methods, such as the ones mentioned in Appendix C.2 or by deploying surrogate models that extrapolate well.
F GUIDELINES FOR CREATING SURROGATE BENCHMARKS
In order to help with the design of realistic surrogate benchmarks in the future, we provide the following list of guidelines:
Data Collection: The data collected for the NAS benchmark should provide (1) good overall coverage, (2) explore strong regions of the space well, and (3) optimally also cover special areas in which poor generalization performance may otherwise be expected. We would like to stress that depending on the search space, a good overall coverage may already be sufficient to correctly assess the ranking of different optimizers, but as shown in Appendix E.3 additional architectures from strong regions of the space allow to increase the fidelity of the surrogate model.
1. A good overall coverage can be obtained by random search (as in our case), but one could also imagine using better space-filling designs or adaptive methods for covering the space even better. In order to add additional varied architectures, one could also think about fitting or more surrogate models to the data collected thus far, finding the regions of maximal predicted uncertainty, evaluate architectures there and add them to the collected data, and iterate. This would constitute an active learning approach.
26

---

Under review as a conference paper at ICLR 2021
GNN GIN RS
XGB RS
GNN GIN mixed
XGB mixed
95
R 0.57
R² = 0.61 T 0.67
R = 0.73 T= T=0.72
R² 0.65 T= 0.67
Train ratio 5%
94
== 0.67
True
92
91
Train ratio 20%
R = 0.73 0.72
R 0067
R² =081
R² = 0083 T= 0.77
94
0.722
0.76
True
93
91
95
Train ratio 60%
R² = 0.75
R² = 0.71 T 0.75
R 0.84 T= 0.79
R² = 0.87
== 0.75
T= 0.82
True
93
92
95
Train ratio 80%
R = 0.73
R 0.72 T 0.76
R 0.86 T= 0.79
R 0.90 T= 0.85
94
T= 0.75
True
92
91
Train ratio 100%
R = 0.75
R² = 0.73
R² = 0.87
R² 0.90 T= 0.85
94
T 0.75
T= 0.76
T 0.80
True
93
92
91
92 94
92
94
94 92 94
Predicted
Predicted
Predicted
Predicted
Figure 23: Scatter plots of the predicted performance against the true performance of the GNN GIN/XGB surrogate models trained with different ratios of training data. "RS" indicates that the training set only includes architectures from random search, "mixed" indicates the training set in- cludes architectures from all optimizers. Training set sizes are identical for the two cases. The test set contains architectures from all optimizers. For better display, we show 1000 randomly sampled architectures (blue) and 1000 architectures sampled from the top 1000 architectures (orange). For each case also show the R.2 and Kendall-T coefficients the whole test set.
27

---

Under review as a conference paper at ICLR 2021
True Benchmark
GIN Surrogate Benchmark
XGB Surrogate Benchmark
validation achieved
x 10
DE
8
RE
TPE
1
BANANAS
RS
6 10
50-1
10a
10
10
107
10a
10
10"
10a
10"
10'
Wallclock Time [s]
Simulated Wallclock Time
Simulated Wallclock Time
Figure 24: Anytime performance of different optimizers the real benchmark (left) and the surro- gate benchmark (GIN (middle) and XGB (right)) when training ensembles only on data collected by random search. Trajectories the surrogate benchmark averaged over 5 optimizer runs.
True Benchmark
GIN Surrogate Benchmark
XGB Surrogate Benchmark
Best validation achieved
10
DE
RE
TPE
x -0
BANANAS
6
S 10
10a
10
207
104
10a
10f 107
Wallclock Time [s]
Simulated Wallclock Time [s]
Simulated Wallclock Time [s]
Figure 25: Anytime performance of different optimizers on the real benchmark (left) and the sur- rogate benchmark (GIN (middle) and XGB (right)) when training ensembles on 47.3% of the data collected from all optimizers. Trajectories on the surrogate benchmark are averaged over 5 optimizer runs.
2. A convenient and efficient way to identify regions of strong architectures to run NAS methods. In this case, the found regions should not only be based on the strong architectures one NAS method finds but rather on set of strong and varied NAS methods (such as, in our case, one-shot methods and different types of discrete methods, such as Bayesian optimization and evolution). In order to add additional strong architectures, one could also think about fitting one or more several surrogate models the data collected thus far, finding the predicted optima of these models, evaluate and add them to the collected data and iterate. This would constitute special type Bayesian optimization.
3. Special areas in which poor generalization performance may otherwise be expected may, as in our case, e.g., include architectures with many parameterless connections, and in particu- lar, skip connections. Other types of failure modes the community learns about would also be useful to cover.
mentioned in the guidelines for using a surrogate benchmark (see Sec- tion benchmarking algorithm that internally uses the same model type as the surrogate model should be avoided. Therefore, to provide a benchmark for diverse set of algorithms, we recommend providing different types of surrogate models with a surrogate benchmark. Also, in order to guard against a possible case of "bias" in a surrogate benchmark (in the sense making more accurate predictions for architectures explored by particular type of NAS optimizer), recommend to provide two versions of a surrogate: one based on all available training architec- tures (including those found by NAS optimizers), and one based only the data gathered for overall coverage (1. above).
Surrogate Models:
Verification: As means to verify surrogate models, stress the importance of leave-one- optimizer-out experiments both for data fit and benchmarking, which simulate the benchmarking of "unseen" optimizers.
28

---

Under review as a conference paper at ICLR 2021
Since most surrogate benchmarks will continue to grow for some time after their first release, to allow apples-to-apples comparisons, we strongly encourage to only release surrogate benchmarks with a version number.
In order to allow the evaluation of multi-objective NAS methods, we encourage the logging of as many relevant metries of the evaluated architectures other than accuracy as possible, including training time, number of parameters, and multiply-adds.
Alongside released surrogate benchmark, we strongly encourage to release the training data its surrogate(s) were constructed on, as well the test data used to validate it.
In order facilitate checking hypotheses gained using the surrogate benchmarks in real experi- ments, the complete source code for training the architectures should be open-sourced alongside the repository, allowing to easily go back and forth between querying the model and gathering new data.
29


Reviews:
[{"review_id": "u0GKAe1JC6", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors train a model on a subset of architectures (~60k) in the DARTS search space and use this model to predict the performance of architectures outside of that subset in DARTS. They then use this as a surrogate for having to perform network training for evaluating NAS algorithms.\n\nNAS benchmarks are a good thing, given that they facilitate NAS research outside of labs with lots of resources. Saying this, I think there are serious issues with this paper.\n\nNAS-Bench-301 is a model that predicts the performance of networks in the DARTS search space. We know from https://arxiv.org/abs/1912.12522v3 that there is a significant lack of variety in the performance of models within this space, and this work compounds that. Additionally, this paper only considers CIFAR-10. This is a step back from NAS-Bench-201 which despite its small size, did contain multiple datasets. The compute used by the authors (training 60k networks) has gone into differentiating a bunch of networks that are all quite good, within a few percentage points of error. I’m not sure how this is of practical use.\n\nA problem with NAS-Bench-101 and 201 is that the search spaces are small (423k, 6k) as the authors point out. NAS-Bench-301 encapsulates the DARTS search space which is much bigger (10^18). However, from https://arxiv.org/abs/1912.12522v3 we see that randomly sampling within this space gives networks between 96.5% and 97.5%. I would argue that it doesn’t matter how large a space is if it is lacking in variety; Every possible network works well enough. In 101/201 we see networks across a much larger range (typically between 80-95% although there are some much lower). I believe this is more interesting from a research perspective as we would like to apply NAS to situations where networks can break (and avoid this happening). The DARTS networks do explore a slightly higher accuracy range but it is not at state-of-art levels, so the added value is not clear.\n\nThe authors are critical of random search, stating (i) `random search stagnates and cannot identify one of the best architectures even after tens of thousands of evaluations`, (ii) NAS-Bench-201 (Dong & Yang, 2020) only contains 6466 unique architectures in total, causing the median of random search runs to find the best architecture after only 3233 evaluations.  However, (i) makes random search sound like it is failing, where on Figure 4 we can see at 10^4 (s) random search is doing very well  - matching or beating all the other techniques.  We know from https://arxiv.org/abs/1806.09055  that random search works well on the DARTS  space (probably due to the lack of variety). (ii) Random search does work well on NAS-Bench-201 but the DARTS algorithm fails, even though it works on this space. The statement in the abstract that using previous benchmarks  “leads to unrealistic results that do not transfer to larger search spaces.” does not tell the whole story, as there appears to be more going on than just the size of the search space. Some results on this large DARTS search space, do not transfer to the smaller search spaces. It seems that there is more to a search space than the number of architectures within it.\n\nIn terms of presentation, this paper is well written, although the figures could be larger. I appreciate that this is problem to keep within the page limit.\n\nNAS benchmark papers are important and shape the direction of research in the field. This paper doesn’t present a benchmark. It provides a model that represents computationally-efficient means of getting network accuracies from the DARTS search space. I appreciate this endeavour and it could be of use outside of this space — being able to map a 10^18 space with 50000 points indicates that the effective size of the space is much smaller, and is highly predictable.\n\nThis space, although large, has very little variety in terms of network accuracy. I believe NAS-Bench-101 and 201 despite their sizes represent more varied search spaces. 201 also covers multiple datasets, whereas 301 only has CIFAR-10 (which it feels like we are saturating on).  I recommend rejection for this paper, as I do not believe it represents a step forward in the way we benchmark our NAS algorithms. We need to develop more interesting search spaces,  rather than advocating exploration of uninteresting ones.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "kPw3Od9dRf3", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "****Update after rebuttal ****\nI am increasing my rating for the paper as they did all the experiments I had asked for and updated the paper accordingly.\n**************************\n\nSummary:\n\nWhile NAS has made tremendous advances in recent past, benchmarking algorithms with respect to each other still remains a challenge. Tabular benchmarks like 101 and 201, take a search space and train all possible architectures in them. While this is possible to do for relatively small search spaces and datasets, this is impractical to repeat for larger search spaces (e.g. DARTS' search space which has 10^18 architectures). \n\nThis work makes the nice observation that a tabular benchmark treats each architecture as an independent random variable and doesn't utilize any similarities between them. Due to similarities in architecture space, knowing the train/val/test accuracy of one architecture tells us a lot about other architectures nearby. So a predictive model trained on a sparse subset of architectures can actually outperform an exhaustive tabular method.  \n\nLots of careful experiments are reported on the DARTS search space to create predictive models which can accurately predict architecture performance (accuracy, latency) and hence can be used as a 'simulator' by NAS algorithms for rapid research and fair comparison.\n\nComments:\n\n- The paper is generally well-written and has thorough clean experiments! Thanks!\n\n- My main concern is the following: The fact that the benchmark has added architectures encountered on the trajectory of well-known performing optimizers bothers me a bit. In the ideal world a benchmark should have no knowledge of any particular solution to the problem. This is true in the case of tabular benchmarks like 101 and 201. I understand the position that a surrogate model should be very good at predicting parts of the architecture space which optimizers are most likely to visit. Can we construct surrogates without knowing anything about any particular optimizers the community may invent in the future?\n\nOne part of an ablation study answering this question has been presented in Appendix E.2 where a model has been only trained on well-performing architectures (above 92% accuracy) and in Figure 21 has been found adequate for predicting the trajectories of BANANAS and Random Search (RS). This begs the question of what if we only used random sampling to fit surrogate models? (Of course coverage methods like adaptive submodularity-based greedy algorithms may result in even better performance). But can we do the easy baselines first for which the authors already have the data:\n\n1.  Train surrogate model using only the 23746 architectures via random sampling and plot the same figures as in Fig 4 center and right. How much worse are they from current ones?\n\n2. (if compute allows): Use up the entire budget of ~60k architectures in Table 2 only from random sampling and plot Fig 4 center and right.  How much worse are they from current ones?\n\n3. (to have a fair comparison of current method to baseline 1 without using much more compute): Keep the total budget 23746 but fill them up in the same proportion from each method as currently in Table 2. For example RS will have 23746/~60k ratio, DE will have 7275/~60k ratio, etc. This will create \"NASBench-301-small\" which will have the same budget as baseline 1 above. Then plot Fig 4 center and right with both this and 1.  \n\nThese might be competitive baselines because the Once-For-All work from Cai et al. ICLR 2020 uses a regressor trained on 16k architectures sampled from a supergraph as a surrogate model to run evolutionary search against and obtain good performance. Also in this paper itself if more than 21500 architectures on 101 are used to train (unclear from the paper whether they were randomly sampled and diverged models rejected or some other technique was used to select them, since it says \"subsets of D^{train}\" but I think they were randomly sampled, right?), then that itself is better than the tabular benchmark.\n\nHappy to be convinced if these are fair baselines or not. (Also possible that these are already included and I missed them. Ther are a lot of experiments :-))\n\n-My other worry is that the LOOO ablation may be misleading since reasonable optimizers may be visiting similar parts of the architecture space hence may give a false sense of extrapolation.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Tp84KeLQByA", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary:\n\nThe authors propose a new benchmark for evaluating surrogate functions for architecture search. According to the authors, existing tabular architecture search benchmarks are insufficient for this purpose due to using overly small search spaces. The main difference of this benchmark and other existing architecture search benchmarks such as NAS Bench 101 and NAS Bench 201 is that they do not attempt to evaluate all the architectures in the search space and do so for a much larger search space (DARTS). The authors then use this new dataset to show that surrogate functions are better than tabular estimators (error wise; although some lower performance architectures were discarded to make this case). Additionally, the authors compare the proposed benchmark (based on surrogate functions) with a real benchmark and observe that the results are qualitatively similar. Finally, the authors show that reevaluate the claim that local search is state-of-the-art for architecture search and find that, using their benchmark, that local search is still state-of-the-art provided that enough computational budget is available for the experiment.\n\nPros:\n+ The paper presents a new dataset for architecture search that does not rely on exhaustive evaluation of all architectures in the search space. The experiments conducted that compare different surrogate functions on this benchmark are solid \n+ The paper suggests that building benchmarks for different search spaces can be accomplished through the surrogate function route where first a dataset of architectures is collected and then it is used to train a surrogate model. The surrogate function is then used as an interface between the search space and the search algorithm. This approach for building benchmarks is general and is likely to be useful in the construction of future benchmarks for architecture search. \n\nCons:\n- The fact that existing benchmarks for architecture search are insufficient for surrogate function evaluation and lead to wrong inferences is perhaps insufficiently supported. It is well known in architecture search work that existing search spaces (DARTS being one of them) have limited performance variability and that much of the performance variation ascribed to different architecture search methods can often be ascribed to differences in search spaces. Is a single search space good enough to mitigate these problems? For example, how do we guarantee that NAS Bench 301 is not just another dataset and guarantee that addresses some of the perceived problems with existing architecture search search spaces and benchmarks?\n\n\nComments: \n- Appendices weren't included in the submission.\n- It would be interesting to discuss how this benchmark should be placed in the context of other existing benchmarks for architecture search to guide future practice.\n- The motivation from Section 2 is rather sparse. It would be warranted to show that this trend is consistent with other metrics such as squared error and Kendall Tau (i.e., showing the ranks of different architectures are also preserved better).\n- Insufficient information about the feature representation that is used for prediction?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "KU2iPND2aQR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThis work filled an important gap in the NAS benchmarks. The previous benchmarks only contain small search space due to the expensive cost of evaluation of neural architecture. In this search space, random search often becomes competitive in the narrow search space. Thus, to provide meaningful comparison, this work provided a benchmark in a large NAS search space (same as in DARTS), and using  surrogate models to predict validation performance of untrained neural architecture. The empirical results suggested using the surrogate benchmarks resulted in similar optimization trajectory as real evaluations and the author also shows one can derive/validate research ideas quickly with the benchmarks.\n \n \n##########################################################################\n\nPros: \n \n1. This work filled an important gap in NAS benchmarks. \n2. The empirical results are very solid; the observation on the noise in the training is very insightful.\n3. The work also contains guidelines for using the benchmarks.\n4. The paper is very well written and contains enough detail for reproducibility.\n \n##########################################################################\n\nCons: \n\nI only have some minor comments:\n\n1. In Section 3.2, first paragraph, the author mentioned that validation and test error are highly correlated. This is clear if the poor performing architectures are included. But in practice, we are only interested in the good performing region, say top 5%, is the correlation still high between validation and test there?\n\n2. In Section 4.2, second paragraph,  I am not sure I understand \"We use train/val/test splits (0.8/0.1/0.1) stratified by the NAS\nmethods used for the data collection\".\n\n3. Given the mean and noise estimation based on the surrogate models, is the assumption there is gaussian and every experiment will draw one value from this gaussian? If so, could you state it clearly in the paper? If not, please clarify.\n\n=====POST-REBUTTAL COMMENTS======== \n\nInitially I had only minor comments and the authors addressed all of them. I will keep my score.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The contributions of this paper lie in two areas: a new benchmarking dataset and a new way to generate benchmarking datasets. Overall, the reviewers are split in their assessment based on which particular area they are focusing on. Reviewers who focus more on evaluating this work as a new benchmarking dataset, correctly point out that the variation within the search space has been shown to be limited and that the evaluation focuses on an overly-studied and toy (by today’s standards) dataset such a CIFAR-10. In terms of choice of dataset, this work is indeed a step backwards from nasbench201, which includes more datasets, although it is a step forward in terms of size of the search space. As one reviewer correctly points out “This paper doesn’t present a benchmark. It provides a model that represents computationally-efficient means of getting network accuracies from the DARTS search space”. The authors argue that the combination of the DARTS search space and its evaluation on CIFAR-10 is the de-facto evaluation standard in the NAS community, which is also true. Ultimately, benchmark datasets somewhat direct the attention of the community and this attention would be better directed elsewhere, not on DARTS+CIFAR-10, as pointed out by some of the reviewers.

On the other hand, this work is as much about a new benchmark as it is about a *protocol* to generate new benchmarks. Specifically, a big part of the appeal and novelty here lie in the idea of training a predictive model from a small subset of architecture evaluations. From this perspective, the authors showed evidence that their approach is sound, economical (in terms of computational cost) and robust to sources of bias in the selection of the architectures to evaluate. The limitation here is that this was only shown in search spaces that where either small or lacking diversity and thus it’s unclear how general its findings are.

Overall, this is very much a paper that could have gone either way in terms of acceptance. It’s a step in the right direction in terms of methodology that can be used to generate reproducible benchmarks in a computationally efficient way. It’s a step backwards in terms renewing focus on measures of performance that (arguably) we have all overfit to. 

Decision: Reject