NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
Authors: Xuanyi Dong
Yi Yang
Venue:  (2020)
Keywords: Neural Architecture Search
AutoML
Benchmark

Abstract: Neural architecture search (NAS) has achieved breakthrough success in a great number of applications in the past few years.
It could be time to take a step back and analyze the good and bad aspects in the field of NAS. A variety of algorithms search architectures under different search space. These searched architectures are trained using different setups, e.g., hyper-parameters, data augmentation, regularization. This raises a comparability problem when comparing the performance of various NAS algorithms. NAS-Bench-101 has shown success to alleviate this problem. In this work, we propose an extension to NAS-Bench-101: NAS-Bench-201 with a different search space, results on multiple datasets, and more diagnostic information. NAS-Bench-201 has a fixed search space and provides a unified benchmark for almost any up-to-date NAS algorithms. The design of our search space is inspired by the one used in the most popular cell-based searching algorithms, where a cell is represented as a directed acyclic graph. Each edge here is associated with an operation selected from a predefined operation set. For it to be applicable for all NAS algorithms, the search space defined in NAS-Bench-201 includes all possible architectures generated by 4 nodes and 5 associated operation options, which results in 15,625 neural cell candidates in total. The training log using the same setup and the performance for each architecture candidate are provided for three datasets. This allows researchers to avoid unnecessary repetitive training for selected architecture and focus solely on the search algorithm itself. The training time saved for every architecture also largely improves the efficiency of most NAS algorithms and presents a more computational cost friendly NAS community for a broader range of researchers. We provide additional diagnostic information such as fine-grained loss and accuracy, which can give inspirations to new designs of NAS algorithms. In further support of the proposed NAS-Bench-102, we have analyzed it from many aspects and benchmarked 10 recent NAS algorithms, which verify its applicability.

Published as a conference paper at ICLR 2020
NAS-BENCH-200 EXTENDING THE SCOPE OF RE- PRODUCIBLE NEURAL ARCHITECTURE SEARCH
Xuanyi Dongit and Yi Yang!
fReLER, CAI, University of Technology Sydney, BBaidu Research
ABSTRACT
Neural architecture search (NAS) has achieved breakthrough success in a great number of applications in the past few years. could be time to take a step back and analyze the good and bad aspects in the field of NAS. A variety of algorithms search architectures under different search space. These searched architectures are trained using different setups, e.g., ppprr-paraettes, data augmentation, regular- ization. This raises a comparability problem when comparing the performance of various NAS algorithms. NAS-Bench-10 101 has shown success to alleviate this problem. In this work, we propose an extension to NAS-Bench-101 101: NAS-Bench- 201 with a different search space, results on multiple datasets, and more diagnos- tic information. NAS- Bench-201 has a fixed search space and provides a unified benchmark for almost any up-to-date NAS algorithms. The design of our search space is inspired from the one used in the most popular cell-based searching al- gorithms, where cell is represented as directed acyelic graph. Each edge here is associated with an operation selected from a predefined operation set. For it to be applicable for all NAS algorithms, the search space defined in NAS WAS-Benchh221 includes all possible architectures generated by 4 nodes and 5 associated operation options, which results in 15,625 neural cell candidates in total. The training log using the same setup and the performance for each architecture candidate are pro- vided for three datasets. This allows researchers to avoid unnecessary repetitive training for selected architecture and focus solely on the search algorithm itself. The training time saved for every architecture also largely improves the efficiency of most NAS algorithms and brings more computational cost friendly NAS com- munity for a broader range of researchers. We provide additional diagnostic in- formation such as fine-grained loss and accuracy, which can give inspirations new designs of NAS algorithms. In further support of the proposed NAS-Bench- 201, we have analyzed from many aspects and benchmarked 10 recent NAS algorithms, which verify its applicability.
INTRODUCTION
The deep learning community undergoing a transition from hand-designed neural architecture (He et al. 2016; Krizhevsky et al.) 20122] Szegedy et al.] 2015) to automatically designed neural archi- tecture (Zoph & Lc| 2017; Pham et al. 2018 Real ct al. 2019 Dong & Yang 2019b] Liu et al.] 2019). In its carly era, the great success of deep learning was promoted by novel neural architec- tures, such as ResNet (He et al. 2016), Inception (Szegedy et al. 2015), VGGNet (Simonyan & Zisserman 2015), and Transformer (Vaswani al. (017). However, manually designing one archi- tecture requires human experts to try numerous different operation and connection choices (Zoph)
& Le 2017]. In contrast architectures that are manually designed, those automatically found by neural architecture search (NAS) algorithms require much less human interaction and expert effort. These NAS- NAS-generated architectures have shown promising results in many domains, such as image recognition (Zoph & Le 2017 Pham et al. [0118] Real et al. 2019), sequence modeling (Pham et al.) 2018 Dong & Yang 201 9b19b] Liu et al. 2019) etc.
Recently, a variety of NAS algorithms have been increasingly proposed. While these NAS methods are methodically designed and show promising improvements, many setups in their algorithms are
Part of this work was done when Xuanyi was research intern with Baidu Research.

---

Published as a conference paper at ICLR 2020
architecture
residual block (stride=2)
residual block (stride=2)
global
image
conv
cell x N
cell
x N
cell x N
avg. pool
cell
cell
zeroize
skip-connect
Ixl conv
cell
cell
3x3 conv
3x3 avg pool
predefined operation set
Top: the macro skeleton of each architecture candidate. Bottom-left: examples of neural cell with 4 nodes. Each cell is directed acyclic graph, where each edge is associated with an operation selected from a predefined operation set as shown in the Bottom-right.
Figure 1:
different. (1) Different search space utilized, e.g., different macro skeletons of the whole archi- tecture (Zoph et al., 2018: Tan et al. 2019) and a different operation set for the micro cell within the skeleton (Pham et al. 2018). etc. (2) After good architecture is selected, various strategies can be employed to train this architecture and report the performance, e.g., different data augmen- tation (Ghiasi et al. 2018 Zhang et al., 2018), different regularization (Zoph et al., 2018), different scheduler (Loshchilov lov & Hutter (017)) 7) and different selections of hyyer--araaeters. (Liu et al.) 2018 Dong & Yang 2019a). (3) The validation set for testing the performance of the selected archi- tecture is not split in the same way (Liu et al.) [209 Pham et al.) 2018). These discrepancies raise a comparability problem when comparing the performance of various NAS algorithms, making difficult conclude their contributions.
(Yin et al. 2019) and NAS-HPO-Bar -erO-Beech (Klein & Hutter 2019) are proposed. However, some NAS algorithms not be applied directly NAS- Bench- 101, and NAS-HPO-Bench only has 144 candidate architectures, which maybe insufficient to evaluate NAS algorithms. To extend these two benchmarks and towards better reproducibility of NAS method!" we propose MAA--Bnnc-200) with fixed cell search space, inspired from the search space used in the most popular neural cell-based searching algorithms (Zoph et al.) 2018 Liu al., 2019). As shown in Figure each architecture consists of predefined skeleton with a stack of the searched cell. In this way, architecture search is transformed into the problem of searching good cell. Each cell is represented as densely-connected directed acyclic graph (DAG) as shown in the bottom section of Figure Here the node represents the sum of the feature maps and each edge associated with operation transforming the feature maps from the source node to the target node. The size of the search space is related to the number of nodes defined for the DAG and the size of choose 4 nodes and 5 representative operation candidates for the operation set, which generates a total search space of 15,625 cells/architectures Fach ar- chitecture is trained multiple times three different datasets. The training log and performance of each architecture are provided for each run. The training accuracy/test accuracy/traiiin loss/test loss after every training epoch for each architecture plus the number of parameters and floating point operations (FLOPs) are accessible.
In response to this problem,
the operation set. In MAS--enchh-20
Hopefully, NAS MAS--Bnnc-2211 will show its value in the field of NAS research. (1) provides a unified benchmark for most up-to-date NAS algorithms including all cell-based NAS methods. With NAS- Bench-201, researchers can focus on designing robust searching algorithm while avoiding tedious hyper-parameter tuning of the searched architecture. Thus, WAS--ench-220 provides relatively fair benchmark for the comparison of different NAS algorithms. (2) It provides the full training log of each architecture. Unnecessary repetitive training procedure of each selected architecture can be avoided (Liu et al. 2018 Zoph & Le 2017) so that researchers can target on the essence of NAS, ie., search algorithm. Another benefit is that the validation time for NAS largely decreases when which provides a computational power friendly environment for more participations in NAS. (3) It provides results of each architecture multiple datasets. The model transferability can be thoroughly evaluated for most NAS algorithms. (4) In NAS-Bench-201, we provide systematic analysis of the proposed search space. We also evaluate 10 recent advanced NAS
testing in WAS-Bench-225
One parallel concurrent work for the similar purpose MAS-BBnch--1SHO Zela et al.
2020
2

---

Published as a conference paper at ICLR 2020
algorithms including reinforcement learning (RL)-based methods, evolutionary strategy (ES)-based methods, differentiable- based methods, etc. We hope our empirical analysis can bring some insights to the future designs of NAS algorithms.
NAS-Bench-200
Our NAS-Bench-201 is algorithm-agnostie Put simply, it is applicable to almost any up-to-date NAS algorithms. In this section, we will briefly introduce our NAS-Bench-200 The search space of NAS- Bench-201 is inspired by cell-based NAS algorithms (Section 2.1). NAS-Bench-201 evaluates each architecture on three different datasets (Section 2.2). All implementation details of WAS-Bench-201 introduced in Section 2.3 NAS-Bench-201 also provides some diagnostic information which can be used for potentially better designs of future NAS algorithms (discussed in Section 2.4)
2.1 ARCHITECTURES IN THE SEARCII SPACE
Macro Skeleton. Our search space follows the design of its counterpart as used in the recent neural
cell-based NAS algorithms (Liu al. 2019 Zoph al. 2018 Pham al. 2018). As shown in the top of Figure 0 the skeleton is initiated with one 3-by-3 convolution with 16 output channels and batch normalization layer (loffe & Szegedy 2015). The main body of the skeleton includes three stacks of cells, connected by a residual block. Each cell is stacked N = 5 times, with the number of output channels as 16. 32 and 64 for the first, second and third stages, respectively. The intermediate residual block is the basic residual block with a stride of 2 (He et al.] (2016), which serves to down- sample the spatial size and double the channels of an input feature map. The shorteut path this residual block consists of a 2-by-2 average pooling layer with stride of and 1-by-1 convolution. The skeleton ends up with a global average pooling layer to flatten the feature map into a feature vector. Classification uses fully connected layer with a softmax layer to transform the feature vector into the final prediction.
Searched Cell. Each cell in the search space is represented as densely connected DAG. The densely connected DAG obtained by assigning a direction from the i-th node to the j-th node (i < j) for each edge in undirected complete graph. Each edge in this DAG is associated with an operation transforming the feature map from the source node to the target node. All possible operations are selected from predefined operation set, shown in Figure F bottom- high In our NAS-Bench-201, the predefined operation set O has L = 5 representative operations: (1) zeroize, (2) skip connection, (3) 1-by-1 convolution, (4) 3-by-3 convolution, and (5) 3-by-3 average pooling layer. The convolution in this operation set is an abbreviation of an operation sequence of ReLU, convolution, and batch normalization. The DAG has = 4 nodes, where each node represents the sum of all feature maps transformed through the associated operations of the edges pointing to this node. We choose V requires 4 nodes. Densely connected DAG does not restrict the searched topology of the cell to be densely connected, since we include zeroize the operation set, which an operation of dropping the associated edge. Besides, since we do not impose the constraint on the maximum number of edges (Ying et al.) 2019). our search space is applicable to most NAS algorithms, including all cell-based NAS algorithms.
allow the search space to contain basic residual block-like cells, which
2.2 DATASETS
We train and evaluate each architecture on CIFAR-10, CIFAR- (Krizhevsky et al. 2009). and ImageNet- - (Chrabaszez al., 2017). We choose these three datasets because CIFAR and ImageNet (Russakovsky et al. 2015) are the most popular image classification datasets.
We split each dataset into training, validation and test sets to provide consistent training and evalu- ation settings for previous NAS algorithms (Liu et al.) 2019). Most NAS methods use the validation set to evaluate architectures after the architecture is optimized on the training set. The validation performance of the architectures serves as supervision signals to update the searching algorithm.
The test set is to evaluate the performance of each searching algorithm by comparing the indicators (e.g., accuracy, model size, speed) of their selected architectures. Previous methods use different splitting strategies, which may result in various searching costs and unfair comparisons. We hope to use the proposed splits to unify the training, validation and test sets for a fairer comparison.
3

---

Published as a conference paper at ICLR 2020
CIFAR-10: It is standard image classification dataset and consists of 60K 32x32 colour images in 10 classes. The original training set contains 50K images, with 5K images per class. The original test set contains 10K images, with 1K images per class. Due to the need of validation set, we split all 50K training images in CIFAR-10 into two groups. Each group contains 25K images with 10 classes. We regard the first group the new training set and the second group as the validation set.
This dataset is just like CIFAR-10. It has the same images as CIFAR- 10 but categorizes each image into 100 fine-grained classes. The original training set on CIFAR-100 has 50K images, and the original test set has 10K images. We randomly split the original test set into two group of equal size - 5K images per group. One group is regarded as the validation set, and another one is regarded as the new test set.
CIFAR-100:
ImageNet-16-122 120: We build ImageNet-16-120 from the down-sampled variant of ImageNet (ImageNet16x 16). As indicated in Chrabaszez et al. (017), down-sampling images in ImageNet can largely reduce the computation costs for optimal hyper-parameter of some classical models while maintaining similar searching results. Chrabaszez et al. (2017) down-sampled the original ImageNet to 16x 16 pixels to form ImageNet16x 16, from which we select all images with label E [1, 120] to construct ImageNet- 1--1661110 200. In sum, ImageNet-16- 1---166110 120 contains 151.7K training im- ages, 3K validation images, and 3K test images with 120 classes.
By default, in this paper, "the training set", "the validation set", "the test set" indicate the new training, validation, and test sets, respectively.
2.3 ARCHITECTURE PERFORMANCE
Training Architectures. order to unify the performance of every architecture, we give the per- In our NAS- WAS-Bench-201, we follow previous
formance of every architecture in our search space.
literature to set up the hyper-parame parammttrasssss and train- ing strategies (Zoph et al., 2018, Loshchilov & Hutter 2017: He et al. 2016). We train each ar- chitecture with the same strategy, which is shown in Table For simplification, we denote all hyper- parameters for training model as a H. and we that
Table 1:
The training syper-parameter set Ht.
optimizer
SGD
initial LR
0.1
Nesterov
V
ending LR
0
momentum
0.9
LR schedule
cosine
weight decay
0.0005
epoch
200
batch size
256
initial channel
16
use Ht to denote the values of we use. Specifically, we train each architecture via Nesterov momentum SGD, using the cross-entropy loss for 200 epochs in total. We set the weight de- cay as 0.0005 and decay the learning rate from 1 to O with a cosine annealing (Loshchilov & Hutter 2017 We use the same Hi on different datasets, except for the data augmentation which is slightly different due to the image resolution. On CIFAR, we use the random flip with probability of 0.5, the random crop x22 patch with 4 pixels padding on each border, and the normalization over RGB channels. On 1e- we with 2 pixels padding on each border. Apart from using H for all datasets, we also use a different hyper-parameter
V
4
N
random flip normalization
P=0.5
random crop
/
similar strategy but random crop 16x 16 patch
Ht for CIFAR-10. It is similar Ht but its total number of training epochs is 12. In this way. we could provide candit-based algorithms (Falkner et al. [0018] Li ct al. 2018) more options for the usage of short training budget (see more details in appendix).
Metries. We train each architecture with different random seeds on different datasets. We evaluate NAS-Bench-201 provides the training, validation,
each architecture A after every training epoch. and test loss as well as accuracy. We show the supported metrics on different datasets in Ta-
Table 2: NAS-Bench-20/ provides the following metrics with Ht. 'Ace.' means accuracy.
ble 2 Users can easily use our API to query the results of each trial of A, which has neg- ligible computational costs. In this way, re- searchers could significantly speed up their searching algorithm these datasets and fo- cus solely on the essence of NAS.
Dataset
Train Loss/Acc, Eval Loss/Acc.
CIFAR- 10
train set train+valid set
valid set test set valid set test set valid set test set
CIFAR-10
CIFAR-100
train set
CIFAR-100
train set
ImageNet-16-122
train set
We list the training/test loss/aceuracies over different split sets on four datasets in Table On CIFAR-10, 10, we train the model on the training set and evaluate it on the validation set. We also train the model on the training and validation set and
ImageNet-166-100
train set
4

---

Published as a conference paper at ICLR 2020
#archit
#data
search space
Supported NAS algorithms
Diagnostic information
[0]
ecttures
-sets
constraint
RL ES
Diff. HPO
WAS-Bench-10N 101
510M
constrain #edges
partial
none none
most
fine-grained
NAS-Bench-
15.6K
3
5
constraint
all
all
all
most
info., param., etc
Table 3: We summarize some characteristics NAS-Bench-100 and NAS-Bench-201. Our NAS- Bench-201 can directly be applicable to almost any up-to-date NAS algorithms. In contrast, as pointed in (Ying et al. 2019), NAS algorithms based on parameter sharing or network morphisms cannot be directly evaluated on NAS-Bench-101. Besides, NAS-Bench-201 provides train/valida- tion/test performance on three (one for NAS-Bench-101) different datasets so that the generality of NAS algorithms can be evaluated. It also provides some diagnostic information that may provide insights to design better NAS algorithms.
evaluate it on the test set. These two paradigm follow the typical experimental setup CIFAR-10 in previous literature (Liu et al. 2018, Zoph et al.] 22018) Liu et al.] [2018] Pham et al. 2018). On CIFAR-100 100 and ImageNet- 16- 120, we train the model on the training set and evaluate it on both validation and test sets.
2.4 DIAGNOSTIC INFORMATION
Validation accuracy is a commonly used supervision signal for NAS. However, considering the expensive computational costs for evaluating the architecture, the signal is too sparse. In our NAS- Bench-201, we also provide some diagnostic information which some extra statistics obtained during training each architecture. Collecting these statistics almost involves no extra computation cost but may provide insights for better designs and training strategies of different NAS algorithms, such platform-awan
NAS (Tan al. 2019), accuracy prediction (Baker et al. 2018), mutation- based NAS (Cai et al. 2018 Chen et al.
2016). etc.
Architecture Computational Costs: NAS-Bench-20/ provides three computation metrics for each architecture - the number parameters, FLOPs, and latency. Algorithms that target on searching these metrics
architectures with computational constraints, such as models edge devices, directly in their algorithm designs without extra calculations.
Fine-grained training and evaluation information. NA.S-Bench-25 tracks the changes in loss and accuracy of every architecture after every training epochs. These fine- ree-grained training and evaluation information shows the tendency of the architecture performance and could indicate some attributes of the model, such as the speed of convergence, the stability, the over-fitting or under-fitting levels, etc. These attributes may benefit the designs of NAS algorithms. Besides, some methods learn to predict the final accuracy of an architecture based on the results of few early training epochs (Baker al.) 2018). These algorithm can be trained faster and the performance of the accuracy prediction can be evaluated using the fine-grained evaluation information.
Parameters of optimized architecture. Our NAS-Benchs each architecture. This can provide ground truth label for hypereetorrk-bate NAS methods (Zhang
releases the trained parameters for
jet al.) 2019; Brock et al.] 2018), which learn to generate parameters of an architecture. Other methods mutate architecture to become another one (Real et al.) 2019 Cai et al. 2018). With
MAAS-Bench-20 201, researchers could directly use the off-the-shelf parameters instead of training from scratch and analyze how to transfer parameters from one architecture to another.
3 DIFFERENCE WITH EXISTING NAS BENCHMARKS
To the best of our knowledge, NAS-Benche 101 (Ying et al. 2019) is the only existing large-scale architecture dataset. Similar to NAS-Bench-201, NAS-Bench-101 also transforms the problem of architecture search into the problem of searching neural cells, represented as DAG. Differently, MAS-Benchh 101 defines operation candidates on the node, whereas we associate operations on the edge as inspired from (Liu al., 2019; Dong & Yang 2019b] Zoph al.) 2018). We summa- rize characteristics of our NAS- Bench-20 201 and NAS-Bench- 101 in Table The main highlights of follows. (1) MAS-Bench-201 is algorithm-agnostic while NAS-Bencht
5

---

Published as a conference paper at ICLR 2020
18
the fictreer bectontnggggoooop ROuHArr (M
onier SCUUSO 1M)
Uhe www.caaart//
CIFAR-
70
70
70
3.31
6200
6017
15600
90.5
023
0.6
0.6
1.5
900
8.6
325
980
0.1
0.4
CIFAR-10N
TRINE
G24 6333
12466
15420
64
35
0.
65
36
Aperansters
50
15.
52
50
ImageNet- 120
%
the 19
reanet
6:3
2.5
undhosctnn 1D
aporerddar (MB)
#peranatio P4S1
Figure 2: Training, validation, test accuracy of each architecture CIFAR- 10, CIFAR-100, and AmageNet-16-1222 We also visualize the results of ResNet in the orange star marker.
101 without any modification only applicable to selected algorithms (Yu et al. 2020 Zela et al. 2020). The original complete search space, based on the nodes in NAS-Bench- 101, extremely huge. So, it is exceedingly difficult to efficiently traverse the training of all architectures. To trade off the computational cost and the size of the search space, they constrain the maximum number of edges in the DAG. However, it is difficuut to incorporate this constraint in all NAS algorithms, such as NAS algorithms based on parameter-sharing (Liu al., 2019 Pham et al., 2018). Therefore, many NAS algorithms cannot be directly evaluated on NAS- Bench- 101. Our MAS-Bench-201 solves this problem by sacrificing the number of nodes and including
possible edges so that our search space is algorithm-agnosicc (2) We provide extra diagnostic information, such architecture com- putational cost, fine-grained training and evaluation time, etc., which give inspirations better and efficient designs of NAS algorithms utilizing these diagnostic information.
NAS-HPO- (Klein Hutter, 2019) evaluated 62208 configurations in the joint NAS and hyper-parameter space for simple 2-layer feed-forward net- work. Since WAS-HPO-Bende has only 144 architectures, it could be insufficient evaluate different NAS algorithms.
GIFAR-10
CIFAR-100
ImageNat
architedure ronking
4 ANALYSIS OF NAS-Bench-201
An overview of architecture performance. The performance of each architecture is shown in Figure [ We show the test accuracy of every architecture in our search space in the left column of Figure 2 The training, validation and test accu- racy with respect to the number of parameters are shown in the rest three columns, respectively. Results show that dif- ferent number of parameters will affect the performance of the architectures, which indicates that the choices of operations are essential in NAS. We also observe that the performance of the architecture can vary even when the number of parameters stays the same. This observation indicates the importance of how the operations/cells connected. We compare the architectures with clas-
2604 5208
10416 archilccurre rankking CIFAR-1O
The ranking of each ar- chitecture on three datasets, sorted by the ranking in CIFAR-10.
Figure 3:

---

Published as a conference paper at ICLR 2020
sical human-designe architecture (ResNet) in all cases, which is indicated by orange star mark. ResNet shows competitive performance in three datasets, however, still has room to improve, Le., about 2% compared to the best architecture in CIFAR-100 and ImageNet- 16- 120, about 1% compared to the best one with the same amount of parameters in CIFAR- 100 and ImageNet- 120.
Architecture ranking three datasets. The ranking of every architecture in search space is shown in Figure 3 where the architecture ranked in CIFAR-10 (x-axis) is ranked as in y-axis in CIFAR-100 and ImageNet-16-120 indicated by green and red markers respectively. The perfor- mance of the architectures shows generally consistent ranking over the three datasets with slightly different variance, which serves test the generality of the searching algorithm.
Correlations of validation and test accuracies. We visualize the correlation between the validation and test accuracy within one dataset and across datasets in Figure E The correlation within one
dataset is high compared to cross-datase correlation. The correlation dramatically decreases as we
only pick the top performing archi- tectures. When we directly transfer the best architecture
one dataset to another (a vanilla strategy), it can not 100% secure a good performance. This phenomena is a call for better transferable NAS algorithms instead of vanilla strategy.
sonmaay
Dynamic ranking of architectures. We show the ranking of the perfor- mance of all architectures in different time stamps in Figure 5 The ranking based on the validation set (y axis)
Figure 4: We report the correlation coefficient between the accuracy on 6 sets, i.c. CIFAR-10 validation set (C10- V). CIFAR-10 test set (C10-T), CIFAR- 1-0 validation set (C100-V), CIFAR-100 test set (C100-T), ImageNet- BepeeNet-16-11 16-120 validation set (I1 (120-V), ImageNet-16. 16-1 120 test set (I120-T).
gradually converges to the ranking based on the final test accuracy (x axis).
CAM 31
CHM
Ta opoch
100, opoch
150g epoch
190g spoch
200, opoch
Figure 5: The ranking of all architectures based the validation accuracy at different time stamps (y axis) sorted by the final test accuracy (x axis).
5 BENCHMARK
In this section, we evaluate 10 recent searching methods on our WAS--Bech-2201 which can serve as baselines for future NAS algorithms in dataset. Specifically, we evaluate some typical NAS algo-
rithms: (I) Random Search algorithms, e.g., random search (RS) (Bergstra & Bengio 2012). random
search with parameter sharing (RSPS) (Li & Talwalkar 2019). (II) ES methods, e.g.. REA (Real)
let al. 2019). (III) RI. algorithms, e.g., RHINFORCE (Williams) 1992). ENAS (Pham et al.] 2018).
(IV) Differentiable algorithms. e.g., first order DARTS (DARTS-VI) (Liu al. 2019). second or- der DARTS (DARTS-V22 GDAS (Dong & Yang 2019b), and SETN (Dong & Yang 2019a). (V) HPO methods, e.g., BOHB (Falkner et al. 2018). We experimented all NAS algorithms on a single GeForce GTX 1080 Ti GPU.
accelerate search evaluation
RS
RSPS VI DARTS-V2
GDAS x
SETN REA REINFORCE ENAS BOHB
x
X
The utility of our WAS-Bench- 201 for different NAS algorithms. We show whether a NAS algorithm can our TAS-Bench-22 to accelerate the searching and evaluation procedure.
Table 4:
7

---

Published as a conference paper at ICLR 2020
Search
CIFAR-10
CIFAR-I00
ImageNet-
16-120
Method
(seconds)
validation
test
validation
test
validation
test
RSPS
8007.13
80.42+3.58
84.07+3.61
52.1245.55
52.31+5.77
27.22+3.24
26.28+13.09
DARTS-VI
11625.77
39.77+00.00
54.30+0..00
15.03++..00
15.61+0.00
16.43+0.00
16.32+00.00
DARTS-V V2
35781.80
39.77:10.00
54.30.10.00
15.03:10.00
15.61=0.00
16.43.1.00
16.32:10.00
GDAS
31609.80 80
89.89110.08
93.61110099
71.34:10.04
70.70:10.00
41.59.11.3
41.71:10.98
SETN
34139.53
84.04.10.28
87.64.1..00
58.86.10.06
59.05.10.24
33.06-0.002
32.52.10.21
ENAS
14058.80 80
37.51+3.19
53.89+0.58
13.37+2.35
13.96=2.33
15.06=1.95
14.84+2.10
RSPS
7587.12
84.16++1.69
87.66+1.69
59.00+4.60
58.33+4.34
31.56+3.22
31.14:33.88
DARTS-VI
10889.87
39.77+00.00
54.30+0.000
15.03+0..00
15.61=0.00
16.43=0.00
16.32+00.00
DARTS-V2'
29901.67
39.77:10000
54.30110.00
15.03
15.61=0.00
16.43=0.00
16.32:10.00
GDAST
28925.91
90.00:10.21
93.51:0.13
71.1410.27
70.61:0.26
41.70=:1.26
41.8410.90
SETNN
31009.81
82.25:15.17
86.1944..63
56.86-17.59
56.87=7.77
32.5413.63
31.90:1..07
ENAST
13314.51
39.77:10.00
54.30:10.00
15.03:10.00
15.61:10.00
16.43.10.00
16.32.10.00
REA
0.02
91.19:20.31
93.92=0.300
7181
71.84=0.99
45.15=0.89
1..54=1.033
RS
0.01
90.93-10.36
93.70-10.36
70.93+1.09
71.04+1.07
44.45=1.10
44.57
REINFORCE
0.12
91.09++0.77
93.85++0.77
1..61
1..71
45.05=11002
45.24
BOHB 3.59
90.82++0.53
93.61+0.52
70.74+1.29
7..85
1..26
44.42+11.99
ResNet
90.83
93.97
70.42
70.86
44.53
43.63
N/A
optimal
91.61
94.37
73.49
73.51
46.77
47.31
We evaluate 10 different searching algorithms in our NAS-Bench-201. The first block shows results parameter sharing based NAS methods. The second block similar to the first one, however, BN layers in the searching cells do not keep running estimates but always use batch statistics. The third block shows results of NAS methods without parameter sharing. Each algorithm uses the training and validation set of CIFAR-10 for searching. We show results of their searched architectures for (1) training on the CIFAR-10 train set and evaluating on its validation set; (2) training the CIFAR-10
Table 5:
validation sets and evaluating CIFAR-10 or ImageNet- 16- 120 train set and evaluating on their validation or test sets. "optimal" indicates the highest mean accuracy for each set. We report the mean and std of 500 runs for RS, REA, REINFORCE, and BOHB and of 3 runs for RSPS, DARTS, GDAS, SETN, and ENAS.
its test set; (3) training the
72
8 92
The accurecy (55)
91
86
REA
REA
REA REINFORCE RANDOM
REINFORCE RANDOM
REINFORCE RANDOM
87
63
BOH3
3OHB
BOHB
85
60
300
350
300
400
The index
The index
The index
CIFAR-10
CIFAR-100
ImageNet-16-120
We show results of 500 runs for RS, REA, REINFORCE, and BOHB CIFAR-I0. The architecture is searched on CIFAR-IO and we report its validation accuracy (solid line) and test accuracy (dashed line) on three datasets. Each individual run is sorted by the validation accuracy of the searched architecture.
Figure 6:
We show the benefits for speed using our NAS-Bench-22 for different NAS algorithms in Table For each NAS algorithm, once the searching procedure finished and the final architecture is found, our NAS-Bench-20/ can directly return the performance of this architecture. With NAS-Bench-201, NAS algorithms without parameter sharing can significantly reduce the searching time into seconds. Notably, it still requires several GPU hours for NAS algorithms with parameter sharing complete the searching.
All algorithms use the training and validation set of CIFAR-10 to search architectures. In Table 5 Figure 6 Figure 7 and Figure g we report the performance of the searched architectures plus the three datasets. We make the following observations: (1) NAS methods with-
optimal architecture out parameter sharing (REA, RS, REINFORCE, and BOHB) outperform others. This be because training model for few epochs with the converged LR scheduler (HF) can provide good relative
8

---

Published as a conference paper at ICLR 2020
90
90
90
2 70
70
60
60
50
40
40
DARTS-VA V1 (VAUI))
30
GDAS
30
GDAS
DARTS (VAUD)
SETN
DARTS-V2 (TEST)
10
The 30
100 150
scarching epoch
200
250
50
100
searching
150
200
250
CIFAR-10
The CIFAR- validation
CIFAR-10 test
Figure 7: Results keeping keep running estimates for BN layers in each searching cell. We use parameter sharing based NAS methods to search the architecture on CIFAR-10. After each search- ing epoch, we derive the architecture and show its validation accuracy (VALID) and test accuracy (TEST) on C.. The O-th epoch indicates the architecture is derived from the randomly ini- tialized architecture encoding.
90-
90-
2 70
70
The accurgy
60
40
40
40
DARTS-V1 (VAUD)
30
DARTS (TEST)
GDAS
GDAS
SETN
DARTS V2 (TEST)
10
10
20
30
40
50
50
100
200
250
50
100
150
200
250
The vearching epoch
scarching opach
epoch
CIFAR-10
The CIFAR-10 validation
The CIFAR-10 test
Figure 8: Results using batch statistics without keeping keep rumming estimates for BN layers in each searching cell. We use parameter sharing based NAS methods search the architecture on CIFAR-10. After each searching epoch, we derive the architecture and show its validation accuracy (VALID) and test accuracy (TEST) on CIFAR-10. The 0-th epoch indicates the architecture is derived from the randomly initialized architecture encoding.
ranking of each architecture. (2) DARTS-VE V1 and DARTS- -2 quickly converge to find the architec- ture whose edges are all skip connection. possible reason is that the original hyper-parameter of DARTS are chosen for their search space instead of ours. (3) The strategy of BN layers can significantly effect the NAS methods with parameter sharing. Using batch statisties are better than keep running estimates of the mean and variance. (4) Using our fine-grained information, REA, REINFORCE and RS can be finished seconds which could significantly reduce the search costs and let researchers focus solely on the search algorithm itself.
Figure 7 and Figure 8 we show the performance of the architecture derived from each algorithm per searching epoch. DARTS-VI -I will gradually over-fit to architccture with all skip-connectee connectonn operations. DARTS-VE V2 can alleviate this problem to some extent but will still over-fit after more epochs. It can further alleviate this problem by using batch statistics for BN layers. We train RSPS, GDAS, SETN, and ENAS five times longer than DARTS (250 epochs vs. 50 epochs). This is because at every iteration, RSPS, GDAS, SETN, and ENAS only optimize of=S parameters of the shared parameters, whereas DARTS optimize all shared parameters. The searched architecture performs similar for GDAS after 50 searching epochs. RSPS and SETN show higher variance of the searched architecture compared to GDAS.
Clarification. We have tried our best to implement each method. However, still, some algorithms might obtain non-optimal results since their typer-parameten might not fit our NAS-Bench-20 We empirically found that some NAS algorithms are sensitive to some hyper-parameters, whereas we try to compare them in a fair way as we can (Please see more explanation in Appendix). If researchers happy to update results according
provide better results with different to the new experimental results. We also welcome more NAS algorithms to test on our dataset and would include them accordingly.

---

Published as a conference paper at ICLR 2020
6 DISCUSSION
How to avoid over-fitting NAS-Bench-2012 Our NAS-Bench-201 provides benchmark for NAS algorithms, aiming to provide a fair and computational cost- cost-friendly environment to the NAS community. The trained architecture and the easy-to-access performance of each architecture might provide some insidious ways for designing algorithms to over-fit the best architecture in our NAS- Bench-201. Thus, we propose some rules which we wish the users will follow achieve the original intention of NAS- AAS-Bench-200 201, a fair and efficient benchmark.
No regularization for specific operation. Since the best architecture is known in our benchmark, specific designs to fit the structural attributes of the best performed architecture are insidious ways to fit our NAS- Bench-201. For example, as mentioned in Section we found that the best architecture with the same amount of parameters for CIFARI 10 on HAS-Bench-201 is ResNet. Restrictions on the number of residual connections is a way to over-fit the CIFAR 10 benchmark. While this can give a good result on this benchmark, the searching algorithm might not generalize to other benchmarks.
2. Use the provided performance. The training strategy affects the performance of the architecture. We suggest the users stick to the performance provided in our benchmark even if is feasible to use other H to get a better performance. This provides a fair comparison with other algorithms.
3. Report results of multiple searching runs. Since our benchmark can help to largely decrease the computational cost for a number of algorithms. Multiple searching runs give stable results of the searching algorithm with acceptable time cost.
Limitation regarding to hyper-paramete optimization (HPO). The performance of an architec- ture depends on the byper-parameters H. for its training and the optimal configuration of H may vary for different architectures. In NAS- WAS-Bench-201, we use the same configuration for all architectures, which may bring biases to the performance of some architectures. One related solution is HPO, which aims to search the optimal hyper-parameter configuration. However, searching the optimal Syper-paramees configurations and the architecture in one shot is too computationally expensive and still is an open problem.
Potential designs using diagnostic information in NAS-Bench-201. As pointed in Section 2.4 different kinds of diagnostic information are provided. We hope that more insights about NAS could be found by analyzing these diagnostic information and further motivate potential solutions for NAS. For example, parameter sharing (Pham et al. (2088) is the crucial technique to improve the searching efficiency, but the shared parameter would sacrifice the accuracy of each architecture. Could we find a better way to share parameters of each architecture from the learned 15,625 models' parameters?
Generalization ability of the search space. It is important to test the generalization of observations much larger search space.
on this dataset. An idea strategy is to do all benchmark experiments Unfortunately, it is prohibitive regarding the expensive computational cost. We bring some results from (Ying et al. 2019) and (Zela et al. 2020) to provide some preliminary evidence of generaliza- tion. In Figure we show the rankings of RS, REA, and REINFORCE is REA > REINFORCE RS) ). This is consistent with results in NAS-Bench- 101, which contains more architecture candi- dates. For NAS methods with parameter sharing, we find that GDAS 2 DARTS 2 ENAS, which is also consistent with results in NAS AAS-Bench-ISHOT1. Therefore, observations from our NAS-Bench- 201 may generalize to other search spaces.
7 CONCLUSION & FUTURE WORK
In this paper, we introduce NAS-Bench-201 that extends the scope of reproducible NAS. In NAS- Bench-201, almost any NAS algorithms can be directly evaluated. We train and evaluate 15,625 architecture three different datasets, and provide results regarding different metrics. We comprehensively analyze our dataset and test some recent NAS algorithms on NAS-Bench-20 to serve baselines for future works. In future, will (I) consider HPO and NAS together and (2) much larger search space. We welcome researchers to try their NAS algorithms on our NAS-Bench- 20/ and would update the paper to include their results.
A Acknowledgements. We thank the ICLR area chair, ICLR reviewers, and authors of 101 for the constructive suggestions during the rebuttal and revision period.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. In International Conference on Learning Representations Workshop (ICLR-W), 2018.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. The Journal of Machine Learning Research (JMLR), 13(Feb):281-3053 2012.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. SMASH: one-shot model archi- tecture search through hypernetworks. In International Conference Learning Representations (ICLR), 2018.
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In AAAI Conference on Artificial Intelligence (AAAI), pp. 2787-2794, 2018.
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. International Conference on Learning Representations (ICLR), 2016.
Patryk Chrabaszez, llya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv: 08819, 2017.
Xuanyi Dong and Yi Yang. One-shot neural architecture search via self-evaluated template net- work. In Proc. of the IEEE International Conference on Computer Vision (ICCV), pp. 3681-3690, 2019a.
Xuanyi Dong and Yi Yang. Searching for a robust neural architecture in four gpu hours. In Proc. of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 1761-1770, 2019b.
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter opti- mization at scale. In The International Conference on Machine Learning (ICML), pp. 1436-1445, 2018.
Golnaz Ghiasi, Tsung-Y Yi Lin, and Quoc V Le. Dropblock: A regularization method for convolu- tional networks. In The Conference on Neural Information Processing Systems (NeurIPS), pp- 10727-10737, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
Aaron Klein and Frank Hutter. Tabular benchmarks for joint architecture and hyperparameter opti- mization. arXiv preprint arXiv:
2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolu- tional neural networks. In The Conference Neural Information Processing Systems (NeurIPS), pp. 1097-1105, 2012.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In The Conference on Uncertainty in Artificial Intelligence (UAI), 2019.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research (JMLR), 18())6675-6816, 2018.

---

Published as a conference paper at ICLR 2020
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proc. of the European Conference Computer Vision (ECCV), pp. 19-34, 2018.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference Learning Representations (ICLR), 2019.
Iya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts, In Inter- national Conference on Learning Representations (ICLR), 2017.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In The Conference on Neural Information Processing Systems Workshop (NeurIPS-W), 2017.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. The International Conference on Machine Learning (ICML), pp. 01995-1144 2018.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In AAAI Conference on Artificial Intelligence (AAAI), pp. 4780- 4789, 2019.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 115(3) 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations (ICLR), 2015.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-9, 2015.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proc. of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 2820-2828, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In The Conference on Neural Information Processing Systems (NeurIPS), pp. 5998-999999999 03-06600811 2017.
Ronald J Williams. Simple statistical gradient- algorithms for connectionist reinforcement learning. Machine Learning, (33-4)229-255 256, 1992.
Chris Ying, Aaron Klein, Estehan Real, Eric Christiansen, Kevin Murphy, and Frank Hutter. Nas- bench-101: Towards reproducible neural architecture search. In The International Conference on Machine Learning (ICML), pp. 7105-7114, 2019.
Kaicheng Yu, Christian Seiuto, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. In International Conference on Learning Repre- sentations (ICLR), 2020.
Arber Zela, Julien Siems, and Frank Hutter. hus-bench-Ishotl Benchmarking and dissecting one shot neural architecture search. In International Conference Learning Representations (ICLR), 2020.
Chris Zhang. Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. In International Conference on Learning Representations (ICLR), 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations (ICLR), 2018
12

---

Published as a conference paper at ICLR 2020
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. In Interna- tional Conference on Learning Representations (ICLR), 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recogmition. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8697-8710, 2018.
CIFAR- 10
CIFAR-100
ImageNet- 16-126111000
120
EPOCHS
TOTAL
validation
test
validation
test
validation
test
6
12 (H+)
0.7767
0.7627
0.8086
0.8095
0.8052
0.7941
12
12 (HL)
0.9110
0.8983
0.9361
0.9368
0.9062
0.8952
12
200 (H
0.7520
0.7396
0.8071
0.8080
0.8167
0.8092
24
200 (H))
0.7705
0.7594
0,8280
0.8290
0.8286
0,8217
100
200 (HL)
0.7938
0.7900
0.8529
0.8540
0.8262
0.8211
150
200 (H()
0.8955
0.8926
0.9239
0.9246
0.8506
0,8425
175
200 (HLT)
0.9834
0.9782
0.9743
0.9744
0.8539
0,8423
200
200 (H()
0.9993
0.9937
0.9672
0.9671
0.8259
0.8124
Table 6: We compare the correlation of different training strategies. The correlation coefficient between the validation accuracy after several training epochs on CIFAR-10 and (1) the validation accuracy of full trained models on the CIFAR-10 training set, (2) the test accuracy CIFAR-10 trained with the training and validation sets, (3) the validation/test accuracy on CIFAR-100 trained with the CIFAR-100 training set, (4) the validation/test accuracy ImageNet-16- 1-2200 trained with the ImageNet- 16- 120 training set. We use the validation accuracy after "EPOCHS" training epochs, where the the cosine annealing converged after "TOTAL" epochs.
A MORE DETAILS OF NAS-Bench-201
Number unique architectures. our NAS-Bench-201, we encode each architecture by a 6- dimensional vector. The i-th value in this vector indicates the operation in the i-th edge in cell. Since we have possible operations, there are =15225 total unique models in this encoding. If identify the isomorphic cell caused by the "skip-connect" operation, there are 12751 unique topology structures. If we identify the isomorphic cell caused by both "skip-connect" and "zeroize" operations, there are only 6466 unique topology structures. Note that, due the numerical error, when given the same inputs, two architectures with the isomorphic cell might have different outputs.
Note that, when we build our NAS-Bench we train and evaluate every architecture without considering isomorphism.
NAS-Bench-201 with bandit-based algorithms. Bandit-based algorithms, such as Hyperband (Li et al. 2018) and BOHB (Falkner et al.) 2018). usually train models with a short time budget. In our WAS-Bench-201. on CIFAR-I0, 10, we provide two options you want obtain the performance of model trained with a short time budget: (1) Results from H², where the cosine annealing converged the 12-th epoch. (2) Results from 7t, where the cosine annealing converged at the 200-th epoch. As shown in Table 6 the performance of these converged networks is much more likely to correlate highly with the performance after larger number of iterations than just taking an earlier point of single cosine annealing trajectory. Therefore, we choose the first option for all NAS algorithms that do use parameter sharing.
B IMPLEMENTATION DETAILS
Based on the publicly available codes, we re-implement 10 NAS algorithms by ourselves to search architectures algorithm below.
NAS-Bench-20 We provide the implementation details of each searching
We consider the searching time of the first order DARTS baseline (about 12000 seconds CIFAR-101 10). When evaluating RS, REINFORCE, ENAS, and BOHB, we set the total time budget 12000 seconds for them. By default, for NAS algorithms with parameter sharing, follow
13

---

Published as a conference paper at ICLR 2020
most srper-parameters from DARTS and do not learn the scale and shift parameters for BN lay- ers in each searching cell. We setup the searching procedure of RSPS, GDAS, SETN, ENAS five times longer than DARTS, because they optimize & of parameters but DARTS optimize all pa- rameters per iteration. Most configurations can be found at https: /ttp//ggttttt..//gg com com/D-X-Y/ AutoDI-Prro wttpr//cccttttteeemmtteeeeeeeeee
gagoonngggg//ssss benchhh rk gos
Random search (RS) (Bergstra & Bengio 2012). We randomly select architectures until the total training time plus the time of one evaluation procedure reaches
the total budget. We use the validation accuracy after 12 training epochs (Ht), which can be obtained directly in our
The acaut way (80
as discussed in Section 2.4 The architecture with the highest vali- dation accuracy selected as the final searched architecture.
Regularized evolution for image classifier architecture search (REA) (Real et al., 2019). We set the initial population size as 10, the number of cycles infinity. The sample size is chosen as 10 from [3. 5, 10], according to Figure We finish the algorithm once the simulated training time of the traversed architecture reaches the time budgets (12000 seconds). We use the validation accuracy after 12 training epochs (H+) the fitness.
g.
sampo-ta--
0%
The index nuns
Figure 9: The effect of differ- ent sample sizes for REA on the CIFAR-10 10 validation set.
REINFORCE (Williams 1992). We follow (Ying et al. 2019) to use the REINFORCE algorithm as
baseline RL method. We an architecture encoding to parame- terize each candidate in our search space as (Liu et al. 2019] Dong & Yang 2019b). We use the validation accuracy after 12 training epochs (H+ as the reward in REINFORCE. The architecture en- coding is optimized via Adam. We evaluate the learning rate from [0.01, 0.02, 0.05, 0.1, 0.2. 0.5] following (Ying et al. 2019). Ac- cording to Figure I0 the learning date is set as The momentum for exponential moving average of 0.9. We finish the training once the simulated training time reaches the time budgets (12000 seconds).
The accuraty 196 9.
U1-0:1
00
Uh-0.20 20
200
430
The index of rums
Figure 10: We evaluate
The first order and second order DARTS (DARTS-VI and DARTS- V2) (Liu et al., 2019). We train the shared parameters via Nesterov momentum SGD, using the cross-entropy loss for 50 epochs in total. We set weight decay as 0.0005 and momentum of 0.9. We decay the learning rate from 0.025 to 0.001 via cosine learning rate scheduler and clip the gradient by 5. We train the ar- chitecture encoding via Adam with the learning rate of 0.0003 and the weight decay of 0.001. We use the batch size of 64. The random horizontal flipping, random cropping with padding, and nor- malization are used for data augmentation. We choose these hyper-parameters following (Liu al. 2019).
the effect of different learning rates for REINFORCE, and report the CIFAR-10 valida- tion accuracy of the searched architecture.
Random search with parameter sharing (RSPS) (Li & Talwalkar 2019). We train RSPS with the similar syper-parameteen as that of DARTS. Differently, we train the algorithm in 250 epochs in total. During each searching iteration, we randomly sample one architecture in each batch training. Each architecture uses the training mode for BN during training and the evaluation mode during evaluation (Paszke et al., 2017). After training the shared parameters, we evaluate 100 randomly selected architectures with the shared parameters. For each architecture, we randomly choose one mini-batch with 256 validation samples to estimate the validation accuracy instead of using the whole validation set to calculate the precise validation accuracy. The one with the highest estimated validation accuracy will be selected. With the size of this mini-batch increasing, the more precise validation accuracy would be obtained and the better architecture would be selected. However, the searching costs will also be increased. We use the size of 256 to trade-off the accuracy and cost.
Gradient-based search using differentiable architecture sampler (GDAS) (Dong & Yang 2019b). We use the most hyper-parameters as that of DARTS but train it for 250 epochs in total. The Gumbel-Softmax temperature is linearly decayed from 10 to O.
Self-Evaluated Template Network (SETN) (Dong & Yang. 2019a). We use the most hyper- parameters that of DARTS but train it for 250 epochs in total. After training the shared parameters, we select 100 architectures with the highest probabilities (encoded by the learned architecture en-
14

---

Published as a conference paper at ICLR 2020
CIFAR-10 Validation Set
Methods
Probability
OSVA (BN with Train)
OSVA (BN with Eval)
DARTS-V V-
0.0779
0.0039
-0.0071
DARTS-V2 0.0862
0.0355
0.0109
SETN 0.0682
0,9049
0.0862
GDAS
0.2714
0.8141
0.2466
The correlation between the probability the one-shot validation accuracy (OSVA) and the ground truth accuracy on the CIFAR-I0 validation set. "BN with Train" indicates that, during evaluation, the mean and variance of BN layers are calculated within each mini-batch. "BN with Eval" indicates that we accumulate mean and variance of BN layers in the training set and use these accumulated mean and variance for evaluation. We report the correlation as the average of 3 runs.
Table 7:
coding). We evaluate these 100 selected architectures with the shared parameters. The evaluation procedure for these 100 architectures are the same as RSPS.
ENAS (Pham et al. 2018). We use two layer LSTM as the controller with the hidden size of 32. and the tanh constant of 2.5 for the sampling logits Following (Pham)
We use the temperature et al. 2018), we also add the the controller's sample entropy to the reward, weighted by 0.0001. We optimize the controller with Adam using the constant learning rate of 0.001. We optimize the network weights with SGD following the learning rate scheduler the original paper and the batch size of 128. We did not impose any penalty to a specific operation.
BOHB (Faknner et al. 2018 We choose to BOHB 201. We follow (Ying et al. 2019) to set up the hyper-parameters for BOHB. We set the number the random fraction to 0%, the 0.3, the bandwidth factor to 3. We finish the algorithm once the simulated training time reaches the time budgets (12000 seconds).
HPO algorithm
NAS-Bench-
of samples for the acquisition function
to
C DISCUSSION FOR NAS WITH PARAMETER SHARING
Parameter sharing (Pham al.) 2018) becomes common technique to improve the efficiency of differentiable neural architecture search methods (Liu et al. 2019 Dong & Yangl 2019b a). The shared parameters are shared over millions of architecture candidates. It almost impossible for the shared parameters to be optimal for all candidates. We hope to evaluate the trained shared parame- ters quantitatively. Specially, we use DARTS, GDAS, and SETN to optimize the shared parameters CIFAR-10. For each architecture candidate, we can calculate its
and the architecture encoding probability of being a good architecture from the architecture encoding following SETN (Dong & Yang, 2019a). In addition, we can also evaluate candidate using the shared parameters on the val- idation set to obtain "the one-shot validation accuracy". is computationally expensive to evaluate the whole validation set. To accelerate this procedure, we evaluate each architec-
all candidates ture on a mini-batch with the size of 2048, and use the accuracy on this mini-batch to approximate "the one-shot validation accuracy".. Ideally, the architecture ranking sorted by the probability the one-shot validation accuracy should be similar to the ground truth ranking. We show the correlation between the proxy metric and the ground truth validation accuracy in Table 7 There several observations: (1) The correlation between the probability (encoded by the architecture encoding) and the ground truth accuracy low. It suggests that the argmax-baa based deriving strategy (Liu al.) 2019) can not secure good architecture. It remains open on how to derive a good architecture after optimizing the shared parameters. (2) The behavior of BN layers important to one-shot valida- tion accuracy. The accumulated mean and variance from the training set are harmful to one-shot accuracy. Instead, each architecture candidate should re-calculate the mean and variance of the BN layers. (3) GDAS introduced Gumbel-softmax sampling when optimizing the architecture encod- ing. This strategy leads to a high correlation for the learned probability than that of DARTS. (4) The uniform sampling strategy for training the shared parameters (Dong & Yang, 2019a) increase the correlation for one-shot accuracy compared to the strategy of the joint optimizing strategy (Dong
& Yang 2019b
Liu et al. 2019).
15

---

Published as a conference paper at ICLR 2020
D DETAILED INFORMATION OF NAS-Bench-201
In MAS-Bench-201 (version 1.0), every architecture is trained at least once. To specific, 6219 architectures are trained once, 1621 architectures are trained twice, 7785 architectures trained three times with different random seeds. Moreover, we are actively training all architectures with more seeds and will continue updating our NAS-Bench-200
The latency in our MAS--Bnch--21 (version 1.0) is computed by running each model on a single GPU (GeForce GTX 1080 Ti) with a batch size of 256. We report the latency on CIFAR-100 and ImageNet- 1- 1--166110 120, and the latency on CIFAR- 10 should be similar to CIFAR-10.
The usage API. We provide convenient APIs access our NAS-Bench-200 201, which can easily installed via "pip install nas-bench-201" Some examples are shown as follows:
from nas_201_apl import MA5Bench201API as API
api API (B2 for i, arch_str in enumeratec
# show every architecturre (i, (api), arch_str))
print (' (((:5d)/(:5d) info = app..pprry.eeta_inf_bb_.l_iaeaee
# get metrics of the 1-th arch dict saving loss/acc cos_dit dos_ticttoo = te accuracyy' }))
'train') res_dict inff..et_mmtrricc(cffffrrr. (res print ('The accuracy - print ('The loss is
dict saving costs
10 print ("the flops is (:.2f) M/ ff
11 print ('The Mparameters is [:.2f] MB' (
12 print ('The latency is [:.3f] sf wfw.mattttttsssccc....... (oo_d 1))
13 # query the index of a specific architecture from API
arch_irneeee index
avg avg_pool_3xe0 3x3-1xx111|| 0 is get results
connect-o nor wtw___xxl_lixippccoeeeee
each trial for a specific architecture
16 results apw.auuueeee query byoody_y_innxxxle bn____iioo((arc_iddee..e.. (arch_index,, 'cifar1001)
17 print (' Theee are [:] trials for this architecture [(:]] on cifar100'.
format (len (results), apí (arch_index))
Please see https::////.mm....mm /tte:://gtttuuu.
for more kinds of usages. The
benchmark data file for API can be downloaded online from https::/drive. http://llvvv...gglee... com/ file/d/1 #1le/d/1sx0Caall
16


Reviews:
[{"review_id": "SyemyjsPKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nResearch into Neural Architecture Search (NAS) has exploded in recent times. But unfortunately the entry barrier into the field is high due to the computational demands of running experiments on even cifar10/100 let alone ImageNet sized datasets. Furthermore there is a reproducibility and fair comparision crisis due to differences in search spaces, training routine hyperparameters, stochasticity in gpu training, etc. This paper proposes a benchmark cell-search space (resnet backbone, 4-node cell space, 5 possible operations) which is algorithm agnostic. They train all possible architectures (15625) in this search space on cifar10/100/Imagenet-16-120 (a reduced version of ImageNet with 120 classes). Thus anyone can now use this pretrained lookup-table to benchmark their search algorithm in seconds on a tiny laptop instead of having to get access to a cluster with hundreds of gpus. By also proposing reference implementations of training architectures the community can use this to fairly benchmark their search algorithms. \n\nThe other such benchmark is NASBench-101 which uses a much more expansive search space but by imposing a limit on the number of edges in the cell (to keep the search space manageable with respect to how many of them they have to train) they leave out algorithms which do weight-sharing (ENAS, DARTS, RANDNas) from being able to use their benchmark. This paper alleviates those constraints and thus brings important algorithm classes to their fold.\n\nComments:\n\n- The paper is very well written. Thanks!\n\n- Minor clarification question: One nice thing of the NASBench paper was the fact that they also reported variance in training with differnt random seeds. I see a line in the 'Metrics' section saying that this is also done but did not find any details on number of trials and whether this was part of the benchmark lookup. I might have missed it somewhere. \n\n- There is another class of search algorithms which grow from small to big cells (if using a cell search space) like EFAS (Efficient Forward Architecture Search by Dey et al and AutoGrow by Wen et al.). Can such algorithms take advantage of this benchmark? I think the answer is yes, because of the 'zeroise' operation but wanted to get the authors' answer.\n\n- Overall I think this is an important contribution to the field and I am assuming that the authors plan to release the benchmark and reference implementations if accepted?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A"}, "extra_text": {}}, {"review_id": "SkgxgbqpKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "--- Updated during response period ---\n\nAuthors successfully answers all my questions. I revise my rating to Accept.\n\n\n-----\n\nSummary:\n\nThis paper proposes another benchmark dataset for neural architecture search. The idea is following the NASBench-101 dataset, that in a given search space, densely sampled all existing architectures and train each of them on three tasks for multiple times, and using the obtained metrics as a tool to evaluate an arbitrary neural architecture search algorithm. The paper also presents comprehensive reports on the statistics, revealing a strong performance correlation between tasks, and evaluate some baseline NAS algorithms.\n\nI think this paper will be valuable to the research community for these reasons: (1) the dataset contains a more geologically complex search space comparing to the original NASBench-101, whose search space is restrained in certain ways; (2) released metrics include more meaningful information rather than single point value in NASbench; (3) it uses 3 datasets rather than 1.\nMy major concerns, which I will detail later, is the phrasing \"algorithm-agnostic\" does not truly reflect the difference between their approach and NASBench-101, and about the architecture search space design details. \n\nAltogether, I think even the technical novelty is incremental, the work is not trivial considering the computational cost. I am willing to improve my score if my concerns are addressed during the rebuttal period. Nevertheless, this dataset is a strong subsidy of existing NASBench-101 and can benefit the research community and serves as an important baseline to evaluate a NAS algorithm. \n\n\nStrength\n\n+ Clear motivation to use an operation-on-the-edge search space that is widely used in NAS domain.\n+ Extensive experiments on evaluating 15K architectures over 3 datasets\n+ Detailed statistics on the search space\n+ Good baseline experiments comparison\n\nMain concerns about this dataset:\n\n- Comparing to NASBench-101 in terms of \"Algorithm Agnostic\", it is in a \"more-or-less\" game but not a \"yes-or-no\" one, so that AA-NAS-Bench does not seem appropriate. In my perspective, this dataset has not shown significant differences for the following reasons.\n\n1. With proper adaptation, both NASBench-101 and the one in this paper are \"algorithm agnostic\". For example, original ENAS is training a reinforcement learning sampler that learns to predict a string with encoding [id1, op1, id2, op2] for each node, where id1, id2 is the IDs of the previous node to connect, op1, op2 is the operation choice for each edge. Since NASBench has operation on output node, one could simply make RL sampler to predict [id1, id2, op1], or another string encoding that suits the search space better. In my perspective, Ying et al. mentioned that many NAS algorithms cannot be directly evaluated on NASBench-101 are because the search space is different, but it does not mean using NASBench-101 is impossible. On the other hand, for some other state-of-the-art algorithms, like Proxyless-NAS on ImageNet, the search space is also different from the one proposed in this paper, but likewise, it does not indicate evaluating Proxyless-NAS on this dataset is impossible. \n\n2. NASBench-101 does impose constraints on maximum edge number equals to 9 with 7 nodes in their space and results in 423K architectures. However, this constraint is no longer applied if you reduce the number of node to 6 (i.e. all possible architectures can be sampled), yet it still contains around 64K architectures, which is more than 15K in the proposed dataset. In this perspective, NASBench is a larger dataset and \"algorithm agnostic\".\n\nTo summarize, I acknowledge the paper's contribution is using an operation-on-the-edge search space that is widely used in previous NAS algorithms while NASBench-101 is using operation-on-the-node space. However, it only makes the proposed dataset \"more algorithm agnostic\" with less effort, and it does not make the previous NASBench-101 \"not\" algorithm agnostic. If using the current name AA-NAS-Bench, I think it is not fair for the NASBench-101, specifically they are 4 times larger after removing the edge number constraints. \n\n- Questions about architecture space design\n1. Why using average pooling instead of max pooling? \n2. How do you compute the total architecture number 15,625 in Table 3? In your setting, with the number of node V=4 densely connected DAG, it should have 6 edges as depicted in Figure 1, and each edge has 5 possible operations, i.e. total number = 6^5 = 7776. I am confused about this point, could author comment more on this number?\n3. Is there any topologically equal architectures in this space? For example, let's name the node 1,2,3,4, and the following two architectures should be the same since input edges are summed before passed to the next node. I listed **non-zeroed** edge as, id1->id2: op\n\nArchitecture 1:\n1->2: conv3x3\n2->4: skip\n1->3: conv1x1\n3->4: skip\n\nArchitecture 2:\n1->2: conv1x1\n2->4: skip\n1->3: conv3x3\n3->4: skip\n\nIf the pruning is not effectively conducted, my worry is the actual number of architectures is smaller.\n\nMinor comments:\n1. DARTS results on the are quite poor as mentioned in this paper that, DARTS will eventually converge to an architecture with all skip connection. However, it could be a simple fix, by tracking the architecture evolution during the search and report the best like early-stopping. Will this improve DARTS results? \n\n2. Since ENAS is the first work using parameter sharing on the NAS problem, could the author add it to the baseline?\n\n3. In table 4, what is the average (94.37 for CIFAR-10) mean in the \"optimal\" column? Is this the mean performance of all architectures? If so, it is quite strange to see all the baselines are selecting architectures worse than the average performance. Or it is the best architecture performance as indicated in the caption? This \"average\" column for \"optimal\" seems confusing.\n\n4. The dynamic ranking of architecture in Figure 5 is very interesting. Architecture ranking seems stable after the 190th epoch. Could the author provide another visualization, showing when stabilization happens in between epoch number 150 and 190? \n\n5. Figure 4, correlation matrix for top 4743 architectures are significantly lower than the full and 1387 ones, is this possible because of repetitive architectures in the space are not pruned? And, what is the reason for number 4743 and 1387?\n\n6. ResNet (star in Figure 2) seems to perform very well. Does this indicates the proposed search space is not much meaningful, considering there are only 1~2% for NAS to improve? ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJg7AweAtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Edit after rebuttals: I have read all other reviews and rebuttals and maintain my assessment.\n----\nSummary: Comparison of neural architecture search algorithms is hindered by the lack of a common measurement procedure. This paper describes a publicly available benchmark on which most recent types of NAS algorithms can be evaluated. It does so by exhaustive calculation of performance metrics on the full combinatorial space of select architectures, on two select datasets. NAS algorithms can then perform search without having to perform evaluation on each node, which shrinks the computational cost of experimentation and benchmarking drastically.\n\nI recommend acceptance, as the resource described in the paper has been created thoughtfully and is useful to the research community, as well as to users of NAS algorithms. The paper is clear about restrictions too, which doesn't hurt.\n\nThe technical details are laid out clearly especially in sec 2.1. It would be interesting to know the computational cost of producing the data. It is useful in practice to have access to different metrics (validation, training and test) for each node, as well as extra diagnostic information.  \n\nThe usefulness of the resources hinges on a few elements, which make its strength and also weakness:\n- choice of tasks and datasets\n- choice of skeleton architecture, fig 1\n- choice of hyperparameters, sec 2.3 (I note there is no regularisation, as discussed in the paper)\nAll of these seem reasonable to me. It is clearly a limitation that hyperparameter search is infeasible to conduct in parallel with architecture search, as pointed out sec 6.\n\nThe principal competitor NAS-Bench-101 is only applicable to specific NAS algorithms, which evidences the need for the present resource. The discussion and comparison in sec3 is fair.\n\nThe discussion of weaknesses, such as possible overfitting patterns, or technical choices, is balanced. \n\n# Minor\nEnglish proofreading is required. \n- Maybe you can attempt a pun on Ananas in the naming?\n- I'm not sure \"fairness\" as in the abstract is the exact core problem; I would call this comparability.\n- sec2 head: \"side information\", I usggest diagnostic information\n- sec2.2 \"and etc\" is a redundant: etc stands for \"and the others\"\n- sec2.4 almost involves almost; target on computation cost; stabability\n- sec 4: has impacts on, parameters keeps the same -> stays, which serves as testing -> to test\n- sec6 tricky ways-> insidious?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new benchmark for architecture search. Reviewers put this paper in the top tier. I encourage the authors to also cite https://openreview.net/forum?id=SJx9ngStPH in their final version. 
Decision: Accept (Spotlight)