NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search

1 INTRODUCTION . The deep learning community is undergoing a transition from hand-designed neural architecture ( He et al. , 2016 ; Krizhevsky et al. , 2012 ; Szegedy et al. , 2015 ) to automatically designed neural architecture ( Zoph & Le , 2017 ; Pham et al. , 2018 ; Real et al. , 2019 ; Dong & Yang , 2019b ; Liu et al. , 2019 ) . In its early era , the great success of deep learning was promoted by novel neural architectures , such as ResNet ( He et al. , 2016 ) , Inception ( Szegedy et al. , 2015 ) , VGGNet ( Simonyan & Zisserman , 2015 ) , and Transformer ( Vaswani et al. , 2017 ) . However , manually designing one architecture requires human experts to try numerous different operation and connection choices ( Zoph & Le , 2017 ) . In contrast to architectures that are manually designed , those automatically found by neural architecture search ( NAS ) algorithms require much less human interaction and expert effort . These NAS-generated architectures have shown promising results in many domains , such as image recognition ( Zoph & Le , 2017 ; Pham et al. , 2018 ; Real et al. , 2019 ) , sequence modeling ( Pham et al. , 2018 ; Dong & Yang , 2019b ; Liu et al. , 2019 ) , etc . Recently , a variety of NAS algorithms have been increasingly proposed . While these NAS methods are methodically designed and show promising improvements , many setups in their algorithms are ∗Part of this work was done when Xuanyi was a research intern with Baidu Research . different . ( 1 ) Different search space is utilized , e.g. , different macro skeletons of the whole architecture ( Zoph et al. , 2018 ; Tan et al. , 2019 ) and a different operation set for the micro cell within the skeleton ( Pham et al. , 2018 ) , etc . ( 2 ) After a good architecture is selected , various strategies can be employed to train this architecture and report the performance , e.g. , different data augmentation ( Ghiasi et al. , 2018 ; Zhang et al. , 2018 ) , different regularization ( Zoph et al. , 2018 ) , different scheduler ( Loshchilov & Hutter , 2017 ) , and different selections of hyper-parameters ( Liu et al. , 2018 ; Dong & Yang , 2019a ) . ( 3 ) The validation set for testing the performance of the selected architecture is not split in the same way ( Liu et al. , 2019 ; Pham et al. , 2018 ) . These discrepancies raise a comparability problem when comparing the performance of various NAS algorithms , making it difficult to conclude their contributions . In response to this problem , NAS-Bench-101 ( Ying et al. , 2019 ) and NAS-HPO-Bench ( Klein & Hutter , 2019 ) are proposed . However , some NAS algorithms can not be applied directly on NASBench-101 , and NAS-HPO-Bench only has 144 candidate architectures , which maybe insufficient to evaluate NAS algorithms . To extend these two benchmarks and towards better reproducibility of NAS methods1 , we propose NAS-Bench-201 with a fixed cell search space , inspired from the search space used in the most popular neural cell-based searching algorithms ( Zoph et al. , 2018 ; Liu et al. , 2019 ) . As shown in Figure 1 , each architecture consists of a predefined skeleton with a stack of the searched cell . In this way , architecture search is transformed into the problem of searching a good cell . Each cell is represented as a densely-connected directed acyclic graph ( DAG ) as shown in the bottom section of Figure 1 . Here the node represents the sum of the feature maps and each edge is associated with an operation transforming the feature maps from the source node to the target node . The size of the search space is related to the number of nodes defined for the DAG and the size of the operation set . In NAS-Bench-201 , we choose 4 nodes and 5 representative operation candidates for the operation set , which generates a total search space of 15,625 cells/architectures . Each architecture is trained multiple times on three different datasets . The training log and performance of each architecture are provided for each run . The training accuracy/test accuracy/training loss/test loss after every training epoch for each architecture plus the number of parameters and floating point operations ( FLOPs ) are accessible . Hopefully , NAS-Bench-201 will show its value in the field of NAS research . ( 1 ) It provides a unified benchmark for most up-to-date NAS algorithms including all cell-based NAS methods . With NASBench-201 , researchers can focus on designing robust searching algorithm while avoiding tedious hyper-parameter tuning of the searched architecture . Thus , NAS-Bench-201 provides a relatively fair benchmark for the comparison of different NAS algorithms . ( 2 ) It provides the full training log of each architecture . Unnecessary repetitive training procedure of each selected architecture can be avoided ( Liu et al. , 2018 ; Zoph & Le , 2017 ) so that researchers can target on the essence of NAS , i.e. , search algorithm . Another benefit is that the validation time for NAS largely decreases when testing in NAS-Bench-201 , which provides a computational power friendly environment for more participations in NAS . ( 3 ) It provides results of each architecture on multiple datasets . The model transferability can be thoroughly evaluated for most NAS algorithms . ( 4 ) In NAS-Bench-201 , we provide systematic analysis of the proposed search space . We also evaluate 10 recent advanced NAS 1One parallel concurrent work for the similar purpose is NAS-Bench-1SHOT1 ( Zela et al. , 2020 ) . algorithms including reinforcement learning ( RL ) -based methods , evolutionary strategy ( ES ) -based methods , differentiable-based methods , etc . We hope our empirical analysis can bring some insights to the future designs of NAS algorithms . 2 NAS-Bench-201 . Our NAS-Bench-201 is algorithm-agnostic . Put simply , it is applicable to almost any up-to-date NAS algorithms . In this section , we will briefly introduce our NAS-Bench-201 . The search space of NASBench-201 is inspired by cell-based NAS algorithms ( Section 2.1 ) . NAS-Bench-201 evaluates each architecture on three different datasets ( Section 2.2 ) . All implementation details of NAS-Bench-201 are introduced in Section 2.3 . NAS-Bench-201 also provides some diagnostic information which can be used for potentially better designs of future NAS algorithms ( discussed in Section 2.4 ) . 2.1 ARCHITECTURES IN THE SEARCH SPACE . Macro Skeleton . Our search space follows the design of its counterpart as used in the recent neural cell-based NAS algorithms ( Liu et al. , 2019 ; Zoph et al. , 2018 ; Pham et al. , 2018 ) . As shown in the top of Figure 1 , the skeleton is initiated with one 3-by-3 convolution with 16 output channels and a batch normalization layer ( Ioffe & Szegedy , 2015 ) . The main body of the skeleton includes three stacks of cells , connected by a residual block . Each cell is stacked N = 5 times , with the number of output channels as 16 , 32 and 64 for the first , second and third stages , respectively . The intermediate residual block is the basic residual block with a stride of 2 ( He et al. , 2016 ) , which serves to downsample the spatial size and double the channels of an input feature map . The shortcut path in this residual block consists of a 2-by-2 average pooling layer with stride of 2 and a 1-by-1 convolution . The skeleton ends up with a global average pooling layer to flatten the feature map into a feature vector . Classification uses a fully connected layer with a softmax layer to transform the feature vector into the final prediction . Searched Cell . Each cell in the search space is represented as a densely connected DAG . The densely connected DAG is obtained by assigning a direction from the i-th node to the j-th node ( i < j ) for each edge in an undirected complete graph . Each edge in this DAG is associated with an operation transforming the feature map from the source node to the target node . All possible operations are selected from a predefined operation set , as shown in Figure 1 ( bottom-right ) . In our NAS-Bench-201 , the predefined operation set O has L = 5 representative operations : ( 1 ) zeroize , ( 2 ) skip connection , ( 3 ) 1-by-1 convolution , ( 4 ) 3-by-3 convolution , and ( 5 ) 3-by-3 average pooling layer . The convolution in this operation set is an abbreviation of an operation sequence of ReLU , convolution , and batch normalization . The DAG has V = 4 nodes , where each node represents the sum of all feature maps transformed through the associated operations of the edges pointing to this node . We choose V = 4 to allow the search space to contain basic residual block-like cells , which requires 4 nodes . Densely connected DAG does not restrict the searched topology of the cell to be densely connected , since we include zeroize in the operation set , which is an operation of dropping the associated edge . Besides , since we do not impose the constraint on the maximum number of edges ( Ying et al. , 2019 ) , our search space is applicable to most NAS algorithms , including all cell-based NAS algorithms . 2.2 DATASETS . We train and evaluate each architecture on CIFAR-10 , CIFAR-100 ( Krizhevsky et al. , 2009 ) , and ImageNet-16-120 ( Chrabaszcz et al. , 2017 ) . We choose these three datasets because CIFAR and ImageNet ( Russakovsky et al. , 2015 ) are the most popular image classification datasets . We split each dataset into training , validation and test sets to provide a consistent training and evaluation settings for previous NAS algorithms ( Liu et al. , 2019 ) . Most NAS methods use the validation set to evaluate architectures after the architecture is optimized on the training set . The validation performance of the architectures serves as supervision signals to update the searching algorithm . The test set is to evaluate the performance of each searching algorithm by comparing the indicators ( e.g. , accuracy , model size , speed ) of their selected architectures . Previous methods use different splitting strategies , which may result in various searching costs and unfair comparisons . We hope to use the proposed splits to unify the training , validation and test sets for a fairer comparison . CIFAR-10 : It is a standard image classification dataset and consists of 60K 32×32 colour images in 10 classes . The original training set contains 50K images , with 5K images per class . The original test set contains 10K images , with 1K images per class . Due to the need of validation set , we split all 50K training images in CIFAR-10 into two groups . Each group contains 25K images with 10 classes . We regard the first group as the new training set and the second group as the validation set . CIFAR-100 : This dataset is just like CIFAR-10 . It has the same images as CIFAR-10 but categorizes each image into 100 fine-grained classes . The original training set on CIFAR-100 has 50K images , and the original test set has 10K images . We randomly split the original test set into two group of equal size — 5K images per group . One group is regarded as the validation set , and another one is regarded as the new test set . ImageNet-16-120 : We build ImageNet-16-120 from the down-sampled variant of ImageNet ( ImageNet16×16 ) . As indicated in Chrabaszcz et al . ( 2017 ) , down-sampling images in ImageNet can largely reduce the computation costs for optimal hyper-parameters of some classical models while maintaining similar searching results . Chrabaszcz et al . ( 2017 ) down-sampled the original ImageNet to 16×16 pixels to form ImageNet16×16 , from which we select all images with label ∈ [ 1 , 120 ] to construct ImageNet-16-120 . In sum , ImageNet-16-120 contains 151.7K training images , 3K validation images , and 3K test images with 120 classes . By default , in this paper , “ the training set ” , “ the validation set ” , “ the test set ” indicate the new training , validation , and test sets , respectively .

Summary: Research into Neural Architecture Search (NAS) has exploded in recent times. But unfortunately the entry barrier into the field is high due to the computational demands of running experiments on even cifar10/100 let alone ImageNet sized datasets. Furthermore there is a reproducibility and fair comparision crisis due to differences in search spaces, training routine hyperparameters, stochasticity in gpu training, etc. This paper proposes a benchmark cell-search space (resnet backbone, 4-node cell space, 5 possible operations) which is algorithm agnostic. They train all possible architectures (15625) in this search space on cifar10/100/Imagenet-16-120 (a reduced version of ImageNet with 120 classes). Thus anyone can now use this pretrained lookup-table to benchmark their search algorithm in seconds on a tiny laptop instead of having to get access to a cluster with hundreds of gpus. By also proposing reference implementations of training architectures the community can use this to fairly benchmark their search algorithms.