NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search

1 INTRODUCTION . While neural architecture search ( NAS ) has attracted a lot of attention due to the effectiveness in automatically designing state-of-the-art neural networks ( Zoph & Le , 2017 ; Zoph et al. , 2018 ; Real et al. , 2017 ; 2019 ) , the focus has recently shifted to making the search process more efficient ( Pham et al. , 2018 ; Elsken et al. , 2019 ; Liu et al. , 2019 ; Xie et al. , 2019 ; Cai et al. , 2019 ; Casale et al. , 2019 ) . The most crucial concept which led to a reduction in search costs to the order of a single function evaluation is certainly the weight-sharing paradigm : Training only a single large architecture ( the one-shot model ) subsuming all the possible architectures in the search space ( Brock et al. , 2018 ; Pham et al. , 2018 ) . Despite the great advancements of these methods , the exact results of many NAS papers are often hard to reproduce ( Li & Talwalkar , 2019 ; Yu et al. , 2020 ; Yang et al. , 2020 ) . This is a result of several factors , such as unavailable original implementations , differences in the employed search spaces , training or evaluation pipelines , hyperparameter settings , and even pseudorandom number seeds ( Lindauer & Hutter , 2019 ) . One solution to guard against these problems would be a common library of NAS methods that provides primitives to construct different algorithm variants , similar to what as RLlib ( Liang et al. , 2017 ) offers for the field of reinforcement learning . Our paper makes a first step into this direction . Furthermore , experiments in NAS can be computationally extremely costly , making it virtually impossible to perform proper scientific evaluations with many repeated runs to draw statistically robust conclusions . To address this issue , Ying et al . ( 2019 ) introduced NAS-Bench-101 , a large tabular benchmark with 423k unique cell architectures , trained and fully evaluated using a one-time extreme amount of compute power ( several months on thousands of TPUs ) , which now allows to cheaply simulate an arbitrary number of runs of NAS methods , even on a laptop . NAS-Bench-101 enabled a comprehensive benchmarking of many discrete NAS optimizers ( Zoph & Le , 2017 ; Real et al. , 2019 ) , using the exact same settings . However , the discrete nature of this benchmark does not ∗Equal contribution allow to directly benchmark one-shot NAS optimizers ( Pham et al. , 2018 ; Liu et al. , 2019 ; Xie et al. , 2019 ; Cai et al. , 2019 ) . In this paper , we introduce the first method for making this possible . Specifically , after providing some background ( Section 2 ) , we make the following contributions : 1 . We introduce NAS-Bench-1Shot1 , a novel benchmarking framework that allows us to reuse the extreme amount of compute time that went into generating NAS-Bench-101 ( Ying et al. , 2019 ) to cheaply benchmark one-shot NAS methods . Our mapping between search space representations is novel to the best of our knowledge and it allows querying the performance of found architectures from one-shot NAS methods , contrary to what is claimed by Ying et al . ( 2019 ) . Specifically , it allows us to follow the full trajectory of architectures found by arbitrary one-shot NAS methods at each search epoch without the need for retraining them individually , allowing for a careful and statistically sound analysis ( Section 3 ) . 2 . We introduce a general framework for one-shot NAS methods that can be instantiated to many recent one-shot NAS variants , enabling fair head-to-head evaluations based on a single code base ( Section 4 ) . 3 . We use the above to compare several state-of-the-art one-shot NAS methods , assess the correlation between their one-shot model performance and final test performance , examine how sensitive they are to their hyperparameters , and compare their performance to that of black-box optimizers used in NAS-Bench-101 ( Section 5 ) . We provide our open-source implementation1 , which we expect will also facilitate the reproducibility and benchmarking of other one-shot NAS methods in the future . 2 BACKGROUND AND RELATED WORK . 2.1 NAS-BENCH-101 . NAS-Bench-101 ( Ying et al. , 2019 ) is a database of an exhaustive evaluation of all architectures in a constrained cell-structured space on CIFAR-10 ( Krizhevsky , 2009 ) . Each cell is represented as a directed acyclic graph ( DAG ) where the nodes represent operation choices and the edges represent the information flow through the neural network ( see also Figure 1 and Section 3.1 ) . To limit the number of architectures in the search space , the authors used the following constraints on the cell : 3 operations in the operation set O = { 3x3 convolution , 1x1 convolution , 3x3 max-pool } , at most 7 nodes ( this includes input and output node , therefore 5 choice nodes ) and at most 9 edges . These constraints , and exploiting symmetries , reduced the search space to 423k unique valid architectures . Each architecture was trained from scratch three times to also obtain a measure of variance . In addition , each architecture was trained for 4 , 12 , 36 and 108 epochs ; for our analysis , we mainly used the results for models trained for 108 epochs , if not stated otherwise . 2.2 NAS-BENCH-102 . Concurrently to this work , Dong & Yang ( 2020 ) released NAS-Bench-102 , which is another NAS benchmark that , differently from NAS-Bench-101 , enables the evaluation of weight-sharing NAS methods . Their search space consists of a total of 15,625 architectures , which is exhaustively evaluated on 3 image classification datasets . Similarly to Zela et al . ( 2020 ) and this work , Dong & Yang ( 2020 ) found that architectural overfitting occurs for DARTS for all their datasets . While NAS-Bench-102 and this work go towards the same direction , they differ in many ways : 1 . They use extensive computation to create a new benchmark ( with 15.625 architectures ) , while we devise a novel reformulation to reuse the even much more extensive computation of the NASBench-101 dataset ( 120 TPU years ) to create three new one-shot search spaces with the larges one containing 363.648 architectures . This required zero additional computational cost . 2 . We show that it is possible to reuse the graph representation in NAS-Bench-101 to run one-shot NAS methods ; this requires changes to the one-shot search space , but allows a mapping which can be used for architecture evaluation . 1https : //github.com/automl/nasbench-1shot1 3 . They evaluate their search space on 3 image classification datasets , while we introduce 3 different search spaces ( as sub-spaces of NAS-Bench-101 ) with growing complexity . 2.3 ONE-SHOT NEURAL ARCHITECTURE SEARCH . The NAS problem can be defined as searching for the optimal operation ( e.g . in terms of validation error of architectures ) out of the operation set O in each node of the DAG and for the best connectivity pattern between these nodes . Designing architectures for optimized accuracy or to comply with resource constraints led to significant breakthroughs on many standard benchmarks ( Pham et al. , 2018 ; Zoph & Le , 2017 ; Brock et al. , 2018 ; Liu et al. , 2019 ; Cai et al. , 2019 ; Elsken et al. , 2019 ) . While early methods were computationally extremely expensive ( Zoph & Le , 2017 ) , the weight-sharing paradigm ( Brock et al. , 2018 ; Pham et al. , 2018 ) led to a significant increase in search efficiency . Here , the weights of the operations in each architecture are shared in a supermodel ( the so-called one-shot model or convolutional neural fabric ( Saxena & Verbeek , 2016 ) ) , which contains an exponential number of sub-networks , each of which represents a discrete architecture . Architectures whose sub-networks share components ( nodes/edges ) also share the weights for these components ’ operations ; therefore , in analogy to DropOut ( Srivastava et al. , 2014 ) , training one architecture implicitly also trains ( parts of ) an exponential number of related architectures . There are a variety of methods on how to conduct NAS by means of the one-shot model ( Brock et al. , 2018 ; Pham et al. , 2018 ; Bender et al. , 2018 ; Liu et al. , 2019 ; Li & Talwalkar , 2019 ) ( see also Appendix B ) , but the final problem is to find the optimal sub-network in this one-shot model . The weight sharing method was used to great effect in DARTS ( Liu et al. , 2019 ) , where it allows a gradient based optimization of both the architectural and the one-shot weights . Subsequent work on DARTS has addressed further lowering the computational and the memory requirements ( Dong & Yang , 2019 ; Xu et al. , 2020 ; Cai et al. , 2019 ; Casale et al. , 2019 ) . One fundamental drawback of the weight sharing method is the fact that the architecture search typically takes place in a lower fidelity model ( e.g. , using less cells and/or cheaper operations ) : the so-called proxy model . After the search , a discrete architecture is derived from the proxy model which is then trained with more parameters — a stage often referred to as architecture evaluation . This poses the question whether the architecture found in the proxy model is also a good architecture in the bigger model , a question studied by several recent works ( Bender et al. , 2018 ; Yu et al. , 2020 ) . 3 A GENERAL FRAMEWORK FOR BENCHMARKING ONE-SHOT NAS . We will now introduce our framework for cheaply benchmarking the anytime performance of oneshot NAS methods . Our main analysis strategy is the following : First , we run the search procedure of various methods and save the architecture weights of the one-shot models for each epoch . Second , we find the discrete architecture at each epoch and query it in NAS-Bench-101 . The last step is not trivial due to the different representations of the search space used in NAS-Bench-101 and standard one-shot methods . Ying et al . ( 2019 ) state that one-shot methods can not be directly evaluated on NAS-Bench-101 . In the following sections we present a mapping between these different search space representations , which eventually enable us to evaluate one-shot methods on NAS-Bench101 . To the best of our knowledge this is a novel contribution of this paper . 3.1 SEARCH SPACE REPRESENTATION . In order to carry out the analysis we propose in this work , we had to construct a search space that only contains discrete architectures that are also contained in NAS-Bench-101 . This allows us to look up any discrete architectures ’ performance in NAS-Bench-101 when the larger model is trained from scratch . Unfortunately , this is non-trivial since the NAS-Bench-101 space does not match the typical space used in one-shot NAS methods . We separately consider the various parts of the search space . Network-Level Topology . In terms of network-level topology , our search spaces closely resemble the models which were evaluated in NAS-Bench-101 . We used the same macro architecture as in NAS-Bench-101 , i.e. , 3 stacked blocks with a max-pooling operation in-between , where each block consists of 3 stacked cells ( see Figure 1 ) . While our final evaluation models exactly follow NAS-Bench-101 in order to be able to reuse its evaluations , our one-shot model only has 16 initial convolution filters , rather than the 128 used in NAS-Bench-101 . This is a common practice to accelerate NAS and used similarly in , e.g. , Liu et al . ( 2019 ) . Cell-Level Topology . The cell-level structure is represented as a DAG , where the input node is the output of a previous cell or the convolutional stem , and the output node is a concatenation of all the previous nodes . In order to have the operation choices still in the intermediate nodes of the DAG , we adapt the choice block motif from Bender et al . ( 2018 ) as depicted in Figure 1 . The edges connecting input , output nodes and choice blocks represent only the information flow in the graph . To have a large and expressive enough search space ( s ) , we introduce the following architectural weights in the DAG edges : • αi , j to edges connecting nodes i < j to choice block j . The input of choice block j is then computed as Ij = ∑ i < j exp ( αi , j ) ∑ i′ < j exp ( α i′ , j ) xi , where xi is the output tensor of node i ( either input node or choice block ) . • γj , k to the edges connecting the input node or choice blocks j < k to the output node k of the cell , where the corresponding feature maps are concatenated : Ok = ⊕j < k exp ( γ j , k ) ∑ j′ < k exp ( γ j′ , k ) xj , where ⊕ is the concatenation operator . Note that the non-linearity applied to the edge weights varies depending on the NAS optimizer used ; e.g . for GDAS ( Dong & Yang , 2019 ) and SNAS ( Xie et al. , 2019 ) it would be a GumbelSoftmax ( Eric Jang & Poole , 2017 ) instead . Choice Blocks . As in Bender et al . ( 2018 ) , each choice block inside the cell can select between the operations in the operations set O of NAS-Bench-101 . In order to find the optimal operation in each choice block via gradient-based one-shot NAS methods , we assign an architectural weight βo to each operation o ∈ O inside the choice block . The output of the choice block j is computed by adding element-wise the latent representations coming from the operations outputs : xj = ∑ o∈O exp ( βo ) ∑ o′∈O exp ( β o′ ) o ( Ij ) , ( 1 ) which is basically the so-called MixedOp in DARTS . NASBench cells contain 1x1 projections in front every operation ( demonstrated in Figure 1 in ( Ying et al. , 2019 ) ) . The number of output channels of each projection is chosen such that the output has the same number of channels as the input . This adaptive choice for the number of channels is incompatible with the one-shot model due to the different tensor dimensionality coming from previous choice blocks . We used 1x1 projections with a fixed number of channels instead .

Summary: This paper introduces a benchmarking framework to evaluate a class of one-shot NAS methods by exploiting the NAS-Bench-101 database. In order to do so, the authors apply techniques from Bender et al. (2018) to match the search spaces in NAS-Bench to one-shot scenarios and weight edges in the search graph allowing for a variable number of edges per cell. These weights are then used to query a larger, discrete architecture from NAS-Bench for the corresponding evaluation errors. The authors motivate the soundness of a unique framework by showing minimal differences between three NAS optimizers.