NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
Authors: Arber Zela
Julien Siems
Frank Hutter
Venue:  (2020)
Keywords: Neural Architecture Search
Deep Learning
Computer Vision

Abstract: One-shot neural architecture search (NAS) has played a crucial role in making
NAS methods computationally feasible in practice. Nevertheless, there is still a
lack of understanding on how these weight-sharing algorithms exactly work due
to the many factors controlling the dynamics of the process. In order to allow
a scientific study of these components, we introduce a general framework for
one-shot NAS that can be instantiated to many recently-introduced variants and
introduce a general benchmarking framework that draws on the recent large-scale
tabular benchmark NAS-Bench-101 for cheap anytime evaluations of one-shot
NAS methods. To showcase the framework, we compare several state-of-the-art
one-shot NAS methods, examine how sensitive they are to their hyperparameters
and how they can be improved by tuning their hyperparameters, and compare their
performance to that of blackbox optimizers for NAS-Bench-101.

Published as a conference paper at ICLR 2020
NAS-BENCH-ISHOTI:
BENCHMARKING AND DISSECTING ONE-SHOT NEURAL ARCHITECTURE SEARCH
Arber Zela'*, Julien Siems Frank Hutter
DDpartment of Computer Science, University of Freiburg Bosch Center for Artificial Intelligence (zelaa, siems), fh] uni monl-freiburg.co de
ABSTRACT
One-shot neural architecture search (NAS) has played crucial role in making NAS methods computationally feasible practice. Nevertheless, there still a lack of understanding on how these weight-sharing algorithms exactly work due to the many factors controlling the dynamics of the process. order to allow scientific study of these components, we introduce a general framework for one-shot NAS that can be instantiated to many recently-introducee variants and introduce general benchmarking framework that draws on the recent large-scale tabular benchmark NAS-Bench- 101 for cheap anytime evaluations of one-shot NAS methods. To showcase the framework, we compare several state-of-the-art one-shot NAS methods, examine how sensitive they are to their hyperparameters and how they can be improved by tuning their hyperparameters, and compare their performance to that of blackbox optimizers for NAS- DAS-Bench-101.
INTRODUCTION
While neural architecture search (NAS) has attracted a lot of attention due to the effectiveness in automatically designing state-of-the-al neural networks (Zoph & Le Zoph et al.) 2018 Real
Zoph 2017 2019), the focus has recently shifted to making the search process more efficient (Pham al.
al.
2018 Elsken al.)
| 2019 Liu et al. 2219 Xie et al. 2019 Cai et al. 2019 Casale et al.) 2019).
The most crucial concept which led to a reduction in search costs to the order of single function
evaluation is certainly the weight-sharing paradigm: Training only single large architecture (the
one-shot model) subsuming all the possible architectures in the search space (Brock et al.) 2018
Pham et al. 2018).
Despite the great advancements of these methods, the exact results of many NAS papers are often
hard reproduce (Li & Talwalkar 2019 Yu al.) 2020 Yang al. 2020). This is result of several factors, such as unavailable original implementations. differences in the employed search spaces, training or evaluation pipelines, hyperparameter settings, and even pseudorandom number seeds (Lindauer & Hutter] 2019). One solution to guard against these problems would be a common library of NAS methods that provides primitives construct different algorithm variants, similar to what as RLlib (Liang et al., 2017) offers for the field of reinforcement learning, Our paper makes a first step into this direction.
Furthermore, experiments in NAS can be computationally extremely costly, making it virtually im- possible to perform proper scientific evaluations with many repeated runs to draw statistically robust conclusions. To address this issue, (Ying al. (2019) introduced NAS-Bench-100 101, large tabular benchmark with 423k unique cell architectures, trained and fully evaluated using a one-time ex- treme amount of compute power (several months on thousands of TPUs), which now allows to cheaply simulate an arbitrary number of runs of NAS methods, even on a laptop. enabled a comprehensive benchmarking of many discrete NAS optimizers (Zoph & Le 2017 Real et al. 2019). using the exact same settings. However, the discrete nature of this benchmark does not.
101
"Equal contribution

---

Published as a conference paper at ICLR 2020
allow directly benchmark one-shot NAS optimizers (Pham al. 2018 Liu al. 2019 Xie al. 2019 Cai et al., 2019). In this paper, we introduce the first method for making this possible.
Specifically, after providing some background (Section 2). make the following contributions:
1. We introduce MAS-Bench--Shoo a novel benchmarking framework that allows us to reuse the extreme amount of compute time that went into generating cheaply benchmark one-shot NAS methods. Our mapping between search space representations novel to the best of our knowledge and it allows querying the performance of found architec- tures from one- shot NAS methods, contrary to what is claimed by Ying et al. (2019). Specifically, it allows us to follow the full trajectory of architectures found by arbitrary one-shot NAS methods at each search epoch without the need for retraining them individually, allowing for careful and statistically sound analysis (Section 3
(Ying al. (209) to
2. We introduce general framework for one-shot NAS methods that can be instantiated to many recent one-shot NAS variants, enabling fair head-to saud-to-head evaluations based on a single code base (Section 4).
3. We use the above to compare several state-of-the fit-the-art art one-shot NAS methods, assess the correla- tion between their one-shot model performance and final test performance, examine how sensitive they are to their hyperparameters, and compare their performance to that of black-box optimizers used in NAS MAS--ennc--Ie 101 (Section
We provide our open-souree implementation which expect will also facilitate the reproducibil- ity and benchmarking of other one-shot NAS methods in the future.
BACKGROUND AND RELATED WORK
2.1 NAS-BENCH-101
101 (Ying et al. 2019) is a database of an exhaustive evaluation of all architectures in a constrained cell-structured space on CIFAR-10 (Krizhevsky (009).. Each cell is represented as a directed acyclic graph (DAG) where the nodes represent operation choices and the edges represent the information flow through the neural network (see also Figure and Section 3.T). To limit the number of architectures in the search space, the authors used the following constraints
the cell: 3 operations in the operation set O = {3x3 convolution, Ixl convolution, 3x3 max-pool]. at most nodes (this includes input and output node, therefore 5 choice nodes) and at most edges.
These constraints, and exploiting symmetries, reduced the search space to 423k unique valid archi- tectures. Each architecture was trained from scratch three times also obtain measure of variance. In addition, each architecture was trained for 4, 12, 36 and 108 epochs; for our analysis, we mainly used the results for models trained for 108 epochs, if not stated otherwise.
2.2
Concurrently to this work, Dong & Yang (220)) released NAS-Bench-10P 102, which is another NAS benchmark that, differently from NAS-Bench-100 101, enables the evaluation of weight-sharing NAS methods. Their search space consists of a total of 15,625 architectures, which is exhaustively evalu- ated on image classification datasets. Similarly to Zela et al. (2000) and this work, Dong & Yang 2020) found that architectural overfitting occurs for DARTS for all their datasets.
While MAS-Bench-109 102 and this work go towards the same direction, they differ in many ways:
1. They use extensive computation to create a new benchmark (with 15.625 architectures), while devise novel reformulation reuse the even much more extensive computation of the NAS- Bench-100 101 dataset ( 120 TPU years) to create three new one-shot search spaces with the larges one containing 663.648 architectures. This required zero additional computational cost.
2. We show that it is possible to reuse the graph representation in HAS-Bench-100 to run one-shot NAS methods; this requires changes to the one-shot search space, but allows a mapping which can be used for architecture evaluation.
2

---

Published as a conference paper at ICLR 2020
Search
Evaluation
Inpus
Call
Call
Blokk
Dan
Discretize
Cho
Discretize
NASBench
Figure I: Overview of the MAAS--Bech--SSA analysis strategy. The one-shot model we construct only contains discrete architectures that are elements of NAS-Bench- 101 (Ying al. 2019). The cell architecture chosen is similar to that Bender et al. (2018), with each choice block containing operation decision. Note that NAS-Bench 101 does not contain separate reduction cell type. Plot on the right from Ying et al. 2019) (Best viewed color).
They evaluate their search space on 3 image classification datasets, while we introduce 3 different search spaces (as sub-spaces of PuS--Bench-10 with growing complexity.
2.3 ONE-SHOT NEURAL ARCHITECTURE SEARCH
The NAS problem can be defined as searching for the optimal operation (e.g. in terms of valida- tion error of architectures) out of the operation set O in each node of the DAG and for the best connectivity pattern between these nodes.
Designing architectures for optimized accuracy to comply with resource constraints led to sig- nificant breakthroughs on many standard benchmarks (Pham et al. 2018 Zoph & Le, 2017, Brock et al. 2018 Liu et al.) 2019 Cai et al., 2019 Elsken et al. 2019). While early methods were compu- tationally extremely expensive (Zoph & Le 2017). the weight-sharin
paradigm (Brock et al. 2018 Pham et al. 2018) led to a signilicant increase in search efficiency. Here, the weights of the opera- tions in each architecture are shared in a supermodel (the so-called one-shot model or convolutional neural fabric (Saxena & Verbeek 2016)), which contains exponential number of sub- sub-networks, each of which represents a discrete architecture. Architectures whose sub-networks share compo- nents (nodes/edges) also share the weights for these components' operations; therefore, in analogy to DropOut (Srivastava et al.) 20144), training one architecture implicitly also trains (parts of) an ex- ponential number of related architectures. There are a variety methods how conduct NAS by means of the one-shot model (Brock et al,, 22118) Pham et al. 2018 Bender et al. 2018: Liu et al.) 2019 Li & Talwalkar 2019) (see also Appendix but the final problem to find the optimal sub-network in this one- shot model.
The weight sharing method was used great effect DARTS (Liu et al. 2019), where it allows gradient based optimization of both the architectural and the one-shot weights. Subsequent work on DARTS has addressed further lowering the computational and the memory requirements (Dong & Yang 2019 Xu et al. 2020 Cai et al. 2019 Casale et al. 2019).
One fundamental drawback of the weight sharing method is the fact that the architecture search typically takes place in lower fidelity model (e.g.. using less cells and/or cheaper operations): the so-called proxy model. After the search, a discrete architecture is derived from the proxy model which is then trained with more parameters This poses the question whether the architecture found in the proxy model is also a good architecture in the bigger model, question studied by several recent works (Bender al. 2018 Yu al. 2020
stage often referred to architecture evaluation.
3

---

Published as a conference paper at ICLR 2020
3 A GENERAL FRAMEWORK FOR BENCHMARKING ONE-SHOT NAS
We will now introduce our framework for cheaply benchmarking the anytime performance one- shot NAS methods. Our main analysis strategy is the following: First, we run the search procedure of various methods and save the architecture weights of the one-shot models for each epoch. Second, find the discrete architecture at each epoch and query in NAS-Bench- trivial due to the different representations of the search space used in NAS Beee--Beeehhieee 101 and standard one-shot methods. Ying al.] (2019) state that one-shot methods cannot directly evaluated on MAS-Bench-101. In the following sections we present mapping between these different search space representations, which eventually enable us to evaluate one-shot methods on NAS-Bench- 101. To the best of our knowledge this is novel contribution of this paper.
The last step is not
3.1 SEARCH SPACE REPRESENTATION
In order to carry out the analysis we propose in this work, we had to construct a search space that only contains discrete architectures that also contained in NAS-Bench-IDe This allows us look up any discrete architectures' performance in NAS-Bench- 101 when the larger model is trained from scratch. Unfortunately, this is non-trivial since the NAS-Bench-101 space does not match the typical space used in one-shot NAS methods. We separately consider the various parts of the search space.
Network-Level Topology. terms of network-level topology, our search spaces closely resemble the models which were evaluated in MusS-Bench-100 We used the same macro architecture as in NAS-Bench-10%) i.e., stacked blocks with max-pooling operation in-between, where each block consists of 3 stacked cells (see Figure D While our final evaluation models exactly follow NAS-Bench-10) 101 in order to be able to reuse its evaluations, our one-shot model only has 16 initial convolution filters, rather than the 128 used in PAS-Bench-101. This is a common practice to accelerate NAS and used similarly in, e.g.. Liu et al. (2019).
Cell-Level Topology. The cell-level structure is represented
DAG, where the input node is the output of a previous cell or the convolutional stem, and the output node is a concatenation of all the previous nodes. In order to have the operation choices still in the intermediate nodes of the DAG, adapt the choice block motif from Bender et al.] (2018) as depicted in Figure The edges connecting input, output nodes and choice blocks represent only the information flow in the graph. To have large and expressive enough search space(s), we introduce the following architectural weights in the DAG edges:
at to edges connecting nodes i < j to choice block j. The input of choice block is then exp(a+-)) x², where x² is the output tensor of node i (cither input computed as 1 = Σiej Eper exp(at's) node or choice block).
git the edges connecting the input node or choice blocks j < k to the output node of the cell, exp(1.*) where the corresponding feature maps are concatenated: O+ = =@ Eyeer explyp wya0,, where @ is the concatenation operator.
Note that the non-linearity applied to the edge weights varies depending on the NAS optimizer used; e.g. for GDAS (Dong & Yang. 2019) and SNAS (Xie al. 2019) it would be Gumbel- Softmax (Eric Jang & Poole 2017) instead.
Choice Blocks. As in Bender al. (008), each choice block inside the cell can select between the operations in the operations set O of -OS-Bench--OL. In order to find the optimal operation in each choice block via gradient-based one-shot NAS methods, we assign an architectural weight g to each operation E O inside the choice block. The output of the choice block j is computed by adding clement-wiss the latent representations coming from the operations outputs:
exp(f9)
x
Σ DEO Eoroo exp o('))
(1)
which is basically the so-called MixedOp in DARTS. NASBench cells contain 1x1 projections in front every operation (demonstrated in Figure in (Ying et al. 2019). The number of output channels of each projection is chosen such that the output has the same number of channels as the
4

---

Published as a conference paper at ICLR 2020
input. This adaptive choice for the number of channels is incompatible with the one-shot model due to the different tensor dimensionality coming from previous choice blocks. We used 1xl projections with fixed number of channels instead.
3.2 EVALUATION PROCEDURE
By means of these additional weights we do not restrict the possible architectures in the search space to contain only a fixed number of edges per cell, as done for example in Zoph et al.) (2018),
Pham et al. (2018), Liu et al. (2019), etc. This requirement would have restricted our architectural decisions heavily, leading only small search spaces.
Table [ shows the characteristics of each search
Table I: Characteristic information of the search spaces.
space. We propose three different search spaces by making different decisions on the number of par- ents each choice block has. The decisions affect the quality and quantity of the architectures contained in each search space. For search spaces note that the sum of the number of parents of all nodes in the search space is chosen to be 9, to match the NAS-Bench- 101 requirement. Search space 1, 2 and 3 have 6240, 29160 and 363648 architectures with loose ends respectively, making search space the largest investigated search space. To the best of our knowledge search space is currently the largest and only available tabular benchmark for one-shot NAS. For details on each search space see Appendix
Search space
3
Node
Node 2
1
Node 3
2
2
1
No. parents
Node 4
2
2
2
Node
2
Output
2
3
2
No. archs.
w/ loose ends
6240
29160
363648
w/o loose ends
2487
3609
24066
A
Given the architectural weights of the cell shown in Figure I we query the test and validation error of the discrete architecture from NAS-Bench-100 101 as follows.
We determine the operation chosen in each choice block by choosing the operation with highest architectural weight.
We determine the parents of each choice block and the output by choosing the top-k edges ac- cording to T for choice block 4 in search space we would choose the top-2 edges as parents).
3. From 1. we construct the operation list and from 2. the adjacency matrix of the cell which we query NAS-Bench-101 for the test and validation error.
Each node in the graph chooses its parents during evaluation following e.g. DARTS (Liu et al. 2019). However, because edges model information flow and the output edges are also architectural decisions there is possibility of node being loose end. These are nodes whose output does not contribute to the output of the discrete cell, as seen in the upper cell under evaluation of Figure [ As result, we can count the number of architectures with without loose ends. Note, that had we chosen the children of each node we could have invalid architectures where node has an output but no input.
A GENERAL FRAMEWORK FOR ONE-SHOT NAS METHODS
Most of the follow-up works of DARTS (Algorithm Tp. which focus on making the search even more efficient and effective, started from the original DARTS codebase and each of them only change very few components compared to DARTS.
5

---

Published as a conference paper at ICLR 2020
Algorithm DARTS
Algorithm 2 PC-DARTS
Algorithm 3 GDAS
I: p Eie, S(af-))a
1: få Mizey S(at))z)
I: = Eic 08 (af
2: of S
2: Oh BiKck B (ty2if))
2: o @e (
3: =² Eoco 3(00)0(1P))
3: x² Euzo
ya M
3: x En (o
4: DAG(L O², ,²))
m DAGiP), o², .²))
4: DAG OF .²))
5: while not converged do
5: while not comvenoo do
5: while oot converged do
6: suupdanetA, V. Crulied
mappianee VA Cvatidd
mupdhae(A V Coatis)
7: mupddaeetu, (evan))
7: mupdateetw, Vw Firain?
7:
cad while
8: end while
8: while
9:
9:
Return
Algorithm 4 Random NAS with Weight shiggtt-sharing
Algorithm 5 ENAS
1: Eiey
1: P Eie; R1
2: of Byck
2: of Djck
3: Eoco [(P)
3: x² Eogo B(2))
4: m -- DAGIP', oF,
4: DAGIP, 0", .
5: while converged
5: while not converged do
6: arch -- sample matformly mndossm)))
6: arch sample woing
7: caupdase wfigh.s....h...oocco......o.... ant. Ctrain)
7: muuddite weiphts sinple inchilecturoorrcc uo Vi cstirie
8:
8:
Updaie controlle
9:
end while
9:
end while
10:
for 100000 do
11:
arch samples aample jandomtan)
11: archesmples ammple/iinng.coirllle.
13: Return srch,sanppes lowest validaaion errce
12:
cnd
13: Return inch E arch samples with lowest valicdation crnce
12:
end
Algorithm 2 and Algorithm 3 highlight these components (relative to DARTS) for PC-DARTS (Xu jet al., 2020) and GDAS (Dong Yang 2019), respectively. For example, when comparing PC- DARTS and DARTS, the only difference in our benchmark is the partial channel connections (line 3 of Algorithm 2 in the choice blocks, which consists of a channel sampling mask M that drops feature maps coming from p. GDAS,
the other hand, replaces the Softmax (S) function in
DARTS by a Gumbel-Softmax (GS), which applies for every architectural weight in A = {a, B3 (lines 1-3 in Algorithm 3). and uses this concrete distribution to sample single paths through the cell during search (line 7 iin Algorithm 33.. Random Search with Weight Sharing (Random WS) (Li & Talwalkar 2019) (Algorithm 4 and ENAS (Pham al.] 2018) (Algorithm (5) do not need the continuous re relaxation in order to conduct the architecture search, instead they sample randomly in RandomWSS WS or from the recurrent neural network controller (line in Algorithm T in ENAS, in order to select the sub-network in the one-shot model to train.
These close correspondences between current one-shot NAS variants provide an opportunity to im- plement all of these variants in the same general code basis. This allows us to (a) automatically guard against any confounding factors when evaluating the strengths and weaknesses of different approaches, and (b) allows us to mix and match the components of different algorithms. We imple- mented all variants in a single code basis, which we are committed to grow into a flexible library of primitives for one-shot NAS methods, and for which we will gladly accept any help the community wants to provide.
Ohe-shot NAS methods in this code basis inherit all the methods and attributes necessary for building the one-shot computational graph from a base parent class. This encapsulation and modularity ensures that all differences in their performance come from a few lines of code, and that all other confounding factors cannot affect these results. This will also facilitate the incorporation other one-shot NAS methods and pinpoint the components that differ in them.
Furthermore, the primitives encoding the search spaces presented in Section 3 are defined separately from the NAS optimizers. This encapsulation will allow researchers to study each of these com- ponents in isolation, experimenting with one of them while being sure that the other one does not change.

---

Published as a conference paper at ICLR 2020
5 NAS-BENCH-ISHOTE AS A BENCHMARK AND ANALYSIS FRAMEWORK
We now demonstrate the use of BAS-Bench- Bench-IShotl as a benchmark for one-shot NAS. We first evaluate the anytime performance of five different one-shot NAS methods: DARTS (Liu et al. 2019), GDAS (Dong& & Yang 2019). PC-DARTS (Xu et al.) 2020). ENAS (Pham et al.) 2018) and Random Search with Weight Sharing (Random WS) (Li & Talwalkar 2019) Afterwards, we investigate the robustness of these one-shot NAS optimizers towards their search hy peprparaaeters and show that these hyperparameters are carefully tuned, the one-shot NAS optimizer can outperform a wide range of other discrete NAS optimizers in our search spaces.
5.1 COMPARISON OF DIFFERENT ONE-SHOT NAS OPTIMIZERS
We ran the NAS search for 50 epochs" using their respective default hyperparameter settings (see Appendix C If not stated otherwise, all the following results were generated by running each ex- periment with six random seeds (0 to 5). All plots show the mean and standard deviation of the test regret queried from MAS--ennch-111 Over the three independent trainings contained in NASBench- 101 for each architecture on each epoch budget we average. The search was done on a single NVIDIA RTX2080Ti using the same python environment. In Figure 2 we report the anytime test regret for each of these methods. Our findings can be summarized as follows:
While the one-shot validation error converged in all cases (except for GDAS due to the temperature annealing) the queried test error from TAS-Bench-101 of the architectures increased at some point, indicating that the architectural parameters overfit the validation set. This phenomenon occurred frequently for DARTS. The same result also previously observed by Zela et al. 2020) on subspaces of the standard DARTS space.
PC-DARTS demonstrated both a stable search and relatively good final performance in all search spaces, notably the best overall performance for search space 3. We attribute this behaviour to the regularization effect present PC-DARTS via the partial channel connections (see /Zela et al.
(2020) and Section 5.3)
Random WS and ENAS mainly explore poor architectures across all three search spaces. This behaviour is result of the small correlation between the architectures evaluated with the one-shot weights and their true performance during architecture evaluation (as queried from NAS-Bench- 101 (see Section 5.2) This correlation directly affects these methods since in the end of search they sample a certain number of architectures (1000 randomly sampled for Random WS and 100 using the learned controller policy for ENAS) and evaluate them using the one-shot model weights in order to select the architecture which is going to be trained from scratch in the final evaluation phase. When running ENAS for 100 epochs in search space 2 (Figure in the appendix) we see that the it performs better than Random WS. ENAS also has stronger correlation between the sampled architectures and the MAS-Bench--I 101 architectures for search space 2 (see Section (.2).
GDAS performs quite robustly across all 3 benchmarks, however, due the temperature annealing of the Gumbel Softmax, it might manifest some premature convergence to a sub-optimal local minimum.
5.2 CORRELATION ANALYSIS
Many one-shot NAS methods, such as ENAS (Pham et al. 2018), NAONet (Luo et al. 2018) or Random WS (Li & Talwalkar] 2019) select the final architecture by means of the one-shot parame- ters. In order to assess if this optimal we computed the correlation between a given architecture's one-shot test error and its respective BAS--ench- 101 test error, for all 4 available budgets in NAS- Bench- 101 on every 10th search epoch. This analysis was performed for all architectures without loose ends in each search space. The only exception is ENAS, for which we decided to evaluate the correlation by sampling 100 architectures (as done by the algorithm after the search has finished to select the one to retrain from scratch) from the controller instead of evaluating every architecture in the search space.
Details on these optimizers can be found Appendix
"This required different amounts of time: DARTS 1st order: 3h, DARTS 2nd order: 4h, GDAS: 1.5h, PC-DARTS: 2h ENAS: 4h, Random-WS: 11h.
7

---

Published as a conference paper at ICLR 2020
PC SAATE
Pa th wore 100.e00000
1w30
(a) Search space
(b) Search space 2
(c) Search space 3
Figure 2: Comparison different one-shot NAS optimizers on the three different search spaces defined on NASBench. The solid lines show the anytime test regret (mean + std), while the dashed blurred lines the one-shot validation error (Best viewed color).
DAIISS sirst order un cutse
GUAS
CD45
CD45
CARTN
PC-ATTS
1855
DVA
EMAS
Runder 155
(a) Search space
(b) Search space 2
(c) Search space 3
Figure 3: Correlation between the one-shot validation error and the corresponding test error for each search space. (Best viewed in color).
As shown in Figure there almost no correlation between the weight sharing ranking and the true one (Spearman correlation coeff. between -0.25 and 0.3) during search for DARTS, PC-DARTS, GDAS and Random WS. Only ENAS shows some correlation for search space and some anticor- relation for search spaces and 3. These results agree with the ones reported by Yu et al. (2020) (who could only do this evaluationx on a small search space) and explain the poor performance of Random WS and ENAS on our benchmarks, since the architectures sampled during evaluation and ranked according to their one-shot validation error are unlikely to perform well when evaluated inde- pendently. To the best of our knowledge this is the first time that an evaluation of this correlation is conducted utilizing such a large number of architectures in the search space, namely 24066 different architectures for search space 3. We added further experiments the correlation between the lower fidelity proxy model used in architecture search and the final model from architecture evaluation in the Appendix H
5.3 ROBUSTNESS OF ONE-SHOT NAS OPTIMIZERS
As already observed by Zela et al. 2020). DARTS tends to become more robust when the inner objective Ltrain in the bi-level optimization procedure has relatively strong regularization factor
8

---

Published as a conference paper at ICLR 2020
Eal"t Bestade
lacird Lobe cood
164
30-
Cpon
Eacc
(a) DARTS
(b) GDAS
(c) PC-DARTS
Figure 4: Iilustration of the impact that Cutout has on the test regret on NAS-Bench-101 and the validation error of the one-shot model using DARTS, GDAS and PC-DARTS on search space 3 (Best viewed in color).
during search. In order investigate the long term behaviour, for this analysis chose every search for 100 epochs instead of 50 epochs. Similarly to Zela et al. (2020), we find that enabling Cutout (DeVries & Taylor| 2017) or increasing the Le factor for the search model weights, has a substantial effect on the quality of the solutions found by the NAS optimizers. See Figure 4 for the Cutout (CO) results and Figure I (in the appendix) for the results with different L2 regularization. Based on these results we observe that:
As DARTS overfits to the validation set the end of search for search space I and 3, applying Cutout during search either keeps the solutions to a good local minimum or reduces the overfitting effect, finding similar the one in Zela et al. (2020).
Interestingly, while Zela et al. 2020) only showed overfitting behavior for DARTS, it may also occur for GDAS (Figure 13 in the appendix) and PC-DARTS (Figure [4] in the appendix), which indicates that this might be intrinsic property of these methods due to the local updates in the architecture space.
PC-DARTS, there is already a strong regularization effect as a result of the partial channel connectivity, which explains the robust behavior of this optimizer and that its results deteriorate on average as we increase L2 regularization.
5.4 TUNABILITY OF ONE-SHOT NAS HYPERPARAMETERS
Next to the hyperparameters used in the evaluation pipeline, one-shot NAS methods have several hyperparameters of their own, such the regularization hyperparameters studied in Section 5.3 learning rates, and other hyperparameters of the search phase.
Naively tuning these hyperparameters with the one-shot validation error as the objective would lead to sub-optimal con- figurations, since, as we saw, this met- ric is not good indicator of generaliza- tion. In our proposed benchmarks, we can tune these hyperparameters of the NAS optimizer to minimize the validation error queried from so, we aim to shed more light onto the in- fluence these hyperparameters have during the one-shot search, and to study the sen- sitivity of the NAS method towards these hyperparameters.
Space 3
sosers perrs
10
bost
10->
101. By doing
10a
10
16a
10
simulated walletock k time [s]
Figure Optimizing the hyperparameters of one-shot optimizers with BOHB on search space 3. (best viewed in color). Results for search space and are shown in Figure 16
To this end, we constructed configura- tion spaces of increasing cardinality, CS1,
CS2 and CS3 (see Appendix(Floor details), which only include hyperparameters controlling the NAS process. We chose BOHB (Falkner et al. (0018) see Appendix F for details) as the hyperparameter

---

Published as a conference paper at ICLR 2020
optimization method and DARTS our NAS method to be tuned across all configuration spaces, since in our benchmarks it was more brittle than PC-DARTS and GDAS. We provide the results in Appendix F22
In order to compare the tunability of NAS methods, we used BOHB to optimize all of DARTS, PC- DARTS, and GDAS, starting from their respective default hyperparameter settings. Figure 5 shows the anytime test regret of the architectures found by the respective NAS method's s configurations tried by BOHB; as the figure shows, PC-DARTS and GDAS start with much more robust hyperparameter settings, but DARTS can also be tuned to perform as well or better. We note that carrying out this optimization on JuS--Bnnh--Jee days to 1 day on 16 GPUs.
reduced the time for tuning DARTS from a simulated 45 GPU
Figure 5 also provides an evaluation of DARTS, GDAS and PC-DARTS compared to the state-of-art discrete NAS optimizers used by Ying et al. (019)) (such as RI. RE, and HPO methods). Since these one-shot NAS methods are much faster than black-box methods, it is possible to tune them online using BOHB and the resulting BOHB-(DART)
GDAS, PC-DARTS] typically still yields better performance over time. Note that we never use the validation set split used to evaluate the individual architectures in NAS-Bench-10 101 during the architecture search. This subset with 10k examples is only used to compute the objective function value that BOHB optimizes. Therefore, the one-shot optimizers use 20k examples for training and 20k for the architectural parameter updates. shows the simulated wall-clock time: tsim = &eearch + train. where seearch
The x-axis in Figures shoos is the time spent during search by each configuration and Itrain is the training time for 108 epochs (queried from NAS-Bench-101 of the architectures selected by the one-shot optimizer. From this experiment, make the following observations:
On all search spaces the architectures BOHB-DARTS found outperformed the architectures found by the default DARTS configuration by factor of 7 to 10.
The found robust configurations did not only avoid overfitting in the architectural level, but they also typically outperformed the architectures found by state-of-art discrete NAS optimizers used by Ying al. 2019) (such as RL, RE, and HPO methods).
The multi-fidelin in BOHB does not only accelerate the hyperparameter optimization procedure, but in this case also allows to determine the sufficient number of epochs to run the NAS optimizer in order get an optimal architecture. In fact, the best configuration on each of the incumbents comes usually from the lowest budget, i.e. 25 search epochs.
The most robust configuration on each of the search spaces are typically the ones with a large reg- ularization factor, relative to the default value in Liu et al. (2019) (see Figure i9 the appendix).
CONCLUSION AND FUTURE DIRECTIONS
We proposed B--.B--enncctttte a set of new benchmarks for one-shot neural architecture search which allows to track the trajectory and performance of the found architectures computationally cheaply. Using our analysis framework, we compared state-of-the-an one-shot NAS methods and inspected the robustness of the methods and how they are affected by different hyperparameters. Our framework allows fair comparison of any one-shot NAS optimizer and discrete NAS optimizers without any confounding factors. We hope that our proposed framework and benchmarks will facil- itate the evaluation of existing and new one-shot NAS methods, improve reproducibility of work in the field, and lead to new insights on the underlying mechanisms of one-shot NAS.
ACKNOWLEDGMENTS
The authors acknowledge funding by the Robert Bosch GmbH, support by the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation program through grant no. 716721, and by BMBF grant DeToL.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Understand- ing and simplifying one-shot architecture search. International Conference on Machine Learn- ing, 2018.
Andrew Brock, Theo Lim, J.M. Ritchie, and Nick Weston. SMASH: One-shot model architecture search through hypemetworks. In International Conference Learning Representations, 2018. URL https::
net forum? Eo
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search target task and hardware. International Conference on Learning Representations, 2019.
Francesco Paolo Casale, Jonathan Gordon, and Nicolo Fusi. Probabilistic neural architecture search. arXiv preprint arXiv:1902.055 16, 2019.
Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 04552, 2017.
Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four gpu hours. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1761 1770, 2019.
Xuanyi Dong and Yi Yang. Nas-bench- 102: Extending the scope of reproducible neural architec- ture search. In International Conference on Learning Representations, 2020. URL https:
//openreview. net fortt//foumma www.miiddddddxDDDDD.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objectivv neural architecture search via lamarckian evolution. In International Conference on Learning Reprexentations, 2019.
Shixiang Gu Eric Jang and Ben Poole. Categorical reparameterization with gumbel-softmax. In International Conference Learning Representations, 2017.
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: Robust and efficient hyperparameter op- timization at scale. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th Inter- national Conference on Machine Learning, volume 80 Proceedings of Machine Learning Re- search, pp. 1437-1446, Stockholmsmässan Stockholm Sweden, 10-15 Jul 2018. PMLR. URL www.innnsmlllll.... wtt.ggss.....rrrrre.. v00/ Bt..tmm....l...m.mm..mm.mm..m...
Hutter, H. Hoos, and K. Brown. An efficient approach for assessing hyperparameter importance. In E. Xing and T. Jebara (eds.), Proceedings of the 31th International Conference on Machine Learning, (ICML' 14), pp. 754-762. Omnipress, 2014.
K. Jamieson and A. Talwalkar. Non-stochastic best arm identification and hyperparameter optimiza- tion. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics (AISTATS), 2016.
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
L. Li, K. Jamieson, G. DeSalvo, Rostamizadeh, and Talwalkar. Hyperband: Bandit-based con- figuration evaluation for hyperparameter optimization. In International Conference on Learning Representations, 2017.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Avis, Israel, July 22-25, 2019, pp. 129, 2019. URL patt:://aual wtt.aaawii..ruua1/a/.... pxoceelinggs/ppper//222
pdf
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Joseph Gonzalez, Ken Gold- berg, and Ion Stoica. Ray rllib: A composable and scalable reinforcement learning library. CoRR, abs/1712.09381. 2017. / 09381

---

Published as a conference paper at ICLR 2020
Marius Lindauer and Frank Hutter. Best practices for scientific research on neural architecture search. arXiv preprint arXiv:1909.02553, 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019.
Renqian Luo, Fei Tian, Tao Qin, and T. M. Liu. Neural architecture optimization. In NeurIPS, 2018.
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoe V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In International Conference on Machine Learning. 2018
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoe V. Le, and Alexey Kurakin. Large-scale scale evolution of image classifiers. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2902-2911, International Conven- tion Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. httttt:: htttt::///ppceedgggg..ooo mmr. press/v70/ wwt.:///reeaaa....a.... html
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe Le. Aging Evolution for Image Classi- fier Architecture Search. In AAAI, 2019.
Shreyas Saxena and Jakob Verbeek. Convolutional neural fabrics, In D. D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 4053-444400 4061. Curran Associates, Inc., 2016.
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way prevent neural networks from overfitting. Journal of Machine Learning Research, 15: 1929-1958, 2014.
B. Williams, Santner, and W. Notz. Sequential design of computer experiments to minimize integrated response functions. Statistica Sinica, 2000.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: stochastic neural architecture search. In International Conference on Learning Representations, 2019.
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong- Pe-darts: Partial channel connections for memory-eflicient architecture search. In International Conference on Learning Representations, 2020. URL/ttps:: / wwt.rriwwwwweeeeeeeeeeeeeeeeeeee forrm?
1d-BJ18634tPr
Antoine Yang, Pedro M. Esperana, and Fabio M. Carlucci. Nas evaluation is frustratingly hard. In International Conference Learning Representations, 2020. htt.hhtttss://peerree
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS- bench-101: Towards reproducible neural architecture search. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7105-7114, Long Beach, Cali- fornia, USA, 09-15 Jun 2019. PMLR. URL httt:::/proceee//eee htwp:///........r.s...ee pt..sssss/..r. ying 9a. html
ml
Kaicheng Yu, Christian Sciuto, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. In International Conference on Learning Repre- sentations, 2020. URL http:tttts//:peeeeeeee wtt.iiiwwwweeeeeeemeom forummid
Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hut- ter. Understanding and robustifying differentiable architecture search. In International Confer- ence on Learning Representations, 2020. URL
htwprerrrreee...eeffouurn....... net fottt/ffrrmmidd
H1 cDNyrKDS
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In Interna- tional Conference on Learning Representations (ICLR) 2017 Conference Track, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition. In Conference on Computer Vision and Pattern Recognition, 2018.
12

---

Published as a conference paper at ICLR 2020
A DETAILS ON SEARCH SPACES
Search space The main characteristic of this search space is that the number of parents for each choice block and output has to be exactly 2 (apart from choice block which is only connected to the input). Because of this requirement one choice block had to be discarded that would exceed requirement to have at most edges. The total distribution test error in shown Figure 6a is the smallest search space discussed in this work.
Search space 2 This search space is related to search space I in that it only consists of 4 in- termediate nodes, in contrast the output has three parents and nodes and 2 only one parent. This increases the number of architectures in this space compared to search space 1. The test error distribution is shown in Figure 6b
Search space 3
All 5 intermediate nodes are used in this search space, making this search space the largest, but also the search space where each node has average the least number of parents. The test error distribution is shown in Figure 6c
B OPTIMIZERS
weighted continuous relaxation over the operations learn architecture by solving a bilevel optimization problem. The training dataset is split in two parts, one used for updating the parameters of the operations the one-shot model, and the other to update the weights appended to operations, that determine the importance of that operation.
DARTS (Liu al. 2019)
For evaluation, choose the parents of each choice block based on the highest architectural weights and the number of parents for that choice block given by Table [ operation from the choice block.
We pick the highest weighted
GDAS (Dong & Yang, 2019) modifies DARTS, such that individual paths are sampled differen- tiably through each cell using Gumbel-Softtax (Eric Jang & Poole 2017) adapt the architecture weights. This reduces the memory overhead created by DARTS as only the sampled paths have to be evaluated. GDAS uses the same search space and evaluation procedure DARTS.
PC-DARTS XXu et al., 2020) reduces the memory overhead by only evaluating a random fraction of the channels with the mixed-ops. The authors argue that this also regularizes the search as lowers the bias towards weight-free operations such as skip-connect and max-poote max-pooling, which are often preferred early on DARTS search. In addition to partial channel connections, the authors propose edge normalization, which adds additional architectural parameters to the edges connecting to an intermediate node. This is done to compensate for the added fluctuations due to the partial channel connections. These additional weights already part of the search spaces proposed in this paper.
Random Search with Weight Sharing (Random WS) (Li Talwalkar 2019) randomly samples architectures from the one-shot model for each training mini-batch and trains only the selected subset of the one-shot model on that mini-batch. Differently from DARTS, PC-DARTS and GDAS,
Desitaaion
Toallider bnteom
Tellera
Tailano
(a) Search space
(b) Search space 2
(c) Search space
Figure 6: Distribution of test error in the search spaces with loose ends.
13

---

Published as a conference paper at ICLR 2020
Random WS does not require validation set, since there architectural weights that need to be updated. For evaluation Random WS samples 1000 architectures from the search space and evaluates each for only small number of batches on the validation set using the optimized weights of the one-shot model corresponding to the sub-networks. Then the architectures with the lowest one-shot validation error chosen and fully evaluated on the validation set. The overall best architecture is returned.
ENAS (Pham al. 20188) similarly to Random WS samples sub-networks of the one-shot model, however by means of a recurrent neural network (RNN) controller rather than randomly. As the search progresses the parameters of the RNN controller are updated via REINFORCE (Williams et al. 2000) using the validation error of the sampled architectures a reward. This way the sampling procedure handled in a more effective way.
C HYPERPARAMETERS
If not stated otherwise the following hyperparameters were used for all our experiments. We used batch size of 96 throughout for DARTS, GDAS and PC-DARTS as the search spaces are small enough to allow it and this reduces the randomness in the training, which makes the comparison between optimizers easier. Random WS was trained with batch size of 64. All other hyperparam- eters were adapted from DARTS (Liu et al. 2019).
D COMPARISON OF OPTIMIZERS OVER DIFFERENT BUDGETS
OANER Disandes
CARTS Sutueded
(a) Search space
(b) Search space 2
(c) Search space 3
Figure 7: Comparison of different One-Shot Neural Architecture optimizers on the three different search spaces defined on NAS-Bench- 101 over 100 epochs.
EMR Sen UIe
Dal's Bet
44204
E4307
24207
(a) Search space
(b) Search space 2
(c) Search space
Figure 8: Comparison of DARTS first and second order on the three different search spaces defindd on NAS-Bench- 101 for 25 epochs.
14

---

Published as a conference paper at ICLR 2020
E REGULARIZATION
E CUTOUT
Interestingly, for GDAS the validation enror of the one-shot model is closer linked the test regret on NASBench as that is the case for DARTS as shown in Figure ! This is particularly striking in search space 2 in which the local minimum attained by the one-shot validation error well aligned with the minimum of the test regret. It is interesting to note that GDAS very quickly overfits on this search space, since the validation error increases usually at around 50 epochs. This may be related to the linearly decreasing temperature schedule for the gumbel-softmax (Eric Jang & Poole 2017) from 10 to as proposed by Dong & Yang (2019). As the temperature decreases the operations with higher architectural weights will be sampled more often leading overfitting the architectural
level as demonstrated by the increasing one-shot validation error. Cutout has little impact on the search phase of GDAS (Figure
Ex30
2wa0-
10"
(a) Search space
(b) Search space
Figure 9: Comparison of the effect of using Cutout during architecture search on GDAS for search space and 2.
Acoel's Besiabet Vaiial
capr
(a) Search space
(b) Search space
Figure 10: Comparison of the effect of using cutout during architecture search on PC-DARTS for search space and 2.
For PC-DARTS (Figure 10 cutout regularization helped find better architectures in particular in search space and This underlines the fact that strong regularization on the architectural level via partial channel connections can be effectively supported by cutout regularization on the training loss. Second order optimization as proposed by DARTS has no significant benefit for PC-DARTS and often decreases the performance.
E.2 L2 REGULARIZATION
Increasing the Le regularization has a positive effect on the found architectures for GDAS in search space 1 and 2 as shown in Figure 13 However, in search space 3 setting the weight decay to 81e-1 has the effect of making the model unstable.
For PC-DARTS lowering the La regularization had overall positive effect across all search spaces (Figure 14). However, this made the training also less stable as demonstrated by search space
(Figure 14a
15

---

Published as a conference paper at ICLR 2020
Sreep
AN Scow (7EA
Lock
(a) DARTS
(b) GDAS
(c) PC-DARTS
Figure 11: Illustration of the impact that weight decay has on the test regret on MAAS-Bench-10 101 and the validation error of the one-shot model using DARTS, GDAS and PC-DARTS on search space (Best viewed in color).
(a) Search space
(b) Search space
Figure 12: DARTS first order w/o cutout trained with different levels of L2 regularization for search space and
The
(a) Search space
(b) Search space
Figure 13: Comparison of the effect of using different values of weight decay during architecture search on GDAS for search space and
Ariges trea
gin Dicp
(a) Search space
(b) Search space
Figure 14: Comparison of the effect of using different values of weight decay during architecture search on PC-DARTS for search space I and 2.
16

---

Published as a conference paper at ICLR 2020
F BOHB DETAILS
BOHB FFlknnr et al. 2018) is a combination of Bayesian Optimization (BO) and Hyperband (HB) (Li al., 2017). Hyperband uses SuccessiivHallvnn (SH) (Jamieson & Talwalkar 2016) to stop poorly performing trainings early. Success Halving starts trainings with an initial budget and ad- vances the top fraction (1/1)) of them the next stage with 7) higher budget. Hyperband uses this subroutine to evaluate many uniformly at random sampled configurations on small budgets. The budgets and scaling factors are chosen such that all SuccessiveHalving evaluations take approxi- mately the same time. BOHB combines Hyperband with Bayesian Optimization by using proba- bilistic model to guide the search towards better configurations. As result, BOHB performs as well as Hyperband during early optimization, but samples better configurations once enough samples are available to build a model.
F.1 SETUP
We ran BOHB for 64 iterations of SuccessiveHalving (Jamieson & Talwalkar 2016) on 16 paral- lel workers, resulting in 280 full function evaluations. In our experiments we use the number of epochs that the one-shot NAS optimizers run the search as the fidelity used by BOHB and optimize the validation eгror after 108 epochs of training queried from MAS-Bench-101. Namely, we use min budget
25 epochs, max budget = 100 epochs and 17 the number of epochs used for the architecture search. Additionally. we never use the validation set split used to evaluate the individual architectures in NAS-Benchi 101 during the architecture search. Therefore, each one-shot NAS optimizer will use 20k examples for training and 20k for search. The x-axis in Figures 13 16 17 18 shows the simulated wall-clock time: tasm = tsearch + trrain where search is the time spent during search by each NAS algorithm configuration and train is the train- ing time for 108 epochs (queried from NAS-Bench-100 Bench-101) of the architectures selected by the NAS optimizers.
BOHB. Note that this is only
We build 3 configuration spaces with different cardinality and which include hyperparameters af- fecting the architecture search process. The spaces follows:
1. CS1 = {L2, CO (0.prob}
2. CS2 = (L2, -r-poob, br}
CS3 = (22, CO.prob, Ir, moment, COlln, bizeh_size, grad archJr, L2)
F.2 RESULTS
smodrod
(a) Search space
(b) Search space 2
(c) Search space 3
Figure 15: Test regret of architectures found with DARTS (Ist order) configurations sampled by BOHB on CSI. All the lines except the BOHB-DARTS one show the tsdd of the best archi- tecture from 500 search repetitions (Best viewed in color).
Interestingly, optimizing on CS2 led not only a more robust configuration of DARTS, but also in outperforming a state-of-the-art discrete NAS optimizer such as Regularized Evolution (RE) (Real) cet al. 2019). The found solutions by BOHB also outperform every other one-shot optimizer used throughout this paper with their default settings. Including the learning rate in the configuration space was crucial to achieve such a performance. Figure 16 shows the results when running BOHB with the same settings on CSL. Note that none of the sampled configurations outperforms RE. On the other hand, increasing the cardinality of the configuration space requires many more samples to build good model. [7 shows the results when optimizing with BOHB CS3. Even though
17

---

Published as a conference paper at ICLR 2020
the learning rate was inside this space, again none of the sampled configurations is better than the discrete NAS optimizers.
Space
Space
KOWN norrs KCHNNEDIS
baxt ragge
10
b0st: rugrot
18-1
20.
10f
304
(a) Search space
(b) Search space 2
Figure 16: Test regret of architectures found with DARTS, GDAS and PC-DARTS (Ist order) con- figurations sampled by BOHB CS2. All the lines except BOHB-DARTS. BOHB BOHB-GDAS and BOHB-PC-DARTE show the meantistd of the best architecture from 500 search repetitions (Best viewed in color).
Space
Space
adttte nublock LIwEX
Aiddond clcc Cree
(a) Search space
(b) Search space 2
(c) Search space 3
Figure 17: Analogous to Figure 15 with the only difference being that here we optimize on CS3.
Space
Space
Spoce
12"
creel
creell
(a) Search space
(b) Search space 2
(c) Search space 3
Figure 18: Test regret of architectures found with DARTS, GDAS and PC-DARTS (2nd order) configurations sampled by BOHB CS2.
F.3 TRANSFERABILITY BETWEEN SPACES.
The following Table 2 shows the performance of the best found configuration search space for 50 epochs by BOHB when transferred to search spaces and The results show the mean and standard deviation of the architectures found by 6 independent search runs with the respective optimizers. We can see that there is no clear pattern on what is transferable where.
G HYPERPARAMETER IMPORTANCE
To better understand the configuration space which we evaluated with BOHB we use functional analysis of variance (fANOVA) (Hutter et al. 2014). The idea is to assess the importance of individ- ual hyperparameters by marginalizing performances over all possible values other hyperparameters could have taken. The marginalization estimates are determined by random forest model which
18

---

Published as a conference paper at ICLR 2020
Table 2: Results of architectures found on search space 1 and 2 with the best found configuration for 50 epochs by BOHB on search space 3.
Test regret
Optimizer
Search space
Search Space 2
Default config.
1.252e-2 + 0.0
0.864e-2 + 0.023e-2
DARTS
Transferred config.
1.045e-2 +: 0.
0.992e-2 + 0.21 1e-2
Default config-
1.252e-2 + 0.0
0.871e:2 + 0.0
GDAS
Transferred config.
0.831e-2 + 0.045e-2
Default config.
1.104e-2 + 0.088e-2
1.133e-2 + 0.404e-2
PC-DARTS
Transferred config.
5.843e-2 4.118e-2
0.992e-2 + 0.087e-2
was trained on all configurations belonging to specific budgets during the BOHB optimization pro- cedure.
Figure 19 shows the interaction between the Cutout (CO) and L2 factor when optimizing on CS2, DARTS Ist order, for search space 1, and 3. It should be noted that the best found configuration involves at least one relatively high value of one of the regularizers in CS2.
2JM
(a) SS 1, Budget: 25 epochs
(d) SS 2, Budget: 25 epochs
(g) SS 3, Budget: 25 epochs
Sonce
Cosce
(b) SS 1, Budget: 50 epochs
(c) SS 2, Budget: 50 epochs
(h) SS 3, Budget: 50 epochs
oue
000
SS 1, Budget: 100 epochs
SS 2, Budget: 100 epochs
(i) 3, Budget: 100 epochs
Figure 19: Parameter importance for two hyperparameters, Cutout (CO) and L2 regularization (CS2) across different training epochs and search spaces (SS).
19

---

Published as a conference paper at ICLR 2020
H CORRELATION BETWEEN THE ARCHITECTURE SEARCH MODEL AND THE ARCHITECTURE EVALUATION MODEL
As further experiment wanted test how strongly the validation error of the models used in architectures search and architecture evaluation are correlated. For this we sampled 150 architectures from search space and trained this architecture the proxy model. During training we compute the Spearman rank correlation between the validation eπror of the proxy model and the full architecture evaluation as queried from NAS-Bench- 101. The results are shown in Figure 20 Note that cells in the proxy was used for all of our previous experiments as it is also used by the NAS-Bench 101 models.
For 9 cells (Figure 20c in the proxy model, we find that increasing the total number of channels leads to stronger correlation between the proxy model and the full architecture. However, increasing beyond 16 channels leads to decrease correlation. For 3 cells (Figure the strongest anytime correlation was interestingly found using only 2 initial channels, with more channels leading to worse performance at the beginning and no better final performance. The results suggest that there exists a set of good combinations between the number of cells and the initial number of channels to get the maximum correlation between the search and evaluation model. As future work we plan to investigate this relationship, which could eventually lead to a more effective bandit-based NAS method.
RCC.
T>x30
(a) cells (1 cell per stack)
(b) cells (2 cells per stack)
(c) 9 cells (3 cells per stack)
Figure 20: In this experiment we varied the total number of cells within [3, 6, 9] and the number of initial channels of the proxy model within [2, 4, 8, 16, 36).
20


Reviews:
[{"review_id": "SkljhB5ndB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this submission, the authors present a benchmark NAS-Bench-1Shot1 for one-shot Network Architecture Search. The presented benchmark reuses the existing NAS-Bench-101 that only contains discrete architectures, and the authors give a method to discretize architectures to match NAS-Bench-101. Also, the authors claim that they introduce a general framework for one-shot NAS methods.\n\nOverall speaking, the technique contribution and novelty of this submission requires further improvement, and I suggest to rejecting this submission. The reasons are detailed as follows:\n\n1) The main contribution of this submission is the benchmark NAS-Bench-1Shot1, which can benefit future research for one-shot NAS. However, this benchmark is based on NAS-Bench-101, and the difference is that NAS-Bench-101 only contains discrete architecture, while the presented NAS-Bench-1Shot1 has a component to discretize architectures and then match to NAS-Bench-101. The novelty may not be enough.\n\n2) The authors claim that they introduce a general framework for one-shot NAS methods. Personally, I think this is over-claimed. There are several variants of DARTS, and the authors just implement these variants and DARTS itself in a unified code base. It would be a true general framework if it can also work with other one-shot NAS methods such as ENAS and the rest ones.\n\nGiven these, this submission requires further improvement, especially in terms of technique contribution and novelty. \n\nOne interesting thing is that the authors try to use HPO method such as BOHB to tune the hyperparameter of NAS methods.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Byxt3MX9qH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a benchmarking framework to evaluate a class of one-shot NAS methods by exploiting the NAS-Bench-101 database. In order to do so, the authors apply techniques from Bender et al. (2018) to match the search spaces in NAS-Bench to one-shot scenarios and weight edges in the search graph allowing for a variable number of edges per cell. These weights are then used to query a larger, discrete architecture from NAS-Bench for the corresponding evaluation errors. The authors motivate the soundness of a unique framework by showing minimal differences between three NAS optimizers. \nIn their analysis, they also corroborate the findings of Sciuto et al. (2019) on a larger scale, showing little generalization of the ranking from one-shot validation to the final architecture. Finally, the authors also use this framework to show that a fine-tuned baseline compares favorably to SOTA methods, and that large regularization factors lead to more robust configurations.\n\nThe paper is well-written and introduces a framework to cheaply compare one-shot NAS optimizers based on expensive computations behind the NAS-Bench-101 benchmark. In addition, the authors provide empirical analyses that reflect generalization and regularization issues with current methods and that could lead towards designing more robust algorithms. \n\nIn my opinion, the the background section might be improved by presenting higher level notions that would lead to an easier understanding by a wider audience. Also, although comprehensible due to the nature of this study, readers not familiar with a variety of previous work might find difficult to parse the paper as many concepts are just referenced.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BkeRiU01qB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposes a benchmark dataset for evaluating One-Short Neural Architecture Search models. It extends on the idea of the NASBench-101 (Ying et. al., 2019) into One-Shot models. \n\nI think this is important work. With all the development of neural archictecture search methods, reproducible research is paramount. This is even more timely considering the fact that these problems are computationally expensive, so it is even more computationally difficult than usual to run exhaustive comparisons among proposed methods. \n\nThe paper is well-written and the design decisions are clearly explained. The comparison of NAS methods is also interesting to read. In general, I think this is a worthy publication. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a new benchmark for architecture search. Reviews were somewhat mixed, but also with mixed confidence scores. I recommend acceptance as poster - and encourage the authors to also cite https://openreview.net/forum?id=HJxyZkBKDr
Decision: Accept (Poster)