How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
Authors: Kaicheng Yu
Rene Ranftl
Mathieu Salzmann
Venue:  (2021)
Keywords: autoML
neural architecture search
NAS
one-shot NAS
weight-sharing NAS
super-net

Abstract: Weight sharing promises to make neural architecture search (NAS) tractable even on commodity hardware. Existing methods in this space rely on a diverse set of heuristics to design and train the shared-weight backbone network, a.k.a. the super-net. Since heuristics substantially vary across different methods and have not been carefully studied, it is unclear to which extent they impact super-net training and hence the weight-sharing NAS algorithms. In this paper, we disentangle super-net training from the search algorithm, isolate 14 frequently-used training heuristics, and evaluate them over three benchmark search spaces. Our analysis uncovers that several commonly-used heuristics negatively impact the correlation between super-net and stand-alone performance, whereas simple, but often overlooked factors, such as proper hyper-parameter settings, are key to achieve strong performance. Equipped with this knowledge, we show that simple random search achieves competitive performance to complex state-of-the-art NAS algorithms when the super-net is properly trained.


Under review as a conference paper at ICLR 2021
HoW TO TRAIN YOUR SUPER-NET: AN ANALYSIS OF TRAINING HEURISTICS IN WEIGHT-SHARING NAS
Anonymous authors Paper under double-blind review
ABSTRACT
Weight sharing promises to make neural architecture search (NAS) tractable even on commodity hardware. Existing methods in this space rely on diverse set of heuristics to design and train the shared- weight backbone network, a.k.a. the super- net. Since heuristics substantially vary across different methods and have not been carefully studied, is unclear to which extent they impact super-net training and hence the weight-sharing NAS algorithms. In this paper, we disentangle super-net training from the search algorithm, isolate frequently-used training heuristics, and evaluate them over three benchmark search spaces. Our analysis uncovers that heuristics negatively impact the correlation between super- net and stand-alone performance, whereas simple, but often overlooked factors, such as proper hyper-parameter settings, are key to achieve strong performance. Equipped with this knowledge, we show that simple random search achieves competitive performance to complex state-of-the-art NAS algorithms when the super-net is properly trained.
several commonly-use
INTRODUCTION
Neural architecture search (NAS) has received growing attention in the past few years, yielding state- of-the-art performance on several machine learning tasks (Liu et al. 2019a Wu et al., 2019 Chen
al. 2020). One of the milestones that led the popularity of NAS weight al. 2019b; Ryoo
sharing (Pham et al. 2018 Liu al. 2019b), which, by allowing all possible network architectures to share the same parameters, has reduced the computational requirements from thousands of GPU hours to just a few. Figure I shows the two phases that are common to weight-sharing NAS (WS-NAS) algorithms: the search phase, including the design of the search space and the search algorithm; and the evaluation phase, which encompasses the final training protocol on the proxy task
While most works focus developing good sampling algorithm (Cai al. 2019 Xie al. 2019) or improving existing ones (Zela et al. 2020a| Nayman et al. 2019 Li et al. (2020), they tend to overlook gloss over important factors related to the design and training of the shared-weight weight backbone network, i.e. the super-net. For example, the literature encompasses significant variations of learning hyper-parameter settings, batch normalization and dropout usage, capacities for the initial layers of the network, and depth of the super-net. Furthermore, some of these heuristics are directly transferred from standalone network training to super-net training without carefully studying their impact in this drastically different scenario. For example, the fundamental assumption of batch normalization that the input data follows a slowly changing distribution whose statisties can be tracked during training is violated in WS-NAS, but nonetheless typically assumed to hold.
In this paper, revisit and systematically evaluate commonly-used super-net design and training heuristics and uncover the strong influence of certain factors on the success of super- net net training. To this end, we leverage three benchmark search spaces, NASBench- 101 (Ying et al. 2019), NASBench- 201 (Dong & Yang. 2020), and DARTS-NDS (Radosavovic et al. 2019), for which the ground-truth stand-alone performance of large number of architectures is available. We report the results of our experiments according to two sets of metrics: i) metrics that directly measure the quality of the super-net, such the widely-adopted super-net accuracy [ and modified Kendall- correlation between the searched architectures and their ground-truth performance, which we refer to as sparse
Proxy task refers to the tasks that neural architecture search aims to optimize on.
"The mean accuracy over small of randomly sampled architectures during super-net training.

---

Under review as a conference paper at ICLR 2021
Propoeed
Weight sharing
Puu Fus
Training protocol
P
P
Search phase
Super-Net
Super-Net
Super-Net
Fixed
Metrics
Metrics
Sus
Search Algorithm
Search Algorithm
Search Algorithm
Jpruzy
Fixed
Jprozy
prouyy
Proxy-task
Proay
prozy
Training protocal
Evaluation
Stand-Alone
Stand-Alone Net
Stand-Alone
Net
Fixed
MCDOerNdE 221
WcGDeech
CARTSANDG
DAITS--DD
Metric
Metric
Metric
Paradigm of NAS
(b) NASBench series
(c) Our work
Figure 1: WS-NAS benchmarking. Green blocks indicate which aspects of NAS benchmarked in different works. A search algorithm usually consists of a search space that encompass many architectures, and a policy select the best one. P indicates training protocol, and f mapping function from the search space to neural network. (a) Early works fixed and compared the metrics on the proxy task, which doesn't 't allow for a holistic comparison between algorithms. (b) The NASBench benchmark series partially alleviates the problem by sharing the stand-alone training protocol and search space across algorithms. However, the design of the weight-sharing search space and training protocol is still not controlled. (c) We fill this gap by benchmarking existing techniques construct and train the weight backbone. We provide controlled evaluation across three benchmark spaces.
Kendall-Tau; ii) proxy metrics such the ability to surpass random search and the stand-alone accuracy of the model found by the WS-NAS algorithm.
Via our extensive experiments (over 700 GPU days), we uncover that (i) the training behavior of super-net drastically differs from that of standalone network, e-g., in terms of feature statisties and loss landscape, thus allowing us to define training factor settings, e.g., for patch-normmllizttoon (BN) and learning rate, that are better suited for super-nets; (ii) while some neglected factors, such as the number of training epochs, have a strong impact on the final performance, others, believed to be important, such as path sampling, only have marginal effect, and some commonly-used heuristies, such the use of low-fidelity estimates, negatively impact it; (iii) the commonly-adopte super-net accuracy is unreliable evaluate the super-net quality.
Altogether, our work is the first systematically analyze the impact of the diverse factors of super-net design and training, and we uncover the factors that are crucial to design super-net, as well as the non-important ones. Aggregating these findings allows us to boost the performance of simple weight-sharing random search the point where it reaches that of complex state-of-the-an NAS algorithms across all tested search spaces. We will release our code and trained models so as to establish solid baseline facilitate further research.
PRELIMINARIES AND RELATED WORK
We first introduce the necessary concepts that will be used throughout the paper. As shown in F [ weight-sharing NAS algorithms consist of three key components: search algorithm that samples an architecture from the search space the form an encoding, a mapping function fprozy that maps the encoding into its corresponding neural network, and training protocol for proxy task P Perozy for which the network is optimized.
To train the search algorithm, generates the shared -eightt network. Note that the mapping fprezy frequently differs from fwss since in practice the final model contains many more layers and parameters as yield competitive results on the proxy task. After fixing fus. a training protocol P.. is required to learn the super-net. In practice, Pars often hides factors that are eritical for the final performance of approach, such
needs to additionally define the mapping function fwo that
2

---

tuning to evaluate their Under review as a conference paper at ICLR 2021
as
settings the use of data augmentation strategies to achieve oftte-of-hhe-ar performance (Liu et al.] 2019b; Chu et al., 2019; Zela et al.) 2020a). Again, Pus may differ from Pprozy, which is used to train the architecture that has been found by the search. For example, our experiments reveal that the learning rate and the total number of epochs frequently differ due to the different training behavior of the super-net and stand-alone architectures.
Many strategies have been proposed to implement the search algorithm, such as reinforcement learn- ing (Zoph & Le 22017] Zoph et al. 2018), evolutionary algorithms (Real et al. 2017 Miikkulainen
et al. 20199; So et al. 2019)]LL et al. 2018 Lu et al. 2018), gradient-based optimization (Liu et al.
2019b Xu et al. 2020: Li et 22020), Bayesian optimization (Kandasamy et al. 2018: 8: Jin et al.
2019:ZZou et al. 2019: Wang et al. | (2020), and separate performance predictors (Liu et al. 20118)
Luo et al.) 2018). Until very recently, the common trend to evaluate NAS consisted of reporting the searched architecture's 's performance on the proxy task (Xie et al.) 2019: Real et al., 2019. Ryoo et al. 2020). This, however, hardly provides real insights about the NAS algorithms themselves, because of the many components involved in them. Many factors that differ from one algorithm to another can influence the performance. In practice, the literature even commonly compares NAS methods that employ different protocols to train the final model.
Li & Talwalkar (2019) and Yu et al. 2020b) were the first to systematically compare different algorithms with the same settings for the proxy task and using several random initializations. Their surprising results revealed that many NAS algorithms produce architectures that do not significantly outperform a candomly-samples architecture. YYngg et al.] (2020) highlighted the importance of the training protocol Pprozy- They showed that optimizing the training protocol can improve the final architecture performance on the proxy task by three percent on CIFAR-10. 10. This non-trivial improvement can be achieved regardless of the chosen sampler, which provides clear evidence for the importance of unifying the protocol to build a solid foundation for comparing NAS algorithms.
In parallel to this line of research, the recent series of "NASBench" works (Ying al. 2019: Zela ct al. 2020bb Dong & Yang 2020) proposed to benchmark NAS approaches by providing a complete, tabular characterization of a search space. This was achieved by training every realizable stand-alone architecture using a fixed protocol Pprony. Similarly, other works proposed to provide a partial characterization by sampling and training sufficient number of architectures in given search space
using a fixed protocol (Radosavovic et al. 2019 Zela et al. 2020a Wang et al. 2020).
While recent advances for systematic evaluation are promising, no work has yet thoroughly stud- ied the influence of the super-net training protocol Pus and the mapping function fus- Previous works (Zela et a 2020a; Li & Talwalkar 2019) performed own algorithms, and focused only on a few parameters. We fill this gap by benchmarking different choices of P..e and fus and by proposing novel variations to improve the super-net quality.
Recent works have shown that sub-nets of super-net training can surpass some human designed models without retraining Yuu et al.) 2020a Cai et al., (220)) and that reinforcement learning can surpass the performance random search (Bender et al.] 2000). However, these findings are still only shown on MobileNet-like search spaces where we only search for the size of convolution kernels and the channel ratio for each layer. This is an effective approach to discover compact network, but it does not change the fact that on cell-based search space super-net quality remains low.
EVALUATION METHODOLOGY
We first isolate 14 factors that need to be considered during the design and training of a super-net, and then introduce the metries to evaluate the quality of the trained super-net. Note that these factors are agnostic to the search policy that is used after training the super-net.
3.1 DISENTANGLING THE SUPER-NET FROM THE SEARCH ALGORITHM
Our goal is to evaluate the influence of the super-net mapping fus and weight-sharing training protocol P.n.- As shown in Figure fws translates an architecture encoding, which typically consists parameters, into neural network. Based
of a discrete number of choices mapping, the super-net is a network in which every sub-path has a one-to-one mapping with an
well-defined
architecture encoding (Pham et al. 2018). Recent works (Xu et al. 2020 Li et al. 2020 Ying et al.
3

---

Under review as a conference paper at ICLR 2021
WS Mapping fux
WS Prosocol Pae
Avtiinoun Freisiigg
paringeeroo Pxx(!
implementation
fidelity
loperparom.
sumplinng
Nacio Paranetass
Jus
Dynamic Channeling
# layer
batch-norm learning rate
FairNAS Random NAS
OFA Conv
train partion
dc
WSBN
busch size
spochs
Random-A
Oo Cride
Drogont
channels
weight decay
Paramitar !
Op on Node/Edge
Learing cal
Table 1: Summary of factors
Figure 2: Constructing super-net
2019) separate the encoding into cell parameters, which define the basic building blocks of network, and macro parameters, which define how cells are assembled into a complete architecture.
Weight-sharing mapping sus- To make the search space manageable, all cell and macro parameters are fixed during the search, except for the topology of the cell and its possible operations. However, the exact choices for each of these fixed factors differ between algorithms and search spaces. We report the common factors in the left part of Table | They include various implementation choices, e.g., the use of convolutions with a dynamic number of channels (Dynamic Channeling), super-convolutionall layers that support dynamic kernel sizes (OFA Kernel) (Cai et al., (020), weight-sharing batch- normalization (WSBN) that tracks independent running statisties and affine parameters for different incoming edges (Luo et al. (2188) and path and global dropout (Pham et al. 2018 Luo et al. 2018 Liu et al. 2019b). They also include the use of low-fidelity estimates (Elsken al. (2019) to reduce the complexity of super-net training. c.g., by reducing the number of layers (Liu et al. (2019b) and
channels (Yang et al.] 2020 Chen et al. (019a), the portion of the training set used for super-net training (Luu et al.) 2019b), or the batch size (Liu et al. 2019b Pham et al. 2018 Yang et al. 2020).
Weight-sharing protocol Pws. Given a mapping fuss different training protocols Pws can be employed to train the super-net. Protocols can differ in the training hyper-parane prrper-praamters and the sampling strategies they rely on. We will evaluate the different prpee-parameter choices listed in the right part of Table ! This includes the initial learning rate, the normalization, the total number of training epochs, and the amount of weight decay.
of batch
We randomly sample one path to train the super- neet net (Guo et al., 2019). which is also known as single-path one-shot (SPOS) Random-NAS (Li & Talwalkar 2019). The reason for this choice is that Random- NAS is equivalent to the initial state of many search algorithms (Liu et al., 20196] to use
Pham et al., 2018: Luo et al., 2018), some of which even freeze the sampler training random sampling to warm-up the super-net (Xu et al. 2020: Dong & Yang 2019b).. Note that we also evaluated two variants of Random-NAS, but found their improvement be only marginal. Please
Appendix C.2 for more detail.
In our experiments, for the sake of reproducibility, ensure that Pws and P Perozy. well as fus and fprozy. are as close to each other as possible. For the apper-prrameters of Pwe. we cross-validans each factor following the order in Table [ and after each validation, use the value that yields the best performance in Pp Pprozy. For all other factors, we change one factor at a time.
Search spaces. We use three commonly-used search spaces, for which large number of stand-alone architectures have been trained and evaluated on CIFAR-10 (Krizhevsky et al.. 2009) to obtain their ground-truth performance. In particular, we use NASBench- 101 YYin et al. (011)) which consists of 423, 624 architectures and is compatible with weight-sharing NAS (Yu al.. 2020b Zela et al. 2020b); NASBench-201 (Dong& & Yang (020). which contains more operations than NASBench-101 but fewer nodes; and DARTS-NDS (Radosavovic et al. 2019) that contains over 10 architectures of which subset of 5000 models was sampled and trained in stand-alone fashion. See Appendix for a detailed discussion.
A.2 A.2
3.2 SPARSE SENDALL-TAU A NOVEL NUP EVALUATION METRIC
We define novel super-net metric, which we name sparse Kendall- Taau It is inspired by the Kendall- Tau metric used by Yu et al. (2020b) to measure the discrepancy between the ordering of stand-alone architectures and the ordering that is implied by the trained seper-net. An ideal super-net should yield the same ordering of architectures as the stand-alone one and thus would lead high Kendall-Tau However, Kendall-Tan -aa is not robust to negligible performance differences between architectures (c.f. Figure 3). To robustify this metric, we share the rank between two architectures if their stand-alone
4

---

Under review as a conference paper at ICLR 2021
accuracies differ by less than a threshold (0.1%
Stard--ltrr Ace.
here). Since the resulting ranks are sparse, we call this metric sparse Kendall-Tai (s-KdT).
Kendall Tau
Note that also compare Kendall-Tau and Spearman correlation in Appendix A.3 and pro- vide implementation details in Appendix A..4
E3333
Rark c0.01
5
original sparse
0.6444
Predec fed
ground pround-futt
Supar-Not
rest
Although, sparse Ken captures the super-net quality well, it may fail in extreme cases, such as when the top-performing archi- tectures are ranked perfectly while poor ones are ordered randomly. To account for such rare situations and ensure the soundness of our analy- sis, we also report additional metrics. We define two groups of metrics to holistically evaluate different aspects of a trained super-net. The first group of metrics directly evaluates the quality of the super-net, including sparse Kendall-Tar and the widely-adopted super-net accuracy. For the super-net accuracy, we report the average accuracy of 200 architectures on the validation set of the dataset of interest. We will refer to this metric simply as accurucy. frequently used (Guo et al.) 2019 Chu al. 2019) to assess the quality of the trained super-net, but will show later that in fact poor predictor of the final stand-alone performance. The metrics in the second group evaluate the search performance The first metric is the probability to surpass random scarch: Given the fround-truth rank of the best architecture found after n runs and the maximum rank Tmazs equal to the total number of architectures, the probability that the best architecture found is better than randomly searched one is given by p = 1 - (1 (r/Tmax))". Finally, where appropriate, we report the stand-alone accuracy of the model that was found by the complete WS-NAS algorithm. Concretely, we randomly sample 200 architectures, select the best models based on the super-net accuracy and query the ground-truth performance. We then take the mean of these architectures Sand-alone accuracy. Note that the same architectures are used to compute the sparse Kendall-Tai
Figure 3: Kendall-Tan vs sparse Kendall-Tau Kendall-Tau is not robust when many architectures have similar performance. Minor performance dif- ferences can lead to large perturbations in the rank- ing. Our sparse Kendall- alleviates this by dis- missing minor differences in performance.
trained super-net.
ANALYSIS
We provide an analysis on the impact of the factors that are shown in Table I across three different search spaces. Note that, in this section, we present the factors that are the most important for performance; our analysis of the remaining factors is provided in Appendix
4.1 EVALUATION OF SUPER-NET
The standalone performance of the architecture that is found by NAS algorithm is clearly the most important metric to judge its merits. However, in practice, one cannot access this metric-we we wouldn't 't need NAS if standalone performance easy to query (the cost of com-
Waman saroeen men
0.6
81-
29-
NiSkeech 201
BARTS MG6
puting stand-an aloone performance discussed in Appendix B.2 Furthermore, stand-alone per- formance invitaably depends the sampling pol- icy, and does not directly evaluate the quality of the super-net (see Appendix B.3). Conse- quently, it is important to rely metrics that are well correlated with the final performance but be queried efficiently. To this end, collect all our experiments and plot the pairwise correlation between final performance, sparse Kendall-Ta -Tuu, and super-net accuracy. As shown in Figure the super-net accuracy has low
60 Sporos kilf
0.5
0.0
Superee Accunry
Sugurmet AeetAcrurr
Speaman
K.T
NASBeach- 101 NASBeach DARTS-NDS
0.55
0.07
0.19
Figure 4: Super-net evaluation. We collect all experiments across 3 benchmark spaces. (Top) Pairwise plots of super-net accuracy, final perfor- mance, and the sparse Kendall-Tai Each point corresponds to statistics computed over a trained super-net. (Bottom) Spearman correlation coeffi- cients between the metrics.
5

---

Under review as a conference paper at ICLR 2021
correlation with the final performance NASBench- 101 and DARTS-NDS, Only NASBench-201 201 does it reach a correlation of 0.52. The sparse Kendall-Tau yields a consistently higher correlation with the final performance. This is evidence that one should not focus too strongly on improving the super-net accuracy. While this metric remains computationally heavy, it serves as a middle ground that is feasible to evaluate in real-world applications.
In the following experiments, we thus mainly rely on sparse Kendall-Tau, and use final search performance as reference only. We report the training details in Appendix B.1 and the complete results of all metrics in Appendix C.6
4.2 BATCH NORMALIZATION IN THE SUPER-NET
Batch normalization (BN) is commonly used in standalone networks to allow for faster and more stable training. It is thus also employed in most CNN search spaces. However, BN behaves differently in the context of WS-NAS, and special care has be taken when using it. In standalone network (c.f. Figure (To))), a BN layer during training computes the batch statistics μı and dg, normalizes
the activations fA as (fa(z) - /B)/oBB, and finally updates the population statistics using moving average. For instance, the mean statistics is updated as j + 7/ + (1 7)lg. At test time, the stored population statistics are used normalize the feature map. In the standalone setting, both batch and population statistics are unbiased estimators of the population distribution
CNN Archtocture
sage pums
RN
Y
Sample Freq.
falz)
Buuca
A2
By contrast, when training a super-net (Figure 5 (Bottom)) the population statistics that are computed based on the running average are not unbiased estimators of the population distri- bution, because the effective architecture before the BN layer varies in each epoch. More formally, let fA, denote the i-th architecture. During training, the batch statistics are computed as n1 = SA, (2,)/m, and the output feature follows the distribution N(up. dy), where the superscript i indicates that the current batch statistics depends on A, only. The population mean statistics is then updated as j + 7/ + (1 - -)) How- ever, during training, different architecture from the super-net are sampled. Therefore, the population mean statistics essen- tially becomes weighted combination of means from different architectures, i.e., j + Σ asslin = Σ Co (x), where o; is the sampling frequency of the i-th architecture. When eval- uating a specific architecture A, at test time, the estimated population statisties thus depend the other architectures in the super-net. This leads to a train-test discrepancy. One solu- tion mitigate this problem by recomputing the statistics on the entire training set before the the final evaluation (Yu & Huang 2019), While the cost of doing so is negligible for a standalone network, NAS algorithms typically sample ~ 100 architectures for evaluation, which makes this approach intractable.
Figure 5: Batch normalization in standalone and super-net training.
Darts NDS
NasBench NasBench
201
101
Super-net Test Accuracy
Our BN
150
Ade
Baseline
100
307
0 143 303
59 100
Frequency
re-calibrate the batch statisties
Figure 6: Validation of BN.
In contrast to Dong & Yang 2020) and Bender et al.] (2020) who use the training mode also during testing, formalize a simple, yet effective, approach to tackle the train-test discrepancy of BN in super-net training: we leave the normalization based on batch statistics during training unchanged, but use batch statisties also during testing. Since super-net evaluation is always conducted over complete dataset, we are free to perform inference in mini-batches of the same size as the ones used during training. This allows to compute the batch statisties on the fly in the exact same way as during training.
Figure Tigureeccome compares standard BN to our proposed modification. Using the tracked population statis- ties leads to many architectures with an accuracy around 10%, i.e., performing no better than random guessing. Our proposed modification allows us to significantly increase the fraction of

---

Under review as a conference paper at ICLR 2021
leaming rate
super aGC.
sparce Kdf
0.20
0.21
0.5
0.10
8.55
0.55
0.00
(n) Stand-alone
(b) Super-net
200
430
266
800
222
489
Figure 7: Loss landscapes.
Figure 8: Learning rate on NASBench-201.
begh-performing architectures. Our results also show that the choice of fixing vs. learning an affine transformation in batch normalization should match the standalone protocol P Prrozy.
4.3 SUPER-NET LOSS LANDSCAPES
The training loss of the super-net encompasses the task losses of all possible architectures. We suspect that the training difficulty increases with the number of architectures represented by the super-net. To better study this, we visualize the loss landscape (Li et al. 2018)) of the standalone network and super-net with 300 architectures. Coneretely, the landscape is computed over the super-net training loss under the single-path one-shot sampling method,
C, 0.) = (o,,0,))) where Vi, = 0,.
(I)
Figure 7 shows that the loss landscape of the super-net less smooth than that of standalone architecture, which confirms our intuition. A smoother landscape indicates that optimization will converge more casily to a good local optimum. With a smooth landscape, one can thus use a relatively large learning. By contrast, a less smooth landscape requires using smaller one.
Our experiments further confirm this observation. In the standalone protocol Porozy- the learning rate is set to 0.2 for NASBench-101 and to 0.1 for NASBench-201 and DARTS-NDS, respectively. All protocols use a cosine learning rate decay. Figure g shows that super-net training requires lower learning rates than standalone training. The same trend is shown for other search spaces in Appendix C.I.
We set the learning rate to 0.025 to consistent across the three search spaces.
4.4 LOWER FIDELITY ESTIMATES LOWER THE RANKING CORRELATION
Reducing memory foot-print and training time by proposing smaller super-nets has been an active research direction, and the resulting super-nets are referred to as lower fidelity es- timates (Elsken et al., 2019). The impact of this approach on the seper-net quality, however, has never been studied systematically over multiple search spaces.. We compare four popular strate- gies in Table - We deliberately prolong the training epochs inversely proportionally to the computational budget that would be saved by the low-fidelity estimates, e.g. if the channel num- ber is reduced by half, train the model for two
Table 2: Low fidelity estimates under same com- putational budget, reporting final search model ac- curacy (FSA) and sparse Kendall- -au (S-KdT) on NASBench-201.
Metrics
Settings
Repeated cells
3
2
S-KdT FSA
0.751 + 0.09
0.692 + 0.18
0.502 +: 0.21
4:
91.95
0.10
0.71
Init Channel
16
S-KdT FSA
0,740 - 0.07
0.677
0.10
0.691
0.15
92.92 1 0.48
92.32 :: 0.37
92.79 1: 0.85
Batch
128
64
times more epoch. Note that this provides an upper bound to the performance of low-fidelity estimates.
S-KdT FSA
0,740 + 0.07
0.728
0.16
0.703 0.16
92.92 + 0.48
92.37 + 0.61
92.35 1: 0.34
Train portion
0.75
0.5
0.25
A commonly-used approach to reduce memory requirements is to decrease the batch size (Yang
S-KdT FSA
0.751 4 0.11
(.742 + 0.12
0.693 + 0.13
92.13 0.51
92.74 - 0.43
91.47 0.81
al. 2020). Surprisingly, lowering the batch size from 256 64 has limited impact on the accuracy, but decreases sparse Kendall-Te and the final searched model's performance, the most important metric in practice.
Another approach is to decrease the number of channels in the first layer (Liu et al. 2019b). This reduces the total number of parameters proportionally, since the number of channels consecutive
7

---

Under review as a conference paper at ICLR 2021
Cin Inchonns
Type
Accuracy
S-KdT
P>-R
Fixed
71.52 1: 6.94
0.22
0.546
91.79 4. 1.72
Shuffle
$1.79 + 10.90
0.17
0.391
90:58 + 1.58
Interpolate
57.53 # 10.05
0.37
0.805
93.35 + 3.27
[Con//2
Cona/a
Baselinei
76.91
10.05
0.22
0.865
89:43 1 4.30
Baseline-22
75.18
Cout charnel
Ours
8.29
0.949
93.65 0.73
See Appendis
details.
(3) Search space
(b) 2 edges
(c) edges
Figure 9: NASBench-100 101 dynamic channel.
Table 3: Dynamic channels on NASBench-101.
layers depends on the first one. Table 2 shows that this decreases the sparse Kendall-Tau from 0.7 to 0.5. By contrast, reducing the number of repeated cells (Pham al. 2018 Chu et al.] (2099) by one has little impact. Hence, to train good super-net, one should avoid changes between fus and Sprozy, but one can reduce the batch size by factor 0.5 and use only one repeated cell.
The last lower- fidelity factor is the portion of training data that is used (Liu et al., 2019b Xu et al. 2020). Surprisingly, reducing the training portion only marginally decreases the sparse Kendall-Tau for all three search spaces. On NASBench-20) keeping only 25% of the CIFAR-10 dataset results in 0.1 drop in sparse Kendall-Tau. This explains why DARTS- methods typically only 50% of the data to train the super-net but can still produce reasonable results.
4.5 DYNAMIC CHANNELING HURTS SUPER-NET QUALITY
Dynamic channeling is an implicit factor in many search spaces (Ying et al. 2019 Cai et al. 2019
Guo al. 2019; Dong & Yang 2019b). It refers to the fact that the number of channels of the intermediate layers depends on the number of incoming edges to the output node. This is depicted by Figure 9 (a): for search cell with intermediate nodes, where X and Y are the input and output node with Cin and Cout channels, respectively. When there are n 2 edges (c.f. Figure 9 (b)), the associated channel numbers decrease so that their sum equals Cout. That is, the intermediate nodes have [Cont/2] channels. In the general case, shown in Figure 9 (c). the number of channels in intermediate nodes is thus [Cout/ni for n. incoming edges. A weight sharing approach has to cope with this architecture-deeendeer fluctuation of the number of channels during training.
Let C' denote the number of channels of given architecture, and Cmaa the maximum number of channels for a node across the entire search space. All existing approaches allocate Cinaze channels and, during training, extract subset of these channels. The existing methods then differ in how they extract the channels: Guo et al. (2019) use a fixed chunk channels, e.g.. [0 : C): Zhang et al. (2018 8) randomly shuffle the channels before extracting fixed chunk; and Dong & Yang (2019a) linearly interpolate the Cmazz channels into C' channels using a moving average across neighboring channels.
Instead of sharing the channels between architectures, propose to disable dynamic channelling completely. As the channel number only depends on the incoming edges, we separate the search space into discrete number of sub-spaces, each with fixed number of incoming edges. As shown in Table 3, disabling dynamic channeling improves the sparse Kendall-Taa and the final search performance by large margin and yields
state of the art NASBench101.
We compose another baseline, where we enable dynamic channeling during super-net training. During validation, we compute the average sparse Kendall-Tau of each sub-space, where sample 200 architectures that shares the same number of channels. We call this baseline-v2. In Table B we can this surpasses the original baseline by significant margin. It further evidence the importance of disabling dynamic channels. Nonetheless, the best is to disable dynamic channeling during both the training and the validation phase.
5 How SHOULD YOU TRAIN YOUR SUPER-NET?
Figure 10 summarizes the influence of all tested factors on the final performance. It stands out that properly tuned hyper-parameters lead the biggest improvements by far. Surprisingly, most other factors and techniques either have a hardly measurable effect or in some cases even lead to worse performance. Based on these findings, here is how you should train your super-net:
8

---

Under review as a conference paper at ICLR 2021
Hyper parameterss
Sampling
Low Fidelity Estimates
Implementation
WASBanch
THE
lamning raae
apector
Bortar
Figure 10: Influence of factors on the final model. We plot the difference in percent between the searched model's performance with and without applying the corresponding factor. For the hyper-parameters of P.s. the baseline is Random NAS, as reported in Table For the other factors, the baseline of each search space uses the best setting of the hyper-parameters. Each experiment was run at least 3 times.
1. Do not use super-net accuracy to judge the quality of your super-net. The sparse Kendall-Tau has much higher correlation with the final search performance.
2. When batch normalization is used, do not use the moving average statistics during evaluation. Instead, compute the statistics the fly over batch of the same size as used during training.
3. The loss landscape of super-nets is less smooth than that of standalone networks. Start from smaller learning rate than standalone training.
4. Do not use other low-fidelity estimates than moderately reducing the training set size to decrease the search time.
5. Do not use dynamic channeling in search spaces that have a varying number of channels in the intermediate nodes. Break the search space into multiple sub-spaces such that dynamic channeling is not required.
Comparison to the state of the art. Table shows that carefully controlling the relevant factors and adopting the techniques proposed in Section 4 allow us to considerably improve
Table 4: Final results. Results on NASBench- 101 and 201 are from Yu et (2020b). and Dong & Yang (020). We report the mcan over 3 runs. Note that NASBench-101 (n
in (Yu al.) 2020b) is identical to our setting. Our new strategy
the performance of Random-NAS. Thanks to our evaluation, we were able to show that sim- ple Random-NAS together with appropriate training protocol Pw. and mapping function Jus yields results that are competitive to and some- times even surpass state-of-the-ee algorithms. Our results provide strong baseline upon which future work can build.
significantly surpasses the random search baseline.
Mathel
NASRmch 101 (i²7)
NASBerch 201
DARTS NDS
DARTS NDS
ENAS DARES V2
91.85 a 0.42
54.30 2 0.00
94.45 2 0.99
97.11
92.21 1: 061
54.30
0.00
94,79
1.1.1
97.37
NAO GDAS
92.59 = 0.59
97.10
93.51 = 0.13
96.23
Randon NAS
89.89
3.89
87.66
1.60
91.33 + +012
96.74
Randon NAS (Ouss)
93.12 0.66
92.71
= 0.15
94.26 0.15
97.08
Results from
Talaakar 2019
"Traied soco Sireal
2019
for 600 coochs
DARTS Lget 2019 noxs
Pham etal 2018 NoO
2018
Randem-NAS Hivali
2019)
Gaass ficdy Ying 2019c)
On NASTlemss Fermmi
aeciilectress fellow Tang Yang

---

Under review as a conference paper at ICLR 2021
REFERENCES
Gabriel Bender, Hanxiao Liu, Bo Chen, Grace Chu, Shuyang Cheng, Pieter-Jan Kindermans, and Quoc V. Le. Can weight sharing outperform random architecture search? an investigation with tunas. In CVPR, 2020.
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware. In ICI.R 2019.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han, Once for all: Train one network and specialize it for efficient deployment. In /CLR, 2020. /Rt/ net/
forum? dy x HKwS.
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In ICCV, 2019a.
Yukang Chen, Tong Yang. Xiangyu Zhang, Gaofeng Meng, Chunhong Pan, and Jian Sun. DetNAS: Neural Architecture Search on Object Detection. NeurIPS, 2019b.
Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li. FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search. arXiv:; 2019.
Xuanyi Dong and Yi Yang. Network pruning via transformable architecture search. Advances in Neural Information Processing Systems, pp. 760-771, 2019a.
Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four gpu hours. In CVPR, 2019b.
Xuanyi Dong and Yi Yang. NAS-Bench-201: Extending the scope of reproducible neural architecture search. In ICLR, 2020.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21, 2019.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single Path One-Shot Neural Architecture Search with Uniform Sampling. ECCV, 2019.
Haifeng Jin, Qingquan Song, and Xia Hu. Auto-Keras: An efficient neural architecture search system. In International Conference on Knowledge Discovery & Data Mining, 2019.
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poezos, and Eric Xing. Neural architecture search with bayesian optimisation and optimal transport. In NewrIPS, 2018.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-10 (canadian institute for advanced research). 2009.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In NeurIPS, 2018.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. UAI, 2019.
Xiang Li, Chen Lin, Chuming Li, Ming Sun, Wei Wu, Junjie Yan, and Wanli Ouyang. Improving one-shot NAS by suppressing the posterior fading. CVPR, 2020.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In ECCV, 2018.
Chenxi Liu, Liang-Chine Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan Yuille, and Li Fei-Fei. Auto-DeepLab- Hierarchical Neural Architecture Search for Semantic Image Segmentation. CVPR, 2019a.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. ICLR, 2019b.
10

---

Under review as a conference paper at ICLR 2021
Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, and Wolfgang Banzhaf. NSGA-NET: A multi-objective genetic algorithm for neural architecture search. arXiv:1810.03522. 2018.
Renqian Luo, Fei Tian, Tao Qin, En-Hong Chen, and Tie-Yan Yan Liu. Neural architecture optimization. In NeurlPS, 2018.
Risto Miikuulanenn, Jason Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, et al. Evolving deep neural networks. In Artificial Intelligence in the Age of Neural Networks and Brain Computing, pp. 293-312. 2019.
Niv Nayman, Asaf Noy. Tal Ridnik, Itamar Friedman, Rong Jin, and Lihi Zelnik. Xnas: Neural architecture search with expert advice. In NeurIPS, 2019.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. ICML, 2018
Ilija Radosavovic, Justin Johnson, Saining Xie, Wan- Yen Lo, and Piotr Dollár. On Network Design Spaces for Visual Recognition. In ICCV, 2019.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc Le, and Alexey Kurakin. Large-scale evolution of image classifiers. ICML, 2017.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. AAAI, 2019.
Michael S. Ryoo, AJ Piergiovanni, Mingxing Tan, and Anelia Angelova. Assemblenet: Searching for multi-stream neural connectivity in video architectures. In ICLR, 2020. URL https://
openreview net./wn.eeeeeee wewee/fooumiideee http://www.ee//oommddddd/sa-///e SJggK 4Ywr
David R. So, Chen Liang, and Quoe V. Le. The evolved transformer. In ICML, 2019.
Linnan Wang, Saining Xie, Teng Li, Rodrigo Fonseca, and Yuandong Tian. Neural architecture search by learning action space for monte carlo tree search. AAAI, 2020. /RL/pttpp://opeeeevvie
net / Eorummrmmii. 7id - Sk R6a
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangging Jia, and Kurt Keutzer. FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search. CVPR, 2019.
Sirui Xie, Hchui Zheng, Chunxiao Liu, and Liang Lin. SNAS: Stochastic neural architecture search. ICLR, 2019.
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi. Qi Tian, and Hongkai Xiong. PC-DARTS: Partial channel connections for memory-efficient architecture search. In ICLR, 2020. URL httphhtttpp::/peeeeeeeee
ICLR, 2020. URL httpe///weee.eeeeeoooooordddddad
Antoine Yang, Pedro M. Esperança. and Fabio M. Carlucci. NAS evaluation is frustratingly hard. In
Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, and Frank Hutter. NAS- Bench- 101: Towards reproducible neural architecture search. ICLR, 2019.
Jiahui Yu and Thomas S. Huang. Network slimming by slimmable networks: Towards one-shot architecture search for channel numbers. ICLR, 2019.
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoe Le. BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models. ECCV, 2020a.
Kaicheng Yu, Christian Sciuto, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. In ICLR, 2020b. URL wtt.::ttps//.mm /tt //tp:://///ppppeeiiiiee net/ mett/forumtn d== 1oF2NFFF
11

---

Under review as a conference paper at ICLR 2021
Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hutter. Understanding and robustifying differentiable architecture search. In ICLR, 2020a. URL https
//openreveew.nee hott//ooormidddllooD////
Arber Zela, Julien Siems, and Frank Hutter. AAS-Bench-1Sholee Benchmarking and dissecting one-shot neural architecture search. In ICLR, 2020b. URL https:://ppeeeeeieeww net/
Forum?2d=SSx9nnsttet
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018.
Hongpeng Zhou, Minghao Yang. Jun Wang, and Wei Pan. BayesNAS: A bayesian approach for neural architecture search. In JCML, 2019.
Barret Zoph and Quoe V. Le. Neural Architecture Search with Reinforcement Learning. ICLR, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018
12


Reviews:
[{"review_id": "_2TMINzcvCj", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper analyzes the effect of different components of the supernet training process w.r.t performance of the final NAS models. Finally it makes some recommendations for good practices for training supernets that eventually lead to better NAS designed models.\n\nStrengths:\n+ The performance of many NAS approaches is critically dependent on the effectiveness of Supernets. As such, a thorough study of supernet training process is timely and useful.\n+ The final recommendations are useful, albeit unsurprising. Standalone or supernet, training hyper-parameters usually play a big role.\n+ The discussion on batch norm is helpful. This is indeed a problem with some kinds of supernet training methods.\n\nWeaknesses:\n- I did not find information on the supernet training process that is used. There are many supernet training method proposed in the literature but it is not clear what is similar or different in the paper training process. I looked for this but could not find it, it is possible I missed it, but this seems to be important information that should not be hard to find.\n- As existing supernet models (e.g., OFA, SPOS, BigNAS) show, sampled subnets can be directly used, with the same level of accuracy compared to training from scratch. This seems to run counter to the main premise of the current paper, which is that the supernet only provides an architecture, which then needs to be trained properly. As such it is not clear how much of the analysis in this paper is applicable to current supernets, and consequently how useful this analysis is.\n- Following from the above, the OnceForAll supernet is one of the most effective one, which does incorporate some of the suggestions from this paper. There does not seem to be much discussion of the OFA supernet apart from the OFA conv.\n- Parts of the paper are very confusing and not easy to understand. As an example, in section 2, $f_{ws}$ is introduced as a mapping but not defined until much later. \n\nOverall, the paper analyzes the effect of different components of the supernet training process. While parts of the paper are interesting, it does not discuss the training process of some of the latest and most effective supernet models. As such it is not clear how useful the findings  of this paper are.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Prg6i551yss", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nNeural Architecture Search (NAS) aims to find a model with the best possible accuracy (or best possible accuracy/size tradeoff) from within a human-defined search space. One popular strategy for speeding up NAS is to train a one-shot model -- a single set of shared weights -- that can then be used to evaluate any candidate architecture within the search space without retraining or fine-tuning. The submission investigates how various decisions made when training a one-shot model can affect its ability to rank different candidate architectures within a search space.\n\nThe submission presents an empirical study, and I believe its main contribution is to provide information about (i) how sensitive one-shot model quality is to hyper-parameter tuning, and (ii) how to tune the hyper-parameters of a one-shot model in order to produce high-quality search results. These results are likely to be of practical interest to people trying to train high-quality one-shot models for neural architecture search, as well as for researchers trying to benchmark and compare different NAS algorithms.\n\nThe submission presents several sets of experiments. The most significant set of experiments (in my opinion) is Figure 11 (\"influence of factors on the final model\"), which shows the effect on the quality of the final searched model when we tune different hyper-parameters of the one-shot model, such as the learning rate, number of training epochs, weight decay, sampling scheme, and batch norm implementation, or apply various heuristics for decreasing the training time and memory usage of the one-shot model. Several additional sets of experiments are presented throughout Sections 4 and 5 of the paper.\n\nIn general, the paper is clear and well-organized, and most of the experiments are well-described; two relatively minor exception are (i) Figure 1, which I had trouble making sense of initially, and (ii) some of the experiments on the right-hand side of Figure 11 (e.g., \"ofa kernel,\" \"op-edge\"), which include citations but where the authors' precise setup doesn't seem to be clearly defined in the main paper.\n\n**Pros:**\n* Conducts a broad set of experiments evaluating the effect of different training decisions on the quality of a one-shot model.\n* The experiments and concrete recommendations made by the paper are likely to be of practical value for practitioners/researchers who want to train high-quality one-shot models.\n* Experiments reported on a variety of different search spaces/benchmark tasks.\n\n**Cons:**\n* Experiments are limited to CIFAR-10.\n* Some unresolved questions about a few of the experiments. (Discussed below.)\n\n**Notes on Paper Rating**: I have some concerns about how a few of the experiments are presented  (detailed below) which would need to be addressed before I felt comfortable advocating for the paper's acceptance. If they are adequately addressed, however, I think the paper could be a valuable resource for NAS practitioners.\n\n## Notes on Empirical Evaluation\nThe submission conducts experiments on (certain subsets of) the NASBench-101 and NASBench-201 benchmark tasks, as well as a third task (attributed to Radosavovic et al., 2019) which the submission refers to as DARTS-NDS. One caveat is that all the conclusions in the paper appear to be drawn based on experiments on the CIFAR-10 image classification dataset. While it's unclear how well the results would generalize to other datasets, I think this is an acceptable limitation, given (i) the broad set of experiments conducted on this dataset, and (ii) most existing NAS Benchmark tasks I'm aware of are built around CIFAR-10.\n\nAt various points in the paper, the quality of the one-shot model is measured according to one of three different metrics: (i) \"sparse Kendall Tau\", which measures the correlation between accuracies measured according to the one-shot model and ground-truth accuracies obtain by training network architectures from scratch, (ii) the ground-truth accuracy of the network architecture which is ranked highest according to the one-shot model, and (iii) the probability that a network architecture found using a one-shot model will be better than one found using a random search baseline without weight sharing. What this means in practice is that different sets of experiments use different metrics. This leads to a slightly inconsistent presentation. For example, the experiments in Section 4.4 (\"lower fidelity estimates\") focus on Sparse Kendall Tau, while the experiments in Section 5 / Figure 11 (\"Influence of factors on the final model\") focus on ground-truth accuracies. However, experiments within the same section/plot are typically comparable to each other.\n\nThe weakest part of the paper is Table 3 (\"Final Results\") which compares the results of the authors' well-tuned one-shot model against previously published results. The raw numbers are compelling, but I had trouble verifying the comparison between the \"Random NAS\" and \"Random NAS (Ours)\" rows of the table. In particular:\n1. The 87.66 accuracy number for \"Random NAS\" in NASBench-201 was obtained by sampling 100 random architectures from the search space and retaining only the one with the highest one-shot model accuracy. It was unclear to me whether \"Random NAS (Ours)\" used the same evaluation protocol.\n2. It was unclear to me how the 91.33 accuracy number for DARTS-NDS Random NAS was obtained, since I was unable to find that number when I skimmed Radosavovic's paper.\n3. If I understand the table caption correctly, the \"NASBench-101\" table header is misleading and should be changed to \"NASBench-101 (n=7)\".\n**Because the improvement of \"Random NAS (Ours)\" over \"Random NAS\" is highlighted in Section 5, these points would need to be addressed before I felt comfortable recommending the paper for acceptance.**\n\n## Additional Notes\nIn Section 4.4: It would be helpful to explicitly state how the learning rate is adjusted when you decrease the batch size, since this can significantly affect the final results.\n\nIn Section 4.5: It would be helpful to update the submission to clarify: if you disable dynamic channels and train a separate model for each possible number of incoming edges, do you include comparisons between candidate architectures with different numbers of incoming edges when you compute the sparse Kendall-Tau? (If not, it seems like the sparse Sparse Kendall Tau numbers for different methods might not be directly comparable to each other.)", "extra_scores": {}, "extra_text": {}}, {"review_id": "HJ69b22hh7B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to analyze 14 commonly used training tricks for weight-sharing based NAS algorithms. The empirical analysis is evaluated on three existing NAS datasets.\n\nPros:\n- A comprehensive analysis of commonly used NAS training tricks.\n- A new metric -- sparse Kendall-Tau -- for the correlation between architectures.\n\nCons:\n- No new ideas are proposed, as the authors are evaluating existing tricks.\n- Most of the conclusions are somehow already revealed in previous NAS works.\n- As an emprical analysis paper, the authors do have some interesting finds. However, could these finds be used to improve existing NAS algorithms (which mostly already integrate multiple different training tricks)? A more concrete example is could the authors use the proposed findings to outperform the recent TuNAS?\n- The authors claimed to evaluate and analyse 14 factors, and use one page to explain the behavior of BN in Sec.4.2. However, \"the proposed modification\" is already mentioned and analyzed in previous papers, such as https://arxiv.org/abs/2001.00326 and https://arxiv.org/abs/2008.06120.\n- The explanation in Sec.4.3 is unclear to me. The learning rate schedule does not consider the commonly used warmup strategy. From Figure 8, it seems 0.01 results in the best correlation, but the authors choose 0.025 without further explanation. The authors mentioned \"requires a smaller lr\", whereas 0.005 is worse than 0.01. Overall speaking, in Sec.4.3, the loss landscapes looks interesting to me, but other analysis is not convincing to me.\n- All the experiments in this paper focus on the DARTS-like search space. There are also some other popular search spaces, such as MobileNet-like search space. It remains a question on how does the conclusion generalize to other search spaces.\n\n\nMinor issues:\n- In Figure 1, to make it self-contained, I would suggest to explain P_{ws}, f_{xxx} in the caption.\n- In Figure 1, \"NAS-Bench-102\" -> \"NAS-Bench-201\"\n- In Figure 1, why use \"Proxy-task Training protocol\" for the evaluation phase? In my opinion, the search is to proxy the evaluation task?\n- Regarding the presentation, f_{xxx} and P_{xxx} in Sec 2 take me a lot of time to understand. If the authors could add some examples when introducing these concepts, it could help readers understand.\n- In Sec.4.4, \"The impact of this approach on the super-net quality, however, has never been studied.\" is somewhat arbitrary. The results of #cells and #channels for DARTS is reported in their rebuttal, and I believe many researchers try these hyperparameters in their experiments but just not mentioned in the paper.\n- In Sec.4.5, the reference before \"linearly interpolate\" should be \"Network Pruning via Transformable Architecture Search\" instead of the cited one.\n- \"SNAS\" is accepted by ICLR 2019, \"FBNet\" is accepted by CVPR,  \"Auto-DeepLab\", etc; I would like to suggest authors cite their official version instead of arxiv version.\n- I'm confused about \"disabling dynamic channeling\". Would the authors mind to explain more or share the codes on how to do it? In addition, it seems this strategy can only be used for NAS-Bench-101 search space, instead of others.\n\nSome questions:\n- I'm confused about Figure 4 - 2/3. How could the x-axis be \"sparse KdT\"? If x-axis is \"sparse KdT\", it means each data architecture has a \"sparse KdT\" value? But if I understand correctly, \"sparse KdT\" is evaluated on a set of pairs, instead of a single data.\n- In Sec.3.2, why does less smoothed loss landscape indicate a smaller learning rate?\n- Would the authors mind adding the results of #channels=32 / #batch size=32/512 / #cells=1,5 in Figure 9?\n\n\n**Post Rebuttal Comments**:\nI appreciate the authors' efforts on the rebuttal, most of my questions are answered and addressed. However, I still have a major concerns. As agreed by the authors' response, this paper lies in a detailed analysis of existing tricks. I thus hope the paper could bring some new insights, but the mentioned \"other important conclusions\" in response is somehow commonly known in the NAS community. This makes the paper like some empirical supplementary material for existing papers instead of a new one. Therefore, I keep my original rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AG5gdh0fLOQ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work analyzes commonly used heuristics for training the supernet in weight sharing NAS. The authors first proposes a new metric, sparse Kendall-Tau, to measure the quality of the supernet. Then extensive experiments are conducted on three NAS benchmarks to empirically evaluate the heuristics, and pick the best settings. To highlight the significance of the training quality of supernet, the author showed that random search, when combined with the best settings, can performs competitive to SOTA results. \n\nStrength:\n\n1. The supernet training quality is an important factor in weight sharing NAS, which, as argued in this paper, has not received enough attention. This work raises the concern about this issue and empirically shows that it can indeed affect the search result significantly. \n\n2. The systematic benchmarking of different heuristics is valuable and serves as a useful guide to apply weight sharing NAS effectively. \n\nWeakness:\n\n1. The experiments are all conducted on relatively small search spaces. However, the search spaces in more realistic settings are usually much larger and might have different properties. For example, a recent work (Bender et al, 2020) showed that, contrary to the results on the small search space, random search significantly underperforms weight sharing results on large search space. So it would be interesting to see if the conclusions in this work still holds on large search spaces. \n\n2. The proposed sparse Kendall-Tau metric is shown to be more robust. But it also seems to be more coarse grained. For example, if the differences of top models are small, they would be put into one group by sparse Kd-T thus neglecting their relative ranking. However, those relatively rankings is also important in getting the best search outcome / selecting the best model. \n\n3. It is probably expected that the low-fidelity estimates would make search results worse since they are trading off search quality with efficiency. It would help to calibrate the comparison based on the search budget. For example, the comparisons in Figure 11 should be done while controlling the search budget for each setting and it would be even better to show multiple comparisons under different search budgets. \n\n4. More information about the loss landscape visualization would be appreciated, especially how the losses are computed for the supernet. Since the supernet can be seen as a larger standalone network, it is unintuitive to me why its loss landscape would be so different.  \n\n5. The presentation could be improved to make the main message more clear. Some specific points below:\n\nFigure 1 is quite confusing. From the description, it seems the main message is that, from (a) to (c), more and more parts in the whole process are controlled, so it would help to align the elements from different sub-figures to make it easier to compare and see the highlighted differences. Right now, it is a bit cluttered and hard to see the main point. It would also help to simplify the elements. For example, the sub-figures seem to be also describing the differences in the search algorithm's relationship with the other components, but it is not very clear and there isn't any discussion about it. \n\nThe use of \"the proxy task\" seems confusing. From my understanding, this term is usually used to refer to a surrogate task, when the target task is too expensive to search on. An example from (Cai et al., 2018): \"they need to utilize~\\emph{proxy} tasks, such as training on a smaller dataset, or learning with only a few blocks, or training just for a few epochs. These architectures optimized on proxy tasks are not guaranteed to be optimal on the target task.\". However, this paper seems to use \"proxy task\" to refer to the target task that the NAS is optimizing for. It would help to clarify the difference in the paper. \n\nThe term \"super-net accuracy\" and how it is computed should be defined early on since it is used quite frequently and could be ambiguous to the reader. \n\nTypo:\n\nAppendix B. \"This is not a reliable metric, as shown in Fig. 9 in the main paper.\" ==> I guess it should be \"Fig. 8\" instead of \"Fig. 9\" since Fig. 9 is discussing \"Low fidelity estimates on NASBench-201\".\n\n\nBender, Gabriel, et al. \"Can Weight Sharing Outperform Random Architecture Search? An Investigation With TuNAS.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\nCai, Han, Ligeng Zhu, and Song Han. \"Proxylessnas: Direct neural architecture search on target task and hardware.\" arXiv preprint arXiv:1812.00332 (2018).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an analysis of different tricks for training the super-network in NAS. While all reviewers see value in some of the many experiments, all reviewers also have substantial criticisms of the paper, and all reviewers gave weak rejection scores.

Looking at the paper myself, I agree with this assessment. Several of the experiments are valuable, but there are also several substantial issues.

One question that confused two reviewers and myself is about using sparse Kendall's tau as a metric that the authors in the rebuttal again state can be computed during super-net training to evaluate the quality, just like super-net accuracy. I don't see how that is possible. Kendall's tau measures the correlation between the ranks of the performances of the stand-alone architectures and the ordering implied by the super-net. Computing this requires access to the performance ranks of the stand-alone architectures. For tabular benchmarks this is of course available, but not in practical NAS applications.

I would also like to echo the concern of AnonReviewer2 that too little information is given to fully understand what is shown in Figure 10.

Some reviewers also questioned inhowfar the results generalize to the setting of the Once-for-all-network or BigNAS. This was not a deciding factor for me, since insights based on NAS-Bench101, 201 and a DARTS-like search spaces are already very useful.

I agree with the reviewers that the authors' use of "proxy" is highly misleading. It is standard to refer to the low-fidelity model used for training as the proxy model. In contrast, the authors use it for the final evaluation model.

Concerning the authors' five final take-aways:
1) I don't see how sparse Kendall's tau is actionable.
2) The batch normalization part is interesting, and I agree with the authors that it is useful to spell this out and analyze it, rather than just having one sentence in the paper as NB-201 and TuNAS, but the attribution that this has been done before is broken. "In contrast to X", rather than "Like X"
3) This is interesting, although I agree with AnonReviewer3 that I'm lacking intuition why a smaller learning rate should be useful for a less smooth space
4) The experiment on low fidelity estimates is very misleading. The proxy settings used during training are already low fidelity evaluations -- for the final evaluation, you would increase the number of channels, number of layers and number of epochs. Stating that the use of low fidelities is not useful is highly misleading. The authors' experiments only shows that the proxy model is already well chosen, and that if you reduce #layers or #channels and proportionally increase #epochs, performance gets worse. I encourage the authors to try searching without this proxy model, and I'm sure they will find that (which correlations might increase) the search process will be far too slow.
5) The insight on dynamic channeling appears very useful to me.

In summary, I recommend rejection and encourage the authors to address the points raised by the reviewers and in this meta-review.
Decision: Reject