Scaling Up Neural Architecture Search with Big Single-Stage Models
Authors: Jiahui Yu
Pengchong Jin
Hanxiao Liu
Gabriel Bender
Pieter-Jan Kindermans
Mingxing Tan
Thomas Huang
Xiaodan Song
Quoc Le
Venue:  (2020)
Keywords: Single-Stage Neural Architecture Search

Abstract: Neural architecture search (NAS) methods have shown promising results discovering models that are both accurate and fast. For NAS, training a one-shot model has became a popular strategy to approximate the quality of multiple architectures (child models) using a single set of shared weights. To avoid performance degradation due to parameter sharing, most existing methods have a two-stage workflow where the best child model induced from the one-shot model has to be retrained or finetuned. In this work, we propose BigNAS, an approach that simplifies this workflow and scales up neural architecture search to target a wide range of model sizes simultaneously. We propose several techniques to bridge the gap between the distinct initialization and learning dynamics across small and big models with shared parameters, which enable us to train a single-stage model: a single model from which we can directly slice high-quality child models without retraining or finetuning. With BigNAS we are able to train a single set of shared weights on ImageNet and use these weights to obtain child models whose sizes range from 200 to 1000 MFLOPs. Our discovered model family, BigNASModels, achieve top-1 accuracies ranging from 76.5% to 80.9%, surpassing all state-of-the-art models in this range including EfficientNets.

Under review as a conference paper at ICLR 2020
SCALING UP NEURAL ARCHITECTURE SEARCH WITH BIG SINGLE-STAGE MODELS
Anonymout moonymous authors Paper under double-blind review
ABSTRACT
Neural architecture search (NAS) methods have shown promising results discov- ering models that are both accurate and fast. For NAS, training one-shot model has became a popular strategy to approximate the quality of multiple architectures (child models) using a single set of shared weights. To avoid performance degra- dation due to parameter sharing, most existing methods have a two-stage workflow where the best child model induced from the one-shot model has to be retrained or finetuned. In this work, we propose BigNAS, an approach that simplifies this workflow and scales up neural architecture search to target a wide range of model sizes simultaneously. We propose several techniques to bridge the gap between the distinct initialization and learning dynamics across small and big models with shared parameters, which enable to train a single-stage model: single model from which we can directly slice high-quality child models without retraining or finetuning. With BigNAS we able to train a single set of shared weights on Im- ageNet and use these weights to obtain child models whose sizes range from 200 1000 MFLOPs. Our discovered model family, BigNASModels, achieve top-1 accuracies ranging from 76.5% to 80.9%, surpassing all state-of-the-ar models in this range including EfficientNets.
INTRODUCTION
Designing network architectures that are both accurate and efficient is crucial for deep learning on edge devices. well known that single neural network architecture can require more than an order of magnitude more inference time if it is deployed on slower device (Yu et al.) (018). This makes appealing to not only search for architectures that are optimized for specific devices, but also to ensure that range of models can be deployed effectively.
In the past, Neural Architecture Search (NAS) methods (Zoph & Le 2016 Zoph et al., 2018 Real et al. 2018) have shown to be excellent at optimizing for single device and latency target (Tan et al. 2019). However, if we wish to target a large array of devices, it becomes prohibitively expensive and time-consuming basic scaling heuristics such as the EfficcinnNet family (Tan & Lej (019). However, this loses on opportunities to optimize models specialized for the diverse performance characteristics of individual devices, Another option would be to use efficient architecture search methods, e.g., e.g., Phha al. (2018); Bender al. (2018); LLu et al. (2018b). However, target multiple devices, must run many searches and retrain all of the searched models from scratch.
separate search for each one. A possible solution to this is
In this work, we search over a big single-stage model that contains both small child models (~200 MFLOPs, comparable to MobileNetV3) and big child models (~1 GFLOPs, comparable to Effi- cientNet B2). Different from existing one-shot methods (Bender et al.) 2018, Liu et af. 2018h Brock et al. 2018 [hha et al.) 2018), our trained single-stage model offers much wider coverage of model capacities, and more importantly, all child models are trained in way such that they simul- taneously reach excellent performance the end of the search phase, without requiring separate retraining step. Architecture selection can be then carried out via a simple coarse-to- to-fine selection strategy. Once an architecture is selected, we obtain child model by slicing the single-stage model for instant deployment w.r.t. the given constraints such as memory footprint and/or runtime latency. The workflow is illustrated in Figure T

---

Under review as a conference paper at ICLR 2020
One-shot model
Progressive shrinking
Single-stage model (ours)
Retrain and deploy
Deploy instantly
(without retraining)
Distillaion
Comparison with several existing workflows. We use nested squares denote models with shared weights, and use the size of the square to denote the size of each model. Workflow in the middle refers the concurrent work from Cai et al. (2019), where submodels are sequentially induced through progressive distillation and channel sorting, We simultaneously train all child models in single-stage model with proposed modifications, and deploy them without retraining or finetuning.
Figure I:
The success of our method heavily relies training high-qualing single-stage model, which is challenging on its own, For example, we find the training loss explodes if the single-stage model is not properly initialized, and bigger child models start overfit before smaller ones plateau. It is particularly nontrivial to simultancously retain good performance on every individual child model due to aggressive parameter sharing during architecture search. We address these challenges through a combination of techniques, including an improved sampling strategy and efficient inplace distilla- tion, and substantially stabilize the single-stage model training through better initialization, learning rate schedule and regularization. Effectiveness of the proposed solutions is backed up by ablation studies.
With the proposed techniques, we are able train a single-stage model on ImageNet and obtain family of child models that simultaneously surpass all the state-of-the-ard models in the range of 200 to 1000 MFLOPs, including EfficientNets BO-B2 (1.6% more accurate under 400 MFLOPs), without retraining or finetuning the child models upon the completion of search. One our child models achieves 80.9% t accuracy 1G FLOPs (four times less computation than ResNet-50).
RELATED WORK
Earlier NAS methods (Zoph & Le 2016 Zoph et al. 2018 Liu et al., 2017 2018a; Real et al., 2018) train thousands of candidate architectures from scratch (on a smaller proxy task) and use their validation performance the feedback to algorithm that learns to focus the most promising regions in the search space. More recent works have sought to amortize the cost by training a single wwer-parameterized one-shot model. Each architecture in the search space only subset of the operations in the one-shot model; these child models can be efficiently ranked by using the shared weights to estimate their relative accuracies (Brock et al., 2018: Pham et al. 2018 Bender et al., 2018
Liu et al., 2018b Cai et al. 2018 Wu et al.) 2019).
As complementary direction, resource-aware NAS methods proposed simultaneously max- imize prediction accuracy and minimize resource requirements such as latency, FLOPs. or memory footprints (Tan al. 2019 Cai al. 2019 Wu 2019 Stamoulis al. 2019 Guo al. 2019
Yu & Huang 2019a).
All the aforementioned approaches require two-stage training: Once the best architectures have been identified (either through the proxy tasks or using one-shot model), they have to be retrained from scratch obtain a final model with higher accuracy. most of these existing works, single search experiment only targets single resource budget or narrow range of resource budgets at time.
To alleviate these issues, Cai et al. (2019) proposed a progressive training approach (OFA) coneur- rently with our work. The idea is pre-train single full network and then progressively distill to obtain the smaller networks. Moreover, a channel sorting procedure is required to progres- sively construct the smaller networks. In proposed BigNAS, however, all the child models in the single-stage model are trained simultaneously, allowing the learning of small and big networks to mutually benefit each other. During the training, we always keep lower-index channels in each layer and inder-index layers each stage for our child models, eliminating the sorting procedure. While OFA focuses limited range of model sizes (for example, models around 300 MFLOPss,,
2

---

Under review as a conference paper at ICLR 2020
our BigNAS is able to handle a wider set of models (from 200 MFLOPs to 1 GFLOPs) and offers better coverage over diverse deployment scenarios and varied resource budgets.
Our work shares high-level similarities with slimmable networks Yu al.) 2018 Yu & Huang. 2019bla) in terms of training a single shared set of weights which can be used for many child models. However, while slimmable networks are specialized to vary the number of channels only, we are able to handle a much larger space where many architectural dimensions (kernel and channel sizes, network depths, input resolutions) are searched simultaneously, subsuming and outperforming the scaling heuristics in EfficientNets (Tan & Le 2019).
3 ARCHITECTURE SEARCH WITH SINGLE-STAGE MODELS
Our proposed method consists of two steps:
1. We train big single-stage model from which we can directly sample or slice different architectures as child models for instant inference and deployment. In contrast to previ- works (Brock et al.) 2018) Pham et al., 2018 [ender et al. 2018 Liu et al.) 2018b] Stammulii et al. 2019; Guo et al. 2019), our training is single-stage as it does not require finetuning the sampled architectures retraining them from seratch the end of search.
2. Architecture selection using a simple course-to-fine selection method to find the most ac- curate model under the given resource constraints (for example, FLOPs, memory footprint and/or runtime latency budgets on different devices).
In the following, we will first systematically study how to train a high-quality single-stage model from five aspects: network sampling during training, inplace distillation, network initialization, con- vergence behavior and regularization. Then we will present a coarse-to-fine approach for efficient resource-aware architecture selection.
3.1 TRAINING A HIGH-QUALITY SINGLE-STAGE MODEL
Training a high-quality single-stage model is important and highly non-trivial due to the distinct initialization and learning dynamics of small and big child models. In this section, we first generalize two techniques originally introduced by Yu & Huang (2199) to simultancously train a set of high- quality networks with different channel numbers, and show that both can be extended to handle a much larger space where the architectural dimensions, including kernel sizes, channel numbers, input resolutions, network depths are jointly searched. We then present three additional techniques to address the distinct initialization and learning dynamics of small and big child models.
Sandwich Rule In each training step, given a mini-batch of data, the sandwich rule (Yu & Huang. 2019b) samples the smallest child model, the biggest (full) child model and N randomly sampled child models (N = 2 in our experiments). It then aggregates the gradients from all sampled child models before updating the weights of the single-stage model. As multiple architectural dimensions are included in our search space, the "smallest" child model is the one with lowest input resolution, thinnest width, shallowest depth, and smallest kernel size (the kernel of the depthwise convolutions in each inverted residual block (Sandler et al.) (0188). The motivation is to improve all child models in our search space simultancously, by pushing up both the performance lower bound (the smallest child model) and the performance upper bound (the biggest child model) across all child models.
Inplace Distillation During the training of a single-stage model, inplace distillation Yu & Huang 2019b) takes the soft labels predicted by the biggest possible child model (full model) to supervise all other child models. The benefit of inplace distillation comes for free our training setting, as we always have access to the predictions of the largest child model in each gradient update step thanks to the sandwich rule. We note that all child models are only trained with the inplace distillation loss, starting from the first training step the end of the training.
During training, input images are randomly cropped as a preliminary data augmentation step. When distilling a high-resolution teacher model into oowrresolution student model, we find that it is helpful to feed the same image patches into both the teacher and the student. In our data preparation, first randomly crop an image with fixed resolution (on ImageNet we use 224), and then apply
3

---

Under review as a conference paper at ICLR 2020
bicubic interpolation to the same patch to transform it into all target resolutions (e.g., 192, 288, 320). In this case, soft labels predicted by the biggest child model (the teacher) are more compatible with the inputs seen by other child models (the students). Therefore this can serve as more accurate distillation signal. Our preliminary results show that sampling different patches even in a same image leads to 0.3% drop on top-1 accuracy for child models.
Initialization Previous weight initialization methods, such as "He Initialization" (He et al. 2015
are deduced from fixed neural networks where the number of input units (the fan-in) is constant. The principal motivation of these initialization methods is to keep the variance of the responses in each layer unchanged, so that the forward information signals and the backward gradients will not be reduced or magnified exponentially as the network goes deeper. For example, He et al. (2015) suggested to initialize the variance of the weights as for convolutions with ReLU activations.
However, the above is ill-fitted for initializing a single-stage model, where n is no longer a constant across different child models with varied kernel sizes and input channels. This issue is exaggerated when we train bigger and deeper single-stage models. In practice, we find the training loss of single-stage model explodes when we use the optimized learning rates for training a normal network. The training starts to work when reduce the learning rate to 30%, but it leads to much worse results (~ 1.0% top-1 accuracy on ImageNet).
Identifying this issue is critical while the solution is quite simple. As all child models in our search space are residual networks, we initialize the output of each residual block (before skip connection) to all-zeros tensor by setting the learnable scaling coefficient y 0 in the last Batch Normal- ization (loffe & Szegedy] 2013) layer of each residual block, ensuring identical variance before and after each residual block regardless of the fan-in. This initialization is originally mentioned in Goyal et al. 2017) which improves accuracy by ~ 0.2% in their setting, yet is more critical in our setting (improving by 1.0%) due the above analyzed initialization issue. We also additionally add skip connection between each stage transitions when either resolutions or channels differ (us- ing 2 x 2 average pooling and/or 1 x convolution if necessary) to explicitly construct an identity mapping (He et al. 2016b).
Convergence Behavior In practice, we find that big child models converge faster while small child models converge slower. Figure 2a shows the typical learning curves during the training of single-stage model, where we plot the validation accuracies of small and a big child model over time. This reveals a dilemma: at training step t when the performance of big child models peaks, the small child models are not fully-trained; and at training step t' when the small child models have better performance, the big child models already overfitted.
5:25
0.82
Small Chic Mocel
Schedule: Expensncialyy Decaying
Big
80.9%
Schooule: Expenencially Decaying with Constant Ending
0.80
70.3
6:20-
0.78
0.76
76.
0.15
Inepeien Top.
0.74
75.4%
0:72
0.50
0.70
Coestur Exding
20000
40020
100000 Number of Training Shops
65020
80020
120000
20000
40000
60002
50000
100000
120000
Number Training Sheps
(a)
(b)
Figure 2: On the left, we show typical accuracy curves during the training process for both small and big child models. On the right, we plot the modified learning rate schedules with constant ending.
To address this issue, we put our focus on the learning rate schedule. We first plot the optimized and widely used exponentially decaying learning rate schedule for MobileNet-series (Howard et al.) 2017 Sandler et al. 2018 Howard et al.) (2019), MNasNets (Tan et al.) 2019) and EfficientNets (Tan) & Le 2019) in Figure 2b We introduce simple modification this learning rate schedule, named
4

---

Under review as a conference paper at ICLR 2020
exponentially decaying with constant ending, which has constant learning rate the end of training when it reaches 5% of the initial learning rate (Figure[bb). brings two benefits. First, with a slightly larger learning rate at the end, the small child models learn faster. Second, the constant learning rate at the end alleviates the overfitting of big child models as the weights oscillate.
Regularization Big child models tend to overfit the training data whereas small child models tend to underfit. In previous work, [Bedder et al.] (018) apply the same weight decay to all child mod- els regardless whether they are small or big. In EfficientNet, (Tan & Le (2019) linearly increase dropout (Srivastava et al. 2014) ratio it moves from the smallest EfficientNet-BO to the biggest EfficientNer-B7. This becomes even more complicated in the context of training big single-stage models, due to the interplay among the small child models and big child models with shared pa- rameters. Nevertheless, we introduce a simple rule that is surprisingly effective for this problem: regularize only the biggest (full) child model (i.e., the only model that has direct access to the ground truth training labels because of inplace distillation). We apply this rule to both weight decay and dropout, and empirically demonstrate its effectiveness in our experiments.
Batch Norm Calibration Batch norm statistics are not accumulated when training the single- stage model as they are ill-defined with varying architectures. After the training is completed, we re-calibrate the batch norm statistics (Yu & Huang 2019b) for each sampled child model for de- ployment without retraining or finetuning any network parameters.
3.2 COARSE-TO-FINE ARCHITECTURE SELECTION
After training a single-stage model, one needs to select the best architectures w.r.t. the resource budgets. Although obtaining the accuracy of a child model is cheap, the number of architecture candidates is extremely large (more than 1022). To address this issue, we propose a coarse-to-fine strategy where we first try to find a rough skeleton of promising network candidates in general, and then sample multiple fine-grained variations around each skeleton architecture of interest.
Specifically, in the coarse-grained phase, we define limited input resolution set, depth set (global depth multipliers), channel set (global width multipliers) and kernel size set, and obtain benchmarks for all child models in this restricted space. This followed by fine- finegrained search phase, where we first pick the best network skeleton satisfying the given resource constraint found in the previous phase, and then randomly mutate its network- wise resolution, stage-wise depth, number of channels and kernel sizes to further discover better network architectures. Finally, we directly use the weights from the single-stage model for the induced child models without any retraining or finetuning. More details will be presented in the experiments.
EXPERIMENTS
In this section, we first present the details of our search space, followed by our main results com- pared with the previous state-of-the-arts in terms of both accuracy and efficiency. Then we conduct extensive ablative study to demonstrate the effectiveness of our proposed modifications. Finally, we show the intermediate results of our coarse-to- soarse-to-fine architecture selection.
4.1 SEARCH SPACE DEFINITION
Following previous resource-aware NAS methods Tan et al. 2019; Tan & Le 2019 Cai et al., 2018 Wu et al. 2019 Howard al. 2019: Wu et al. 2019), our network architectures consist of stack with inverted bottleneck residual blocks (MBConv) (Sandler et al.) 2018). The detailed search space summarized in Table I For the input resolution dimension, we sample from set (192, 224, 288, 320]. By training on different input resolutions, we find our trained single-stage model is able to generalize to unseen input resolutions during architecture search or deployment (e.g., 208, 240, 256, 272, 304, 336) after BN calibration. For the depth dimension, our network has seven stages (excluding the first and the last convolution layer). Each stage has multiple choices of the number of layers (e.g., stage 5 can pick any number of layers ranging from 2 to 6). Following slimmable networks (Yu et al.] 2018)) that always keep lower-index channels in each layer, we always keep lower-index layers in each network stage (and their weights). For weight sharing on the kernel size
5

---

Under review as a conference paper at ICLR 2020
Table I: MobileNetV2-based search space.
Stage
Operator
Resolution
#Channels
#laayess
Kemel Sizes
Conv
192 192 320 320
32-40
MBConv1
96 x 96 160
160
16 24
1-2
3
MBConv6
96 x 96 160
160
24 32
2-3
3
MBConv6
48 48 80 80
40
2-3
3,5
MBConv6
24
40
3,
MBConv6
12 12 20 20
112 128
2-
3,5
6
MBConv6
12 12 20 20
216
2-6
3,5
MBConv6
6 6 10 x 10
320 352
1-2
3,5
Conv
10 x
1280 1408
dimension in the inverted residual blocks, a x depthwise kernel defined to be the center of 5 5 depthwise kernel. Both kernel sizes and channel numbers can be adjusted layer-wise. The input resolution network-wise and the number of layers is stage-wise configuration in our search space.
4.2
MAIN RESULTS ON IMAGENET
BiMAsMMdd-1
Group
Model Family
Params
FLOPS
Top-1
8D
MobileNet MobileNet
1.3
150h1
63.3
60.8
BigNASMMddi
200M FLOPs
4.1M
207h
73.0
MobileNetVa uos
78-
MNnsNet a
3.5M
315M
75.2
ImageNet Top-1 Accuracy
Ouces For-All
4.4M
327M
15.0
numoet
76
PigNAAModel
BigNASModel-8
4.5M
242M
76.5
NASNet
72.8
MobaleNerV2 1.x
5.3
509M
74.4
400M FLOPs
MobileNetV3
6.1M
350M1
76.0
BigNASModel
76.7
EErrieuNNet
5,330
390b1
77.3
EfficientNet
MobileNetv3
RigN SModel-M
5.5M
418M
78.9
MobaleNetVL L.s
4.234
58901
70.9
AutoSili-MobillNeet2 800M PLOPs
NASNet
5.3M
66411
64.0
73.1
Onco-For-All
EEiriessNet
7.8M
731M
79.2
70
MobileNetV2
BigN SModel
6.7M
659M
79.6
NASNet
MobileNetV1
10000 FLOPS
BigNASModcl-L
EliciessNet is
9.5M
9.2M
1040M
105081
80.9
80.3
68
200
400
600
1000
Millions of Multiply-Adeed Adds (MFLOPs)
Figure 3: Main results of BigNASModels on ImageNet.
We train our big single-stage model on ImageNet (Deng et al. 2009) using same settings as Tan
al. 2019 Tan & Le 2019; (Hwwrd et al.) (017). We use RMSProp optimizer with decay 0.9 and momentum 0.9; batch normalization with post-calibration YYu & Huang 2019b); weight decaying
factor 5; initial learning rate 0.256 that decays by 0.97 every epochs. We also use the swish activation (Ramachandran et al.] 2017) and fixed AutoAugment VO policy (Cubuk et al.) 2018) following EfficientNets (Tan & Le 2019). In addition to these training settings, we train our big single-stage model with all techniques proposed in Section 3.1 The learning rate is truncated to constant value when it reaches 5% its initial learning rate (i.e., 0.0128) until the training ends. We apply dropout only training the full network with dropout ratio 0.2, and weight decaying only full network once in each training iteration. To train the single-stage model, we adopt the sandwich sampling rules and inplace distillation proposed by Yu & Huang (019h)). After the training.

---

Under review as a conference paper at ICLR 2020
simple coarse-to- fine architecture selection to find the best architecture under each interested resource budgets. We will show the details coarse-to- cosse-to-finn fine architecture selection in Section 4.4
We show the performance benchmark of model family, named BigNASModels, in Figure On the left we show the visualization FLcur-Accuracy benchmarks compared with the previ- state-of-the-arts including MobileNetV1 (Howard al. 2017), NASNet (Zoph al. 2018 MobileNetV2 (Sandler et al.) 2018), AutoSlin-Mobileeett Yu & Huang 2019a), MNasNet (Tan
al., 2019), MobileNetV3 (Howard et al. 2019), EfficientNet (Tan Le| 2019) and concurrent work Once-For-All (Cai et al. 2019). We show the detailed benchmark results on the right table. For small-sized models, our BigNASModel-S achieves 76.5% accuracy under only 240 MFLOPs, which is 1.3% better than MobileNetV3 in terms of similar FLOPs, and 0.5% better than ResNet-50 (He) al. 2016a) with 17 fewer FLOPP. For medium-sized models, our BigNASModel-M achieves 1.6% better accuracy than EfficientNet BO. For large-sized models, even when ImageNet classifica- tion accuracy saturates, our BigNASModel-L. still has 0.6% improvement compared with Efficient- Net B2. Moreover, instead of individually training models of different sizes, our BigNASModel-S, BigNASModel-M and BigNASModel-L are sliced directly from one pretrained single-stage model, without retraining finetuning.
4.3
ABLATION STUDY
0.-
Imsgete Tep-
Imugoh Top's
0.3
Tram boor
02
0.1
staalaato 30% Losning
0.1
Ie retulivatonn 36% Loamiing Rane
Baplods
Mndired aladiin Lasning fatts
arcatratio Luaning faro
0.0
0.0
6050
L0020
5020
L0000
Number Traiing
Nuumbrr Trassing Staps
Figure 4: Focusing on the start of training. Ablation study on different initialization methods. We show the validation accuracy of small (left) and big (right) child model.
0.50
0.00
Imagrtet Top- Accunyy
Imaghhet top- Ao0-
0.75
0.75
0:5
0.65
Nown relaiiatee toome foxe
tutlubition 93% toaring
insaiiation
rntaltatian Leuring
0.50
0,60
20003
40363
65000
83000
300003
120303
20003
40320
63530
5020n
100000
120020
Kamber of Traning Steps
Rumber of ming Steps
Figure 5: Focusing on the end of training. Ablation study on different initialization methods. We show the validation accuracy of small (left) and big (right) child model.
Ablation Study on Initialization. Previous weight initialization methods (He et al. 2015) are deduced from fixed neural networks, where the numbers of input units is constant. However, in a stngle-stage model, the number of input units varies across the different child models. In this part, we start with training a single-stage model using He Initialization (He et al.] 2013) designed for fixed neural networks. As shown in Figure the accuracy of both small (left) and big (right) child
7

---

Under review as a conference paper at ICLR 2020
models drops to zero after few thousand training steps during the learning rate warming-up (Goyal et al. 2017). The single-stage model is able to converge when we reduce the learning rate to the 30% of its original value. If the initialization is modified according to Section the model learns much faster at the beginning of the training (shown in Figure 4) and has better performance at the end of the training (shown in Figure Moreover, we are also able train the single-stage model with the original learning rate hyper-parameter, which leads to much better performance for both small (Figure left) and big (Figure right) child models.
632
0.82
680
0.80
inwyyten Tops Acuueyy
0.78
0.79
0.76
0.76
6.74
0.74
6.72
0.72
0.70
0.70
0.68
0.68
Small Medel ne Child Model
0.66
Ci st Model
0.66
20002
.0002
620-32
0:0000
100000
120203
20000
40350
60300
100020
120020
Number of Theining Stepe
Number of Thi ming Steps
Figure 6: The validation accuracy curves during the training process for both small and big child models before (left) and after (right) our modifications.
Ablation Study on Convergence Behavior: During the training of a single-stage model, the big child models converge faster and then overfit, while small child models converge slower and need more training. In this part, we show the performance after addressing this issue in Figure 6 We apply the proposed learning rate schedule exponentially decaying with constant ending on the right. The detailed learning rate schedules shown in Figure 2b We also tried many other learning rate schedules with an exhaustive
sweep, including linearly decaying (Ma et al.) 2018 Yu & Huang 2019b) and cosine decaying (Loshchilov & Hutter 2016
He al. 2019). But the
performances are all worse than exponentially decaying.
0.70
0.32
0.76
0.80
imspeen Topa M400R
Imppolet 3cor
6.72
0.78
0.58
0.76
0.74
0.6
helve Repu aetzatitiinnn Modited Regulartzzion
Natoe Segutartontion Moctted Regulartzaiion
052
0.22
33000
23030
70003
20083
30002
70003
Figure 7: The validation accuracy of small (left) and big (right) child model using different regu- larization rules.
Ablation Study on Regularization. Big child models are prone to overfitting on the training data whereas small child models are prone to underfitting. In this part, we compare the effects of the reg- ularization between two rules: (1) applying regularization on all child models (Bender et al. 2018), and (2) applying regularization only on the full network. Here the regularization techniques we consider are weight decay with factor 1e 5 and dropout with ratio 0.2 (the same used in training previous state-of-the-an mobile networks). In Figure of both small (left) and big (right) child models using different regulanizatin rules. On the left, the performance of small child models is improved by large margin (+0.5 top-1 accuracy) as it has
show the performance
8

---

Under review as a conference paper at ICLR 2020
less regularization and more capacity to fit the training data. Meanwhile the right, found the performance of the big child model is also improved slightly (+0.2 top-1 accuracy).
4.4 COARSE-TO-FING ARCHITECTURE SELECTION
After the training of a single-stage model, we use coarse-to-fin architecture selection to find the best architectures under different resource budgets. During the search, the evaluation metrics can be flexible including predictive accuracy, FLOPs, memory footprint, latency on various different devices, and many others. is noteworthy that we pick the best architectures according to the predictive accuracy on training set, because we used all training data for obtaining our single-stage model (no retraining from scratch), and the validation set of ImageNet (Deng et al.) 2009) is being used as "test set" in the community. In this part, we first show an illustration of our coarse-to-fing architecture selection with the trained big single-stage model in Figure [ The search results are based FLOPs-Accuracy benchmarks (as FLOPs are more reproducible and independent of the software version, hardware version, runtime environments and many other factors).
0.82
0.795
inagehett Top-3 AcUracy
0.80
Imagelee Tap-1 AcCUFaCy
0.292
0.78
0.790
11
0.77
0.76
0.788
Coarsie grained Architecture Selection
Fine grained Architecture Selection
200
400
600
80B
1000
1200
1400
1600
1800
300
550
600
650
700
750
860
850
900
Millons of Mutiily-Adi (MP)
Millonz of Nutply Adds (MFLOPS)
Figure 8: Benchmark results of coarse-to-fine architecture selection. The red dot in coarse-grained architecture selection is picked and mutated for fine-grained architecture selection.
During the coarse-to-fine architecture selection, we first find rough skeletons of good candidate networks. Specifically, in the coarse selection phase, we pre-define five input resolutions (network- wise, [192, 224, 256, 288, s20)), four depth configurations (stage-wise via global depth multipliers (Tan) & Le 2019)), two channel configurations (stage-wisg wise via global width multipliers (Howard al. 2017) and four kernel size configurations (stage-wise). and obtain all of their benchmarks (shown in perform fine-gras uineggrineed grid search by varying its configurations (shown in Figure Jon the right). For example. under FLOPs near 600M first pick the skeleton of the red dot shown in Figure 8 We then perform additional fine-grained architecture selection by randomly varying the input resolutions, depths, channels and kernel sizes slightly. We note that the coarse-to-fine architecture selection is flexible and not very exhaustive in our experiments, yet it already discovered fairly good architectures shown in Figure 8 on the right. For the FLOPss near 650M, we finally select the child model with input resolution 256, depth configuration (12222441) channel configuration (32:1624448888 1222163522210888 1tl.::/48.88..2223322222.. and kernel size configuration (1383:33) After training of the single-stage model, the post-search step is highly parallelizable and independent of training.
Figure 8
on the left). Then under interested latency budget,
CONCLUSION
We presented a novel paradigm for neural architecture search by training a single-stage model, from which high-quality child models of different sizes can be induced for instant deployment without retraining or finetuning. With several proposed techniques, we obtain a family of BigNASModels as slices in a big pre- pre-trained single-star stage model. These slices simultancously surpass all state-of art ImageNet classification models ranging from 200 MFLOPs to 1G FLOPs. We hope our work can serve further simplify and scale up neural architecture search.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jazefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg. Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg. Martin Wicke, Yuan Yu, and Xiaogiang Zheng. TensorFlow: Lcree-scale machine learn- ing on heterogencous systems, 2015. URL hitps: /www www.wwwwsooo....ooooooo org/ Software available from tensorflow.org.
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Understand- ing and simplifying one-shot architecture search. In International Conference on Machine Learn- ing. pp- 549-558, 2018.
Andrew Brock, Theo Lim, J.M. Ritchie, and Nick Weston. SMASH: One-shot model architecture search through hyperneworks. In International Conference Learning Representations, 2018. URL
ew net met//fot forrmm i dieimmmdd=rdeee d=rydeCEhs-
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprini arXiv: 00332, 2018.
Han Cai, Chuang Gan, and Song Han. Once for all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.0979) 2019.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment:
Learning augmentation policies from data. arXiv preprint vrXiv:/805.09500 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 248-255, leee, 2009.
Priya Goyal, Piotr Dollár, Ross Girshick, Picter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint (idi:::706.066777 2017.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. arXiv preprint arXiv:1904.00422 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEFE international conference on computer vision, pp. 1026-1033 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer. 2016b.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 558-567, 2019.
Andrew Howard. Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. arXiv preprint arXiv: 19050244, 2019.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: /t0l::/704.00000 1704.04661, 2017.
10

---

Under review as a conference paper at ICLR 2020
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceed- ings of the European Conference on Computer Vision (ECCV), pp. 19-34, 2018a.
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hier- archical representations for efficient architecture search. arXiv preprint urXiv:1711.0000 2017.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv
preprint arXiv:/806.0900 09055, 2018b.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608. 03988, 2016.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 116-131, 2018.
Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint 1FXXi:/802.03268 2018.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv: 1710.05941 2017.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. arXiv preprint arXiv:/802.0/544 2018.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen. In- verted residuals and linear bottlenecks: Mobile networks for classification, detection and segmen- tation. arXiv preprint /rXiv:/801.. 08381, 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Iyy Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958, 2014.
Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Single-path nas: Designing hardware-efficient convnets in less than hours. arXiv preprint arXiv:1904.0280 2019.
Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. Proceedings of the International Conference on Machine Learning (ICML), 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoe V Le. Mnasnet: Platform- aware neural architecture search for mobile. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang. Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-awee efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 10734-10742, 2019.
Jiahui Yu and Thomas Huang. Network slimming by slimmable networks: Towards one-shot archi- tecture search for channel numbers. arXiv preprint arXiv: 1903. 2019a.
Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques. arXiv preprint arXiv:1903.05/5: 05144 2019b.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. arXiv preprint 182,0888,, 2018.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint 1.01588. 2016.
11

---

Under review as a conference paper at ICLR 2020
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
A ARCHITECTURES OF BIGNASMODEL
We show the architecture visualization of the single-stage model and child models BigNASModel- BigNASModel-M, BigNASModel 660M, BigNASModel-L in Figure 9 The child models are directly sliced from the single-stage model without retraining or finetuning. Compared with the com- pound model scaling heuristic (Tan Le 2019), our child models have distinet architectures across all dimensions. For example, comparing BigNASModel-L with EfficientNet-B2, the EfficientNet- B2 has input resolution 260, channels ers (23314452)). Our BigNASModel-L achieves 80.9% top-1 accuracy under 1040 MFLOPs, while EfficientNet-B2 achieves 80.3% top-1 accuracy under 1050 MFLOPs.
kernel sizes (3333:35: and stage lay-
B IMPLEMENTATION DETAILS
We implement all training and coarse-to- architecture selection algorithms on TensorFlow frame- work (Abadi al. 2015). All of our experiments conducted on 8 x 8 TPUv3 pods. For ImageNet experiments, we use a total batch size 4096. Our single-stage model has sizes from 200 to 2000 MFLOPs, from which we search architectures from 200 to 1000 MFLOPs. To train single-stage model, it roughly takes 36 hours.
Training on TPUs requires defining static computational graph, where the shapes of all tensors in that graph should be fixed. Thus, during the training we are not able to dynamically slice the weights, select computational paths sample many input resolutions. To this end, here we provide the details of our implementation for training single-stage models on TPUs. On the dimensions of kernel sizes, channels, and depths, we use the masking strategy to simulate the weight slicing or path selection during the training (i.e., we mask out the rest of the channels, kernel paddings, or the entire output of a residual block). On the dimension of input resolutions, in each training iteration, our data pipeline provides same images with four fixed resolutions ((192, 224, 288, 320}) which are paired with the model sizes. The smallest child model is always trained on the lowest resolution, while the biggest child model is always trained the highest resolution. For all other resolutions the models are randomly varied on kernel sizes, channels, and depths. By this implementation, our trained stngle-stage model is able to provide high-quality child models across all these dimensions. For inference, we directly declare a child model architecture and load the sliced weights from the single-stage model. To slice the weights, we always lower-index channels in each layer, lower- index layers in each stage, and the center 3 x 3 depthwise kernel from a 5 x 5 depthwise kernel.
For the data prefetching pipeline, we need multiple image input resolutions during the training. We first prefetch a batch of training patches with fixed resolution (on ImageNet we use 224) with data augmentations, and then resize them with bicubic interpolation to our target input resolutions (e.g., 192, 224, 288, 320). We note that during inference, the single-crop testing accuracy is reported. Importantly, for testing data prefetching pipeline, we also prefetch 224 center crop first and then resize to the target resolution to avoid the inconsistency.
During the training. we use cross-replica (synchronized) batch normalization following Efficient- Nets (Tan & Le 2019). To enable this, we also have to use stateless random sampling function since naive random sampling function
daatiionn'lldd Ileads to different sampled values across different TPU cores. The input seed of stateless random sampling functions is the global training step plus current layer index so that the trained single-stage model can provide child models with different seyer-wise/stagee wise configurations.
http:///ww.rrorrro.ooooo///ohh/r form
12

---

Under review as a conference paper at ICLR 2020
Model
MipNASMooke-S
BigNASModel-M
HigNASMModelrodii
BigNASModehL.
256:256322
Carw 20
Car 30
Gen
Cr
Stage
MDComns 20
VICeesL 30
MDConsd30 2c
MBCooed 20
ViCeri 20
MiKonn 20
Wiceen 20
Stage
MiCoosdTo TD
NiConniSo S6
ViKeniG
SIKCOn 2G
Stuge
MiCoossho PD
36:5:90
MBoax130 2d
MaCHnn 30
Stage
MbComv2 Tx0
1515212
Mibonw
ViKciiG
MICanV 3G
MiCooid
VICEIVI 20
MaCoii 20
Stage
Macorid
MICHIV 2G
Mitoomn AM/
MEDDev 3O
Macoousho Pr
MiCoone Sc
Wiccond
Micooio De
Miccev 26
VOCIVV-30
MBCIii 20
MDCIYS30
Stage
MiConis
MiCcowid 7d
MBconi33 20
MIConi 20
Stage?
Corw tsl
Com ist
Carv 2c
Care ici
Corv ici
Figure 9: Architecture visualization the single-stage model and child models BigNASModel BigNASModel-M. BigNASModel 660M, BigNASModel-L. All child models are directly sliced from the single-stage model without retraining or finetuning.
13


Reviews:
[{"review_id": "rkl2qQn3YH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a search for neural architectures with different resource requirements by training a single model only. Furthermore, models found at the end of the search require no additional post-processing and are ready for deployment. A weight-sharing technique is applied to make this happen. The authors argue that there are multiple adaptions required to make it work. This includes the child model sampling strategy, use of model distillation, weight initialization, learning rate schedule, regularization and batch norm calibration.\n\nThe work seems to be an incremental extension of Yu & Huang (2019b) and phrased as a NAS algorithm. Many techniques considered vital for the proposed method make use of techniques proposed by Yu & Huang (2019b) (sandwich rule, inplace distillation, batch norm calibration and the way how weights are shared). Other required techniques are either proposed by others (initialization) or very simple (learning rate schedule and regularization). The authors claim that they extend the work by Yu &Huang (2019b) \"to handle a much larger space where the architectural dimensions, including kernel sizes, channel numbers, input resolutions, network depths are jointly search\". They never clarify why this is a non-trivial step and they might want to point this out in their rebuttal.\n\nBesides this, the authors did a very good job. The paper is well-written, references are given wherever needed and all the closest related work is covered sufficiently. The experimental part conducts several ablation studies which supports their various decisions. Unfortunately, all results reported use heavy data augmentation which makes a comparison to other methods (besides EfficientNet) impossible. 600M is considered the upper bound for mobile architectures by the NAS community. Unfortunately, no such model is considered making it even harder to compare to existing NAS methods. The EfficientNet numbers reported don't match the ones reported in the original paper as far as I see. A red dot in Figure 3 could be added for the BigNASModel with 660M parameters.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyx1C_yMcr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for architecture search in deep neural networks in order to identify scaled-down networks that can operate on resource limited hardware. The approach taken in this paper is different from other approaches, which train a single big model then fine tune smaller models for specific hardware, or use distillation to train progressively smaller models. Here, a single model is trained in a manner that allows subsequent slicing to smaller models without additional training required. The authors employ a variety of strategies to get this to work well, including specific initialization techniques, regularization methods, learning schedules, and a coarse-to-fine optimization method to obtain the smaller models. The authors demonstrate SotA performance on ImageNet relative to other techniques.\n\nOverall, this was a well-written paper, and the results appear convincing. I would have liked a little bit more explanation about the implementation details though. As someone knowledgeable about deep ANNs, but not an expert in NAS for efficiency, I was not very clear on a couple of items. Specifically, I think it would be good to clarify the following:\n\n1) The authors say that they use a masking strategy to implement slicing during training. So, do I take it that they effectively do a separate pass through the network with different masks to implement the different sized models? If so, do you then simply accumulate gradients across all masks to do the updates?\n\n2) When the authors say they perform a fine grained grid search by varying the configurations, what is meant by this exactly? Do you mean that you do a search through different slicings of the big model to discover the best smaller architecture? What does it mean to do a grid search through binary masks? Or, is there a continuous hyperparameter that determines the masking which you do the grid search on? Maybe I’m just not understanding well at all, but even still, in that case, it indicates that this could be clarified in the manuscript.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Bye2FkL2tB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Parameter sharing (PS) is an important approach to speed up neural network search (NAS), which further allows the development of differential architecture search (e.g., DARTS) methods. However, PS also deteriorates the performance of learning models on the validation/testing set. \n\nThis paper first changes the search space from a DAG (micro+marco) in e.g., DARTS to a stacked one based on MBConv; and then, propose to use several tricks to train the super-net well. Finally, a search method is constructed for the supernet to find the desired architectures.\n\nOverall, the paper is too experimental. The method is an ensemble of existing approaches, i.e., every single component in the paper has been visited in the literature. Expect for experimental results, I do not see many general lessons we can learn from the paper. Finally, why the proposed method can be better than others is not well-explained and clarified. \n\nPlease see the questions below:\n\nQ1. Is NAS a method only for ImageNet? Can the method generalize to more applications/datasets?\n- While ImageNet is a good dataset for CV experiments, I think NAS should be a method for deriving architectures with certain requirements.\n- So, with so many tricks proposed in the paper, I wish authors can carry on experiments on other data sets as well, e.g., CIFAR and MNIST, which can still be preferred.\n\nQ2. On motivation, could authors explain more about the difficulties of combining all these techniques? \n- Each method is brought from some other paper, what motivate authors to combine them together? What makes them believe this is possible?\n\nQ3. On presentation, could authors draw a figure of the search space in the main text and give an overall algorithm for Section \"3.2 COARSE-TO-FINE ARCHITECTURE SELECTION\". It is hard for a reader to see novelties there.\n\nQ4. \"We also use the swish activation (Ramachandran et al., 2017) and fixed AutoAugment V0 policy\" - are all other compared methods using swish activation and AutoAugment V0 policy?\n\nQ5. How about the search efficiency of the proposed method? Only the accuracy is reported in the paper.\n\nQ6. Could authors give STD (i.e., gray area to represent STD) in Figure 4, 5 and 7? Some curves are too close, I am not sure they are statistically different.\n\nQ7. How is the performance of the super-net? \n\nQ8. Could the authors add an ablation study on this point? -  \"The motivation is to improve all child models in our search space simultaneously, by pushing up both the performance lower bound (the smallest child model) and the performance upper bound (the biggest child model) across all child models.\" \n- It is important to avoid fine-tune\n- From the paper, I am not sure whether the problem is solved by changing the space or the proposed training method (See Q1).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a NAS method that avoids having to retrain models from scratch and targets a range of model sizes at once. The work builds on Yu & Huang (2019) and studies a combination of many different techniques.
Several baselines use a weaker training method, and no code is made available, raising doubts concerning reproducibility.

The reviewers asked various questions, but for several of these questions (e.g., running experiments on MNIST and CIFAR) the authors did not answer satisfactorily. Therefore, the reviewer asking these questions also refuses to change his/her rating. 

Overall, as AnonReviewer #1 points out, the paper is very empirical. This is not necessarily a bad thing if the experiments yield a lot of insight, but this insight also appears limited. Therefore, I agree with the reviewers and recommend rejection.
Decision: Reject