SINGLE PATH ONE-SHOT NEURAL ARCHITECTURE SEARCH WITH UNIFORM SAMPLING
Authors: Zichao Guo
Xiangyu Zhang
Haoyuan Mu
Wen Heng
Zechun Liu
Yichen Wei
Jian Sun
Venue:  (2020)
Keywords: Neural Architecture Search
Single Path

Abstract: We revisit the one-shot Neural Architecture Search (NAS) paradigm and analyze its advantages over existing NAS approaches. Existing one-shot method (Benderet al., 2018), however, is hard to train and not yet effective on large scale datasets like ImageNet.  This work propose a Single Path One-Shot model to address the challenge in the training.  Our central idea is to construct a simplified supernet, where all architectures are single paths so that weight co-adaption problem is alleviated. Training is performed by uniform path sampling. All architectures (and their weights) are trained fully and equally.
Comprehensive experiments verify that our approach is flexible and effective.  It is easy to train and fast to search.  It effortlessly supports complex search spaces(e.g., building blocks, channel, mixed-precision quantization) and different search constraints (e.g., FLOPs, latency).  It is thus convenient to use for various needs. It achieves start-of-the-art performance on the large dataset ImageNet.

Under review as a conference paper at ICLR 2020
SINGLE PATH ONE-SHOT NEURAL ARCHITECTURE SEARCH WITH UNIFORM SAMPLING
Anonymous authors Paper under double-blind review
ABSTRACT
We revisit the one-shot Neural Architecture Search (NAS) paradigm and analyze its advantages over existing NAS approaches. Existing one-shot method (Bender et al., 2018). however, hard to train and not yet effective on large scale datasets like ImageNet. This work propose Single Path One-Shot model to address the challenge in the training. Our central idea is to construct a simplified supernet, where all architectures are single paths so that weight co-adaption problem is al- leviated. Training is performed by uniform path sampling. All architectures (and their weights) are trained fully and equally.
Comprehensive experiments verify that our approach is flexible and effective. It is easy to train and fast search. It effortlessly supports complex search spaces (c.g., building blocks, channel, mixed- mixed-precision quantization) and different search constraints (c.g., FLOPs, latency). It is thus convenient to for various needs. It achieves start-of-the-an performance on the large dataset ImageNet.
INTRODUCTION
Deep learning automates feature engineering and solves the weight optimization problem. Neural Architecture Search (NAS) aims to automate architecture engineering by solving one more problem, architecture design. Early NAS approaches (Zoph et al., 2018; Zhong et al., 201 8a;;; Liu et al., 2018a; Real et al., 2018: 8; Tan et al., 2018) solves the two problems in a nested manner. A large number of architectures are sampled and trained from scratch. The computation cost is unaffordable on large datasets.
Recent approaches (Wu et al., 2018a; Cai et al., 2018; Liu et al., 2018b; Xie et al., 2018; Pham al., 2018; Zhang et al., 2018c; Brock et al., 2017; Bender et al., 2018) adopt a weight sharing strategy to reduce the computation. A supernet subsuming all architectures is trained only once. Each architecture inherits its weights from the supernet. Only fine-tuning is performed. The computation cost greatly reduced.
Most weight sharing approaches use a continuous relaxation to parameterize the search space (Wu al., 201 Cai et al., 2018; Liu et al., 2018b; Xie et al., 2018: Zhang al., 2018c). The archi- tecture distribution parameters are jointly optimized during the supernet training via gradient based methods. The best architecture is sampled from the distribution after optimization. There are two issues in this formulation. First, the weights in the supernet are deeply coupled. It is unclear why inherited weights for a specific architecture are still effective. Second, joint optimization introduces further coupling between the architecture parameters and supernet weights. The greedy nature of the gradient based methods inevitably introduces bias during optimization and could easily mislead the architecture search. They adopted complex optimization techniques to alleviate the problem.
The one-shot paradigm (Brock al., 2017: Bender et al., 2018) alleviates the second issue. defines the supernet and performs weight inheritance in similar way. However, there is architecture relaxation. The architecture search problem is decoupled from the supernet training and addressed in separate step. Thus, it is sequential. It combines the merits of both nested and joint optimization approaches above. The architecture search is both efficient and flexible.
The first issue is still problematic. Existing one-shot approaches (Brock et al., 2017; Bender et al., 2018) still have coupled weights in the supernet. Their optimization is complicated and involves sensitive hyper parameters. They have not shown competitive results large datasets.

---

Under review as a conference paper at ICLR 2020
This work revisits the one-shot paradigm and presents new approach that further eases the train- ing and enhances architecture search. Based on the observation that the accuracy of an architecture using inherited weights should be predictive for the accuracy using optimized weights, propose that the supernet training should be stochastic. All architectures have their weights optimized si- multaneously. This gives rise uniform sampling strategy. To reduce the weight coupling in the supernet, a simple search space that consists of single path architectures is proposed. The training hyperparameter-fiee and easy converge.
This work makes the following contributions.
1. We present a principled analysis and point out drawbacks in existing NAS approaches that use nested and joint optimization. Consequently, we hope this work will renew interest in the one-shot paradigm, which combines the merits of both via sequential optimization.
2. We present a single path one-shot approach with uniform sampling. It overcomes the draw- backs of existing one-shot approaches. Its simplicity enables rich search space, including novel designs for channel size and bit width, all addressed in a unified manner. Architec- ture search is efficient and flexible. Evolutionary algorithm is used to support real world constraints easily, such as low latency.
Comprehensive ablation experiments and comparison to previous works on a large dataset (Ima- geNet) verify that the proposed approach is state-of-the-art in terms of aceuracy, memory consump- tion, training time, architecture search efficiency and flexibility.
2 REVIEW OF NAS APPROACHES
Without loss of generality, the architecture search space A is represented by directed acyclic graph (DAG). A network architecture a subgraph a E A, denoted as (a,, w) with weights w.
Neural architecture search aims to solve two related problems. The first is weight optimization, Wa = arg min Luain (N(a, ())
w
where Cuain (.) is the loss function on the training set.
The second is architecture optimization. It finds the architecture that is trained on the training set and has the best accuracy on the validation set, as
a arg Aax (N(a, wa)),
aEA
where (-) is the accuracy on the validation set.
Early NAS approaches perform the two optimization problems in a nested manner (Zoph & Le, 2016; Zoph et al., 2018; Zhong et al., 2018;;;; Baker al., 2016). Numerous architectures are sampled from AA and trained from scratch as in Eq. (1). Each training is expensive. Only small dataset (e.g.. CIFAR 10) and small search space (e.g, a single block) are affordable.
Recent NAS approaches adopt weight sharing strategy (Cai et al., 2018; Liu et al., 2018b; Wu et al., 2018a; Xie al., 2018; Bender al., 2018; Brock et al., 2017; Zhang et al., 2018c; Pham et al., 2018). The architecture search space A is encoded in a superner', denoted as WMA, W), where W the weights in the supernet. The supernet trained once. All architectures inherit their weights directly from W. Thus, they share the weights in their common graph nodes. Fine tuning of an architecture is performed in need, but no training from scratch is incurred. Therefore, architecture search is fast and suitable for large datasets like ImageNet.
Most weight sharing approaches convert the discrete architecture search space into a continuous one (Wu et al., 201 Sa; Cai et al., 2018; Liu et al., 2018b; Xie et al., 2018; Zhang et al., 201 2018c). Formally, space A is relaxed to A(0), where 0 denotes the continuous parameters that represent the distribution of the architectures in the space. Note that the new space subsumes the original one, A C A(0). An architecture sampled from A(0) could be invalid in A.
"Supernet" is used as general concept in this work. It has different names and implementation in previous approaches.
2

---

Under review as a conference paper at ICLR 2020
An advantage of the continuous search space is that gradient based methods (Liu al., 2018b; Cai et al., 2018; Wu et al., 2018a; Véniat & Denoyer, 2018; Xie et al., 2018; Zhang et al., 2018c) is feasible. Both weights and architecture distribution parameters are jointly optimized.
(0" Wo-) = arg min Leroin (A((0) W)).
(3)
8.W
After optimization, the best architecture a" is sampled from A(0*). ). Note that it could be invalid in A. so, it is validated (e.g., by binarization of (Liu et al., 2018b)). then inherits the weights from We. and is fine-tuned.
Optimization of Eq. (3) challenging. First, the weights of the graph nodes the supernet depend on each other and become deeply coupled during optimization. For specific architecture, it inherits certain node weights from W. While these weights are decoupled from the others, it is unclear why they still effective.
Second. joint optimization of architecture parameter 0 and weights W introduces further complex- ity. Solving Eq. (3) inevitably introduces bias certain areas in and certain nodes in W during the progress of optimization. The bias would leave some nodes in the graph well trained and others poorly trained. With different level of maturity in the weights, different architectures are actually non-comparable. However, their prediction accuracy is used as guidance for sampling in A(0) (e.g.. used as reward in policy gradient (Cai et al., 2018)). This would further mislead the architecture sampling. This problem is in analogy to the "dilemma of exploitation and exploration" problem in reinforcement learning. To alleviate such problems, existing approaches adopt complicated opti- mization techniques (see Table 7 for summary). Nevertheless, there lacks a comprehensive evalu- ation of their effectiveness (Li & Talwalkar, 2019).
Task constraints Real world tasks usually have additional requirements network's memory consumption, FLOPs, latency, energy consumption, etc. These requirements only depends on the architecture a, not the weights Wa- Thus, they called architecture constraints in this work. A typical constraint is that the network's latency is no more than a preset budget, as
Latency(a") < Latmax-
Note that it is challenging to satisfy Eq. (2) and Eq. (4) simultaneously for most previous approaches. Some works augment loss function Lerain in Eq. (3) with seft loss terms that consider the architecture latency (Cai et al., 2018; Wu et al., 201 8a; Xie et al., 2018; Véniat & Denoyer, 2018) 8). However, is hard, not impossible, to guarantee a hard constraint like Eq. (4).
OUR SINGLE PATH ONE-SHOT APPROACH
As analyzed above, the coupling between architecture parameters and weights is problematic. This is caused by joint optimization of both. To alleviate the problem, natural solution is to decouple the super net training and architecture search in two sequential steps. This leads to the so called one-shot approaches (Brock et al., 2017; Bender et al., 2018).
In general, the two steps are formulated as follows. Firstly, the supernet weight is optimized as
WA arg min Cuain W)).
w
Compared to Eq. (3), the continuous parameterization of search space is absent. Only weights are optimized. Secondly, architecture searched is performed as
arg maxACC (N(a, Wa(a))) aEA
(6)
During search, each sampled architecture inherits its weights from WA Wa(a). The key dif- ference of Eq. (6) from Eq. (1) and (2) is that architecture weights are ready for use. Evaluation of ACC..r(.) only requires inference. Thus, the search is very efficient.
The search is also flexible. Any adequate search algorithm is feasible. The architecture constraint like Eq. (4) can be exactly satisfied. Search can be repeated many times the same supernet once
3

---

Under review as a conference paper at ICLR 2020
drop rate 0.01
60.055
Validation top 1accrr 00
60.09
stop rate 0.05
srop rate 0.1
Validation top.1 accuracy
69.9%
40.0%
68.4%
68.2%
10.0%
28.0%
Exolutton
20000
60000
90000
120000
150000
12
16
20
Training
Evolution
Figure 1: Comparison of single path strategy and Figure Evolutionary vs. random architecture search.
drop path strategy
trained, using different constraints (e.g., 100ms latency and 200ms latency). These properties absent in previous approaches. These make the one-shot paradigm attractive for real world tasks.
One problem in Sec. 2 still remains. The graph nodes' weights in the supernet training in Eq.( 5) are coupled. is unclear why the inherited weights (a(a) are still good for an arbitrary architecture a.
The recent one-shot approach (Bender al., 2018) attempts decouple the weights using "path dropout" strategy. During an SGD step in Eq. (5), each edge in the supernet graph is randomly dropped. The random chance is controlled via a dropout rate parameter. In this way, the adaptation of the node weights is reduced during training. Experiments in (Bender et al., 2018) indicate that the training very sensitive to the dropout rate parameter. This makes the supernet training hard. A carefully tuned heat-up strategy is used. In our implementation of this work, we also found that the validation accuracy is very sensitive the dropout rate parameter.
Single Path Supernet and Uniform Sampling. Let restart to think about the fundamental principle behind the idea of weight sharing. The key to the success of architecture search in Eq. (6) validation set using inherited weight W (a(a) (without
is that, the accuracy of any architecture extra fine tuning) is highly predictive for the accuracy that the weight W (a to approximate the optimal weight Wa in Eq. (1). The quality of the approximation depends on how well the training loss Lusin (N(a, (a())) is minimized. This gives rise to the principle that the supernet weights W should be optimized in way that all architectures in the search space are optimized simultaneously. This is expressed as
that is fully trained, Ideally, this requires
WA arg min Em-I(A) [CuainNVaa, W(a))).
where I(44) is a prior distribution of a E A. Note that Eq. (7) is implementation of Eq. (5). In each step of optimization, an architecture a is randomly sampled. Only weights (()) are activated and updated. So the memory usage is efficient. In this sense, the supernet no longer valid net- work. It behaves as a stochastic superner (Véniat & Denoyer, 2018). This is different from (Bender al., 2018).
To reduce the co-adaptation between node weights, we propose supernet structure that each archi- tecture is single path, shown in Fig.3. Compared the path dropout strategy in (Bender al., 2018), the single path strategy is hyperparameter-free We compared the two strategies within the same search space (as in this work). Note that the original drop path in (Bender al., 2018) may drop all operations in block, resulting in short cut of identity connection. In our implementation, is forced that one random path is kept in this case since our choice block does not have an identity branch. We randomly select sub network and evaluate its validation accuracy during the training stage. Results Fig. show that drop rate parameters matters a lot. Our single path strategy corre- sponds using drop rate 1. It works the best, which also verifies the benefits of weight decoupling by our single path strategy.
The prior distribution I(A) is important. In this work, empirically find that uniform sampling good. This is not much of surprise. recent work also finds that purely random search is competitive several SOTA NAS approaches(( Li & Talwalkar (2019)). We also experimented with
4

---

Under review as a conference paper at ICLR 2020
variant that samples the architectures uniformly according to their constraints, named uniform constraint sampling. Specifically, we randomly choose a range, and then sample the architecture repeatedly until the FLOPs of sampled architecture falls in the range. This is because real task usually expects to find multiple architectures satisfying different constraints. In this work, we find the uniform constraint sampling method is slightly better. So use it by default in this paper.
We note that sampling a path according to architecture distribution during optimization is already used in previous weight sharing approaches (Pham et al., 2018; Véniat & Denoyer, 2018; Wu al., 2018a; Cai et al., 2018; Xie et al., 2018; Zhang et al., 20; Yao et al., 2019; Dong & Yang, 2019; Stamoulis al., 2019). The difference is that, the distribution [(AA is a fixed priori during our training (Eq. (7)), while it is learnable and updated (Eq. (3)) in previous approaches (e.g. RL (Pham et al., 2018), policy gradient (Véniat Denoyer, 2018; Cai et al., 2018), Gumbel Softmax (Wu et al., 2018a; 8a; Xie et al., 2018), APG (Zhang et al., 2018c)). As analyzed in Sec. 2, the latter makes the supernet weights and architecture parameters highly correlated and optimization difficult.
Comprehensive experiments in Sec. 4 show that our approach achieves better results than the SOTA methods. Note that there is no such theoretical guarantee that using fixed prior distribution is inherently better than optimizing the distribution during training. Our better result likely indicates that the joint optimization in Eq. (3) is too difficult for the existing optimization techniques.
Supernet Architecture and Novel Choice Block Design. Choice blocks are used to build stochastic architecture. Fig. 3 illustrates an example case. A choice block consists of multiple architecture choices. For our single path supernet, each choice block only has one choice invoked at the same time. A path is obtained by sampling all the choice blocks.
The simplicity of our approach enables to define different types of choice blocks to search various architecture variables. Specifically, we propose two novel choice blocks to support complex search spaces.
Quantizer
Feuturn litWid
Choiko llok
Convolutional Layer
Quantiner
Kemel Weights
Chotes Stock
Whng Br
Crotes Bloc
escirau carres
Figure 3: Choice blocks for our Figure 4: Choice block for Figure 5: Choice block for single path supernet.
channel number search. mixed-precision quantization search.
Channel Number Search. We propose new choice block based on weight sharing, as shown in Fig. 4. The main idea is to preallocate a weight tensor with maximum number of channels, and the system randomly selects the channel number and slices out the corresponding subtensor for convolution. With the weight sharing strategy, we found that the supernet can converge quickly.
In detail, assume the dimensions of preallocated weights are (max-c_out, max ksize). For each batch in supernet training, the number of current output channels c. cut is randomly sampled. Then, we slice out the weights for current batch with the form Weights|: c_out, c_in, :]. which is used to produce the output. The optimal number of channels is determined in the search step.
Preeesson Quantization Search. In this work. We design a novel choice block to search the bit widths of the weights and feature maps, shown in Fig. 5. We also combine the channel search space discussed earlier to our mixed-precision quantization search space. During supernet training. for each choice block feature bit width and weight bit width are randomly sampled. They are determined in the evolutionary step. See Sec. 4 and Fig. 5 for details.
Evolutionary Architecture Search. For architecture search in Eq. (6), previous one-shot works (Brock et al., 2017; Bender et al., 2018) use random search. This is not effective for a large search space. This work uses evolutionary algorithm. Note that evolutionary search in NAS is used in (Real et al., 2018), but it is costly as each architecture is trained from scratch. In our search, each architecture only performs inference. This is very efficient.
5

---

Under review as a conference paper at ICLR 2020
The algorithm is elaborated in Algorithm 1. For all experiments, population size P 50, max iterations T = 20 and = 10. For crossover, two randomly selected candidates are crossed to produce one. For mutation, randomly selected candidate mutates its every choice block with probability 0.1 to produce a new candidate. Crossover and mutation are repeated to generate enough new candidates that meet the given architecture constraints. Before the inference of an architecture, the statistics of all the Batch Normalization (BN) (loffe & Szegedy, 2015) operations are recalculated on random subset of training data (20000 images on ImageNet). It takes few seconds. This is because the BN statistics from the supernet are usually not applicable to the candidate nets. This is also referred in (Bender et al., 2018).
Fig. 2 plots the validation accuracy over generations, using both evolutionary and random search methods. It is clear that evolutionary search more effective. Experiment details are in Sec. 4.
The evolutionary algorithm is flexible in dealing with different constraints in Eq. (4), because the mutation and crossover processes can be directly controlled to generate proper candidates to satisfy the constraints. Previous RL-based (Tan et al., 2018) and gradient-based (Cai et al., 2018; Wu et al., 2018a; Véniat & Denoyer, 2018) methods design tricky rewards or loss functions to deal with such constraints. For example, (Wu et al., 2018a) 8a) uses a loss function CE(a, wa) a (og(LA((()) to balance the accuracy and the latency. It is hard to tune the hyper parameter 3 to satisfy a hard constraint like Eq. (4).
Summary. The combination of single path supernet. uniform sampling training strategy, evolu- tionary architecture search, and rich search space design makes our approach simple, efficient and flexible Table in Appendix performs a comprehensive comparison of our approach against previ- ous weight sharing approaches on various aspects. Ours is the easiest train, occupies the smallest memory, best satisfies the architecture (latency) constraint, and easily supports large datasets. Ex- tensive results in Sec. 4 verify that our approach is the state-of-theeart.
4 EXPERIMENT RESULTS
Dataset. All experiments are performed on ImageNer (Russakovsky et al., 2015). We randomly split the original training set into two parts: 50000 images for validation (50 images for each class exactly) and the rest as the training set. The original validation set used for testing, on which all the evaluation results are reported, following (Cai et al., 2018).
Training. For the training of the supernet and retraining of the best architecture (after evolutionary search) from scratch, schedule, etc.) as (Ma et al., 2018). The batch size is 1024. Supernet is trained for 120 epochs (150000 iterations) and the best architecture for 240 epochs (300000 iterations). Training uses 8 NVIDIA GTX 10801i GPUs.
the same settings (including data augmentation strategy, learning rate
Search Space: Building Blocks. First, we evaluate our method the task of building block selection, i.e. to find the optimal combination of building blocks under a certain complexity con- straint. Our basic building block design is inspired by state-of-the-art manually-designed network ShhfffeNett v2 (Ma et al., 2018). Table I shows the overall architecture of the supernet. There are 20 choice blocks in total. Each choice block has 4 candidates. namely "choice_3" "choice_5" "choice_ 7" and "choice_s x" respectively (see Fig. 6 in Appendix for details). They differ in kernel sizes and the number of depthwise convolutions. The size of the search space is 420
We use FLOPs < 330M as the complexity constraint, as the FLOPs of plenty of previous networks lies in [300,330], including manually-desie designed networks (Howard al., 2017; Sandler et al., 2018; Zhang et al., 201 2018b; Ma et al., 2018) and those obtained in NAS (Cai et al., 2018; Wu et al., 201 8a; Tan et al., 2018).
Table 2 shows the results. For comparison, we set up series of baselines as follows: 1) select certain block choice only (denoted by "all choice_ * entries); note that different choices have differ- ent FLOPs, thus we adjust the channels to meet the constraint. 2) Randomly select some candidates from the search space. 3) Replace our evolutionary architecture optimization with random search used in (Brock et al., 2017; Bender et al., 2018). Results show that random search equipped with our single path supernet finds an architecture only slightly better that random select (73.8 vs. 73.7). does mean that single path supernet is less effective. This is because the random search is too

---

Under review as a conference paper at ICLR 2020
input shape
block
channels
repeat
stride
model
HLOPs
top-1 1 acc( (b)
224" x
3 3 conv
16
2
all chnice 3
324M
73.4
112 16
CB
64
4
2
all choice_
321M
73.5
56 x 61
CB
160
4
2
all choice 7
327M
73.6
28 x 160
CB
320
all choice_x
326M
73.5
14a
CB
640
7 640
x
1024
1
random select (5 times)
~020
~73.7
7 x 1024
GAP
SPS random search
323M
73.8
1024
fc
1000
ours (fully--quipped)
319M
74.3
Table 1: Supernet architecture. CB -chice Table 2: Results of building block search. SPS block. GAP global average pooling. The -single path supernet. "stride" column represents the stride of the first block in each repeated group.
naive to pick good candidates from the large search space. Using evolutionary search, our approach finds out an architecture that achieves superior accuracy (74.3) over all the baselines.
Search Space: Channels. Based on our novel choice block for channel number search, we first evaluate channel search on the baseline structure "all choice_3" (refer to Table 2): for each building block, we search the number of "mdd-channels" (output channels of the first 1xl conv in each build- ing block) varying from 0.2x to 1.6x (with stride 0.2), where "k-x" means & times the number of default channels. Same as Sec. 4, we set the complexity constraint FLOPs < 330M Table (first part) shows the result. Our channel search method has higher accuracy (73.9) than the baselines.
To further boost the accuracy, we search building blocks and channels jointly. There are two alter- natives: 1) rumning channel search on the best building block search result of Sec. 4; 2) searching on the combined search space directly. In our experiments, we find the results of the first pipeline is slightly better. As shown in Table 3, searching in the joint space achieves the best accuracy (74.7 % acc.), surpassing all the previous tftte-of-the-aae manually-designed (Ma et al., 2018; Sandler et al., models (Tan al., 2018: Zoph et al., 2018: Liu al., 2018a;b; Cai
2018) or cotoomaticall-ssarcchi et al., 2018; Wu et al., 201 8a) under the complexity of ~ 300M FLOPs.
Comparison with State-of-the-arts in Table 3 shows our method is superior. Nev- ertheless, the comparisons could be unfair be- cause different search spaces and training meth- ods are used in previous works (Cai et al., 2018). To make direct comparisons, we bench- mark our approach to the same search space of (Cai et al., 2018; Wu et al., 201 20) In addition, we retrain the searched models reported in (Cai al., 2018: Wu et al., 201 8a) under the same settings to guarantee the fair comparison.
Results
Model
PLOPs/ PLOPP/Praass
Top.
all choice
324 IM
73,4
rand sel. channels (5 times)
323M/3.2M
73.1
choice channel search
329M/1 AM
rand sol. blocks + channels block seurch
~- 325M/3 2M
73.4
319M3 3M
74.3
search channel search
328M/3 4M
MobaleNet (0.75g)
et (2017)
325M016M
MobileNet V2 (1 (1.0g) Sandler et al. (2018)
300M/3 AM
72.0
ShuffleNet V2 (1.5x) Ma et al. (2018)
299M/3 SM
72.6
NASNET-A Zoph et al. (2018)
56-MU5 3M
74.0
PNASNET Liu al. 82)
588MS IM
74,2
MinsNerr Tan (2018)
317M4 2M
DARTS Liu et al. (2018b)
595M4.7M 7.
73.1
Proxyless-R (mabile)* Cai (2018)
320M/
(24.6)
The search space and supernet architecture in ProxylessNAS (Cai et al., 2018) is inspired by MobileNet v2 (Sandler et al., 2018) and Mnas- Ner (Tan et al., 2018). It contains 21 choice blocks; each choice block has choices dif- ferent building blocks and one skip layer). The size of the search space 721, FBNei (Wu et al., 201 8a) also uses a similar search space.
FBN
(2018a)
295M44.5M
(441)
Table 3: Results of channel search. * Perfor- mances reported in the form "xx (y)", where "x" means the accuracy retrained by us and "y" means accuracy reported by the original paper.
Table reports the accuracy and complexities (FLOPs and latency on our device) of 5 models searched by (Cai et al., 2018; Wu et al., 2018a), as the baselines. Then, for each baseline. search method runs under the constraints same FLOPs or same latency, respectively. Results shows that for all the cases baselines. We also point out that since the target devices in (Cai et al., 2018; Wu et al., 201 2a) are different from ours, the reported results may be sub-optimal on platform.
method achieves comparable higher accuracy than the counterpart
Furthermore, worth noting that all our 10 architectures Table 4 are searched on the same supernet, justifying the flexibility and efficiency of our approach to deal with different complexity

---

Under review as a conference paper at ICLR 2020
constraints: supernet is trained once and searched multiple times. In contrast, previous methods (Wu et al., 8a; Cai et al., 2018) have to train multiple supernets under various constraints. According to Table 6, searching is much cheaper than supernet training.
baseline network
FLOPs
lateney
top- I acc(
top- acc
top-1 aoc( ours (same latency)
baseline
ours (same FLOPs)
FBNeE-A (Wu et al., 2018a)
249M14.33
13ms
(73.0)
73.2
73.3
FBNet-B (Wu et al., 2018a)
295M/4.5M
17ms
74.1 (74.1)
74.2
74.8
FBNet-C (Wu et al., 2018a)
375M/5.5M
19ms
74,9 (74.9)
75.0
75.1
Proxyless-R (mobile) (Cai el al., 2018)
320M/4.0M
17ms
74.2 (74.6)
74.5
74.8
Proxyless (GPU) (Cai et al., 2018)
465M/5.3M
22ms
74.7 (75.1)
74.8
75.3
Table 4: Compared with state-of-the-art NAS methods (Wu et al., 2018a: 8a; Cai et al., 2018) using the same search space. The latency evaluated on single NVIDIA Titan XP GPU, with batchsize 32. Accuracy numbers in the brackets are reported by the original papers: others are trained by us. All our architectures are searched from the same supernet via evolutionary architecture optimization.
Application: Mixed-Precision Quantization. We evaluate our method on R8sNet-18 and ResNer-34 as common practice in previous quantization works (e.g. (Choi et al., 2018; Wu et al., 201 8b; Liu et al., 2018c; Zhou et al., 2016; Zhang et al., 2018a)). Following (Zhou et al., 2016; Choi et al., 2018; Wu et 201 we only search and quantize the res-blocks, ex- cluding the first convolutional layer and the last fully-connected layer. In the search space, choices of weight and feature bit widths include ((1,2), (2,2) 2), (1, 4), (2, 4). (3, 4), (4) As for channel search, "bottleneck channels" (i.e. the output channels of the first convolutional layer in each residual block) in (0.5z, 1.0x, 1.5x). where "k-x" means k times the number of original channels. The size of the search space is (3 x 6)* 18 where N is the number of choice blocks (N = 8 for ResNet- 18 and N 16 for ResNet-34). Note that for each building block we use the same bit widths for the two convolutions. We use PACT (Choi et al., 2018) as the quantization algorithm.
Method
BitOPs
lopl-acc(15
Methad
BitoPs
topl acce
ResNet- N Rodls
70.9
ReaNet. 34
Foat
75.0
2W2A
6.32G
65.6
2W2A
13.21G
70.8
curs
6.21G
66.4
ours
13.11G
3WIA
1421G
68.3
3W3A
29.720
72.5
DNAS
15.62G
68.7
DNAS
73.2
ours
13,496
69.4
ours
28.78G
73.9
4W4A
25.27G
69.3
4W4A
52,83G
73.5
DNAS
25.70G
70.6
DNAS
57.31G
74.0
ours
24.31G
70.5
ours
51.92G
74.6
Table 5: Results of mixed-precisiin quantization search. "kWkA" means k-bit quantization for all the weights and activations. DNAS (Wu et al., 2018b).
search the number of
Table 5 reports the results. The baselines are denoted as &WRA (A: = 2, 4), which means uniform quantization of weights and activations with bits.. Then, our search method runs under the con- straints of the corresponding BitOps. We also compare with recent mixed-precision quantization search approach (Wu et al., 2018b). Results shows that our method achieves superior accuracy in most cases. Also note that all our results for ResNet-18 and 34 are searched on the same supernet. This is very efficient.
Search Cost Analysis. The search cost is
matter of concern NAS methods. So we an- alyze the search cost of our method and previ- ous methods (Wu et al., 2018a; Cai et al., 2018) (reimplemented by us). We use the search space of our building blocks to measure the memory cost of training supernet and overall time cost. All the supernets are trained for 150000 iter- ations with a batch size of 256. All models are trained with 8 GPUs. The Table 6 shows that our approach clearly uses less memory than other two methods because of the single path supernet. And our approach is much more efficient overall although we have an extra search step that costs less than GPU day. Note Table 6 only com- pares single run. In practice, our approach is more advantageous and more convenient multiple searches are needed. As summarized in Table 7, it guarantees to find out the architecture satisfying constraints within one search. Repeated search is easily supported.
Method
Proxyless
FBNet
Ours
GPU memory cost (8 GPUs in total)
37G
63G
24G
Training time
15 Gds
20 Gds
Gds
Search time
0
0
<I Gds
Retrain time
16 Gds
16 Gds
16 Gds
Total time
Gds
Gds
Gds
Table 6: Search Cost. Gds GPU days
when
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architec- tures using reinforcement learning. arXiv preprint 1.02/16 2016.
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Understand- ing and simplifying one-shot architecture search. International Conference on Machine Learn- ing. pp. 549-558. 2018.
Andrew Brock, Theodore Lim, James Ritchie, and Nick Weston. Smash: one-shot model archi- tecture search through hypernetworks. arXiv preprint / 2017.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.00332. 2018.
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural 2018.
networks. arXiv preprint
Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four gpu hours. In Pro- ceedings of the IEEE Conference Computer Vision and Patlern Recognition, pp. 1761-1770, 2019.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 170.000704. 2017.
Sergey loffe and Christian Szegedy. Batch normalization:
Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 1.X2::/502.0366.. 2015.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search.
arXiv preprint 110.::100.0766000 2019.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceed- ings of the European Conference on Computer Vision (ECCV), pp. 19-34, 2a.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.0000 2018b.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1 1-bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European Conference on Computer Vision (ECCV), pp- 722-737. 2018c.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 116-131, 2018
Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:/802.03268 2018.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. arXiv preprint arXiv: 802.01548, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-553 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018

---

Under review as a conference paper at ICLR 2020
Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Single-path nas: Designing hardware- convnets in less than hours. arXiv preprint
2019.
Mingxing Tan, Bo Chen, Ruoming Pang. Vijay Vasudevan, and Quoc V Le. Mnasnet: Platform- aware neural architecture search for mobile. arXiv preprini urXiv::807) 1626, 2018.
Tom Véniat and Ludovic Denoyer. Learning time/memory-efffciinn deep architectures with bud- geted super networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3492-3500, 2018.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. arXiv preprint arXiv:1812.03444 2018a.
Bichen Wu, Yanghan Wang. Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Mixed precision quantization of convnets via differentiable neural architecture search. arXiv preprint arXiv:1812.00009,, 2018b.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. arXiv preprint arXiv:1812.09922 2018.
Quanming Yao, Ju Xu, Wei-Wei Tu, and Zhanxing Zhu. Differentiable neural architecture search via proximal iterations. arXiv preprint 13X1v:1905.11577, 2019.
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 365-382, 2018a.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6848-6856, 2018b.
Xinbang Zhang, Zehao Huang, and Naiyan Wang. You only search once: Single shot neural archi- tecture search via direct sparse optimization. arXiv preprint
2018c.
Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Practical block-wise neural network architecture generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2423-2432 2018a.
Zhao Zhong, Zichen Yang. Boyang Deng, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Blockqnn Efficient block-wise neural network architecture generation. arXiv preprint arXiv: /808. 05584, 2018b.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 1606.06606.66/60 2016.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arX05:161..01578, 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
10

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
Algorithm 1: Evolutionary Architecture Search
Input:
supernet weights W. population size architecture constraints iteration T, validation dataset Dual
Output: the architecture with highest validation accuracy under architecture constraints Topk := =
Po := ynitialize.population.
P/2:
Crossover number
P/2:
Mutation number
prob 0.1;
Mutation probability
for =1:7 do
ACC;-1 == Inf ereeeeee (errenceeWWrrr Doat, Pi-
Topk Udaale I opk (Topk, ACC,,
10
Prossover Crossover Topk, n)C); mmttattin == Mutation (nopa,on(Toop, m, prob, C);
11
12
P. = Perossover U mmuaaiion
13
end
14
Return the architecture with highest accuracy in Topk;
Approach
Supemet optimization
Architecture search
Hyper parameiers
superact Training
Memory consumption
supernet training
How
ImagoNet
ENAS (Pham al,, 2018)
Altemative Rl. and time tuning
Ene tuning
Short-ime
Single patti
RL system
None
No
(Véniat Denoyer, 2018)
Stochastic super
policy gradient
cost penally
Woight
Single path
Not guaranteed
traiming.
DARTS (Lis et 20186)
Gradient- hased. dropout
Weight suxiliary loss
Pith dropout rate.
Whole supermet
None
Transfer
Proxyless (Cai et al., 2018)
Stochastic relecation of the discrete search + policy gradicnt
Scaling factor of latency loss
Two puths
Soft constraint in maining. Not guaranteed.
Yes
FBNet (Wa al., 2018a)
Stochastic relecation the discrrti search to differeniable
Temperature parameter in Gumhee softmax. Coefficient in constraint loss
Whole supernet
Soft constraint in mainingg Not guaranteed.
Yes
optimication Gumbel sofimax.
SNAS (Xie et al., 2018)
Same as FBNet
Same FRNes
Whole supernet
Soll constraint in training.
Nut guaranieed.
Transter
SMASH
Hypernetwook
Random
Hypeneiesingla Faib
None
One ShotShot [Bender al., 2018) Path drupout Unifarm pach
Random
Drop rale
W Whole supernet
Not investigaiod
Yes
Guaranteed in searching.
Ours
sampling
Evolution
None
Single path
Support multiple constraines,
Yes
Table Overview and comparison SOTA weight sharing approaches. Ours the easiest train, occupies the smallest memory, best satisfy the architecture (latency) constraint, and easily supports the large dataset. Note that those approaches belonging to the joint optimization category (Eq. (3)) have "Supernet optimization" and "Architecture search" columns merged.

---

Under review as a conference paper at ICLR 2020
Channels Split
WxaDW
Channels Split
Channels Spåit
Channels Split
Ix]Cony
Wonny
Ix]Conv
IxIConv
CxnCo
3xDDWennv
5xnDWeonv
Wx7DWeomv
3x3DWoan
ExlConv
IxTConv
IxICone
Lxx
Concat
Concal
Concat
Concal
(a) Choice blocks with stride=1
WxnDWcoou (snide=2)
CRlCoov
5x5D comy (sande=2)
IxICony
DW (stride=2)
Ix|Conv
3e3DW
Woony (stride=2)
5x5DWeone (strid==2)
Te7DWeony (stride=2)
IxICoss
1x3DWooew (stide-2)
3xDWeony
IxlConv
IxlCoa
IxlConv
Ix]Conv
IxTConv
IxlC
IxICony
Concat
Conent
Cuncal
3x3DWenny
Channels Shaffle
Channels Shuffle
Channel Shuffi
Ix]Conv
Concat
(b) Choice blocks with stride
Shutfie
Figure 6: Choice blocks used in Sec. 4. From left to right Choice_3, Choice Choice_7, Choice
(a) Block search result
Choico
Choice
1.66%
10
8.8%
(b) Channel search result
6%
LD
1.2%
L.0%
0.8%
ec) Block search channel search result
Figure 7: Structures of searched architectures in Sec. 4. (a) Result of building block search. (b) Result of channel search on all_choice.3 structure. (c) Result of channel search on best building block search structure.
12

---

Under review as a conference paper at ICLR 2020
hins hC
Sint
Mile
NBB 717
(a) Searched result with same FLOPs of FBNet-A
KIBE TC
LIBE
MBS
MILU
(b) Searched result with same FLOPs of FBNet-B
Yaa
VSS Sa
Milte
Mile
Mile
MBS MY
MILE
MBte
(c) Searched result with same FLOPs of FBNettC
13
MBS NT
MBs
(d) Searched result with same FLOPs of Proxyless-Rmoobiee
T
MBB P47
HIR 3:
NIRE 38
hills 3.
hith Te
hith Tr
hibl
(e) Searched result with same FLOPs of Proxyless (GPU)
Figure 8: Structures of searched architectures under FLOPs constraints by using ProxylesSNAS search space, see Table 4 for details.
13

---

Under review as a conference paper at ICLR 2020
1000 hr
VEB8 hd
342
Mittce Te
MB
(a) Searched result with same latency of FBNet-A
hill32
MIBE
hilli MIN
NBB
(b) Searched result with same latency of FBNet-B and Proxyless-R(mobine R(mobile)
HIRE 1E
HBB M
MIL
540
(c) Searched result with same latency FBNet-C
MIIT7
Mine TC
MBLSv
Mite 3:
(d) Searched result with same latency of Proxyless(GPU)
Figure 9: Structures of searched architectures under GPU latency constraints by using PrexylessNAS search space, Table 4 for details.
14

---

Under review as a conference paper at ICLR 2020
1.5x
4W4A
(a) Searched result with 6.21 GBitOS
3W4A
(b) Searched result with 13.49 GBitOPs
1.5x
1.5x
1.5x
2W4A
(c) Searched result with 24.31 GBitOPs
1.5x
1.0%
1.0%
1.5x
1.0x
1.5x
1.5x
1.5x
1.5x
1.5x
1W4A
(d) Searched result with 13.11 GBitOPs
1.5x
1.5x
1.5x
L.N%
1.0%
1.0x
1.0%
1.5%
1.5x
1.5%
2W2A
(e) Searched result with 28.78 GBitOPs
IW2A
1.0%
1.5x
1.0%
1.0%
1.5x
1.5%
1.0%
1.5x
1.5x
1.5x
1.5x
1.5x
1.5x
(f) Searched result with 51.92 GBitOPs
Figure 10: Searched architectures of joint searching channel size and bit width under BitOPs con- straints, see Table 5 for details. (a) (c) are searched based on Resnet18, (d) (f) are searched based on Resnet34.
15


Reviews:
[{"review_id": "Sygdboybqr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors revise the one-shot NAS algorithm in this work. One-shot NAS that employs a supernet to share the weights between subnets is an efficient NAS algorithm. Authors develop a new training paradigm to train the supernet sufficiently. Specifically, they uniformly sample a single path from supernet at each iteration to make the training effective and stable.\n\nPros:\n1.\tImprove the one-shot NAS by uniform path sampling.\n2.\tExperiments demonstrate effectiveness.\n\nCons:\n1.\tAuthors only report the performance of the best architecture after fine tuning. It is interesting to see the performance of subnets after obtaining the supernet. Is the better subnet in supernet still better than others after fine tuning?\n2.\tGiven the large number of single paths, it is hard to train each one sufficiently within a supernet. Authors may demonstrate how one-shot NAS can address the problem.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Sklsio4str", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new one-shot NAS approach. Parameter updating and structure updating are optimized separately. In the process of parameter optimization, different from the previous methods, the network samples the structure according to the uniform probability distribution. This sampling method can avoid the coupling caused by optimizing the structural parameters at the same time. After training the supernet, the network uses the genetic algorithm to search the structure.\n\nPros:\n+ This paper is well-written and easy to follow. \n+ The method proposed in this paper is very efficient in the search stage and saves memory. During the searching stage, the constraints of the model can be restricted by the genetic algorithm. \n+ This method could directly search architectures on the large scale dataset.\n+ The results are promising. Experiments are performed on various structures and including quantization layers. \n\nStill, I have several minor concerns regarding the algorithm and experiments.\n\n1. Why does the author think that the supernet needs to be trained by single path all the time? In the architecture searching space, under the same computational constraints, some models perform well, and some do not. Why do the good models and the worse models use the same probability to sample? Wouldn't it require much more time to optimize the supernet? In the experiment part, the author compares a related work FBNet, which uses the structure parameter \\theta and temperature parameter \\tau to control the sampling. When the temperature \\tau is high, the distribution of the samples degenerates into the single path sampler, and the accuracy is used to optimize the parameter \\theta, which makes \\theta tend to select the well-performed models. With the decrease of the temperature \\tau, the probability of sampling well-performed architectures is higher than the worse-performed models. Therefore, what is the advantage of the single path sampling method over \\tau controlled sampler? It seems that gradually pruning the worse-performed search space would accelerate searching time. More analysis is preferred rather than the superior results from the experiments.\n\n2. Is identity in the search space? For at the last of Page4, the author said 'our choice block does not have an identity branch', and the listed architecture on page 13 has the identity.\n\n3. Is the supernet training for 120 epochs necessary? Is the rank of network structures become steady when training for 120 epochs. Because for the subsequent EA-based network structure search, the loss scale is not important, but the sorting (ranking) is important. (This beyond the scope of this paper, but relates to the NAS area)\n\n4. What is the number of parameters for all the listed models in Tab. 3 & Tab. 4? Listing the parameters makes it easier for the followers.\n\n5. What kind of structure do the mixed-precision networks use? The original resnet module, or the bireal resnet module [r1]?\n\n6. While recalculating the statistics of all the BN,  is backpropagation required or just run the inference without gradient backpropagation.\n\n7. The author uses the approximate complete set of Imagenet to train supernet. If we directly inherit the parameter from the supernet, can we accelerate the training of searched good structure? Will Top-1 acc lower than training from scratch?\n\n[r1] Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "S1e5ibyY5B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*UPDATE* I have read the other reviews and authors' responses. All the reviewers agree that improving single-shot NAS is an important problem, and that sampling single-paths can be a plausible approach for it that avoids weight coupling. Consequently, I have updated my rating to weak accept. I think the paper can be substantially stronger though.\nThe key claim that set this paper apart from other single-path NAS approaches is that they use a fixed distribution (in particular, the uniform distribution) to sample from unlike others like FBnet who use a trainable distribution. They argue that uniform is parameter-free whereas trainable distributions introduce additional parameters that need to be trained. Their findings motivate a natural follow-up question: could we use a different fixed distribution? Reviewer2 has a similar question in their review. Perhaps a distribution that is weighted according to (some proxy of) how much computational resource the networks take to train? Could such a distribution also be parameter-free and give good benefit over uniform distribution without needing to be updated during supernet training? Such an analysis of the prior distribution will make the paper even stronger.\n\n\nThe paper studies a sequential optimization approach to neural architecture search that can provide some benefit over nested or joint approaches. The core challenge in sequential approaches (which first train the weights of a supernetwork; then search through possible architectures which inherit appropriate weights from the supernetwork) is that the weights for a giant network may not be optimal for the weights of a sub-network encountered during subsequent architecture search. The core benefit of such an approach compared to nested approaches is that the subsequent search phase only needs to perform network inference with inherited weights; not train a sub-network from scratch.\nThe primary contribution is to fix a prior distribution over architectures and sample from them when training the supernetwork. This simple fix helps the weights of the giant network be more useful when inherited into any sub-networks during architecture search.\nThe paper will be substantially stronger with a careful study of the choice of the prior distribution and how it affects (a) the rejection sampling step needed to ensure the sampled architectures satisfy complexity constraints, and (b) the performance of the eventual neural architecture search procedure.\nAnother experiment that will be valuable is to rigorously validate the hypothesis that reducing weight coupling in the supernetwork training is crucially linked to improving the downstream architecture search performance.\n\nMinor (writing comments):\nIntroduction: \"Complex optimization techniques are adopted.\" This statement is awkward. Does this paper specifically adopt more complex methods to address the shortcomings of gradient methods. Or, does the community broadly research more complex methods as a consequence (and you are advocating for a return to simpler gradient methods)?\nPg5: \"we randomly choice a range\" -> \"choose\"\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a simple NAS method based on sampling single paths of the one-shot model based on a uniform distribution. Next to the private discussion with reviewers, I read the paper in detail. 

During the discussion, first, the reviewer who gave a weak reject upgraded his/her score to a weak accept since all reviewers appreciated the importance of neural architecture search and that the authors' approach is plausible. 
Then, however, it surfaced that the main claim of novelty in the paper, namely the uniform sampling of paths with weight-sharing, is not novel: Li & Talwalkar already introduced a uniform random sampling of paths with weight-sharing in the one-shot model in their paper "Random Search and Reproducibility in NAS" (https://arxiv.org/abs/1902.07638), which was on arXiv since February 2019 and has been published at UAI 2019. This was their method "RandomNAS with weight sharing".

The authors actually cite that paper but do not mention RandomNAS with weight sharing. This may be because their paper also has been on arXiv since March 2019 (6 weeks after the one above), and was therefore likely parallel work. Nevertheless, now, 9 months later, the situation has changed, and the authors should at least point out in their paper that they were not the first to introduce RandomNAS with weight sharing during the search, but that they rather study the benefits of that previously-introduced method.

The only real novelty in terms of NAS methods that the authors provide is to use a genetic algorithm to select the architecture with the best one-shot model performance, rather than random search. This is a relatively minor contribution, discussed literally in a single paragraph in the paper (with missing details about the crossover operator used; please fill these in). Also, this step is very cheap, so one could potentially just run random search longer. Finally, the comparison presented may be unfair: evolution uses a population size of 50, and Figure 2 plots iterations. It is unclear whether each iteration for random search also evaluated 50 samples; if not, then evolution got 50x more samples than random search. The authors should fix this in a new version of the paper.

The paper also appears to make some wrong claims in Section 2. For example, the authors write that gradient-based NAS methods like DARTS inherit the one-shot weights and fine-tune the discretized architectures, but all methods I know of actually retrain from scratch rather than fine-tuning. Also, equation (3) is not what DARTS does; that does a bi-level optimization.
In Section 3, the authors say that their single-path strategy corresponds to a dropout rate of 1. I do not think that this is correct, since a dropout rate of 1 drops every connection (and does not leave one remaining). All of these issues should be rectified.

The paper reports good results on ImageNet. Unfortunately, these may well be due to using a better training pipeline than other works, rather than due to a better NAS method (no code is available, so there is no way to verify this). On the other hand, the application to mixed-precision quantization is novel and interesting.

AnonReviewer2 asked about the correlation of the one-shot performance and the final evaluation performance, and this question was not answered properly by the authors. This question is relevant, because this correlation has been shown to be very low in several works (e.g., Sciuto et al: "Evaluating the search phase of Neural Architecture Search" (https://arxiv.org/abs/1902.08142), on arXiv since February 2019 and a parallel ICLR submission). In those cases, the proposed approach would definitely not work.

The high scores the reviewers gave were based on the understanding that uniform sampling in the one-shot model was a novel contribution of this paper. Adjusting for that, the real score is much lower and right at the acceptance threshold. After a discussion with the PCs, due to limited capacity, the recommendation is to reject the current version. I encourage the authors to address the issues identified by the reviewers and in this meta-review and to submit to a future venue. 
Decision: Reject