Once-for-All: Train One Network and Specialize it for Efficient Deployment
Authors: Han Cai
Chuang Gan
Tianzhe Wang
Zhekai Zhang
Song Han
Venue:  (2020)
Keywords: Efficient Deep Learning
Specialized Neural Network Architecture
AutoML

Abstract: We address the challenging problem of efficient inference across many devices and resource constraints, especially on edge devices.  Conventional approaches either manually design or use neural architecture search (NAS) to find a specialized neural network and train it from scratch for each case, which is computationally prohibitive (causing $CO_2$ emission as much as 5 cars' lifetime) thus unscalable. In this work, we propose to train a once-for-all (OFA) network that supports diverse architectural settings by decoupling training and search, to reduce the cost. We can quickly get a specialized sub-network by selecting from the OFA network without additional training. To efficiently train OFA networks, we also propose a novel progressive shrinking algorithm, a generalized pruning method that reduces the model size across many more dimensions than pruning (depth, width, kernel size, and resolution). It can obtain a surprisingly large number of sub-networks ($> 10^{19}$) that can fit different hardware platforms and latency constraints while maintaining the same level of accuracy as training independently. On diverse edge devices, OFA consistently outperforms state-of-the-art (SOTA) NAS methods (up to 4.0% ImageNet top1 accuracy improvement over MobileNetV3, or same accuracy but 1.5x faster than MobileNetV3, 2.6x faster than EfficientNet w.r.t measured latency) while reducing many orders of magnitude GPU hours and $CO_2$ emission. In particular, OFA achieves a new SOTA 80.0% ImageNet top-1 accuracy under the mobile setting ($<$600M MACs). OFA is the winning solution for the 3rd Low Power Computer Vision Challenge (LPCVC), DSP classification track and the 4th LPCVC, both classification track and detection track. Code and  50 pre-trained models (for many devices & many latency constraints) are released at https://github.com/mit-han-lab/once-for-all. 

Published as a conference paper at ICLR 2020
ONCE-FOR-ALL. TRAIN ONE NETWORK AND SPE- CIALIZE IT FOR EFFICIENT DEPLOYMENT
Han Cai', Chuang Gan', Tianzhe Wang's , Zhekai Zhang', Song Han "aassachusetts Institute of Technology. "MIT-IBM Watson Al Lab (hancai, chuangg, songhan]emit...
ABSTRACT
We address the challenging problem of efficient inference across many devices and resource constraints, especially on edge devices. Conventional approaches either manually design or use neural architecture search (NAS) to find specialized neural network and train it from scratch for each case, which computationally prohibitive (causing CO2 emission as much as 5 cars' lifetime Strubell et al. (2019)) thus unscalable. In this work, we propose to train a once-for-all (OFA) network that supports diverse architectural settings by decoupling training and search, reduce the cost. We can quickly get a specialized sub-network by selecting from the OFA network without additional training. To efficiently train OFA networks, we also propose a novel progressive shrinking algorithm, a generalized pruning method that reduces the model size across many more dimensions than pruning (depth, width, kernel size, and resolution). It can obtain a surprisingly large number of sub- networks (> 10 that can fit different hardware platforms and latency constraints while maintaining the same level of accuracy as training independently. On diverse edge devices, OFA consistently outperforms state-of-the-are (SOTA) NAS methods (up to 4.0% ImageNet topl accuracy improvement over MobileNetV3, same accuracy but 1.5x faster than MobileNetV3, 2.6x faster than EfficientNet w.r.t measured latency) while reducing many orders of magnitude GPU hours and CO22 emission. In particular, OFA achieves a new SOTA 80.0% ImageNet top- accuracy under the mobile setting (<600M MACs). OFA is the winning solution for the 3rd Low Power Computer Vision Challenge (LPCVC), DSP classification track and the 4th LPCVC, both classification track and detection track. Code and 50 pre-trained models (for many devices & many latency constraints) are released at attps:/ thub. com/mii han Lab/ Lab/once- for-ai all.
INTRODUCTION
Deep Neural Networks (DNNs) deliver state-of-the-art accuracy in many machine learning applica- tions. However, the explosive growth in model size and computation cost gives rise to new challenges on how to efficiently deploy these deep learning models diverse hardware platforms, since they have to meet different hardware efficiency constraints (e.g., latency, energy). For instance, one mobile application on App Store has to support diverse range of hardware devices, from high-end Sam- sung Note 10 with a dedicated neural network accelerator to 5-year-old Samsung S6 with much slower processor. With different hardware resources (e.g., on-chip memory size, #arithmetic units), the optimal neural network architecture varies significantly. Even running on the same hardware, under different battery conditions or workloads, the best model architecture also differs a lot.
Given different hardware platforms and efficiency constraints (defined as deployment scenarios), researchers either design compact models specialized for mobile (Howard al., 2017; Sandler et al., 2018; Zhang et al., 2018) or accelerate the existing models by compression (Han et al., 2016; He et al., 2018) for efficient deployment. However, designing specialized DNNs for every scenario is engineer-expensive and computationally expensive, either with human-based methods NAS. Since such methods need to repeat the network design process and retrain the designed network from scratch for each case. Their total cost grows linearly the number of deployment scenarios increases, which will result in excessive energy consumption and CO2 emission (Strubell et al., 2019). It makes them unable handle the vast amount of hardware devices (23. 14 billion IoT devices till

---

Published as a conference paper at ICLR 2020
train once-for-all network
Previpus: O(N) design cost
OFA
MobileNerVS
Ours: O(1) design cost
76.1
Train Oniin
do
Design Cost
Top-1 ImageNes Aos (%)
75
Get Many
00
75.2
specialized sub-nets
16x-1300x reduction
73
Train Four Times,
71
70.0
70.4
Oet Four
0
20
40
80
80
direct deploy (no retrain]
Number Deployment Scenarios
67.4
67
Tiny HIRE (AloT)
12
15
18
21
24
Cloud AI
Mobile AI
Different Hardwaree
Constraint
Samsung Note10 Latency (ms)
Figure I: Left: single once-for-all network is trained to support versatile architectural configurations including depth, width, kernel size, and resolution. Given a deployment scenario, a specialized sub- network is directly selected from the once-for-all network without training. Middle: this approach reduces the cost of specialized deep learning deployment from O(N) to O(1). Right: once-for-all. network followed by model selection can derive many accuracy- trade-offs by training only once, compared to conventional methods that require repeated training.
2018') and highly dynamic deployment environments (different battery conditions, different latency requirements, etc.).
This paper introduces a new solution tackle this challenge designing a once-for-all network that can be directly deployed under diverse architectural configurations, amortizing the training cost. The inference is performed by selecting only part of the once-for-all network. It flexibly supports different depths, widths, kernel sizes, and resolutions without retraining, A simple example of Once-for-All (OFA) is illustrated in Figure I (left). Specifically, we decouple the model training stage and the neural architecture search stage. In the model training stage, we focus on improving the accuracy of all sub-networks that are derived by selecting different parts of the once-for for-all network. In the model specialization stage, we sample subset of sub-networks to train accuracy predictor and latency predictors. Given the target hardware and constraint, a predictor-gut guided architecture search (Liu et al., 2018 8) is conducted get a specialized sub-network, and the cost is negligible. As such, we reduce the total cost of specialized neural network design from O(N) to O(1) (Figure middle).
However. training the once-for-all network is a non-trivial task, since it requires joint optimization of the weights to maintain the accuracy of a large number of sub-networks (more than 10 in our experiments). It is computationally prohibitive to enumerate all sub-networks get the exact gradient in each update step, while randomly sampling a few sub- neb-networks in each step will lead to significant accuracy drops. The challenge that different sub-networks are interfering with each other, making the training process of the whole once-for-all network inefficient. To address this challenge, we propose a progressive shrinking algorithm for training the once-for-all network. Instead of directly optimizing the once-for-all network from scratch, we propose to first train the largest neural network with maximum depth, width, and kernel size, then progressively fine-tune the once-for-al all network to support smaller sub-networks that share weights with the larger ones. As such, it provides better initialization by selecting the most important weights of larger sub-networks, and the opportunity to distill smaller sub-networks, which greatly improves the training efficiency. From this perspective, progressive shrinking can be viewed as a generalized network pruning method that shrinks multiple dimensions (depth, width, kernel size, and resolution) of the full network rather than only the width dimension. Besides, it targets on maintaining the accuracy of all sub-networks rather than a single pruned network.
We extensively evaluated the effectiveness of OFA on ImageNet with many hardware platforms (CPU, GPU, mCPU, mGPU, FPGA accelerator) and efficiency constraints. Under all deployment scenarios, OFA consistently improves the ImageNet accuracy by significant margin compared to SOTA hardware-aware NAS methods while saving the GPU hours, dollars, and CO2 emission by orders of magnitude. On the ImageNet mobile setting (less than 600M MACs), OFA achieves new SOTA 80.0% topl accuracy with 595M MACs (Figure 2). To the best of our knowledge, this is the first time that the SOTA ImageNet topl accuracy reaches 80% under the mobile setting.
2

---

Published as a conference paper at ICLR 2020
14x reduccioo
80.0% Top-1
595M
MACs
Once-for-All (ours)
InceptionV3
Xoeption
EfficienNNet
NASNet-A
ResNetXI-50
ImageNes Top-1 accurayy (%%
DPN-92
MBNetV3
DenseNet-169
ResNetXt-
ProxylessNAS
DenseNet-264
AmosbaNet
DenseNet-19 121
75
MBNetV2
ResNet-101
PNASNet
InceptionV2
ResNet-50
ShuffleNet
DARTS
2M
4M
16M
32M
64M
IGCV3DD
Model Size
The higher the better
Ø
MobileNetV1 (MBNetV1)
Handcralted
AutoML
lower the better
2
5
6
MACs (Billion)
Figure 2: Comparison between OFA and state-of-the-arr CNN models on ImageNet. OFA provides 80.0% ImageNet topl accuracy under the mobile setting (< 600M MACs).
RELATED WORK
Efficient Deep Learning. Many efficient neural network architectures are proposed to improve the hardware efficiency, such as SqueezeNet (landola et al., 2016), MobileNets (Howard et al., 2017; Sandler et al., 2018), ShuffleNets (Ma et al., 2018; Zhang et al., 2018), etc. Orthogonal to architecting efficient neural networks, model compression (Han al., 2016) is another very effective technique for efficient deep learning, including network pruning that removes redundant units (Han et al., 2015) or redundant channels (He al., 2018; Liu et al., 2017), and quantization that reduces the bit width for the weights and activations (Han et al., 2016; Courbariaux et al., 2015; 5; Zhu et al., 2017).
Neural Architecture Search. Neural architecture search (NAS) focuses on automating the architec- Cai et al., 201 Liu et al., 2019). Early NAS methods (Zoph et al., 2018 8; Real et al., 2019; Cai et al., 201 search for high- accuracy architectures without taking hardware efficiency into consideration. Therefore, the produced architectures (e.g., NASNet, AmoebaNet) are not efficient for inference. Recent hardware-aware NAS methods (Cai et al., 2019; Tan et al., 2019; Wu et al., 2019) directly incorporate the hardware feedback into architecture search. Hardware-DNN co-design techniques (Jiang et al., 2019b;;; Hao et al., 2019) jointly optimize neural network architectures and hardware architectures. As a result, they improve inference efficiency. However, given new inference hardware platforms, these methods need to repeat the architecture search process and retrain the model, leading to prohibitive GPU hours, dollars, and CO2 emission. They are not scalable to a large number of deployment scenarios. The individually trained models do not share any weight, leading to large total model size and high downloading bandwidth.
ture design process (Zoph & Le, 2017; Zoph al., 2018; Real al., 2019;
Dynamic Neural Networks. To improve the efficiency of given neural network, some work explored skipping part of the model based on the input image. For example, Wu et al. (2018); Liu & Deng (2018); Wang et al. (2018) learn controller gating modules adaptively drop layers; Huang et al. (2018) introduce early-exit branches in the computation graph; Lin et al. (2017) adaptively prune channels based on the input feature map; Kuen al. (2018) introduce stochastic downsampling point to reduce the feature map size adaptively. Recently, Slimmable Nets (Yu et al., 2019; Yu & Huang, 2019b) propose to train model to support multiple width multipliers (e.g.. different global width multipliers), building upon existing human-designe neural networks (e.g.. MobileNerV2 0.35, 0.5, 0.75, 1.0). Such methods can adaptively fit different efficiency constraints runtime, however, still inherit pre-designed neural network (e.g., MobileNet-v2), which limits the degree of flexibility (e.g., only width multiplier can adapt) and the ability in handling new deployment scenarios where the pre-designed neural network is not optimal. In this work, in contrast, we enable much more diverse architecture space (depth, width, kernel size, and resolution) and a significantly larger number of architectural settings (1019 v.s. 4 (Yu et al., 2019)). Thanks to the diversity and the large design
3

---

Published as a conference paper at ICLR 2020
Full
Elastic Resolution
Full
Elastic Kernel Size
Full
Elastic
Full
Elastic Width
Depth
Partial
Partial
Partial
Partial
Figure 3:
Illustration of the progressive shrinking process to support different depth D, width W, kernel size K and resolution R. It leads large space comprising diverse sub-networks (> 10
space, we can derive new specialized neural networks for many different deployment scenarios rather than working top of an existing neural network that limits the optimization headroom. However, it more challenging to train the network to achieve this flexibility, which motivates us to design the progressive shrinking algorithm tackle this challenge.
3 METHOD
3.1 PROBLEM FORMALIZATION
Assuming the weights of the once-for-all network as Wo and the architectural configurations as {arch;), we then can formalize the problem as
min Wo
Σ Lual (CWW,,
(1)
where C(W,, oy arch.) denotes selection scheme that selects part of the model from the once-for-all network W. to form a sub-network with architectural configuration arch. The overall training objective is to optimize Wo to make each supported sub-network maintain the same level of accuracy as independently training network with the same architectural configuration.
3.2 ARCHITECTURE SPACE
Our once-for-all network provides one model but supports many sub-networks of different sizes, covering four important dimensions of the convolutional neural networks (CNNs) architectures, i.c., depth, width, kernel size, and resolution. Following the common practice of many CNN models (He et al., 2016; Sandler et al., 2018; Huang et al., 2017), we divide a CNN model into a sequence of units with gradually reduced feature map size and increased channel numbers. Each unit consists of a sequence of layers where only the first layer has stride 2 if the feature map size decreases (Sandler et al., 20. All the other layers in the units have stride
We allow each unit to use arbitrary numbers of layers (denoted as elastic depth); For each layer, we allow to use arbitrary numbers of channels (denoted as elastic width) and arbitrary kernel sizes (denoted as elastic kernel size). In addition, we also allow the CNN model to take arbitrary input image sizes (denoted as elastic resolution). For example, in our experiments, the input image size ranges from 128 to 224 with stride 4; the depth of each unit chosen from (2, 3, 4) the width expansion ratio in each layer is chosen from (3. 4, 6): the kernel size chosen from (3, 5, 7}. Therefore, with units, we have roughly ((3 x 3)2 (3 3) (3 3)4)5
1029 different neural network architectures and each of them can be used under 25 different input resolutions. Since all of these sub-networks share the same weights (i.e., W.) (Cheung al., 2019), we only require 7.7M parameters to store all of them. Without sharing, the total model size will be prohibitive.
3.3 TRAINING THE ONCE-FOR-ALL ALL NETWORK
Naïve Approach. Training the once-for-all all network can be cast as multi-objective problem, where each objective corresponds to one sub-ne network. From this perspective, naïve training approach to directly optimize the once-for-all network from scratch using the exact gradient of the overall objective, which is derived by enumerating all sub-networks in each update step, as shown in Eq. (1). The cost this approach is linear to the number of netorss. Therefore, it only applicable to scenarios where limited number of sub-networks are supported (Yu al., 2019), while in case, is computationally prohibitive to adopt this approach.
Another naive training approach is sample few sub-networks in each update step rather than enumerate all of them, which does not have the issue of prohibitive cost. However, with such large number of sub-networks that share weights, thus interfere with each other, we find it suffers from
4

---

Published as a conference paper at ICLR 2020
Network Pruning
Train the full model
Shrink the model (only width)
Fine-tune the small
single pruned network
Progressive Shrinking
Fine-tune both large and small sub-nets
Train the full model
Shrink the model (4 dimensions)
once-for-all network
Figure 4: Progressive shrinking be viewed generalized network pruning technique with much higher flexibility. Compared to network pruning, it shrinks more dimensions (not only width) and provides much more powerful once-for-all network that can fit different deployment scenarios rather than a single pruned network.
7x7
5x5
3x3
unit
iiKe shrink the depth
unit
I
unit
O1
O2
Transform Matrix 25x25
Transform Matrix 9x9
03
train with full depth
shrink the depth
Figure 5: Left: Kerel transformation matrix for elastic kernel size. Right: Progressive shrinking for elastic depth. Instead of skipping each layer independently, we keep the first D layers and skip the last (4 D) layers. The weights of the early layers are shared.
significant accuracy drop. the following section, we introduce a solution to address this challenge. i.e., progressive shrinking.
The once-for-all network comprises many sub-networks different sizes where small sub-networks are nested in large sub-networks. To prevent interference between the sub-networks, we propose enforce a training order from large sub-networks small sub-networks in progressive manner. We name this training scheme progressive shrinking (PS). An example of the training process with PS is provided in Figure 3 and Figure 4, where we start with training the largest neural network with the maximum kernel size (e.g., 7), depth (e.g., 4), and width (e.g., 6). network to support smaller sub-networks by gradually adding them into the sampling space (larger sub-networt may also be sampled). Specifically, after training the largest network, we first support clastic kernel size which can choose from (3, 5, 7} at each layer, while the depth and width remain the maximum values. Then, support elastic depth and elastic width sequentially, as is shown in Figure 3. The resolution elastic throughout the whole training process, which is implemented by sampling different image sizes for each batch of training data. We also use the knowledge distillation technique after training the largest neural network (Hinton et al., 2015; Ashok al., 2018; Yu & Huang, 2019b). It combines two loss terms using both the soft labels given by the largest neural network and the real labels.
Progressive Shrinking.
Next, we progressively fine-tune
Compared the naïve approach, PS prevents small sub-networks from interfering large sub-networks. since large sub-networks are already well-trained when the once-for-all network is fine-tuned to support small sub-networks. Regarding the small sub-networks, they share the weights with the large ones. Therefore, PS allows initializing small sub-networks with the most important weights of well-trained large sub-networks, which expedites the training process. Compared to network pruning (Figure 4), PS also starts with training the full model, but it shrinks not only the width dimension but also the depth, kernel size, and resolution dimensions the full model. Additionally. PS fine-tunes both large and small sub-nett networks rather than single pruned network. As result. PS provides a much more powerful once-for-all network that can diverse hardware platforms and efficiency constraints compared to network pruning. We describe the details of the PS training flow follows:
5

---

Published as a conference paper at ICLR 2020
channel importance
channel importance
0.02
0.82
channel sorting
channel sorting
reorg
channel
0.15
reorg
0.11
sorting
0.85
0.46
+O
0.63
O1
On
+01 progressively shrink the width
train with full width
progressively shrink the width
Figure 6: Progressive shrinking for elastic width. In this example, we progressively support 4. 3, and 2 channel settings. We perform channel sorting and pick the most important channels (with large L1 norm) to initialize the smaller channel settings. The important channels' weights are shared.
Elastic Kernel Size (Figure 5 left). We let the center of a 7x7 convolution kernel also serve as 5x5 kernel, the center of which can also be a 3x3 kernel. Therefore, the kernel size becomes elastic. The challenge is that the centering sub-kernels (e.g., 3x3 and 5x5) are shared and need to play multiple roles (independent kernel and part of a large kernel). The weights of centered sub-kernels may need to have different distribution magnitude as different roles. Forcing them to be the same degrades the performance of some sub-networks. Therefore, we introduce kernel transformation matrices when sharing the kernel weights. We separate kernel transformation matrices for different layers. Within each layer, the kernel transformation matrices are shared among different channels. As such, we only need 25 x 25 9 x 706 extra parameters store the kernel transformation matrices in each layer, which is negligible.
Elastic Depth (Figure 5 right). To derive a sub-network that has layers in a unit that originally has N layers, we keep the first D layers and skip the last N - D layers, rather than keeping any D layers as done current NAS methods (Cai al., 2019; Wu al., 2019). As such, one depth setting only corresponds to one combination of layers. In the end, the weights of the first D layers are shared between large and small models.
Elastic Width (Figure 6). Width means the number of channels. We give each layer the flexibility to choose different channel expansion ratios. Following the progressive shrinking scheme, we first train a full-width model. Then we introduce a channel sorting operation to support partial widths. It reorganizes the channels according to their importance, which is calculated based on the LI norm of channel's weight. Larger L1 norm means more important. For example, when shrinking from a 4-channel-layer to 3-channel-layer, we select the largest 3 channels, whose weights are shared with the 4-channel-layer (Figure 6 left and middle). Thereby, smaller sub-networks initialized with the most important channels on the once-for-all network which is already well trained. This channel sorting operation preserves the accuracy of larger sub-networks.
3.4 SPECIALIZED MODEL DEPLOYMENT WITH ONCE-FOR-ALL NETWORK
Having trained once-for-all network, the next stage is to derive the specialized sub sub-network for given deployment scenario. The goal is search for neural network that satisfies the efficiency (e.g., latency, energy) constraints on the target hardware while optimizing the accuracy. Since OFA decouples model training from neural architecture search, we do not need any training cost this stage. Furthermore, we build wewrmllnntwooo--wwiee network architecture, providing a quick feedback for model quality. eliminates the repeated search cost by substituting the measured accuracy/latency with predicted accuracy/latency (twins).
to predict the latency and accuracy given a neural
Specifically, we randomly sample 16K sub-networks with different architectures and input image sizes. then measure their accuracy on 10K validation images sampled from the original training set. These [architecture, accuracy] pairs are used to train an accuracy predictor to predict the accuracy of model given its architecture and input image ? We also build latency lookup table (Cai et al., 2019) on each target hardware platform to predict the latency. Given the target hardware and latency constraint, conduct evolutionary search (Real et al., 2019) based on the neural-network. twins to get a specialized sub-network. Since the cost of searching with neural-network -twwiss is negligible,
Details of the accuracy predictor is provided in Appendix A.

---

Published as conference paper at ICLR 2020
w/o PS
w/ PS
ImageNet Top-1 Aco (%)
3.5%
3.4%
3.4%
3.3%
73
3.5%
2.8%;
2.5%
D=2
D=2
D=4
D=4
W=3
W=3
W=6
W=6
W=3
W=3
W
W=6
K=3
K=7
K=3
K=7
K=3
K=7
K=3
K=7
Sub-networks under various architecture configurations D: depth, W: width, K: kernel size
Figure 7: ImageNet topl accuracy (%) performances of sub-networks under resolution 224 224. "(D d. W = w, K = k)" denotes a sub-network with d layers in each unit, and each layer has an width expansion ratio w and kernel size
we only need 40 GPU hours to collect the data pairs, and the cost stays constant regardless of #deployment scenarios.
EXPERIMENTS
In this section, we first apply the progressive shrinking algorithm to train the once-for-all network on ImageNet (Deng al., 2009). Then we demonstrate the effectiveness of our trained once-for-all network on various hardware platforms (Samsung S7 Edge, Note8, Note10, Google Pixel1, Pixe12, LG G8, NVIDIA 1080Ti, V100 GPUs, Jetson TX2, Intel Xeon CPU, Xilinx ZU9EG, and ZU3EG FPGAs) with different latency constraints.
4.1 TRAINING THE ONCE-FOR-ALL NETWORK ON IMAGENET
Training Details. We use the same architecture space as MobileNetV3 (Howard et al., 2019). For training the full network, use the standard SGD optimizer with Nesterov momentum 0.9 and weight decay 35 The initial learning rate is 2.6, and we use the cosine schedule (Loshchilov & Hutter, 2016) for learning rate decay. The full network trained for 180 epochs with batch size 2048 on 32 GPUs Then we follow the schedule described in Figure to further fine-tune the full network? The whole training process takes around 1,200 GPU hours on V100 GPUs. This one-time training cost that can be amortized by many deployment scenarios.
Results. Figure reports the topl accuracy of sub-networks derived from the once-for-all networks that are trained with our progressive shrinking (PS) algorithm and without PS respectively. Due to space limits, we take 8 sub-networks for comparison, and each of them denoted as "(D W w. K =>"" It represents sub-network that has d layers for all units, while the expansion ratio and kernel size are set to w and & for all layers. PS can improve the ImageNet accuracy of sub-networks by a significant margin under all architectural settings. Specifically, without architecture optimization, PS can achieve 74.8% topl accuracy using 226M MACs under the architecture setting (D=4, W=3, K=3), which is par with MobileNerV3-LLan In contrast, without PS, it only achieves 71.5%. which is 3.3% lower.
4.2 SPECIALIZED SUB-NETWORKS FOR DIFFERENT HARDWARE AND CONSTRAINTS
We apply our trained once-for-all network get different specialized sub-networks for diverse hardware platforms: from the cloud to the edge. On cloud devices, the latency for GPU is measured with batch size 64 on NVIDIA 1080Ti and V100 with Pytorch 1.0+cuDNN. The CPU latency is measured with batch size 1 on Intel Xeon E5-2690 v4+MKL-DNN. On edge devices, including mobile phones, we use Samsung, Google and LG phones with TF-Lite, batch size I; for mobile GPU,
Implemmntatton details he found in Appendix B.
7

---

Published as a conference paper at ICLR 2020
Model
ImageNet Topl (%)
MAC's
Mobile
lalency
Search cost (GPU hours)
Training cost
(GPU bours)
GPL hours
Total cost (NV
COye (lbs)
10)
AWS
MobileNatV215 [31]
72.0
300M
06ms
0
I30N
6k
1.7k
S18.4K
MobileNetV2 #1200
73.5
300M
66ms
1200N
13.6k
$146.9L
NASNei-A [44]
74.0
564M
48,000N
1920k
544.5k
$5875,2K
DARTS [25]
73.1
595M
96.V
250N
14k
4.0%
$42.8K
MnasNet [33]
74.0
317M
70ms
40,000N
.,00
453.8K
$4896600 OK
FBNet-C [36]
74.9
375M
216N
360N
23k
6.5%
$70.4k
ProxylessNAS [4]
74.6
320M
200.N
300.V
20k
5.7k
561
SinglePathNAS [8]
74.7
328M
288 24.V
384.V
17k
4.8k
S52.0k
AutoSlim [38]
74.2
30SM
63ms
180
300
12K
3.4k
S36.7k
MohileNetV3-- [15]
75.2
219M
58ms
180.V
7.2k
1.8k
S22.2K
OFA w/o PS
72.4
2
59ms
40
1200
1.2k
0.34K
$3.7k
1200
OFA w/ PS #25
76.4
230M
58ms
40
1200 + 25.V
2.2k
0.62k
$6.7k
OFA w/ PS #75
76.9
230M
58ms
40
1200 + 75.N
4.2k
1.2k
$13.0K
aFA #75
80.0
40
1200 75.V
4.2k
122K
S13.0k
Table 1: Comparison with SOTA hardware-awand NAS methods on Pixel1 phone. OFA decouples model training from neural architecture search. The search cost and training cost both stay constant as the number of deployment scenarios grows. "#25" denotes the specialized sub-networks are fine-tuned for 25 epochs after grabbing weights from the once-for-all network. "COge" denotes CO2 emission which is calculated based on Strubell et al. (2019), AWS cost is calculated based on the price of on-demand P3. 16xlarge instances.
Total cost (bs CO2 emission), N 40
ProxylessNAS
5.7k
FBNet
6.5k
MnssNett
454k
OFA
|0.34k
1300x
Figure 8: OFA saves orders of magnitude design cost compared to NAS methods.
we use Jetson TX2 with Pytorch 1. L.0+cuDNN, batch size of 16; for embedded FPGA, we use Xilinx ZU9EG and ZU3EG FPGAS with Vitis Al batch size 1.
Comparison with NAS Mobile Devices. Table reports the comparison between OFA and ttate-of-the-art hardware-aware NAS methods on the mobile phone (Pixell). OFA is much more efficient than NAS when handling multiple deployment scenarios since the cost of OFA constant while others are linear to the number of deployment scenarios (N). With N = 40, the total CO2 emissions of OFA is 16x fewer than ProxylesNNAS, 19x fewer than FBNet, and 1,300 fewer ImageNet,
than MnasNet (Figure 8). Without retraining, OFA achieves 76.0% topl accuracy which is 0.8% higher than MobileNetV3-Large while maintaining similar mobile latency. We can further improve the topl accuracy 76.4% by fine-tuning the specialized sub-netwe sub-network for 25 epochs and to 76.9% by fine-tuning for 75 epochs. Besides, we also observe that OFA with PS can achieve 3.6% better accuracy than without PS.
OFA under Different Computational Resource Constraints. Figure summarizes the results of OFA under different MACs and Pixel1 latency constraints. OFA achieves 79.1% % ImageNet top1 accuracy with 389M MACs, being 2.8% more accurate than EfficientNet-Bi that has similar MACs. With 595M MACs, OFA reaches new SOTA 80.0% ImageNet topl accuracy under the mobile setting (<600M MACs), which is 0.2% higher than EfficientNet-B22 while using 1.68x fewer MACs. More importantly, OFA runs much faster than EfficientNets on hardware. Specifically, with 143ms Pixell latency. OFA achieves 80.1% % ImageNet topl accuracy, being 0.3% more accurate and 2.6x faster than EfficientNet-B2 We also find that training the searched neural architectures from scratch cannot reach the same level of accuracy as OFA, suggesting that not only neural architectures but also pre-trained weights contribute to the superior performances of OFA.
Figure 10 reports detailed comparisons between OFA and MobileNetV3 on six mobile devices. Remarkably, OFA can produce the entire trade-off curves with many points over wide range of latency constraints by training only once (green curve). It is impossible for previous NAS methods (Tan et al., 2019; Cai al., 2019) due to the prohibitive training cost.
8

---

Published as a conference paper at ICLR 2020
* OFA
OFA- Train from scratch
EfficientNet
81
Top-1 1 ImageNet Acc (%)
1.68x MACs
Top-1 ImageNet Acc (%)
80.1
latency reduction
80
reduction
79.8
80
79.8
79.1
79.8
78.8
79
76.7
2.8% higher accuracy
78.2
77.8
77.9
78
78.0
/7.6
3.8% higher
$7,,0
77
76.4.
76.3
76.3
76
200
400
600
800
1,000
1,200
50
150 200 250 300 350 400 Google Pixel1 Latency (ms)
100
MACs (M)
Figure 9: OFA achieves 80.0% top accuracy with 595M MACS and 80.1% top1 accuracy with 143ms Pixell latency, setting a new SOTA ImageNet topl accuracy on the mobile setting.
025
OFA
MobileNaiV3
77
77
Top ImageNent Acc (%)
75
74.7
Top- ImageNet Ace (%)
75
76.1
Top-1 ImageNer Aca (Mg)
75
75.5
76.8
78.1
73.6
75.2
73
78.8
72.8
73.3
70.5
-00
#70.4
71
70.4
71
700
89
67
67.4
67.4
25
55
70
100
13
15
23
Samsung Edge Latency (ms)
Samsung Note8 Latency (ms)
Samsung Note10 Latency (ms)
77
75.8
77
78.4
Top-1 Imageter Ace (%6
75
74.7
73.3
co
73.3
75.2
Top-1 ImageNett Ace
73
73.4
00000
73,3
76.2
ImageNNet Acc
73
73.0
73,3
71.5
71
71
71
770.4
#7704
69
67
67.4
67
467.4
42
23
28
33
38
43
48
53
58
68
68
10
13
16
19
22
25
Google Pixel1 Latency (ms)
Google
Pixel2
Latency (ms)
LG G8 Latency (ms)
Figure 10: OFA consistently outperforms MobileNetV3 on mobile platforms.
OFA for Diverse Hardware Platforms. Besides the mobile platforms, extensively studied the effectiveness of OFA on six additional hardware platforms (Figure 11) using the ProxylcssNAS architecture space (Cai et al., 2019). OFA consistently improves the trade-off between accuracy and latency by a significant margin, especially on GPUs which have more parallelism. With similar latency MobileNetV2 0.35, "OFA #25" improves the ImageNet top1 accuracy from MobileNetV2's 60.3% to 72.6% (+12.3% improvement) on the 1080Ti GPU. Detailed architectures of our specialized models shown in Figure 14. It reveals the insight that using the same model for different deployment scenarios with only the width multiplier modified has limited impact on efficiency improvement: the accuracy drops quickly as the latency constraint gets tighter.
OFA for Specialized Hardware Accelerators. There has been plenty of work on NAS for general- purpose hardware, but little work has been focused on specialized hardware accelerators. We quantitatively analyzed the performance of OFA on two FPGAs accelerators (ZU3EG and ZU9EG) using Xilinx Vitis AI with 8-bit quantization, and discuss two design principles.
Principle 1: memory access is expensive, computation is cheap. An efficient CNN should do as much computation with small amount of memory footprint. The ratio is defined as the arithmetic intensity (OPs/Byte). The higher OPs/Byte, the less memory bounded, the easier to parallelize. Thanks to OFA's diverse choices of sub-network architectures (1029) (Section 3.3), and the OFA

---

Published as a conference paper at ICLR 2020
O25.
OFA
MnasNet
MobileNerV2
Slimmable Nets
77
75.3
76.4
77
75.3
78.1
77
767
Top-1 ImagelNet (%)
73
72.6
73
71.6
73
72.0
$0.8
66
66
66
62
62
8603
000.3
60.9
22
26
30
11
15
19
NVIDIA 1080Ti Latency (ms)
NVIDIA V100 Latency (ms)
Intel Xeon CPU Latency (ms)
Batch Size 64
Batch Size =
Batch Size -
77
77
76.4
Top-1 ImageNet Acc (%)
70.3
69.4
69.6
69
89
$7.0
69
86.0
66
88
66
860
58
58
50.1
58
30
45
60
75
90
105
1.5
2.0
2.5
3.0
3.5
4.0
3.0
4.0
5.0
6.0
7.0
8.0
Jetson TX2 Latency (ms) Batch Size
Xilinx ZU9EG FPGA Latency (ms) Batch Size (Quantized)
Xilixx ZUGEG FPGA Latency (ms) Batch Size (Quantizee))
16
Figure 11: Specialized OFA models consistently achieve significantly higher ImageNet accuracy with similar latency than non-specialized neural networks on CPU, GPU, mGPU, and FPGA. More remarkably, specializing for new hardware platform does not add training cost using OFA.
MobileNet-v2
MnasNet
OFA (Ours)
50.0
160.0
80.0
Arithmetic Intensity
ZUBEG FPGA
ZU3EG FPGA
(OPS/Byte)
37.5
(GOPS/s)
120.0
(GOPS/s)
60.0
25.0
80.0
40.0
12.5
40.0
20.0
0.0
0.0
0.0
Figure 12: OFA models improve the arithmetic intensity (OPS/Byte) and utilization (GOPS/s) compared with the MobileNetV2 and MnasNet (measured results Xilinx ZU9EG and ZU3EG FPGA).
model twin that can quickly give the accuracy/latency feedhack (Section 3.4), the evolutionary search can automatically find a CNN architecture that has higher arithmetic intensity. As shown in Figure 12, OFA's arithmetic intensity is 48%/43% higher than MobileNetV2 and MnasNet (MobileNetV3 is not supported by Xilinx Vitis AI). Removing the memory bottleneck results in higher utilization and GOPS/s by 70%-90% pushing the operation point the upper-right in the roofline model (Williams et al., 2009), as shown in Figure 13. (70%-90% looks small in the log scale but that is significant).
Principle 2: the CNN architecture should be co-designed with the hardware accelerator's cost model. The FPGA accelerator has specialized depth-wise engine that is pipelined with the wisn engine. The pipeline throughput perfectly matched for 3x3 kernels. As result, OFA's searched model only has 3x3 kernel (Figure 14, on FPGA, despite 5x5 and 7x7 kernels are also in the search space. Additionally, large kernels sometimes cause "out BRAM" error on FPGA, giving high cost. On Intel Xeon CPU, however, more than 50% operations large kernels. Both FPGA and GPU models are wider than CPU, due to the large parallelism of the computation array.
CONCLUSION
We proposed Once-for-All (OFA), new methodology that decouples model training from architecture search for efficient deep learning deployment under large number of hardware platforms. Unlike
10

---

Published as a conference paper at ICLR 2020
10a
(Ours) MobileNet-v2
OFA (Ours)
MobileNet-v2
10a
MnasNet
MnasNet
GOPS/s
GOPS/s
10
102
104
10
100
10a
10a
10a
10
10a
10a
10a
OPS/Byte
OPS/Byte
(a) on Xilinx ZU9EG FPGA
(b) on Xilinx ZUSEG FPGA
Figure 13: Quantative study of OFA's roofline model on Xilinx ZU9EG and ZU3EG FPGAs (log scale). OFA model increased the arithmetic intensity by 33%/43% and GOPS/s by 72%/92% these two FPGAs compared with MnasNet.
Comn 3X3
3x3
MB4 3x3
3x3
MB5 3x3
MBS 3x3
MBS 3x3
9x3
MBS 3x3
MBS 3x3
MBB 3x3
Pooling
MBS
MSS
164x786
(a) 4ms latency Xilinx ZU3EG (batch size 1).
Cony 3x3
MBB 3x3
MB4 MB44555
5x55
MB3 5x5
MB3
MB3
MB3
MB4 Sx5
MB4
MBS 313
MB4 5x5
Pooling FC
MB4
144x144
(b) 10.9ms latency on Intel Xeon CPU (batch size = 1).
MB1 3x3
MBB4 3x3
MB6 3x3
MB3 3x3
MB3 5x3
3x3
MB4 5x5
MB3 3x3
MB6 3x3 3x3
MB6 3x3
MB6 7x7
MB4 7xT
MBS 3x3
5x5
Pooling FC
MBB
NB6
MB4
146x144
(c) 14.9 44 latency on NVIDIA 1080TΙ (batch size = 64).
Figure 14: OFA design specialized models for different hardware and different latency constraint. "MB4 3x3" means "mobile block with expansion ratio 4, kernel size 3x3". FPGA and GPU models are wider than CPU model due larger parallelism. Different hardware has different cost model, leading to different optimal CNN architectures. OFA provides unified and efficient design methodology.
previous approaches that design and train neural network for each deployment scenario, designed a once-for-all network that supports different architectural configurations, including elastic depth, width, kernel size, and resolution. It reduces the training cost (GPU hours, energy consumption, and CO2 emission) by orders of magnitude compared to conventional methods. To prevent sub-networks of different sizes from interference, we proposed progressive shrinking algorithm that enables a large number of sub-network to achieve the same level of accuracy compared to training them independently. Experiments demonstrated the effectiveness of our approach. OFA provides an automated ecosystem to efficiently design efficient neural networks with the hardware cost model the loop.
diverse range of hardware platforms and efficiency constraints
ACKNOWLEDGMENTS
We thank NSF Career Award #1943349, MIT-IBM Watson AI Lab, Award, Samsung, Intel, Xilinx, SONY, AWS Machine Learning Research Award for supporting this
Research
11

---

Published as a conference paper at ICLR 2020
research. We thank Samsung, Google and LG for donating mobile phones. We thank Shuang Wu and Lei Deng for drawing the Figure 2.
REFERENCES
Anubhav Ashok, Nicholas Rhinehart, Fares Beainy, and Kris M Kitani. N2n learning: Network to network compression via policy gradient reinforcement learning. In ICLR, 2018.
Han Cai, Tianyao Chen, Weinan Zhang. Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In AAAI, 2018a. 3
Han Cai, Jiacheng Yang, Weinan Zhang, Song Han, and Yong Yu. Path-level network transformation for efficient architecture search. In ICML, 2018b. 3
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In ICLR, 2019, URL wwt.h::/ttt/ttttee http:a///aaxv..rrrpp/////2.2.... 00332 00..2.pdf. 3, 8.
Brian Cheung. Alex Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Superposition of many models into one. NeurIPS, 2019.
Matthicu Courbariaux, Yoshua Bengio, and Scan-Piems David. Binaryconnect: Training deep neural networks with binary weights during propagations. In NeurIPS, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 7
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path shothot neural architecture search with uniform sampling. arXiv preprint vrXiv:1904.0044 2019. 8
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NeurIPS, 2015.
Song Han, Huizi Mao, and William J Dally. Deep compression: with pruning, trained quantization and huffman coding. In ICLR, 2016. 1,3
Compressing deep neural networks
Cong Hao, Xiaofan Zhang, Yuhong Li, Sitao Huang, Jinjun Xiong, Kyle Rupnow, Wen-mei Hwu, and Deming Chen. Fpga/dnn co-design: An efficient design methodology for lot intelligence on the edge. 2019 56th ACM/IEEE Design Automation Conference (DAC), pp. 1-6. IEEE, 2019.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In ECCV, 2018. 1,3
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.022. 02531, 2015. 5
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, al. Searching for mobilenetv3. In ICCV 2019, 2019. 7,8
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: /744v0170..0486,, 2017. 1,3
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017. 4
Gao Huang, Danlu Chen, Tianhong Li. Felix Wu, Laurens van der Maaten, and Kilian Q Weinberger. Multi-scale dense networks for resource efficient image classification. In ICLR, 2018. 3
12

---

Published as a conference paper at ICLR 2020
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf. William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. arXiv preprint arXiv: arXfv:1602.07366) 2016.
Weiwen Jiang, Lei Yang, Edwin Sha, Qingfeng Zhuge, Shouzhen Gu, Yiyu Shi, and Jingtong Hu, Hardware/software co-exploration of neural architectures. arXiv preprint arXiv: (i5i:::907.0465000 2019a.
Weiwen Jiang, Xinyi Zhang, Edwin H-M Sha, Lei Yang, Qingfeng Zhuge, Yiyu Shi, and Jingtong Hu. Accuracy vs. efficiency: Achieving both through fyga-implementation aware neural architecture search. In Proceedings of the 56th Annual Design Automation Conference 2019, pp. 1-6, 2019b. 3
Jason Kuen, Xiangfei Kong. Zhe Lin, Gang Wang, Jianxiong Yin, Simon See, and Yap-Peng Tan. Stochastic downsampling for cost-adjustable inference and improved regularization in convolutional networks. In CVPR, 2018. 3
Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In NeurIPS, 2017.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In ECCV, 2018. 2
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In ICLR, 2019. 3,
Lanlan Liu and Jia Deng. Dynamic deep neural networks: Optimizing accuracy-efficiency trade-offs by selective execution. In AAAI, 2018. 3
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In ICCV, 2017.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 033983, 2016. 7
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In ECCV, 2018. 3
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. AAAI, 2019. 3,
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018. 1, 1,3, 4,
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. In ACI., 2019. 8
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2820-2828, 2019. 3,
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. ECCV, 2018.
Samuel Williams, Andrew Waterman, and David Patterson. Roofline: An insightful visual perfor- mance model for floating-point programs and multicore architectures. Technical report, Lawrence Berkeley National (lBBLL, Berkeley, CA (United States), 2009. 10
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangging Jia, and Kurt Keutzer. Fbnet: Hardware-awar efficient convnet design via differentiable neural architecture search. In CVPR, 2019. 6.
Zuxuam Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In CVPR, 2018.
13

---

Published as a conference paper at ICLR 2020
Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers. 2019a.
arXiv preprint
Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques. In ICCV, 2019b. 3,5
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In ICLR, 2019. 3,4
Xiangyu Zhang. Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: extremely efficient convolutional neural network for mobile devices. In CVPR, 2018. 1,3
Chenzhuo Zhu, Song Han, Huizi Mao, and William Dally. Trained ternary quantization. In ICLR, 2017. 3
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018. 3,
A DETAILS OF THE ACCURACY PREDICTOR
We use three-layer feedforward neural network that has 400 hidden units in each layer as the accuracy predictor. Given a model, we encode each layer in the neural network into one-hot vector based its kernel size and expand ratio, and assign zero vectors layers that skipped. Besides, we have an additional one-hot vector that represents the input image size. We concatenate these vectors into large vector that represents the whole neural network architecture and input image size, which then fed to the three-layer feedforward neural network to get the predicted accuracy. In our experiments, this simple accuracy prediction model can provide very accurate predictions. At convergence, the poot-mean-squall еπоо (RMSE) between predicted accuracy and estimated accuracy on the test is only 0.21%. Figure 15 shows the relationship between the RMSE of the accuracy prediction model and the final results (i.e., the accuracy of selected sub-networks). We can find that lower RMSE typically leads to better final results.
75.5
Acc of Selected Sub-net (%)
74.6
73.8
72.9
72.0
15
20
RMSE of Acc Prediction Model (%)
Figure 15: Performances of selected sub-networks using different accuracy prediction model.
B IMPLEMENTATION DETAILS OF PROGRESSIVE SHRINKING
After training the full network, we first have one stage of fine-tuning to incorporate elastic kernel size. In this stage (i.e.. K [7, 5, 3)), sample one sub-network in each update step. The network is fine-tuned for 125 epochs with an initial learning rate of 0.96. All other training settings are the same training the full network.
14

---

Published as a conference paper at ICLR 2020
Next, have two stages of fine-tuning to incorporate elastic depth. We sample two sub-networks and aggregate their gradients each update step. The first stage (i.e., D E [4, 3)) takes 25 epochs with initial learning rate of 0.08 while the second stage (i.e., D [4, 14,3,22 2)) takes 125 epochs with an initial learning rate of 0.24.
Finally, have two stages of fine-tuning incorporate elastic width. We sample four sub-networks
and aggregate their gradients in each update step. The first stage (i.e., W [6, 4) takes 25 epochs with an initial learning rate of 0.08 while the second stage (i.e., W E [6, 4, 3)) takes 125 epochs with an initial learning rate of 0.24.
15


Reviews:
[{"review_id": "HygXKK8RKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this manuscript, authors propose an OFA NAS framework. They train a supernet first and then finetune the elastic version of the large network. After training, the sub-networks derived from the supernet can be applied for different scenarios directly without retraining. The motivation is clear and interesting. My concerns are as follows.\n1.\tWhen sampling sub-networks, a prediction model is applied to predict the accuracy of networks. It is interesting to show the accuracy of the prediction model itself and how it will influence the final selection.\n2.\tThe results compared in Table 2 are outdated. Authors should at least add the result of MobileNetV3.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bygt_gUQYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this papers, the authors learn a Once-for-all net. This starts as a big neural network which is trained normally (albeit with input images of different resolutions). It is then fine-tuned while sampling sub-networks with progressively smaller kernels, then lower depth, then width (while still sampling larger networks occasionally, as it reads). This results in a network from which one can extract sub-networks for various resource constraints (latency, memory etc.) that perform well without a need for retraining.\n\nThis paper is well written, and the results are very good. However there are serious problems that need addressing.\n\nThe method as described *is not reproducible*. The scheduling of sampling subnetworks is alluded to on page 4, and that's it. It is essential that the authors include their exact subnet sampling schedule e.g. as pseudocode with hyperparameters. There is no point doing good work if other researchers cannot build off it. \n\nOn another reproducibility note, as far as I can tell, the original model isn't given. There would be no harm in adding this to the appendix. \n\nFigure 1 is misleading, as we don't find out until later in the paper that Once For All #25 means that each of these points was finetuned for a further 25 epochs (which on ImageNet is non-trivial). This defeats the narrative of the paper (once-for-all plus some fine-tuning isn't exactly once-for-all).\n\nIs there a reason why the progressive shrinking goes resolution->kernel->depth->width? Was this just the permutation that worked best? I would be curious as to why this is.\n\nFor elastic width, I wasn't sure why the \"channel sorting operation preserves the accuracy of larger sub-networks\". Could you please elaborate?\n\nKudos on adding CO2 emissions in Table 2, I hope this gets reported more often.\n\nIn the introduction, the authors talk about iPhones and then the hardware considered is Samsung and Google. A minor note, but it seems inconsistent.\n\nAnother minor note, in Table 2, (Strubell et al) should be out of the brackets, as it is part of the sentence.\n\nGiven that there are 10^19 subnetworks that can be sampled, it would be nice to see more than 3-4 appear on a plot. This makes it seem like they might have been cherry-picked. Sampling a few 100/1000 subnets and producing some Pareto curves would be both interesting and insightful.\n\nPros\n-------\n- Good results\n- Well written\n- Neat idea\n\nCons\n-------\n- Training details are obfuscated. This paper should not be accepted without them.\n- Very few subnetworks of the vast quantity that exist are observed.\n\nIn conclusion, I am giving this paper a weak reject, as it is currently impossible to reproduce, and as such, is of no use to the community. If the authors remedy this I will gladly raise my score.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BklpzCV0FB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tries to tackle the problem of searching best architectures for specialized resource constraint deployment scenarios. The authors basically take a two-step approach: First train a large network including all the small networks with weight sharing and some specially designed trick (e.g., progressive shrinking). Second, use prediction based NAS method to learn the performance/inference prediction module, from which the good sub architecture corresponding to a particular scenario is obtained. The experiments show that the proposed method is promising.\n\nPros:\n\n\n1. It is an interesting new paradigm that tries to solve AutoML for different deployment scenarios “once for all”.  AFAIK there is no prior works thinking in this way.\n2. It is useful and encouraging to see the proposed method achieves satisfactory performances, on par with the current best method specially designed for different deployment environment, while the computational cost is reduced by a large margin. \n3. Paper is clearly written and easy to understand.\n\nCons:\n\n1. The motivation towards “progressive shrinking (PS)” is not that clear. It seems natural to train a large network, and from it to train sub structures, since overparameterization helps NN training. However, it is hard to imagine that training from large to small could eliminate the “interfering” of subnetworks, let alone “while maintaining the same accuracy as independently trained networks”.  To me it is neither theoretically nor empirically supported (Please note the training of subnetworks definitely affect the learnt weights of the big one through weight sharing). In particular, the subnetworks with weight sharing could achieve the same, or even better performances compared with those non shared counterparts, which seems too good to be true.\n    1. A possible explanation might be that the overparameterization brings additional gain in the optimization process of each small network, especially with the help of knowledge distillation. If that is true, an additional ablation study should be done to separate the benefits of PS, and the disadvantage of weight sharing (i.e., interfering). \n2. I see no statements about code release. If a clear, and TIMELY code (for the SEARCH phase, not only for the Eval phase) release could be done, then at least from the perspective of application, the impact of this paper could be further enhanced.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new method for neural architecture search, except it's not exactly that because model training is separated from architecture, which is the main point of the paper. Once this network is trained, sub-networks can be distilled from it and used for specific tasks.

The paper as submitted missed certain details, but after this was pointed out by reviewers the details were satisfactorily described by the authors. 

The idea of the paper is original and interesting. The paper is correct and, after the revisions by authors, complete. In my view, this is sufficient for acceptance.
Decision: Accept (Poster)