Fast and Efficient Once-For-All Networks for Diverse Hardware Deployment
Authors: Jun Fang
Li Yang
Chengyao Shen
Hamzah Abdel-Aziz
David Thorsley
Joseph Hassoun
Venue: ICLR 2022 Submitted (2022)
Keywords: neural architecture search
computer vision
convolutional neural networks

Abstract: Convolutional neural networks are widely used in practical application in many diverse environments. Each different environment requires a different optimized network to maximize accuracy under its unique hardware constraints and latency requirements. To find models for this varied array of potential deployment targets, once-for-all (OFA) was introduced as a way to simultaneously co-train many models at once, while keeping the total training cost constant. However, the total training cost is very high, requiring up to 1200 GPU-hours. Compound OFA (compOFA) decreased the training cost of OFA by 2$\times$ by coupling model dimensions to reduce the search space of possible models by orders of magnitude, while also simplifying the training procedure.

In this work, we continue the effort to reduce the training cost of OFA methods. While both OFA and compOFA use a pre-trained teacher network, we propose an in-place knowledge distillation procedure to train the super-network simultaneously with the sub-networks. Within this in-place distillation framework, we develop an upper-attentive sample technique that reduces the training cost per epoch while maintaining accuracy. Through experiments on ImageNet, we demonstrate that, we can achieve a $2\times$ - $3\times$ ($1.5\times$ - $1.8\times$) reduction in training time compared to the state of the art OFA and compOFA, respectively, without loss of optimality.

Under review as a conference paper at ICLR 2022
FAST AND EFFICIENT ONCE-FOR-ALL NETWORKS FOR DIVERSE HARDWARE DEPLOYMENT
Anonymous authors Paper under double-blind review
ABSTRACT
Convolutional neural networks are widely used in practical application in many diverse environments. Each different environment requires different optimized network to maximize accuracy under its unique hardware constraints and latency requirements. To find models for this varied array of potential deployment tar- gets, once-for-all (OFA) was introduced as a way to simultancously co-train many models once, while keeping the total training cost constant. However, the to- tal training cost is very high, requiring up to 1200 GPU-hours. Compound OFA (compOFA) decreased the training cost of OFA by 2x by coupling model dimen- sions to reduce the search space of possible models by orders of magnitude, while also simplifying the training procedure.
In this work, we continue the effort to reduce the training cost of OFA methods. While both OFA and compOFA use pre-trained teacher network, we propose an in-place knowledge distillation procedure to train the super-network simultane- ously with the sub-networks. Within this in-place distillation framework, we de- velop an upper-attentive sample technique that reduces the training cost per epoch while maintaining accuracy. Through experiments on ImageNet, we demonstrate that, we can achieve 2 3x (1.5x 1.8x x) reduction in training time compared to the state of the art OFA and compOFA, respectively, without loss of optimality.
INTRODUCTION
Convolutional neural networks (CNNs) are overwhelmingly successful in many machine learning applications. These applications may have different inference constraints (e.g., latency) and are de- ployed in different hardware platforms that range from server-grade platforms to edge devices such as smartphones. Optimal network architectures need to be designed to meet the requirement for a target deployment scenario. However, naively designing specialized architecture for each scenario is very expensive as requires to fully retrain the model each time. This is an excessively expensive process in terms of the required machine learning expertise, time, energy and CO2 emission.
Recently, researchers have proposed efficient methods which are based on training a super net- work only once. Then, for specific deployment scenario, sub-network is sampled from the super-network that meet the deployment constraints with the best accuracy. The weight of the sam- pled network shared with original super network, hence retraining is not required. Once-for-all (OFA) (Cai et al.) 2020) is among the first methods proposed to tackle this problem. The OFA method trains a once-for-all network that jointly optimizes the accuracy of large number of sub- networks (more than 19 sampled from the once-for-all network. Each sub-network is selected from the once-for-all network where layer depths, channel widths. kernel sizes and input resolu- tion are scaled independently. Such scaling provides a family of CNNs networks with different computation and representation power to flexibly support deployment under diverse platforms and configurations. With this massive search space, OFA co-trains all the sub-networks by complex four-stage progressive training process which is prohibitively expensive and costs around 1200 GPU hours.
Compound OFA (CompOFA) (Sahni et al. 2020) builds upon the original OFA by shrinking the design space of possible sub-networks. This is done by only considering networks whose dimensions are coupled. This reduces the number of possible models by 17 orders of magnitudes, from 10 19 down to 243. Sahni et al. (2020) demonstrate that this smaller design space is sufficient, most

---

Under review as a conference paper at ICLR 2022
OFA
teecher training
elastic
elestic depth
CompOFA
training
compound training
1OFA
wermup phase
In-place distilation
Comparison of training schedules for OFA, CompOFA, and fOFA. Length on the horizon- tal axis is proportional to the number of epochs in each phase. For OFA, "elastic kernel," "elastic width," and "elastic depth" are the phases training specified in Cai et al. (2020) that are not used in CompOFA and fOFA.
Figure I:
sub-networks in the original OFA design space are far from the optimal accuracy-latency frontier. With this smaller space, the training procedure can be simplified well these suboptimal sub- networks are no longer influencing the training process. CompOFA reduces the four stages of the original OFA process to stages, and this optimization speeds up the training time of CompOFA by a factor of 2 over OFA.
However, 2x faster than OFA's 1200 GPU-hours still 600 GPU-hours. Even with this significant improvement, the training cost remains vary expensive, especially when effects on the environment are considered (Strubell et al. 2019). While some of this cost can be mitigated by improvements in hardware efficiency and the continued development of specialized platforms for training CNNs, algorithmic enhancements still have a large role to play. While CompOFA greatly simplifies the pro- gressive shrinking training procedure used in OFA, it is still dependent on pre-training supernet- work to act as a teacher for the sub-network co-training process, which uses knowledge distillation Hinton al. (2015). Due to the optimizations in the co-training process, training the supernetwork in CompOFA requires more than half (180 out of 330) of the total training epochs.
In this work, we propose several optimizations to the mor-ffo-all training process that produces one-stage training algorithm for fast and efficient neural architecture search. The key features of our method are:
We co-train all of the sub-networks from scratch without pre-training a teacher network, using the concept of in- in-place distillation (Yu et al.) (2020). The largest network we train using in in-place distillation is smaller than the pre-trained teacher network used in Cai et al. (2020) and (ahni et al. (2020).
During the co-training process, we develop an upper-attentive sampling method which al- ways sample the full-sized sub-net each iteration to help co-train the rest sub-networks.
Before co-training, we use upper-attentive warmup technique which trains only the full- sized sub-net for a few epochs before co- co-training to further improve the performance.
With these optimizations, we can decrease the number of sampled sub-networks in each iteration of training, further improving performance.
The benefits of our proposed fast OFA (fOFA) method are shown in Figure | Furthermore, since our method has only single stage, we can easily increase the training time and improve the accuracy of previous methods while still requiring less training time.
The rest of this paper is organized follows. We describe related work in more detail in Section and illustrate our method in depth in Section 3 and finish the paper with conclusions in
We report on our experimental results in Section
RELATED WORK
Neural architecture search (NAS) aims to automatically find the optimal network architecture given hardware constraints, such FLOPs or latency. Early NAS works mainly adapted reinforcement learning (Zoph et al. 2018; Zoph & Le 2016), evolutionary search (Real et al., 2019 2017). or
sparse connection learning Kim et al. 2018) to sample different architectures. However, each sampled architecture needed to be trained from scratch, resulting in a huge and intractable computing cost. More recent NAS works greatly reduce the cost by training pver-parameterized network
2

---

Under review as a conference paper at ICLR 2022
named super-network, and then sample various sub-networks which share the weights with the super-network. Such super- based methods can be further divided into two main categories as follows:
2.1 TWO-STAGE TRAINING
The main idea of the two-stage training methods (Berman et al. 2020 Bender et al. 2018 Brock al., 2017] Guo et al., 2019 Liu et al., 2018 Pham al., 2018) is that after searching for best architectures in the first stage of training, the best architectures then have to be retrained from scratch to obtain a final model. Generally, a single two-stage search experiment can only target a single resource budget narrow range of resource budgets at time, which is inefficient.
2.2 ONE-STAGE: TRAINING
To alleviate the inefficiency of two-stage training, Once-For-All (OFA) (Cai et al. 2020) was pro- posed to jointly train various sub-networks of the super-network in single stage. By doing so, the sub-networks could be directly deployed into different hardware platform without retraining. However, to support an extremely large number sub-networks (i.e., 10²h), such one-stage training involves multi-steps gradually add more sub-networks by using the proposed progressive shrink- ing technique. Moreover, OFA also needs to train a single full-sized teacher network (same size super-network) from scratch firstly to guide the training of the sub-ne sub-networks by using knowl- edge distillation. Thus, due to the complex training procedure, OFA still suffers from a prohibitive training cost, requiring around 1200 GPU hours.
More recently, inspired by studies on neural network design spaces Tan & Le 2019 Radosavovic
et al. 2020 CompOFA (Sahni al. 2020) proposes compound sub-networt scaling method, which couples the depth and width configuration of the sampled sub-networks to constrain the search space, reducing the space to only 243 number of sub-networks without losing accuracy. Although CompOFA achieves a 2 training cost reduction compared with OFA, it follows the similar training procedure that also needs train a teacher model from scratch. A similar approach is investigated in Yang et al. (2020). which couples network width and input resolution into a single mutual learning framework.
In addition, bigNAS Yu et al. 2020) proposed replacing OFA's multi-step training by single step, namely one-shot NAS, challenging the usual practice of progressive training in OFA. The idea of the one-shot NAS is to jointly train sub-networks from scratch directly by using the sandwich rule and in-place distillation techniques proposed for slimmable networks (Yu et al., 2018). Based on bigNAS, Wang et al. (2020) points out the unnecessary updates on sub-optimal models in one-stage training. and uses attention mechanisms to push the Pareto front. However, the primary objective of these works is to obtain better accuracy, thus still suffer from high training cost, e.g., bigNAS needs over 2300 TPU hours for (110²2) models. In addition, (Li et al. 2021) works to improve the trade-off between accuracy and computation complexity based for slimmable networks by introduc- ing a dynamic gating mechanism and in- in-place ensemble bootstrapping to increase training stability.. However, requires a one more gating training step, resulting in larger training cost.
Our approach also follows the direction of the one-shol model. Differentiating from bigNAS, the primary objective this work reduce the training cost without loss of Pareto optimality under the design space of OFA and CompOFA.
3 METHODS
3.1 BUILDING THE SEARCH SPACE
A neural network A is a function that takes an input set X and generates a set of outputs ó (A. In this work, focus fixed input set (i.e., ImageNet), and thus write the network output as 5 (A). In the supervised learning setting, the performance of the neural network evaluated against set of labels Yp.
Following the standard practice in neural architecture search, we limit our neural network space to the set of architectures that consists of sequence of blocks B1, B2y..., Bun where 5 is
3

---

Under review as a conference paper at ICLR 2022
D copies
SeparableConv20 Activation
BatchNorm
Activation
Squeeze Excitation
BetchNorm
inverted bottleneck
depth convolution
W
Figure 2: The search space used for fOFA, based on the architecture space MobileNetV3 (Howard et al., 2019). The dimension K refers to the size of the convolutional kernel, W to the channel expansion ratio, and D to the number of repetitions of the block.
typical value. Each block is based on the inverted residual in the architecture space of MobileNetV3 (Howard et al. 2019). block is parameterized by three dimensions: the depth (number layers in the block) D, the width (channel expansion ratio) W, and the convolution kernel size K. This search space illustrated in Figure 2
To reduce the size of the search space, use the same coupling heuristic CompOFA (Sahni al., 2020]: that is, if there are n choices for the depth dimension and n choices for the width dimension, sample the ith largest depth w; whenever the block. While OFA uses an elastic kernel that allows for different kernel sizes within blocks, follow COmpOFA and use a fixed kernel size within each block. We call the network where the values of K, D, and W are each their largest possible value the full-sized network or super-net, and the network created by any other choice of these values sub-network.
sample the ith largest depth d; for each layer in
As in CompOFA, we choose three possible values for D E {2, 3, 4} and three possible values for W [3, 4, 6} and fix the kernel size to that of (Howard et al. 2019), that is, K in the first, third, and fourth blocks, and K = 5 in the second and fifth blocks. Thus, with five blocks, we have 35 = 243 models in our search space.
In neural architecture search, the input resolution can vary as well, up maximum size of 224 x 224 for ImageNet (Deng et al. 2009). In this work, we use an elastic resolution, where input images resized to be square with dimension in the set {128, 160, 192, 224].
3.2 ONCE-FOR-ALL TRAINING
Both OFA and CompOFA use knowledge distillation to guide the super-net co-training procedure. In general, co-training all the sub-networks with a teacher model can be considered as multi-objective optimization problem, which can be formulated
min Σ NTo,
(I)
where V denotes the weights of the full-sized network, Nr. is the additional pre-trained teacher model, and No, is random sub-network of N where a, specified the sub-network architecture. The loss function is
f NN, YD) (8 YD) C( &
where denotes the distillation weight. This optimization function aims to co-train all the sub- networks during the training using both the target label and output of the teacher network using knowledge distillation. However, because there are so many sub-networks, it is not practical to compute this loss function in its entirety. So, following the approach of OFA and CompOFA, randomly sample n sub-networks in each training iteration. The loss function thus reformulated

---

Under review as a conference paper at ICLR 2022
full-sized network
randomly selected sub-networks
Batch
Batch 2
Batch
Figure 3: In upper attentive sampling, the largest possible model (in our case with D = 4 and W 6 for all blocks, shown red) selected during each batch of the training process. The other models selected at each batch are randomly chosen from all possible sub-networks. Upper attentive sampling differs from "sandwich model" of Yu et al. (2020) in that the smallest possible model (shown in blue) need not be selected at each batch.
N' min Σ c. (Wrand((a)) Wr, YD).
(3)
where n is typical value of the number sub-networks to sample.
3.3 IN-PLACE DISTILLATION
Requiring the training of teacher model Ny adds significant overhead to the total training time, as teacher training must completed before training subnets can begin. (Sahni er al. 2020) reports that teacher training takes up 17.6% of the wall time for OFA and 35.0% for CompOFA.
In this work, we propose to climinate training the teacher model and instead co-training the sub- networks from scratch. If we remove Ny from the loss function above, we reformulate random sampling loss function
min
Σ c (Nrand(aa))) YD). N
(4)
where L (Na,, YD) = ( (NVa. YD) for any network Na,- However, this naive sampling method
results in significant accuracy drops if co-training sub-networks from scratch. To improve accuracy,
BigNAS Yu et al. 2020) uses the "sandwich model" from Yuu & Huang. 2019). wherein the largest and smallest possible sub-networks always sampled. Its loss function
n-2
min [[NNnax YD) + ((randd(aa)) (Naxx) + L (Nuin: Nwaxe)
(5)
1=1
where Nuxx denotes the full-sized network and Nmin denotes the smallest sub-network. The full- sized network is thus trained in parallel with the smaller models.
3.4 UPPER ATTENTIVE SAMPLING
The sandwich model proposed for BigNAS applies to a high training cost scenario where 10 models are being evaluated. In our scenario, with only 243 models, find that including the
5

---

Under review as a conference paper at ICLR 2022
Table 1: Training schedule for fOFA. We replace the lengthy teacher training phase with in-place distillation, preceded by a short warmup phase. We also decrease the size of the teacher kernels from K 7 K K 5, described in Section 3.1.
Method
Phase
K
D
W
N sample
Epochs
Teacher
7
4
6
180
CompOFA
Compound
3/5
2,
3, 4,
25
Compound
3/5
2, 2,3,4 4
3, 3,4,6
4
125
Warmup
3/5
4
6
fOFA
In-place Distillation
3/5
4
6
180
(simultaneous)
3/5
2,
3, 4
3,
4, 6
2,3
180
smallest model Miin adversely affects overall accuracy. address this issue, we develop new upper-attentive sampling method, which always samples the full-sized sub-network in each iteration, and n random sub-networks. The loss function upper-attentive sampling is:
min (C(Miaes YD) + Σ C (Wrand(a.)) Nmax))
(6)
Where Wmax represents the largest sub-network. During training, the largest sub-network is max- imized only with respect to the ground truth labels, while the additional sub-networks are trained with respect to the output of the largest sub-network.
A schematic of upper attentive sampling is shown in Figure With upper attentive sampling, we may either choose to replace the smallest model with randomly selected sub-network, or choose to remove it entirely, effectively reducing the number of sampled networks by 1 when compared to CompOFA or BigNAS. Intuitively, removing the smallest network will result in faster training than replacing it, but may have a negative impact on accuracy. We study both the removing and replacing options in Section
3.5 WARMUP PHASE
Because the full-sized sub-network. Muax is a soft target for the other sub-networks, we find training benefits from a warmup phase so that the initial target for the smaller sub-networks is not random at the start. We find that first training the largest sub-network for few epochs provides good results and is still much faster than training a teacher from scratch for 180 epochs.
3.6 SUB-NETWORK SELECTION PROCEDURE
Again, following Sahni al. 2020) and (Cai et al. 2020), we evolutionary search (Real al. 2019) to retrieve specific sub- networks that are optimized for a given hardware target. This search finds trained networks that maximize accuracy subject to the target latency FLOP constraint. For hardware targets such as the Samsung Note10, latency can be estimated using look-up table
estimator from (Cai et al. 2020).
4 RESULTS
4.1 TRAINING SETUP
We performed our experiments on an NVIDIA DGX-A100 server with 8 GPUs. Experiments were run in version 21.03 of the NVIDIA NGC pytorch containe which includes Python 3.8, pytorch 1.9.0, and NVIDIA CUDA 11.2. Horovod version 0.13.3 was used for multi-GPU training.
https://docs. nvidia a https//iww.....coooooooooooonnnn ease- ease-notes/
21 11-03.htm html

---

Under review as a conference paper at ICLR 2022
The training schedule for fOFA is listed in Table | and comparisons with CompOFA and fOFA shown in Figure [ CompOFA requires 330 total epochs, of which over half (180) are dedicated to training the full-size teacher model. For fOFA, we require 185 total epochs. Of these, only 5 epochs are used for warming up the full-size model advance of in-place distillation, wherein the supernet and the randomly selected networks trained simultaneously.
Directly following CompOFA (Sahni et al. 2020 we perform the model search over the Mo- bileNetV3 space with expansion ratio 1. We use 8 GPUs, batch size of 256 per GPU, and learning rate of 0.325. For fair comparison, all other hyper-parameters are set to the same values as OFA and CompOFA, including a cosine learning rate schedule, momentum of 0.9, batch-norm momentum of 0.1, weight decay of 3e-5, label smoothing of 0.1, dropout rate of 0.1. Also, as fOFA trained from scratch instead of fine-tuning on the pre-trained teacher model, gradient clipping threshold of 1.0 is adapted to make the training stable, following bigNAS (Yu et al. 2020).
4.2 TRAINING RESULTS
Table 2: Mean Top-1 Accuracy on ImageNet
Method
Epochs
Training Cost (GPU h)
Mean Top-1 Accuracy
OFA
605
672 (1.0x)
75.5
CompOFA
330
336 (2.0x)
75.4
fOFA (n=3)
185
184 (3.7x)
75.5
IOFA (n=4)
185
216 (3.1x)
75.5
1OFA (n=4)
300
350 (1.9x)
75.6
Table 2 shows the average accuracy over the generated models. For 1OFA, = 3 means that the smallest model from the sandwich rule of (Yu et al.) 2020) has been removed, and we are training with the largest model and two randomly selected sub-networks. = 4 means that the smallest model has been replaced, and we are training with the largest model and three randomly selected nub-networks.
For CompOFA and fOFA, the mean top-1 accuracy is computed over all 243 models generated by the training process. Since OFA has extremely large number of sub-networks, average by selecting the same 243 models that are used in CompOFA and fOFA. We see that while CompOFA is 2.0x faster than OFA, with 185 epochs, fOFA is further 1.55x faster than CompOFA if we sample four sub-networks during training, and further 1.83x faster if we sample only three sub-neut sub-networks. In both cases, our accuracy is equal to OFA and 0.1 % greater than CompOFA. If we sample four sub-networks and extend our training time to approximately match the number of GPU-hours required for CompOFA, we generate an average accuracy 0.1 % greater than OFA.
calculate this
4.3 HARDWARE LATENCY
Table 3: Top-1 Accuracy on Lateney Constrained Models for Samsung Notel0
Method
Epochs
Latency
Constraint
15 ms
20 ms
25 ms
30 ms
OFA
605
71.93
73.95
74.94
75.41
compOFA
330
72.08
73.94
74.94
75.58
fOFA (n=3)
185
72.02
74.06
75.06
75.60
fOFA (n=4)
185
71.74
73.78
74.77
75.47
Table 3 shows the performance of the once-for-all methods for the hardware deployment scenario of a Samsung N0.e10. We used the latency estimator for the Note 10 CPU provided Cai et al. (2020). The latency thresholds of 15, 20, 25, and 30 milliscconds are selected to match the latency targets
II Cai et al. 2020 results are reported with expansion ratio of 1.2, so the numbers reported here are identical.
7

---

Under review as a conference paper at ICLR 2022
used in (ahni et al. (2020). At latencies larger than 20 ms, fOFA with = 3 is more accurate than other methods while also having the smallest training cost. At 15 ms, fOFA is slightly less accurate than compOFA, but is still x faster in training time.
Table 4: Top-1
Accuracy on Latency Constrained Models for GPU platforms.
Method
Epochs
A100 Latency Constraint
Pascal Latency Constraint
4 ms
ms
ms
10 ms
15 ms
25 ms
35 ms
40 ms
OFA
605
72.55
76.12
77.02
77.11
73.86
76.11
77.00
77.10
compOFA
330
72.78
76.26
77.40
77.49
72.84
76.27
77.45
77.52
fOFA (n=3)
185
73.63
76.08
77.39
77.55
74.46
76.28
77.37
77.46
fOFA (n=4)
185
73.75
76.33
77.21
77.28
74.51
76.29
77.15
77.36
Table 4 (left) shows the performance of the methods NVIDIA A100 GPU. In this setting, the latency measured directly using the CompOFA code fOFA with n = 4 has the highest accuracy the strictest latency constraints (4 ms and 6 ms), while fOFA with = 3 performs best 10 ms latency. fOFA (n=3) and compOFA have nearly identical accuracy at ms. In Table [ (right), we also show the performance of the methods for earlier-gemerationn GPU (Nvidia Pascal) with more relaxed latency constrains. The results show similar trends to the A 100 GPU where fOFA is superior the strictest constraints and has similar accuracy CompOFA high constraints.
Table 5: Top-1 Accuracy Latency Constrained Models for AMD EPYC 7763 CPU Method
Epochs
Latency
Constraint
22 ms
25 ms
28 ms
31 ms
OFA
605
74.34
74.92
74.92
76.05
compOFA
330
72.77
74.65
75.17
76.35
IOFA (n=3)
185
73.55
75.01
75.38
75.78
1OFA (n=4)
185
73.44
74.69
75.59
75.85
Table shows the performance of the methods on CPU. Again, latency is measure directly using the CompOFA code. In this setting, we find that fOFA achieves the highest accuracy at medium constraints (25 ms and 28 ms), while compOFA achieves the best accuracy 31ms, and OFA as 22 ms.
5 DISCUSSION
Figure ssows the trade-off between model accuracy and number of floating-point operations for compOFA and IOFA with n = 4. On average, 1OFA is ~0.1% more accurate that CompOFA, as listed in Table 2 and achieves greater accuracy on models with lower FLOP counts, agrecing Despite replacing the sandwich rule with upper-attentive sampling, the smallest model in the search space has 0.9% greater accuracy in fOFA.
with results in Tables 3
We also ran iOFA using the sandwich rule (Yu & Huang (2019) on the same hyperparameter space. With the sandwich rule, the average accuracy over the search space was 0.3% lower than with upper- attentive sampling. Furthermore, the decrease in accuracy was greater on models with higher FLOP counts, and less on models with lower FLOP counts. To explain these observations, we note that the upper bound of the CompOFA search space is significant lower than that the MobileNetv2 search space from Yu et (202)) In BigNAS, the largest model in the search space required 1.8 GFLOPs while the largest output model, BigNASModel-Xtt -L, required only 1.04 GFLOPs. In contrast, the largest model in the CompOFA search space uses 447 MFLOPs and the models we selected for GPU deployment in T appproach this upper limit.
In our experiments, we set the convolution kernel sizes for each block to those used in MobileNetV3 (Howard et al.] (2019) for all models in the search space, including the teacher. We also experi- mented increasing the size of the teacher model, using K for each block in the the teacher, and
https:://gi
gat.ch 3m) httpt/eehhssssmm/ooooooa/aaa/aae httpa//m/co/////////aaaae
8

---

Under review as a conference paper at ICLR 2022
78
Top-1 accuracy
75
74
73
72
fOFA (n-4)
compOFA
71
150
200
250
300
350
400
450
MFLOPS
Figure 4: Comparison of floating point operation accuracy for each of the 243 models in the compOFA searchspace, for compOFA (orange) and fOFA with n = (blue).
found that this results in the average accuracy decreasing to 74.8%. From this result, propose that an overly large teacher model, while providing a higher upper bound on accuracy, may not be effective for training smaller submodels, and that when the teacher model is closer in size the submodels, upper-attentive sampling sufficient to achieve good accuracy throughout the search space.
When upper-attentive sampling is used in combination with in-place distillation, the warm-up phase essential that the initial target for sub-model training better than random. After five epochs of warm-up. the teacher model has an accuracy of 47.42% in our experiments, providing a reasonable starting point for training. OFA and CompOFA, this warmup phase not needed because the teacher model is already fully trained.
CONCLUSION
In this work, we introduce fast once-for-all (fOFA) and demonstrate how this methodology can reduce expensive training cost for neural architecture search below 200 GPU-hours. This is done by combining the approaches of CompOFA (Sahni et al. 2020) and BigNAS (Yu et al. 2020) by using a reduced state space to only consider models close to the Pareto-optimum, as in CompOFA, and using in-place distillation, training from OFA. While these methods work well together, to achieve optimal performance it was necessary to develop methods such as upper attentive sampling and apply warmup phase to achieve optimal results. Our results show that we can achieve the same accuracy as OFA with a speed-up of 3.1 x-1x7x77x, xx, and similar accuracy CompOFA with a speed-up of x.5x
in BigNAS, to eliminate the requirement of expensive teacher
As we continue this work, we look to further investigate the relationships between training heuristics and the choice of the network search space, that for their performance and determine which methods lead to most efficient training. We also hope further investigate multiple network search spaces develop new methods for finding training schedules and network sampling approaches that optimize accuracy for both benchmark tasks such as Imagenet classification and novel tasks such as segmentation, detection, and transfer learning.
better understand the theoretical reasons
REFERENCES
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Understand- ing and simplifying one-shot architecture search. In International Conference on Machine Learn ing, pp. 550-559, 2018.
Maxim Berman, Leonid Pishchulin, Ning Xu, Matthew B Blaschko, and Gérard Medioni. Aows: Adaptive and optimal network width search with lateney constraints. In Proceedings of the

---

Under review as a conference paper at ICLR 2022
IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 11217-11226. 2020.
Andrew Brock, Theodore Lim, James Ritchie, and Nick Weston. Smash: one-shot model archi- tecture search through hypernetworks. arXiv preprint arXiv: wwxivv/708.053444 2017.
Han Cai, Chuang Gan, Tiazhe Wang, Zhekai Zhang. and Song Han. Once for all: Train one network and specialize it for efficient deployment. In International Conference Learning Representations, 2020. URL httttttts///aattt http://wwxwwooog..p..../ owg/pdf 9t9p:///////99..0....... pdf
Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 255.255. leee, 2009.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. arXiv preprint arXiv:1904.00420, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint 1XXiv:5503.025:50 2015.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE International Conference on Computer Vision, pp. 1314-1324, 2019.
Eunwoo Kim, Chanho Ahn, and Songhwai Oh. Nestednet: Learning nested sparse structures in deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8669-8678, 2018.
Changlin Li, Guangrun Wang. Bing Wang. Xiaodan Liang, Zhihui Li, and Xiaojun Chang. Dynamic slimmable network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6607-8617, 2021.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint 1rXv::1806.09055 2018.
Hieu Pham, Melody Y Guan, Barret Zoph, Quoe V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.03268. 2018.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10428-10436, 2020.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoe V Le, and Alexey Kurakin. Large-scand scale evolution of image classifiers. In International Conference on Machine Learning, pp. 2902-2911. 1. PMLR, 2017.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedingy of the aaui conference artificial intelligence, volume 33, pp. 4789-4789, 2019.
Manas Sahni, Shreya Varshini, Alind Khare, and Alexey Tumanov. Compofa-compound once- for-all networks for faster multi multi-platform deployment. International Conference on Learning Representations, 2020.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint 10f:::90000224333 2019.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, pp. 6105-6114 14, 2019.
Dilin Wang, Meng Li. Chengyue Gong, and Vikas Chandra. Attentivenas: Improving neural archi- tecture search via attentive sampling. arXiv preprint arXiv:2011.090) 2020.
10

---

Under review as a conference paper at ICLR 2022
Taojiannan Yang, Sijie Zhu, Chen Chen, Shen Yan, Mi Zhang, and Andrew Willis. Mutualnet: Adaptive convnet via mutual learning from network width and resolution. In European conference computer vision, pp. 299-315. Springer, 2020.
Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1803-1811,
2019.
Jiahui Yu, Linjie Yang. Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks.
arXiv preprint arXiv: 1812.008812 2018.
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural archi- tecture search with big single-stage models. arXiv preprint arXiv:2003.11142 2020.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint arXiv:1611.01578, 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.


Reviews:
[{"review_id": "x8yexWIWoz2", "reviewer": "Reviewer_vCfG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The proposed method improved the training speed without loss of accuracy.\n2. The paper is easy to follow.\n\nWeakness:\n1. The proposed method is of very limited novelty and technical contribution. Looking at the contribution part, (1) in-place distillation is used in Slimmable Network. (2) The upper-attentive sampling is basically Sand-wich rule in Slimmable Network, the only difference is that the author doesn’t sample the smallest sub-network. (3) The warm-up training is also a very incremental and commonly-used technique. The proposed method is basically a combination of previous techniques.\n2. The results seem to be not consistent with that reported in the paper. For example, in Table 3, the performance of OFA seems to be lower than that reported in the original paper.\n3. Some missing related works: \\\n[1] NestedNet: Learning Nested Sparse Structures in Deep Neural Networks \\\n[2] MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution \\\n[3] Dynamic Slimmable Network \n", "summary_of_the_paper": "This paper proposed a method to train a once-for-all network, where one network can run at different resource constraints. The method is based on previous methods, and the author further improved the training speed by around 1.5x - 1.8x without loss of performance. The method is evaluated on ImageNet classification.", "summary_of_the_review": "Overall, I think the paper has limited technical contribution, and the training speedup is not surprising (less than 2x). If it is a combination of techniques but significantly improved the training speed (say 5x-10x), which makes such network training easily-doable for everyone, then I would say it is a large contribution. Currently, I think the author need to further improve their method.\n\n======== Post Rebuttal ========\n\nThank the author for the response. My concern about the novelty and significance of the method and results still remain. I agree that combining existing techniques may need more tuning. But this is hard to be considered as enough contribution, especially given that the performance is not significantly improved. Therefore, I will keep my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hwutAuaS7k", "reviewer": "Reviewer_cquh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this manuscript, the authors propose a new framework for training the once-for-all (OFA) networks. This framework uses an in-place distillation schema to replace the teacher training in the former OFA training fashion and makes the overall training time faster. The paper is overall well-written with good performance.\n\nMy concerns are as follows.\n\n- The novelty for this paper is limited: this paper claims four contributions they made, but the idea for training without the teacher network borrowed from the in-place distillation paper without much modification. The upper-attentive schema simply samples the biggest network in each phase, which can be regarded as \"merging“ the training for \"teacher network\" and \"elastic network\" in one phase, but the author does not show the performance boost for the schema (compared to without using upper-attentive).\n- The OFA aims to alleviate search cost across different hardware platforms, but the comparison in the experiment only include little platform. The author should prepare more experiments across different platforms (mobile, CPU, GPU, FPGA, etc...).\n- It is not clear how each component in this paper contributes to the final performance for the reduction of training time, the author should do an ablation study to justify this.\n\nI will raise my vote if the concerns are addressed.", "summary_of_the_paper": "In this manuscript, the authors propose a new framework for training the once-for-all (OFA) networks. This framework uses an in-place distillation schema to replace the teacher training in the former OFA training fashion and makes the overall training time faster. The paper is overall well-written with good performance.", "summary_of_the_review": "The paper is overall well-written, but the novelty for this paper is kind of limited. The author should prepare more experiments to justify the performance and address the technical contribution for this paper.\n\nI will raise my vote if the concerns are addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XmFjijGQ2dx", "reviewer": "Reviewer_RKoZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- Overall this paper is well written and easy to follow.\n- The reduction in training time is indeed decent\n\nWeakness:\n- Limited novelty.\n- Lacking the deeper explanation of the improved results.\n", "summary_of_the_paper": "This work proposes new methods of Once-for-all(OFA) network training methods, which significantly reduce the training time as compared to the existing OFA methods. Specifically, this work:\n- Eliminate the need of pretraining the teacher network with in-place distillation\n- Develop an upper-attentive sampling method which always sample the full-sized sub-net at each iteration to help co-train the rest sub-networks\n- Use an upper-attentive warmup technique which trains only the full sized sub-net for a few epochs before co-training to further improve the performance\n\nWith these optimizations, this work can decrease the number of sampled sub-networks in each iteration of training, further reducing the total training cost.\n", "summary_of_the_review": "- **Limited novelty**.\nIt seems to me the two major contributions in-place distillation and upper-attentive sampling are already proposed by the existing works. It is a little unclear what further modifications the authors did other than stitching these two techniques together. Would be great if the authors can give some stress here considering the improvements in the results.\n\n- **Lacking the deeper explanation of the improved results.** \nIn the results part, it seems to me that the proposed methods just magically improve as compared to on the baselines. Not clear the underlying reasons and insights for them. Would be nice for the authors to point out as I might miss them.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Pj72hxNKoiH", "reviewer": "Reviewer_vTmv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The training cost saving is clear and significant while maintaining the same level of accuracy as OFA and CompOFA.\n2. The proposed method can potentially benefit real-world deep learning applications. \n\nWeaknesses:\n[major]\n1. Limited novelty. Although this paper presents good results, the novelty of the proposed method is a bit weak. For example, in-place distillation is not new. Upper-attentive sampling and upper-attentive warmup are more like engineering tricks. \n2. This paper only presents results on one design space (mobilenetv3). But, in practice, MobileNetV3 is not supported or not efficient in many cases (e.g., on FPGA, GPU, etc). Showing the generalizability of the proposed method on other design spaces (e.g., MobileNetV2) is important. \n\n[minor]\n1. On ImageNet, it seems that only supporting 243 sub-networks is sufficient for NAS. But it is unclear whether this strategy also works under the transfer learning setting. \n", "summary_of_the_paper": "This manuscript aims to reduce the training cost of once-for-all networks. The proposed method is built upon CompOFA that reduces the number of sub-networks within the OFA network to 243 by only considering networks whose dimensions are coupled. This paper introduces several techniques to further reduce the training cost, including in-place distillation, upper-attentive sampling, and upper-attentive warmup.", "summary_of_the_review": "This paper presents several simple methods to reduce the training cost of once-for-all networks, showing clear and significant training cost saving without losing accuracy. But, I find the novelty of the proposed method is a bit weak, and the generalizability of the method on other design spaces and transfer learning is unclear. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper exposes a method to reduce the training cost of once-for-all networks.
Overall this paper is well written and easy to follow, and the experimental section shows a clear reduction of training time on the examples used.
However, the reviewers point out that the experimental section could benefit from adding more design spaces, and have a better explanation of the results. More importantly, three out of four reviewers agree that the novelty of this work is too low for the submission to be accepted, with the fourth one only giving a score of 6 (and also noting the lack of novelty). I therefore recommend reject for this paper.
Decision: Reject