CompOFA – Compound Once-For-All Networks for Faster Multi-Platform Deployment
Authors: Manas Sahni
Shreya Varshini
Alind Khare
Alexey Tumanov
Venue: ICLR 2021 Poster (2021)
Keywords: Efficient Deep Learning
Latency-aware Neural Architecture Search
AutoML

Abstract: The emergence of CNNs in mainstream deployment has necessitated methods to design and train efficient architectures tailored to maximize the accuracy under diverse hardware and latency constraints. To scale these resource-intensive tasks with an increasing number of deployment targets, Once-For-All (OFA) proposed an approach to jointly train several models at once with a constant training cost. However, this cost remains as high as 40-50 GPU days and also suffers from a combinatorial explosion of sub-optimal model configurations. We seek to reduce this search space -- and hence the training budget -- by constraining search to models close to the accuracy-latency Pareto frontier. We incorporate insights of compound relationships between model dimensions to build CompOFA, a design space smaller by several orders of magnitude.  Through experiments on ImageNet, we demonstrate that even with simple heuristics we can achieve a 2x reduction in training time and 216x speedup in model search/extraction time compared to the state of the art, without loss of Pareto optimality! We also show that this smaller design space is dense enough to support equally accurate models for a similar diversity of hardware and latency targets, while also reducing the complexity of the training and subsequent extraction algorithms. Our source code is available at https://github.com/gatech-sysml/CompOFA

Published as a conference paper at ICLR 2021
COMPOFA: COMPOUND ONCE-FOR-ALL NETWORKS FOR FASTER
DEPLOYMENT
Manas Sahni, Shreya Varshini, Alind Khare, Alexey Tumanov Georgia Institute of Technology (sahnimanas, shreyavarshini, alindkhare, (tumanov@@gatechi
ABSTRACT
The emergence of CNNs in mainstream deployment has necessitated methods design and train efficient architectures tailored to maximize the accuracy under diverse hardware & latency constraints. To scale these resource-intensive tasks with an increasing number of deployment targets, Once-For-All (OFA) proposed an approach to jointly train several models at once with constant training cost. However, this cost remains as high as 40-50 GPU days and also suffers from a combinatorial explosion of sub-optimal model configurations. We seek to re- duce this search space and hence the training budget - by constraining search to models close to the accuracy-latency Pareto frontier. We incorporate insights of compound relationships between model dimensions to build CompOFA, design space smaller by several orders of magnitude. Through experiments on ImageNet, we demonstrate that even with simple heuristics training time") and 216x speedup in model search/cxtraction time compared to the state of the art, without loss of Pareto optimality! We also show that this smaller design space is dense enough to support equally accurate models for a similar di- versity of hardware and latency targets, while also reducing the complexity of the training and subsequent extraction algorithms
achieve 2x reduction in
INTRODUCTION
CNNs are emerging in mainstream deployment across diverse hardware platforms, latency require- ments, and/or workload characteristics. The available processing power, memory, and latency re- quirements may vary vastly across deployment platforms say, from server-grade GPUs to low- power embedded devices, cycles of high or low workload, etc.
Since model accuracies tend to increase with computational budget, it becomes vital to build models tailored to each deployment scenario, maximizing accuracy constrained by the desired model infer- ence latency. These efficient models lie close the Pareto-frontier of the accuracy-latency trade- off. Building such models (either manually or by searching) and then training them are resource- intensive tasks requiring massive computational resources, expertise in both ML and underlying systems, time, dollar cost, and CO2 emissions every time they are performed. Repeating such inten- sive processes for each deployment target is prohibitively expensive w.r.t. multiple metrics of cost and this does not scale.
Once-For-All (OFA) (Cai al. 2020) proposed to address this challenge by decoupling the search and training phases through a novel progressive shrinking algorithm. OFA builds a family of 19 models of varying depth, width, kernel size, and image resolution. These models are jointly trained in single-shot via sharing of their intersecting weights. Once trained, search techniques can extract specialized sub-networks that meet specific deployment targets a task that can then be indepen- dently repeated on the same trained family.
This massive search space leads to a training cost that remains prohibitively expensive. Though the cost be amortized over a number of deployment targets, it' still significant reaching 1200 GPU hours for OFA. The search space arises from training every possible model combination, and many
and, therefore, dollar cost and CO2 emissions
"Our source code available at httph:ttp////. //gi thub. http://ccoo/accc///oeeee sysm] /CompOF'

---

Published as a conference paper at ICLR 2021
(6) Conventional:
Design, train, deploy single nst per deployment larget
$$
Depth
Width (W)
Co-trained model
family
Train
R (2300 68)
(234)x
J.w. (3.6))
Progresstively Co-Train Subnets
Teinneine Family
(e) CompOFA:
Panto-uplimelly diversity
Match Pareto
optimality diversity the cost
Suboptiml models
Tmin Teacher
E Comporad (2.34) EA4GH Co-Tmin All Subnets
Co-Trained
Model Family
Figure/ 1: (a): Conventional methods require expensive designing & training per deployment plat- form, which is infeasible scale. (b): OFA co-trains a family of subnetworks of teacher supernet. However, combinatorial explosion of depth (D) and width (W) compels progressive, phased training requiring 1200 GPU hours. (c): CompOFA exploits the insight of compound couplings between D) & W to vastly simplify the search space while maintaining Pareto optimality. The smaller space can be trained in half the time without phases, and gives equally performant and diverse model families
of these models may lie well below the accuracy-latency Pareto frontier, as represented in Figure (b). This exhaustive approach misses opportunities for any accuracy- or latency-guided exploration in such a vast space, thus suffering a clear inefficiency. These sub-optimal models not only go un- utilized but also add training interference, which necessitates longer phased training to stabilize their simultaneous optimization. Finally, searching & extracting an optimal model from this space only be done via indirect estimators of their accuracy and latency, all model combinations cannot be enumerated.
On the other hand, we argue that such large search space in unnecessary for two reasons. First, common practices as well as empirical studies (Tan & Le 2019: Radosavovic et al. 2020) have shown that model dimensions such depth, width, and resolution not orthogonal models that follow some compound couplings between these dimensions produce a better accuracy-latena trade-off than those with unconstrained settings. Informally, increasing model capacity along one dimension (say, depth) is helped by an accompanying increase along another dimension (say, width). Secondly, a much coarser latency granularity (order of ms) is sufficient for practical systems de- ployment.
In this work we propose CompOFA - a model design space leveraging compound couplings between model dimensions, and demonstrate the following:
1. Utilizing the insight of compound coupling, we show that simple, easy to implement heuris- ties can capture models close to the Pareto frontier (depicted in Fi.. This enables us to reduce OFA's 10 models to just 243 CompOFA, and still train a model family with equally good accuracy-latency tradeoff.
2. We show that this tractable design space directly reduces interference in training, which allows to reduce training duration and cost by 2x.
3. Once trained, CompOFA's simplicity avails itself to easier extraction that's faster by 216x.
4. Despite the size reduction, we show that the latency granularity is sufficient to cover the same range and diversity of hardware targets as OFA.
5. Finally, the generality of CompOFA's insights is validated by training it another base architecture, achieving similar gains.
2 RELATED WORK
Efficient neural network design has been an active area of research due to the high computational complexity of CNNs. NAS is increasingly used to guide replace previously manual design pro-
2

---

Published as a conference paper at ICLR 2021
cesses. Early NAS techniques (Zoph al. 2018: Zoph & Le 2016) sampled several architectures
and trained them from scratch every time, making them extremely compute-hungyy The technique of weight-sharing has emerged as one way to address these inefficiencies (Berman al. 2020)
Bender et al., 2018 [Brock et al. 2017 Guo et al., 2019 Liu et al. 2018 Pham et al. (018). These methods slice candidate sub-networks from larger super--nttwork, thereby sharing weights between them. Simultancously, latency-guided NAS methods (Tan et al. 2019 Cai et al. 22018 88 Berrman et al.)
2020 Stamoulis et al. 2019; Wu et al. 2019) have sought incorporate model complexity into their search to find efficient models for given target latency on given hardware target.
Nevertheless, these methods yield a single model per rum both search and training must be repeated for new deployment targets. With compute costs reaching as high as O(10*) GPU hours. this linear scaling infeasible for an ever-growing need for multi-platform, multi-latency deployment. Once- For-All (OFA) (Cai et al., 2020) proposed to reduce this cost by using weight-sharing for large model family that collectively supports diverse range of latencies. They perform one-time training of (112 sub-networks, which can then be independently searched to support a given deployment target later, thus amortizing the training cost. However, this cost is still prohibitively expensive, reaching 1200 GPU hours. The unnecessarily large search space complicates both training and searching, stemming from an uninhibited combinatorial explosion of model configurations.
On the other hand, empirical studies on neural network design spaces (Tan & Le 2019 Radosavovic ett al. 2020) have recently shown that model dimensions (e.g., depth, width, resolution) are not independent underlying relations between them can be used obtain optimal accuracy- -attncyy trade-off. In other words, the number of degrees of freedom in the architecture search space can be reduced without loss of Pareto optimality. This insight forms the basis of our work constrain the design space of OFA networks while maintaining the same quality (w.r.t. accuracy) as well as diversity of models with reduced train and search costs.
Dynamic Neural Networks with weight-sharing across models of varying latencies have been ex- plored before (Yu et al., 2018: Yu & Huang, 2019) but with much fewer models (e.g. Yu et al. 2018) and few dimensions (e.g. only width) which make for much sparser and narrower support for diverse latency targets. We explore middle ground between these works and[Cai et al. (2020) to build design spaces that are tractable yet sufficiently diverse to support many latency targets and varying in multiple model dimensions.
Yu et al. 2020) proposed replacing OFA's multi-stage training by a single-stage one, challenging the usual practive of progressive training in one-shot NAS. However, its training cost remains high at over 2300 TPU hours for ((10 models. Contemporary to our work, Wang et al. (020) point out similar wasted computation on sub-optimal models in one-shot NAS, and use attention mechanisms to push the Pareto front. Our approach instead focuses on achieving the same Pareto frontier as OFA with a smaller budget. We also emphasize architectural insights to show that an intractably large cardinality is not necessary in the first place simple heuristics identify good models while enabling a host of other cost savings.
3 MOTIVATION
3.1 DESIGN SPACE PARAMETRIZATION
Consider a network architecture A composed of m aicco-architectural blocks, B1, Bz,..., Bm- Each block is parametrized by its depth, per-layer width, and per-layer kernel size as (dd,, W,, K.). Here d; denotes the number of layers (depth) in the block, and W, and K, are lists denoting the width & kernel sizes of each of these d, layers.
Aone-FoorAll(Cai et al.) 2020) builds family of networks N1, of varying accuracies and latencies. The weights of common layers and channels of a block B, are shared across all the net- works. The "block" used in OFA is the Inverted Residual from MobileNetV3 (Howard et al.) 2019) and hence "width" here refers to the chanmel expansion ratio. d,, Wij, kij are sampled independently from sets D = [2, 3, 4, W [3, 4, 6], K [3, 5, 7] respectively.
In OFA, each of these dimensions is treated as orthogonal. Hence, the resulting number of possible networks is enormous, with O11219) models for 5 Caoc et al. 2020).
3

---

Published as a conference paper at ICLR 2021
Accurary Healmap
Latency Healmap
71:776
73:512
74.700
6.756
7,900
12:127
Deahh
25
74.588
75. 890
76.760
11.203
15.338
23.436
23
76. 660
157
21:044
11
71
12
Whtth
Wisth
Figure 2: Accuracy and latency heatmaps for varying uniform depth and expansion ratios and fixed kernel size=5, as measured in MobileNetV3 architecture. Latency is measured in ms on NVIDIA RTX 2080 Ti GPU with BS=64.
3.2 COMPOUND RELATION OF MODEL DIMENSIONS
The combinatorial explosion from just independent model dimensions of D, W, K yields model family with an enormous number of models. While the aim for jointly training "every possible model" in this large design space is to support diverse range of hardware platforms, note the following concerns that arise from this:
1. Model dimensions are not orthogonal Scaling model dimensions like depth, width, and resolution to higher FLOP regimes is a common practice and Tan & Le (2019) systematically showed that compound relations ex- ist between these dimensions for achieving an optimal accuracy -latencyy trade-off. In partic- ular, increasing model dimensions in accordance with compound scaling rule gives better results than doing so independently. RRadosvvovic et al. (2020) elevated this concept to the model population level and found that a quantized linear relation between model depth and width yielded design spaces with a better concentration of well-performing models. These works solidified the common practice of jointly increasing model depth and width. Yet, all these other sub-optimal models are still included in the search space when all dimensions are sampled independently.
2. Large model families complicate training & searching
The interference between OFA's sub-networks complicates their simultaneous optimization and necessitates techniques like progressive shrinking (Cai et al. 2020), increasing training time. Further, extracting models for a desired target cannot rely on simple enumeration and instead needs rely predicting model accuracy/latency. This requires additional training for these accuracy predictors specific to the trained model family and collecting latency look-up tables (Cai et al.) 2018). With a more tractable cardinality, could rely on more standard, faster approaches during training and achieve "off-the-shelf" usability during search.
3. Hardware latency differences below a certain granularity are noisy
Finally, the difference between unique architectures' accuracy or latency needs be distin- guishable during search. The original OFA design space covers FL.OP range of 120-560 MFLOPs and within this range there exist ( architecture choices. Even if these models were uniformly distributed into buckets of FLOP, we would still have O(10"1) models in each FLOP target. Note that these results do not account for variable resolution, which would otherwise further multiply the number of choices at each FLOP target. On any hardware, inference time below a certain threshold is expected to be indistinguishable from noise. This threshold can vary depending application context hardware (e.g. it can be upto 1-5 ms in ML serving scenarios). Irrespective of the threshold, having these many models with such fine-grained differences in their compute requirement is well below the minimum error resolution at which they can be meaningfully compared. This motivates to consider that a design space that is sparser by several orders of magnitude might still support the same density and range of deployment targets.
Figure 2 shows heatmaps comparing accuracy and latency of OFA sub-networks with uniform depth and width for a fixed kernel size. A closer look into these heatmaps, depicts a monotonically in- creasing gradient along the diagonal of each of these dimensional heatmaps. This observation
4

---

Published as a conference paper at ICLR 2021
showcases that when the configurations increased along both the depth and width dimensions retrieve a class of sub-networks that achieve a better accuracy-lateney trade-off as opposed to those formulated through a single dimensional configuration change. Thereby, emphasizing the existence of a coefficient that dictates the relationship between depth and width configurations of a model. This suggests that by quantitatively conforming to this coupling between the model dimensions, can aim toward reducing the network design space of an architecture without sacrificing its accu- racy, Subsequently, this further implies that OFA is unnecessarily training a lot more models than required and we additionally show that OFA's model distribution suffers from high variance as well (Appendix(A.) (A.I). This leads us toward solution that leverages these insights to prune out redundant and ineffective sub-networks otherwise generated by the unconstrained OFA architecture.
4 COMPOUND OFA
4.1 COUPLING HEURISTIC
Motivated by the above observations, we constrain the network design space using a simple heuristic: depth and width dimensions should increase decrease together, not independently. Specifically, in each block, whenever we sample the 7th largest depth d, E D, we correspondingly sample the 7th largest width wa W for all dę layers in the block. For instance, with D (2 4] and W = [3. 4, 6). each block can have either two layers of channel expansion ratio 3. or three layers of ratio 4, or four layers of ratio 6. Once this modified search space is defined by the heuristic, the method of extracting given model configuration remains the same we slice out sub-network of the specified dimensions from the largest network, thus sharing common weights between all sub-networks.
This significantly reduces the degrees of freedom the design space. An additional consequence of this is that all layers within a block now have the same width, further reducing the architecture-cvt Nevertheless, we will show in Section 5 that the design space is still diverse and dense enough to provide models for different deployment requirements.
We fix the kernel size by default, but for fair comparison we also show a variant with Clastic-Kernel create simplified search space with kernel sizes fixed to either 3 or 5 in
With fixed-kernel, each block, to match the original setting used in MobileNetV3. Thereby in this design space, depth (or width) alone can fully specify block's is configuration. For 5 blocks, this yields family of 3 = 243 models. In the elastic-kernel design space, we expand the kernel size as done in OFA, sampled from K [3. 5, 7] per layer. This results in a family of (32 + 3 + 3+)5 se 10t0 models. Unless otherwise specified, we use "CompOFA" to refer to the fixed-kernel design space.
As we show in the following section, the source of our training speedup the reduction in cardinality of the search space. The input resolution to the model affects the model inference time but does not affect the number of unique trainable architectures. Hence, we keep the resolution elastic, which allows using one architecture to support multiple latencies for "free", without increasing our search space cardinality (and soon, training budget).
4.2 TRAINING SPEEDUP
Cai et al. (2020) proposed "progressive shrinking" approach to train the OFA sub-networks in a phased manner, starting with the largest model and then progressively including smaller sub- networks in training. At each batch, diffeernnt sub-network is sampled for forward/backwart pass. If the expected time to complete one epoch in phase p is E(tp) and the training is run for ep number of epochs, then the total time to train model family becomes T Tjmiiy Epe Phases Cp x B(tp).
Our goal is to reduce Tammity while achieving the same level of accuracy-latenee trade-off and supporting the same range of deployment scenarios. Since we wish to train models of similar latency targets, we try to keep (t,,)) unchanged and target reduction in the number of phases reduce Tfamily-
While the number of models does not explicitly factor in T family. note that it implicitly affects the count & duration of training phases progressive shrinking was required in the first place due to interference between large number of models. The maximum number of simultaneously trainable
5

---

Published as a conference paper at ICLR 2021
Table 1: Training schedule, duration, and GPU hour comparisons for OFA and CompOFA. Com- pOFA reduces the training time of OFA by 50% with fixed kernel space. The columns K, D, W represent the sets of possible model dimensions (as in Section [.IIp While CompOFA allows all sub-networks to be trained after the teacher network, OFA progresses to the full search space with multiple phases of similar duration. See Appendix A.3 for CompOFA with Elastic Kernel
(a) Once-For-All Cai et al
2020 (Elastic Kerne))
Phase
D
N Naampic
Epochs
Wall Time
GPU Hours
Teacher
7
4
180
28h 45m
172h 30m
Elastic Kernel
3.5.7 7
125
26h 51m
16th 06m
Elastic Depth-1
3.5.7
3.
25
7h 46m
46h 36m
Elastic Depth-2
3.5.7
2,3.4
6
2
125
38h 32m
23th 12m
Elastic Width-1
3.
2,
4,
25
10h 06m
60h 36m
Elastic Width-2
3,5,7
2,
3,
125
51h 03m
306h 18m
Total
605
163h 03m
978h 18m
(b) CompOFA (Fixed Kemel)
Phase
K
D
W
N Nammpe
Epochs
Wall Time
GPU Hours
Teacher
7
180
28h 45m
172h 30m
Compound
2.
3.4. 6
25
8h 43m
52h 18m
Compound
2,3, 4
4,
125
44h 47m
268h 42m
Total
330
82h 15m
493h 30m
models in CompOFA's search space is smaller by 17 orders of magnitude. With this significantly smaller family size, we find that the interference between these models is reduced to the extent that we are now able remove progressive shrinking altogether, and instead achieve the same accuracy with trainable sub-networks included training (see Section 5).
In Section 5 we show that this allows to reduce the training time by 50%. Additionally, we show an ablation to confirm that this reduction in phases is only possible due to the reduced number of models in CompOFA, and not in the original large design space of OFA.
EXPERIMENTS
5.1 TRAINING SETUP
We train the full-sized CompOFA network (D = [4], W = [6]) on ImageNet (Deng et al. 2009) using the same base architecture of MobileNetV3 as OFA. We use batch size of [536 and learning rate of 1.95 to train on NVIDIA V100 GPUs. All other training hyperparameters are kept the same OFA for accurate comparison.
Next, for CompOFA with fixed kernel sizes, we unlock all permissible model configurations (D, W [22, 3), (3, 4), (4, 6)) in stage, as opposed to progressively adding more configu- rations. We train this for 25 epochs with an initial learning rate of 0.06, followed by 120 epochs with an initial learning rate of 0.18 obtain our final network. We sample Naample
models in each batch and aggregate their gradients before each optimizer step. The full network serves as teacher model for knowledge distillation (Hinton al. 2013). The comparison of the possible configurations and epoch durations for each training phase is summarized in Table
5.2 TRAINING TIME AND COST
By using just one stage of training after the teacher model (in the default fixed kernel setting), instead of multiple stages of the same duration, we able reduce the training cost of CompOFA by 50%. Our reproduction of OFA's training scheme on 6 GPUs takes 978 GPU hours while CompOFA- Kernel finishes in 493 GPU hours, shown in Table [ Table 2 shows the translation of this time reduction in terms of dollar cost and CO2 emissions. Note that the total duration of an epoch with the same design space configuration is comparable,
do not target speedup by training smaller

---

Published as a conference paper at ICLR 2021
Comparing OFA and CompOFA on training monetary cost, CO2 emission and average search duration. Monetary cost is based on hourly price of 1 NVIDIA V100 on Google Cloud. CO2 emission estimation is based on Strubell et al. (2019). Search time reported for an average over latency targets, without the use of latency estimators
Table 2:
Method
Cost
CO2 emission
Avg. Search Time
OFA
S2.4k
277 lbs
4.5 hours
CompOFA (Flastic Kemee) CompOFA (Fixed Kerme))
S1.7k
196 lbs
2.25 hours
$1.2k
138 lbs
75 seconds
27.4
77.4
15.5
77.2
75.8
74.0
76.6
11.2
76.4
Blsii Compona
OFA
Fland CompoTA
11.
Elasii CampoOt
Flaed CompORn
72.5
+
0TA
Esols Curpora CompotA
76.2
OFA
Flond Compont
Blasu Conponn
14
16
20
20
40
15
25
32
350
400
450
500
550
500
Xian CPU Labenry (m))
2360 Latenzy Jmo)
Sammung Ladenzy (mn)
FLOPS
Figure 3: CompOFA networks consistently achieve comparable and higher ImageNet accuracies for similar lateney and FLOP constraints CPU, GPU and mobile platforms within 1-5ms granulari- ties.
models. Instead, the speedup stems largely from halving the number of epochs, by way of reducing the phases for training. This reduction is turn possible due a smaller, constrained design space.
5.3 ACCURACY-LATENCY TRADE-OFF
After training CompOFA networks, we carried out evolutionary search (Real et al. 2019) re- trieve specialized sub-networks for diverse hardware deployment scenarios, as done in OFA. The evolutionary search fetches trained sub-networks that maximize accuracy subject a target effi ciency constraint (latency FLOPs). Similar to OFA, we use a 3-layer accuracy predictor common across all platforms. For latency, we use lookup-table based latency estimator for Samsung Note 10 CPU provided by Cai et al. (020). For other hardware platforms namely NVIDIA GeForce RTX 2080 GPU and the Intel(R) Xcon(R) Gold 6226 CPU we measured actual latency with batch size of 64 and respectively. The estimated accuracies of the best models returned by this search were then verified on the actual ImageNet validation set.
Figure B reports the performances of iccuracy-lateng, trade-off for CompOFA and OFA networks. We observe that the best model CompOFA for evaluated latency targets are at least as accurate, despite their significantly smaller family size and training budget. This result validates intuition behind the simple heuristic that aids creating smaller design space without losing out on Pareto- optimality or density the generated models.
30
40
Curmltatio Trabaaliyy
8.50
0.25
OFA
CompOFA
0.00
27
Cosst. (rror
Classtt.
Classit Error
Classi. Emro119)
Clessi: Emoo
Figure 4: CDF comparisons of 50 randomly sampled models sampled latency buckets of 5ms each. CompOFA has higher fraction of its population better than given classification error.
7

---

Published as a conference paper at ICLR 2021
ketwok Anpttaakee
ChA CompaRA
Imtgeneet Topt Acoomay
20
25
30
Lakeroy (m6)
Lakency ((ms))((ms)
Figure 5: Random sampling of 20 models in each latency bucket of 5ms for CompOFA & OFA actual latency (left) and bucketed latency (right). CompOFA yields higher average accuracy, i.c. as population it has a higher concentration of accurate models. The shaded regions in the left plot show the 95% confidence interval of the average accuracy.
OFA With
OFA W/
CompOFA With PS
CompOFA Weo PS
Cumulative Probat ity
0.8
19.3
ImageNed Top-1
700
76.2
0.2
CompOFA
OFA
25
26
28
062, We3
D-4, W-6
D-2, We3
D=4, W-8
Classif, Error (%6)
Comparing accuracies with and without progressive shrinking (PS). OFA suffers upto 3.7% accuracy drop when progressive shrinking is removed. For CompOFA, no such accuracy drop is seen when compared to the longer training with progressive shrinking. Right: CDF of model configurations common to OFA & CompOFA. CompOFA does not lose ac- curacy despite its smaller cardinality and smaller training budget.
Figure 6: Left and Center:
5.4 DESIGN SPACE COMPARISON
Apart from individual models, we further evaluate CompOFA at population level through a statis- tical sampling of design space as introduced by Radosavovic et al, (2020) 2019). Using Samsung Note 10 as sample hardware, we divide the supported range of latencies into buckets of 5ms each. For each latency bucket, we randomly sample 50 models OFA and CompOFA. Figure 4 plots the cumulative distribution function (CDF) of classification error for each of these latency buckets. In each bucket, the CDFs depict the fraction of models that exceed a given accuracy on the x-axis. CompOFA's CDFs lie above those of OFA, showing that CompOFA has higher fraction of accurate models. Next, Figure 5 we randomly sample 20 models in each bucket for each search space, and show that the average accuracy of randomly picked model from CompOFA is higher than that in OFA. The takeaway of both these evaluations that CompOFA yields a better concentration of accurate models and fewer sub-optimal models i.e. more accurate overall model population. Note that this is different from the best accuracy per latency target, where CompOFA matches OFA but with half the training budget.
5.5 EFFECT OF NUMBER PHASES
Cai et al. (2020) showed that OFA networks suffer top-1 accuracy drop of up to 3.7% when progressive shrinking is removed, showing its role in resolving interference. With 191 models interfering with each other for simultaneous optimization of their weights. phased approach is needed to stabilize training by gradually train model configurations. In CompOFA, our primary method of reducing the training time is by reducing the number of phases in training. We repeat
8

---

Published as a conference paper at ICLR 2021
similar ablation to compare the effect of progressive shrinking in OFA & CompOFA. For both design spaces, we train with & without progressive shrinking and compare the accuracies of common sub- networks. Figure 6 shows CompOFA achieves the same accuracy with and without progressive shrinking. We attribute this to lower interference between sub-networks in a design space smaller by 17 orders of magnitude.
Next, we build a CDF of model configurations in CompOFA and then extract the same sub-networks from OFA. Figure 6 shows that CompOFA still maintains slightly higher accuracy despite com- paring the same models in both spaces. Thus, models discarded in the process of constraining the design space do not add the accuracy of OFA or CompOFA the retained models can achieve the same (if not better) accuracy independent of any potential collaboration from these discarded models.
5.6 GENERALIZING TO OTHER ARCHITECTURES
The guiding principle behind our hcuristic namely, the optimality of coupling depth & width over the unconstrained search space is expected to apply to other architectures, datasets, and tasks. We demonstrate similar savings in training budget with our heuristic applied to a different base architecture.
We train OFA and CompOFA with the base architecture changed from MobileNetV3 to Proxyless-
NAS (Cai al. 2018). We keep the same depths and widths (D [2. 3, 4J. W = [3, 4, 6)) and build the search spaces of OFA & CompOFA using the cartesian product or the heuristic, respectively. In CompOFA, we fix the kernel size in all layers. Both networks use a width multiplier of 1.3.
The training hyperparameters, schedule, and search spaces are identical to those described in Section 5.1 and Table [ Hence, CompOFA again requires half the training epochs and thererfore half the training time, cost, and CO2, emissions. Appendix A.2 shows CDFs of models common to both PrA-PPoxylesSNAS & CompOFA-ProxylessNN ProxylessNAS, showing that the CompOFA has the same or marginally higher accuracies for the same model configurations despite half the training budget.
5.7 SEARCH TIME
The intractable cardinality of OFA necessitated the use of latency estimators proxy to real latency measurement. With a significantly smaller design space, this need is removed avoiding the time and effort of building these latency tables that eases "off-the-shelf" practical usability of CompOFA. We introduce a simple memoization technique during the evolutionary search to cache the measured latencies of model architectures and hence avoid its re-measurement, which is only practical with the smaller search space. Table 2 reports the average run time of NAS algorithms for a single latency target with this optimization added both CompOFA and OFA, reducing the search time to just 75 seconds for CompOFA Fixed-Kermel We also show that the evolutionary search converges in fewer iterations for CompOFA, in Appendix(A.4
6 CONCLUSION
To conclude, we have built upon Once-For-All (OFA) networks to propose CompOFA a design space for OFA networks using compound couplings between model dimensions, which speeds up the process of one-shot training and neural architecture search with hardware latency constraints.
We show that intractably large architectural search spaces are unnecessary for both accuracy and di- versity of models. By leveraging the insight of compound couplings we introduced a simple heuristic that vastly shrinked the search space without losing on Pareto optimality. Despite its sparsity, this smaller search space is sufficiently dense to support the same diversity and range of deployment targets.
This smaller cardinality reduces interference in weight-shared training which allows CompOFA to reach the same accuracy in half the training budget. Once trained, CompOFA's tractability lends it- self to easier extraction which is faster by 216x without the time or effort to build latency estimators. This improves the ease of "off-the-shell" usability of our method in real-world settings. Finally, we apply our heuristic on another model and show that CompOFA continues uncover similar gains.

---

Published as a conference paper at ICLR 2021
REFERENCES
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Understand- ing and simplifying one-shot architecture search. International Conference on Machine Learn- ing. pp. 550-559, 2018.
Maxim Berman, Leonid Pishchulin, Ning Xu, Matthew B Blaschko, and Gérard Medioni. Aows: Adaptive and optimal network width search with lateney constraints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11217-11226, 2020.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Smash: one-shot model archi-
tecture search through hypernetworks. arXiv preprint arXiv: (:Xiv:/700.00 708.05534 2017.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint (rXiv:/812.00333 2018.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once for all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations, 2020. URL htwp:: /atxx: wwwp:/wwwwwvggp////////
J. Deng, W. Dong, R. Socher, L.-J Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. CVPR09, 2009.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. arXiv preprint arXiv:1904.00420, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0255: 2015.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE International Conference on Computer Vision, pp. 1314-1324, 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv 2018.
preprint 1806vv/800..900
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.03266 2018.
Ilija Radosavovic, Justin Johnson, Saining Xie, Wan- Yen Lo. and Piotr Dollár, On network design spaces for visual recognition. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1882-1890, 2019.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10428-10436, 2020.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780-4789, 2019.
Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Single-path nas: Designing hardware-efficient convnets in less than 4 hours. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 481-497 Springer, 2019.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.022 02243, 2019.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference Machine Learning, pp. 6105-6114, 2019.
10

---

Published as a conference paper at ICLR 2021
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe Le. Mnasnet: Platform-awaee neural architecture search for mobile. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2820-28222 2019.
Dilin Wang, Meng Li, Chengyue Gong, and Vikas Chandra. Attentivenas: Improving neural archi- tecture search via attentive sampling. arXiv preprint arXiv:2011.09011 2020.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang. Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware- efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10734-10742, 2019.
Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE International Conference Computer Vision, pp. 1803-1811, 2019.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks.
arXiv preprint arXiv: 1et..08922..00 2018.
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoe Le. Bignas: Scaling up neural archi- tecture search with big single-stage models. arXiv preprint arXiv:2003. 2020.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprini arXiv:1611.0157,, 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
A APPENDIX
A.1 EXTRA MODELS IN OFA
Figure depicts the stratification of the latency levels into multiple buckets of size 5ms each and uniformly sampled 100 sub-networks of OFA for each bucket. This box-plot helps in uncovering the considerable variance between the maximum and minimum accuracies of the models of the same latency bucket in OFA. This variance is mainly caused due to OFA's enormous search space that contains redundant models that do not improve the overall accuracy of the model distribution belonging to a certain latency bucket.
A.2 COMPOFA- PROXYLESSNAS
1.0
ve Probability
0.6
0.4
0.2
CompOFA-Proxyless
OFx-Ploxs
0.0
24
26
28
(%)
Figure 7: Cumulative distribution function of accuracies of model configurations common to OFA & CompOFA with the base architecture changed from MobileNerV3 to ProxylessNAS. Despite the change architectre,, the same heuristic allows CompOFA to train the same or marginally higher accuracies with half the training budget.
11

---

Published as a conference paper at ICLR 2021
A.3 TRAINING SCHEDULE FOR COMPOFA-ELASTER KERNEL
Table 3 shows the training schedule for CompOFA with Elastic Kernel. Compared to OFA, the
training budget is reduced by %..
Table 3: CompOFA (Elastic Kernel)
Phase
K
D
W
Nsamplc
Epochs
Wall Time
GPU Hours
Teacher
180
28h 45m
172h 30m
Elastic Kernel
3,5,
4
6
125
26h 51m
161h 06m
Compound
3.5,
3,
3, 4,
25
9h 21m
56h 06m
Compound
3,5.7
2,
3.
4
3, 4,
4
125
48h 01m
288h 06m
Total
455
112h 58m
677h 48m
A.4 FASTER CONVERGENCE NAS
An evolutionary algorithm is set to converge after certain number of iterations (N) beyond which the fitness value of the population (P) does not improve significantly. OFA runs NAS with setting of N 500 iterations for population of size |P| 100. Figure demonstrates that the search time of NAS could further be reduced by reducing the number of iterations to N 300 and = 50 for CompOFA Elastic-Ken Kemiel and CompOFA Fixed-Kerne' respectively, without losing out on their Pareto-optimality. Coupled with the removal the latency predictor, these CompOFA-specins optimizations to the search are effective in reducing the search time and improving direct usability of CompOFA.
Figure 8: Distribution of accuracies of the randomly sampled models from OFA
12

---

Published as a conference paper at ICLR 2021
76.0
ImageNet Top-1 Accuracy (%)
75.5
75.0
74.0
735
73.0
OFA
72.5
Elastic CompOFA Fixed CompOFA
72.0
15.0
17.5
20.0
22.5 25.0
30.0
Intel Xeon CPU Latency (ms)
Figure 9: ImageNet accuracies and latencies on Samsung Notel0 with reduced NAS iterations for CompOFA.
13


Reviews:
[{"review_id": "zbZCPnpS8ue", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a compounding strategy for constraining the search spaces for once-for-all (OFA) network training framework. This is motivated by observations in prior work (particularly the EfficientNet by Mingxing Tan and Quoc Le) that certain compound relations exist between network dimensions (widths, depths, resolutions) for achieving optimal accuracy-latency trade-offs. In this work, network depths and widths are compoundly coupled. The resulting method called CompOFA reduces the search spaces significantly from 10^19 candidates to 3^5 (CompOFA) or 10^10 (CompOFA-Elastic) candidates, thus cutting the training costs by 50% or 30%. The models found with CompOFA/Elastic for specific hardware latencies are also more accurate than those found with OFA, despite requiring less training costs.\n\nPaper's strengths\n- Both smaller search spaces and the avoidance of progressive shrinking further reduce the training costs of an already efficient framework (OFA) while not sacrificing prediction performance is impressive. This makes it more practically useful.\n- The proposed compounding strategy is well-motivated by existing findings and nicely integrated to the OFA framework. It also clearly demonstrates that (human) prior knowledge is still very relevant in automated search for efficient networks.\n- The models searched by CompOFA match the performance of (or in some cases, outperform) OFA models, both in terms of best models given certain latencies and sampled models at population levels.\n- The paper is generally well-written and includes sufficient references to prior work.\n\nPaper's weaknesses\n- The pre-defined coupling configurations [D:2, W:3], [D:3, W:4], [D:4, W:6] are heuristically chosen based on the default search space originally proposed by the OFA paper. CompOFA works well because potentially these numbers happened to quite effective. In the more realistic settings (e.g., other datasets, tasks) where we do not have access to the prior numbers, it may be difficult to achieve good performance with a small number of coupling configurations, and OFA may work better to explore different combinations in a larger search space.\n\n- The compound coupling only considers network depths and widths, and only a small modification to the design space is proposed to achieve that. If the proposed method considered image resolution, the paper would be more complete in the context of compound scaling and it would have a stronger contribution.\n\n- In the OFA paper, smaller models are generated by shrinking the larger models, such that the smaller models (partially) retain the weights of the larger models. In this paper, all the smaller models are generated and trained at the same time. However, there is no information in this paper on how they are derived from the teacher network, e.g., whether they partially share the weights with teacher or they are distinctly initialized.\n\n- While it is interesting that CompOFA models could outperform OFA models in some cases, this paper does not make it clear why this is the case. CompOFA's search space is supposedly a subset of OFA's search space. The models trained by CompOFA should be covered by OFA. \n\n- Code is not provided in the submission for reproducibility and there is no promise of code release.\n\nMinor comment\n- Sec 5.2 mentions \"by using just one stage of training after the teacher model\" but the paper should explicitly mention that this applies only to CompOFA (fixed kernel). CompOFA (elastic kernel) in Table 4 actually still requires multiple stages.\n\nThis paper's ideas and contributions are nice-to-have but they are far from being groundbreaking.\n\n##post-rebuttal##\n\nI share the concerns of others reviewers that this paper has a limited novelty and narrowed scope but I think the authors have addressed other issues/concerns quite well. In my opinion, the key contributions mostly come from the insight and experimental results, and less so on the heuristic itself. Thus, I would adjust my rating to 7.", "extra_scores": {}, "extra_text": {}}, {"review_id": "oh5N63Hhpik", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Deploying models on multiple target end points with different hardware spec requires training model variants of an underlying architecture that meet the latency and other resource constraints on the device. OFA is a framework that helps this search for architectures like mobilenetv3 by training a model with large components and then training multiple instances of smaller models wherein components have reduced size so as to fit multiple target devices. Due to the large state space of parameter sizes for subcomponents, this search can take a while.\n\nThis paper proposes a heuristic to bring down the size. The main premise here is that searching for various subcomponents size parameters as if they were orthogonal design choices is not optimal. Good choices for parameters, for example the depth and the width are correlated. Therefore in place of searching as though these were orthogonal, the authors propose a correlated search to reduce the number of possibilities. This halves the overall search time, which is substantial saving given the time of the overall search. \n\nThe experiments are comprehensive on backing up the premise for the limited choice of data set and architecture considered. They show consistent improvements in overall training time  while showing the quality of resulting architectures is comparable to OFA methodology. This is done by showing the accuracy and size distribution of models obtained with this method vs OFA. \n\nMy concern with this paper is that the heuristic, while useful in this case, is not particularly interesting. Nor does the paper establish that this is applicable to other architectures. The result is that the scope of this paper is narrow. I would like more evidence to demonstrate that there are generalizable principles here. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hrd5pZSH08h", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Overview:\nThis paper is aiming at optimizing the OFA method in neural network model searching and training. Though CNN models perform well for many prolems, there is a serious shortage of this method. Researchers need to build and train a new model for every new problem, which will cost a lot of time and resources. A recent work, OFA, proposed a new method to partly solve this problem by training a family of models at a same time by parameter sharing. However, the OFA still have some problems. The searching space of OFA is too huge so the training process still takes too long and cost lots of resources. Thus this paper addressed a new solution, CompOFA, to speed up the training and try to get a balance between the accuracy and latency by building constraints between dimensions of model searching space and \"progressive shrinking\" approach.\n\nStrengths:\n1. This paper gives out obvious evidence of their basic insight. They use a heatmap to show the trade-off relationship between the width and depth of different models.\n2. This paper uses real data and SOTA works to do evaluation, makes their result more convincing.\n\nWeaknesses:\n1. The design idea and motivation of this paper is not well addressed. How do you decide to build the constraints? Why the trade-off is necessary? As far as I know, most of AI models care most about their accuracy, not latency, Unless the model is really too huge and slow (obviously the evaluation part of this paper only includes very small models with ms level latency). Because the inference speed can be easily increased by adding more GPUs. Since the inference job does not need to do synchronization, its is easily to achieve the linear scale out ratio. The paper just use a very arbitrary way to determine the constraints between these dimensions. How do you know that this is a good trade-off that researchers want?\n2. The design part is too brief. The first subsection just tell us \"We decide to build a constraint like this\", but why? The second subsection about speeding up the training is too ambiguous. Using some pseudo-code or program chart can help readers to understand your work's logic and innovations.\n3. The evaluation part is not solid enough to support this papers claim at the introduction. It claims to reduce searching space from 10^19 to 243. However, it still costs 50% training times compared to the original OFA method. And the training results also does not show apparent gains. The accuracy gain in most cases is lower than 0.4% while the overall accuracy does not exceed 80%. It will be better to add a classical light-weight CNN to be compared with, such as AlexNet, VGG or ResNet.\n\nOther Comments:\nThis paper's idea is quite creative and valuable that adding some constraints to the searching space will help us to remove many unnecessary models and improve the overall efficiency. However, the current design of the solution is not mature enough. It is necessary to add theoretical proofs for important design choices and convince readers that the choice is reasonable. And the evaluation part could also be improved by adding some classic models to compare.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8zn1FpJH1YV", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\nThis paper proposes to reduce the search space of OFA (a training scheme for obtaining networks for various deployment requirements) by scaling or shrinking the depth and width dimensions in NAS search space together. Because of the reduction, the multiple training phases of OFA is also less important and can be simplified to reduce training time.\n\n\n#### Technical\n* (+) The key observation made in this paper makes sense to me that the search space of OFA is over-sufficient and might be unnecessary for the target problem -- deployment in various environment with different latency/hardware constaints\n* (+) the experiments of this paper looks good, the improvement over OFA in terms of training speedup and energy saving is impressive.\n\n* (-)  I do appreciate the good engineering and intensive empirical results revealed by this paper but the technical novelty of this paper seems to me very increamental compared to OFa, though the arguments made in section 3.2 makes total sense me. For examples, the authors argued the model dimensions are not orthogonal, and the latency requirements only need to be satisfied in a way (granularity) that are below many thresholds. I think a better way to materialize these arguments is to explicitly characterize how these two observations should be incorporated into the architecture space, e.g., establishing direct correspondences between the design of the architecture space and the latency requirement, or by revealing how different dimension of architecture space are coupled (either theoretically or empirically ) and affect the design of the space. However, the paper ends up with a (not so intuitive) approach  -- that slightly modifies the space by coupling of two dimensions of the space, and starts discussing that by coupling these two dimensions, progressive multil-phase training can be eliminated to improve training.\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on  once-for-all (OFA) network training towards developing accurate models for different hardware platforms and varying latency constraints. The paper proposes an approach to significantly reducing model search space and thus training costs without losing in predictive performance. The paper is well-written, and the authors provide a thorough and convincing response to the concerns raised by some of the reviewers.
Decision: Accept (Poster)