PareCO: Pareto-aware Channel Optimization for Slimmable Neural Networks
Authors: Rudy Chin
Ari S. Morcos
Diana Marculescu
Venue:  (2021)
Keywords: Adaptive Neural Network
Slimmable Neural Network
Channel Optimization
Neural Architecture Search
Convolutional Neural Network
Image Classification

Abstract: Slimmable neural networks provide a flexible trade-off front between prediction error and computational cost (such as the number of floating-point operations or FLOPs) with the same storage cost as a single model. They have been proposed recently for resource-constrained settings such as mobile devices. However, current slimmable neural networks use a single width-multiplier for all the layers to arrive at sub-networks with different performance profiles, which neglects that different layers affect the network's prediction accuracy differently and have different FLOP requirements. Hence,  developing a principled approach for deciding width-multipliers across different layers could potentially improve the performance of slimmable networks. To allow for heterogeneous width-multipliers across different layers, we formulate the problem of optimizing slimmable networks from a multi-objective optimization lens, which leads to a novel algorithm for optimizing both the shared weights and the width-multipliers for the sub-networks. We perform extensive empirical analysis with 15 network and dataset combinations and two types of cost objectives, i.e., FLOPs and memory footprint, to demonstrate the effectiveness of the proposed method compared to existing alternatives. Quantitatively, improvements up to 1.7% and 8% in top-1 accuracy on the ImageNet dataset can be attained for MobileNetV2 considering FLOPs and memory footprint, respectively. Our results highlight the potential of optimizing the channel counts for different layers jointly with the weights for slimmable networks.

Under review as a conference paper at ICLR 2021
PARECO: PARETO-AWARS CHANNEL OPTIMIZATION FOR SLIMMABLE NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Slimmable neural networks provide flexible trade-off front between prediction eтror and computational cost (such as the number of floting-poinn operations FLOPs) with the same storage cost as a single model. They have been proposed recently for resource-constrained settings such mobile devices. However, current slimmable neural networks use a single hidth-multiiplie for all the layers to arrive at sub-networks with different performance profiles, which neglects that differ- ent layers affect the network's prediction accuracy differently and have different FLOP requirements. Hence, developing principled approach for deciding width- multipliers across different layers could potentially improve the performance slimmable networks. To allow for heterogeneous width-multiplle across different layers, we formulate the problem of optimizing slimmable networks from a multi- objective optimization lens, which leads to a novel algorithm for optimizing both the shared weights and the width- multipliesss for the networss..s. We perform extensive empirical analysis with 15 network and dataset combinations and two types of cost objectives, i.e., FLOPs and memory footprint, to demonstrate the effectiveness of the proposed method compared to existing alternatives. Quan- titatively, improvements up to 1.7% and 8% in top-1 accuracy on the ImageNet dataset can be attained for MobileNetV2 considering FLOPs and memory footprint, respectively. Our results highlight the potential of optimizing the channel counts for different layers jointly with the weights for slimmable networks.
INTRODUCTION
Slimmable neural networks have been proposed with the promise of enabling multiple neural networks with different trade-offs between prediction error and the number of floating-poing operations (FLOPs), all at the storage cost of only a single neural network (Yu et al. 2019). This is stark contrast to channel pruning methods (Berman et al. 2020 Yu & Huang 2019a Guo et al. 2020 Molchanov jet al. 2019) that aim for a small standalone model. Slimmable neural networks are useful for applications mobile and other resource- -or.urrcee-ottttieeeddeeeeee devices. As an example, the ability to deploy multiple versions of the same neural network would alleviate the maintenance costs for applications which support number of different mobile devices with different memory and storage constraints, as only one model needs to be maintained. Similarly, one can deploy a single model which is configurable at run-time to dynamically cope with different latency or accuracy requirements. For example, users may care more about power efficiency when the battery of their devices is running low while the accuracy of the pettowee- application may be more important otherwise.
A slimmable neural network is trained by simultaneously considering networks with different widths (or channel counts) using a single set of shared weights. The width child network is specified by a real number between 0 and 1, which is known as the "width "width--mltiplic" multiplier" (Howard et al. (017). Such a parameter specifies how many channels per layer to use proportional to the full network. For example, a width-multiplier of 0.35x represents network that has channel counts that are 35% of the full network for all the layers. While specifying child networks using a single width-m mutth-multipplir for all the layers has shown empirical success (Yu & Huang 2019 9bl Yu et al. (2019), such a specification neglects that different layers affect the network's 's output differently (Zhang et al. 2019) and have different FLOP requirements (Gordon et al.) 2018), which may lead to sub-optimal results. In similar setting, as demonstrated in the model pruning literature (Gordon et al. 2018 Liu et al. 2019bb Morcos al. 2019 Renda et al. 2020), having different pruning ratios for different layers of

---

Under review as a conference paper at ICLR 2021
Conventional Sliimable Training
TwoStage Siiimabbe Training
Channel Pruning
PareCO (Ours)
Weight Training
Weight Training
Weight Training
Pruning and training
SGO
SGD
SGD
CORCO CONSENO COADSROp
GoODo
CORno
Arch Searching
GOOD
CORDe
Foxd weights aptimtzefoee
GOORD GARRO
DORO
One model for one target sparsity
Shared weights archiledtuna specified by single global width muttipller
Shared weights, archileeures spcoiled by random layer-wiss muttipliers
Arch Searching
Fiell oppigtss.cccinsss and
specifidd by joint optimization
Shared weights, archtectures
Figure I: Schematic overview comparing our proposed method with existing alternatives and chan- nel pruning. Channel pruning has a fundamentally different goal compared to ours, i.e., training slimmable nets. PareCO jointly optimizes both the architectures and the shared weights.
the network can further improve results over single ratio across layers. This raises an interesting question: How should we obtain these non-uniform widths for slimmable nets?
To achieve non-uniform width- mullippiers across layers, one consider using techniques from the neural architecture search (NAS) literature (Cai et al.] 2020; Yu et al.) (020), which we call TwoStage training. Specifically, one can first train supernet with weight-sharing by uniformly sampling width-multiplier for each layer. After this procedure converges, one can use multi-objective optimization methods to search for width given the trained weights. However, width optimization has a much larger design space than that considered in existing methods for NAS, Specifically, each layer can have hundreds of choices (since there are hundreds of channels for each layer). This makes unclear if such a training technique is suitable for channel optimization"
As an alternative to existing techniques, we take multi-objectio optimization viewpoint, aiming to jointly optimize the width width-multiplice multipliers for different layers and the shared weights in slimmable neural network. schematic view of the differences among the conventional slimmable training. TwoStage training, and our proposed method is shown in Figure [ The contributions of this work are three-fold. First, through a multi-objective optimization lens, we provide the first principled formulation for jointly optimizing the weights and widths of slimmable neural networks. The proposed formulation is general and can be applied to objectives other than prediction error and FLOPs YYu & Huang 2019b; Yu al. 2019). Second, we propose Pareto-aware Channel Optimization PareCO, a novel algorithm which approaches the intractable problem formulation in an approximate fashion using stochastic gradient descent, of which the conventional training method proposed for universally slimmable neural networks (Yu & Huang 2019b) is a special case. Finally, we perform extensive empirical analysis using 15 network and dataset combinations and two types of cost objectives to demonstrate the effectiveness of the proposed algorithm over existing techniques.
2 RELATED WORK
2.1 SLIMMABLE NEURAL NETWORKS
Slimmable neural networks Yu et al. 2019) enable multiple sub-networks with different compression ratios to be generated from a single network with one set weights. This allows the FLOPs of network to be dynamically configurable at run- time without increasing the storage cost of the model weights Based on this concept, better training methodologies have been proposed to enhance the performance of slimmable networks Yuu & Huang 2019b). One can view a slimmable network dynamic
Both OFA (Cai et al. 2020) and BigNAS Yu et al. 2020) mainly use the pre-defiend channel counts and search for kernel sizes, depth, and input resolution. Specifically, channel counts refer to expansion ratios only for OFA while BigNAS only considers small range of channel counts near the pre-defined ones.
2

---

Under review as a conference paper at ICLR 2021
computation graph where the graph can be constructed dynamically with different accuracy and FLOPs profiles. With this perspective, one can go beyond changing just the width of the network. For example, one can alter the network's 's sub- sub-graphs (Ruiz& & Verbeek 2019), network's 's depth (Bolukbasi et at. 2017] 2017 Elbayad et al. 2019 Huang et al. 2017 Li |l. 2019a Kaya et al. 2019), and network's kernel sizes and input resolutions (Cai al., 2020: Yu et al. 20220). Complementing prior work
primarily focusing on generalizing slimmable networks to additional architectural paradigms, our work provides the first principled multi-objective formulation for optimizing slimmable networks with tunable architecture decisions. While our analysis focuses on the network widths, our proposed formulation can be easily extended other architectural parameters.
2.2 NEURAL ARCHITECTURE SEARCH
A slimmable neural network can be viewed as an instantiation of weight-sharing. In the literature for neural architecture search (NAS), weight-sharing is commonly adopted to reduce the search cost (Liu
eet al. 2018: Stamoulis er al., 2019; Guo et al. 2019, [Bender et al., 20188) Berman et al. [020 Yu & Huang 2019a). Specifically, NAS methods use weight-sharing as proxy for evaluating the
performance of the sub-networks to reduce the computational cost of iterative training and evaluation.
However, NAS methods are concerned with the architecture of the network and the found network is
re-trained from scratch, which is different from the weight-sharing mechanism adopted in slimmable networks where the weights are used for multiple networks during test time.
Multi-objectiin optimization has also been adopted in NAS literature (Dong et al. 2018 Cheng et al. 2018; Iqbal et al., 2020; Lu et al. 2019 Elsken et al., 2018). However, a crucial difference of the present work compared to these papers is that we are interested in learning a single set of weights from which multiple FL.OP configurations can be used (as in slimmable networks) rather than finding architectures independently for each FLOP configuration that can be trained from scratch freely. Put another way, in our setting, both the shared weights and the searched architectures optimized jointly, whereas in prior work, only searched architectures were optimized.
2.3
CHANNEL PRUNING
Reducing the channel or filter counts for a pre-trained model also known as channel pruning. In channel pruning, the goal is to find a single small model that maximizes the accuracy while satisfying some resource constraints. Several studies have investigated how to better characterize redundant
channels in a post-training fashion given pre-trained model (Li et al. 2016 He et al. 2019 Liu et al.,
2017 Ye et al. 2018 Molchanov et al. 2016: 2019: Aflalo et al. 2020). Besides post-processing
perspective to channel pruning, prior work has also investigated channel pruning via an optimization
lens. Specifically, channel pruming methods based on Lasso (Wen et al. 2016 Liu et al.) 2017]
Gordon et al.] 20188 Yun al. 2019), stochastic to (Louizos et al. 2017 7), and ADMM (Li et al. 2019b] Yang et al. 2020) have been developed. Liu et al. (Liu et al. 2019b) later show that channel counts for different layers are more important for the performance of channel pruning, As a result, several studies have investigated pruning via an architecture search perspective (Yu & Huang 2019a
Berman al. 2020 Ma al. 2019 Chin et al. 2020 Liu et al. 2019a).
While channel pruning also optimizes for non-uniform widths, the goal of channel pruning is crucially different from ours. The key difference is that channel pruning is concerned with a single pruned model while slimmable neural networks require a set of models to be trained using weight sharing. As a result, is not clear if those techniques are suitable for finding non-uniform widths for slimmable nets. Moreover, using channel pruning to obtain a set of pruned models can be computationally expensive. Specifically, the state-of-the-art differentiable pruning method takes roughly 2x the training time for searching the channel counts for a given target FLOPs (Guo et al. 2020). That is, if we want 20 uniformly distributed FLOPs between 20% and 100%, it takes roughly 24x the training time of the 100% FLOPs network just to obtain architectures. In contrast, our formulation is targeting slimmable nets directly with a training cost similar to that of conventional slimmable networks, which is 4 x the training time of standalone model.
3

---

Under review as a conference paper at ICLR 2021
3 METHODOLOGY
In this work, we are interested in jointly optimizing the network widths and network weights. Ultimately, when evaluating the performance of a slimmable neural network, we care about the trade-off curve between multiple objectives, e.g., theoretical speedup and accuracy. This trade-off' curve is formed by evaluating multiple networks with width configurations sampled from width sampling distribution. Viewed from this perspective, the sampling distribution should be optimized in such a way that the resulting networks have a better trade-off curve (i.e., larger area under curve), which is a multi-objective optimization problem. This section formalizes this idea and provides an algorithm to solve in an approximate fashion.
3.1 PROBLEM FORMULATION
Intuitively, goal is to optimize the shared weights maximize the area under the best trade-off curve between the accuracy and theoretical speedup obtained by optimizing network's widths. Since accuracy is not differentiable w.r.t. the shared weights, we switch objectives from accuracy and theoretical speedup to cross-entropy loss and FLOPs, respectively. In this setting, the objective becomes to minimize the area under curve. To arrive at such an objective, we start by defining the notion of optimality in minimizing multiple objectives (such as the cross-entropy loss and FLOPs).
Definition 1 (Pareto frontier) Let f(x) (fi(a)....... . (x)) be vector of responses from K different objectives. Define vector inequality a < y as I, S y1 V E [K] with at least one inequality being strict. We call set of points P a Pareto frontier if f(x) < fzy) for any x E P and y & P.
With this definition, we essentially want the loss for the shared weights to be the area under the curve formed by the Pareto frontier. To do so, we need an actionable way to obtain the Pareto frontier and we make use of the following Lemma:
Lemma 3.1 (Augmented Tchebyshev Scalarization (Section 1.3.3 in (Nakayama et al. 2009 )
Define scalarization of K objectives as
(x()) = max (.(((()) - -F))+ + Σ \f((()), KE(K]
{E[K]
where X is weightings among objectives, Js is a baseline constant such that (Ji(z) - (1) O x and >> the Pareto frontier can be specified via P (arg mine (x(x) AA E A AK-1 where K 1-1 is a K-1 simplex.
The second term of equation is the commonly used weighted sum scalarization and it can be depicted as a line in the objectives space. However, minimizing it alone not sufficient if the Pareto curve is non-convex (Nakayama et al.) 2009). This calls for the first term, which can be depicted as the axis-aligned lines of a non-positive orthant in the objectives space. With Lemma 3.T one can obtain the Pareto frontier by solving multiple augmented Tchebyshev scalarized optimization problems with different As. A À vector can be interpreted as a weighting on the objectives, which is used to summarize multiple objectives into single scalar. For instance, consider the case in which the cross-entropy loss and FL.OPs are the two objectives of interest. If taking ÀCE -> and AFLOeS - O, the scalarized objective is then dominated by the cross-entropy loss and are effectively seeking width configurations that minimize the cross entropy loss. In contrast, if taking AcE + 0 and *PLOPS + 1, we are then effectively seeking width configurations that minimize FLOPs. With the scalarization, we have the following theorem for summarizing the area under curve quantitatively:
Theorem 3.2 Optimizing network weights O to minimize area under Pareto curve formed by cross- entropy loss (fcr) and FLOPs (fFLors) can be done approximately with the following objective:
M
Σ m=1
(= 0, x,y),
arg min 0
M
where x, are the sampled training input and label, respectively; M is the number of Monte Carlo sample; (m) is FLOPs target sampled uniformly between the lowest and the highest FLOPs, a()
4

---

Under review as a conference paper at ICLR 2021
maps weighting À to the corresponding Pareto channel configuration (i.e., a(x) arg mine (x(c) where T and X are defined in Lemma 3. and g(.) maps certain FLOPs back to the corresponding X, (i.e., 9 (frLOr, o aa) )
The proof is in Appendix Intuitively speaking, the objective to minimize the cross entropy loss for the Pareto-optimal channel configurations (given the current 0, æ. across different FLOPs uniformly. While equation 2 precisely defines our goal, solving g(-) and a(-) can be intractable since the functions are usually highly non-convex with respect to channel configurations and do not have analytical gradient information that admits first-order optimization algorithms. To tackle these challenges, we propose to model both fcE and FELOPS via Gaussian Processes (GPs) (Rasmussen 2003). This allows (-)) to be approximated via Bayesian Optimization (BO) (Srinivas et al. 2009) with low overhead, which in enables g(-) to be implemented using binary search with fFLOP
Concretely, binary search is done by repeating the following: start with X, solve a(),) using BO, and decrease or increase 1 depending on the FLOPs of the current architecture fFlops(a(A))
Algorithm 1: PareCO
Input :Model parameters 0, lower bound for width- multipliers wh E [0, 1J. number of full iterations F. number of gradient descent updates n, number of À samples M
Output:Trranne parameter 0, approximate Pareto front.N /N
H = (Historical minimizers ?)
for
I...F do
y. sample data()
fcr. fF.OP. = fCE ( (H: 0, w, y). SFLOP(H) (Calculate the objectives for each ē E H)
9= BuildGP-UCBB H. fCE. JHON (Build acquisition func. via BoTorch Balandat al. 2019
widths =
for m = I...M do C (OBS( g. H. fcs, fnLos) (Algorithm) appeh
2
10
end
11
7( = 7 widths widtts..ppend((ur))
(update historical data) (smallest width for the sandwich rule in Yu & Huang 2079b
12
for = I...I do
14
crummableTraining 0, widths) (line 3-16 of Algorithm in Yu Huang 20/9h
15
end
16
N W=nonDominnaeddoon(HH fcE, FFLOP))
17
end
3.2
APPROXIMATION VIA MULTI-OBJECTIVE BAYESIAN OPTIMIZATION
The goal is to approximate ax-) and g(-) quickly with good quality. In this realm, Bayesian Op- timization is known to be sample-efficien for doing so. Specifically, are hoping to solve arg min. (c, 0, x, y) efficiently using Bayesian Optimization where here denotes the chan- nel configurations. Instead of modeling Tx directly, we propose to model fcr and fFLOPs using two GPs. This design choice is mainly based on the fact that we would like to solve g(.) using binary search. More specifically, by having two GPs, the GPs can be reused across different X. On the other hand, if we directly model Tx we have to fit a GP for every X, which can be costly.
Having a separate GP for each objective of interest turns out have regret for optimization if a proper acquisition function is adopted (Paria et al., 2019). As a result, we follow Paria et al. (2019) and adopt the Upper Confidence Bound (UCB) Srinivas al. 2009) acquisition function for optimizing the problem of interest. To optimize the target objective. Bayesian Optimization proceeds sequentially. To begin, we start with training data H4-1 (ci..... Ctto) and their function responses f:(ãe) V E (CE, FLOPss, l E [t + o) where t denotes the current timestamp in the sequential optimization and o is the initial size of the training data. Then, GPs are fitted for both objectives and acquisition functions for both GPs are combined using the augmented Tchebyshev scalarization with a specified X, which is then optimized to obtain the next point C-t+11- This minimization is tractable because it minimizes the surrogate function instead of the unknown function. Under properly set hyperparameters for UCB, it is known that this procedure introduces regret (Paria et al. 2019). In other words, if allocate enough time for BO, iee.
5

---

Under review as a conference paper at ICLR 2021
def
t + oo, and set the output of BO to be = Ctto. the method provides a close approximation, i.e.,
Tx (C, 0, æ, y) mine (c, 0, y).
3.3 APPROXIMATION VIA TEMPORAL SIMILARITY
Equipped with BO, solving equation 22 can be done by running the following three steps iteratively: (1) sample M target FLOPs, (2) solve the corresponding channel configurations using BO with binary search, and (3) perform step of stochastic gradient descent. While BO provides a tractable mean for us to solve for Pareto-optimal channel configurations, is still computationally expensive to run BO for every iteration of stochastic gradient descent. Thus, we propose to exploit temporal similarity for making the optimization practical. Specifically, we notice that & would not be drastically different across few iterations since gradient descent itself relies first-order Taylor approximation. As a result, we propose the following two approximations. First, instead of having many sequential only perform
queries in one subroutine of Bayesian optimization, to H for future Bayesian optimization. Note that for each query in BO, the cross-entrops loss will be reevaluated for each H build faithful Gaussian Processes. Our second approximation is to perform n SGD updates as opposed to one before another query for Bayesian optimization. We further provide theoretical analysis for approximation via temporal similarity Appendix
query and store the query
E
The proposed algorithm is Pareto-aware as the derivation stems from minimizing area under the Pareto curve. In short, PareCO has three steps: (1) build surrogate functions (i.e., GPs) and acquisition functions (i.e., UCBs) using historical data H and their function responses, (2) sample target FL.OPs and solve for the corresponding widths (i.e., 2) via binary search with one query of BO, and (3) perform gradient descent steps using the solved widths. One can recover slimmable training (Yu & Huang 2019b) by replacing lines 8 with randomly sampling single width-multiplie for all the layers and setting n = in line 13. The outputs of PareCO are both the shared weights and the Pareto-optima widths. To obtain the Pareto-optimal widths, we use non-dominated sort based on the training loss and FLOPSs for e E H.
PareCO Based on this preamble, we present our algorithm, PareCO, in Algorithm 1
Algorithm 2: Bayesian Optimization with Binary Search (BOBS)
Input :Aquisiiionn functions g. historical data H. fCE. frlors, search precision Output configurations é
{ = 10 (A small positive number according to Nakaaama et al. 2009
SFLOP. = (eniformm(fLLoP.mmi. fFLOPsma)
(Sample a target FLOPs)
AFLOrS Amin, Amox = 0.5, 0,
while [
> do
binary search
FullModsiFLLP
== arg mine [CF (((a(c) - g.) + +BE A.g.(c)
(FLOP)(2) > SFOOP then
Amin AFLOPS
AmLOR: = (AHuue + 3/2
clse
10
Amax AHLORS
11
AELORS = (AHLuSs + Amii) 1/2
12
end
13
end
EXPERIMENTS
4.1
PERFORMANCE GAINS INTRODUCED PARECO
For all the PareCO experiments in this sub-section, set such that PareCO only visits 1000 width configurations throughout the entire training [ = 1000). Also, we set M to be 2, which follows the conventional slimmable training method (Yu & Huang 2019b) that samples two width configurations in between the largest and the smallest widths. As for binary search, we conduct at most 10 binary searches with to 0.02, which means that the binary search terminates if the FLOPs difference is

---

Under review as a conference paper at ICLR 2021
Top1 Test Eiro (80
Top-1 Toot Enro (%)
Top- Teat Error (%)
Too-1 Tost Emor
(8)
(%)
FLOPA
(%)
(a) ResNet20 C10
(b) ResNet32 C10
(c) ResNet44 C10
(d) ResNet56 C10
Top-1 Test TestEroi Enor (%)
Top-1 Tost Emto (M)
(S)
Top-1 Tast Emer (%)
8
FLOPs (56)
FLOPs (%)
FLOPs (%)
FLOPs (%)
(c) ResNer20 C100
(f) ResNet32 C100
(g) ResNet44 C100
(h) ResNet56 C100
Top-1 Test Error (%)
Top-1 Test Error (%)
Top-1 Test (%)
Top-1 Test Error (%)
FLOPs (%6
FLOPs
FLOPs (%)
FLOPs
(i) x ResNet22 C100
(i 3eRNNet2 C100
(k) 2eReNNe120 C100
5x ResNet20 C100
Swets
Imaggeherr Trpe Emner (4)
Imogetes Top Enor
Twedhage
500
752
3000
3250
1000
3750
(m) MohileNetV2 ImageNet
(n) MobileNerV3 ImageNet
(0) ResNet18 ImageNet
Figure 2: Comparisons among PareCO, Slim, and TwoStage. C10 and C100 denote CIFAR- For the CIFAR dataset, perform three trials for each method and plot the mean and standard deviation. PareCO is better or comparable to Slim. The numerical results for ImageNet are detailed in Table I in Appendix G
within a two percent margin relative to the full model FLOPs. On average, the procedure terminates by using 3.4 binary searches for results on ImageNet. The dimension of e network-des deeeeddedeee and is specified in Appendix B and the training hyperparameters are detailed in Appendix [D] To arrive at the final set of architectures for PareCO, we use non-dominated based the training loss and FLOPs for E H.
We consider three datasets: CIFAR-10. CIFAR-100, and ImageNet. To provide informative com- parisons, we verify our implementation for the conventional slimmable training with the reported numbers in Yuu & Huang 2019b) using MobileNetV2 on ImageNet. Our results follow closely to the reported numbers shown in Figure 2m, which makes our comparisons on other datasets convincing.
compare to following baselines:
the conventional slimmable training method (the universally slimmable networks Yu & Huang (2019b)). We select 40 architectures uniformly distributed across FLOPs and run non-dominated sort using training loss and FLOPs to arrive at the points to be plotted.
Slim:
TwoStage:
disjoint optimization that first trains the model with weight sharing, then search methods to find architectures that work well given the trained weights (similar to OFA (Cai et al. 2020) and BigNAS YYu et al. 2020)). To compare fairly with PareCO,

---

Under review as a conference paper at ICLR 2021
we use multi-objective Bayesian optimization for the search. After optimization, we run a non-dominated sort for all the visited architectures H. using training loss and FLOPs to arrive at the points to be plotted.
Compared to Slim, the proposed PareCO has demonstrated much better results across various networks and datasets. This suggests that channel optimization can indeed improve the efficiency of slimmable networks. Compared to TwoStage, PareCO is better or comparable across networks and datasets. This suggests that training network weights by uniformly sampling architectures regardless of Pareto- efficiency can be sub-optimal. More importantly we find that such a training techniques result in low performances for all the sub-networks in MobileNetV3, which is potentially due to conflicts in the optimization process with large number of sub-networks and small network capacity. From the perspective of training overhead, PareCO introduced minor overhead compared to Slim due to the temporal similarity approximation. More specifically, on ImageNet, PareCO incurs approximately 20% extra overhead compared to Slim.
Note that the performance among these three methods similar for the CIFAR-10 dataset. This is plausible since when a network is more over-panee wwer-parameterized there are many solutions to the optimization problem and it is easier find solutions with the constraints imposed by weight sharing. In contrast, when the network is relatively less over-parameterized, compromises have to be made due the constraints imposed by weight sharing. In such scenarios, PareCO outperforms Slim significantly, as it can be seen in CIFAR- 100 and ImageNet experiments. We conjecture that this because PareCO introduces a new optimization variable wwidth-multiplierss which allows better compromises to be attained. Along similar lines, from the experiments with ResNets on CIFAR-100 (Figure 2e to Figure 2h), we find that shallower models tend to benefit more from joint channel and weight optimization than their deeper counterparts.
As FLOPs may not necessarily reflect latency improvements since FLOP does not capture memory accesses, we in addition plot latency-vs. -errer for the data in Figure 2m in Figure 3 The latency is measured on a single V100 GPU using a batch size of 128. When visualized in latency, PareCO still performs favorably compared to Slim and TwoStage for MobileNetV2 ImageNet.
Lastly, to demonstrate the generality of PareCO, we consider another cost objective that is critical for on-device machine learning, i.e., inference memory footprint (Yu et al. 2019). Inference memory footprint decides whether a model is executable or not on memory ceettry-consraained devices. We detailed the memory footprint calculation in Appendix [F] Since PareCO is general, we can replace the FLOPs calculation with memory footprint calculation to optimize for memory-vs, -error. As shown in Figure 4 PareCO significantly outperform other alternatives. Notably, PareCO outperforms Slim by up 8% top-1 accuracy for MobileNetV2. Such drastic improvement comes from the fact that memory footpring depends only on the largest layer. As a result, slimming all the layers equally to arrive at networks with smaller memory footprint (as done in Slim) is less than ideal since only one layer contributes to the reduced memory.
MohlieNet22 ImagaNet
ResNet-18 ImageNet
Iragpewe Tap-1 Enee 280
Ne:00
Tws
TwsStage
680
4.5
(Milytes)
Infwerce fextprir (Mlytwo
Figure 3: A latency-vs-eeroo view of Figure 2m
Figure 4: Prediction eiror inference memory footprint for MobileNetV2 and ResNet-18 on ImageNet.
4.2 ABLATION STUDIES
In this subsection, we ablate the hyperparameters that are specific to PareCO to understand their impact. We use ResNet20 and CIFAR-100 for the ablation with the results summarized in Figure 5
Binary search Without binary search, one can also consider sampling À uniformly from the A K-1 which does not require any binary search and is easy implement. However, the issue with this
8

---

Under review as a conference paper at ICLR 2021
Additonnal Traiiing FLOPs (%)
307
Top Test Emar (%)
Froquency
FLOPs (%)
FLOPs (%)
FLOPs (%)
(a) Impact of binary search (BS).
(b) Histogram FLOPs (c) Performance for differ- (d) Additional overhead over Slim for different n.
for H w/ and w/o BS. ent n.
Figure Ablation study for binary search and the number of gradient descent updates per full iteration using ResNet20 and CIFAR-100. Experiments are conducted three times and we plot the mean and standard deviation.
sampling strategy is that uniform sampling À does not necessarily imply uniform sampling in the objective space, e.g., FLOPs. As shown in Figure 5a and Figure 5b sampling directly in the A space results in non-uniform FLOPs and worse performance compared binary search.
Number of samples in BO We reduce the number of samples in BO by increasing the number of gradient descent updates. In previous experiments, we have = 313, which results in |H|] 1000. Here, we ablate n to 156, 626, 1252, 3128 such that | = 2000, 500, 250, 100, respectively. With larger n, the algorithm introduce a worse approximation since there are overall less iterations put into Bayesian optimization. As shown in Figure 5c| we observe worse results with higher n. On the other hand, the improvement introduced by lower n saturates quickly. The training overhead of PareCO as function of compared to Slim is shown in Figure 5d where the dots are the employed n.
5 CONCLUSION
In this work, we propose to tackle the problem of training slimmable networks via a multi-objective optimization lens, which provides novel and principled framework for optimizing slimmable networks. With this formulation, we propose a novel training algorithm, PareCO, which trains slimmable neural networks by jointly learning both channel configurations and the shared weights. In our empirical analysis, we extensively verify the effectiveness of PareCO over existing techniques on 15 dataset and network combinations and types of cost objectives, i.e. FLOPs and memory footprint. Our results highlight the potential of optimizing the channel counts for different layers jointly with the weights and demonstrate the power of such techniques for slimmable networks.
REFERENCES
Yonathan Aflalo, Asaf Noy, Ming Lin, Itamar Friedman, and Lihi Zelnik. Knapsack pruning with inner distillation. arXiv preprint arXiv:2002.08255: 2020.
Maximilian Balandat, Brian Karrer, Daniel R Jiang, Samuel Daulton, Benjamin Letham, Andrew Gor- don Wilson, and Eytan Bakshy. Botorch: Programmable bayesian optimization in pytorch. arXiv preprint 191.::191000660 2019.
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Understanding and simplifying one-shot architecture search. In Jennifer Dy and Andreas Krause (eds.), Proceed- ings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 550-559, Stockholmsmässan Stockholm Sweden, 10-15 Jul 2018. PMLR. /ttphhttt://ppreeeeiime
ml prwss wttpasss/v/bbbbeeeeee 8a httml
Maxim Berman, Leonid Pishchulin, Ning Xu, Gérard Medioni, et al. Aows: Adaptive and optimal network width search with latency constraints. Proceedings IEEE CVPR, 2020.
Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 527-536. JMLR. org, 2017.

---

Under review as a conference paper at ICLR 2021
Han Cai, Chuang Gan, Tianzhe Wang. Zhekai Zhang. and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations, 2020.
An-Chieh Cheng, Jin- Dong. Chi-Hung Hsu, Shu-Huan Chang, Min Sun, Shih-Chiel Chang, Jia-Yo Pan, Yu-Ting Chen, Wei Wei, and Da-Cheng Juan. Searching toward pareto-optimal device- aware neural architectures. In Proceedings of the International Conference on Compuler-Aided Design, pp. 1-7, 2018.
Ting-Wa Fing-Wu Chin, Ruizhou Ding, Cha Zhang, and Diana Marculescu. Towards efficient model compres- sion via learned global ranking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Jin-Dong Dong, An-Chieh Cheng, Da-Cheng Juan, Wei Wei, and Min Sun. Dpp-net: Device- aware progressive search for pareto-optimal neural architectures. In Proceedings of the European Conference Computer Vision (ECCV), pp. 517-531, 2018.
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. arXiv preprint 1rXiv:I910.
2019.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient mbltiiobjecttvve neural architecture search via lamarckian evolution. arXiv preprint arXiv:1804. 09081, 2018.
Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang, and Edward Choi. Morphnet: Fast & simple resource--onstrained structure learning of deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1586-1595, 2018.
Shaopeng Guo, Yujie Wang, Quanquan Li, and Junjie Yan. Dmep: Differentiable markov channel pruning for neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1539-1547, 2020.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. arXiv preprint arXiv:1904.00420. 2019.
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4340-4349, 2019.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang. Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1314
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: /ee../4444.41/1/ 2017
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Wein- berger. Multi-scale dense networks for resource efficient image classification. arXiv preprint 1733::1703.09844, 2017.
Md Shahriar Iqbal, Jianhai Su. Lars Kotthoff, and Pooyan Jamshidi. Flexibo: Cost-aware multi- objective optimization of deep neural networks. arXiv preprint arXiv:2001. 006588, 2020.
Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. Shallow-Deer Networks: Understanding and mitigating network overthinking. In Proceedings of the 2019 International Conference on Machine Learning (ICML), Long Beach, CA, Jun 2019.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.0877 087100 2016.
Hao Li, Hong Zhang, Xiaojuan Qi, Ruigang Yang. and Gao Huang. Improved techniques for training adaptive deep networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1991-1900, 2019a.
10

---

Under review as a conference paper at ICLR 2021
Tuanhui Li. Baoyuan Wu, Yujiu Yang, Yanbo Fan, Yong Zhang, and Wei Liu. Compressing convolutional neural networks via factorized convolutional filters. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3977-3986, 2019b.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search, arXiv preprint arXiv: 09055, 2018.
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang- Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE International Conference Computer Vision, pp. 5296-3305, 2019a.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2736-27622222 1734,4744, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference Learning Representations, 2019b. URL htttps://ppenreiil net.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1.0 regularization. arXiv preprint arXiv: 111v:11712.03122 2017.
Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, and Wolfgang Banzhaf. Nsga-net: neural architecture search using multi-objective genetic algorithm. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 419-427, 2019.
Xingchen Ma, Amal Rannen Triki, Maxim Berman, Christos Sagonas, Jacques Cali, and Matthew B Blaschko. bayesian optimization framework for neural network compression. In Proceedings of the IEEE International Conference on Computer Vision, pp. 10274-10283, 2019.
Bertil Matérn. Spatial variation, volume 36. Springer Science & Business Media, 2013.
Pavlo Molchanov, Stephen Tyree. Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.0644/ 2016.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11264-111 11272, 2019.
Ari Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. In Advances in Neural Information Processing Systems, pp. 4933 4943, 2019.
Hirotaka Nakayama, Yeboon Yun, and Min Yoon. Sequential approximate multiobjective optinization using computational intelligence. Springer Science & Business Media, 2009.
Biswajit Paria, Kirthevasan Kandasamy, and Barnabás Póczos. flexible framework for multi- objective bayesian optimization using random scalarizations. In Amir Globerson and Ricardo Silva (eds.), Proceedings of the Thirty- Conference on Uncertainty Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, pp. 267. AUAI Press, 2019, hRtp:tt:://aaa ua 20920019/rocceiingg
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luea Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8026-8037. Curran Associates, Inc., 2019. URL http:: htttp::/ppppss..tttte/ wwt.::ess.ipsssccccceeeeeeeeeeee cc/paerr/ 901 5 Sis---yttrch-aaa mperat ve sty e h gh www.-pprooooeeeeeeeeeeee leapeddeeeeeeeeeenneeeee learning library.
pdf
Carl Edward Rasmussen, Gaussian processes in machine learning. In Summer School on Machine Learning, pp. 63-71. Springer, 2003.
11

---

Under review as a conference paper at ICLR 2021
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020. URL
net/ htttps://ppenreaatt wttprrrrwweeeeeeeeooooueeaaaaa2 forumiidd-$
Adrià Ruiz and Jakob Verbeek. Adaptative inference cost with convolutional neural mixture models. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1872-1881, 2019,
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Llan-CChiet Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian process opti- mization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995, 2009.
Dimitrios Stamoulis, Ruizhou Ding. Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Single-path nas: Designing hardware-efffcient convnets in less than 4 hours. arXiv preprint arXiv: 02877, 2019.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity deep neural networks. In Advances in neural information processing systems, pp. 2074-2082, 2016.
Haichuan Yang, Shupeng Gui, Yuhao Zhu, and Ji Liu. Learning sparsity and quantization jointly and automatically for neural network compression via constrained optimization. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition, 2020.
Jianbo Ye, Xin Lu, Zhe Lin, and James Z. Wang. Rethinking the smaller- wwww--less-inoommeeeee assumption in channel pruning of convolution layers. In International Conference on Learning Representations, 2018. htt /tttttps://openrevvieee
net. 2tte://formmmi===44444403
Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers. 8, 2019a.
arXiv preprint arxiv:
Jiahui Yu and Thomas S Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1803-1811, 2019b.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In
International Conference on Learning Representations, 2019. URL pttLlittts///peeeeiiiee
met//orunm www.mmmdddllaa.cmm
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan,
Thomas Huang. Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural architec- ture search with big single-stage models. arXiv preprint arXiv:2003.11142 2020.
Jihun Yun, Peng Zheng, Eunho Yang, Aurelie Lozano, and Aleksandr Aravkin. Trimming the & regularizer: Statistical analysis, optimization, and applications to deep learning. In International Conference on Machine Learning, pp. 7242-7251, 2019.
Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are all layers created equal? arXiv preprint arXiv:/902.011 1902.0199, 2019.
12

---

Under review as a conference paper at ICLR 2021
A PROOF FOR THEOREM 3.2
Let O be the network weights, D be the training data distribution, and æ and be the training input and label. Furthermore, let (-) be function that maps from the corresponding Pareto channel configuration (i.e., a(A) = arg ming Tx(x)) 9(-) be function that maps a certain FLOPs back to the corresponding A, (i.e. 9 (frLops aa)"), and fce be the cross entropy loss that takes channel configuration, weights and data, one can further capture the area under curve by Riemann integration over FLOPs:
Sr [c](a(g()))) 0,
A(O, 8. v)
(3)
(cE(a(g(z:)), 0, 2, -44),
where the approximation is done via Riemann sum with uniform partitions over FLOPs. As result, our formal objective for optimizing slimmable neural network is as follows:
arg min E (..y)~D( [A(,, x. y) = arg min A(0, x. v)
(4)
arg min
0, -2)
2))
(5)
arg min
(cl(ag(2z)))00..
(6)
arg min
(c 0,0,1)
arg min E
M
arg min
1 M
fcela(g(=(m)), v.) m=1
Σ
(z.
where equation and equation are done via Monte Carlo sampling and equation 5ls by equation 3
As for equation we make use that (#++1 - zi) a positive constant across i, which does not affect arg min. Likewise, we can multiply positive constant 1 to arrive at equation 7
B WIDTH PARAMETERIZATTION
For ResNets with CIFAR, has six dimensions and is denoted by C1:6 [0.316, 1], i.e., one parameter for each stage and one for each residual connected layers in three stages. More specifically, the network is divided into three stages according the output resolution, and result, there three stages for all the ResNets designed for CIFAR. For example, in ResNet20, there are 7, 6, and 6 layers for each of the stages, respectively. Also, the layers that added together via residual connection have to share the same widtth-multiplier, which results in one width-multiplier per stage for the layers that are connected via residual connections.
For MobileNerV2, C1:25 E [0.42, 1]. and therefore there is one dimension for each independent convolutional layer. Note that while there are in total 52 convolutional layers MobileNetV2, not all of them can be altered independently. More specifically, for layers that are added together via residual connection, their widths should be identical. Similarly. the depth-wise convolutional layer should have the same width as its preceding point-wise convolutional layers. The same logic applies to MobileNetV3, which has 47 convolutional layers (excluding suneeze-and-excitatinnn layers) and C1 E [0.42, 1J. In MobileNetV3, there are squeeze-and-extee excittiionn (SE) layers and we do not alter the width for the expansion layer in the SE layer. The output width the SE layer is set to be the same as that of the convolutional layer where the SE layer is applied to. Note that there is no concept of expansion ratio for the inverted residual block in MobileNets in our width optimization. More specifically, the convolutional layer that acts upon expansion ratio is in itself just convolutional layer
13

---

Under review as a conference paper at ICLR 2021
with tunable width. Also, we do not quantize the width to be multiples of 8 as adopted in the previous work (Sandler et al. [018:: Yu & Huang, 2019b). Due to these reasons, our 0.42x MobileNetV2 has 59 MFLOPs. which has the same FLOPs as the 0.35 MobileNetV2 in (Yu & Huang 2019b Sandler et al. 2018).
C WIDTH DIFFERENCES
In Figure we visualize the widths learned by PareCOO and contrast them with Slim for MobileNetV2 and MobileNetV3. Note that both PareCO and Slim are slimmable networks with shared weights and from the top row to the bottom row represent three points on the trade-off curve for Figure 2m and Figure 20
AAcc:
1.5, AMFLOPs: -1.3 (Acc.: MFLOPs:
0.5, AMFLOPs:
(Acc.: 67.7, MFLOPs: 58)
Skmm
Wishmulggius
0.5
Pam2D
0.6
0.6
Willemo
6.4
0.0
0.0
Layer
AAcc: 1.7, AMPLOPs: -2.5 (Acc.: 69.9, MFLOPs: 166)
AAcC. 1.0, AMFLOPS: 0.5 (Acc.: 70.0, MFLOPs: 85)
Rineco
0.8
94
0.4
0.0
10
50
30
Layer Index
Layer
AAcc.: 0.9, AMFLOPs: 0.8 (Ace.: 70.6, MFLOPs: 213)
4Acc.: 0.3, AMFLOPs: .0.9 (Acc.: 73.2, MFLOPs: 183)
1.0
1.0
0.8
0.6
Wid Wallemm
0.4
0.0
10
99
Layer index
Layer index
(a) MobileNetV2 ImageNet
(b) MobileNetV3 ImageNet
Figure 6: Comparing the widthh-multipliers between PareCO and Slim. The title for each plot denotes the relative differences (PareCO- Slim) and the numbers in the parenthesis are for PareCO.
D TRAINING HYPERPARAMETERS
We use Py PyTorch Paszke et al. 2019) as our deep learning framework and we use BoTorch (Balandat et al. 2019) for the implementation of MOBO-RS, which works seamlessly with PyTorch. More specifically, for the covariance function of Gaussian Processes, we use the commonly adopted Matérn Kernel (Matérn 2013) without changing the default hyperparameters provided in BoTorc Similarly, we use the default hyperparameter provided in BoTorch for the Upper Confidence Bound acquisition function. To perform the optimization of line 6 in Algorithm 2 we make use of the API "optimize acqf"' provided in BoTorch. As a reference, with a single 1080Ti GPU, one can train CeeeCCoRReee220 on CIFAR- with around 3 hours. On the other hand, with 8 V100 GPUs on a single machine, one can train
on ImageNet with 19 hours.
The training hyperparameters for the independent models are 0.1 initial learning rate, 200 training epochs, 0.0005 weight decay, 128 batch size, SGD with nesterov momentum, and cosine
CIFAR
14

---

Under review as a conference paper at ICLR 2021
learning rate decay. The accuracy on the validation set is reported using the model at the final epoch. For slimmable training, we keep the same exact hyperparameters but train 2x longer compared to independent models, i.e., 400 epochs.
ImageNet Our training hyperparameters follow that of (Yu & Huang 2019b). Specifically, we use initial learning rate of with 5 epochs linear warmup (from 0 to 0.5), linear learning rate decay (from 0.5 to 0), 250 epochs, 4e weight decay, 0.1 label smoothing, and we use SGD with 0.9 nesterov momentum. We use batch size of 1024. For data augmentation, we use the "RandomResizedCropp and "RannommHorizontallll and use random scale between 0.25 to 1. For MobileNerV3, we use the default scale parameters, Le.. from 0.08 to 1. The input resolution we use is 224. Besides scaling and horizontal flip, we follow (Yu & Huang 2019b) and use color and lighting jitters data augmentataion with parameter of 0.4 for brightness. contrast, and saturation; and 0.1 for lighting. These augmentations can be found in the official repository of Yu & Huang 2019b The entire training is done using 8 NVIDIA V100 GPUs.
APIs in PyTTorc.. For MobileNetV2 we follow (Yu & Huang 2019b)
E THEORETICAL ANALYSIS FOR TEMPORAL APPROXIMATION
The intuition behind the proposed approximation in Section 3.3 is the similarity for 0 across train- ing iterations. In an extreme case, if we hold constant throughout the training procedure, the approximation is equivalent to the original multi-objective BO. With that said, 0 changes gradually throughout training. To proceed with further theoretical understanding, we assume the network f(x 0) is L-LLipschtzz More formally,
f(a 0') ((e,0'+1) < LIJ0' 1' vO', ,
(10)
Now, consider using stochastic gradient descent to update the weights 0, i..., 0 = 0- a'g g
where g is the gradient of loss with respect to the weights and a' is the learning rate iteration t. Since f is L-Lipschitz, we have | < L. Assuming using an exponential decaying learning rate with a factor 7 < 1, we can further upper bound the functional differences across n iterations of gradient descents as follows:
[+n
f(x,0') - f(x, gct))= s
Σ im
1g S na'L.
(11)
Aligning with our intuition, the analysis reveals that larger implies poorer approximation. In multi- objective Bayesian optimization (Paria et al.] 2019). the hyperparameter is searched over stationary objectives. In our case, due to temporal approximation, our cross entropy changes over time and the change is upper-bounded by na' L. As result, we can plug such an upper bound in the regret bound analysis of Bayesian optimization (Paria al. 2019) understand how n. a, and y affect the optimality of Bayesian optimization. Specifically, we upper bound fæz 0¹) with f(x, @***) ) + na' L and use in Lemma and Lemma from from|Paria (Paria al. (2019) in Appendix B.1. With such technique, regret bound will have the following overhead in addition to the original regret bound in equation (14) of Paria et al. 2019
2a LLE|LA]
(12)
where we have utilized the geometric progression of the exponential learning rate decay. other words, without decaying learning rate, the overhead can be unbounded. This analysis reveals that larger learning rate a and n results in a worse regret bound.
F INFERENCE MEMORY FOOTPRINT CALCULATION
To demonstrate the generality of proposed PareCO, we in addition consider optimizing for the trade- off curve between prediction error and inference memory footprint. The inference memory footprint
ht / thub. www.////ahhaaeooooo Yu/sl i immable http://aaaaaaaaaaaaaaaaa////ooo/ hetp
1.43
15

---

Under review as a conference paper at ICLR 2021
critical factor when it comes to deploying deep CNNs onto resource-constraine devices such as mobile phones or cicro-controllers as motivated in the original slimmable neural network paper (uu et al. 2019). We single image per batch to calculate the memory footprint. Specifically the inference memory footprint is characterized as follows:
FM = Win x H x Cn
FM W out X H C
Weights'= Ky Ky x C x / G
(13)
Skip'= = W x Hout x Sskip
MEM= max (FM,, FM + Weights' SSkp')
where FM in and FM out denote the input and output feature map sizes of layer 1, Weights' denotes the size of the weights in layer and Skip' denotes the memory cost of storing the feature maps from skip connections. W and H represent the width and height of the feature map. K and K₁, denote the kernel size. Lastly, Cin. Cout and G denote the input channel, output channel, and the number of groups for convolutional layer I.
G NUMERICAL RESULTS FOR IMAGENET
We summarize the numbers from Figure 2m and Figure 2o in Table for future work to compare easily.
MobileNeeV2
Mo
MFLOP
Indeperdenlly trained (Sondler et 2018
Sim
ParCOO
MELOPS
Independenlly trained Howwadd
2010
Sim
FareCO
107
61.4
61.5 (-I0.T]
401
64.2
71
61.9
63.0 (-1.1)
42
65.8
65.9 (+0.1)
97
65.4
69
68.8
102
64.7
65.5 (+0.8)
73
68.1
68.8 (40.7)
136
67.1
68.2 (+1.1)
84
69.0
70.0 (+1.0)
149
67.6
69.1 (-1.5)
118
71.0
71.4 (40.4)
169
68.2
69.9 (-1.7)
121
71.0
71.6 (+0.6)
73.3
244
70.5
71.0 (+0.5)
183
73.0
73.2 (+0.2)
300
718
72.0
72.1 (+0.1)
217
75,2
75.5
73.7 (40.2)
Table 1: MobileNetV2 and MobileNetV3 on ImageNet. The number in the parenthesis for PareCO the improvements compared to the corresponding Slim. Bold represents the highest accuracy of given FLOPs.
16


Reviews:
[{"review_id": "xr74pb6OwGo", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper extends on the existing network slimming approach, computing a heterogenous width for each layer. The layer width is computed by solving a multi objective optimization problem based on pareto distribution.\n\nPros:\n\nThe paper is well motivated and well written.\nThe experiments are run across mutliple datasets and models. And also compared with existing approaches in the literature.\nCons:\n\nThe main aspect lacking in the paper is the novelty. In the three steps of the approach explained in \"Pareco\" section in Algorithm 1, the first and third step already exist in Network slimming approaches. And Step 2 - solving for 'c' is a multi objective optimization which directly exists in BoTorch library. So, I find lack of novelty and do not learn much or take away much reading this paper.\n\nAlso, by additionally solving for 'c', I do not realize what is the value add in terms of performance. In resnet-like architectures, there is no significant improvement in accuracy- as shown in Fig. 2 (a) to (l). The results in Mobilenet is slightly better as compared to NS - ~1 to 1.5% improvement, which is not significant, in my opinion. This is further established by interpreting, Fig 3(c) - for all different values of n (unless extremely high), we achieve a similar performance by the proposed approach.\n\nThus, I believe this paper is basically a hyperparameter tune over existing NS approaches, results in marginally not-so-sure significant improvement in results.", "extra_scores": {}, "extra_text": {}}, {"review_id": "UAYJIj9XqBM", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper directly extends the “Slimmable network” by using per-layer width multipliers to allow more flexible network configurations. PareCO mainly aims to optimize both width multipliers and shared weights while considering Pareto-optimal. (Accuracy & Speed) Because the problem is intractable, the authors adopt the Bayesian optimization model of CE loss and FLOPs. To improve search speed, several methods (temporal similarity, binary search) are also introduced. Compared to previous work (SLIM), PareCO consistently achieves better Pareto-optimal performance.\n\nThe paper is clearly written and has some significance. Related works are well established. Additional techniques, such as storing query for BO, binary searching, restrict to 1000 configurations, makes the proposed algorithm practical. The experimental result seems quite promising.\n\nIn my opinion, the novelty of the paper is not heterogenous width multiplier nor supernet-like training. Many works state that per-layer channel sparsity is important, and the idea of per-layer width multiplier is not new. I think this paper has some novelty, especially the “Pareto-Aware” part, but is not significant.\n\nMinor questions/suggestions:\n\n1.\tSeveral works in NAS target Pareto-Optimal architecture. Some use Bayesian optimization.  Consider adding them to the related works section... [Jin-Dong Dong, 2018] DPP-Net, [Jin-Dong Dong, 2018] PPP-Net, [An-Chieh Cheng, 2018] Searching Toward Pareto-Optimal Device-Aware Neural Architectures, [Md Shahriar Iqbal, 2020] FlexiBO, etc.\n\n2.\tIsn’t the CE re-evaluation part for every query in history too costly? Especially for a big dataset like ImageNet. Is this the reason for “20% extra overhead compared to Slim”?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EK0YMIqJrSq", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new multi-objective optimization method for slimmable neural networks and jointly optimizes network architecture and weights. However, the use of multi-objective optimization methods for networks is not new. The advantage of this paper over existing work in this regard is not clear. Besides, the key term “Pareto-aware” in the title is not clearly defined. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0iqjsjwtm2", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a multi-objective optimization approach to jointly train both channel configurations and shared weights of slimmable networks. It is theoretically proven that by minimizing the cross entropy loss for the Pareto-optimal channel configurations, the joint optimization can be approximately accomplished. Based on the objective, this paper proposes several approximation algorithms and sampling strategies to obtain a target network with better trade-off curve.\n\nExperiments are thoroughly conducted on a range of network settings and datasets. The results experimentally show that the channel optimization can improve the performance and efficiency of slimmable networks. This paper is logically organized and the motivation is based on sufficient theoretical supports.\n\nA few constructive criticisms or concerns as follows:\n1.\tThe approximation algorithms and theorem 3.2 in this paper points out several ways to approximate the joint-learning objective. However, the authors did not provide descriptions about the intuition or advantages of the proposed approximation methods. Detailed theoretical analysis on the discrepancy between approximation and the ideal objective had better to be provided.\n2.\tExperiments cannot directly support the authors’ conclusion “less over-parameterized networks benefit more from joint channel and weight optimization” because the experiments cannot eliminate the impact of other variables.\n3.\tIt might be difficult to reproduce this research and the experiment results since the search space is huge and the authors did not report the search cost (e.g. GPU-days and memory cost).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper extends the work of “slimmable networks” in that it aims to find a single set of weights suitable for multiple FLOP/accuracy tradeoff (or memory/accuracy tradeoff). The main novelty of the paper is in adapting known techniques from bayesian optimization (BO) to the setting at hand, resulting in a modified training technique. The experiments show a performance lift when compared against the original slimmable networks, as well as other approaches called “two stage” that alternate between optimizing the weights and the architecture.
The paper provides a practical approach to an important problem yielding non-trivial results. The main weakness of the paper seems to be its novelty. Although it is not possible to naively apply the multi-objective optimization with NAS techniques, the reviews seem to indicate that the innovation required to do so is not sufficient to meet the ICLR bar. This is indeed a borderline case, but given the competing papers, my tendency is towards rejecting the paper.

Decision: Reject