Differentiable Architecture Compression
Authors: Shashank Singh
Ashish Khetan
Zohar Karnin
Venue:  (2020)

Abstract: In many learning situations, resources at inference time are significantly more constrained than resources at training time. This paper studies a general paradigm, called Differentiable ARchitecture Compression (DARC), that combines model compression and architecture search to learn models that are resource-efficient at inference time. Given a resource-intensive base architecture, DARC utilizes the training data to learn which sub-components can be replaced by cheaper alternatives. The high-level technique can be applied to any neural architecture, and we report experiments on state-of-the-art convolutional neural networks for image classification. For a WideResNet with 97.2% accuracy on CIFAR-10, we improve single-sample inference speed by 2.28X and memory footprint by 5.64X, with no accuracy loss. For a ResNet with 79.15% Top-1 accuracy on ImageNet, we improve batch inference speed by 1.29X and memory footprint by 3.57X with 1% accuracy loss. We also give theoretical Rademacher complexity bounds in simplified cases, showing how DARC avoids over-fitting despite over-parameterization.

Under review as a conference paper at ICLR 2020
DIFFERENTIABLE ARCHITECTURE COMPRESSION
Anonymous authors
Paper under double-blind review
ABSTRACT
In many learning situations, resources at inference time are much more con- strained than resources at training time. This paper studies a general paradigm. called Differentiable ARchitecture Compression (DARC), that combines model compression and architecture search to learn models that are resource-efficien at inference time. Given essuucce-nntensiv base architecture, DARC utilizes the training data to learn which sub-components can be replaced by cheaper alter- natives. The high-level technique can be applied to any neural architecture, and we report experiments on statc-of-ther tf-he--rrt ahe convolutional neural networks for image classification. For WideResNet with 97. 97.2% accuracy on CIFAR-10 10, we improve single-sample inference speed by 2.28 x and memory footprint by 5.64x, with no accuracy loss. For ResNet with 79.15% Topl accuracy on ImageNet, we im- prove batch inference speed by 1.29x and memory footprint by 3.57x with 1% accuracy loss. We also give theoretical Rademacher complexity bounds simpli- fied cases, showing how DARC avoids overfitting despite over-parameteriizaion.
INTRODUCTION
In machine learning, resources inference time often much constrained than training time. For example, while neural networks for computer vision and natural language processing (NLP) are routinely trained using GPUs, trained networks are often deployed on embedded systems or mobile devices with limited memory and computational power. As another example, it is common to train a model that will be applied continuously production; while training occurs for limited time, the machine performing inference may run indefinitely, and so learning more efficient model can directly reduce costs associated with hardware or energy usage. As a result, many recent papers have studied deep model compression and acceleration. Most of these papers provide resource efficient model components (Jaderberg et al., 2014 Zhang et al.) [2016] Wu et al., 2017) or methods to prune or quantize parameters (LeCun et al. 1990; Polyak & Wolf 2015 Li et al. 2016a He ct al.
2017 Luo & Wu 2017 Luo et al. 2017 Zhuang et al. 2018).
We propose general paradigm called Differentiable ARchitecture Compression (DARC) for learn- ing in the context of constrained resources at inference-tine Rather then suggesting a specific cheap component and using it blindly throughout neural network, trying tune layer hyperparame- ters as in network pruning or quantization, our approach is inspired by Neural Architecture Search (NAS) (Zoph & Le 2016 Liu et al. 201 |ham et al. 2018 Kandasamy al., 2018 Liu al.) 2018b). DARC starts with resource- resource-intensive network design and uses training data to learn which components can be replaced with efficient alternatives, while maintaining model output quality. The resource requirement of the final model is controlled via regularization term, which can be flexibly defined depending on the objective, such as minimizing inference time or memory footprint.
DARC has clear intuitive advantage when compared to methods that quantize or prune parameters; these approaches are inherently restricted in their search space. They cannot replace layer with a structurally different layer, or replace two or more layers with a shallow alternative. As examples, it might be the case that convolutional layer cannot be pruned without hurting performance but can be replaced with a seephhwsse-spprrabe convolution, weight pruning, but could be replaced with a more efficient self-attention layer. DARC, applicd to deep networks, offers way search rich space in the context of model compression.
LSTM layer might not be amenable to
The high-level idea is to partition the network into components, and explore alternatives to these components simultaneously. Replacing all components simultaneously is crucial, as it provides a data-driven way to decide which components can be replaced with cheap alternatives. Indeed, replacing layers blindly may sacrifice too much prediction performance, be seen in Mo-

---

Under review as a conference paper at ICLR 2020
bileNets (Howard et al. 2017 Sandler al., 2018), which exclusively use depthwise separrabe convolutions; although computationally efficient, even the most accurate MobileNet model is far less accurate at ImageNet classification than, say, ResNet50.
We frame the problem of learning good replacement components as a sparse ensemble learning problem. This view allows us to draw guidelines from simpler analyzable cases, leading to simple, gradient-based learning scheme that avoids over-fitting and is fast enough to apply DARC directly large datasets such as ImageNet. This contrasts from most NAS methods, which first learn an architecture on a small dataset and then fit this architecture on the larger dataset.
networks commonly used in computer vision. By applying our tech- niques on the ResNet Architecture (He et al. 2016), we outperform state-of-the-an models on both ImageNet and CIFAR- 10 datasets, in terms of accuracy vs. throughput and accuracy vs. model size. A few results from our compression framework: for WideResNet model achieving 97.2% accu- racy on CIFAR-10 10, we improve single-sample inference speed by 2.28 x and memory footprint by 5.64x, with no loss in accuracy. For modified ResNer50 model with 79.15% Topl accuracy on ImageNet, we improve inference speed by 1.29x and memory footprint by 3.57x with 1% loss in accuracy. Both base models are publicly available from the GluonCV Model Zoo (Mod) 2018). Our experiments empirically demonstrate an intuitive observation that 'you get what you optimize for', in that models minimizing model size tend to be quite different from those maximizing throughput.
We present experiments
We note that, while our experiments are limited to image classification, DARC is applicable to any deep learning architecture, including models with recurrent cells or transformers, sufficiently modular learning algorithm, as described in Section 2 and any task with a well-defined objective function in which we would like reduce inference costs. We this work as a proof-of- concept for capabilities of the DARC framework, and the results in this paper give strong indication that DARC can be applied NLP architectures, or optimized for metrics other than those we try here (e.g. for latency on devices other than GPU, or energy consumption).
indeed any
GENERAL SETTING AND DARC ESTIMATOR
The intuition and motivation for our method starts with task of model selection. Given a task and function families corresponding to candidate models, we are interested in finding the best model type for the task. We relax this combinatorial optimization problem to more tractable differentiable optimization problem (whence the name "differentiable architecture compression"), by allowing convex combinations of these candidates. This can be thought of as constrained form of ensemble learning in which weights are restricted to represent a convex combinations of individual learners.
Then, we posit a budget constraint: each model type has an associated cost (e.g., memory consump- tion or latency), and the overall cost of the ensemble is the sum of costs of the used models. Our task is then learn a convex ensemble over a subset of candidates, with total cost within budget. We now formalize this approach, with modifications to address technical challenges as they arise.
In the sequel, for any positive integer J, [J] = (1, .... J) denotes the set of positive integers at most J, and A == {I E [0, 1] Eje( 2; = } denotes the probability simplex over J elements.
Consider the conventional supervised learning setting, in which we have an i.i.d. training dataset (X1, Yi), ..., (Xn, Yn) IID P(x.y) from some joint distribution Pix,Y) on X' x V. Fix loss function L: : R x y + [0, oo). and a hypothesis class H of RRvalued functions. We would like to leam function h: X + R, h E H. that minimizes the risk R(h)= == ((x,)) [C (h(X), Y)]-
The usual empirical risk minimization (ERM) estimator is TIRRM == arg min R((h) where, for any hypothesis h E H. R(h)- 1 resource-constrained objective, we impose a few structural assumptions on our hypothesis class H:
x E L (M(X,), Y2) denotes the empirical risk. To derive our
(A1) H. = Conv UsE|o] H, is the convex hull of a union of J classes Hı...., H.J. (A2) Each class H. class has known cost C2 2 0 of using hypothesis hG H, at test-time.
(A3) Costs are additive: hypothesis Σ ayıy E H has cost Ce,(a) = Ej= Cy4(a)>0)-
(A4) We have known budget B ' 0 for the final model test-time.
2

---

Under review as a conference paper at ICLR 2020
As show in Section these assumptions arise naturally in architecture compression. Given assumptions equation Al equation A4 the constrained ERM estimate is ğ EjEly āzhţ where
(āo: ho) = arg min R
Σ aghy JENJ]
subject to Ct (a) S.
(I)
nEA oEA-MieEH
The above estimator is difficult (NP-hard) compute, due the non-smooth, non-convex budget constraint Ct. (a) & B. Since this constraint bounds the to norm of (weighted by C), the usual remedy would be to relax the constraint to one on the & norm of a (weighted by C), namely Ce, (a) == ΣjEμa Gjaz I B. Unfortunately, due to the constraint that a lies in the probability simplex A (which implies ΣjEμ Oj =1) the constraint is insufficient induce sparsity on O. Fortunately, sparse optimization on A' is well-studied, with many solutions proposed (Pilanci jet al. 2012; Kyrillidis et al., 2013 Li et al., (016bb). Due to ease implementation, we adopt simple but effective solution proposed by (Kyrillidis al. 2013), which involves alternating gradient updates with a projection operation PA. R A,, given by Pa)(a) ||o II where
a+ (max(0, } ..., max a,}) R.Y.
PA easy to compute, enforces the simplex constraint a E A exactly, and induces sparsity on For an intuition of how this works, one can note that V|
a, so that the update a/||a/] = a - (1 1/||a[|11 [|] can be viewed as a gradient step for minimizing - -|aa]/// with adaptive step size (1 1/ [a]l1). As technicality,
note that the projection Pa (a) is undefined when a E R__ has no positive components. However, for realistic gradient step sizes this never occurs, since, after each gradient update, ΣSE[n] Oj
O(n).
Finally, natural initial point for our procedure is one where (a) B. hence we re- express the constraint Ce, (a) < B as penalty (a), Since the value of A correspond- ing to is not known priori, we iteratively increase until the solution of the optimiza- tion problem satisfies the budget constraint. The resulting DARC procedure is shown Algo- rithm [ We note that the "stopping criterion" for the inner loop can as simple as fixed num- ber of training epochs (as in our experiments), or a more sophisticated early-stopping criterion.
Algorithm 1: DARC algorithm for general hypotheses
Data: Training Data {(,,, Y:) J candidates hi,an,.... j,,,, with initial parameters Wi, Wj and costs CI, .... Cj 2 initial cost penalty parameter An > budget B. Result: W1, .... Wj such that h = ΣjE(J ayhj,, has small risk R(h) and cost Ce, (a) S B <- (1/J.... 1/J), while (a) > B do
while stopping criterion
met do (a, Wi. WJ) + (a, Wt, .... W)) 17 (Va.e ( ESEpyyashhou) Cei(a)):
+ Pas (a):
end
2A
end
3 APPLYING DARC TO DEEP NETWORKS
DARC can be applied in myriad of ways compress deep neural networks. In all of these ways, the basic premise is to intelligently replace components of the network with cheaper components.
Consider Neural Network (NN) with L layers. For layer É, let W; be the parameters of the layer, and gr be the function mapping inputs and parameters to the output (in layers having no parameters, W can be empty token). For example, for fully connected layer, W is matrix, the input is vector, and g is the matrix- vector multiplication function. We can write the NN as a function:
f(x) gu (Wz. (W-1 ... 91 (W1,,)) To apply DARC, consider set of replacement candidates (gr.2, Wz.). (ge,J, J. We.J, for each layer & (with gt,1 Wr,1 denoting the original function and weight the layer). For each candi- date j in layer DARC takes as input associated cost Ct 2 0. Examples of such costs include
3

---

Under review as a conference paper at ICLR 2020
parameters count, FLOPs, or latency, which are usually easy to calculate or estimate experimentally. Applying DARC to neural network compression then involves four main steps:
1. Layerwise Continuous Relaxation: First, we replace each ge, We with weighted average
(eW, ax, = Ejz It. (W 2)), where ag E A The original network is replaced by f(z) = (W, T1 (W (1.))......
2. DARC Model Initialization: Before training the DARC model, we need to initialize the weights and the parameters of the compression candidates. We initialized the a parameters as uniform vectors a₂ (1/Jg, L/ ..., 1/Jz). The other option we considered weight on the original candidate (ag = (1, 0, ..... ........,) 0)), so that the initial model was equivalent to the original model being compressed. However, this makes the gradient of the loss 0 with respect to all parameters of the compression candidate, preventing these from training. Furthermore, the non-convex regularization discourages the weights of ag shift towards a value that makes use of the compression candidates. As for candidate parameters, we initialized each compression candidate to mimic the original layer, which we know gives good prediction results. In some cases, this can be done analytically (e.g. via PCA for lower dimensional conneccd layers); more generally, this can be done via SGD, training the new candidate to minimize squared loss between its outputs and those of the original layer It, (W,,,, Since this is only for initializa- tion, it suffices to use small training sample and crude optimization procedure.
to put all
3. Training the Relaxed Model: We minimize the empirical risk, simultaneously over the mixture weights (as) and the candidate weights (Ws) as described in Algorithm
4. Selecting Sub-Model: As discussed above, for sufficiently large X, Algorithm
I converges to
a solution with small (weighted) lo norm; i.c., ae will have a small number of non-zero entries. Thus, we remove candidate gr,j (and its weight ore.)) from the network if oke,j = 0.
During optimization, we jointly optimize a and the model parameter on the same data, contrast- ing from other gradient based NAS approaches (Liu et al. 2018b) that split data into two training sets, optimize model parameters on one and a weights on the other. In Section 4 we analyze the Rademacher complexity of procedure in simple setting and show that under the condition that the original model class defined by gr,1 is richer than the alternatives, optimizing all parameters jointly does not hurt generalization guarantees when compared the original optimization objective where J2= = 1. Unlike for NAS, this condition holds naturally for model compression.
Efficient Approximate Convolutions Computation in most deep networks used in computer vi- sion problems, such as image classification, image segmentation, and object detection, is dominated by convolutional layers. This has motivated several papers on efficient approximations to convolu- tion. such as depthwise-separabe convolution (Jaderberg et al.) 2014 Zhaang et al.) [0166] Howard et al. 2017). bottleneck convolution (Sandler et 2018), and shifts (Wu et al. 2017).
In a standard convolution layer we have k x l filters for every input and output channel. Denot- ing the output channels by Y, and the input channels by X,. the i'th output channel is defined as Fi... Here Fi, is the appropriate filter and is the convolution operator. Restrict- Y, = -,,,, Xj -, ing the discussion to the setting where the number of output and input channels are the same, a fully-grouped convolution is a more constrained alternative in which the filter is h x but each output is computed based on single input channel. A depthwise-separable convolution consists of a full-grouped convolution followed by a standard 1 x 1 convolution. In most setting this op- eration requires less compute and memory resources. A shift layer is even cheaper alternative to septhwise-separabbe where the F,'s are fixed and have only a single non-zero element, resulting in computational complexity equivalent to a single 1 x convolution. We use DARC to compress CNNs by considering alternatives from among the above options, for each convolution layer.
4 THEORETICAL RESULTS
Here, we discuss generalization power models learned by DARC. We restrict our attention to the simple case of learning an ensemble of models; described below, the result has implications for our algorithm for training DARC. This setting actually applies not only for DARC but also for various NAS methods such DARTS (Liu et al.) 2018b) or ENAS (Pham et al. 2018). Indeed, these methods aim to choose one out of several options in each layer. While these methods differ in how this ensemble is learned, our generalization bound is independent of the learning technique.
4

---

Under review as a conference paper at ICLR 2020
Recall that DARC learns convex combination of functions from J classes Hi..... H, Here, analyze generalizability of this process via Rademacher complexity Bartlett & Mendelson (2003):
Definition 1 (Rademacher Complexity). Let H be a class of functions mapping - I& and let E N. Denote by X (X1,..,, Xn) IID samples from X. Let uniform random vector in {-1,1}". The Rademacher complexity of H is R(H) ,x,xT suPMCh 7. Eie(m) ach((xn.))
It is well known that the Rademacher complexity of a class H. is equal to that of the convex hull of H. Based on this fact, for h (hi...., bound on the difference between true risk R and empirical risk R:
........,)))) h))
E IT;=1 H,, h Σi= azhy, we have generalization
Theorem 1. Suppose we jointly estimate a, a,,,,,,,, hJi i.e. (a,hi....,Ts)
argmin, E aN, Ei= [(x h(X.), Yi). Let ((h(x),)) = 1((0))/ be 0-1 loss. Then, w.p. 2 - 6 (over training samples), R (a. - Refo (a. R)) s R8 (Usern) H.) + log 1/6
Since Theorem follows from standard Rademacher generalization bounds (e.g.. (Bartlett & Mendelson 2003 Theorem 5(b))), we omit its proof. According to Theorem | generalization error depends on standard / /log / term and Rademacher complexity of the union of classes H1...., H.. If Hi,..., H. are diverse, this union can be quite rich and so R (U))),)HH) might be large, leading to overfitting. However, consider an example where Hi is the family of full convolu- tions, H2 is the family of depthwise-separabl convolutions, H4s is the family of sparse convolutions, etc. Here, we actually have My = UJEμi Hy; thus, Rademacher complexity is simply that of the original model (i.e., with J = 1). Formally:
Corollary 1. Suppose that every sub-model is contained in Hi: i.e. H2, .... Hy C. H4. Then, the Rademacher complexity of DARC is at most that of the original model: R (H) S 9 (H1).
Even if Hi ç Uje(s) Hl, in the setting of model compression the alternative families Hj. > 1 are cheaper replacements for H2 suggesting that R(H,) R(U)e Hyy) This observation motivates shows us that there is no need to split the training set, train the model
our learning framework parameters on one split and the control parameters on the other, as in many NAS papers (Cai et al.,
2018
Liu et al., 2018b).
We note that does not motivate a change in the learning framework of NAS. A key difference be- tween NAS and DARC that candidate models HI, H. in NAS are intentionally diverse; their union is much richer than any individual. This translates to large A (UsE(ı) Nt,). motivating need to avoid jointly optimizing a and model parameters on a single training set. When keeping validation set aside for training a, given the limited number of update steps typical in NAS pa- pers, generalization eтror may be closer to the setting of fixed a, wherein Rademacher complexity bounded by E, R(H,) (Cortes al. 2014). potentially much smaller than in Theorem
5 EXPERIMENTAL RESULTS
We applied DARC to a number of deep networks from the GluonCV Model Zoo Mod 2018) for image classification on the CIFAR-10 and ImageNet (Russakovsky et al. 2013) datasets; this see- tion presents quantitative and qualitative results. We report three performance metries (model size, single-sample throughput, and batch throughput), and we specifically considered using DARC to minimize two of these (model size and batch throughput). Below, "DARC(S)" denotes DARC with a model (S)ize penalty, and "DARC(T)" denotes DARC with batch (T)hroughput penalty.
Choice of Compression Candidates For each convolutional layer, besides the original (full) con- volution, we considered 3 compression candidates: a (fully-grouped) seppaaise-separable convolu- tion with 3 x 3 kernels (abbreviated henceforth as "3x3DS"), full convolution with 1 x kernels ("IxIFC"), and a 2-layer candidate consisting of 3x3DS layer followed immediately by a 1xC layer ("3x3+Ixll Ix1"). In ResNet50, which was already implemented using bottleneck convolutions (He) et al. 2016) consisting of a sequence of 1, x 3, and convolutions, the entire bottleneck
5

---

Under review as a conference paper at ICLR 2020
sub-network was treated as a single component (i.e., all 3 convolutions were replaced with single block from the above mentioned alternatives). We intentionally limited the choice of alternatives to maintain a simple system that enjoy high throughput without special implementation. Our precise choice of alternatives is motivated by them already being an established component for deep networks, proven to work in some settings even without architecture search.
DARC Training Details To maximize fairness when comparing with models in the GluonCV Model Zoo (Mod) 2018), most aspects of training DARC were based on the training scripts provided publicly by the Model Zod] Due to space constraints, these implementation details are decrihed in Appendix A; few specific differences from these scripts are described below.
Student-Teacher Initialization: As described in Section 3 we initialized compression candidates to mimic original layers using student-teacher training. While this training had to be performed sepa- rately for each compression candidate in each layer of the original model, since each compression candidate has few parameters, each candidate's 's training converged quite quickly. Thus, in CIFAR-10 10 experiments, we simply ran epoch of the entire training dataset; in ImageNet experiments, we ran only 1000 batches. A relatively large step size of 0.1 was used, since teacher-student initialization only for initialization and fine-tuning can be performed during model-selection
Main Training Phase: As noted in Algorithm [ training occurred in blocks of epochs (20 epochs/block for CIFAR-10, 10 for ImageNet), with the compression penalty À increased after each block, to obtain spectrum of compressed models. For each dataset, penalty type, and model size, the initial value of À was selected to roughly balance the orders of magnitude of the empirical loss and the regularization term at the beginning of training. After each block, we: 1) remove candidates j with 0, = 01, 2) save (for evaluation later) a copy of the DARC model, in which, in any layer with multiple non-zero a entries, all but the most expensive remaining candidate are removed, and 3) decrease learning rate 1) and increase compression penalty (each by a factor of 2).
This "iterative compression" was repeated until only one compression candidate per layer remained in the DARC model. Finally, each saved model was fine-tuned for 20 epochs using only prediction loss (i.e., with λ 0). This procedure enabled to obtain sequence of compressed models progressively increasing compression levels. Moreover, this "warm-starting improved compression speed since we only perform a total of 30 epochs per value, rather than the 100 epochs needed for convergence at high levels of compression.
5.1 CIFAR-10 RESULTS
By all metries, DARC gave the best results when applied very wide models such as the WideRes- Net series (specifically, the WideResNet16_10, WideResNer28_10 and WideResNer40.3 _ mod- els (Mod) 2018)). Moreover, unlike results on other ResNets, results on WideResNets were rel- atively similar for both DARC(S) and DARC(T); both versions of DARC selected the 3x3+1x1 candidate for every layer. The reason for this is that very wide convolutions in WideResNet models can be replaced by depthwise separable convolutions with essentially no loss in accuracy, and im- provement in throughput for both batches and for single samples (1.4-1.6x for each) and memory footprint (4-6x). In the case of WedeResNer16 6 10, DARC produces a model with latency (single sample throughput) comparable to on of the fastest CIFAR-10 model (ResNer20_ v1; see Figure I), while having accuracy within 0.8% of the best model DesNNeXt29116x444 3.2% accuracy points above the performance of ResNet20_v1. For complete CIFAR-10 results see Appendix tables 3-4.
5.2 IMAGENET RESULTS
For ImageNet we compressed several ResNet models. To present the size compression results we provide Table 2 comparing accuracy change as function of parameter reduction. We compared to previous published results compressing ResNet50 on ImageNet. To our knowledge these are the state-of-the-art results among those compressing ResNet50 on ImageNet. For compression as aggressive as 3x, we incur an accuracy drop of 0.86% while the baseline suffers a drop of 3.26%. Table [ gives throughputs of our compressed ResNet34 and ResNet50 models and throughputs of state-of -fte-o-ftthhrrrr competing models, on identical hardware; compressed models outperform these competing models, due to our initialization based on pre-traince ResNet34 and ResNet50 models.
train imagenet. train minnn mi xup ci far Far10.py

---

Under review as a conference paper at ICLR 2020
While further details, including other compressed versions, available in the Appendix (Figures and and Table 5), Tobpe@co
compares prediction performance of DARC(S) with that of state-of-the- art network pruning techniques applied to ResNet50 on ImageNet, various compression levels. In the light (1 x) compression regime, the result of (Zhuang et al. 2018) outperforms ours. The work of (Zhuang et al. 2018) complements ours, in that their novelty in warm-starting the compressed alternative not only to mimic the original, but also to be informative w.r.t. the label. Since this work does have an architecture search component, we suggest that future work combine this clever warm--tal with an architecture search component such as DARC. Once the compression becomes more aggressive, DARC outperforms the baseline, likely since in that regime (smaller network with more training epochs), a good architecture is more important than good warm-start.
Model
Accuracy (Top-1)
Throughput (224 x 224px/s)
ResNet34 Mod 2018
74.4
205
ResNe150 Mod
2018
79.1
148
ProxylessNAS (Cai et al.
2018
75.1
196
MobileNetV2 (Sandler el al. 2018)
72.0
1 164
Nets (Tan al. 2019)
74.0
164
DARTS Liu et al. 20186)
73.1
(ours)
73.9
234
DARC-ResNet50 (ours)
76.8
208
Table I: Comparison of ResNec34/ResNet50 compressed by DARC(T) with models from state-of- the-art competing methods (as reported by Cai et al. (2018) in an identical runtime environment).
5.3 DISCUSSION OF COMPRESSED ARCHITECTURES
In both ResNets and WideResNets, model size tends to be dominated by a small number of the largest layers in the model, and is relatively insensitive the depth of the network. Thus,
ResNet-50
Compression
Top1/Top5
Disc
1.51 x
+0.39/ 00.14
DARC(S)
1.63x
-0.51/-0.16
ThiNet
2.06x
-1.87/-1.11
significant compression can be achieved by re- placing these large layers with 3x3+1x1 can- didates, which offer compression of nearly 9x (for 3 x convolutions). Since the sizes (num- ber of convolutional kernels) of ResNet layers increases from bottom to top (i.e., from input to output) DARC(S) tends to first replace the top-most layers (i.e., layers closest to the out- put) of the network with 3x3+1x Ixl candidates, proceeding towards the bottom of the network as the compression parameter À increases.
Disc
2.06x
-1.06/-0.61
DARC(S)
2.05x
-0.63/-0.18
Disc
2.94x
-3.26/-1.58
DARC(S)
3.01x x
-0.86/
DARC(S)
3.57x
-0.99/
DARC(S)
6.14x
-6.09/ 4.85
Table 2: Comparison of DARC and state-of-the- art pruning methods, Disc (Zhuang et al. 2018) and ThiNet (Luo al. 2017). when compress- ing ResNet-50 trained ImageNet for ResNet-50. (S)ize denotes models optimized by DARC to minimize model size. Top1/ accuracy of pre- trained model were 79.15% respectively.
In contrast, model latency is relatively uni- formly distributed throughout the layers of the network - the time taken to compute each con- volutional layer scales only weakly with the number, size, and grouping of filters. Thus, in ResNets (but not in WideResNets) 3x3+1xl can- didates, which replace 1 layer with smaller layers, tend offer little or no benefit in throughput; of the compression candidates we considered, only the smallest (3x3DS) candidates offer significant acceleration (typically of about 2x) over full convolutions. As result, the acceleration offered by DARC(T) scales primarily with the number of layers that can be replaced by 3x3DS layers. More- over, the replaced layers tend to be scattered throughout the network (rather than clustered near the output of the network). As noted earlier, in WideResNets, each layer is large that the 3x3+1x1 candidate is much faster than full convolution, and DARC(T) selects this candidate for all layers.
Overall, we see that the optimal compression strategy depends whether one is optimizing for size or for speed. This implies that some sort of intelligent replacement of components, as DARC, is needed to obtain reliable performance improvements (as opposed to, say random baselines that have been shown to perform well in pruning). This parallels recent work (Cai et al. 2018) showing (for NAS) that optimizing for speed on different hardware (GPU, CPU, mobile) leads different

---

Under review as a conference paper at ICLR 2020
models. While we focused GPU throughput, note that the speedup of depthwise-sepiann convolution over full convolution is typically larger on CPU and embedded devices than on GPU devices. Thus, DARC should also produce efficient architectures for these alternative hardware.
RELATED WORK
Work on compressing deep networks has abounded in recent years, with diverse approaches includ-
ing pruning (LeCun et al. [190; Polyak & Wolf 2015 Li et al.) 2016a He et al.] 2017;: Luo &
Wu 2017: Luo et al. 2017 Zhuang et al. 2018). low-rank factorization (Jaderberg al. 2014
Zhang et al., 2016 Howard et al., 2017). fast approximate convolutions (Bagherinezhad et al. 2017
Wu et al. 2017). knowledge distillation (Hinton et al. 2015, Romero et al. 2014), and quantiza-
tion (Gong et al. 2014 Han al. 2015 Zhou et al. 2017 Lin et al. 2016); (Chegg et al., 2018)
survey common approaches. In contrast, DARC searches a richer space of alternative models and has the ability to replace multiple components by single one (e.g. replacing bottleneck sub-network of 3 convolutions with a single convolution). Indeed, though beyond the scope of this paper, DARC can incorporate complement many of these methods. Our initialization of compression candidates by mimicking the original layer is also reminiscent of knowledge distillation.
Another closely-related line of papers concerns Neural Architecture Search (NAS) (Pham et al.
2018 Kandasamy et al., 2018 Liu et al. [2018b] Gordon et al. [018 Cai et al. 2018). The most relevant papers are (Liu al. 2018b), (Gordon al.) 2018), and (Cai al. 2018). who all use sparse linear component weighting scheme similar to DARC. (Liu et al.) (0188) focused on pure architecture search, in which the goal simply find an architecture maximizing prediction per- formance, without consideration of inference-time efficiency. (Gordon et al., 2018) do not aim to replace layers with general alternatives but rather discover parameters a data-driven way; their experiments are restricted to results in channel pruning. Recently, (Cai et al. 2018) performed NAS with a latency regularization term similar to ours.
Our methods differ from these NAS papers in two main ways (summarized in Table 3). First, our architecture search is guided by an established base model that was already tested and proven use- ful. This distinction allows simpler and more efficient learning scheme (motivated in Section 4 that avoids iterating between training sets, to optimize model and a parameters. Furthermore, start- ing with pre-trained model allows us not only reach effective architecture, but start the weight parameters. As evidence of the advantage of starting with a pretrained model, our compressed model ResNet50(() on ImageNet has Topl accuracy 2 3% more than models ob- tained by these NAS papers; thus, it seems that starting architecture search with a highly accurate base model can improve the efficiency/accuracy trade-off of the learned model. Another distinction from gradient- based NAS results is our use of sparsity- sparsity-induiing regularization. Previous methods make the choice between the candidates via softmax layer; this restricts the output to be convex combination of inputs without optimizing sparsity. Since these methods also aim to find sparse combination, they might benefit from a non-convex regularization term as DARC.
Table 3:
Comparison of DARC with NAS methods. "SGD" denotes stochastic gradient descent. "RL" denotes reinforcement learning.
Method
Optimizer
Efficient architecture
Resists Overfit- ting
Exploits pre- training
Iterative com- pression
DARTS (Liu et al. 2018b)
SGD
X
X
ENAS Pham et al. 2018)
RL
X
X
ProxylesNNAS Ca et al. (2018)
SGD/RL
x
X
x
MNasNet (Tan et al. 2019)
SGD
V
FBNet (Wu et al. 2019
SGD
/
X
X
DARC (proposed method)
SGD
/
7 CONCLUSIONS AND FUTURE WORK
We have shown that simple DARC implementation, with only depthwise-separable approx- imations as compression candidates, can be used to compress large state-of ofate-of-the-art deep networks, improving inference speed and memory footprint. Intelligently making only some layers of the net-
8

---

Under review as a conference paper at ICLR 2020
work depthwise -eephwise-separabee results in compressed models with much better predictive performance than simply making all convolutions depttwiss-separabbl,, as in (Howard et al. 2017).
While depthwise- sepaarable convolutions casily implemented in existing deep learning packages and already offer substantial compression, future work may benefit from more sophisticated approx- imate convolutions, with efficient implementations. For example, shift operations (Wu et al.) 2017) are promising, as they require no stored parameters and replace the slow multiplications convo- lution with fast indexing. Another venue worth pursuing is compress model into shallower version. Although there are a few ways this could be attempted, such as an Identity candidate or replacing entire blocks of layers, it unclear which technique would work best. Finally, we hope to apply DARC to models other than CNNs (e.g., models with recurrent cells or transformers).

---

Under review as a conference paper at ICLR 2020
REFERENCES
Gluon Model Zoo: Classification.
https::/ggius ov. mxnet 1o/model 200/
classi fication. html, 2018. Accessed: 2018-12-10.
Hessam Bagherinezhad, Mohammad Rastegari, and Ali Farhadi. Lenn: Lookup-based convolutional neural network. In Proc. IEEE CVPR. 2017.
Peter I.. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. J. Mach. Learn. Res., 3:463-4822 March 2003. ISSN 1532-4435, URL
http://dde acm. org/ htt//iitatio witaaiii...........eio cfm dimmiid--4919 944919. 944944
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search target task and hardware. arXiv preprint / /et::/0322.., 2018.
Yu Cheng, Duo Wang. Pan Zhou, and Tao Zhang. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Signal Processing Magazine, 35 (1):126-136, 2018.
Corinna Cortes, Mehryar Mohri, and Umar Syed. Deep boosting. In International Conference on Machine Learning, pp. 1179-1187, 2014.
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional net- works using vector quantization. arXiv preprint arXiv: /4/2.6/ 15, 2014.
Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang, and Edward Choi. Mor- phnet: Fast & simple cesource-constraie structure learning of deep networks. In IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2018.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In International Conference on Computer Vision (ICCV), 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253/1 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1744441704..4661, 2017.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. In Proceedings of the British Machine Vision Conference. BMVA Press, 2014.
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric Xing. Neural architecture search with bayesian optimisation and optimal transport. arXiv preprint 1802.07/ 2018.
Alex Krizhevsky and Geoff Hinton. Convolutional deep belief networks on cifar-10. Unpublished manuscript, 40(7), 2010.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Anastasios Kyrillidis, Stephen Becker, Volkan Cevher, and Christoph Koch. Sparse projections onto the simplex. In International Conference on Machine Learning, pp. 235-243, 2013.
10

---

Under review as a conference paper at ICLR 2020
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-699869 0088605,11 1990.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint wxXiv:160.008711 2016a.
Ping Li, Syama Sundar Rangapuram, and Martin Slawski. Methods for sparse and low-rank recovery under simplex constraints. arXiv preprint arXiv: arXiv:/605.00500 /605.00577 2016b.
Darryl Lin, Sachin Talathi, and Sreekanth Annapureddy. Fixed point quantization of deep convolu- tional networks. In International Conference on Machine Learning, pp. 2849-2858, 2016.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceed- ings of the European Conference on Computer Vision (ECCV), pp. 19-34, 2018a.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint 0r06001806.00055,, 2018b.
Jian-Hao Luo and Jianxin Wu. An entropy-based pruning method for emn compression. arXiv preprint vrrXiv:776.000 05791, 2017.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In 2017 IEEE International Conference on Computer Vision (ICCV), pp- 5068-5076. IEEE, 2017.
Hicu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv: / 008.000...22.. 2018.
Mert Pilanci, Laurent E Ghaoui, and Venkat Chandrasekaran. Recovery of sparse probability mea- sures via convex programming, In Advances in Neural Information Processing Systems. pp. 2420- 2428, 2012.
Adam Polyak and Lior Wolf. Channel-level acceleration of deep face representations. JEEE Access, 3:2163-2175, 2015.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.65501 2014.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-22) 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In 2018 JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4510 4520. IEEE, 2018.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. MnasNet: Platform-aware neural architecture search for mobile. In CVPR, pp- 2820-2828, 2019.
Bichen Wu, Alvin Wan, Xiangyu Yue, Peter Jin, Sicheng Zhao, Noah Golmant, Amir Gholaminejad, Joseph Gonzalez, and Kurt Keutzer. Shift: A zero flop, zero parameter alternative spatial convolutions. arXiv preprint 17X10:1711.08141, 2017.
Bichen Wu et al. FBnet: Hardware-aware efficient convnet design via differentiable neural architec- ture search. In CVPR, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David I Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXin: 2017.
Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE transactions on pattern analysis and machine intelligence, 38(10):1943111 2016.
11

---

Under review as a conference paper at ICLR 2020
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantiza- tion: Towards lossless cnns with low- www-prcission weights. arXiv preprint arXiv:1702.03044, 2017.
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrimination-awanl channel pruning for deep neural networks. In Advances in Neural Information Processing Systems, pp. 983-894,20 2018.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint arXi::1611.01578, 2016.
A DARC IMPLEMENTATION DETAILS
In this section, we provide further details about the implementation of DARC used in our experi- ments.
Environment Details We implemented DARC Apache MXNet 1.3.1 using Python 3.6 and CUDA 9.0. Experiments were run on AWS EC2 p3.8x 3..xlarge and p3. 16xlarge machines, which re- spectively features 4 and NVIDIA Tesla-V100 GPUs. CIFAR- 10 models were each trained with I GPU. Smaller ImageNet models (8esNet18 and ResNet34) were trained using 4 GPUs, while the larger ResNet50 was trained using 8 GPUs.
Training Details Following the original script used to train models in the GluonCV Model Zoo, we utilized mixup training (Zhang et al., 2017). and optimized cross-entropy loss with Nesterov accelerated stochastic gradient descent (NAG) with (default) momentum parameter 0.9. As noted in the main paper, for student-teacher initialization, we used relatively large learning rate rj = 0.1. Thereafter, for model-selection, we began with initial learning rate of 0.01, which then halved after each traning block.
To minimize training time, training batch sizes were selected to be large possible without exceeding GPU memory during training. This resulted in batch sizes (per training GPU) of 256 for ResNets CIFAR-10, 128 for WideResNets CIFAR-10 10, 64 for ResNet18 and ResNet34 on ImageNet, and 32 for ResNet50 on ImageNet.
For each dataset, penalty type, and model size, the initial value of X was selected to roughly balance the orders of magnitude of the empirical loss and the regularization term at the beginning of training. For CIFAR-10 experiments with size penalization, the initial value of the À compression penalty was set to À = 10-5 x L, where L is the number of layers to which DARC was applied (i.c., the number of full convolutions in the original model). For CIFAR-10 experiments with latency penalization, we used X = 10 x L. For ImageNet experiments, we used = 10 x L with size penalization and L, 10 x L for latency penalization.
A.1 MEASURES OF MODEL PERFORMANCE
Computational Performance As estimate of model size, we report the size (on disk) of the parameter file created by MXNet when saving the model; this correlates well with both the number of parameters in the model and the footprint of the model in RAM or GPU memory. Since through- puts are inherently noisy, we report average inference times over 1000 batches. Though multiple GPUs were used for training DARC, all inference times computed using single Tesla V100 GPU. We used batch size to estimate single-sample throughput and batch size 256 to estimate batch throughput. The cost of each compression candidate (i.e., number of parameters for DARC(S) or latency for DARC(T)) was calculated or estimated based on the student model trained during initialization.
Prediction Performance On 100, we used standard ("Topl") prediction accuracy. On Im- ageNet, additionally used (TTpp5") accuracy, the fraction of test images for which the correct label is among the five labels considered most probable by the model. We note that these are the standard performance used for these datasets (Krizhevsky & Hinton 2010 Krizhevsky al. 2012
12

---

Under review as a conference paper at ICLR 2020
B SUPPLEMENTARY RESULTS
This section provides detailed numerical results of our experiments:
Throughput as a function of batch size
for original models (solid lines) and compressed models (dotted lines)
ResNet20_v v1
Throughout (CIFAR-10 Image/sec)
ResNet20_ v2
WideResNet16 10
WideResNet28 10
10*
WideResNet40 8
103
100
10
102
103
Batch Size
Figure 1: Throughputs of original and compressed WideResNet models, and original ResNet20 models, various batch sizes.
13

---

Under review as a conference paper at ICLR 2020
Model
Top-1
Model Size
Throughput (1 Im/batch)
Throughput (256 Im/batch)
ResNet20_v1 (0)
92.9
1.05
709.62
53020.76
ResNet20 vl (S)
92.1
0.90
682.85
53078.90
ResNet20 vl (S)
92.0
0.78
706.91
52822.23
ResNer20_vl (S)
90.2
0.60
699.80
53206. 18
ResNet20_v1 (S)
89.0
0.26
723.89
53058 ,22
ResNet20.v22())
92.7
1.05
792.03
52889.67
ResNer20.v2 (S)
92.1
0.85
665.98
53619.09
ResNe:20.v2
91.4
0.67
671.05
54148
ResNet20.v2 (S)
91.0
0.48
700.55
54144.82
ResNe:20.v2 (S)
90.3
0.26
663.76
56124.30
ResNe:20. v2 (S)
89.1
0.25
739.73
55545.69
ResNe156_vl (0)
94.2
3.31
345.89
18930.77 70
ResNet56_v1 (S)
93.4
1.60
369.71
19974.62
ResNet56.v1 (S)
93.3
1.34
348.71
20330.08
ResNet56 v1 (S)
92.9
1.18
347.43
20711.30
ResNe:56 v1 (S)
92.8
0.60
354.43
21155.00
ResNet56_vl (S)
92.7
0.59
343.84
21648.
ResNe:56 (0)
94.6
3.30
395.35
19266.32
ResNe156_v2 (S)
93.4
1.12
382.69
20138.1 11
ResNe156_v2 (S)
93.2
1.12
360.47
20587.73
ResNe:56_v2 (S)
93.1
0.93
353,93
20874.39
ResNe156_v2 (S)
92.9
0.71
355.93
20993 .0
ResNet56_v2 (S)
92.7
0.63
335.27
21578.8 .89
ResNet56 v2 (S)
92.5
0.59
330.72
22315.29
ResNet110_w1 (0)
95.2
6.68
239.89
9704.98
ResNet1 (S)
94.1
2.84
213.33
10391.16
ResNetl (S)
94.1
2.38
198.18
10704.93
ResNetl (S)
94.1
1.10
209.27
10858.91
ResNer1 (S)
93.9
1.10
203.75
11084.
10_v1 (S)
93.7
1.09
191.06
11245.70
ResNet110 (O)
95.5
6.68
220.21
9942.27
ResNerl10_v22e (S)
94.3
3.13
210.58
10393. 19
ResNetl1 (S)
94.3
2,08
209.11
10523,53
ResNetl (S)
94.2
1.97
199.03
10568.06
ResNe:l (S)
93.9
1.21
214.44
10740.52
ResNet1 (S)
93.7
1.14
219.05
11029.77 722
ResNetl v2 (S)
93.7
1.09
209.54
11320.08
WideRes.Net16_100 (O)
96.7
65.34
345.19
4914.29
11deResNNe166 10 (S)
96.6
17.11
608.58
6932.60
WideRessNet16 (S)
96.5
16.23
631.89
7546.51
WideResNee228 (0)
97.1
139.24
186.63
2525,49
WideResNNr228 (S)
97.2
26.42
423.48
3918.92
WideRessNer28100 (S)
97.1
24.67
425.66
4311.35
WideResNet40.88 (O)
97.3
136.47
156.06
2617.89
WideResNet40 8 (S)
97.3
21.31
358.66
4258.24
Table 4: Results of applying -ize-penalized DARC CIFAR-10 models. (O)riginal Denotes original model taken from the GluonCV Model Zoo (Mod) 2018). (S)ize denotes a model optimized by DARC to minimize model size. (T)hroughput denotes model optimized by DARC maximize model throughput. Model Size is provided in MB on disk. Throughput numbers are provided in 32px x32px images/second.
14

---

Under review as a conference paper at ICLR 2020
Model
Top-1
Model Size
Throughput (1 Im/batch)
Throughput (256 Im/batch)
ResNet20 vl (0)
92.9
1.05
684.32
52239.12
ResNet20 v1 (T)
91.4
1.03
749.98
34743.19
ResNer20_v1 (T)
90.6
0.93
1144.10
48822.26
ResNet20_v1 (T)
88.3
0.92
1233.44
69868.6
ResNee20.x2(O)
92.7
1.05
737.07
52336.75 75
ResNe:20.v2
91.9
1.01
691.04
60078.59
ResNe:20.v2 (T)
90.7
0.83
1178.61
59269.61
ResNe:20.v2 (T)
90.2
0.83
1275.66
70498.54
ResNet56_v1 (0)
94.2
3.31
377.22
18966.31
ResNet56_vl (T)
93.5
3.05
389.27 .1
19175.07
ResNe:56_v1 (T)
92.9
2.97
398.94
24985.88
ResNe156.v1 (T)
92.5
2.94
399.47
25682.29
(esNe:36.v2(O)
94.6
3.31
389.76
18966.31
(T))
94.4
3.16
398.09
20293.55
ResNe156_v2 (T)
94.3
3.16
409.56
20382.47
ResNet56 v2 (T)
94.3
3.15
409.78
21752.16
ResNet110_vl (O)
95.2
6.68
213.59
9615.24
ResNet1 (T)
94.5
5.50
222.80
10161.07
ResNe:l (T)
93.1
5.48
229.85
12565.77
ResNer1 (T)
90.3
5.47
237.86
13032.99
ResNetl (O)
95.5
6.68
174.33
9160.58
ResNetl10.w2 v2
94.9
5.35
225.93
10892.03
ResNetl 10 v2 (T)
94.5
5.20
226.46
11331 .41
ResNet1 10 v2 (T)
94.3
5.19
233.13
11472.62
WideResNet16 10 (O)
96.7
65.34
349.98
5019,10
6ideRes.Net6...c (T)
96.1
16.23
601.01
7699.13
WideRessNer28.10 (O)
97.2
139.24
158.52
2553,99
WideRessNet28110 (T)
96.8
24.67
455.45
4339.26
WideResNett0.88 (O)
97.3
136.47
155.67
2643.25
WideResNNet0.38 (T)
96.4
21.31
395.71
4269.20 20
Table 5: Results of applying peeellpen DARC to CIFAR-10 -0 models. (O)riginal Denotes an
original model taken from the GluonCV Model Zoo (Mod) 2018). (S)ize denotes model optimized by DARC to minimize model size. (T)hroughput denotes a model optimized by DARC to maximize model throughput. Model Size provided in MB disk. Throughput numbers provided in 32px x32px images/second.
15

---

Under review as a conference paper at ICLR 2020
Model
Top-1
Model Size
Throughput (1 Im/batch)
Throughput (256 Im/batch)
ResNet18 (O)
70.9
44.64
333.86
4044.52
ResNet18_vl (S)
69.7
37.47
391.47
4142.90
ResNet18_vl (S)
69.6
30.72
399.27
4196.04
ResNet18_vl (S)
68.3
14.10
421.47
4281.12
ResNet18_vl (T)
69.9
42.43
417.64
4296.34
ResNet18 vI (T)
69.6
38.98
449.36
4473.29
ResNet18 vl (T)
67.9
36.77
457.60
5173.91
ResNet34.vI (0)
74.4
83.23
205.15
2441.79
ResNet34.vl (S)
73.5
37.68
216.68
2480.66
ResNer34_vl (S)
72.4
31.56
220.29
2543.27
ResNet34_v1 (T)
73.9
70.66
233.92
2693.20
ResNet34_v1 (T)
73.2
60.77
245.57
3289.19
ResNet50 v1 (0)
79.1
97.79
147.61
1242.42
ResNet50 vl (S)
78.6
59.93
150.82
1257.20
ResNe:50_vl (S)
78.5
47.71
157.79
1263.03
ResNet50_v1 (S)
78.3
32.49
161.64
1298.97
ResNet50.v1 (S)
78.2
27.42
167.92
1359.69
ResNe150_vl (S)
73.1
15,93
175.34
1492.43
ResNe:50_vl (T)
78.3
92.57
175.68
1518.25
ResNe150_vl (T)
78.2
71.19
199.76
1602.78
ResNe:50_vl (T)
76.8
69.29
208.01
1682.92
ResNer101_v1 (0)
77.2
170.54
100.25
724.79
ResNe:152.vl10) (O)
78.1
230.49
69.27
500.19
MobileNet1.0(O)
69.5
16.24
597.59
4393.48
MobileNen0.75 (0)
66.2
9.94
627.24
6161.22 .25
MobileNet0.5 (0)
61.2
5.13
662.31
9805.38
MobileNerVV. (O)
70.2
13.53
369.75
3615.16
MobileNer12.0075 (O)
68.1
10.15
376.56
4739.72
MobileNerV2..05500
64.0
7.59
398.03
7065.31
VGG11 (O)
66.9
506.83
318.53
912.44
VGG13 (0)
68.0
507.54
262.04
613.30
VGG16 (O)
71.5
527.79
205.29
458.85
VGG19 (0)
72.9
548.05
173.37
363.46
NenseN (O)
74.0
30.80
119.88
1104.38
DenseNet161 (0)
76.9
110.31
63.33
508.16
1en6eNet169 (O)
75.5
54.65
81.86
88 1,48
DenseNet201 (O)
76.6
77.30
62.83
673.
Table 6: Results of applying DARC to ImageNet models. (O)riginal Denotes original model taken from the GluonCV Model Zoo (Mod) 2018). (S)ize denotes model optimized by DARC minimize model size. (T)hroughput denotes a model optimized by DARC maximize model throughput. Model Size is provided in MB on disk. Throughput numbers are provided in 224px x224px images classifeddseccod.
16

---

Under review as a conference paper at ICLR 2020
Compression of ImageNet Models using Size Penalization
0.800
Original Models
0.775
Compressed Models
CenseNect.0
0.750
Top-1 Accuracy
0.700
@haieeaeep 1.0
0.650
0.625
0.600
10
10
Model (MB disk)
Acceleration of ImageNet Models using Size Penalization
0.800
eslet322 yi
0.775
eateton
0.750
Top-1 Accuracy
0.725
Te Mosockez2
VIGLE
0.675
GLL
Mosleliet
0,650
0.625
Original Compressed Models
Nowt 5
0.600
10a Model Throughput (images/second at 256 images/batch)
104
Acceleration of ImageNet Models using Size Penalization
0.800
testesto
0.775
AareeNetaee
Centellec00
0.750
Dense
Top-1 Accuracy
ALS
ResketR v
0.700
MAinNee2210
0.675
011
Hole
0.650
Nosleler2.00
0.625
Original Models
Compressed Models
0.600
Model Latency (images/second image/batch)
Figure 2: Compression of ImageNet models, in terms of size, throughput, and latency, when size is used the computational penalty in DARC.
17

---

Under review as a conference paper at ICLR 2020
Compression of ImageNet Models using Speed Penalization
0.800
Original Models
0.775
Compressed Models
0.750
Prooleet221
Top-1 Accuracy
0.725
0.700
@haikteep 1.0 MAsilelLL
Mosleteee 75
0.650
0.625
0.600
10
10
Model (MB disk)
Acceleration of ImageNet Models using Speed Penalization
0.800
Maclerto vi
sseet52 yi
0.775
eatetot at
0.750
DensehNut221
Top-1 Accuracy
0.725
NoseetR
0.700
VIGL3
0.675
GLL
Mosleliet
0,650
0.625
Original Compressed Models
Now 5
0.600
10a Model Throughput (images/second at 256 images/batch)
104
Acceleration of ImageNet Models using Speed Penalization
0.800
Mastent0
Nelee322
0.775
Dentellec00
0.750
V8019
Top-1 Accuracy
GAGLS
Reckett v1
0.700
0.675
011
0.650
Hoslewer2_05
0.625
Original Models
Compressed Models
0.600
Model Latency (images/second image/batch)
Figure 3: Compression of ImageNet models, in terms of size, throughput, and latency, when empir- ical latency is used as the computational penalty in DARC.
18


Reviews:
[{"review_id": "ryeWeSmCFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new architecture search method called \"DARC\" that utilizes a differentiable objective function. Since a naive formulation of architecture search is reduced to a combinatorial optimization which is not differentiable, the optimization requires much computational cost. To overcome this difficulty, this paper proposes a L1-norm relaxation and apply such relation in a layer-wise manner. The method shares a similar spirit with NAS, but the proposed model is more like \"model selection\" from a fixed candidates, and thus there is a Rademacher complexity guarantee. The effectiveness of DARC is justified by thorough numerical experiments.\n\nAlthough the idea is rather straight-forward, the effectiveness of the method is well supported by the thorough experiments. In particular, it works as a model compression method and shows a favorable performances compared with SOTA methods.\n\nThe pros and cons are summarized as follows.\nPros:\n- The proposed method is simple and rather easy to implement.\n- The numerical experiments show the proposed method gives favorable performances compared with the existing methods.\n\nCons:\n- The idea itself is rather straight-forward.\n- The theoretical analysis is instructive but its derivation does not require new techniques.\n- Compared with NAS, the proposed method should prepare a set of candidates which restrict the search space. This ensures generalization but limits its flexibility.\n\nMore comments:\n- I could not see how efficient the method is in terms of memory. It prepares several models in each layer, thus it requires large memories. Can it be performed on more large networks?\n- The setting of C_j affects the result. How did you set C_j in the experiments?\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyl64KDMqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem of limited resources that may generally be available at inference time (as compared to the lack of constraints during training time). The paper addresses architecture search as well as model compression by simultaneously optimizing all layers/weights/submodules of the network. The approach is motivated well in terms of comparisons to other work: Although I am not an expert in this particular area of model compression, the intuitive comparisons and mentions of prior work felt satisfying and informative. This problem is definitely timely, and the paper showed results of improved inference speed and memory footprint on both a smaller 10-way classification task as well as a larger 1000-way one. \n\nMy current decision is a weak accept, for a well-written paper and an interesting problem, but in the presence of some concerns and suggestions as listed below. \n\n(1) My main concern is that without weight regularization, this optimization problem seems ill-posed. Looking at algorithm 1 specifically, consider the case where j=10 and the learned alphas are indeed sparse, like (0.1, 0, 0,…, 0, 0.9). The cost of J=0 here might be very high (i.e., large, slow, expensive model) and cost of J=9 might be very low (i.e., small fully-connected network that basically maps to an identity function). Here, the cost during optimization could be very low, and the results could be very accurate, but the weights of J=0 may be very high such that the entire work of getting the right answer is from this model. More generally, a low alpha can be attained for any model by simply increasing the network weights themselves. This doesn’t seem to be taken care of anywhere in this paper; if not addressed in any way, it would be a large reason for me to suggest rejecting this work.\n\n(2) For the experiments, both of the chosen tasks were classification. Although one was smaller/easier and one was larger/harder, it would serve to be much more convincing if a task requiring regression and/or higher-dimensional output were tested. For these classification tasks, I can imagine that for certain model changes, the decision boundaries may remain the same even if the network itself is changing in detrimental ways. Thus, a task such as an image-to-image depth perception task (or other tasks in this category) would solidify the results and make it more convincing that DARC does indeed help.\n\nMinor: \n\n(a) The motivation of going from L0 to L1 for the cost constraint seems relevant also to the choice of h being the convex combination of different models. Can the motivation and problem setup address these motivations together, instead of separately?  \n\n(b) Although this paper defined R(h) and it also defined h=(\\sum (alpha_j * h_j), it would still be helpful for the reader if the paper could explicitly define R(\\sum (alpha_j * h_j)) somewhere. Although I know what was intended was to compose these h_j functions as sequential stages to pass data through, the sum notation seems a bit misleading like you may be averaging the final predictions/outputs from different models and computing the loss on these averaged results. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByesugqTtr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed a new gradient-based architecture search method that tries to find more efficient alternatives starting from the pre-trained model. The approach is similar to DARTS (Liu et al., 2019) with a budget constraint, such as size and throughput. One major difference is to modify the update of architectural parameters, i.e., mixing weights of all candidate operations, to induce the sparsity rather than to keep the weighted sum of all possible operations. Another difference is that it starts from a well-defined architecture with pre-trained weights. It is simple to apply, but hard to tell. The recognized strengths and concerns are as follows.\n\n< Strengths >\n1. The proposed update method to induce the sparsity of the architecture during the search seems applicable to all gradient-based search methods, and more important for the budget-constrained search minimizing the discrepancy between the ensemble architectures during the search and the final architecture derived after the search.\n2. The setup to initialize with the well-trained model seems practically useful rather than starting the search with a random initial model.\n3. The proposed methods seem easy to apply.\n\n< Concerns & Questions >\n1. The algorithm does not seem efficient because it continues training iteratively while increasing the strength (\\lambda) of the L1 regularizer by the estimated cost until the budget constraint is met.\n2. No details to calculate the cost C_l0(\\alpha) for each architecture candidate (e.g., the throughput cost of the architecture with a 3x3 DS operation in layer 1).\n3. No number of operations (e.g., # FLOPs, # MACs) is reported. Since the throughput is strongly dependent on the underlying hardware, the number of operations also needs to be shown as a more general estimation of the model inference latency in various hardware devices.\n4. The models under comparison are out-dated. It needs to be compared with the latest models designed with computational efficiency in mind, e.g., EfficientNet (Tan et al. 2019a), MixNet (Tan et al. 2019b), MobileNet V3 (Howard et al., 2019).\n\n- Liu et al., DARTS: differentiable architecture search, ICLR 2019\n- Tan et al., EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks, ICML 2019a\n- Tan et al., MixConv: Mixed Depthwise Convolutional Kernels, arxiv:1907.09595, 2019b\n- Howard et al., Searching for MobileNetV3, arxiv:1905.02244, 2019", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is on the borderline. A rejection is proposed due to the percentage limitation of ICLR.  
Decision: Reject