Year: 2019

Abstract:
In many learning situations, resources at inference time are much more constrained than resources at training time. This paper studies a general paradigm, called Differentiable ARchitecture Compression (DARC), that combines model compression and architecture search to learn models that are resource-efficient at inference time. Given a resource-intensive base architecture, DARC utilizes the training data to learn which sub-components can be replaced by cheaper alternatives. The high-level technique can be applied to any neural architecture, and we report experiments on state-of-the-art convolutional neural networks for image classification. For a WideResNet with 97.2% accuracy on CIFAR-10, we improve single-sample inference speed by 2.28× and memory footprint by 5.64×, with no accuracy loss. For a ResNet with 79.15% Top1 accuracy on ImageNet, we improve batch inference speed by 1.29× and memory footprint by 3.57× with 1% accuracy loss. We also give theoretical Rademacher complexity bounds in simplified cases, showing how DARC avoids overfitting despite over-parameterization.

heading: 1 INTRODUCTION
text: In machine learning, resources at inference time are often much more constrained than at training time. For example, while neural networks for computer vision and natural language processing (NLP) are routinely trained using GPUs, trained networks are often deployed on embedded systems or mobile devices with limited memory and computational power. As another example, it is common to train a model that will be applied continuously in production; while training occurs for a limited time, the machine performing inference may run indefinitely, and so learning a more efficient model can directly reduce costs associated with hardware or energy usage. As a result, many recent papers have studied deep model compression and acceleration. Most of these papers provide resource efficient model components (Jaderberg et al., 2014; Zhang et al., 2016; Wu et al., 2017) or methods to prune or quantize parameters (LeCun et al., 1990; Polyak & Wolf, 2015; Li et al., 2016a; He et al., 2017; Luo & Wu, 2017; Luo et al., 2017; Zhuang et al., 2018).
We propose a general paradigm called Differentiable ARchitecture Compression (DARC) for learning in the context of constrained resources at inference-time. Rather then suggesting a specific cheap component and using it blindly throughout a neural network, or trying to tune layer hyperparameters as in network pruning or quantization, our approach is inspired by Neural Architecture Search (NAS) (Zoph & Le, 2016; Liu et al., 2018a; Pham et al., 2018; Kandasamy et al., 2018; Liu et al., 2018b). DARC starts with a resource-intensive network design and uses training data to learn which components can be replaced with efficient alternatives, while maintaining model output quality. The resource requirement of the final model is controlled via a regularization term, which can be flexibly defined depending on the objective, such as minimizing inference time or memory footprint.
DARC has a clear intuitive advantage when compared to methods that quantize or prune parameters; these approaches are inherently restricted in their search space. They cannot replace a layer with a structurally different layer, or replace two or more layers with a shallow alternative. As examples, it might be the case that a convolutional layer cannot be pruned without hurting performance but can be replaced with a depthwise-separable convolution, or an LSTM layer might not be amenable to weight pruning, but could be replaced with a more efficient self-attention layer. DARC, applied to deep networks, offers a way to search a rich space in the context of model compression.
The high-level idea is to partition the network into components, and explore alternatives to these components simultaneously. Replacing all components simultaneously is crucial, as it provides a data-driven way to decide which components can be replaced with cheap alternatives. Indeed, replacing layers blindly may sacrifice too much prediction performance, as can be seen in Mo-
bileNets (Howard et al., 2017; Sandler et al., 2018), which exclusively use depthwise-separable convolutions; although computationally efficient, even the most accurate MobileNet model is far less accurate at ImageNet classification than, say, ResNet50.
We frame the problem of learning good replacement components as a sparse ensemble learning problem. This view allows us to draw guidelines from simpler analyzable cases, leading to a simple, gradient-based learning scheme that avoids over-fitting and is fast enough to apply DARC directly to large datasets such as ImageNet. This contrasts from most NAS methods, which first learn an architecture on a small dataset and then fit this architecture on the larger dataset.
We present experiments on networks commonly used in computer vision. By applying our techniques on the ResNet Architecture (He et al., 2016), we outperform state-of-the-art models on both ImageNet and CIFAR-10 datasets, in terms of accuracy vs. throughput and accuracy vs. model size. A few results from our compression framework: for a WideResNet model achieving 97.2% accuracy on CIFAR-10, we improve single-sample inference speed by 2.28× and memory footprint by 5.64×, with no loss in accuracy. For a modified ResNet50 model with 79.15% Top1 accuracy on ImageNet, we improve inference speed by 1.29× and memory footprint by 3.57× with 1% loss in accuracy. Both base models are publicly available from the GluonCV Model Zoo (Mod, 2018). Our experiments empirically demonstrate an intuitive observation that ‘you get what you optimize for’, in that models minimizing model size tend to be quite different from those maximizing throughput.
We note that, while our experiments are limited to image classification, DARC is applicable to any deep learning architecture, including models with recurrent cells or transformers, or indeed any sufficiently modular learning algorithm, as described in Section 2, and any task with a well-defined objective function in which we would like to reduce inference costs. We see this work as a proof-ofconcept for capabilities of the DARC framework, and the results in this paper give a strong indication that DARC can be applied to NLP architectures, or optimized for metrics other than those we try here (e.g. for latency on devices other than GPU, or energy consumption).

heading: 2 GENERAL SETTING AND DARC ESTIMATOR
text: The intuition and motivation for our method starts with a task of model selection. Given a task and J function families corresponding to candidate models, we are interested in finding the best model type for the task. We relax this combinatorial optimization problem to a more tractable differentiable optimization problem (whence the name “differentiable architecture compression”), by allowing convex combinations of these candidates. This can be thought of as a constrained form of ensemble learning in which weights are restricted to represent a convex combinations of individual learners.
Then, we posit a budget constraint: each model type has an associated cost (e.g., memory consumption or latency), and the overall cost of the ensemble is the sum of costs of the used models. Our task is then to learn a convex ensemble over a subset of candidates, with total cost within budget. We now formalize this approach, with modifications to address technical challenges as they arise.
In the sequel, for any positive integer J , [J ] = {1, 2, ..., J} denotes the set of positive integers at most J , and ∆J := { x ∈ [0, 1]J : ∑ j∈[J] xj = 1 } denotes the probability simplex over J elements.
Consider the conventional supervised learning setting, in which we have an i.i.d. training dataset (X1, Y1), ..., (Xn, Yn)
IID∼ P(X,Y ) from some joint distribution P(X,Y ) on X × Y . Fix a loss function L : R × Y → [0,∞], and a hypothesis class H of R-valued functions. We would like to learn a function h : X → R, h ∈ H that minimizes the risk R(h) := E(X,Y )∼P(X,Y ) [L (h(X), Y )]. The usual empirical risk minimization (ERM) estimator is ĥERM := arg minh∈H R̂(h) where, for any hypothesis h ∈ H, R̂(h) := 1n ∑n i=1 L (h(Xi), Yi) denotes the empirical risk. To derive our resource-constrained objective, we impose a few structural assumptions on our hypothesis classH: (A1) H = Conv ⋃ j∈[J]Hj is the convex hull of a union of J classesH1, ...,HJ .
(A2) Each classHj class has a known cost Cj ≥ 0 of using a hypothesis hj ∈ Hj at test-time. (A3) Costs are additive: hypothesis h = ∑J j=1 αjhj ∈ H has cost C`0(α) = ∑J j=1 Cj1{αj>0}. (A4) We have a known budget B ≥ 0 for the final model at test-time.
As we show in Section 3, these assumptions arise naturally in architecture compression. Given assumptions equation A1-equation A4, the constrained ERM estimate is ĝ = ∑ j∈[J] α̂j ĥj , where
(α̂0, ĥ0) := arg min α∈∆J ,ĥj∈Hj R̂ ∑ j∈[J] αjhj  , subject to C`0(α) ≤ B. (1) The above estimator is difficult (NP-hard) to compute, due to the non-smooth, non-convex budget constraint C`0(α) ≤ B. Since this constraint bounds the `0 norm of α (weighted by C), the usual remedy would be to relax the constraint to one on the `1 norm of α (weighted by C), namely C`1(α) := ∑ j∈[J] Cjαj ≤ B. Unfortunately, due to the constraint that α lies in the probability
simplex ∆J (which implies ∑ j∈[J] αj = 1), the `1 constraint is insufficient to induce sparsity on α. Fortunately, sparse optimization on ∆J is well-studied, with many solutions proposed (Pilanci et al., 2012; Kyrillidis et al., 2013; Li et al., 2016b). Due to ease of implementation, we adopt a simple but effective solution proposed by (Kyrillidis et al., 2013), which involves alternating gradient updates with a projection operation P∆J : RJ\RJ− → ∆J , given by P∆J (α) = α+ ‖α+‖1 , where α+ = (max{0, α1}, ...,max{0, αj}) ∈ RJ+.
P∆J is easy to compute, enforces the simplex constraint α ∈ ∆J exactly, and induces sparsity on α. For an intuition of how this works, one can note that ∇‖α‖22/2 = α, so that the update α/‖α‖1 = α − (1− 1/‖α‖1)α can be viewed as a gradient step for minimizing −‖α‖22/2 with adaptive step size (1− 1/‖α‖1). As a technicality, we note that the projection P∆J (α) is undefined when α ∈ RJ− has no positive components. However, for realistic gradient step sizes η, this never occurs, since, after each gradient update, ∑ j∈[J] αj ≥ 1−O(η).
Finally, a natural initial point for our procedure is one where C`1(α) > B, hence we reexpress the constraint C`1(α) ≤ B as a penalty λC`1(α). Since the value of λ corresponding to B is not known a priori, we iteratively increase λ until the solution of the optimization problem satisfies the budget constraint. The resulting DARC procedure is shown Algorithm 1. We note that the “stopping criterion” for the inner loop can be as simple as a fixed number of training epochs (as in our experiments), or a more sophisticated early-stopping criterion.
Algorithm 1: DARC algorithm for general hypotheses Data: Training Data {(Xi, Yi)}ni=1, J candidates h1,w1 , ..., hj,wj with initial parameters w1, ..., wj and costs c1, ..., cj ≥ 0, initial cost penalty parameter λ0 > 0, budget B. Result: α,w1, ..., wj such that h = ∑ j∈[J] αjhj,wj has small risk R(h) and cost C`0(α) ≤ B
1 α← (1/J, ..., 1/J), λ← λ0 ; 2 while C`0(α) > B do 3 while stopping criterion is not met do 4 (α,w1, ..., wJ)← (α,w1, ..., wJ)− η ( ∇α,w1,...,wJ R̂ (∑ j∈[J] αjhj,wj ) + C`1(α) ) ; 5 α← P∆J (α); 6 end 7 λ← 2λ 8 end

heading: 3 APPLYING DARC TO DEEP NETWORKS
text: DARC can be applied in a myriad of ways to compress deep neural networks. In all of these ways, the basic premise is to intelligently replace components of the network with cheaper components.
Consider a Neural Network (NN) with L layers. For layer `, let W` be the parameters of the layer, and g` be the function mapping inputs and parameters to the output (in layers having no parameters, W can be an empty token). For example, for a fully connected layer, W is a matrix, the input x is a vector, and g is the matrix-vector multiplication function. We can write the NN as a function:
f(x) = gL(WL, gL−1(WL−1, · · · g1(W1, x) · · · )), (2) To apply DARC, we consider a set of replacement candidates (g`,2,W`,2), . . . , (g`,J` ,W`,J`) for each layer ` (with g`,1,W`,1 denoting the original function and weight of the layer). For each candidate j in layer `, DARC takes as input an associated cost C`,j ≥ 0. Examples of such costs include
parameters count, FLOPs, or latency, which are usually easy to calculate or estimate experimentally. Applying DARC to neural network compression then involves four main steps: 1. Layerwise Continuous Relaxation: First, we replace each g`,W` with a weighted average g̃`(W̃`, α`, x) = ∑J` j=0 α`,jg`,j(W`,j , x), where α` ∈ ∆J` . The original network is replaced
by f̃(x) = g̃L(W̃L, αL, · · · g̃1(W̃1, α1, x) · · · ). 2. DARC Model Initialization: Before training the DARC model, we need to initialize the α
weights and the parameters of the compression candidates. We initialized the α parameters as uniform vectors α` = (1/J`, 1/J`, ..., 1/J`). The other option we considered was to put all weight on the original candidate (α` = (1, 0, ..., 0)), so that the initial model was equivalent to the original model being compressed. However, this makes the gradient of the loss 0 with respect to all parameters of the compression candidate, preventing these from training. Furthermore, the non-convex regularization discourages the weights of α` to shift towards a value that makes use of the compression candidates. As for candidate parameters, we initialized each compression candidate to mimic the original layer, which we know gives good prediction results. In some cases, this can be done analytically (e.g. via PCA for lower dimensional fully-connected layers); more generally, this can be done via SGD, training the new candidate to minimize squared loss between its outputs and those of the original layer g`,1(W`,1, x). Since this is only for initialization, it suffices to use a small training sample and crude optimization procedure.
3. Training the Relaxed Model: We minimize the empirical risk, simultaneously over the mixture weights (αs) and the candidate weights (W̃ s) as described in Algorithm 1.
4. Selecting a Sub-Model: As discussed above, for sufficiently large λ, Algorithm 1 converges to a solution with small (weighted) `0 norm; i.e., α` will have a small number of non-zero entries. Thus, we remove candidate g`,j (and its weight α`,j) from the network if α`,j = 0.
During optimization, we jointly optimize α and the model parameter on the same data, contrasting from other gradient based NAS approaches (Liu et al., 2018b) that split data into two training sets, optimize model parameters on one and α weights on the other. In Section 4 we analyze the Rademacher complexity of our procedure in a simple setting and show that under the condition that the original model class defined by g`,1 is richer than the alternatives, optimizing all parameters jointly does not hurt generalization guarantees when compared to the original optimization objective where J` = 1. Unlike for NAS, this condition holds naturally for model compression.
Efficient Approximate Convolutions Computation in most deep networks used in computer vision problems, such as image classification, image segmentation, and object detection, is dominated by convolutional layers. This has motivated several papers on efficient approximations to convolution, such as depthwise-separable convolution (Jaderberg et al., 2014; Zhang et al., 2016; Howard et al., 2017), bottleneck convolution (Sandler et al., 2018), and shifts (Wu et al., 2017).
In a standard convolution layer we have k × k filters for every input and output channel. Denoting the output channels by Yi and the input channels by Xj , the i’th output channel is defined as Yi = ∑ j Xj ∗ Fi,j . Here Fi,j is the appropriate filter and ∗ is the convolution operator. Restricting the discussion to the setting where the number of output and input channels are the same, a fully-grouped convolution is a more constrained alternative in which the filter is k × k but each output is computed based on a single input channel. A depthwise-separable convolution consists of a full-grouped convolution followed by a standard 1 × 1 convolution. In most setting this operation requires less compute and memory resources. A shift layer is an even cheaper alternative to depthwise-separable where the Fi’s are fixed and have only a single non-zero element, resulting in computational complexity equivalent to a single 1 × 1 convolution. We use DARC to compress CNNs by considering alternatives from among the above options, for each convolution layer.

heading: 4 THEORETICAL RESULTS
text: Here, we discuss generalization power models learned by DARC. We restrict our attention to the simple case of learning an ensemble of models; as described below, the result has implications for our algorithm for training DARC. This setting actually applies not only for DARC but also for various NAS methods such as DARTS (Liu et al., 2018b) or ENAS (Pham et al., 2018). Indeed, these methods aim to choose one out of several options in each layer. While these methods differ in how this ensemble is learned, our generalization bound is independent of the learning technique.
Recall that DARC learns a convex combination of functions from J classes H1, . . . ,Hj . Here, we analyze generalizability of this process via Rademacher complexity Bartlett & Mendelson (2003):
Definition 1 (Rademacher Complexity). Let H be a class of functions mapping X → R and let n ∈ N. Denote by Xn1 = (X1, ..., Xn) n IID samples from X . Let σ a uniform random vector in {−1, 1}n. The Rademacher complexity ofH is R(H) = Eσ,Xn1 [ suph∈H 1 n ∑ i∈[n] σih(Xn,i) ] .
It is well known that the Rademacher complexity of a classH is equal to that of the convex hull ofH. Based on this fact, for h = (h1, . . . , hJ) ∈ ∏J j=1Hj , α ·h = ∑J j=1 αjhj , we have a generalization
bound on the difference between true risk R and empirical risk R̂: Theorem 1. Suppose we jointly estimate α, h1, ..., hJ ; i.e., ( α̂, ĥ1, ..., ĥJ ) :=
arg minhj∈Hj ,α∈∆J :C·α≤B ∑n i=1 L (α · h(Xi), Yi). Let L(h(x), y) = 1{f(x)6=y} be 0-1 loss.
Then, w.p. ≥ 1−δ (over n training samples), R ( α̂ · ĥ ) − R̂ ( α̂ · ĥ ) ≤ R (⋃ j∈[J]Hj ) + √ log 1/δ n .
Since Theorem 1 follows from standard Rademacher generalization bounds (e.g., (Bartlett & Mendelson, 2003, Theorem 5(b))), we omit its proof. According to Theorem 1, generalization error depends on a standard √ (log 1/δ)/n term and Rademacher complexity of the union of classes
H1, ...,HJ . If H1, ...,HJ are diverse, this union can be quite rich and so R ( ∪j∈[J]Hj ) might be large, leading to overfitting. However, consider an example where H1 is the family of full convolutions,H2 is the family of depthwise-separable convolutions,H3 is the family of sparse convolutions, etc. Here, we actually have H1 = ⋃ j∈[J]Hj ; thus, Rademacher complexity is simply that of the original model (i.e., with J = 1). Formally:
Corollary 1. Suppose that every sub-model is contained in H1; i.e., H2, ...,HJ ⊆ H1. Then, the Rademacher complexity of DARC is at most that of the original model: R (H) ≤ R (H1).
Even ifH1 ( ⋃ j∈[J]Hj , in the setting of model compression the alternative familiesHj , j > 1 are
cheaper replacements for H1 suggesting that R(H1) ≈ R( ⋃ j∈[J]Hj). This observation motivates our learning framework – it shows us that there is no need to split the training set, train the model parameters on one split and the control parameters on the other, as in many NAS papers (Cai et al., 2018; Liu et al., 2018b).
We note that does not motivate a change in the learning framework of NAS. A key difference between NAS and DARC is that candidate models H1, ...,HJ in NAS are intentionally diverse; their union is much richer than any individual. This translates to large R (⋃ j∈[J]Hj ) , motivating a need to avoid jointly optimizing α and model parameters on a single training set. When keeping a validation set aside for training α, given the limited number of update steps typical in NAS papers, generalization error may be closer to the setting of fixed α, wherein Rademacher complexity is bounded by ∑ j αjR(Hj) (Cortes et al., 2014), potentially much smaller than in Theorem 1.

heading: 5 EXPERIMENTAL RESULTS
text: We applied DARC to a number of deep networks from the GluonCV Model Zoo (Mod, 2018) for image classification on the CIFAR-10 and ImageNet (Russakovsky et al., 2015) datasets; this section presents quantitative and qualitative results. We report three performance metrics (model size, single-sample throughput, and batch throughput), and we specifically considered using DARC to minimize two of these (model size and batch throughput). Below, “DARC(S)” denotes DARC with a model (S)ize penalty, and “DARC(T)” denotes DARC with batch (T)hroughput penalty.
Choice of Compression Candidates For each convolutional layer, besides the original (full) convolution, we considered 3 compression candidates: a (fully-grouped) depthwise-separable convolution with 3 × 3 kernels (abbreviated henceforth as “3x3DS”), a full convolution with 1 × 1 kernels (“1x1FC”), and a 2-layer candidate consisting of a 3x3DS layer followed immediately by a 1x1FC layer (“3x3+1x1”). In ResNet50, which was already implemented using bottleneck convolutions (He et al., 2016) consisting of a sequence of 1× 1, 3× 3, and 1× 1 convolutions, the entire bottleneck
sub-network was treated as a single component (i.e., all 3 convolutions were replaced with a single block from the above mentioned alternatives). We intentionally limited the choice of alternatives to maintain a simple system that can enjoy high throughput without special implementation. Our precise choice of alternatives is motivated by them already being an established component for deep networks, proven to work in some settings even without architecture search.
DARC Training Details To maximize fairness when comparing with models in the GluonCV Model Zoo (Mod, 2018), most aspects of training DARC were based on the training scripts provided publicly by the Model Zoo1. Due to space constraints, these implementation details are decribed in Appendix A; a few specific differences from these scripts are described below.
Student-Teacher Initialization: As described in Section 3 we initialized compression candidates to mimic original layers using student-teacher training. While this training had to be performed separately for each compression candidate in each layer of the original model, since each compression candidate has few parameters, each candidate’s training converged quite quickly. Thus, in CIFAR-10 experiments, we simply ran 1 epoch of the entire training dataset; in ImageNet experiments, we ran only 1000 batches. A relatively large step size of 0.1 was used, since teacher-student initialization is only for initialization and fine-tuning can be performed during model-selection.
Main Training Phase: As noted in Algorithm 1, training occurred in blocks of epochs (20 epochs/block for CIFAR-10, 10 for ImageNet), with the compression penalty λ increased after each block, to obtain a spectrum of compressed models. For each dataset, penalty type, and model size, the initial value of λ was selected to roughly balance the orders of magnitude of the empirical loss and the regularization term at the beginning of training. After each block, we: 1) remove candidates j with αj = 0l, 2) save (for evaluation later) a copy of the DARC model, in which, in any layer with multiple non-zero α entries, all but the most expensive remaining candidate are removed, and 3) decrease learning rate η and increase compression penalty λ (each by a factor of 2).
This “iterative compression” was repeated until only one compression candidate per layer remained in the DARC model. Finally, each saved model was fine-tuned for 20 epochs using only prediction loss (i.e., with λ = 0). This procedure enabled us to obtain a sequence of compressed models at progressively increasing compression levels. Moreover, this “warm-starting” improved compression speed since we only perform a total of 30 epochs per λ value, rather than the > 100 epochs needed for convergence at high levels of compression.

heading: 5.1 CIFAR-10 RESULTS
text: By all metrics, DARC gave the best results when applied to very wide models such as the WideResNet series (specifically, the WideResNet16 10, WideResNet28 10, and WideResNet40 8 models (Mod, 2018)). Moreover, unlike results on other ResNets, results on WideResNets were relatively similar for both DARC(S) and DARC(T); both versions of DARC selected the 3x3+1x1 candidate for every layer. The reason for this is that very wide convolutions in WideResNet models can be replaced by depthwise separable convolutions with essentially no loss in accuracy, and improvement in throughput for both batches and for single samples (1.4-1.6× for each) and memory footprint (4-6×). In the case of WideResNet16 10, DARC produces a model with latency (single sample throughput) comparable to on of the fastest CIFAR-10 model (ResNet20 v1; see Figure 1), while having accuracy within 0.8% of the best model (ResNeXt29 16x64d), 3.2% accuracy points above the performance of ResNet20 v1. For complete CIFAR-10 results see Appendix tables 3-4.

heading: 5.2 IMAGENET RESULTS
text: For ImageNet we compressed several ResNet models. To present the size compression results we provide Table 2 comparing accuracy change as a function of parameter reduction. We compared to previous published results compressing ResNet50 on ImageNet. To our knowledge these are the state-of-the-art results among those compressing ResNet50 on ImageNet. For compression as aggressive as 3×, we incur an accuracy drop of 0.86% while the baseline suffers a drop of 3.26%. Table 1 gives throughputs of our compressed ResNet34 and ResNet50 models and throughputs of state-of-the-art competing models, on identical hardware; our compressed models outperform these competing models, due to our initialization based on pre-trained ResNet34 and ResNet50 models.
1train imagenet.py, train mixup cifar10.py
While further details, including other compressed versions, are available in the Appendix (Figures 2 and 3 and Table 5), Table 2 compares prediction performance of DARC(S) with that of state-of-theart network pruning techniques applied to ResNet50 on ImageNet, at various compression levels. In the light (1.5×) compression regime, the result of (Zhuang et al., 2018) outperforms ours. The work of (Zhuang et al., 2018) complements ours, in that their novelty is in warm-starting the compressed alternative not only to mimic the original, but also to be informative w.r.t. the label. Since this work does not have an architecture search component, we suggest that future work combine this clever warm-start with an architecture search component such as DARC. Once the compression becomes more aggressive, DARC outperforms the baseline, likely since in that regime (smaller network with more training epochs), a good architecture is more important than a good warm-start.

heading: 5.3 DISCUSSION OF COMPRESSED ARCHITECTURES
text: In both ResNets and WideResNets, model size tends to be dominated by a small number of the largest layers in the model, and is relatively insensitive to the depth of the network. Thus, significant compression can be achieved by replacing these large layers with 3x3+1x1 candidates, which offer compression of nearly 9× (for 3× 3 convolutions). Since the sizes (number of convolutional kernels) of ResNet layers increases from bottom to top (i.e., from input to output) DARC(S) tends to first replace the top-most layers (i.e., layers closest to the output) of the network with 3x3+1x1 candidates, proceeding towards the bottom of the network as the compression parameter λ increases.
In contrast, model latency is relatively uniformly distributed throughout the layers of the network – the time taken to compute each convolutional layer scales only weakly with the
number, size, and grouping of filters. Thus, in ResNets (but not in WideResNets) 3x3+1x1 candidates, which replace 1 layer with 2 smaller layers, tend to offer little or no benefit in throughput; of the compression candidates we considered, only the smallest (3x3DS) candidates offer significant acceleration (typically of about 2×) over full convolutions. As a result, the acceleration offered by DARC(T) scales primarily with the number of layers that can be replaced by 3x3DS layers. Moreover, the replaced layers tend to be scattered throughout the network (rather than clustered near the output of the network). As noted earlier, in WideResNets, each layer is so large that the 3x3+1x1 candidate is much faster than full convolution, and DARC(T) selects this candidate for all layers.
Overall, we see that the optimal compression strategy depends on whether one is optimizing for size or for speed. This implies that some sort of intelligent replacement of components, as in DARC, is needed to obtain reliable performance improvements (as opposed to, say random baselines that have been shown to perform well in pruning). This parallels recent work (Cai et al., 2018) showing (for NAS) that optimizing for speed on different hardware (GPU, CPU, or mobile) leads to different
models. While we focused on GPU throughput, we note that the speedup of depthwise-separable convolution over full convolution is typically larger on CPU and embedded devices than on GPU devices. Thus, DARC should also produce efficient architectures for these alternative hardware.

heading: 6 RELATED WORK
text: Work on compressing deep networks has abounded in recent years, with diverse approaches including pruning (LeCun et al., 1990; Polyak & Wolf, 2015; Li et al., 2016a; He et al., 2017; Luo & Wu, 2017; Luo et al., 2017; Zhuang et al., 2018), low-rank factorization (Jaderberg et al., 2014; Zhang et al., 2016; Howard et al., 2017), fast approximate convolutions (Bagherinezhad et al., 2017; Wu et al., 2017), knowledge distillation (Hinton et al., 2015; Romero et al., 2014), and quantization (Gong et al., 2014; Han et al., 2015; Zhou et al., 2017; Lin et al., 2016); (Cheng et al., 2018) survey common approaches. In contrast, DARC searches a richer space of alternative models and has the ability to replace multiple components by a single one (e.g. replacing a bottleneck sub-network of 3 convolutions with a single convolution). Indeed, though beyond the scope of this paper, DARC can incorporate or complement many of these methods. Our initialization of compression candidates by mimicking the original layer is also reminiscent of knowledge distillation.
Another closely-related line of papers concerns Neural Architecture Search (NAS) (Pham et al., 2018; Kandasamy et al., 2018; Liu et al., 2018b; Gordon et al., 2018; Cai et al., 2018). The most relevant papers are (Liu et al., 2018b), (Gordon et al., 2018), and (Cai et al., 2018), who all use a sparse linear component weighting scheme similar to DARC. (Liu et al., 2018b) focused on pure architecture search, in which the goal is simply to find an architecture maximizing prediction performance, without consideration of inference-time efficiency. (Gordon et al., 2018) do not aim to replace layers with general alternatives but rather discover parameters in a data-driven way; their experiments are restricted to results in channel pruning. Recently, (Cai et al., 2018) performed NAS with a latency regularization term similar to ours.
Our methods differ from these NAS papers in two main ways (summarized in Table 3). First, our architecture search is guided by an established base model that was already tested and proven useful. This distinction allows a simpler and more efficient learning scheme (motivated in Section 4) that avoids iterating between training sets, to optimize model and α parameters. Furthermore, starting with a pre-trained model allows us not only to reach an effective architecture, but to warm start the weight parameters. As evidence of the advantage of starting with a pretrained model, our compressed model ResNet50(T) on ImageNet has Top1 accuracy ≥ 3% more than models obtained by these NAS papers; thus, it seems that starting architecture search with a highly accurate base model can improve the efficiency/accuracy trade-off of the learned model. Another distinction from gradient-based NAS results is our use of sparsity-inducing regularization. Previous methods make the choice between the candidates via a softmax layer; this restricts the output to be a convex combination of inputs without optimizing sparsity. Since these methods also aim to find a sparse combination, they might benefit from a non-convex regularization term as in DARC.

heading: 7 CONCLUSIONS AND FUTURE WORK
text: We have shown that even a simple DARC implementation, with only depthwise-separable approximations as compression candidates, can be used to compress large state-of-the-art deep networks, improving inference speed and memory footprint. Intelligently making only some layers of the net-
work depthwise-separable results in compressed models with much better predictive performance than simply making all convolutions depthwise-separable, as in (Howard et al., 2017).
While depthwise-separable convolutions are easily implemented in existing deep learning packages and already offer substantial compression, future work may benefit from more sophisticated approximate convolutions, with efficient implementations. For example, shift operations (Wu et al., 2017) are promising, as they require no stored parameters and replace the slow multiplications in convolution with fast indexing. Another venue worth pursuing is to compress a model into a shallower version. Although there are a few ways this could be attempted, such as an Identity candidate or replacing entire blocks of layers, it is unclear which technique would work best. Finally, we hope to apply DARC to models other than CNNs (e.g., models with recurrent cells or transformers).

heading: A DARC IMPLEMENTATION DETAILS
text: In this section, we provide further details about the implementation of DARC used in our experiments.
Environment Details We implemented DARC in Apache MXNet 1.3.1 using Python 3.6 and CUDA 9.0. Experiments were run on AWS EC2 p3.8xlarge and p3.16xlarge machines, which respectively features 4 and 8 NVIDIA Tesla-V100 GPUs. CIFAR-10 models were each trained with 1 GPU. Smaller ImageNet models (ResNet18 and ResNet34) were trained using 4 GPUs, while the larger ResNet50 was trained using 8 GPUs.
Training Details Following the original script used to train models in the GluonCV Model Zoo, we utilized mixup training (Zhang et al., 2017), and optimized cross-entropy loss with Nesterov accelerated stochastic gradient descent (NAG) with (default) momentum parameter 0.9. As noted in the main paper, for student-teacher initialization, we used a relatively large learning rate η = 0.1. Thereafter, for model-selection, we began with an initial learning rate of η = 0.01, which was then halved after each traning block.
To minimize training time, training batch sizes were selected to be as large as possible without exceeding GPU memory during training. This resulted in batch sizes (per training GPU) of 256 for ResNets on CIFAR-10, 128 for WideResNets on CIFAR-10, 64 for ResNet18 and ResNet34 on ImageNet, and 32 for ResNet50 on ImageNet.
For each dataset, penalty type, and model size, the initial value of λ was selected to roughly balance the orders of magnitude of the empirical loss and the regularization term at the beginning of training. For CIFAR-10 experiments with size penalization, the initial value of the λ compression penalty was set to λ = 10−5×L, where L is the number of layers to which DARC was applied (i.e., the number of full convolutions in the original model). For CIFAR-10 experiments with latency penalization, we used λ = 104 × L. For ImageNet experiments, we used λ = 10−8 × L with size penalization and L = 104 × L for latency penalization.
A.1 MEASURES OF MODEL PERFORMANCE
Computational Performance As an estimate of model size, we report the size (on disk) of the parameter file created by MXNet when saving the model; this correlates well with both the number of parameters in the model and the footprint of the model in RAM or GPU memory. Since throughputs are inherently noisy, we report average inference times over 1000 batches. Though multiple GPUs were used for training DARC, all inference times were computed using a single Tesla V100 GPU. We used batch size 1 to estimate single-sample throughput and batch size 256 to estimate batch throughput. The cost of each compression candidate (i.e., number of parameters for DARC(S) or latency for DARC(T)) was calculated or estimated based on the student model trained during initialization.
Prediction Performance On CIFAR-10, we used standard (“Top1”) prediction accuracy. On ImageNet, we additionally used (“Top5”) accuracy, the fraction of test images for which the correct label is among the five labels considered most probable by the model. We note that these are the standard performance used for these datasets (Krizhevsky & Hinton, 2010; Krizhevsky et al., 2012).

heading: B SUPPLEMENTARY RESULTS
text: This section provides detailed numerical results of our experiments:

Review summaries:
The authors proposed a new gradient-based architecture search method that tries to find more efficient alternatives starting from the pre-trained model. The approach is similar to DARTS (Liu et al., 2019) with a budget constraint, such as size and throughput. One major difference is to modify the update of architectural parameters, i.e., mixing weights of all candidate operations, to induce the sparsity rather than to keep the weighted sum of all possible operations. Another difference is that it starts from a well-defined architecture with pre-trained weights. It is simple to apply, but hard to tell. The recognized strengths and concerns are as follows.

This paper proposes a new architecture search method called "DARC" that utilizes a differentiable objective function. Since a naive formulation of architecture search is reduced to a combinatorial optimization which is not differentiable, the optimization requires much computational cost. To overcome this difficulty, this paper proposes a L1-norm relaxation and apply such relation in a layer-wise manner. The method shares a similar spirit with NAS, but the proposed model is more like "model selection" from a fixed candidates, and thus there is a Rademacher complexity guarantee. The effectiveness of DARC is justified by thorough numerical experiments.