LCS: Learning Compressible Subspaces for Adaptive Network Compression at Inference Time
Authors: Maxwell Horton
Elvis Nunez
Anish Prabhu
Anurag Ranjan
Ali Farhadi
Mohammad Rastegari
Venue: ICLR 2022 Submitted (2022)
Keywords: network subspace
compression
post-training
pruning
quantization
efficient

Abstract: When deploying deep learning models to a device, it is traditionally assumed that available computational resources (compute, memory, and power) remain static. However, real-world computing systems do not always provide stable resource guarantees. Computational resources need to be conserved when load from other processes is high or battery power is low. Inspired by recent works on neural network subspaces, we propose a method for training a "compressible subspace" of neural networks that contains a fine-grained spectrum of models that range from highly efficient to highly accurate. Our models require no retraining, thus our subspace of models can be deployed entirely on-device to allow adaptive network compression at inference time. We present results for achieving arbitrarily fine-grained accuracy-efficiency trade-offs at inference time for structured and unstructured sparsity. We achieve accuracies on-par with standard models when testing our uncompressed models, and maintain high accuracy for sparsity rates above 90% when testing our compressed models. We also demonstrate that our algorithm extends to quantization at variable bit widths, achieving accuracy on par with individually trained networks.

Under review as a conference paper at ICLR 2022
LCS: LEARNING COMPRESSIBLE SUBSPACES FOR ADAPTIVE NETWORK COMPRESSION AT INFERENCE TIME
Anonymous authors Paper under double-blind review
ABSTRACT
When deploying deep learning models device, is traditionally assumed that available computational resources (compute, memory, and power) remain static. However, real-world computing systems do not always provide stable resource guarantees, Computational resources need to be conserved when load from other processes is high or battery power is low. Inspired by recent works on neural network subspaces, we propose a method for training a compressible subspace of neural networks that contains a fine-grained spectrum of models that range from highly efficient to highly accurate. Our models require retraining, thus our subspace of models can be deployed entirely on-device to allow adaptive network compression inference time. We present results for achieving arbitrarily fine- grained accuracy-efficiency trade-offs at inference time for structured and unstruc- tured sparsity. We achieve accuracies on-par with standard models when testing our uncompressed models, and maintain high accuracy for sparsity rates above 90% when testing our compressed models. We also demonstrate that our algo- rithm extends to quantization at variable bit widths, achieving accuracy on par with individually trained networks.
INTRODUCTION
Deep neural networks are deployed to variety of computing platforms, including phones, tablets, and watches (Dhar et al., 2019). Networks are generally designed to consume fixed budget of resources, but the compute resources available on a device can vary while the device is in use. Computational burden from other processes, well battery life, may influence the availability of resources to a neural network. Adaptively adjusting inference-time load is beyond the capabilities of traditional neural networks, which are designed with fixed architecture and a fixed resource usage.
A simple approach to the problem of providing an eecuracy-efficienn trade-off is to train multiple networks of different sizes. In this approach, multiple networks are stored on the device and loaded into memory when needed. There a breadth of research in the design of efficient architectures that can be trained with difffernnt capacities, then deployed on a device (Howard et al.) 2017 Sandler et al. 2018 Howard et al. 2019). However, there are a few drawbacks with using multiple networks provide an accuracy-efliciency trade-off:: (1) requires training and deploying multiple networks (which induces training- time computational burden and on on-device storage burden), (2) it requires all compression levels to specified before deployment, and (3) requires new networks to be loaded into memory when changing the compression level, which prohibits real-time adaptive compression.
Previous methods address the first issue in the setting of structured sparsity by training a single network conditioned to perform well when varying numbers of channels are pruned YYu et al.) 2018
Yu & Huang 2019). However, these methods require BatchNorm (Ioffe & Szegedy 2015) statistics to be recalibrated for every accuracy-efficiency configuration before deployment. This requires users to know every compression level in advance and requires storage of extra parameters (which can be problematic for low-compute devices). Another method trains larger network that can be queried for efficient subnetworks (Cai et al. 2020), but requires users to reload network weights, and to know every compression level before deployment.

---

Under review as a conference paper at ICLR 2022
Learning Compressible Subspaces
Efficiency-Accuracy Trade-Off
Network subspace (*)),, compression level calculator ?(a), compression function f [[w,),, stochastic sampling function e
Input:
Accuracy
[0. 1 dataset D, loss function L, regulariza- tion function
Replace BatchNorm layers with GroupNorm. while w has not converged do for batch B in do
Efficiency
Sample vector an
1 for do
# compute loss on batch
fww (a)). ((a)))
fw (a.)))) 7(a))
1 ++ [(l(w" (a), 7(a)), B)
end for apply regularization
+- (+=r(w")
*
(22)
backpropagate loss
apply gradient update to w end for
Linear Subspace
n *
(a
end while
return w
(a)
(b)
Figure 1: (a) Depiction of our method for learning a linear subspace of networks w parameterized by [2], When compressing with compression function f and compression level y, we ob- tain a spectrum of networks which demonstrate an efficiency- accurray trade-off. (b) Our algorithm.
We address the issue of deploying model that can be compressed in real-time without specifying the compression levels beforehand. Inspired by Wortsman et al.) (202I). we formulate this problem computing adaptively compressible network subspace (Figure Ta). We train a linear subspace using two sets of weights, conditioning them so that a convex combination of them produces an linear subspace contain an arbitrarily fine- the-ggraied efficiency-accuran trade-off (meaning, any compression level within predefined limits can be chosen), and (2) replacing BatchNorm (Ioffe & Szegedy 2015) with GroupNorm [Wu & He 2018) to avoid BatchNorm recalibration.
accurate network. We expand Wortsman al. (202T) by (1) specializing
Our contributions as follows. (1) We introduce method, Learning Com- pressible Subspaces (LCS), for training models that can be compressed with arbitrarily fine-grained compression levels in real-time after deployment. To our knowledge this has not been done be- fore. (2) We demonstrate new use for neural network subspaces, specializing at one end for high-accuracy and the other end for high-efficiency.
Contributions:
(3) We provide an empirical evaluation of method using structured sparsity, unstructured sparsity, and quantization. (4) We open source our code at httt::://tttthheeeeeeeee for. review
RELATED WORK
Architectures Demonstrating Elficiency-Accuracy Trade-Offs: Several recent network architec- tures include a hyperparameter to control the number of filters in each layer (Howard et al. 2017 Sandler et al., 2018: Howard et al. 2019) the number of blocks in the network (TTn & Le 2019). In Once For All (Cai et al.) 2020). the need for individually training networks of different sizes is circumvented. Instead, single large network is trained, then queried for efficient subnetworks.
We differ from these methods in two ways. First, we compress on-device without specifying the compression levels before deployment. Previous works require training separate networks (or in
2

---

Under review as a conference paper at ICLR 2022
Table 1: Our method with a linear subspace (LCS+L) and a point subspace (LCS+P) compared to LEC (Liu et al. 2017). NS Yu et al. 2018), and US (Yu & Huang 2019). Note that "Arbitrary Compression Levels" refers to arbitrarily fine-grained post-deployment compression. |w| denotes the number of network parameters, |6] denotes the number of BatchNorm parameters, and n denotes the number of compression levels for models that don't It support arbitrarily fine-grained compression.
LCS+P
LCS+L
LEC
NS
US
No Retraining No Norm Recalibration
x
x
x
X
Arbitrary Compression Levels
x
x
x
Stored Parameters
2 w
n
|
|b|
||||
n
the case of Once For All, querying a larger model for compressed network) before deployment. Second, we don't require deploying a set of weights for every compression level.
A method for post-training quantization to variable bit widths appears in Shkolnik et al. (2220). but it relies on calibration data and cannot be run on low-compute edge devices. Thus, compression levels must be specified before deployment. They also present method for quantization-awarl training to variable bit widths, but most of their results keep activation bit widths fixed, whereas we vary it.
Adaptive Compression: Recent works train single neural network which can be adaptively con- figured at inference time to execute at different compression levels. These methods are the closest to our work. In Learning Efficient Convolutions (LEC) (Liu et al. 2017), the authors train a sin- gle network, then fine-tune it at different structured sparsity rates (though they also present results without fine-tuning, which is closer our use case). Other methods train single set of weights conditioned to perform well when channels are pruned, but require recalibration (or preemptive stor- age) of BatchNorm (Toffe Szegedy, 2015) statistics at each sparsity level. These methods include Network Slimming (NS) Yu et al. 2018) and Universal Slimming (US) (Yu & Huang 2019). Sim- ilar methods train a single network to perform well at various levels of quantization by storing extra copies of BatchNorm statistics (Guerra et al. 2020).
We differ from these methods by avoiding the need to recalibrate or store BatchNorm statistics (Section 3.3h and by allowing for fine-grained selection of compression levels at inference time, neither of which have been done before (Table [.. Additionally, our method is broadly applicable to variety of compression methods, whereas previous works focus single compression method.
Other Wost-Trannin Compression Methods: Other works have investigated post-training com- pression. In Nagel et al. (2019), method is presented for compressing 32 bit models to 8 bits, though it has not been evaluated in the low-bit regime. It involves running an equalization step and assuming a Nony-BatchNorm- ReLU network structure. A related post-training compression method is shown in |Horton et al. (2020), which shows results for both quantization and sparsity. However, their sparsity method requires a lightweight training phase. We differ from these methods in pro- viding wal-timee post-deployment compression for both sparsity and quantization without making assumptions about the network structure.
Sparsity is well-studied approach to reduce neural network compute. Recent works have achieved state-of-the-ar accuracy using simple sesnnitude-based pruning (Han et al, 2015 Zhu & Guptal 2017) [See et al., [016)] Narang et al. 2017). When experimenting with struc- tured sparsity, we simply remove filters from the network, as described in Yu et al. (2018). When investigating unstructured sparsity, we use TopK pruning (Zhu & Gupta 2017).
Sparse Networks:
Network quantization represents the weights and/or activations of neural net- works with lower-bit integral representations, rather than 32-bit floating point. This reduces model size and memory bandwidth and can accelerate inference. Our work follows the affine quantization scheme presented in Jacob et al. (2018), in which a scale and offset applied to integer tensor to approximate the original floating-pon tensor. However, our method is general enough to utilize any other quantization scheme.
Quantized Networks:
Neural Network Subspaces: The idea of learning a neural network subspace is introduced in Worts- man et al. (021) (though another formulation was introduced concurrently in Benton et al. 2022
3

---

Under review as a conference paper at ICLR 2022
Multiple sets of network weights treated the corners of simplex, and optimization pro- cedure updates these corners to find a region in weight space in which points inside the simplex correspond to accurate networks. We note that Wortsman al. (202I) recalibrate BatchNorm statis- ties before evaluation, which we avoid.
COMPRESSIBLE SUBSPACES
3.1 COMPRESSIBLE LINES
In our method, we train neural network subspace (Wortsman al. [021] that contains a spectrum of networks that each have a different efficiency-
trade-off. In Wortsman et al. (2021), an architecture is chosen, and its collection of weights is denoted by w. A linear subspace is parame- terized by two sets of network weights, W1 and w2. At training time, a network is sampled from the line defined by this pair of weights, w (a) = aw? + (1 a a)W2, where a E [O, 1]. The forward and backward pass of gradient descent are computed using w" (a), and gradient updates are backprop- agated to wh and w2- cosine distance regularization term used to ensure that w/ and w2 don't t converge to a single point.
We propose training a linear subspace of networks a similar manner, but we also bias the subspace to contain high-accuracy solutions at one end and high-efficiency solutions at the other end. To accomplish this, we introduce a parameter y used to control the compression level and a compression function f(w, 7) used to compress model's weights using compression level y. The parameter y induces different compression levels at different positions in the linear subspace, and therefore can be considered a function 7 7(a). When training, first sample network from the subspace, then compress it, obtaining a network with weights faw (()).. We then perform a standard forward and backward pass of gradient descent with backpropagating gradients to wi and w2-
In summary, our training method adjusts the standard subspace training method to include the appli- cation of a compression function f at compression level r(x). Note that we apply the compression after sampling the model w" (a) from the subspace, since we want the sampled model to be com- pressed (rather than Wi and w2) for efficient inference. We experiment with formulating our com- pression function as structured sparsity, unstructured sparsity, and quantization (Section 4). In all cases, the cost of computing f small compared the cost of network forward pass (Section A.1
3.2 COMPRESSIBLE POINTS
In Section 3.1 we discussed formulating our subspace as a line connected by two endpoints in weight space. This formulation requires additional storage resources deploy the subspace (Ta- [. since an extra copy of network weights is stored. For some cost-efficient computing devices, this storage may be significant. To climinate this need, we propose training degenerate subspace with single point in weight-space (rather than two endpoints). We still use a E [0, to control our compression ratio, but our subspace is parameterized by a single set of weights, w (a) w. The compressed weights are now expressed f(w (a), (()) plying varying levels of compression during each forward pass. The cosine distance regularization term included in the linear subspace is no longer needed because only one set of weights is trained.
f(,, 7a)).. This corresponds to ap-
This method still produces subspace of models the sense that, for each value of a, we obtain a different compressed network f(w, 7())). However, no longer use different endpoints of linear subspace to specialize one end of the subspace for accuracy and the other for efficiency. Instead, we condition set of network weights to tolerate varying levels of compression.
3.3 CIRCUMVENTING BATCHNORM RECALIBRATION
Previous works that train a compressible network require an additional training step to calibrate BatchNorm (Ioffe & Szegedy 2015) statistics, as in Universal Slimming (Yu & Huang 2019). Al- ternatively, they require storing separate BatchNorm statistics for each compression level, as in Slimmable Networks (Yu et al. 2018). This precludes both methods from evaluating at arbitrar- ily fine- mrae-grannd compression levels after deployment (Table TT. We seek to eliminate the need for recalibration or storage.
4

---

Under review as a conference paper at ICLR 2022
PPecResNet20 (CIFAR-IP)
Target Sparsity 0.0
Target Sparsity 0.888
Sparsity
0.961
0.937
Density
0,888
5
0.493
0.0
0
0.0
0.00
0.10
MAD Mean
MAD Mean
(e)
(b)
Sparsity
0.961
MAD Mean
0.937
0.1
0.1
0,888
0.493
0.0
0.0
10
15
5
10
15
Layer
Layer (d)
(c)
Test Accuracy
0.75
0.50
0.50
0.25
0,25
0.06
0.04
Mean (c)
(f)
Figure 2: Analysis of observed batch-wise means jú and stored BatchNorm means during testing for models trained with TopK unstructured sparsity. The models are trained with different target sparsities and evaluated with various inference- time sparsities. (a)-(b): The distribution of 4 pul across all layers. (c)-(d): The average value of | ju] for individual layers. (e)-(f): The correlation between the average of І ju] and test set error. Note that in (b) and (d), sparsities of 0 and 0.493 produce near-identical results, thus those curves are overlapping.
To understand the need for recalibration in previous works, recall that BatchNorm layers store the per-channel mean of the inputs μ and the per-channel variance of the inputs σ². The recalibration step is needed to correct la and σ², which are corrupted when a network is adjusted. Adjustments that corrupt statistics include applying structured sparsity, unstructured sparsity, and quantization.
In Figure we analyze the inaccuracies of BatchNorm statistics for two models trained with specific unstructured sparsity levels and tested with a variety of inference-time unstructured sparsity levels.
We calculate the differences between stored BatchNorm means μa and the true mean of a batch fa during the test epoch of cPreResNe120 (He al., 2016) model on CIFAR-10 (Krizhessky 2009) In Figure 2 and Figure Pp. we show the distribution of mean absolute differences (MADs) F
across all layers of the models. Models have lower BatchNorm errors when evaluated near sparsity levels that match their training-time target sparsity. Applying mismatched levels of sparsity shifts the distribution of these errors away from In Figure and Figure 21, we show the average of [14 jul across the test set for each of the BatchNorm layers. Across layers, the lowest error is achieved when the level of sparsity matches training. In Figure - and Figure 2 we show the average of Іμ fil and the corresponding test set accuracy for various levels of inference-time sparsity. We find that the increased error in BatchNorm is correlated with decreased accuracy. See Figure and Figure in the Appendix for similar analyses with structured sparsity and quantization.
Thus, BatchNorm layers' stored statistics can become inaccurate during inference- timee compres- sion, which can lead accuracy degradation. To circumvent the need for BatchNorm, we adjust our
5

---

Under review as a conference paper at ICLR 2022
networks to use GroupNorm (Wu & He 2018). This computes alternative normalization over g groups of channels rather than across a batch. It doesn't 't require maintaining a running average of there are stored statisties that can be corrupted
the mean and variance across batches of imput, the network changes. Formally, let be an index into a 4-dimensional image tensor, with compo- nents i [Ex, & ;rr indexing the batch, channel, height, and width, respectively. The set S; of indices used to compute the normalization of batch element i is
S = {k|KN = iN. Like * g/a] = Lic g/c]}.
(1)
Batch element z; is normalized by subtracting the mean of elements indexed by S, and dividing by the standard deviation of elements indexed by S.. GroupNorm typically uses the hyperparameter g 32, but it also includes InstanceNorm (Ulyanov et al.) 2016) (in which y c, where e is the number of channels) as a special case. We use g = c in structured sparsity experiments, since the number of channels is determined dynamically and is not always divisible by 32. We use 32 in all other experiments | affine transformation, with BatchNorm.
After applying this normalization, we also apply a learned per-channel
3.4 BIASED SAMPLING
The endpoints of a learned linear subspace typically demonstrate slightly reduced accuracy com- pared to the center of the line, as observed in Wortsman et al. (202T). Inspired by this observation, we use biased sampling to choose network endpoints more frequently when training a linear sub- space, to ensure that endpoints are accurate.
similar algorithmic choice made in Yu & Huang (2019), which proposes using a "sandwich method" for training. This method involves performing forward and backward passes at the max- imum and minimum sparsity levels, as well as a few randomly chosen sparsity levels in between. After this round of forward and backward passes, the gradient update is applied.
To implement our sampling of a, we use a stochastic function ox, Ω - [0, 1]", where S2 represents the state of the stochastic function. For each batch, sample backward passes using compressed networks (as" (a:)) for i E {l..... n.,, where a; is the 7th element of cx. The stochastic function used varies based on the compression function chosen. See
Ot., We perform n forward and
Section 1 (structured sparsity), Section 4.2 (unstructured sparsity), and Section 4.3 (quantization)
for more details. Our full method for training compressible subspace is shown in Figure lb
4 EXPERIMENTS
We present results in the domains of structured sparsity, unstructured sparsity, and quantization. We train using Pytorch (Paszke et al. 2019) on Nvidia Tesla V100 GPUs. On CIFAR-10 (Krizhevsky) 2009). we experiment with the pre-activation version of ResNet20 (He et al. 2016) presented in the PyTorch version of the open-source code provided by [Lu et al. (2017). We abbreviate it as "cPreResNer200" to denote that it's pre-activation network designed for usage with CIFAR- We use a pre-activation version because our Learning Efficient Convolutions (LEC) LLu et al. 2017) baseline requires it. We follow hyperparameter choices in Wortsman et al. 2022 for our methods and baselines. We warm up to our initial learning rate of 0.1 (or 0.025 in the case of quantization, for stability) over 5 epochs, then decay to 0 over 195 epochs using cosine schedule. We use batch size of 128 on a single GPU. We set the weight decay to 5 x 10-4, We run 3 trials for each experiment and report the mean and standard deviation (which is often too small to be visible). Our baseline cPreResNet (with BatchNorm) achieves an accuracy of 91. 01.69% on CIFAR-10.
For ImageNet (Deng al. 2009) experiments, we ResNet18 (He et al. 2016) and VGG19 (Si- monyan & Zisserman] 2014). We use the version of VGG19 provided by the same repository as our Ner2ResNet20 implementation. This implementation modifies VGG19 slightly by adding Batch- Norm layers and removing the last two fully connected layers. Again, we follow hyperparameter settings in Wortsman et al. (202I) for our methods and baselines. We warm up to our initial learning rate of 0.1 (or 0.025 in the case of quantization, for stability) over 5 epochs, then decay to 0 over 85 epochs using cosine schedule. We set the weight decay to x 10-5 We use a batch size of 128 for
We set g for the first few layers of cPreResNet20, hecause it has fewer than 32 channels.

---

Under review as a conference paper at ICLR 2022
cPrcResNet20 (CIFAR-10)
ResNet18 8 (ImageNet)
0.6
NS
Text Accuracy
0.75
US
LEC
0.4
0.50
LCS+P+IN
LCS+L-IX
0.2
0.55
0.0
0.5
0.0
0.5
Sparsity
Sparsity
Figure 3: Our method for structured sparsity using linear subspace (LCS+L+IN) and point sub- space (LCS+P+IN) compared to Universal Slimming (US) (Yu & Huang 2019), Network Slimming (NS) Yu et al. 2018), and Learning Efficient Convolutions (LEC) (Liu al. (017). LEC does not provide an open-source implementation of their method for ResNet18, so we omit it. We do not allow fine-tuning recalibration.
ResNet18 with a single GPU, and a batch size of 256 for VGG19 with 4 GPUs. Unlike when train- ing with CIFAR-10, 10, we run single trial for each experiment due to the increased computational expense of training on ImageNet. Our baseline ResNet18 (with BatchNorm) achieves an accuracy of 70.72% ImageNet, and our VGG19 (with BatchNorm) achieves 62.21%.
4.1 STRUCTURED SPARSITY
For structured sparsity, our method follows the algorithm in Figure Tb Our compression level caleu- lator y(a) [0. 1] + j0.25, 1.0] is the unique affine transformation over its domain and range. Our compression function 7(w,)) prunes away fraction - of the input and output channels in each layer (except the input the first layer and the output of the last layer). This is the same compres- sion scheme used in Network Slimming (NS) (Yu et al.] 2018) and Universal Slimming (US) YYu & Huang 2019). Our stochastic sampling function samples a as (0.25, 1.0, (0.25, 1). U (0.25, I) where U(abb) b) samples uniformly in the range (a, bj. This choice mirrors the "sandwich rule" used in US. As discussed in Section [3.3] we use a special case of GroupNorm (Wu & He 2018) known as InstanceNorm (Ulyanov et al. (016) since the number of channels in the network varies. See Section A.3 for additional details on our linear subspace setup.
When running baselines, we do not allow recalibration of BatchNorm (offe & Szegedy 2015) statistics (as used in US), or storage of a predefined set of BatchNorm statistics (as used in NS), since it would contradict our goal of adaptive compression to arbitrarily fine- ane-grained sparsity levels at inference time. We instead use a single BatchNorm layer, using only the channels not removed by the structured pruning step. Similarly, we do not allow fine-tuning (normally used in LEC).
The main difference between our point method and US is that we use InstanceNorm instead of Batch- Norm. The main difference between US and NS is that US includes the sandwich rule described above. Instead, NS samples at predefined sparsity levels [0.25, 0.50, 0.75, 1] at each iteration.
We present results for structured pruning ResNets in Figure (see Section A.5 for VGG19 re- sults and runtime characteristics). Our method produces a stronger efficiency-accuracyy trade-off than competing methods. The poor performance of LEC (Liu et al. (017)) occurs because this method ex- pects weights to be fine-tuned after sparsity is applied, which we do not allow. The dips and spikes shown for NS occur because this method trains at discrete width factors of [0.25, 0.50, 0.75, 1). which roughly correspond to sparsities of (0.9375, 0.75, 0.4325, 0J. This method shows stronger accuracies at these sparsities, with the exception of 0.9375. at which point, inaccuracies Norm statistics are more extreme (see Figure in the Appendix). Similarly, US peaks in accuracy at moderate sparsity rates, but decreases in accuracy at high sparsity rates. We believe this strong decrease for higher sparsity rates occurs because the BatchNorm channels used in this case are also used at all other sparsity rates, thus the stored statistics don't 't accurately reflect the statistics of the highly-sparse network.
Batch-
7

---

Under review as a conference paper at ICLR 2022
cPreResNet20 (CIFAR-10)
ResNet18 (ImageNet)
TopK-0.04
Test Accuracy
0.75
Topk-0,02
Topk-0.01
0.4
0.50
Topk-00000
LCS+I+GN
0.2
LCS+PIGN
0.25
0.0
0.94
0.96
0.98
0.92
0.94
Spansity
Sparsily
Figure 4: Our method for unstructured sparsity using a linear subspace (LCS+L+GN) and a point subspace (LCS+P+GN) compared to networks trained for a particular TopK target. The TopK target refers to the fraction of weights that remain unpruned during training.
Interestingly, our method with linear subspace (LCS+L+IN) and our method with a point subspace (LCS+P+IN) achieve very similar performance. We believe that the extra weights in LCS+L+IN are not necessary because individual channels are able to specialize, since some of them are only used for larger subnetworks.
4.2 UNSTRUCTURED SPARSITY
For unstructured sparsity, our method follows the algorithm in Figure Tb Our compression level calculator 7a))
a. Our compression function f(w, 7) is Topk sparsity (Zhu Gupta 2017), which removes a fraction of the weights with the smallest absolute value from each layer (except the input and output layer). Our stochastic sampling function samples one value for a in [0.005, 0.05) (see Section A.4 for additional details regarding sampling and TopK's 's warmup phase). We choose this interval because our networks are shown to tolerate high sparsity levels, losing almost no accuracy even 90% sparsity. See Section A.5 for experiments with broader ranges of sparsity, and for runtime characteristics of sparse models.
To our knowledge, fine-grained adaptive compression has been explored for unstructured spar- sity. Thus, we compare to standard networks (using BatchNorm) trained at specific sparsity levels and evaluated variety of sparsity levels. See Figure 4 for results. We find that our baselines always achieve highest accuracy at the sparsity level that matches their training-time TopK value, as expected. Accuracy quickly degrades other sparsity levels. By contrast, our method is able to achieve high accuracy for a wide variety of sparsity levels. Interestingly, our line method usually achieves higher accuracy at lower sparsity rates in Figure 4 but this pattern does not hold over wider sparsity range (Section A.5). Further investigation into this phenomena is future work.
4.3 QUANTIZATION
For quantization, our method follows Algorthm Our compression level calculator is 7(a) lb
2 + 6a. Our compression function f(w,7) 7) is affine quantization as described in (Jacob al. 2018
Our stochastic sampling function samples one value for uniformly over the set [1/6, 2/6,... .... 6/6
This corresponds to training with bit widths through 8 (we avoid lower bit widths circumvent training instabilities we encountered in baselines). We train without quantizing the activations for the first 80% of training, then add activation quantization for the remainder of training. Weights are quantized throughout training.
We present results for our method in Figure 5 comparing to models trained at a fixed bit width and evaluated at variety of bit widths. See Section A.5 for ResNet18 results, and for memory usage characteristics of models. Generally, baselines achieve high accuracy at the bit width at which they were trained, and reduced accuracy at other bit widths. Our method using linear subspace (LCS+L+GN) achieves high accuracy at all bit widths, matching or exceeding accuracies of indi- vidual networks trained for target bit widths. In the case of VGG19, we found that our accuracy
8

---

Under review as a conference paper at ICLR 2022
cPreResNet20 (CIFAR-I0)
VGG19 (ImageNet)
Bir W
Tes. Accurncy
0.75
Bii Widli-6
Bir Widthr4
Bir Width-3
0.50
LCS+P+GN
LCS GN
0.25
Bit Width
Bit Width
Figure 5: Our method for quantization using linear subspace (LCS+L+GN) and point subspace (LCS+P+GN) compared to networks trained for particular bit width target.
cPreResNet20 (CIFAR-10)
LCS -------N- Structured
LCS+L-GN- Unstructured
LCS+L+GN- Quantized
0.8
Test Accuracy
0.6
0.4
0.2
0.0
0.5
0.5
1.0
4
Sparsity
Sparsity
Bit Width
Learned line
Reversed line
Figure 6: Standard evaluation a linear subspace with network (a), (a), (Learned line), and evaluation when evaluating with reversed compression levels, f(w (a), ( (11=)) a)) (Reversed line).
exceeded the baselines. We believe part of the increase is due to GroupNorm demonstrating improved results on this network compared to BatchNorm (which does not happen with ResNets, as reported in Wu & He (2018))
4.4 FURTHER ANALYSIS
In Figure 6 we provide additional experimental evidence that our linear subspace method (LCS+L) trains subspace specialized for high-accuracy one end and high-efficiency at the other end. We plot the validation accuracy along our subspace, as well as the validation accuracy along our subspace when compressing with (aw" weights that were trained for low compression levels are evaluated with high compression levels, and vice versa. We see that this leads to a large drop in accuracy, confirming that our method has conditioned one side of the line to achieve high accuracy at high sparsities, and the other side of the line to achieve high accuracy at low sparsities.
= f(w (a), - a)). In other words, the
5 CONCLUSION
We present method for learning a compressible subspace of neural networks. Our subspace can be compressed real-time without retraining and without specifying the compression levels before deployment. We produce variety of models with fine- -en-ggrined efficiency- trade- off. We demonstrate that our generic algorithm can be applied to structured sparsity, unstructured sparsity, and quantization.

---

Under review as a conference paper at ICLR 2022
6 REPRODUCIBILITY STATEMENT
We open source our code to reproduce our results (including baselines) at https://withheldd for. review
REFERENCES
Gregory Benton, Wesley Maddox, Sanae Lotfi, and Andrew Gordon Gordon Wilson. Loss surface simplexes for mode connecting volumes and fast ensembling, In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. PLLR,, 18-24 Jul 2021. https:
V/proceedings mlr. www.:ppeesssvv.... http://////1///aaaaaaaaaaaaaaa..
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. fnce-for-all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Sauptik Dhar, Junyao Guo, Jiayi Liu, Samarth Tripathi, Unmesh Kurup, and Mohak Shah. On- device machine learning: An algorithms and learning theory perspective. arXiv preprint 1rXiv:1911.0062, 2019.
Luis Guerra, Bohan Zhuang, Ian Reid, and Tom Drummond. Switchable precision neural networks. arXiv preprint arXiv:2002.02815. 2020.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. arXiv preprint 12Xxv:5506.026200 2015.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Maxwell Horton, Yanzi Jin, Ali Farhadi, and Mohammad Rastegarii Layer data- CNN com- pression. CoRR, abs/201 1.09058, 2020.
Andrew Howard, Mark Sandler, Grace Chu, Liang- Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukum Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE/CVF International Conference Computer Vision, pp. 1314-1324, 2019.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.0481. 2017.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-456. PMLR, 2015.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew G. Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient anteger-arithmeticonnl inference. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2704-2713. 13, 2018
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. 2017 IEEE International Con- ference Computer Vision (ICCV), pp. 2755-2763, 2017.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias conrection. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 1325-1334, 2019.
10

---

Under review as a conference paper at ICLR 2022
Sharan Narang, G. Diamos, Shubho Sengupta, and Erich Elsen. Exploring sparsity in recurrent neural networks. ArXiv, abs/1704.05119, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary VeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. L'Alché-Buue E. Fox, and R. Garnett (cds.), Advances in Neural Information Processing Systems 32, pp. 8024-8033 8035. Curran Associates, Inc., 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Abigail See, Minh-Thangg Luong, and Christopher D. Manning. Compression of neural machine translation models via pruning. In CoNLL, 2016.
Moran Shkolník, Brian Chmiel, Ron Banner, Gil Shomron, Yury Nahshan, Alex Bron- stein, and Uri Weiser. Robust quantization: One model to rule them all. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5308-5317. Curran Asso- ciates, Inc., 2020. URL 3948ead6 339eadd22224421666333333333
meurrps httppss.cppppp/pppppepp cc/pppper/222///1e
pdf
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint
2014.
Mingxing Tan and Quoc Le. EfficientNer: Rethinking model scaling for convolutional neural net- works. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th In- ternational Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6105-6114. PMLR, 09-15 Jun 2019.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv:1607.0802) 2016.
Mitchell Wortsman, Maxwell C Horton, Carlos Guestrin, Ali Farhadi, and Mohammad Rastegari. Learning neural network subspaces. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11217-11227. PMLR, 18-24 Jul 2021.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), September 2.18.
Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1803-1811, 2019.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In International Conference on Learning Representations, 2018.
Michael Zhu and Suyog Gupta. To prune, not to prune: exploring the efficacy of pruning for model compression. arXiv preprint 17Xiv:1710.01878, 2017.
A APPENDIX
A. COMPUTATIONAL COST OF COMPRESSION
We justify claim that network compression method is real-time. For a typical convolutional layer without downsampling, the computational cost is (d, d2 H, W, k,), where d, is the number of input channels, d2 is the number of output channels, H, is the input tensor height, H2 is the input
11

---

Under review as a conference paper at ICLR 2022
tensor width, Ks is the kernel height, and kw is the kernel width. For our line method, we must calculate w (a), which takes ( dy for each convolution, which is negligible compared to the convolution. For our point method, calculating w (a) requires no computation.
We must also calculate the compressed network, (a), Ma))). Our structured sparsity method (described in Section (4.1) takes (1) for each convolution, since we only need mark each layer with the number of convolutional filters to ignore. Our unstructured sparsity method (TopK, de- scribed in Section 4.2p takes O(d,
for each convolution (to calculate the threshold, then discard parameters below the threshold). Our quantization method (Jacob et al. 2018) takes dzkn,kw) for each convolution (to calculate the affine transform parameters and apply them). In all cases, the complexity of computing f(w" (a):?(a) is lower than the cost of the convolutional forward passes, meaning our method can considered real-time.
A.2 FURTHER BATCHNORM ANALYSIS
In Figure 2 we analyzed the inaccuracies of BatchNorm statistics for models trained with unstruc- tured sparsity. We show similar analysis for the of Universal Slimming (US) (Yu & Huang.
2019) and Network Slimming (NS) (Yu et al. 2018) in Figure 7 We show the case of quantization
Figure 8
A.3 STRUCTURED SPARSITY DETAILS
When training our method with a line in the structured sparsity setting, we do not use two sets of weights (e.g. W1 and w2, Section 3.1) for convolutional filters. Instead, we only use two sets of weights for affine transforms in GroupNorm (Wu & He 2018) layers. For the convolutional filters. single set of weights, similar our point formulation (and similar to US (Yu & Huang, 2019) and NS (Yu et al.) (201)). By contrast, we use two sets of weights for convolutional filters as well as for affine transforms when experimenting with unstructured sparsity (Section 4.2)
instead
and quantization (Section
The reason for only using one of convolutional filters in the structured sparsity setting is that the filters themselves are able to specialize, even without an extra copy of network weights. Some filters are only used in larger networks, so they can learn to identify different signals than the filters used in all subnetworks. Note that this filter specialization argument does not apply to our unstructured or quantized settings.
In preliminary experiments, we found that using a single set of weights for convolutions in our structured sparsity experiments gave a slight improvement over using two sets of weights (roughly 2% for cPreResNer20 (He et al. 2016)) slight difference may be attributed to the subspace training framework's ' regularization term (which increases cosine distance between W1 and w2), which discourages identical filters from being used in both Wa and W2 even if they are optimal.
CIFAR-10 (Krizhevsky 2009)). We hypothesize that this
A.4 UNSTRUCTURED SPARSITY DETAILS
It is typical to include warmup phase when training models with TopK sparsity (Zhu & Gupta. 2017). In our baselines in Section 4.2 we increase the sparsity level from 0% to its final value over the first 80% of training epochs. For our method, sparsity values fall within range, there is no single target sparsity value to warm up to.
For our point method (LCS+P), we simply train for the first 80% of training with the lowest sparsity value in our sparsity range. We finish training by sampling uniformly between the lowest and highest sparsity levels.
For our line method (LCS+L), our choice of sparsity level is tied to our choice of weight-space parameters through a. We implement warmup by simply adjusting x(a) to apply less sparsity early in training, warming up to our final sparsity rates over the first 80% of training.
In detail, let Qmia and Omax correspond
minimum and maximum alpha values (for example, in Section 4.2) amin = 0.005, and amax = 0.05). As motivated in Section we bias sampling of towards the endpoints of line. We set [Omin]] with 25% probability, [Omax
12

---

Under review as a conference paper at ICLR 2022
crrcResNe120 (CIFAR-10)
NS
US
Spinsit
0.955
0.856
1.0
Danily
0,606
0.5
0.455
0.0
0.0
0.233
0.0
0,2
0.4
0.0
0.2
0.1
0.5
MAD Moan (4)
MAD Mean (b)
Spinsit
0.1
0.955
0.56
MAD
0.2
0.2
0,606
0.455
0.0
0.0
0233
10
15
10
15
Layer
Layer (4)
(6)
0.75
0.75
0.50
0.10
0.15
0.075
0.100
0.125
0.150
MAD Meam
MAD Meam (0
(0)
Figure 7: Analysis of the mean absolute difference between observed batch-wise means and stored BatchNorm means during testing for cPreResNet models trained with NS (Yu et al. 2018) or US Yu & Huang 2019). (a)-(b): The distribution of ІA fu across all layers. (c)-(d): The average value of 14 μ for each individual BatchNorm layer. (c)-(f): The correlation between the average of [12 fi) and test set error.
13

---

Under review as a conference paper at ICLR 2022
cPreResNe120 (CIFAR-10)
Target Bit Width =
Target Bit Width =
Bit Width
3
10
Density
6
5
8
0
0.0
0.1
0.2
0.0
0.2
0.4
MAD Mean (a)
MAD Mean
(b)
Bit Width
3
MAD Mean
0.2
0.2
4
6
0.1
0.1
8
0.0
0.0
10
15
10
Layer
Layer (d)
(c)
Test Accuracy
0.75
0.75
0.50
0.50
0.25
0.02
0.04
0.06
0.10
0.12
0.14
0.16
MAD Mcan
MAD Mcan
(c)
(1
Figure 8: Analysis of the mean absolute difference between observed batch- wise means ja and stored BatchNorm means a during testing for cPreResNet models trained with different quantization bit widths. (a)-():: The distribution of μ çul across all layers. (c)-(d): The average value of [14 - fu] for each individual BatchNorm layer. (f)-(f): The correlation between the average of f ju| and test set error.
14

---

Under review as a conference paper at ICLR 2022
VGG19 (ImageNet)
0.7
US LEC
0.6
LCS-LAIN
0.5
LCS+PHIN
Test Accuracy
0.4
0.3
0.2
0.1
0.0
0.0
0.2
0.4
0.6
0.8
Spursity
Figure 9: Our method for structured sparsity using linear subspace (LCS+L+IN) and a point sub- space (LCS+P+IN), compared to Universal Slimming (US) (Yu Huang 2019) Network Slimming (NS) Yu et al. 2018), and Learning Efficient Convolutions (LEC) (Liu et al. 2017).
with 25% probability, and a [U' (Amin d'ax )] with 50% probability, where U(a,b) b) samples uniformly in the range (a, b). To warm up sparsity rates, we choose
d = max (1 c/t, ((a) (1 a1 d).
(2)
(3)
where is the current iteration number, and / is the total number of iterations in the first 80% of training. At the beginning of training, = 1, and ((x) 0, corresponding to sparsity level for all values of a. Once 80% of training is finished, 0, and 7(a) I for the remainder of training. This corresponds to our final sparsity range.
A.5 ADDITIONAL RESULTS
Structured Sparsity: We present results for VGG19 on ImageNet in the structured sparsity set- ting in Figure 9 Our method produces better efficiency-accuracy trade-off than baselines. Note that VGG19 with GroupNorm (Wu & He 2018) had a higher baseline accuracy than VGG19 with BatchNorm (loffe & Szegedy 2015) (as noted in Section 4.3)
We also provide a table demonstrating the memory, flops, and runtime of structured sparsity models in Table 2
Unstructured Sparsity: We present results for a wider range of sparsity values in our unstructured sparsity experiments in Figure 11 We change sparsity limits from [0.005, 0.05] (in Section 4.2)
to [0.025, 1.0j. All other training details are unchanged. We find that our method is able to produce higher accuracy over a wider range of sparsities than baselines.
We also provide a table of unstructured sparsity results in the high sparsity regime (Table 3 and the wider sparsity regime (Table 4h. showing memory usage and FLOPS.
Quantization: We present quantization results for ResNet18 in Figure I0 We find that our models approach the accuracy of models trained single bit width, and our models generalize better to other bit widths.
We also provide table of quantization results in Table showing memory usage of the models.
15

---

Under review as a conference paper at ICLR 2022
ResNet18 (ImageNet)
0.7
Bit W
Bi Walth
0.6
Bit Widthma
Bit Wilth-3
0.5
LCS-P-GN
Test Aecuracy
LCS+LIGN
0.4
0.3
0.2
0.1
0.0
4
5
Bit Width
Figure 10: Our method for quantization using a linear subspace (LCS+L+GN) and a point subspace (LCS+P+GN) compared to networks trained for particular bit width target.
Results for structured sparsity. Note that models particular architecture and sparsity level all have the same runtime characteristics (memory, FLOPS, and runtime), so we only report one value. Runtime was measured on MacBook Pro (16-inch, 2019) with a 2.6 GHz 6-Core Intel Core i7 processor and 16GB 2667 MHz DDR4 RAM. Memory consumption refers the size of model weights in the currently executing model.
Table 2:
Sparsity (%)
0
43.491
60.614
74.655
85.614
93.491
cPreResNer20 (CIFAR-10)
FLOPS (x 10%)
33.75
19.07
13.29
8.55
4.85
2.2
Memory (MB)
0.87
0.49
0.34
0.22
0.12
0.06
Runtime (ms)
3.13
1.83
1.64
1.28
Acc (ICC++P+N)
87.51
86.07
84.46
82.02
78.39
75.96
Ace (I.SSI++N)
88.32
85.69
83.69
80.84
77.5
75.31
Acc (US)
70.62
83.13
81.11
62.18
40.04
21.81
Ace (NS)
72.87
75.46
$7.09
70.07
16.86
19.76
Sparsity (%)
0.0
42.91
59.89
73.88
84.89
92.91
ResNet18 (ImageNet)
Fl.OPS (x 10%)
1814.1
1042.66
736.42
483.16
282.89
135.61
Memory (MB)
46.72
26.67
18.74
12.2
7.06
3.31
Runtime (ms)
45.85
30.34
22.51
14.31
9.84
6.02
Acc (LCS+P+IN)
63.32
60.21
57.42
53.77
48.75
44.62
Acc (LCS+L+IN)
64.21
59.3
56.18
52.73
48.35
44.5
Acc (US)
58.91
60.39
53.76
44.72
22.51
8.34
Ace (NS)
50.63
57.58
22.93
48.52
0.84
1.34
Sparsity (%)
0.0
43.28
60.35
74.37
85.35
93.28
VGG19 (ImageNet)
FLOPS (x 10%)
19533.52
11008.50
7656.48
4911.33
2773.1
1241.81
Memory (MB)
82.12
46.58
32.56
21.04
12.03
5.52
Runtime (ms)
388.4 49
246.81
172.64
105.77
60.0
29.55
Acc (LCS+P+IN)
66.77
64.47
62.11
58.35
53.45
49,5
Acc (LCS+L+IN)
67.7
64.16
61.37
57.6
52.81
49.44
Ace (US)
36.27
43,99
42.17
36.5
20.42
7.07
Acc (NS)
34.05
44.91
18.44
36.26
0.57
0.14
16

---

Under review as a conference paper at ICLR 2022
cPreResNet20 (CIFAR-10)
TopK=09
TopK-0.5
0.8
TopK-0.1
TopK-0.022
LCS+PHGN
Test Accuracy
LCS+L+GN
0.6
0.4
0.2
0.0
0.2
0.4
0.6
0.8
1.0
Sparsity
ResNet1 (ImageNet)
0.7
TopK-09
TopK=0.5
0.6
TopK-0.1
TopK-0.025
0.5
LCS+P
Test Accuracy
LCS+LAGN
0.4
0.3
0.2
0.1
0.0
0.2
0.6
0.8
Sparsity
VGG19 (ImageNet)
0.7
TopK-099
TopK-055
0.6
TapK-0.1
TapK-0.025
0.5
LCS+P+GN
Test Accuracy
LCSILIGN
0.4
0.3
0.2
0.1
0.0
0.0
0.2
0.4
0.8
Sparsity
Figure 11: Our method for unstructured sparsity using linear subspace (LCS+L++NN and a point subspace (LCS+P+GN) compared to networks trained for particular TopK target. The TopK target refers to the fraction of weights that remain unpruned during training.
17

---

Under review as a conference paper at ICLR 2022
Table 3: Results for unstructured sparsity in the high sparsity regime. Note that models of particu- lar architecture and sparsity level all have the same runtime characteristics (memory and FLOPS), only report one value. Runtime was measured because it requires specialized hardware. So, we follow the standard practice of only reporting memory and flops. Memory consumption refers to the size of model weights the currently executing model.
Sparsity (%)
95.66
96.15
96.64
97.14
97.63
98.
cPrcResNe120 (CIFAR-10)
FLOPS (x 1x%)
1.46
1.29
1.13
0.96
0.79
0.63
Memory (MB)
0.04
0.03
0.03
0.02
0.02
0.02
Acc (LCS+P+GN)
70.18
69.22
67.74
63.78
54.91
24.92
Acc (LCS+L+GN)
73.2
69.5
64.42
54.77
42.62
30.61
Acc (TopK=0.04)
14.83
12.83
10.8
9.66
10.01
9.79
Acc (TopK=0.02)
10.25
10.53
78.7
10.56
10.05
10.09
Acc (TopK=0.01)
10.02
9.97
9.96
10.64
59.2
10.79
Acc (T6pK=0.005)
10.0
10.08
9.81
10.03
10.0
41.44
Sparsity (%)
92.67
93.15
93.62
94.1
94.58
95.06
ResNet18 (ImageNet)
FLOPS (x106)
169.42
160.94
152.46
143.98
135.51
127.03
Memory (MB)
3.42
3.2
2.98
2.76
2.53
2.31
Acc (LCS+P+GN)
51.5
51.1
50.37
48.62
44.8
30.69
Acc (LCS+L+GN)
58.56
57.01
54.72
50.98
45.01
39.32
Acc (Topk=0.004)
5.96
0.9
0.18
0.11
0.1
0.1
Acc (TopK=0.02)
24.66
45.37
59.92
3.84
0.1
0.11
Acc (TopK=0.01)
0.12
0.1
0.1
0.11
53.95
0.11
Acc (TopK=0.005)
0.1
0.12
0.11
0.1
0.11
46.35
Table 4: Results for unstructured sparsity in the wide sparsity regime. Note that models of partic- ular architecture and sparsity level all have the same runtime characteristics (memory and FLOPS), so we only report one value. Runtime was not measured, because it requires specialized hardware (so most unstructured pruning works report memory and flops). Memory consumption refers the size of model weights in the currently executing model.
Sparsity (%)
0.0
49.31
86.29
91.22
93.68
96.15
cPreResNe120
FLOPS (x10))
33.75
17.11
4.62
2.96
2.13
1.29
(CIFAR-10)
Memory (MB)
0.87
0.44
0.12
0.08
0.05
0.03
Acc (LCC+P+GN)
89.64
89.34
82.34
75.24
67.55
47.1
(LCS+L+GN)
85.97
84.12
80.86
78.44
75.0
72.81
Acc (TopK=0.9)
91.66
83.17
10.54
10.0
9.76
10.0
Acc (TopK=0.5)
91.16
91.17
10.64
10.14
10.0
10.0
Acc (TopK=001)
40.74
40.74
78.56
39.54
11.67
10.26
Acc (TopK=0.025)
9.64
9.64
10.65
10.0
9,98
82.15
Sparsity (%)
0.0
47.77
83.59
88.37
90.76
93.15
ResNet18 (ImageNet)
FLOPS (x109)
1814.1
966.32
330.49
245.72
203.33
160.94
Memory (MB)
46.72
24.4
7.66
5.43
4.32
3.2
Acc (LCS+P+GN)
69.25
69.12
64.53
60.36
55.58
41.48
Acc (LCS+L+GN)
66.36
64.0
61.3
59.72
57.55
56.58
Acc (TopK=0.9)
70.57
63.17
0.1
0.1
0.1
0.1
Ace (TopK=0.55
70.15
70.15
0.24
0.17
0.1
0.12
Ace (TopK=0.1)
58.21
58.21
66.44
29.93
Acc (TopK=0.025)
0.12
0.12
0.13
0.17
61.33
Sparsity (%)
0.0
48.75
85,31
90.19
92.62
95.06
VGG19 (ImageNet)
FLOPS )
19533.52
9822.65
2539.51
1568.42
1082.88
597.34
Memory (MB)
82.12
42.09
12.06
8.06
6.06
4.06
Ace (LCS+P+GN)
70.0
69.45
62.11
55.62
48.58
25,87
Ace (LCSSL++N)
65.1
62.85
58.85
56.07
52.04
52.8
Acc (TopK=0.9)
61.72
56.67
0.1
0.0
0.0
0.1
Acc (Topk=0.55
61.07
61.07
0.15
0.09
0.06
0.1
Ace (TopK=0.1)
7.91
7.91
50.13
16.8
0.13
0.09
Acc (Topk=0.025)
0.09
0.09
0.1
0.16
3.91
46.66
18

---

Under review as a conference paper at ICLR 2022
Table 5: Results for quantization. Note that models of a particular architecture and quantization bit width all use the same memory, only report one value. Runtime not measured, because requires specialized hardware. Memory consumption refers to the size model weights in the currently executing model.
Bit Widths
8
7
5
4
Memory (MB)
0.22
0.19
0.17
0.14
0.11
0.08
Acc (LCS+P+GN)
89.97
90.0
89.88
89.26
86.25
65.26
cPreResNet20 (CIFAR-10)
Ace (LCS+L+GN)
87.44
88.09
87.9
86.93
84.2
75.16
Ace (Bit Width=8)
91.36
91.02
90.47
87.98
65.91
16.65
Acc (Bit Width=6)
91.07
90.89
91.26
87.12
63.1
18.78
Acc (Bit Width=4)
84.93
84.65
84.77
82.37
88.22
25.19
Acc (Bit Width=3)
55.71
55.56
57.01
55.66
44.83
73.89
Memory (MB)
I1.69
10.23
8.77
7.31
5.84
4.38
ResNet18 (ImageNet)
Acc (LCS+P+GN)
63.59
63.51
63.15
61.82
55,89
5.48
Ace (LCSS++GN)
65.79
66.31
64.4
60.89
55.16
43.98
Ace (Bit Width=8)
70.36
69.2
67.18
41.67
0.54
0.08
Acc (Bit Width=6)
66.8
66.26
69.17
44.0
0.43
0.1
Acc (Bit Width=4)
9.82
9.57
8.57
4.84
59.39
0.1
Acc (Bit Width=3)
0.1
0.12
0.09
0.1
0.12
25.61
Memory (MB)
20.542
17.974
13,406
12.839
10.271
7.703
VGG19 (ImageNet)
Ace (LCS+P+GN)
57.83
57.74
57.24
55.64
47.11
0.25
Acc (LCS+L+GN)
63.72
64.27
62.08
57.87
50.05
33.46
Acc (Bit Width=8)
60.85
59.32
56.13
39.73
0.3
0.09
Acc (Bit Width=6)
56.52
55.35
59.89
23.24
0.16
0.1
Acc (Bit Width=4)
0.13
0.13
0.14
0.11
51.33
0.1
Acc (Bit Width=3)
0.12
0.1
0.1
0.12
0.09
20.72
19


Reviews:
[{"review_id": "pddY3qvZKc2", "reviewer": "Reviewer_jBcH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The novelty of this paper looks limited. It is an extension of recent work on learning network subspaces. Meanwhile, compared to existing works with adaptive networks, the advantages of this work are the finer-grained compress level and no need to recalibrate BN. The benefit of such improvements looks trivial.\n\nMany vital details are missing in this paper. The detailed form of the compression level function $\\gamma(\\alpha)$ and the compression function $f(\\omega, \\gamma)$ are not provided. These are the core of the algorithm and the authors need to show them. \n\nAlso, the authors do not explain what determines the dimension n of the stochastic function $\\alpha$. If $\\alpha$ controls the sparsity at the level of each weight, n will be extremely large, and the training overhead of the proposed algorithm is extremely large because they need to do n forward passes and backward passes for each batch. Even $\\alpha$ controls the sparsity at the layer level, the training overhead will still be formidable. I cannot find much information about n in this paper. The authors need to provide more details about the dimension n of $\\alpha$.", "summary_of_the_paper": "This paper proposes to balance the inference accuracy and efficiency by training a subspace of neural networks and then adapting the network within the subspace at inference time.", "summary_of_the_review": "Many important details are missing in this paper. For example, the finer-grained compression level is a major selling point of this paper, but the authors did not even provide the compression level function $\\gamma(\\alpha)$.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "N-G9v9L6wr_", "reviewer": "Reviewer_FDfN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: the paper is well motivated, as adaption to runtime available resource is important. \n\nWeaknesses:\n* The method is a direct extension of the learning subspace method.\n* There are important details missing from the paper. E.g., there is no info on how to generate a alpha via the state of hardware in runtime. This creates severe difficulty in understanding and reproducing the method.\n* There is no material to measure hardware performance. For example, only the accuracy of the classification models are given, but memory bandwidth, latency or FPS are not available to quantitatively measure the advantage. \n* The paper is not properly peer-compared. For example, the work is not well compared with pruning and quantification methods.\n\nQuestions for the Author(s):\n* please elaborate on the definition of the compression function f and the intuition behind?\n* how to choose the hyper-parameter alpha in a hardware run-time?\n* What will happen to the arch of a model, if pruning is also performed? \n* If using this method in a hardware, how to change the quantization meta-parameters(scale and zero-point) accordingly?\n\n    ", "summary_of_the_paper": "The paper present a method for learning a compressible subspace of neural networks that contains a fine-grained spectrum of models that range from highly efficient to highly accurate. The proposed method allows choosing the proper point of trade-off between accuracy and efficiency at inference time, according to the available resources. There are also efforts to reduce the runtime tweaking overhead like replacing BatchNorm with GroupNorm.", "summary_of_the_review": "The paper proposes a method that reasonably extends the learning subspace method to allow performing accuracy-efficiency tradeoff according to runtime available resource. The method has been evaluated on several classification tasks and find to be useful.\n\nHowever, the paper does not clearly explain how the compression is performed, with important details like choice of alpha missing. The measurement of speedup is not that quantitative, lacking realworld test stats. It is very difficult to evaluate the contribution of this paper under these conditions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "G1OaXu_Jhkr", "reviewer": "Reviewer_EG95", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and easy to follow. The ideas of constructing a linear subspace and using the function $f(w(\\alpha), \\gamma (\\alpha))$ to perform compression during inference are novel. The analysis of BN parameters in adjustment provides quantitative analysis in this area. The experiments are exhaustive and can well support their ideas.\n\nHowever, as the paper claims, they bias the subspace to contain high-accuracy solutions at one end and high-efficiency solutions at the other end. In my understanding, two endpoints are using the same network architecture. How to train a network to obtain $w_1$ and $w_2$ in this case?", "summary_of_the_paper": "The paper proposes to learn compressive subspaces which can adaptively compress the network during inference. It constructs either linear subspace or a single endpoint for compression. It replaces BN with GroupNorm to avoid re-calibrating during inference or after adjustment. Their method is evaluated in three different scenarios: structured sparsity, unstructured sparsity, and quantization.", "summary_of_the_review": "I think the paper is well written; the method is novel and interesting; the experiment can well support the claims. Just need to clarify some details", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "09Hl29L8661", "reviewer": "Reviewer_Dpge", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "=== Strengths ===\n\nTable 1 is useful in summarizing the related works and the claimed advantages for LCS.\n\nThe experiment section listed a significant amount of detail on hyperparams and methods.\n\nThe small discussion and figures on batch norm stats shifts were interesting and helpful for motivating group and instance norm for this application.\n\nThis area of adaptive inference is becoming more and more important with larger models and specialized big-little architectures.\n\n=== Weaknesses ===\n\nIt is unclear how these networks switch between models at inference time, which of course should depend on the type of compression used.  For sparsity, this seems like it would require dynamically pruning the model at inference time which seems very dangerous. For unstructured sparsity, this may require special hardware for taking advantage of that unstructured sparsity. For quantization, this may require hardware can support fine-grained switching of the quantization bitwidth.\n\nMy understanding is that batch norm stats have to be recomputed for NS and US in a post-training way but not necessarily at inference time. It doesn't seem fair to avoid this step since it can be done before model deployment and takes a fraction of the training time. Leaving it out in the evaluation also nullifies the comparison if this is not fair.\n\nThe claim that other methods need additional batch norm params and cannot support fine-grained compression level is mostly correct, but the importance of this seems overstated. In practice, it seems more reasonable to chose a smaller subset of model configurations that can be fully tested before deployment, and the batch norm params should be nearly negligible compared to the weights. Also, quantization LCS does of course limit the number of compression levels, and structured pruning LCS limits the number of compression levels to the number of channels (which is similar to US).\n\nThe existence of gamma and alpha together is confusing to follow. Since the parameterization in linear, it seems like only one of these should be necessary.\n\nThere should be other works included for building robust compressible models, e.g. Robust Quantization (Neurips20).\n\nThe writing is clear but repetitive in some areas. For example, I believe there are 5 sentences talking about being inspired by Wortsman in the first few pages.\n\n=== Questions ===\nIn the Related Works, the neural subspace method is described as operating on simplices, but the description in Section 3.1 seems to be on lines. Is this deliberate?\n\nPlease correct me if I'm wrong but isn't the unstructured compressible point method Dropout? There might be differently weighted probabilities and dynamic dropout probabilities, but they seem fundamentally the same. \n\nFor quantization, what hardware supports dynamic fine-grained switching from 3-8 bits? How are the pruned channels or pruned individual weights chosen at runtime in an adaptive way? ", "summary_of_the_paper": "This paper introduces learnable compressible subspaces, which attempts to learn a set of models that can be switched at inference time to adapt to different resource requirements. This work is motivated by previous work in neural subspaces and slimmable networks. It is evaluated on CIFAR10 and ImageNet and compared against other recent works for adaptable inference models. These results show under certain conditions LCS can maintain higher accuracies at larger sparsities compared to other works.", "summary_of_the_review": "This paper is an interesting proposal that attempts to apply the ideas of neural subspaces to produce a set of compressed models at varying points on the accuracy / efficiency curve. Yet, these methods in the end seem more about learning robust compressible models and stray far from the original neural subspace idea, especially with compressible points. In my current understanding, these networks seem to have no demonstrated advantage to universally slimmable networks, which have a simpler validation process, runtime switching method, and more intuitive training procedure. The comparison against these networks and others needs to be better justified since I currently do not understand why fine-tuning is not allowed. If I misunderstood the method significantly, I would be willing to increase my score, but currently I suggest rejecting the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a method for adaptive network compression at inference time. However, the paper contains various issues raised by the reviewers that needs to be addressed.
Decision: Reject