Year: 2021

Abstract:
When deploying deep learning models to a device, it is traditionally assumed that available computational resources (compute, memory, and power) remain static. However, real-world computing systems do not always provide stable resource guarantees. Computational resources need to be conserved when load from other processes is high or battery power is low. Inspired by recent works on neural network subspaces, we propose a method for training a compressible subspace of neural networks that contains a fine-grained spectrum of models that range from highly efficient to highly accurate. Our models require no retraining, thus our subspace of models can be deployed entirely on-device to allow adaptive network compression at inference time. We present results for achieving arbitrarily finegrained accuracy-efficiency trade-offs at inference time for structured and unstructured sparsity. We achieve accuracies on-par with standard models when testing our uncompressed models, and maintain high accuracy for sparsity rates above 90% when testing our compressed models. We also demonstrate that our algorithm extends to quantization at variable bit widths, achieving accuracy on par with individually trained networks.

text: When deploying deep learning models to a device, it is traditionally assumed that available computational resources (compute, memory, and power) remain static. However, real-world computing systems do not always provide stable resource guarantees. Computational resources need to be conserved when load from other processes is high or battery power is low. Inspired by recent works on neural network subspaces, we propose a method for training a compressible subspace of neural networks that contains a fine-grained spectrum of models that range from highly efficient to highly accurate. Our models require no retraining, thus our subspace of models can be deployed entirely on-device to allow adaptive network compression at inference time. We present results for achieving arbitrarily finegrained accuracy-efficiency trade-offs at inference time for structured and unstructured sparsity. We achieve accuracies on-par with standard models when testing our uncompressed models, and maintain high accuracy for sparsity rates above 90% when testing our compressed models. We also demonstrate that our algorithm extends to quantization at variable bit widths, achieving accuracy on par with individually trained networks.

heading: 1 INTRODUCTION
text: Deep neural networks are deployed to a variety of computing platforms, including phones, tablets, and watches (Dhar et al., 2019). Networks are generally designed to consume a fixed budget of resources, but the compute resources available on a device can vary while the device is in use. Computational burden from other processes, as well as battery life, may influence the availability of resources to a neural network. Adaptively adjusting inference-time load is beyond the capabilities of traditional neural networks, which are designed with a fixed architecture and a fixed resource usage.
A simple approach to the problem of providing an accuracy-efficiency trade-off is to train multiple networks of different sizes. In this approach, multiple networks are stored on the device and loaded into memory when needed. There is a breadth of research in the design of efficient architectures that can be trained with different capacities, then deployed on a device (Howard et al., 2017; Sandler et al., 2018; Howard et al., 2019). However, there are a few drawbacks with using multiple networks to provide an accuracy-efficiency trade-off: (1) it requires training and deploying multiple networks (which induces training-time computational burden and on-device storage burden), (2) it requires all compression levels to be specified before deployment, and (3) it requires new networks to be loaded into memory when changing the compression level, which prohibits real-time adaptive compression.
Previous methods address the first issue in the setting of structured sparsity by training a single network conditioned to perform well when varying numbers of channels are pruned (Yu et al., 2018; Yu & Huang, 2019). However, these methods require BatchNorm (Ioffe & Szegedy, 2015) statistics to be recalibrated for every accuracy-efficiency configuration before deployment. This requires users to know every compression level in advance and requires storage of extra parameters (which can be problematic for low-compute devices). Another method trains a larger network that can be queried for efficient subnetworks (Cai et al., 2020), but requires users to reload network weights, and to know every compression level before deployment.
We address the issue of deploying a model that can be compressed in real-time without specifying the compression levels beforehand. Inspired by Wortsman et al. (2021), we formulate this problem as computing an adaptively compressible network subspace (Figure 1a). We train a linear subspace using two sets of weights, conditioning them so that a convex combination of them produces an accurate network. We expand on Wortsman et al. (2021) by (1) specializing our linear subspace to contain an arbitrarily fine-grained efficiency-accuracy trade-off (meaning, any compression level within predefined limits can be chosen), and (2) replacing BatchNorm (Ioffe & Szegedy, 2015) with GroupNorm (Wu & He, 2018) to avoid BatchNorm recalibration.
Contributions: Our contributions are as follows. (1) We introduce our method, Learning Compressible Subspaces (LCS), for training models that can be compressed with arbitrarily fine-grained compression levels in real-time after deployment. To our knowledge this has not been done before. (2) We demonstrate a new use for neural network subspaces, specializing at one end for high-accuracy and at the other end for high-efficiency. (3) We provide an empirical evaluation of our method using structured sparsity, unstructured sparsity, and quantization. (4) We open source our code at https://withheld.for.review.

heading: 2 RELATED WORK
text: Architectures Demonstrating Efficiency-Accuracy Trade-Offs: Several recent network architectures include a hyperparameter to control the number of filters in each layer (Howard et al., 2017; Sandler et al., 2018; Howard et al., 2019) or the number of blocks in the network (Tan & Le, 2019). In Once For All (Cai et al., 2020), the need for individually training networks of different sizes is circumvented. Instead, a single large network is trained, then queried for efficient subnetworks.
We differ from these methods in two ways. First, we compress on-device without specifying the compression levels before deployment. Previous works require training separate networks (or in
the case of Once For All, querying a larger model for a compressed network) before deployment. Second, we don’t require deploying a set of weights for every compression level.
A method for post-training quantization to variable bit widths appears in Shkolnik et al. (2020), but it relies on calibration data and cannot be run on low-compute edge devices. Thus, compression levels must be specified before deployment. They also present a method for quantization-aware training to variable bit widths, but most of their results keep activation bit widths fixed, whereas we vary it.
Adaptive Compression: Recent works train a single neural network which can be adaptively configured at inference time to execute at different compression levels. These methods are the closest to our work. In Learning Efficient Convolutions (LEC) (Liu et al., 2017), the authors train a single network, then fine-tune it at different structured sparsity rates (though they also present results without fine-tuning, which is closer to our use case). Other methods train a single set of weights conditioned to perform well when channels are pruned, but require recalibration (or preemptive storage) of BatchNorm (Ioffe & Szegedy, 2015) statistics at each sparsity level. These methods include Network Slimming (NS) (Yu et al., 2018) and Universal Slimming (US) (Yu & Huang, 2019). Similar methods train a single network to perform well at various levels of quantization by storing extra copies of BatchNorm statistics (Guerra et al., 2020).
We differ from these methods by avoiding the need to recalibrate or store BatchNorm statistics (Section 3.3) and by allowing for fine-grained selection of compression levels at inference time, neither of which have been done before (Table 1). Additionally, our method is broadly applicable to a variety of compression methods, whereas previous works focus on a single compression method.
Other Post-Training Compression Methods: Other works have investigated post-training compression. In Nagel et al. (2019), a method is presented for compressing 32 bit models to 8 bits, though it has not been evaluated in the low-bit regime. It involves running an equalization step and assuming a Conv-BatchNorm-ReLU network structure. A related post-training compression method is shown in Horton et al. (2020), which shows results for both quantization and sparsity. However, their sparsity method requires a lightweight training phase. We differ from these methods in providing real-time post-deployment compression for both sparsity and quantization without making assumptions about the network structure.
Sparse Networks: Sparsity is a well-studied approach to reduce neural network compute. Recent works have achieved state-of-the-art accuracy using simple magnitude-based pruning (Han et al., 2015; Zhu & Gupta, 2017; See et al., 2016; Narang et al., 2017). When experimenting with structured sparsity, we simply remove filters from the network, as described in Yu et al. (2018). When investigating unstructured sparsity, we use TopK pruning (Zhu & Gupta, 2017).
Quantized Networks: Network quantization represents the weights and/or activations of neural networks with lower-bit integral representations, rather than 32-bit floating point. This reduces model size and memory bandwidth and can accelerate inference. Our work follows the affine quantization scheme presented in Jacob et al. (2018), in which a scale and offset are applied to an integer tensor to approximate the original floating-point tensor. However, our method is general enough to utilize any other quantization scheme.
Neural Network Subspaces: The idea of learning a neural network subspace is introduced in Wortsman et al. (2021) (though another formulation was introduced concurrently in Benton et al. (2021)).
Multiple sets of network weights are treated as the corners of a simplex, and an optimization procedure updates these corners to find a region in weight space in which points inside the simplex correspond to accurate networks. We note that Wortsman et al. (2021) recalibrate BatchNorm statistics before evaluation, which we avoid.

heading: 3 COMPRESSIBLE SUBSPACES

heading: 3.1 COMPRESSIBLE LINES
text: In our method, we train a neural network subspace (Wortsman et al., 2021) that contains a spectrum of networks that each have a different efficiency-accuracy trade-off. In Wortsman et al. (2021), an architecture is chosen, and its collection of weights is denoted by ω. A linear subspace is parameterized by two sets of network weights, ω1 and ω2. At training time, a network is sampled from the line defined by this pair of weights, ω∗(α) = αω1 + (1−α)ω2, where α ∈ [0, 1]. The forward and backward pass of gradient descent are computed using ω∗(α), and gradient updates are backpropagated to ω1 and ω2. A cosine distance regularization term is used to ensure that ω1 and ω2 don’t converge to a single point.
We propose training a linear subspace of networks in a similar manner, but we also bias the subspace to contain high-accuracy solutions at one end and high-efficiency solutions at the other end. To accomplish this, we introduce a parameter γ used to control the compression level and a compression function f(ω, γ) used to compress a model’s weights ω using a compression level γ. The parameter γ induces different compression levels at different positions in the linear subspace, and therefore can be considered a function γ ≡ γ(α). When training, we first sample a network from the subspace, then compress it, obtaining a network with weights f(ω∗(α), γ(α)). We then perform a standard forward and backward pass of gradient descent with it, backpropagating gradients to ω1 and ω2.
In summary, our training method adjusts the standard subspace training method to include the application of a compression function f at a compression level γ(α). Note that we apply the compression after sampling the model ω∗(α) from the subspace, since we want the sampled model to be compressed (rather than ω1 and ω2) for efficient inference. We experiment with formulating our compression function as structured sparsity, unstructured sparsity, and quantization (Section 4). In all cases, the cost of computing f is small compared to the cost of a network forward pass (Section A.1).

heading: 3.2 COMPRESSIBLE POINTS
text: In Section 3.1, we discussed formulating our subspace as a line connected by two endpoints in weight space. This formulation requires additional storage resources to deploy the subspace (Table 1), since an extra copy of network weights is stored. For some cost-efficient computing devices, this storage may be significant. To eliminate this need, we propose training a degenerate subspace with a single point in weight-space (rather than two endpoints). We still use α ∈ [0, 1] to control our compression ratio, but our subspace is parameterized by a single set of weights, ω∗(α) = ω. The compressed weights are now expressed as f(ω∗(α), γ(α)) ≡ f(ω, γ(α)). This corresponds to applying varying levels of compression during each forward pass. The cosine distance regularization term included in the linear subspace is no longer needed because only one set of weights is trained.
This method still produces a subspace of models in the sense that, for each value of α, we obtain a different compressed network f(ω, γ(α)). However, we no longer use different endpoints of a linear subspace to specialize one end of the subspace for accuracy and the other for efficiency. Instead, we condition one set of network weights to tolerate varying levels of compression.

heading: 3.3 CIRCUMVENTING BATCHNORM RECALIBRATION
text: Previous works that train a compressible network require an additional training step to calibrate BatchNorm (Ioffe & Szegedy, 2015) statistics, as in Universal Slimming (Yu & Huang, 2019). Alternatively, they require storing separate BatchNorm statistics for each compression level, as in Slimmable Networks (Yu et al., 2018). This precludes both methods from evaluating at arbitrarily fine-grained compression levels after deployment (Table 1). We seek to eliminate the need for recalibration or storage.
To understand the need for recalibration in previous works, recall that BatchNorm layers store the per-channel mean of the inputs µ and the per-channel variance of the inputs σ2. The recalibration step is needed to correct µ and σ2, which are corrupted when a network is adjusted. Adjustments that corrupt statistics include applying structured sparsity, unstructured sparsity, and quantization.
In Figure 2, we analyze the inaccuracies of BatchNorm statistics for two models trained with specific unstructured sparsity levels and tested with a variety of inference-time unstructured sparsity levels. We calculate the differences between stored BatchNorm means µ and the true mean of a batch µ̂ during the test epoch of a cPreResNet20 (He et al., 2016) model on CIFAR-10 (Krizhevsky, 2009). In Figure 2a and Figure 2b, we show the distribution of mean absolute differences (MADs) |µ− µ̂| across all layers of the models. Models have lower BatchNorm errors when evaluated near sparsity levels that match their training-time target sparsity. Applying mismatched levels of sparsity shifts the distribution of these errors away from 0. In Figure 2c and Figure 2d, we show the average of |µ− µ̂| across the test set for each of the BatchNorm layers. Across layers, the lowest error is achieved when the level of sparsity matches training. In Figure 2e and Figure 2f, we show the average of |µ − µ̂| and the corresponding test set accuracy for various levels of inference-time sparsity. We find that the increased error in BatchNorm is correlated with decreased accuracy. See Figure 7 and Figure 8 in the Appendix for similar analyses with structured sparsity and quantization.
Thus, BatchNorm layers’ stored statistics can become inaccurate during inference-time compression, which can lead to accuracy degradation. To circumvent the need for BatchNorm, we adjust our
networks to use GroupNorm (Wu & He, 2018). This computes an alternative normalization over g groups of channels rather than across a batch. It doesn’t require maintaining a running average of the mean and variance across batches of input, so there are no stored statistics that can be corrupted if the network changes. Formally, let i be an index into a 4-dimensional image tensor, with components i = [iN , iC , iH , iW ] indexing the batch, channel, height, and width, respectively. The set Si of indices used to compute the normalization of batch element i is
Si = {k|kN = iN , bkC ∗ g/cc = biC ∗ g/cc}. (1)
Batch element xi is normalized by subtracting the mean of elements indexed by Si and dividing by the standard deviation of elements indexed by Si. GroupNorm typically uses the hyperparameter g = 32, but it also includes InstanceNorm (Ulyanov et al., 2016) (in which g = c, where c is the number of channels) as a special case. We use g = c in structured sparsity experiments, since the number of channels is determined dynamically and is not always divisible by 32. We use g = 32 in all other experiments 1. After applying this normalization, we also apply a learned per-channel affine transformation, as with BatchNorm.

heading: 3.4 BIASED SAMPLING
text: The endpoints of a learned linear subspace typically demonstrate slightly reduced accuracy compared to the center of the line, as observed in Wortsman et al. (2021). Inspired by this observation, we use biased sampling to choose network endpoints more frequently when training a linear subspace, to ensure that endpoints are accurate.
A similar algorithmic choice is made in Yu & Huang (2019), which proposes using a “sandwich method” for training. This method involves performing forward and backward passes at the maximum and minimum sparsity levels, as well as a few randomly chosen sparsity levels in between. After this round of forward and backward passes, the gradient update is applied.
To implement our sampling of α, we use a stochastic functionαn : Ω→ [0, 1]n, where Ω represents the state of the stochastic function. For each batch, we sample α ∼ αn. We perform n forward and backward passes using compressed networks f(ω∗(αi), γ(αi)) for i ∈ {1, ..., n}, where αi is the ith element ofα. The stochastic function used varies based on the compression function chosen. See Section 4.1 (structured sparsity), Section 4.2 (unstructured sparsity), and Section 4.3 (quantization) for more details. Our full method for training a compressible subspace is shown in Figure 1b.

heading: 4 EXPERIMENTS
text: We present results in the domains of structured sparsity, unstructured sparsity, and quantization. We train using Pytorch (Paszke et al., 2019) on Nvidia Tesla V100 GPUs. On CIFAR-10 (Krizhevsky, 2009), we experiment with the pre-activation version of ResNet20 (He et al., 2016) presented in the PyTorch version of the open-source code provided by Liu et al. (2017). We abbreviate it as “cPreResNet20” to denote that it’s a pre-activation network designed for usage with CIFAR-10. We use a pre-activation version because our Learning Efficient Convolutions (LEC) (Liu et al., 2017) baseline requires it. We follow hyperparameter choices in Wortsman et al. (2021) for our methods and baselines. We warm up to our initial learning rate of 0.1 (or 0.025 in the case of quantization, for stability) over 5 epochs, then decay to 0 over 195 epochs using a cosine schedule. We use a batch size of 128 on a single GPU. We set the weight decay to 5 × 10−4. We run 3 trials for each experiment and report the mean and standard deviation (which is often too small to be visible). Our baseline cPreResNet (with BatchNorm) achieves an accuracy of 91.69% on CIFAR-10.
For ImageNet (Deng et al., 2009) experiments, we use ResNet18 (He et al., 2016) and VGG19 (Simonyan & Zisserman, 2014). We use the version of VGG19 provided by the same repository as our cPreResNet20 implementation. This implementation modifies VGG19 slightly by adding BatchNorm layers and removing the last two fully connected layers. Again, we follow hyperparameter settings in Wortsman et al. (2021) for our methods and baselines. We warm up to our initial learning rate of 0.1 (or 0.025 in the case of quantization, for stability) over 5 epochs, then decay to 0 over 85 epochs using a cosine schedule. We set the weight decay to 5×10−5. We use a batch size of 128 for
1We set g = c for the first few layers of cPreResNet20, because it has fewer than 32 channels.
ResNet18 with a single GPU, and a batch size of 256 for VGG19 with 4 GPUs. Unlike when training with CIFAR-10, we run a single trial for each experiment due to the increased computational expense of training on ImageNet. Our baseline ResNet18 (with BatchNorm) achieves an accuracy of 70.72% on ImageNet, and our VGG19 (with BatchNorm) achieves 62.21%.

heading: 4.1 STRUCTURED SPARSITY
text: For structured sparsity, our method follows the algorithm in Figure 1b. Our compression level calculator γ(α) : [0, 1] → [0.25, 1.0] is the unique affine transformation over its domain and range. Our compression function f(ω, γ) prunes away a fraction 1− γ of the input and output channels in each layer (except the input to the first layer and the output of the last layer). This is the same compression scheme used in Network Slimming (NS) (Yu et al., 2018) and Universal Slimming (US) (Yu & Huang, 2019). Our stochastic sampling function samples α as [0.25, 1.0, U(0.25, 1), U(0.25, 1)], where U(a, b) samples uniformly in the range [a, b]. This choice mirrors the “sandwich rule” used in US. As discussed in Section 3.3, we use a special case of GroupNorm (Wu & He, 2018) known as InstanceNorm (Ulyanov et al., 2016) since the number of channels in the network varies. See Section A.3 for additional details on our linear subspace setup.
When running baselines, we do not allow recalibration of BatchNorm (Ioffe & Szegedy, 2015) statistics (as used in US), or storage of a predefined set of BatchNorm statistics (as used in NS), since it would contradict our goal of adaptive compression to arbitrarily fine-grained sparsity levels at inference time. We instead use a single BatchNorm layer, using only the channels not removed by the structured pruning step. Similarly, we do not allow fine-tuning (normally used in LEC).
The main difference between our point method and US is that we use InstanceNorm instead of BatchNorm. The main difference between US and NS is that US includes the sandwich rule described above. Instead, NS samples at predefined sparsity levels [0.25, 0.50, 0.75, 1] at each iteration.
We present results for structured pruning on ResNets in Figure 3 (see Section A.5 for VGG19 results and runtime characteristics). Our method produces a stronger efficiency-accuracy trade-off than competing methods. The poor performance of LEC (Liu et al., 2017) occurs because this method expects weights to be fine-tuned after sparsity is applied, which we do not allow. The dips and spikes shown for NS occur because this method trains at discrete width factors of [0.25, 0.50, 0.75, 1], which roughly correspond to sparsities of [0.9375, 0.75, 0.4325, 0]. This method shows stronger accuracies at these sparsities, with the exception of 0.9375, at which point, inaccuracies in BatchNorm statistics are more extreme (see Figure 7 in the Appendix). Similarly, US peaks in accuracy at moderate sparsity rates, but decreases in accuracy at high sparsity rates. We believe this strong decrease for higher sparsity rates occurs because the BatchNorm channels used in this case are also used at all other sparsity rates, thus the stored statistics don’t accurately reflect the statistics of the highly-sparse network.
Interestingly, our method with a linear subspace (LCS+L+IN) and our method with a point subspace (LCS+P+IN) achieve very similar performance. We believe that the extra weights in LCS+L+IN are not necessary because individual channels are able to specialize, since some of them are only used for larger subnetworks.

heading: 4.2 UNSTRUCTURED SPARSITY
text: For unstructured sparsity, our method follows the algorithm in Figure 1b. Our compression level calculator is γ(α) = 1 − α. Our compression function f(ω, γ) is TopK sparsity (Zhu & Gupta, 2017), which removes a fraction γ of the weights with the smallest absolute value from each layer (except the input and output layer). Our stochastic sampling function samples one value for α in [0.005, 0.05] (see Section A.4 for additional details regarding sampling and TopK’s warmup phase). We choose this interval because our networks are shown to tolerate high sparsity levels, losing almost no accuracy even at 90% sparsity. See Section A.5 for experiments with broader ranges of sparsity, and for runtime characteristics of sparse models.
To our knowledge, fine-grained adaptive compression has not been explored for unstructured sparsity. Thus, we compare to standard networks (using BatchNorm) trained at specific sparsity levels and evaluated at a variety of sparsity levels. See Figure 4 for results. We find that our baselines always achieve highest accuracy at the sparsity level that matches their training-time TopK value, as expected. Accuracy quickly degrades at other sparsity levels. By contrast, our method is able to achieve high accuracy for a wide variety of sparsity levels. Interestingly, our line method usually achieves higher accuracy at lower sparsity rates in Figure 4, but this pattern does not hold over a wider sparsity range (Section A.5). Further investigation into this phenomena is future work.

heading: 4.3 QUANTIZATION
text: For quantization, our method follows Algorthm 1b. Our compression level calculator is γ(α) = 2+6α. Our compression function f(ω, γ) is affine quantization as described in (Jacob et al., 2018). Our stochastic sampling function samples one value forα uniformly over the set {1/6, 2/6, ..., 6/6}. This corresponds to training with bit widths 3 through 8 (we avoid lower bit widths to circumvent training instabilities we encountered in baselines). We train without quantizing the activations for the first 80% of training, then add activation quantization for the remainder of training. Weights are quantized throughout training.
We present results for our method in Figure 5, comparing to models trained at a fixed bit width and evaluated at a variety of bit widths. See Section A.5 for ResNet18 results, and for memory usage characteristics of models. Generally, baselines achieve high accuracy at the bit width at which they were trained, and reduced accuracy at other bit widths. Our method using a linear subspace (LCS+L+GN) achieves high accuracy at all bit widths, matching or exceeding accuracies of individual networks trained for target bit widths. In the case of VGG19, we found that our accuracy
even exceeded the baselines. We believe part of the increase is due to GroupNorm demonstrating improved results on this network compared to BatchNorm (which does not happen with ResNets, as reported in Wu & He (2018)).

heading: 4.4 FURTHER ANALYSIS
text: In Figure 6, we provide additional experimental evidence that our linear subspace method (LCS+L) trains a subspace specialized for high-accuracy at one end and high-efficiency at the other end. We plot the validation accuracy along our subspace, as well as the validation accuracy along our subspace when compressing with f̃(ω∗(α), γ(α)) ≡ f(ω∗(α), γ(1 − α)). In other words, the weights that were trained for low compression levels are evaluated with high compression levels, and vice versa. We see that this leads to a large drop in accuracy, confirming that our method has conditioned one side of the line to achieve high accuracy at high sparsities, and the other side of the line to achieve high accuracy at low sparsities.

heading: 5 CONCLUSION
text: We present a method for learning a compressible subspace of neural networks. Our subspace can be compressed in real-time without retraining and without specifying the compression levels before deployment. We can produce a variety of models with a fine-grained efficiency-accuracy tradeoff. We demonstrate that our generic algorithm can be applied to structured sparsity, unstructured sparsity, and quantization.

heading: 6 REPRODUCIBILITY STATEMENT
text: We open source our code to reproduce our results (including baselines) at https://withheld. for.review.

heading: A APPENDIX
text: A.1 COMPUTATIONAL COST OF COMPRESSION
We justify our claim that our network compression method is real-time. For a typical convolutional layer without downsampling, the computational cost isO(d1d2H1W1khkw), where d1 is the number of input channels, d2 is the number of output channels, H1 is the input tensor height, H2 is the input
tensor width, kh is the kernel height, and kw is the kernel width. For our line method, we must calculate ω∗(α), which takes O(d1d2khkw) for each convolution, which is negligible compared to the convolution. For our point method, calculating ω∗(α) requires no computation.
We must also calculate the compressed network, f(ω∗(α), γ(α)). Our structured sparsity method (described in Section 4.1) takes O(1) for each convolution, since we only need to mark each layer with the number of convolutional filters to ignore. Our unstructured sparsity method (TopK, described in Section 4.2) takes O(d1d2khkw) for each convolution (to calculate the threshold, then discard parameters below the threshold). Our quantization method (Jacob et al., 2018) takes O(d1d2khkw) for each convolution (to calculate the affine transform parameters and apply them). In all cases, the complexity of computing f(ω∗(α), γ(α)) is lower than the cost of the convolutional forward passes, meaning our method can be considered real-time.
A.2 FURTHER BATCHNORM ANALYSIS
In Figure 2, we analyzed the inaccuracies of BatchNorm statistics for models trained with unstructured sparsity. We show a similar analysis for the case of Universal Slimming (US) (Yu & Huang, 2019) and Network Slimming (NS) (Yu et al., 2018) in Figure 7. We show the case of quantization in Figure 8.
A.3 STRUCTURED SPARSITY DETAILS
When training our method with a line in the structured sparsity setting, we do not use two sets of weights (e.g. ω1 and ω2, Section 3.1) for convolutional filters. Instead, we only use two sets of weights for affine transforms in GroupNorm (Wu & He, 2018) layers. For the convolutional filters, we instead use a single set of weights, similar to our point formulation (and similar to US (Yu & Huang, 2019) and NS (Yu et al., 2018)). By contrast, we use two sets of weights for convolutional filters as well as for affine transforms when experimenting with unstructured sparsity (Section 4.2) and quantization (Section 4.3).
The reason for only using one set of convolutional filters in the structured sparsity setting is that the filters themselves are able to specialize, even without an extra copy of network weights. Some filters are only used in larger networks, so they can learn to identify different signals than the filters used in all subnetworks. Note that this filter specialization argument does not apply to our unstructured or quantized settings.
In preliminary experiments, we found that using a single set of weights for convolutions in our structured sparsity experiments gave a slight improvement over using two sets of weights (roughly 2% for cPreResNet20 (He et al., 2016) on CIFAR-10 (Krizhevsky, 2009)). We hypothesize that this slight difference may be attributed to the subspace training framework’s regularization term (which increases cosine distance between ω1 and ω2), which discourages identical filters from being used in both ω1 and ω2 even if they are optimal.
A.4 UNSTRUCTURED SPARSITY DETAILS
It is typical to include a warmup phase when training models with TopK sparsity (Zhu & Gupta, 2017). In our baselines in Section 4.2, we increase the sparsity level from 0% to its final value over the first 80% of training epochs. For our method, sparsity values fall within a range, so there is no single target sparsity value to warm up to.
For our point method (LCS+P), we simply train for the first 80% of training with the lowest sparsity value in our sparsity range. We finish training by sampling uniformly between the lowest and highest sparsity levels.
For our line method (LCS+L), our choice of sparsity level is tied to our choice of weight-space parameters through α. We implement our warmup by simply adjusting γ(α) to apply less sparsity early in training, warming up to our final sparsity rates over the first 80% of training.
In detail, let αmin and αmax correspond to our minimum and maximum alpha values (for example, in Section 4.2, αmin = 0.005, and αmax = 0.05). As motivated in Section 3.4, we bias sampling of α towards the endpoints of our line. We set α = [αmin ] with 25% probability, α = [αmax ]
cPreResNet20 (CIFAR-10)
with 25% probability, and α = [U(αmin , αmax )] with 50% probability, where U(a, b) samples uniformly in the range [a, b]. To warm up our sparsity rates, we choose
d = max (1− c/t, 0) (2) γ(α) = (1− α)(1− d), (3)
where c is the current iteration number, and t is the total number of iterations in the first 80% of training. At the beginning of training, d = 1, and γ(α) = 0, corresponding to a sparsity level of 0 for all values of α. Once 80% of training is finished, d = 0, and γ(α) = 1− α for the remainder of training. This corresponds to our final sparsity range.
A.5 ADDITIONAL RESULTS
Structured Sparsity: We present results for VGG19 on ImageNet in the structured sparsity setting in Figure 9. Our method produces a better efficiency-accuracy trade-off than baselines. Note that VGG19 with GroupNorm (Wu & He, 2018) had a higher baseline accuracy than VGG19 with BatchNorm (Ioffe & Szegedy, 2015) (as noted in Section 4.3).
We also provide a table demonstrating the memory, flops, and runtime of structured sparsity models in Table 2.
Unstructured Sparsity: We present results for a wider range of sparsity values in our unstructured sparsity experiments in Figure 11. We change our sparsity limits from [0.005, 0.05] (in Section 4.2) to [0.025, 1.0]. All other training details are unchanged. We find that our method is able to produce a higher accuracy over a wider range of sparsities than our baselines.
We also provide a table of unstructured sparsity results in the high sparsity regime (Table 3) and the wider sparsity regime (Table 4), showing memory usage and FLOPS.
Quantization: We present quantization results for ResNet18 in Figure 10. We find that our models approach the accuracy of models trained at a single bit width, and our models generalize better to other bit widths.
We also provide a table of quantization results in Table 5, showing memory usage of the models.

Review summaries:
This paper introduces learnable compressible subspaces, which attempts to learn a set of models that can be switched at inference time to adapt to different resource requirements. This work is motivated by previous work in neural subspaces and slimmable networks. It is evaluated on CIFAR10 and ImageNet and compared against other recent works for adaptable inference models. These results show under certain conditions LCS can maintain higher accuracies at larger sparsities compared to other works.

The paper present a method for learning a compressible subspace of neural networks that contains a fine-grained spectrum of models that range from highly efficient to highly accurate. The proposed method allows choosing the proper point of trade-off between accuracy and efficiency at inference time, according to the available resources. There are also efforts to reduce the runtime tweaking overhead like replacing BatchNorm with GroupNorm.

The paper proposes to learn compressive subspaces which can adaptively compress the network during inference. It constructs either linear subspace or a single endpoint for compression. It replaces BN with GroupNorm to avoid re-calibrating during inference or after adjustment. Their method is evaluated in three different scenarios: structured sparsity, unstructured sparsity, and quantization.