LilNetX: Lightweight Networks with EXtreme Model Compression and Structured Sparsification
Authors: Sharath Girish
Kamal Gupta
Saurabh Singh
Abhinav Shrivastava
Venue: ICLR 2023 poster (2023)
Keywords: Quantization
Model Compression
Sparsity
Pruning

Abstract: We introduce LilNetX, an end-to-end trainable technique for neural networks that enables learning models with specified accuracy-rate-computation trade-off. Prior works approach these problems one at a time and often require post-processing or multistage training which become less practical and do not scale very well for large datasets or architectures. Our method constructs a joint training objective that penalizes the self information of network parameters in a latent representation space to encourage small model size while also introducing priors to increase structured sparsity in the parameter space to reduce computation. When compared with existing state-of-the-art model compression methods, we achieve up to 50% smaller model size and 98% model sparsity on ResNet-20 on the CIFAR-10 dataset as well as 37% smaller model size and 71% structured sparsity on ResNet-50 trained on ImageNet while retaining the same accuracy as those methods. We show that the resulting sparsity can improve the inference time of the models by almost 1.8 times the dense ResNet-50 baseline model. Code is available at https://github.com/Sharath-girish/LilNetX.


Published as a conference paper at ICLR 2023
LILNETX: LIGHTWEIGHT NETWORKS WITH EXTREME MODEL COMPRESSION AND STRUCTURED SPARSIFICATION
Sharath Girish', Kamal Gupta', Saurabh Singh² & Abhinav Shrivastava'
Uniivesitty of Maryland, College Park (sgirish, kampta, abhinaya ccs. undd.udddee edu
GGoogle Research saurabbsingh@@googleeccoo
ABSTRACT
We introduce LilNetX, an end-to-end trainable technique for neural networks that enables learning models with specified accuracy wttr--cooooooocopptttttii
trade- off. Prior works approach these problems one at a time and often require post- processing multistage training. Our method, the other hand, constructs joint training objective that penalizes the self-information of network parame- ters in latent representation space to encourage small model size, while also introducing priors to increase structured sparsity in the parameter space to reduce computation. When compared with existing sate--of-hheart model compression methods, we achieve up to 50% smaller model size and 98% model sparsity on ResNet-20 on the CIFAR- 10 dataset as well as 31% smaller model size and 81% structured sparsity on ResNet-50 trained on ImageNet while retaining the same accuracy as these methods. The resulting sparsity can improve the inference time dense ResNet-50 model. Code is
by factor of almost 1.86x in comparison ceatthhgrrrise girigh/rriiiie MerishhLLiNNe.. available at https: / /tap:s:///ttiie github. com/ Sharath
INTRODUCTION
86 Reduction FLOPs
80
LiINetX (Ours)
Recent research in deep neural networks (DNNs) has shown that large performance gains can be achieved on variety of real world tasks sim- ply by employing larger parameter-heavy and computationally intensive architectures (He et al.,
60
Frankle (2018)
40
Lin al. (2019) Wirdemann a) (2019)
30
Ne
Oktay (2020)
2016; Dosovitskiy et al., 2020). However, as DNNs proliferate in the industry, they often need to be trained repeatedly, transmitted over the net- work to different devices, and need to perform under hardware constraints with minimal loss in accuracy, all at the same time. Hence, finding ways reduce the storage size of the models the devices while simultaneously improving their run-time is of utmost importance. This paper pro- poses general-purpor -urpose neural network training framework jointly optimize the model parame- ters for accuracy, the model size on the disk, and computation, on any given task.
100
Figure Our method jointly optimizes for size on disk and structured sparsity. We compare various approaches using ResNet-50 architec- ture on ImageNet and plot FLOPs (y-axis) vs. size (x-axis) for models with similar accuracy. Prior model compression methods optimize for either quantization tives. Our approach, LilNetX, enables training while optimizing for both compression (model size) well computation (structured spar- sity). Refer Table for details.
or pruning (A) objec-
Over the last few years, research on training smaller and efficient DNNs has followed two seemingly parallel tracks with different goals. One line of work focuses model compression to deal with storage and communication network bot- tlenecks when deploying big models or large number of small models. While they achieve high levels of compression in terms of memory, their focus is not reducing computation. These works either require additional algorithms with some form post hoc training (Yeom et al., 202 1) or quan- tize the network parameters the cost of network performance (Courbariaux al., 2015; Li et al., 2016). The other line work focuses on reducing computation through various model pruning tech- niques (Han et al., 2015; Frankle & Carbin, 2018; Evei et al., 2020). Their focus is decrease the

---

Published as a conference paper at ICLR 2023
number of Floating Point Operations (FLOPs) of the network at inference time, while still achieving some compression due to fewer parameters. Typically, the cost of storing these pruned networks on disk is much higher than dedicated model compression works.
In this work, we bridge the gap between the two lines of work and show that is indeed possible to train neural network while jointly optimizing for both the compression reduce disk space as well as structured sparsity to reduce computation (Fig. 1). We maintain quantized latent representations for the model weights and penalize the entropy of these latents. This idea of reparameterized quan- tization (Oktay et al., 2020) is extremely effective in reducing the effective model size on the disk. However, it requires the full dense model during the inference. To address this shortcoming, we introduce priors to encourage structured and unstructured sparsity in the representations along with key design changes. Our priors reside the latent representation space while encouraging sparsity in the model space. More specifically, we use the notion of slice sparsity, a form of structured spar- sity where K x K slice fully zero for a convolutional kernel of size and channels. Unlike unstructured sparsity which has irregular memory access and offers a little practical speedup, slice- structured sparsity allows for removing entire kernel slices per filter, thus reducing channel size for the convolution of each filter. Additionally, it is more fine tne-grained than fully structured channel/fileer sparsity works (He et al., 2017; Mao et al., 2017) which typically lead to accuracy drops.
Extensive experimentation on three standard datasets shows that our framework achieves high levels of structured sparsity in the trained models. Additionally, the introduced priors show gains even in model compression compared previous state-of-the-art. By varying the weight of the priors, we establish trade-off between model size, sparsity, and accuracy. Along with model compression, we achieve inference speedups by exploiting the sparsity in the trained models. We dub our method LilNetX Lightweight Networks with EXtreme Compression and Structured Sparsification. Our contributions are summarized below.
We introduce LilNetX, an algorithm to jointly perform model compression and structured sparsi- fication for direct computational gains in network inference. Our algorithm can be trained end-to- end using a single joint optimization objective without any post-hoc training or post-processing.
With extensive ablation studies and results, we show the effectiveness of our approach while outperforming existing approaches in both model compression and pruning, in most network and dataset setups, obtaining inference speedups in comparison to the dense baselines.
2 RELATED WORK
Typical model compression methods usually follow some form of quantization, parameter pruning, or both. Both lines of work focus on reducing the size of the model on the disk, and/or increas- ing the speed of the network during the inference time, while maintaining an acceptable level of classification accuracy. In this section, we discuss prominent quantization and pruning techniques.
Model pruning: A plethora of works show that large number of network weights can be pruned without significant loss in performance (LeCun et al., 1990; Reed, 1993; Han et al., 2015). Methods such as the Lottery Ticket Hypothesis (Frankle & Carbin, 2018), adapted by various works (Savarese et al., 2020; Frankle et al., 2019; Malach et al., 2020; Girish al., 2021; Chen et al., 2021; 2020; Desai et al., 2019; Yu et al., 2020) prune models, while reaching the dense network performance, but are iterative and perform unstructured pruning. Other works prune at initialization (Lee et al., 2018; Wang et al., 2020; Liu & Zenke, 2020; Tanaka et al., 2020) and avoid multiple iterations, but show accuracy drops compared to the dense models (Frankle al., 2020). On the other hand, struc- tured sparsity via filter/channel pruning offers practical speedups at the cost of accuracy (Wen et al., 2016; He al., 2017; Huang& & Wang, 2018). Yuan et al. (2020) obtain almost no drops of network performance with structured sparsification but have lower levels of model compression rates due to storage of floating point weights. Other works operate on intermediate levels of structure such as N:M structured sparsity (Zhou et al., 2021) and block sparsity (Narang et al., 2017). Niu et al. (2020) is the closest to ours in terms of pruning structure utilizing slice sparsity, along with an even finer pattern pruning. They show that such structure can be exploited for inference speedups. They, how- ever, require predefining a filter pattern set and heuristics for determining layerwise sparsity. They also optimize for auxiliary variables and have additional training costs due to the dual optimization subproblem (Ren et al., 2019). In contrast, our algorithm uses a single objective to jointly optimize for sparsity and model compression with very little impact on training complexity.
2

---

Published as a conference paper at ICLR 2023
Model Weights W
Quantiee Iatent representations W
Sparse weights
Speess blents
Decoder
Reshape
Commpress
End-to-end
Image x
compute
joint optimization
Logits
Laoc
Standard CNN
Figure 2: Overview of our approach. A standard CNN comprises of a sequence of convolutional and fully-connected layers. We reparameterize the parameters W of each of these layers as W in a quantized latent space. Our decoder is such that sparsity the quantized latents translate to sparsity in CNN parameters. Further, we organize each parameter tensor as a set of slices (depicted as colored bands) corresponding to different channels. Proposed training loss exploits this structure to encourage slice sparsity and jointly optimize for accuracy accuracy-comprest coompeessionncompttteeo
Quantization methods discretize the parameters of a network to small, finite set of values, to store them efficiently using entropy coding methods (Rissanen & Langdon, 1981). Earlier methods uniformly quantize weights to binary or tertiary representations (Courbariaux al,, 2015; Li et al., 2016; Zhou et al., 2018; Zhu et al., 2016; Rastegari et al., 2016; Hubara et al., 2017). Several other works focus on non-uniform Scalar Quantization (SQ) techniques (Tung & Mori, 2018; Zhou et al., 2017; Nagel et al., 2019; Banner et al., 2018; Wu et al., 2016; Zhang et al., 2018; Oktay al., 2020). Vector Quantization( VQ) (Gong et al., 2014; Stock et al., 2019; Wang et al., 2016; Chen et al., 2015; 2016) on the other hand, is a more general technique, where the representers can take any value. VQ can be done by clustering of CNN layers at various compression-accuran trade-offs (Faraone et al., 2018; Son et al., 2018), hashing (Chen et al., 2015; 2016), or residual quantization (Gong et al., 2014). Quantization works focus on reducing bit widths (Zhao et al., 2019; Jain et al., 2020) which has the effect of high model compression (Young et al., 2021). A few works do provide inference speedups (Hubara et al., 2016; Jacob et al., 2018) by utilizing lower bit arithmetic but require custom hardware. In comparison, we focus on jointly optimizing for compression and computational gains, and leave optimization for lower precision arithmetic for future. We show the benefits of our approach in terms of even smaller model size compared to these quantization works along with computational gains while maintaining high levels of accuracy in $5.
Model quantization:
3 APPROACH
We consider the task of classification using a convolutional neural network (CNN), although our approach be trivially extended to other tasks such object detection generative modeling. Given a dataset of N images and their corresponding labels {x, y J.N our goal is to train a CNN with parameters θ that are jointly optimized the model by minimizing the number of bits required to store the model on disk, and 3) minimize the computational cost of inference in the model by maximizing model sparsity. To keep our method end-to-end trainable, we formulate it as minimization of a joint objective that allows for an accuracy- caapression-computtae trade-off as
1) maximize classification accuracy, 2) compress
((O)) = Luce + + Leompress ()) ))= + Leompune (
(1)
For the task of classification, the accuracy term ( is the usual cross-entropy loss. Given an image, it maximizes the probability assigned to the target label. The compression term Leompress ()) encourages the model to have a small disk size. We reparameterize the model weights using quantized latent representation space which are then stored on disk. The compression term, while encouraging smaller model size, doesn't 't lead to computational gains as the decoded model param- eters are still dense. Our computation term Leompute (+) addresses this issue by introducing a struc- tured iparsity-inducing loss. Our framework allows the structured sparsity in latent weight space to directly translate to the structured sparsity in the decoded model weights. In our experiments, demonstrate significant speedups in the inference time of our model with sparse weights using off-the off-the-shelf libraries. Refer to Fig. 2 for a high-level overview of our approach. In the following sections, describe the compression and computation terms in more detail.
3

---

Published as a conference paper at ICLR 2023
3.1 COMPRESSION TERM
We formulate our compression term by building upon prior works that incorporate entropy penalty on parameter representation during training (Ballé et al., 2018; Oktay al., 2020). The model's weights and biases, θ are reparameterized as quantized latent representations which are compressed while the network parameters are implicitly defined as a transform of the latent representations.
We represent the set of model parameters ④ = [W].'' W2, be,..., WN, bd} where N is the
total number of layers in the network, and W ba represent the weight and bias parameters of the hthh layer. Each of these parameters can take continuous values during inference. However, these parameters are stored using quantized latent representations belonging to a corresponding set 4 W bi, W D2
WN, B²}. For each convolutional layer, W is a weight tensor of dimensions Cn x Cou x K K, where Cin the number of input channels, and Cou is the number of output channels, and K' denotes the filter width and height. The corresponding quantized latent representation W is represented by two-dimensional (2D) matrix of size CinCuu x K2 For each dense layer, W is a tensor of dimension Ci Cont and its corresponding latent representation W matrix of dimension Cin Cout x 1. All the biases can be represented in the same way as dense layers. Each latent representation is quantized 2D matrix W gCaCuuxi where / = 1 for dense weights (and biases) while 1 K2 for convolutional weights. Each row/slice W; from W represents a sample drawn from I-dimensional discrete probability distribution. In order decode parameters from latent space & to model space O Iearnable affine transforms y are introduced.
W;- = V scale (
stift
W = rrehhaee ([W,
W ...), i (I...CCinCo.) (2)
Where W; represents the 7thh row/slice of W and W; the corresponding decoded slice. & scale R and L shin E R are the affine transformation parameters. Different kinds of layers use differ- ent pairs of transform parameters ( saal: Ve staa). i.e,, different convolutional layers have their own transform, dense layers have their own, and so on.
As + consists of discrete parameters which are difficult to optimize, continuous surrogates W maintained for each quantized parameter W. W is thus simply obtained by rounding the elements of W to the nearest integer. A straight-through estimator (Bengio et al., 2013) is used to back- propagate the gradients from the classification loss to W. Bit-rate minimization is achieved by enforcing an entropy penalty the surrogates W. For a given surrogate W E Rdxx with samples of dimension I, we add uniform noise n ~ L (- 2: 1 and fit probability models {qj. E [1...[]} as proposed in Ballé al. (2018). The entropy of model weights can now be minimized directly by minimizing the negative log-likelihood which serves as an approximation to the self-information (Eq. (3)). The compression term is then the sum of all the self-information terms
Cr
II II ( + +ma)), I(W) - log, a( (W), Cemmpress i= ==1
( =
Σ (())
(3)
SE
Where Ny is a hyper-parameter specifying relative weight of the compression loss. After training, only the quantized representations W are stored by arithmetic coding using the probability tables from 9j- We discard the continuous surrogates post training. During inference, we load the
quantized latents W and decode them using the leartt decoder to obtain the continuous model weights W which is used in the model's 's forward pass.
3.2 SPARSITY PRIORS
The compression term described in the previous section encourages smaller representation of the model in terms of the number of bits required to store the model on disk. However, once the model is decoded for inference, it can be fully dense with no reduction in terms of computation in single forward pass of the model. To address this, we introduce a few key changes and then formulate computation term structured sparsity priors that lead to reduced computation. We formulate all the priors in the latent representation space to decouple from the affine transform parameters I scale, V sin and to be consistent with the Leompress (+) term that is also applied in the same space.
4

---

Published as a conference paper at ICLR 2023
We observe from Eq. (2) that even if all the elements of the latent W, are zero, the resulting trans- formed slice W may still be non-zero. In order to enforce structural sparsity in the parameters in the model space W, require each K x K slice W, be O, the vector. However, this is only zero vector itself or lies in the null space v scale- We notice that the latter does not occur in most practical situations especially when the vector W, is discrete. There- fore, we remove the shift parameter L shift and make the affine transform a purely linear transform.
possible if ( sbin + W;
(4) Note that the j"h element in W; is zero only if the j row of v seale is orthogonal to W; or W, = 0. The former tend to be rare nonexistent in practice due to Lcaat being real-valuess and W; being discrete. Thus, any single non-zero element in W, causes the transformed model vector W, to be nonzero and does not yield any sparsity in the model space. Loosely, we get W; = 0 + W; = O.
W
L scale Ws, W. Z', L scale R
Since a sparse model contains a majority of zeros, its weight distribution should peak at zero. However, the loss in Eq. (3) does not neces- sarily enforce zero latents, allowing for any non-zero constant value. A zero-mean Gaussian is one such distribution that enforces zero-centered latents. It also enforces other useful properties such as unimodality and symmetry, typically observed in trained uncompressed model weight distributions (Appendix B). Similar to Eq. (3), Gaussian prior can viewed as a compression penalty but with a Gaussian weight distribution. This corresponds to the l2 norm penalty on W.
Unstructured sparsity of latents with Gaussian prior:
Cu
qu (W) = II II
#W and Lu(W) - logg QU W) 2
(5)
j=1
Cau 1
ggaussian ( =v >
Σ
Σ WeB
Weyle
(6)
WeB
j=1
Where Ày is the tradeoff parameter controlling how closely the weights follow Gaussian distribu- tion compared to the the fully factorized distribution from the probability models in Eq. (3). A lapla- cian prior, resulting in an l penalty, can also be used but the difference between the two distributions is negligible due the effects of the quantization on W. We experimented with both priors and ob- served similar performance as expected (refer Appendix G). Through experiments in Sec. 4, we show that this prior not only enforces model sparsity by design but also improves model compression.
Structured sparsity of latents with group lasso: The Gaussian prior encourages individual weight values in the latents to be close to zero. However, from Eq. (4), we see that entire slices W; should be zero vectors to improve sparsity in the model space. Note that each slice W. can be represented group belonging W, the set of groups (or slices). Thus, to enforce individual slices go zero as a whole, we propose to use a group sparsity regularization (Yuan Lin, 2006) on each W; slice follows.
CaxCa
Cgruup (
As
Σ
Σ
Wc#
where p accounts for varying group sizes and Às is the tradeoff parameter for the structured sparsity.
3.3 JOINT OPTIMIZATION OBJECTIVE
The overall loss function is the combination of cross-entropy loss (for classification), self- information, and regularization for structured and unstructured sparsity of the latents follows:
Computaiion
Accuracy
Disk Size
Car
CaxCom
Σ logply/x; W) + A Σ (( Au Σ (x.g)>DD
W
Σ We+
Σ
vpl Willa
Web
Cross Entropy
Pammeter Emtrgyy
Spursity
Spunaiy
5

---

Published as a conference paper at ICLR 2023
The above objective is fully differentiable and can be minimized end-to-end by any gradient-based optimizer. Note that while there are three tradeoff parameters, each of them is relatively indepen- dent and intuitively controls different aspects of the network. X controls for model size, Ay for enforeing a zero-mean prior necessary for sparsity, and Xş for slice sparsity and hence computation during network inference. l2 weight decay model weights is typically used for improving generalization. However, our motivation for a Gaussian prior is for the objective de- tailed in Sec. 3.2. Furthermore, our group lasso prior and the Gaussian prior are applied in the latent representation space rather than the model space as in Wen et al. (2016). Due to the presence of quantization and decoding, the effect of these priors on training significantly different than when applying these priors directly to the model weights. We analyze the effect of these priors in Sec. 4.
regularization
3.4 IMPLEMENTATION DETAILS
Datasets. We consider three datasets in our experiments. CIFAR-10 and CIFAR-100 datasets (Krizhevsky et al., 2009) consist of 50000 training and 10000 test color images each of size 32 x 32. For large scale experiments. we use ILSVRC2012 (ImageNet) dataset (Deng et al., 2009). It has 1.2 million images for training, 50000 images for the test and 1000 classes.
Network Architectures. For CIFAR- 10/100 datasets, we show results using- VGG-16 16 (Simonyan & Zisserman, 2014) and ResNet-20 with a width multiplier of 4 (ResNet-20-4) (He et al., 2016). VGG-16 a commonly used architecture consisting of 13 convolutional layers kernel size 3 x and 3 dense or fully-connected layers. Dense layers are resized to adapt to CIFAR's 32 32 image size, as done in baseline approaches. ResNet-20-4 consists of ResNet groups, each with 3 residual blocks. All convolutional layers of size 3 3, along with final dense layer. For the ImageNet experiments, we use ResNet- 1850 networks with one 7 x 7 convolutional layer, multiple 3 and 1 x 1 convolutional layers, and the final dense layer. We also experiments with MobileNet-V2 (Sandler et al., 2018), containing depthwise separable convolutions and inverted Bottleneck blocks.
We use the Adam optimizer (Kingma& & Ba, 2014) for updating all parameters of our models. The entropy model parameters are optimized with learning rate of 10-4 for all our experiments. The remaining parameters are optimized with a learning rate of 0.01 for CIFAR-10 experiments and a learning rate of 0.02 for ResNet- 8850 on ImageNet with a cyclic schedule. Our model compression results are reported using the torchac library (Mentzer et al., 2019) which does arithmetic coding of the weights given probability tables for the quantized values which obtain from the probability models. We do not compress the biases and batch normalization (BN) parameters and include the additional sizes from these parameters well the parameters L scale when reporting model size. Parameter groups for different networks based on types of convolutional/dense layers are provided in the appendix (Appendix E).
4 ANALYSIS
Recall from Eq. (8), we proposed two sparsity coefficients: Ay for
Tap-1 Accuracy (%)
CIFAR-10
CIFAR-10
95.75
93.50
93.50
unstructured sparsity, and Äs for structured sparsity. Note that the
1u
93.25
0.0
93.00
0.0
sparsity terms not only improve the model's inference speed (by
36:07
36:06
36-06
92.50
92.50
150
200
75
80
increasing slice sparsity and re- ducing the number of FLOPs) but also reduce the entropy of la- tent weights W as most of the weights become zero after quan- tization. By varying the two sparsity coefficients, one for each sparsity term, we obtain different points the Pareto curves for accuracy vs. model size trade-off and accuracy vs. slice sparsity trade-off. In this section, we study each of these trade-offs extensively. We use slice sparsity as a proxy for computational complexity
Model Size (KB)
Slice Sparsity (25
(a)
(b)
Figure 3: Effeec of unstructured sparsity coefficient A We show plots for 3 values of Xu varying Àş to obtain the Pareto curves CIFAR-10 10. Higher Àu improves accuracy- tradeoff (left) by a good margin and also the accuracy-slice spar- sity (right) by small amount. Shaded areas represent the con- fidence intervals of the regression fit. Best viewed in color.
size

---

Published as a conference paper at ICLR 2023
or inference speed in this section for brevity, and revisit the computational complexity in terms of actual wall-clock inference time in Sec. 5.2. We use a constant compression coefficient λ = 10 model size (bit-rate), and the slice sparsity (%) by keeping one of λ fixed and varying the other. We analyze model sizes for the compressed parameters in this section as the remaining parameter sizes are constant. All results in this section are obtained by averaging over 3 runs with varying random seeds.
We show Pareto curves of model performance (accuracy)
4.1 EFFECT OF UNSTRUCTURED SPARSITY REGULARIZATION
Fig. shows model performance for three different values Ay, while varying As for obtaining the Pareto trade-off curve for each case. Fig. 3(a) and 3(b) show the impact the top-1 accuracy as a function of model size and slice sparsity respectively. We see that increasing Ay (blue circles to orange crosses green squares) improves the accuracy vs. model size tradeoff curve by fair margin while also slightly improving the accuracy vs. slice sparsity curve. This is to be expected as higher Å leads higher number of zeros in the latent representations leading to lesser entropy and consequently lower model size. We also see a marginal gain in slice sparsity even though the Gaussian prior doesn't 't directly optimize for it as higher number of zeros eventually leads more slices entirely going to zero. Thus, we see that the Gaussian prior helps with improving model size while also marginally improving slice sparsity.
4.2 EFFECT OF STRUCTURED SPARSITY REGULARIZATION
Fig. 4 shows model performance
for three values of X. while
Top-1 Accuracy (%)
QIAR-10
CHEAR-10
varying AØ to obtain the Pareto curves. Again, Fig. 4(a) and 4(b)
93.5
93.5
show the impact on the top-1 ac- curacy as a function of model size and slice sparsity respec- tively. The highest value of Äs (green squares) shows a marked improvement of the accuracy- sparsity curve in Fig. 4(b). This shows that the group lasso regu- larization is effective in improv- ing the slice sparsity of the model necessary for computational ben- efits. Additionally, we see a small improvement in the accuracy- model size curve as higher Äs promotes a higher number of zero slices which indirectly leads to lower entropy of the overall latent representations.
93.0
As
93.0
0.0
0.0
92.5
92.5
3e07
36:06
06
92.0
92.0
200
90
Model Size 0CB)
Slice Sparsity (8)
(a)
(b)
Figure 4: Effeet of structured sparsity coefficient As- We
show plots for 3 values of Às varying Au to obtain the Pareto curves. Fig. (a) and (b) correspond to Accuracy-Mode Size trade-off and Accuracy-Slice Sparsity trade-off respectively on CIFAR-10. Highest Às improves the accuracy-model sparsity tradeoff curve showing the importance of group lasso regular- ization for improving slice sparsity. Shaded areas represent the confidence intervals of the regression fit. Best viewed color.
4.3 STRUCTURED UNSTRUCTURED SPARSITY REGULARIZATION
While both structured and unstructured sparsity regularization help improve slice sparsity, the latter optimizes for it indirectly. Fig. 5 shows the effect of λu Às on unstructured and slice sparsity of the latent representations. As expected, we observe that increasing Àu (corresponding to larger points), for a fixed As. increases the sparsity of the latent representations while indirectly increasing slice sparsity as well. However, as we increase Äs we notice that the plots shift upwards implying higher structural sparsity for any fixed value of unstructured sparsity. Therefore, we see that our structured sparsity prior is effective in forcing non zero weights to lie in fewer weight slices thus leading to higher structured sparsity, ultimately leading to speedups in model inference. Note that while the structured sparsity constraint directly optimizes for the slice sparsity, the unstructured sparsity promotes sparsity within group/slice, akin to Simon et al. (2013). Unstructured sparsity has strong effect on model size due to large number of zeros and lower entropy as we show in Sec.4. while structured sparsity constraint directly affects slice sparsity as shown above and in S.c.4.2.
7

---

Published as a conference paper at ICLR 2023
Table 1: Comparison of our approach against other model compression techniques. We show two
cases of our method: Best and Extreme. Best corresponds to our best model in terms of error rate and compression factor while Extreme is matching the range of error of baselines if exists. We achieve higher compression along with the added computational benefits of high slice sparsity. "
implies that the work performs pruning but does not report numbers in the paper.
Algosithm
(MB)
Broor (Top-1 56)
Sparsity (16)
ResNet- 18 (ImogoNet)
Algurithm
(KB)
(Top-1 %%)
Sparsity (%)
Uncompressed
(1
30.0
Dubey et al, (2018)
3.11 (15x)
32.0
VGG-16 (CIFAR-10)
Young al. (2021)
(17x)
31.8
Uncompressed
60 MB (1x)
6.6
Okay et al. (2020)
1.97 (24x)
30.0
0.0
Louizos et al. (3017)
525 (116x)
9.2
94.5
LaINeXX (Best)
1.58 (30x)
30.1
33.3
Wiedemann (2019)
960 (62x)
92.4
LaNNeXX (Fsneme)
(54x)
32.3
65.2
Havasi et al. (2015)
168 (152x)
10.0
0.0
ResNet 50 (ImageNet)
Okia (2020)
101 (590x)
LiINetX (Best)
129 (465%)
7.4
97.4
Uncompressed
102.00 (1x)
23.7
LilNetX (Extreme)
76 (800k)
10.0
99.2
Lan al. (2019)
39.91 (3x4
27.7
39.2
Frankle al. 2019))
20.40
24.0
49.1
ResNet-20 40-4 (CIFAR-1O)
Dubey et al. (2018)
6.16 (16x)
26.0
Cncomppeessdd
17.2 (1x)
5.5
0.0
Wiedemann al. (201))
6.06 (17x)
25.9
25.4
Okby at (2020)
128 (134k)
8.8
0.0
Okny al. (2020)
(19x)
26.0
LilNetX (Besi)
139 (123x)
6.0
88.5
Young et al. (2021)
4.39 (21x)
25.8
0.0
LilNetX (Extreme)
(282x)
97.9
LilNerX (Rest)
25.4
66.7
LiNNetX (Extreme)
2.96 (34x)
25.8
81.7
ResNet-20-4 (CIFAR-100)
MobileNet-V2 (ImageNet)
Uncomppessed
17.2 (1x)
LilNetX (Best)
125 (137x)
27.0
86.6
Uncompresced
(1.00((x))
32.7
0.0
LilNetX (Extreme)
(226k)
30.9
To et al. (2020)
10.30
32.7
25.0
Wang (2019)
(15x)
33.5
LaNNerX (Rost)
0.67 (21x)
32.8
56.8
LiNNetX (Extreme)
0.56 (25x)
33.3
64.7
EXPERIMENTS
CIFAR-10
All our models are trained from scratch. CIFAR-10/100
experiments are trained for 200 epochs. We use the FFCV library (Leclerc et al., 2022) for faster ImageNet training, with batch size of 512 for Res5et-18/50 split across 4 GPUs. We train R 18/5 for 35 epochs to keep the range of the uncompressed network accuracies simi- lar to other works for fair comparison. We show strong performance in terms of model compression and sparsity outperforming existing model compression works while converging faster with relatively fewer epochs.
Sice Sparsity (%6)
0.95
0.0
0.90
7e:07
3e-06
0.85
0.80
0.96
0.97
0.99
Sparsity (Latents) (96)
Figure 5: Structured vs. unstructured sparsity regularization. Higher Äs im- proves slice sparsity (y-axis) for a given level of unstructured sparsity (x-axis).
5.1 COMPARISON WITH COMPRESSION METHODS
As discussed in Sec. 2, existing approaches for model compression follow either quantization, pruming, or both. We compare with the state of the art methods in each of these two categories. Among model quantization methods, use Oktay et al. (2020) and Young et al. (2021) for comparison, with the latter offering speedups via mix-precision inference albeit on specialized hardware. Our results are summarized in Table 1. Unless otherwise noted, we use the numbers reported by the original papers. Since we do not have access to many prior art models, we compare using slice sparsity. For the CIFAR-10 dataset, we achieve the best results in compression while also achieving lower Top-1 error rate for both VGG-16 and ResNet- 20-4. For VGG-16 16 we obtain the best performance in the error range of t7% at 129K B which is a 465x compression compared to the baseline model. At the ~ 100 error range, we outperform Oktay et al. (2020) in terms of model compression and also with 99.2% slice sparsity. For ResNet-20-4, compared to Oktay et al. (2020), we achieve almost twice the compression rate at similar error rate of 8.5%% simultaneously achieving extremely high levels of slice sparsity (97.9%). Similar results hold for the case of CIFAR-100 where we achieve 137x compression in model size with 86.7% slice sparsity and little to no drop in accuracy compared to the uncompressed model.
For ResNet-18 trained on ImageNet, model with almost equal error rate outperforming Oktay et al. (2020). The compressed network achieves sparsity of 33.3%. For an extreme compression case, we achieve higher levels of com-
achieve 30x compression as compared to the uncompressed
8

---

Published as a conference paper at ICLR 2023
pression (54x) and sparsity (~65%) at the cost of ~2% accuracy compared uncompressed model. For ResNet-50, our best model achieves a compression rate of 26x, along with 66.7% slice sparsity. An extreme case of our model achieves a higher compression rate of 34x with sparsity of 81.7% compared to the next best work of Young et al. (2021) with rate of 21 x at a similar error rate. We provide additional baseline comparisons for ResNet-50 in the supplementary material.
Finally, for MobileNet- V2 which is already lightweight and optimized for computational efficiency, we still achieve 21 x compression compared to the dense model along with 56.8% slice sparsity with almost no drop in accuracy. This is especially beneficial for MobileNets which consist of Depth- wise Separable Convolutions as removing entire 2D slices of the convolutional weight allows for directly removing the input activation map's 's corresponding channels leading to lower FLOPs. We outperform other baselines at a similar accuracy both in terms of model compression and sparsity.
We conclude that our framework outperforms pression by a fair margin while achieving network weights sparsification for computational gains.
(SOTA) approaches in model com-
5.2 UTILIZING SLICE SPARSITY FOR INFERENCE SPEEDUPS
While we show compres- sion gains with respect to the SOTA in Sec. 5.1, here we highlight the computa- tional gains we get through
ResNet-20 CIFAR-10
ResNet--50 50 on ImageNet
Spocdue
3.0
LiNeetX (Extreme)
Tap-1
92.5
90.0
TTp Speedup
2:50
2.25
2,00
Top-1 Acc 74.2%
LiNNex (Extreme)
Top Accuracy
87.5
2.0
1.75
slice sparsity. We measure the speedups of the trained sparse models by obtain- ing the ratio of the through- put (images/second) of the sparse compressed models to that of the dense uncom- pressed models. Speedups single
1.5
85.01
1.50
82.5
1.25
80.0
40
Siice Sparsity (%)
Slice Sparsity
Figure 6: Speedups vs. Slice Sparsity (%). Speedup is the ratio of CPU throughput for the sparse models to the dense models. For both cases of ResNet-20 (right), we obtain practical inference speedups utilizing the sparsity of the compressed models.
CIFAR-10 (left) and ResNet-50 on ImageNet
measured core of an AMD EPYC
7302 16-Core Processor with a batch size of 16. We show inference results for two cases: ResNet-20 on CIFAR- 10 and ResNet-50 on ImageNet. As high slice sparsity provides the added benefits of full structured sparsity (when all slices in filter/channel are zero), we show speedups for ResNet- 20 by utilizing fully structured sparsity, removing entire filters or input channels of weight tensors which are all zeros. Niu et al. (2020) show that slice sparsity, along with pattern sparsity, provide in- ference speedups mobile devices via compiler-assister optimizations. However, due to the lack of an open- code base, we utilize the DeepSparse engine (Kurtz et al., 2020) which also exploits this sparsity for CPU inference speedups. Results shown in Fig. 6. For CIFAR-10 (left), even when only exploiting fully structured sparsity, we achieve nearly 2x levels of speedups for 95% slice sparsity. Speedups scale almost exponentially for sparsity >55% For ImageNet (right), obtain 1.86x speedup compared to the dense uncompressed model at 81.7% slice sparsity and even faster inference times for sparsity>855% Therefore, our framework offers practical inference speedups via slice sparsity with no hardware modifications, along with high levels of model compression.
CONCLUSION
We propose a novel framework for training a deep neural network, while simultaneously optimizing the model size to reduce storage cost, and structured sparsity, to reduce computation cost. To the best of our knowledge, this is the first work model compression that add priors for structured pruning of weights in quantized latent representation space.
Experiments on three datasets and three network architectures show that our approach achieves state- th-the-art performance in terms of simultancous compression and reduction in computation which directly translate to inference speedups. We also perform extensive ablation studies to verify that the proposed sparsity priors allow us to easily control the which is important consideration for the practical deployment of models.
peapprssioonccopptttiiini trade-off,

---

Published as a conference paper at ICLR 2023
REFERENCES
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational
image compression with scale hyperprior. arXiv preprint urXiv:/802.01466, 2018.
Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. Post-training 4-bit quantization of convolution networks for rapid-deployment. arXiv preprint arXiv:1810.05723, 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.34422 2013. 2013.
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser-Nam Lim, and Abhinav Shrivastava. Nerv: Neural representations for videos. arXiv preprint arXiv:21 1ttl::2/10.1333
10. 13903, 2021.
Chen, Frankle, S Chang, S Liu, Y Zhang. Wang, and M Carbin. The lottery ticket hypothesis for pre-trained bert networks. In NeurlPS, 2020.
Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In International conference on machine learning, pp- 2285-2294. PMLR, 2015.
Wenlin Chen, James Wilson, Stephen Tyree Kilian Q Weinberger, and Yixin Chen. Compress- ing convolutional neural networks in the frequency domain. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1475-1484, 2016.
Matthieu Courbariaux, Yoshua Bengio, and Jean- Pinrree David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems, pp. 3123-3131, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icce, 2009.
Shrey Desai, Hongyuan Zhan, and Ahmed Aly. Evaluating lottery tickets under distributional shifts. arXiv preprint arXiv:1910.12700 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint 0rXhv:2010.11929 2020.
Abhimanyu Dubey, Moitreya Chatterjee, and Narendra Ahuja. Coreset-based neural network com- pression. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 454-470, 2018.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference Machine Learning, pp. 2943-2952. PMLR, 2020.
Julian Faraone, Nicholas Fraser, Michaela Blott, and Philip HW Leong. Syq: Learning symmet- ric quantization for efficient deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4300-4309, 2018
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: /80300330 2018.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv:1903.0161/ 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Pruning neural networks at initialization: Why are we missing the mark? arXiv preprint arXiv:2009.0857 2020.
10

---

Published as a conference paper at ICLR 2023
Sharath Girish, Shishira R Maiya, Kamal Gupta, Hao Chen, Larry Davis, and Abhinav Shrivastava. The lottery ticket hypothesis for object recognition. In Proceddings of the IEEE/CVE Conference Computer Viston and Pattern Recognition, pp. 762-771, 2021.
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional net- works using vector quantization. arXiv preprint arXiv: 2014.
Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient neural networks. arXiv preprint arXiv: 1506.02626, 2015.
Marton Havasi, Robert Peharz, and José Miguel Hermández-Lobato Minimal random code learning: Getting bits back from compressed model parameters. arXiv preprint 1810.04440000 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE international conference on computer vision, pp. 1389-1397, 2017.
Zchao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pp. 304-320, 2018.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv,, and Yoshua Bengio. Binarized neural networks. Advances in neural information processing systems, 29, 2016.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Jour- nal of Machine Learning Research, 18(1):6869-689) 2017.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient tneger-arithmetic-onn inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704-271 2713. 2018.
Sambhav Jain, Albert Gural, Michael Wu, and Chris Dick. Trained quantization thresholds for accurate and efficient fixed-point inference of deep neural networks. Proceedings of Machine Learning and Systems, 2:1 12-128. 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. 2009.
Learning multiple layers of features from tiny images.
Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Nir Shavit, and Dan Alistarh. Inducing and exploiting acti- vation sparsity for fast inference on deep neural networks. Hal Daumé and Aarti Singh (eds.). Proceedings of the 37th International Conference Machine Learning. volume 119 of Proceedings of Machine Learning Research, pp. 5533-5543 PMLR, 13-18 Jul 2020. URL https:/ httpp:://pocceste /ttp://po/eeee////ooo ngs ml press v1 1 9 9/kkrtzz20a .tml.
Guillaume Leclere, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry. ffev. https::///.. // thub. com/ / walfffe//fffeee 2022.
Yann LeCun, John S Denker, and Sara Solla. Optimal brain damage. In Advances neural information processing systems, pp. 598-6598--5 1990.

---

Published as a conference paper at ICLR 2023
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02344 2018.
Fengfu Li, Bo Zhang, and Bin Liu. Ternary weight networks. arXiv preprint arXiv:1605.04771 2016.
Shaohui Lin, Rongrong Ji, Yuchao Li, Cheng Deng, and Xuelong Li. Toward compact convnets via structure-sparsity regularized filter pruning. IEEE transactions on neural networks and learning systems, 31(2):574-555 588, 2019.
Tianlin Liu and Friedemann Zenke. Finding trainable sparse networks through neural tangent trans- fer. In International Conference on Machine Learning, pp. 6336-6347 PMLR, 2020.
Christos Louizos, Karen Ullrich, and Max Welling. Bayesian compression for deep learning, arXiv preprint arXiv:1705. 700.0660550866, 2017.
Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In International Conference on Machine Learning, pp. 6682- 6691. PMLR, 2020.
Huizi Mao, Song Han, Jeff Pool, Wenshuo Li, Xingyu Liu, Yu Wang, and William Dally. Exploring the regularity of sparse structure in convolutional neural networks. arXiv preprint arXiv:1705.0892. 2017.
Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, and Luc Van Gool. Practical full resolution learned lossless image compression. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1325-1334, 2019.
S. Narang al. Block-sparse recurrent neural networks. arXiv: 1711:1711.002785 2017.
Wei Niu, Xiaolong Ma, Sheng Lin, Shihao Wang. Xuehai Qian, Xue Lin, Yanzhi Wang, and Bin Ren. Patdnn: Achieving real-time dnn execution on mobile devices with patternbbaed weight pruning. Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 907-922, 2020.
D. Oktay et al. Scalable model compression by entropy penalized reparameterization. In ICLR, 2020.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 542. Springer, 2016.
Russell Reed. Pruning algorithms-a survey. IEEE transactions on Neural Networks, 4(5):740-747, 1993.
Ren, Tianyun Zhang, Shaokai Ye, Jiayu Li, Wenyao Xu, Xuehai Qian, Xue Lin, and Yanzhi Wang. Admm-nn: An aggoritmm-hardwane co-design framework of dans using alternating direc- tion methods of multipliers. Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 925-938, 2019.
Jorma Rissanen and Glen Langdon. Universal modeling and coding. IEFE Transactions on Infor- mation Theory, 27(1):12-25 1981.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification. Advances in Neural Information Processing Systems, 33:1 1880-11390 2020.
12

---

Published as a conference paper at ICLR 2023
Noah Simon, Jerome Friedman, Trevor Hastie, and Robert Tibshirani. A aparse-group lasso. Journal of computational and graphical statistics, 22(2):231-244 2013.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Sanghyun Son, Seungjun Nah, and Kyoung Mu Lee. Clustering convolutional kernels to compress deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 216-232, 2018.
Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, and Hervé Jégou. And the bit goes down: Revisiting the quantization of neural networks. arXiv preprint #rXiv:190.00566 2019.
Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. arXiv preprint arXiv:2006.0544 2020.
Cheng-Hao Tu, Jia-Hong Lee, Yi-Ming Chan, and Chu-Song Chen. Pruning depthwise separa- ble convolutions for mobilenet compression. In 2020 International Joint Conference on Neural Networks (IJCNN). pp. 1-8. IEEE, 2020.
Frederick Tung and Greg Mori. Clip-q: Deep network compression learning by in-parallel pruning- quantization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7873-7882, 2018
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376, 2020.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quan- tization with mixed precision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8612-8620, 2019.
Yunhe Wang, Chang Xu, Shan You, Dacheng Tao, and Chao Xu. Canpack: Packing convolutional neural networks in the frequency domain. In NIPS, volume 1, pp. 3, 2016.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances in neural information processing systems, 29:2074-2082, 2016.
Simon Wiedemann, Heiner Kirchhoffer, Stefan Matlage, Paul Haase, Arturo Marban, Talmaj Marine, David Neumann, Ahmed Osman, Detlev Marpe, Heiko Schwarz, et al. Deepcabac: Context-adaptive binary arithmetic coding for deep neural network compression. arXiv preprint arXiv:1905.08318, 2019.
Jiaxiang Wu, Cong Leng, Yuhang Wang. Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4828, 2016.
Seul-Ki Yeom, Philipp Seegerer, Sehastian Lapuschkin, Alexander Binder, Simon Wiedemann, Klaus-Robert Müller, and Wojeiech Samek. Pruning by explaining: A novel criterion for deep neural network pruning. Pattern Recognition, 115 115:107899, 2021.
Sean Young, Zhe Wang, David Taubman, and Bernd Girod. Transform quantization for cnn com- pression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Yu, Edunov S, Y Tian Y, and Morcos. Playing the lottery with rewards and multiple languages: lottery tickets in RI. and NLP. In ICLR, 2020. URL https: http::::///ppeeeiiiiie net /formm id S1XXXRVEWH.
Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68 18:497777 2006.
Xin Yuan, Pedro Savarese, and Michael Maire. Growing efficient deep networks by structured continuous sparsification. arXiv preprint arXiv:2007. /5353, 2020.
13

---

Published as a conference paper at ICLR 2023
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. Proceedings of the European conference computer vision (ECCV), pp. 365-382. 2018.
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting. In International conference on machine learning. pp. 7543-7552. PMLR, 2019.
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantiza- tion: Towards lossless cnns with low-precision weights. arXiv preprint 1702 2017.
Aojun Zhou, Anbang Yao, Kuan Wang, and Yurong Chen. Explicit loss-error-awanl quantization for low-bit deep neural networks. In Proceedings of the IEEE conference on computer vision and patter recognition, pp. 9426-9444 2018.
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hong- sheng Li. Learning n: m fine-grained structured sparse neural networks from scratch. arXiv preprint weXiv:2102.044 04010, 2021.
Chenzhuo Zhu, Song Han, Huizi Mao, and William Dally. Trained ternary quantization. arXiv preprint arXiv:1612.01064, 2016.
14

---

Published as a conference paper at ICLR 2023
APPENDIX
A ADDITIONAL BASELINE COMPARISON
We compare our approach with additional pruning and quantization approaches in Table 2 for ResNet-50 trained on ImageNet. We see that we continue to achieve high levels of model com- pression along with slice sparsity for inference speedups. Yuan et al. (2020) achieve high levels of sparsity but are unstructured requiring dedicated hardware obtain speedups. A similar case holds for the quantization approaches of Zhao et al. (2019); Jain et al. (2020) which can obtain inference speedups but with hardware optimized for 4-bit and 8-bit integer arithmetic. Additionally, they typ- ically require post-hoc training stages (Jain et al., 2020) to improve performance after quantization while our approach is a single stage trained end-to-end.
Table 2: Comparison of our approach with other pruning and quantization approaches for ResNet-50 trained on ImageNet. We continue achieve the most compression along with high slice spar- sity." denotes that the sparsity is unstructured and do not directly translate to computational benefits.
Size (MB)
Error
Spursity (%)
Algorithm
(Top-1 %)
Re (ImageNet)
Uncompressed
102.00 (1x)
23.7
0.0
Savarese et al. (2020)
8.36 (12x)
24.5
91.8*
Yuan al. (2020)
38.76 (3x)
24.8
38.8
Zhao et al. (2019) (4 bit)
12.75 (8x)
33.8
0.0
Zhao et al. (2019) (8 bit)
25.5 (4x)
25.3
0.0
Jain al. (2020) (4 w/ retraining)
12.75 (8x)
25.6
0.0
Jain et al. (2020) (8 w/o retraining)
25.5 (4x)
25.7
0.0
Jain et al. (2020) (8 bit w/ retraining)
25.5 (4x)
24.6
0.0
LilNetX (Best)
3.96 (26x)
25.4
66.7
LilNetX (Extreme)
2.96 (34x)
25.8
81.7
B HISTOGRAM OF WEIGHTS FOR DENSE UNCOMPRESSED MODEL
We obtain the histogram of weights of the various types of layers of a dense uncompressed ResNet- 50 model trained on ImageNet with only the cross entropy loss. We do not apply any weight decay in order to avoid enforcing any distribution on the weights. Results are shown in Fig. 7. We show histograms for 1, 3 3, 7 x 7 convolutions 7 x 7 convolutions, we pick a random dimension from a 9-dimensional or a 49-dimensional slice respectively, to highlight the histograms, as single probability model is fit to each dimension as shown in Eq. (3). We see that the distributions naturally follow unimodality and are more or less zero-centered even without any weight decay regularization. The 7 x 7 convolution weight distri- bution is less continuous due to relatively fewer weight values per dimension (192) but still weakly exhibits the property of unimodality and symmetry. This shows that networks trained with vanilla cross entropy loss prefer such distributions naturally. However, the probability models in Eq. (3) do not enforce any such distribution and can take on any random distribution. Thus, enforcing a Gaus- sian prior as proposed in Sec. 3.2 promotes unimodality and symmetry of the weight distributions which can be beneficial for network performance.
well as for the dense layer. For 3 x 3 and
C HISTOGRAM OF WEIGHTS FOR QUANTIZED LATENTS
To provide insights into the effect of our quantization, we visualize the histogram of the quantized latents as well for different weight groups. Results are shown in Fig. 8. We see that we obtain high levels of Os on almost all weight groups spanning different types of convolutional layers as well as the final dense layer. Fewer number of zeros are present in the initial 7 x 7 convolution similar to the uncompressed weights as shown in Fig. highlighting its importance in the network. Additionally, high amount of elements are zeros in 3 x 3 convolutions highlighting their redundancy and potential for compression compared to other convolutional layers the dense layer.
15

---

Published as a conference paper at ICLR 2023
1xl convolutions
3x3 convolutions - Din.
120000
300000
M0000
000000
60000
200300
40000
300000
20000
0
0.4
-0.2
Weight
Weight value
7x7 canvolutions Dim. 14
Dense/FC
20
300000
frequency
Fretueny
00000
10
60000
-0.25
Figure 7: Histogram of the weights for various types of layers of ResNet-50 model trained ImageNet without weight decay. For x 3 or 7 x convolutions we pick a random dimension of a 9-dimensional or 49-dimensional slice highlight the weight distribution for each dimension. Note the unimodal and zero-centered nature of each of these distributions, even without enforcing weight decay, highlighting the importance of the Gaussian prior as proposed in Sec. 3.2
D COMPARISON OF MODEL COMPRESSION WITH ENTROPY CODING AND SPARSE MATRIX FORMATS
Instead of entropy coding, the sparse matrices additionally be compressed using sparse matrix formats. We choose the two popular formats of Compressed Sparse Row (CSR) or Coordinate Format(COO). Results summarized in Table 3 for our best run for ResNet-50 shown in Table in the main paper. We see that entropy coding far outperforms the sparse formats of CSR and COO with COO obtaining better compression rates than CSR. This is expected as CSR/COO achieves high levels of compression only with extremely high levels of sparsity. With an unstructured sparsity level of ~80% storing only the non zero weights itself (and not their indices) provides maximum compression of 5x.
Table Sparse formats: Comparison of the effect of entropy coding vs. sparse matrix formats of CSR, COO on model compression of ResNet-50 trained on ImageNet. We show the model size in MB the latent weights along with the sparsity of the model weights.
Entropy Coding
CSR
COO
Slice Sparsity (%)
Unstructured Sparsity (%)
3.96 (26x)
57 (2x)
30 (3x)
66.7
78.8
E PARAMETER GROUPS FOR VARIOUS NETWORKS
We share weight decoders and probability models for different parameter groups of a network which can be seen being drawn from similar weight distributions. This limits the overhead in storing the weights of the corresponding decoders. We list the types of parameter groups for each network as follows:
16

---

Published as a conference paper at ICLR 2023
93
0.4
Quanlon peppetis
Weight Iastogoum for weight group 2x1 Cony
Wiight halogram for weight group Linear
Figure 8: Histogram of the quantized latents for various types of layers of a ResNet-50 model trained on ImageNet. For 3 x 7 convolutions we pick random dimension of 9-dimensional or 49-dimensional slice to highlight the weight distribution for each dimension. Note the high level of Os obtained due our sparsity priors.
VGG-16 consists of parameter group for each dense layer and parameter group for all convolutions leading to four weight decodees/probability models for each parameter group.
For ResNet-20-4 we use zero padding shortcut type A as defined in He et al. (2016), which leads to only parameter groups, one for the final dense layer and the other for convolutions.
For ResNet-18 18 trained ImageNet, use three parameter groups, for the initial 7x7 convolution, 3 x 3 convolutions, as well as the dense layer.
ResNer-50 consists of an additional parameter group for x convolutions compared to ResNet-18.
MobileNet-V2 V2 consists of parameter groups for the initial 3x3 convolution, final dense layer and the remaining 3x3 convolution.
F STANDARD ERROR FOR MULTIPLE RUNS
Sec. 5 in the main paper shows results when averaged across 3 seeds. this section, additionally provide the standard етгогs across the 3 random seeds. Results are summarized in Fig. 9 for the two datasets of CIFAR- 10/100. CIFAR-10 10 shows little to no standard error both in the x-axis (model size) and y-axis (top-1 validation accuracy). This suggests that the training is stable for different random seeds. For CIFAR-100 however, we observe large error the top-1 validation accuracy. We attribute this to the slow convergence for CIFAR- 100 also highlighted in Fig. 10.
different runs for ResNet-20-4 trained on the CIFAR- 100 dataset with varying values of Às and An. Results are shown in Fig. 10 when trained for 200 epochs. We see that validation accuracy (on the right y-axis) continues to increase towards the end of training between 190-200 epochs. At the same time, validation loss (on the left y-axis) also decreases. This suggests that the model hasn't 't fully converged by the end of 200 epochs. We hypothesize that this is artifact of the dataset as well the cosine decay schedule
CIFAR- R-AR-10 100 Convergence: We analyze the convergence
17

---

Published as a conference paper at ICLR 2023
CIFAR-10
CIFAR-100
Top-1 Accuracy (%)
72.5
93
72.0
Ay
0.0
07
3e 06
91
70.5
50
1000 150
200
00
100 Model Size (KB)
80
120
Model Size (KB)
Figure 9: Scatter plots with horizontal and vertical emror bars for 2esNet-20-4 trained CIFAR- 10/100. For a different random seed, model size changes leading to the error bar in the x-axis while the vertical bar represents the top-1 validation accuracy little variance in CIFAR-10 10 and slightly higher for CIFAR-100 due to slow convergence as shown Fig. 10
the y-axis. There is very
where learning rate decreases drastically towards the end of training and is not maintained for longer for better convergence.
Convergence (CIFAR-100)
4.5
Top-1 Validation (%)
50
Validation
Run
Run 2
2.5
2.0
20
75
100
125
150
175
200
Epoch Number
Figure 10: Convergence plots for 3 ResNet-20-4 runs on CIFAR-100. We see that loss (left axis) well top-1 validation accuracy (right axis) do not stabilize towards the end of training and respectively decrease/increase sharply suggesting that the training has not fully converged.
18

---

Published as a conference paper at ICLR 2023
G l2 vs. 1 vs. L0 NORM
In this section, we analyze the effect of different types of norm for both individual weights and groups. For individual weights, compare the l2 norm with the 4 norm while for the group norm, we compare the l2 norm with the (oo norm (1 weight norm is same as 4 group norm due to sum of absolutes). Results are summarized in Fig. 11 where top/bottom rows are for CIFAR-10/100 respectively. We see that l2 group norm outperforms its ls counterpart for both datasets. However, norm has little additional effect in terms of l2 weight norm. Additionally, the 12 group norm yields lesser slice sparsity for a given sparsity (c.g) highlighting the importance of ls for high structured sparsity. While So leads to higher sparsity, it also shows higher model size for given slice sparsity. Thus, there is an inherent tradeoff for 600 which leads to more sparsity but also larger model sizes (d.h).
H INITIALIZATION OF CONTINUOUS SURROGATES
The initialization of the continuous surrogate W of a latent space weight W and the decoder ma- trix plays an important in the neural network training. Naive He initialization (He et al., 2015) commonly used in training ResNet classifiers does not work in our case since small values of W get rounded to zero before decoding. Such initialization results in zero gradients for updating the parameters and the loss becomes stagnant. To overcome this issue, we propose a modification to the initialization of the different parameters. In our framework, recap that the decoded weights used in forward pass are obtained using
W = reshape W )
where W is a matrix in ZCnCou and v is a matrix in Zlxt (where | 1 for dense weights (and biases) while / = K2 for convolutional weights).
Our goal is to initialize W and T such that the decoded weights W follow He initialization. First, since W is rounded nearest integer (to obtain latent space weights W), we assume its elements to be drawn from a uniform distribution in [-b, bj where b > 0.5 in order to enforce atleast some non-zero weights after rounding to nearest integer. Next, we take the elements of to be a normal distribution with mean 0 and variance v.
Assuming the parameters to be i...d., and Var( (X) denoting the variance of any individual element in matrix X,
Var W) = 1 x Var V
X Var W)
(10)
Assuming a RELU activation, with f denoting the total number of channels (fan-in or fan-out) for layer, LHS of Eq. (10), using the He initializer becomes 7. RHS on the other hand can be obtained analytically
(2b
-
f
=
X
12
101
+1
24
b
(1I)
2
If ((26 +
Eq. (11) gives us a relationship between b (defining the uniform distribution of W) and v (defining the normal distribution of Vr Note that and f values are constant and known for each layer.
For weight decoder corresponding to a parameter group, the maximum value of in that group enforces the smallest value of b which should be above a minimum limit bmin- Denoting faax as the maximum fan-in or fan-out value for a parameter group. we get
24
1f Imax (22mmin + 1)2 -1)
Juax /
((2mmi + +122 - 1) +1
(12)
2
19

---

Published as a conference paper at ICLR 2023
The hyperparameter imin then refers to the minimum boundary any latent space parameter take in the network. By calculating the values of based on fmax: min and for various parameters based on the corresponding value of f. we then initialize the elements of W to be drawn from uniform distribution the interval [-b, b] and elements of y to be drawn from (00, v).
Note that f = fuax = b = bmin which shows that the minimum boundary corresponds to the layer with maximum channels (fan-in or fan-out) f.
By choosing an appropriate value of buin we obtain good initial values of the gradient which allows the network to converge well as training progresses. Umin offers an intuitive way of initializing the discrete weights. Too small value leads to most of the weights being set to zero while too large a value can lead to exploding gradients.
practice, we find that this initialization approach works well for Cifar experiments. For ImageNet experiments, we assume normal distribution instead of uniform distribution for W with sufficiently high variance for the network train.
I LICENSE
Table 4: Licenses of datasets.
Dataset
License
CIFAR-10 Krizhevsky et al. (2009)
MIT MIT BSD 3-Clause
CIFAR- 100 Krizhevsky et al. (2009)
ImageNet Deng et al. (2009)
Table 4 lists all datasets used and their licenses.
20

---

Published as a conference paper at ICLR 2023
CIFAR-10
CIFAR-10
x
Top-1 Accuracy (%)
4x
X
X
Au
93
93
L2
X
L1
92
92
As
Linf
L2
91
91
50
100
150
200
80
90
Model Size (KB)
Slice Sparsity
(a)
(b)
CIFAR-10
CIFAR-10
Slice Sparsity (96)
95
955
Au
90
Slice Sparsity
L1
85
x
As
Linf
80
80
75
0.94
0.96
0,98
1.00
50
100
150
200
Sparsity (Latents) (%)
Model Size (KB)
(c)
(d)
CIFAR-100
CIFAR-100
x
x
x
Au
72.0
Top-1 Accuracy
72.0
L2
X
L1
71.5
71.5
X
As
71.0
71.0
X
Linf
L2
70.5
70.5
60
80
100
120
140
85
90
95
Model Size (KB)
Slice Sparsity
(c)
(f)
CIFAR-100
CIFAR-10
Slice Sparsity (%)
95.0
x/
(96)
95.0
X
92.5
92.5
L1
90.0
90.0
Às
87.5
87.5
Linf
L2
85.0
85.0
0.97
0.98
0.99
60
80
100
120
140
Sparsity (Latents) (%)
Model Size (KB)
(h)
Figure 11: Comparison of l2 vs. livs. (x0 norm for various metrics of sparsity and size for both CIFAR-10 (top row) and 110 (bottom row). We that l2 group norm does better than loo group norm in terms accuracy vs model-size or slice sparsity (a.b.e,f). I weight norm has little additional effect compared to la weight norm. Ix2 favors higher slice sparsity for the same level of sparsity (c.g). Too tends to result in higher model size for given slice sparsity but also higher slice sparsity given model size, which shows the tradeoff between compression and sparsification.
21


Reviews:
[{"review_id": "trB-U1kgAje", "reviewer": "Reviewer_TxzT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new framework to jointly optimize the model size and inference acceleration.  Compared to previous methods, This paper can train the sparse model considering the model size and computation efficiency simultaneously using an end-to-end manner. The unified framework is novel and elegant.", "summary_of_the_review": "Na", "strength_and_weaknesses": "Strength:\n\nThe paper is well-written, and code is also provided for reproductivity.\n\nCompared to the previous pruning method, this paper is more practical, because mode deployment needs to consider model size and memory.\n\nThis paper analysis part is Thorough.\n\n\n\n\nWeaknesses:\n\nIn terms of the model's size, the author are encouraged to discuss different sparse formats (CSR, COO, N:M) [1] to encode the non-zero elements.\n\n\n\n\n\n[1]. STen: An Interface for Efficient Sparsity in PyTorch \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qA5a5182PbO", "reviewer": "Reviewer_1dzT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work adopted a combination of losses that jointly regulate model size, weight decay, and structured sparsity to achieve reduced weight encoding bits and slice sparsity. The authors show promising results in reducing model storage size in their experiments.", "summary_of_the_review": "Overall this work presented promising results but can use extra improvement.", "strength_and_weaknesses": "Strength:\nThe authors show that their approach can give a significantly better storage size compression ratio when compared with earlier works.\n\nWeakness:\nSome statements in the manuscript are not well supported. Moreover, there is a lack of comparison with a proper baseline, especially for the acceleration experiments.\n\nIt is not true: \"Most quantization works only optimize for model compression and not for inference speedups.\". It is a standard industry practice to use quantized models, and the use case of an optimized model only for storage size compression does exist but is limited. \n\nIt is unclear how much of the performance gain is from the training procedure and fine-tuning. Additional ablation experiments would make the paper stronger. \n\nThe slice-sparsity approach can be considered a particular case of block sparsity, and I believe there is probably a need for additional justification for this new approach. \n\nThe uncompressed MobileNet-v2 accuracy presented in the paper is incorrect. It should be 28 instead of 32.7", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ok4F-6rUuMT", "reviewer": "Reviewer_HHQc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an algorithm to optimally compress a model while preserving performance and improving inference speed. The algorithm achieves compression by modelling the weights as quantized latent representations sampled from a Gaussian prior which are further optimized using an entropy penalty. In order to ensure inference speedup, the algorithm also leverages a structured sparsity based loss. The authors show the efficacy of their algorithm on CIFAR-10/100 and Imagenet and claim better model compression due to the analogous behavior of the two constraints.", "summary_of_the_review": "Given the limited novelty and missing comparisons with modern structured sparsity enforcing algorithms, I am currently inclined towards a weak reject. The performance boosts due to the weight sampling mechanisms are unclear given that recent results in bit level compression have shown almost no loss in accuracy with significant speedups. However, i am open to changing my mind if the authors can provide reasonable rebuttals to my concerns.", "strength_and_weaknesses": "**Strengths**\n\n1. The problem of model compression which preserving inference speed is important for realtime and edge applications, and therefore interesting to the community.\n2. The authors show comparable results to existing compression approaches (Oktay et al. especially) while also adding sparsity as a constraint. \n3. The algorithm cleverly uses a zero mean prior to further enforce sparsity.\n\n**Weaknesses**\n1. I am not sure why the loss contains both a group sparse and sparsity enforcing constraint. These are obviously analogous constraints, and ideally the algo should be able to achieve the same result by adjusting the structured sparsity constraint. Fig. 3 shows some effects of sparsity on the model size, but it does not seem to be due to any change in slice sparsity. Could the authors clarify what Fig.3 is supposed to represent?\n\n2.  The paper also does not compare with more modern sparsification methods (for eg. Continuous Sparsification [1], structured continuous sparsification[2]) which achieve similarly sparse models while retaining performance. It would be fair to compare model sizes and accuracies of standard sparsification algorithms as well. \n\n3. I also suggest the authors add bit-quantization based approaches to the comparisons so that their results can be placed in the context of common compression methods. A few simple baselines of 8-bit or 4-bit quantization could help readers validate the need for training models with the proposed algorithm.\n\nRefs\n\n[1] Savarese et al., Winning the Lottery with Continuous Sparsification\n\n[2] Yuan et al., Growing Efficient Deep Networks by Structured Continuous Sparsification", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LpzXrh20jXa", "reviewer": "Reviewer_gqW8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to simultaneously learn model compression and sparsity in a latent space. Experiments on CIFAR-10 and ImageNet show that the proposed methods significantly outperforms previous competitive methods on compression and pruning only.", "summary_of_the_review": "The code idea is novel and experimental results are impressive, though contributions of the latent space of quantized weights are not clearly introduced.", "strength_and_weaknesses": "Pros.\n1) I think the core idea is interesting and reasonable: building a latent space of quantized model parameters for simultaneously learning model compression and sparsity (both unstructured and structured) with continuous weights maintained for sparsity learning.\n2) Experimental results on CIFAR-10 and ImageNet in Table 1 are impressive. The proposed method achieves much better compared to previous methods, from perspective of both compression rate and accuracy.\n\nCons.\nMy major concern is that the whole paper (including both the method description and experiments) does not clearly introduce contribution of quantization to the proposed method. To be specific: \n1) After reading Section 3, I am still not clear how the model weights are quantized during training. The Equation (8) does not show processing on quantization. \n2) In Section 4 and Section 5, the role/contribution of quantization to the experimental results is not formally analyzed. Maybe ablation studies are needed.\n\nThe paper clearly describes unstructured and structured sparsity but not quantization. According to my experiences on DNN model compression, quantization performs well on both model compression and acceleration, while structured spasity performs better on acceleration compared to unstructured sparsity. Anyway, I am not clear about the specific contribution of the latent space of quantized weights.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster