Scalable Model Compression by Entropy Penalized Reparameterization
Authors: Deniz Oktay
Johannes Ballé
Saurabh Singh
Abhinav Shrivastava
Venue:  (2020)
Keywords: deep learning
model compression
computer vision
information theory

Abstract: We describe a simple and general neural network weight compression approach, in which the network parameters (weights and biases) are represented in a “latent” space, amounting to a reparameterization. This space is equipped with a learned probability model, which is used to impose an entropy penalty on the parameter representation during training, and to compress the representation using a simple arithmetic coder after training. Classification accuracy and model compressibility is maximized jointly, with the bitrate–accuracy trade-off specified by a hyperparameter. We evaluate the method on the MNIST, CIFAR-10 and ImageNet classification benchmarks using six distinct model architectures. Our results show that state-of-the-art model compression can be achieved in a scalable and general way without requiring complex procedures such as multi-stage training.

Published as a conference paper at ICLR 2020
SCALABLE MODEL COMPRESSION BY ENTROPY PENALIZED REPARAMETERIZATION
Deniz Oktay" Princeton University
Johannes Ballé Google Research Mountain View, CA, USA jbaile@google. com
Princeton, NJ, USA doktayêcs www.ssppiiinctteeeece
Saurabh Singh
Abhinav Shrivastava
Google Research
University of Maryland, College Park College Park, MD, USA abhinavêcs, umd edu
Mountain View, CA, USA
sau saurabhssinhh@googll... com
ABSTRACT
We describe a simple and general neural network weight compression approach, in which the network parameters (weights and biases) are represented in a "la- tent" space, amounting to a reparameterization. This space is equipped with a learned probability model, which is used to impose entropy penalty the pa- rameter representation during training, and to compress the representation using simple arithmetic coder after training. Classification accuracy and model com- pressibility is maximized jointly, with the bitrate-accuracy trade-off specified by a hyperparameter. We evaluate the method the MNIST, CIFAR-I0 and ImageNet classification benchmarks using six distinct model architectures. Our results show that state-of- of-the-art model compression can be achieved in a scalable and general way without requiring complex procedures such as multi-stage training.
INTRODUCTION
Artificial neural networks (ANNs) have proven be highly successful on a variety of tasks, and as a result, there is an increasing interest in their practical deployment. However, ANN parameters tend to require a large amount space compared to manually designed algorithms. This can be problem- atic, for instance, when deploying models onto devices over the air, where the bottleneck is often network speed, or onto devices holding many stored models, with only few used time. To make these models more practical, several authors have proposed to compress model parameters (Han et al., 2016, Louizos, Ullrich, et al., 2017; Molchanov al., 2017] Havasi et al., 2019). While other desiderata often exist, such as minimizing the number of layers filters of the network, focus here simply on model compression algorithms that 1. minimize compressed size while maintaining an acceptable classification accuracy, 2. are conceptually simple and easy implement, and 3. can be scaled easily to large models.
Classical data compression in Shannon sense (Shannon, 1948) requires liscrete-valued data (i.e., the data can only take on a countable number of states) and probability model on that data known both sender and receiver. Practical compression algorithms are often lossy, and consist of two steps. First, the data is subjected to (re))quantization. Then, a Shannon-style entropy coding method such arithmetic coding (Rissanen and Langdon. 1981) is applied to the discrete values, bringing them into a binary representation which can be easily stored or transmitted. Shannon's 's source coding theorem establishes the entropy of the discrete representation as a lower bound on the average length of this binary sequence (the hit rate), and arithmetic coding achieves this bound asymptotically. Thus, entropy is an excellent proxy for the expected model size.
The type of quantization scheme affects both the fidelity of the representation (in this case, the precision the model parameters, which turn affects the prediction accuracy) as well as the bit
Work performed during the Google AI Residency Program. (http///g.colairesdeenee))

---

Published as a conference paper at ICLR 2020
W2
U2
w/w
2011
W1
Figure 1: Visualization of representers in scalar quantization vs. reparameterized quantization. The axes represent two different model parameters (e.g., linear filter coefficients). Dots are samples of the model parameters, discs are the representers. Left: in scalar quantization, the representers must be given by a Kronecker product scalar representers along the cardinal axes, even though the dis- tribution of samples may be skewed. Right: in reparameterized scalar quantization, the representers are still given by a Kronecker product, but in transformed (here, rotated) space. This allows a better adaptation of the representers to the parameter distribution.
rate, since a reduced number of states coincides with reduced entropy. ANN parameters are typically represented as floating point numbers. While these technically have a finite (but large) number of states, the best results in terms of both accuracy and bit rate are typically achieved for a significantly reduced number of states. Existing approaches to model compression often acknowledge this by quantizing each individual linear filter coefficient an ANN to a small number pre-determined values (Louizos, Reisser, et al., 2019; Baskin et al., 2018; F. Li et al., (0016). This is known as scalar quantization (SQ). Other methods explore vector quantization (VQ), closely related to k- means clustering. in which each vector of filter coefficients is quantized jointly (Chen, J. Wilson, et al., 2015] Ulirich et al., 2017). This is equivalent to enumerating a finite set of representers (representable vectors), while in SQ the set of representers given by the Kronecker product of representable scalar elements. VQ is much more general than SQ, in the sense that representers can be placed arbitrarily: if the set of useful filter vectors all live in a subregion of the entire space, there is no benefit in having representers outside of that region, which may be unavoidable with SQ (fig. [ left). Thus, VQ has the potential to yield better results, but it also suffers from the "curse of dimensionality": the number of necessary states grows exponentially with the number of dimensions, making computationally infeasible to enumerate them explicitly, hence limiting VQ to only handful of dimensions in practice. One of the key insights leading to this paper is that the strengths of SQ and VQ can be combined by representing the data a "latent" space. This space can be arbitrary rescaling, rotation, otherwise warping of the original data space. SQ in this space, while making quantization computationally feasible, can provide substantially more flexibility in the choice of representers compared to the SQ in the data space (fig. [ right). This is in analogy to recent image compression methods based on autoencoders (Ballé, Laparra, et al., 2017 Theis al., 2017).
The contribution of this paper is two-fold. First, we propose a novel end-to-end trainable model com- pression method that uses scalar quantization and entropy penalization reparameterized space of model parameters. The reparamcterization allows to use efficient SQ. while achieving flexi- bility in representing the model parameters. Second, we provide state-of-the-an results on a variety of network architectures on several datasets. This demonstrates that more complicated strategies in- volving pretraining, multi stati--tage training, sparsification, adaptive coding, etc., as employed by many previous methods, are not necessary to achieve good performance. Our method scales to modern large image datasets and neural network architectures such as ResNet-50 on ImageNet.
ENTROPY PENALIZED REPARAMETERIIATIIN
We consider the classification setup, where given dataset D { (x), yN) consisting of pairs examples x; and corresponding labels y;- We wish to minimize the expected negative log-likeli on D, cross-entropy classification loss, over O the set of model parame-
2

---

Published as a conference paper at ICLR 2020
W
W
b
W
ccon
/nias
Tenny
fbius
ddense
W
b²
W
b²
W
ac
logits
conv1
conv2
classifier
Figure 2: Classifier architecture. The d tensors (annotated with a tilde) are stored in their com- pressed form. During inference, they read from storage, uncompressed, and transformed via into θ the usual parameters of a convolutional or dense layer (denoted without a tilde).
fconv
fennse
W HW
W reshaped to HxW xIxO
W
W
reshiped
IxO
bbb
W
V
10X10
IXI
Figure 3: The internals of feww and fdense in our experiments for layer k, annotated with the di- mensionalities. In feoav, H, W, I, O refer to the convolutional height, width, input channel, output channel, respectively. For Sdense. and refer to the number of input and output activations. For Jeame, we use affine transform, while for fäems we use a scalar shift and scale, whose parameters are captured in V. Note that in both cases, the number of parameters of f itself (labeled as Φ) is significantly smaller than the size of the model parameters it decodes.
ters:
0 = arg min
Σ
- log ply x; O),
(1)
(w,y)~
where ply x; 0) is the likelihood our model assigns to a dataset sample (a, y). The likelihood function is implemented using an ANN with parameters θ = [W), b", W b² ba...., WNJ, where W and b denote the weight (including convolutional) and bias terms at layer respectively.
Compressing the model amounts to compressing each parameter in the set θ Instead of compressing each parameter directly, we compress reparameterized forms of them. To be precise, we introduce the reparameterizations fbizs such that
22. b²...., WNJ and parameter decoders feoms faenses [W1, D., W
W Sconv (W
layer ls is convolutional,
W
k faense W
if layer k is fully connected. layer has a bias.
(3)
b
foias (bt)
We can think of each parameter decoder f as mapping from reparameterization space to parameter space. For ease of notation, we write FF {foonv faeuse- foiss) and @ F(Ø). The parameter decoders themselves may have learnable parameters, which we denote V Our method is visually summarized in figs. and 3
2.1 COMPRESSING WITH ENTROPY CODING
In order to apply Shannon-style entropy coder efficiently to the reparameterizations Q. need a discrete alphabet of representers and associated probabilities for each representer. Rather than handling expressive of representers, in VQ, choose to fix them to the integers, and
3

---

Published as a conference paper at ICLR 2020
achieve expressivity via the parameter decoders F instead.
& (i.e. a W or representing a weight or bias, respectively) is matrix in gdx interpreted as consisting of d samples from a discrete probability distribution producing vectors of dimension f. We fit factorized probability model
Each reparameterization g
A.
( - IIII (:(().)) j=1
(5)
to each column of Φ. using & different probability models for each corresponding parameter decoder (the form of qi is described the next section). Fitting of probability models is often done by minimizing the negative logllikeihood.. Assuming 6 follows the distribution g. Shamnon's 's source coding theorem states that the minimal length of a bit sequence encoding & is the self-information of Φ under q:
(6) which is identical to Shannon cross entropy to an expectation operator, and identical to the neg- ative log likelihood up constant factor. By minimizing I over g and during training, we thus achieve two goals: 1) we fit to the model parameters in a maximum likelihood sense, and 2) we directly optimize the parameters for compressibility.
[(Φ) = logz (()),
After training, we design an arithmetic code for q. and use it to compress the model parameters. This method incurs only a small overhead over the theoretical bound due to the finite length of the bit sequence (arithmetic coding is asymptotically optimal). Practically, the overhead amounts to less than 1% of the size of the bit sequence; thus, self-information is an excellent proxy for model size. Further overhead results from including description of V, the parameters of the parameter decoders, as well as of itself (in the form of table) in the model size. However, these can be considered constant and small compared to the total model size, and thus do not need to be explicitly optimized for.
The overall loss function is simply the additive combination of the original cross-entropy classifica- tion loss under reparameterization with the self-information of all reparameterizztions::
L(
Σ - loggr x; F())) A
ΣI())
I(Φ)
(m.9)~~)
ΦE
We refer to the second term (excluding the constant X) as the rate loss. By varying À across different experiments, we can explore the Pareto frontier of compressed model size vs. model accuracy. To compare our method to other work, we varied À such that our method produced similar accuracy, and then compared the resulting model size.
2.2 DISCRETE OPTIMIZATIOON
Since & is discrete-valued, need to make some further approximations in order to optimize L over it using stochastic gradient descent. To get around this, maintain continuous surrogates क.
For optimizing the classification loss, we use the strriigh-tthrough gradient estimator (Bengio et al., 2013)), which provides biased gradient estimate but has shown good results in practice. This consists of rounding the continuous surrogate to the nearest integer during training, and ignoring the rounding for purposes of backpropagation. After training, we only keep the discretized values.
In order to obtain good estimates for both the rate term and its gradient during training, we adopt relaxation approach previously described by Ballé, Minnen, al. (2018, appendix 6.1); the code is provided as an open source library[] In a nutshell, the method replaces the probability mass functions qi with set of non-parametric continuous density functions, which are based on small ANNs. These density models are fitted to âj. + Nj.., where nj. ~ (1 1) is i.i.d. uniformly distributed additive noise. This turns out to work well in practice, because the negative log likelihood of these noise-affected variates under the continuous densities approximates the self inforrrfooattin I:
I(Φ) R
log, (0,5 + n3,s)
(8)
4

---

Published as a conference paper at ICLR 2020
where q: denote the density functions. Once the density models are trained, the values of the proba- bility mass functions modeling & are derived from the substitutes qi and stored in a table, which is included in the model description. The parameters of q; are no longer needed after training.
2.3 MODEL PARTITIONING
A central component of our approach is partitioning the set of model parameters into groups. For the purpose of creating a model compression method, we interpret entire groups of model pa- rameters as samples from the same learned distribution. We define fully factorized distribution (()) = Ioa (Φ)) and introduce parameter sharing within the factors d of the distribution that correspond to the same group, as well as within the corresponding decoders. These group assign- ments are fixed a priori. For instance, in fig. 2 W and W can be assumed to be samples of the same distribution, that is ww Tw2- We also the same parameter decoder fconv to decode them. Further, each of the reparameterizations & is defined as a rank-2 tensor (a matrix), where each row corresponds to a "sample" from the learned distribution. The operations in f apply the same transformation to each row (fig. 3) As an example, in foone, each spatial / x W matrix of filter coefficients is assumed to be a sample from the same distribution.
Our method can be applied analogously to various model partitionings. In fact, in our experiments, we vary the size the groups, i.c., the number of parameters assumed i..d., depending on the total number of parameters of the model (9). The size of the groups parameterizes a trade-off between compressibility and overhead: if groups consisted of just one scalar parameter each, compressibility would be maximal, since q would degenerate (i.e., would capture the value of the parameter with certainty). However, the overhead would be maximal, since F and q would have a large number of parameters that would need to be included in the model size (defeating the purpose of compression). On the other hand, encoding all parameters of the model with one and the same decoder and scalar distribution would minimize overhead, but may be overly restrictive by failing to capture distribu- tional differences amongst all the parameters, and hence lead to suboptimal compressibility. We describe the group structure of each network that we use in more detail in the following section.
EXPERIMENTS
For our MNIST and CIFAR-10 experiments, we evaluate method by applying it to four dis- tinct image classification networks: LeNet300-100 (Lecun et al., 1998] and LeNet-5-Caffd on MNIST (LeCun and Cortes, 2010). well VGG-I6 (Simonyan and Zisserman, 2015 and ResNet-20 (He et al., 2016b Zagoruyko and Komodakis, 2016) with width multiplier 4 (ResNet-20- 4) on CIFAR- 10 (Zagoruyko and Komodakis, 2016). For our ImageNet experiments, we evaluate our method on the ResNet- 18 and ResNet-50 (He et al., 2016a) networks. We train all our models from scratch and compare them with recent sfate-of-the-arr methods by quoting performance from their respective papers. Compared to many previous approaches, we do not initialize the network with pre-trained or pre-sparsified weights.
We found it useful to use two separate optimizers: one to optimize the variables of the probability models q, and one to optimize the reparameterizations & and variables of the parameter decoders v. While the latter is chosen to be the same optimizer typically used for the task/architecture, the former is always Adam (Kingma and Ba, 2015) with a learning rate of 10-4, We chose to always use Adam, because the parameter updates used by Adam are independent of any scaling of the objective (when its hyper-parameter is sufficiently small). In our method, the probability model variables only get gradients from the entropy loss which is scaled by the rate penalty X. Adam normalizes out this scale and makes the learning rate of the probability model independent of X and of other hyperparameters such as the model partitioning.
3.1 MNIST EXPERIMENTS
We apply our method to two LeNet variants: LeNet300-100 and LeNet5-Caffe and report results We train the networks using Adam with a constant learning rate of 0.001 for 200,000
in table |
5

---

Published as a conference paper at ICLR 2020
iterations. To remedy some of the training noise from quantization, we maintain exponential moving average (EMA) of the weights and evaluate using those. Note that this does not affect the quantization, quantization is performed after the EMA variables are restored.
LeNec300-100 consists of 3 fully connected layers. We partitioned this network into three parameter groups: one for the first two fully connected layers, one for the classifier layer, and one for biases. LeNet5-Caffe consists of two 5x5 convolutional layers followed by two fully connected layers, with max pooling following each convolutional layer. We partitioned this network into four parameter groups: One for both of the convolutional layers, one for the penultimate fully connected layer, one for the final classifier layer, and one for the biases.
As evident from table I for the larger LeNet300-100 model, our method outperforms all the base- lines while maintaining comparable error rate. For the smaller LeNet5-Caffe model, our method is second only to Minimal Random Code Learning (Havasi et al., 2019). Note that in both of the MNIST models, the number of probability distributions
in every parameter group, includ- ing in the convolutional layers. To be precise, the W for the convolutional weights W will be H. W. 7. O 1. This is a good trade-off, since the model is small to begin with, and having = 5-555 = 25 scalar probability models for 5 x 5 convolutional layers would have too much overhead.
For both of the MNIST models, we found that letting each subcomponent of F be a simple dimension-wise scalar affine transform (similar Saease in fig. 3. was sufficient. Since each & is quantized to integers, having flexible scale and shift leads to flexible SQ, similar to Louizos, Reisser, al. (2019). Due to the small size of the networks, more complex transformation functions would lead to too much overhead.
3.2 CIFAR- 10 EXPERIMENTS
We apply our method to VGG-16 (Simonyan and Zisserman, 2015) and ResNet-20-4 (He et al., 2016b) Zagoruyko and Komodakis, 2016) and report the results in table [] For both VGG-16 and ResNet-200-4, we use momentum 0.9 with an initial learning rate of 0.1, and decay by 0.2 at iterations 256,000, 384,000, and 448,000 for total of 512,000 iterations. This learning rate schedule was fixed from the beginning and was not tuned in any way other than verifying that our models' training loss had converged.
VGG-16 consists of 13 convolutional layers of size 3 x followed by 3 fully connected layers. We split this network into four parameter groups: one for all convolutional layers and one each all fully connected layers. We did not compress biases. We found that the biases in 32-bit floating point format add up about 20 KB, which we add to our reported numbers.
ResNet-20-4 consists of ResNet groups with residual blocks each. There is also an initial con- volution layer and final fully connected classification layer. We partition this network into two parameter groups: one for all convolutional layers and one for the final classification layer. We again did not compress biases and include them in our results; they add up to about 11 KB.
For convolutions in both CIFAR-10 models, l - 0 x = feom and fäense are exactly as pictured in fig. 3 To speed up training, we fixed vow diagonal scaling matrix multiplied by the inverse real-valued discrete Fourier transform (DFT). We found that this particular choice performs much better than SQ, or than choosing a random but fixed orthogonal matrix in place of the DFT (fig. 4 From the emror vs. rate plots, the benefit of reparameterization in the high compression regime evident. VGG-16 and 2esNet-20-4 both contain batch normalization (loffe and Szegedy, 2015) layers that include moving average for the mean and variance. Following Havasi al. (2019), do not include the moving averages in our reported numbers. We do, however, include the batch normalization bias term 3 and let it function the bias for each layer (y is set to a constant 1).
3.3 IMAGENET EXPERIMENTS
For the ImageNet dataset (Russakovsky et al., 2015), we reproduce the training setup and hyper- parameters from He et al. (2) We put all 3 x 3 convolutional kernels in a single parameter group. similar to in our CIFAR experiments. In the case of ResNet-50, we also group all x convolutional kernels together. We put all the remaining layers in their groups This gives

---

Published as a conference paper at ICLR 2020
Model
Algorithm
Size
Error (Top-1)
Uncompressed
1.06 MB
1.6%
Bayesian Compression (GNJ) (Louizos, Ulirich, al., 2017
18.2 KB (58x)
1.8%
LeNec300-100
Bayesian Compression (GHS) (Louizos, Ullrich, al..
2017
18.0 KB (59x)
2.0%
(MNIST)
Sparse Variational Dropout (Molchanov et al., 2017
(113x)
1.8%
Our Method (SQ)
8.56 KB (124x)
1.9%
Uncompressed
1.72 MB
0.7%
Spurse Variational Dropout (Molchanov et al.. 2017
4.71 (365x)
1.0%
LeNet5-Caffe
Bayesian Compression (GHS) (Louizos, Ullrich, ct al
2017
2.23 KB (771x)
1.0%
(MNIST)
Minimal Random Code Learning (Havasi al..
2019
1.52 KB (1110x)
1.0%
Our Method (SQ)
2.84 KB (606x)
0.9%
Uncompressed
60 MB
6.6%
Bayesian Compression (Louizos, Ullrich, al.
2017
525 KB (116x)
9.2%
VGG-16
DeepCABAC (Wiedemann, Kirchhoffer, al.,
2019
960 KB (62.5x)
9.0%
(CIFAR-10)
Minimal Random Code Learning (Havasi
2019
417 (159x)
6.6%
Minimal Random Code Learning (Havasi et al.
2019
168 KB (452x)
10.0%
Our Method (DFT)
101 KB (590x)
10.0%
Uncompressed
17.2
5%
ResNet-20-4
Our Method (SQ)
176 KB (97x)
10.3%
(CIFAR-10)
Our Method (DFT)
128 KB (134x)
8.8%
Uncompressed
46.7
30.0%
ResNct-18
AP Coreset-S (Duhey ct al., 2018
3.11 MB (15x)
32.0%
(ImageNet)
Our Method (SQ)
2.78 MB (17x)
30.0%
Our Method (DFT)
1.97 MB (24x)
30.0%
Uncompressed
102 MB
25%
ResNet-50 (ImageNet)
AP + Coreset-S (Dubey al., 2018 DecpCABAC (Wiedemann, Kirchhoffer, al., 2019 Our Method (SQ)
6.46 MB (16x)
26.0%
6.06 MB (17x)
25.9%
(17x)
26.5%
Our Method (DFT)
5.49 MB (19x)
26.0%
Table 1: Our compression results compared to the existing state of the art. Our method is able VGG-16, and ResNet-
to achieve higher compression than previous approaches in LeNet300- 18/50, while maintaining comparable prediction accuracy. We have reported the models that have the closest accuracy to the baselines. For the complete view of the trade-off refer to figs, 4a and 4b For VGG-16 and ImageNet experiments, we report median of three runs with fixed entropy penalty. For ResNet-20-4, we report the SQ and DFT points closest to 10% error from fig. 4b6] Note that the values reproduce here for MRC are the corrected values found in the OpenReview version of the publication.
total of 4 parameter groups for ResNet-50 and 3 groups for ResNet-18, Analogously to the CIFAR experiments, we compare SQ using random orthogonal or DFT matrices for reparameterizing the convolution kernels (fig. 4a
4 DISCUSSION
Existing model compression methods are typically built on a combination of pruning, quantization, or coding. Pruning involves sparsifying the network either by removing individual parameters or higher level structures such as convolutional filters, layers, activations, etc. Various strategies for pruning weights include looking the Hessian (Cun et al., 1990) or just their lp norm (Han al., 2016). Srinivas and Babu (2015) focus on pruning individual units, and H. Li et al. (2017) prunes convolutional filters. Louizos, Ullrich, al. (017)) and Molchanov al. (2017). which we compare to in our compression experiments, also prune parts of the network. Dubey et al. (2018) describe dimensionality reduction technique specialized for CNN architectures. Pruning a simple approach to reduce memory requirements as well as computational complexity, but doesn't inherently tackle the problem of efficiently representing the parameters that are left. Here, we primarily focus on the latter: given model architecture and task. we' re interested in finding set of parameters which can be described in compact form and yield good prediction accuracy. Our work is largely orthogonal to the pruning literature, and could be combined if reducing the number of units is desired.

---

Published as a conference paper at ICLR 2020
0.37
0.6
Our Method (SQ)
Our Method (5Q)
0.36
Our Method (Random Ortho]
0.5
Our Method (Random Ortho)
Classfficttion Error
0.35
Method
Clansfiication Error
Method
0.34
0.4
0.32
0.31
02
0.30
0.29
0.1
+9P
0.28
0.0
05
10
15
2.0
2-5
3.0
355
4.0
4.5
5.0
Model Size (Bytes)
1e6
Model Size (Bytes)
1e5
(a) ResNet-18 (ImageNet)
(b) ResNeet2204 (CIFAR-10)
Figure 4: Error vs. rate plot for R8sNet-18 on ImageNet and ResNet-20-4 on CIFAR-10 10 using SQ. DFT transform, and random but fixed orthogonal matrices. The DFT is clearly beneficial in com- parison to the other two transforms. All experiments were trained with the same hyper-parameters (including the set of entropy penalties), only differing the transformation matrix.
Quantization involves restricting the parameters to a small discrete set of values. There is work in binarizing ternarizing networks (Courbariaux et al., 2015) F. Li et al., 2016; Zhou et al., 20118) via either straight-through gradient approximation (Bengio et al., 2013) or stochastic rounding (Gupta et al., 2015). Recently, Louizos, Reisser, al. (2019) introduced new differentiable quantization procedure that relaxes quantization. We use the straight- through heuristic, but could possibly use other stochastic approaches to improve methods. While most of these works focus uniform quantization, Baskin et al. (2018) also extend to non-uniform quantization, which our generalized transformation function amounts
Han al. (2016) and Ulirich et al. (2017) share weights and quantize by clustering, Chen, J. Wilson, et al. (2015) randomly enforce weight sharing, and thus ef- fectively perform VQ with pre-determined assignment of parameters to representers. Other works also make the observation that representing weights in the frequency domain helps compression; Chen, J. T. Wilson, et al. (2016) randomly enforce weight sharing the frequency domain and Wang et al. (2016) use K-means clustering in the frequency domain.
Coding (entropy coding, or Shannon-style compression) methods produce a bit sequence that can allow convenient storage or transmission of trained model. This generally involves quantization as a first step, followed by methods such as Huffman coding (Huffman, 1952), arithmetic coding (Ris- sanen and Langdon, [981), etc. Entropy coding methods exploit known probabilistic structure of the data to produce optimized binary sequences whose length ideally closely approximates the cross entropy of the data under the probability model. In many cases, authors represent the quan- tized values directly as binary numbers with few digits (Courbariaux et al., 2013 F. Li et al., 2016 Louizos, Reisser, et al., 2019), which effectively leaves the probability distribution unexploited for minimizing model size; others do exploit it (Han et al., 2016)). Wiedemann, Marban, al. (2018) formulate model compression with an entropy constraint, but use (non-reparameterizze) scalar quantization. Their model significantly underperforms all the state-of-the-art models that we compare with (table Mp. Some recent work has claimed improved compression performance by skip- ping quantization altogether (Havasi et al., 2019). Our work focuses on coding with quantization.
the values
Han et al. (2016) defined their method using four-stage training process: 1. training the origi- nal network, 2. pruning and re-training, 3. quantization and re- re-training, and 4. entropy coding. This approach has influenced many follow-up publications. In the same vein, many current high- performing methods have significant complexity in implementation or require multi-stage training process. Havasi et al. (2019) requires several stages of training and retraining while keeping parts of the network fixed. Wiedemann, Kirchhoffer, et al. (2019) require pre-sparsification of the net- work, which is computationally expensive, and use a more complex context-adaptive variant of arithmetic coding which may be affected by MPEG patents. These complexities prevent meth- ods from scaling to larger architectures or decrease their practical usability. In contrast, our method requires only single training stage followed by a royalty-free version of arithmetic coding. In
8

---

Published as a conference paper at ICLR 2020
addition, our code is publicly available
Our method has parallels to recent work in learned image compression (Ballé, Laparra, et al., 2017 Theis et al., 2017) that uses end-to-end trained deep models for significant performance improve- ments in lossy image compression. These models operate in an autoencoder framework, where sealar quantization is applied in the latent space. Our method can be viewed as having just a decoder that is used to transform the latent representation into the model parameters, but no encoder.
5 CONCLUSION
We describe a simple model compression method built on two ingredients: joint (i.e., end-to-end) optimization of compressibility and task performance in only single training stage, and reparam- eterization of model parameters, which increases the flexibility of the representation over scalar quantization, and is applicable to arbitrary network architectures. We demonstrate that state-of-the- art model compression performance can be achieved with this simple framework, outperforming methods that rely on complex, multi-stage training procedures. Due to its simplicity, the approach is particularly suitable for larger models, such as VGG and especially ResNets. In future work, we may consider the potential benefits of even more flexible (deeper) parameter decoders.
REFERENCES
Ballé, Johannes, Valero Laparra, and Eero Simoncelli (2017). "End-to-end Optimized Image Compression". In: Proc. of 5th Ini. Conf. on Learning Representations. URL: https /
openreview net/forum? 3idrummmddd
Ballé, Johannes, David Minnen, et al. (2018). "Variational image compression with a scale hyper- prior". In: Proc. of 6th Int. Conf. on Learning Representations. UtL:/tttpp:://ppeeeeiiii net/forum? rkG0FMZRk
Baskin, Chaim et al. (2018). "UNIQ: Uniform Noise Injection for the Quantization of Neural Net- works". In: arXiv preprint arXivvI804: 10969.
Bengio, Yoshua, Nicholas Léonard, and Aaron Courville (2013). "Estimating or propagating gradi- ents through stochastic neurons for conditional computation", In: arXiv preprint 13X8. 3432. Chen. Wenlin, James Wilson, et al. (July 2015). "Compressing Neural Networks with the Hash- ing Trick". In: Proceedings of the 32nd International Conference on Machine Learning. Ed. by Francis Bach and David Blei. Vol. 37. Proceedings of Machine I Learning Research. Lille, France: URL: http: : / pwwceeeo ngs mlr press vv7 / chenc15.
PMLR, pp. 2294. html
Chen, Wenlin, James T. Wilson, et al. (2016). "Compressing Convolutional Neural Networks in the Frequency Domain". In: KDD.
Courbariaux, Matthieu, Yoshua Bengio, and Jean-Pierre David (2015). "BinaryConnect: Training
Deep Neural Networks with binary weights during propagations" In: Advances in Neural Infor- mation Processing Systems 28. Ed. by C. Cortes et al. Curran Associates, Inc., pp. 3123-3131. Cun, Yann Le, John S. Denker, and Sara A. Solla (1990). "Optimal Brain Damage". In: Advances in Neural Information Processing Systems. Morgan Kaufmann, pp. 598-605.
Dubey, Abhimanyu, Moitreya Chatterjee. and Narendra Ahuja (2018). "Coreset-Basce Neural Net- work Compression". In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 454 470.
Gupta, Suyog et al. (2015). "Deep Learning with Limited Numerical Precision". In: Proceedings of the 32Nd International Conference on International Conference on Machine Learning Vol- ume 37, ICML' 15. Lille, France: JMLR.org. Rrg.. pp. 1737-1746.
URL: http::/t/// // acm. org/
citation.
3045303)
Han, Song, Huizi Mao, and William J. Dally (2016). "Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding". In: 4th International Confer- ence on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. Ed. by Yoshua Bengio and Yann LeCun. URL: /ttp://tt///xxxooo///oo//o
1510.00143
Refer to in httpsp///gttaa...c./////o/o/oo//

---

Published as a conference paper at ICLR 2020
Havasi, Marton, Robert Peharz, and José Miguel Hernández-Lobato (2019). "Minimal Random
Code Learning: Getting Bits Back from Compressed Model Parameters". . In: International Con-
ference on Learning Representations, URL: https: : / openreview nnt / fooum? ? iid
E1f0YiCctm
He, Kaiming et al. (2016a). "Deep residual learning for image recognition". In: Proceedings of the
IEEE conference on computer vision and pattern recognition, pp. 770-778.
Science,,, (2016b). "Identity Mappings in Deep Residual Networks", In: Lecture Notes in Computer Science,
pp. 630-645. DOI: 10. 1 007 97 3- 31 31-9--99-444 949 0_
Huffman. David A. (Sept. 1952). "'A Method for the Construction of Minimum-Redundanee Redundancy Codes".
In: Proceedings of the Instiutte of Radio Engineers 40.9, pp. 1098-1101.
loffe, Sergey and Christian Szegedy (2015). "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift". In: Proceedings of the 32nd International Con- ference on International Conference on Machine Learning Volume 37. ICML' 15. Lille, France: JMLR.org, pp. 448-456.
Kingma, Diederik P. and Jimmy Ba (2015). "Adam: A Method for Stochastic Optimization". In: arXiv e-prints. Presented at the 3rd Int. Conf. on Learning Representations. arXiv: 1412. 6980
LeCun, Yann and Corinna Cortes (2010). "MNIST handwritten digit database". In: URL: http: wwom/exdde txdbb htndb/mnist//
7/yann. lecun Com/
Lecun. Yann et al. (1998). "Gradient-based learning applied to document recognition". In: Proceed-
ings of the IEEE. pp. 2278-2324.
Li. Fengfu, Bo Zhang, and Bin Liu (2016). Ternary Weight Networks. arXiv: 1605 04711
[cs
Li, Hao et al. (2017). "Pruning filters for efficient convnets". In: International Conference on Learn- ing Representations (ICLR).
Louizos, Christos, Matthias Reisser, et al. (2019). "Relaxed Quantization for Discretized Neu-
ral Networks". In: International Conference on Learning Representations. URL: https: /
openreview , net/ net/forumts Forumrorriiioee id= i
Louizos, Christos, Karen Ullrich, and Max Welling (2017). "Bayesian compression for deep learn- ing". In: Advances in Neural Information Processing Systems, pp. 3288-3298.
Molchanov, Dmitry, Arsenii Ashukha, and Dmitry Vetrov (2017). "Variational Dropout Sparsifies
Deep Neural Networks". In: Proceedings the 34th International Conference on Machine Learn- ing Volume 70. ICML' Sydney, NSW, Australia: JMLR. org, pp. 2498-2507.
Rissanen, Jorma and Glen G. Langdon Jr. (1981). "Universal modeling and coding". In: IEEE Trans- actions on Information Theory 27.1. DOI: 10 1 / 1188111000 626622
Russakovsky, Olga et al. (2015). "ImageNet Large Scale Visual Recognition Challenge". In: Inter- national Journal of Computer Vision (IJCV) 115.3, pp. 211-252. DOT: 1C. 1007 s112633
015-0816-1 6-y
Shannon, Claude E. (1948). "A Mathematical Theory of Communication" In: The Bell System Tech-
nical Journal 27.3. DOI: 10. 1002/1.1538 7305. 1948. tb01338..... x
Simonyan, K. and A. Zisserman (2015). "Very Deep Convolutional Networks for Large-Scale Image Recognition".. In: International Conference on Learning Representations.
Srinivas, Suraj and R. Venkatesh Babu (2015). "Data-free Parameter Pruning for Deep Neural Net- works". In: Proceedings of the British Machine Vision Conference 2015, BMVC 2015, Swansea, UK, September 7-10, 2015. Ed. by Xianghua Xie, Mark W. Jones, and Gary K. L. Tam. BMVA Press. pp. 31.1-31.12. DO1: 10. 5244/ C. 29.31 29.31
URL: htt.:: /ol::///ooi.. org 10. 5244/C.
Theis, Lucas et al. (2017). "Lossy Image Compression with Compressive Autoencoders".. In: Proc. of 5th Int. Conf. on Learning Representations. URL.:
net/forummfff
id=rJiNww9gg
Ulrichh, Karen, Edward Meeds, and Max Welling (2017). "Soft Weight-Sharing for Neural Network Compression". . In: International Conference on Learning Representations (ICLR). Wang, Yunhe et al. (2016). "CNNpack: Packing Convolutional Neural Networks in the Frequency Domain". In: Advances in Neural Information Processing Systems 29: Annual Conference on
Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 253-
261. URL:: http:: : papers nips cc / paper 6390- cnnpack -aapppckkkppkkkka i ng
convolut
networkss -nnthee-e - fechee-rreeencc--oo domain
Wiedemann, Simon, Heiner Kirchhoffer, al. (2019). DeepCABAC: Context-adaptive binary arith-
metic coding for deep neural network compression. arXiv: 1905. 08318 [cs. LG]]
10

---

Published as a conference paper at ICLR 2020
Wiedemann, Simon, Arturo Marban, et al. (2018). Caaropy-Constrainee Training of Deep Neural
Networks. arXiv: [ 1812 07520 Tes.LG)
Zagorukko, Sergey and Nikos Komodakis (2016). "Wide Residual Networks". In: Procedings of the
British Machine Vision Conference 2016. DOT: 10. 52.4/..30. 87 URL: http::ttpppttm / //xx doi.
org/11.5544/C. 30 87,
Zhou, Aojun et al. (June 2018). "Explicit Loss Aware Quantization for Low-Bit Deep Neural
Networks". In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).


Reviews:
[{"review_id": "ryeXVdBRKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a compression algorithm for neural networks. It uses a linear projection to map the weights and biases to a latent space where they are quantized using a learnt coding distribution. The method is simple yet it shows strong performance of a variety of compression benchmarks.\n\nOriginality: The core idea is related to (Balle et al. 2018), but the paper puts its own twist to it with the projections and applies it to model compression. It is certainly an interesting direction to explore.\n\nPresentation: The paper is well written. It is easy to read and understand. It properly cites prior works and contains all the technical details. I appreciate that the authors fit the paper into the recommended 8 pages.\n\nImpact: The method shows very strong performance in terms of compression ratio, but its unclear whether the compressed model can be used to speed up inference. Currently the main use case would be saving storage/bandwidth.\n\nQuestions:\n- In section 2.2, the paper talks about the form of the coding distribution. It has d components and l dimensions. How is d determined?\n- Section 2.1, How are the weights grouped for coding? is every filter its own group? If the experiments use different groups for different models, how is it decided which model uses which approach?\n- \\phi was fixed for the CIFAR10/ImageNet experiments. Could you provide further insight into why this choice was made?\n\nAssessment:\nWhile the idea is not groundbreaking, it is very well presented and evaluated and shows strong performance.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1xcYOqhKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a novel, information theoretic approach to learning compressed neural networks, whereby model parameters $\\theta$ are expressed in an (approximately discrete) latent space as $\\theta = f_{\\psi}(\\phi)$. During training, \"reparameterizations\" $\\Phi$ are treated as draws from a generative distribution $q(\\phi)$. As per the source coding theorem, penalizing the entropy of $q$ (more accurately, of $\\hat{q}$) directly promotes compressible choices of $\\Phi$ and, therefore, of $\\Theta$. After training, arithmetic coding of $q$ is used to compress $\\Phi$. Empirical results on a suite of common benchmarks demonstrates the proposed algorithm gracefully trades off between model performance and size.\n\n\nFrequency Domain Interpretation:\n  Here, I am operating off of the assumptions that, in Sect 3.2 paragraph 3, $\\phi_{w}$ should be $\\psi_{w}$. If so, can \"Our Method (DFT)\" also be interpreted as frequency domain SQ (of convolutional filters)? When trained on natural images, CNN filters are typically \"smooth\" (see any visualization thereof) and this smoothness translates as a prior of sorts on $\\Phi_{w}$. This insight was previously explored in [1, 2] for purposes of compressing CNNs. Does evidence of this manifest in your experiments? For example, do the empirical distributions of high- and low-frequency components of $\\phi_{w}$ differ?\n\n\nFeedback:\n  The authors propose a general framework where in (seemingly) arbitrary decoders $f$ can be used for purposes of \"reparameterization\". It is therefore somewhat disappointing that, in practice, $f$ is restricted to affine transformations and, in some cases, even fixed. Memory issues notwithstanding, it is unclear how well complex decoders $f$ could be jointly learned with (surrogate) encodings $\\tilde{\\Phi}$ and density models $\\tilde{q}$.\n\n  The lack of comparison between the proposed method's test-time throughput and that of its baselines leaves open questions regarding potential real-world use cases. Much of the time, methods for compressing neural networks divide their attention between both memory footprint and prediction speed. Outright ignoring this aspect of the problem seems odd. As a reviewer, I would much rather see the authors take a proactive approach in addressing this issue.\n\n\nQuestions:\n  - What do results look like if convolutional decoders $f_{\\text{conv}}$ are jointly learned during training?\n  - How sensitive is the algorithm to different groups assignments? Can assignments be reliably made by simply looking at the architecture?\n\n\nMinor Comments:\n  - The general notation of the paper is cumbersome at times, can $\\phi$ to be changed to, e.g., $\\tilde{\\theta}$? Similarly, can notation such as $\\theta_{k, W}$ be simplified as $W^{k}$?\n\n  - Section 2 jumps around a bit. Consider trying to order things \"chronologically\" such that training comes before compression? By the same token, details regarding \"Model Partitioning\" could be move to the end of the section such that the preceding material just refers to $\\theta$ more abstractly?\n\n  - Some additional details regarding training (esp. discretization and model distributions $\\tilde{q}$) would be appreciated.\n\n\n[1] \"Compressing convolutional neural networks in the frequency domain\", Chen et al. 2016\n[2] \"CNNpack: Packing convolutional neural networks in the frequency domain\", Wang et al. 2016", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1eIk0dc9B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces an end-to-end method for the neural network compression, which is based on compressing reparametrized forms of model parameters (weights and biases). Shannon-style entropy coder is applied to the reparametrizations.\n\nReparameterized forms are tensors stored in the compressed format. During inference, they are uncompressed into integer tensors and transformed via parameter decoders into weights/biases of convolutional and dense layers.\n\nDuring training, model parameters are manually partitioned into groups, and within a group they are considered as samples from the same learned distribution. Similarly, parameter sharing is introduced among the corresponding parameter decoders.\nLoss function, which is minimized during training is a sum of rate loss (self-information of all reparametrizations) and cross-entropy classification loss under reparametrization. A trade-off between compressed model size and model accuracy can be explored by varying a constant before the rate loss.\n\nDuring optimization, several tricks are applied. “Straight-through” gradient estimator is used to optimize the loss function over discrete-valued reparametrizations by means of stochastic gradient descent. Relaxation is used to obtain good estimates for both the rate term and its gradient during training.\n\nThe proposed idea is well-founded and intuitive. The proposed method is extensively evaluated on different classification network architectures and datasets. It provides good compression while retaining a significant portion of accuracy.\n\nIn the experiments, It'd be interesting to see a comparison on more efficient networks like MobileNets, ShuffleNets on ImageNet dataset. Also, I wonder whether under the same compression rate the proposed method outperforms DeepCompression (Han et al., 2015) in terms of accuracy? (for example, for LeNets and VGG-16)\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes a simple method for neural network compression by applying Shannon-type encoding. This is a fresh and nice idea, as noted by reviewers. A disadvantage is that the architectures on ImageNet are not the most efficient ones. Also, the review misses several important works on low-rank factorization of weights for the compression (Lebedev et. al, Novikov et. al).   But overall, a good paper.
Decision: Accept (Poster)