Title: Towards the Limit of Network Quantization
Authors: Yoojin Choi
Mostafa El-Khamy
Jungwon Lee
Venue: ICLR 2017
Keywords: Theory
Deep learning

Abstract: Network quantization is one of network compression techniques to reduce the redundancy of deep neural networks. It reduces the number of distinct network parameter values by quantization in order to save the storage for them. In this paper, we design network quantization schemes that minimize the performance loss due to quantization given a compression ratio constraint. We analyze the quantitative relation of quantization errors to the neural network loss function and identify that the Hessian-weighted distortion measure is locally the right objective function for the optimization of network quantization. As a result, Hessian-weighted k-means clustering is proposed for clustering network parameters to quantize. When optimal variable-length binary codes, e.g., Huffman codes, are employed for further compression, we derive that the network quantization problem can be related to the entropy-constrained scalar quantization (ECSQ) problem in information theory and consequently propose two solutions of ECSQ for network quantization, i.e., uniform quantization and an iterative solution similar to Lloyd's algorithm. Finally, using the simple uniform quantization followed by Huffman coding, we show from our experiments that the compression ratios of 51.25, 22.17 and 40.65 are achievable for LeNet, 32-layer ResNet and AlexNet, respectively.


Review discussion:
1) For the case of Huffman coding we see that Uniform quantization works very well actually compared to the Iterative ECSQ. Clarification - end of section 5.3 "Note that one can use Hessian-weighted mean..." - so what is used in the end, Hessian weighted mean or just a non-weighted mean? I would like to see a comparison of the two. If uniform quantization with non-weighted mean works, then this part of the paper boils down to contain no contribution as this method is straight-forward and already exists.
2) I don't understand the focus on storing parameters in one gigantic array or storing them individually for each layer. Isn't this just an implementation detail and for the sake of argument we can always just pretend they are in one gigantic array?
Other comments
- I think the paper is way too long for what it is. ICLR doesn't have a page limit but strongly suggests 8 pages while this is 17 pages long. Plenty of places where things can be summarized more efficiently, e.g. just beofre 6.1 seems like duplicated text of 6.1, many places keep saying the same thing like that it is assumed that pruning is performed first, etc.
- I would emphasise more the fact that this hessian weighting allows you to not tune per-layer compression rates as it's one of the main positives of the paper, it is explained in the text but it is not mentioned neither in the abstract nor in the conclusions.
Dear Reviewer,
1) We agree that uniform quantization is a straightforward method, but we would like to emphasize that it has not been shown previously in the literature that uniform quantization is actually one of the most efficient methods (with some approximation) for network quantization when entropy coding (e.g, Huffman coding) is employed. Please note that k-means clustering followed by Huffman coding was proposed in the deep compression paper (Han et al., 2015a). We identified in this paper that simple uniform quantization outperforms k-means clustering when Huffman coding follows.
In particular, we provided a theoretical understanding why uniform quantization is the best quantization scheme when entropy coding (e.g., Huffman coding) is employed; this follows from the connection of the network quantization problem to the ECSQ problem in information theory. We also would like to point out that uniform quantization is not always good. If fixed-length coding is employed instead of entropy coding, then Hessian-weighted k-means clustering is the best clustering scheme, which is also identified in this paper. For some hardware implementation, fixed-length coding may be preferable due to simplicity and Hessian-weighting can be beneficial for such cases.
The presented uniform quantization results were obtained with non-weighted mean, not Hessian-weighted mean. We evaluated uniform quantization with Hessian-weighted mean, which slightly outperforms uniform quantization with non-weighted mean, but we did not include the results in the paper. We would like to note that after fine-tuning quantized values (cluster centers), the performance difference between uniform quantization with non-weighted mean and uniform quantization with Hessian-weighted mean becomes marginal since cluster centers are fined-tuned and likely converge to the same local optimum.
As your suggestion, we will include the comparison of uniform quantization results with non-weighted mean and Hessian-weighted mean when we revise the paper.
2) You are right. It is an implementation issue.
However, we would like to note that the compression ratio can be different depending on how we implement this in particular when we use layer-by-layer clustering and quantization.
Suppose that we use one array for storing all binary encoded parameters in a network and one lookup table for binary codewords and quantized values. In this case, if we employ layer-by-layer clustering and quantization, then we need to assign some (additional) bits to each binary codeword for layer indication, which hurts the compression ratio. If we quantize all parameters of a network together, then we do not need additional overhead for layer indication when storing binary encoded parameters.
Thus, in this case, quantizing all network parameters together is better and Hessian-weighting can be used to address the different impact of the quantization errors across layers.
For layer-by-layer clustering and quantization, it is beneficial to use separate arrays and separate lookup tables for different layers since the layer information can be excluded in binary codewords. We note that Hessian-weighting can still provide gain even for layer-by-layer quantization since it can address the different impact of the quantization errors within each layer as well.
3) We will try to reduce the number of pages as your suggestion when we revise the paper.
4) Thank you for pointing this out. We will emphasize the advantage of hessian weighting more in the abstract and conclusions as your suggestion when we revise the paper.
We appreciate you for your valuable comments and suggestions. We will prepare a revision and upload it soon.
Best,
Yoojin
The paper analyzes the impact of quantization of the NN performance. Have you considered the impact of re-training part of the networks with quantized parameters?
I ask this because Quantization typically leads to zero gradients, but for some particular designs (which might be included in the proposed formulation, which is presented in a quite general setup), re-training after quantization part of the network is quite beneficial. See, for instance, in a relatively shallow setup, "Image classification with the fisher vector: Theory and practice", Sanchez et al., IJCV 2013. I don't see how these possible interactions between (re-)training and quantization are handled by the proposed formulation. Maybe it would be good to clarify this point.
Dear Reviewer,
Thank you for your comment. We agree that it is an interesting problem to investigate and analyze the impact of retaining in network quantization.
However, in this paper, we did not consider the interactions between retraining and quantization. In this paper, we analyze the expected loss due to quantization of all network parameters assuming no further retraining and focus on finding optimal network quantization schemes that minimize the performance loss while maximizing the compression ratio. After quantization, however, in our experiments, we fine-tuned the quantized values (cluster centers) so that we can recover the loss due to quantization and improve the performance further. Our experimental results showed that this final fine-tuning is also quite beneficial. We will clarify this point when we prepare a revision. We appreciate you for your valuable question and look forward to further comments.
Best,
Yoojin
Dear Reviewers,
We have upload a revised paper. Here is a list of summary for our revision.
1) We added a performance comparison of uniform quantization with non-weighed mean and uniform quantization with Hessian-weighted mean in Appendix A.3 in the revised paper. We also added a sentence for this in lines 2-4 of the last paragraph of Section 5.3 in the revised paper.
2) We revised the second paragraph of Section 4.5 in the original paper and clarified it as follows:
"We note that quantization of all layers is proposed under the assumption that all of binary encoded quantized parameters in a network are simply stored in one single array. Under this assumption, if layer-by-layer quantization is employed, then we need to assign some (additional) bits to each binary codeword for layer information (layer index), and it hurts the compression ratio. If we quantize all parameters of a network together, then we can avoid such additional overhead for layer indication when storing binary encoded quantized parameters.
Thus, in this case, quantizing all layers together is beneficial and Hessian-weighting can be used to address the different impact of the quantization errors across layers.
For layer-by-layer quantization, it is advantageous to use separate arrays and separate lookup tables for different layers since layer information can be excluded in each of binary codewords for network parameters. Hessian-weighting can still provide gain even in this case for layer-by-layer quantization since it can address the different impact of the quantization errors of network parameters within each layer as well."
3) We added the following sentence in the abstract as your suggestion (see Abstract lines 12-15 in the revised paper).
"Hessian-weighting properly handles the different impact of quantization errors not only within layers but also across layers and thus it can be employed for quantizing all layers of a network together at once; it is beneficial since one can avoid layer-by-layer compression rate optimization."
We also added the following sentence in Section 7 as your suggestion (see Section 7 lines 8-11 in the revised paper).
"Hessian-weighting is beneficial in quantizing all of the network parameters together at once since it can handle the different impact of quantization errors properly not only within layers but also across layers;
thus, using Hessian-weighting, we can avoid layer-by-layer compression rate optimization."
4) We reduced the number of pages from 17 to 15 (excluding reference and appendices).
- We removed the third paragraph in page 2 of the original paper.
- We removed the last two sentences in the first paragraph of Section 2 of the original paper.
- We removed the last three sentences in the first paragraph of Section 3 of the original paper.
- We removed the last three sentences in the first paragraph of Section 4 of the original paper.
- We removed the second to the fifth paragraphs in Section 5 of the original paper.
- We revised the paragraphs right before Section 6.1 and Section 6.1 as well in order to remove any duplications.
- We moved the second table in Table 1 of the original paper to Appendix A.4 in the revised paper since it is extra information.
5) We added the following sentence at the end of Section 4.1 of the revised paper for clarification.
"We note that we do not consider the interactions between retraining and quantization in our formulation. In this paper, we analyze the expected loss due to quantization of all network parameters assuming no further retraining and focus on finding optimal network quantization schemes that minimize the performance loss while maximizing the compression ratio. After quantization, however, in our experiments, we fine-tune the quantized values (cluster centers) so that we can recover the loss due to quantization and improve the performance further."
Thank you again for your comments and suggestions. We look forward to further comments.
Best,
Yoojin
The paper has two main contributions:
1) Shows that uniform quantization works well with variable length (Huffman) coding
2) Improves fixed-length quantization by proposing the Hessian-weighted k-means, as opposed to standardly used vanilla k-means. The Hessian weighting is well motivated, and it is also explained how to use an efficient approximation "for free" when using the Adam optimizer, which is quite neat. As opposed to vanilla k-means, one of the main benefits of this approach (apart from improved performance) is that no tuning on per-layer compression rates is required, as this is achieved for free.
To conclude, I like the paper: (1) is not really novel but it doesn't seem other papers have done this before so it's nice to know it works well, and (2) is quite neat and also works well. The paper is easy to follow, results are good. My only complaint is that it's a bit too long.
Minor note - I still don't understand the parts about storing "additional bits for each binary codeword for layer indication" when doing layer-by-layer quantization. What's the problem of just having an array of quantized weight values for each layer, i.e. q[0][:] would store all quantized weights for layer 0, q[1][:] for layer 1 etc, and for each layer you would have the codebook.
So the only overhead over joint quantization is storing the codebook for each layer, which is insignificant. I don't understand the "additional bit" part.
But anyway, this is really not a important as I don't think it affects the paper at all, just authors might want to additionally clarify this point (maybe I'm missing something obvious, but if I am then it's likely some other people will as well).
Dear Reviewer,
We thank you for your kind review and suggestion. We will prepare a revision as follows and upload it soon.
1) We will try to reduce the length of the paper more as your suggestion when we prepare a revision.
2) Thank you for pointing this out again. We will clarify the 'additional bits' part more when we revise the paper. What we wanted to say was that for layer-by-layer quantization, it is beneficial to use separate arrays and separate lookup tables for different layers. It is because the same codeword can be used for representing different values in different layers, which enhances the compression rate.
Best,
Yoojin
This paper proposes a novel neural network compression technique.
The goal is to compress maximally the network specification via parameter quantisation with a minimum impact on the expected loss.
It assumes pruning of the network parameters has already been performed, and only considers the quantisation of the individual scalar parameters of the network.
In contrast to previous work (Han et al. 2015a, Gong et al. 2014) the proposed approach takes into account the effect of the weight quantisation on the loss function that is used to train the network, and also takes into account the effect on a variable-length binary encoding of the cluster centers used for the quantisation.
Unfortunately, the submitted paper is 20 pages, rather than the 8 recommended. The length of the paper seems unjustified to me, since the first three sections (first five pages) are very generic and redundant can be largely compressed or skipped (including figures 1 and 2).
Although not a strict requirement by the submission guidelines, I would suggest the authors to compress their paper to 8 pages, this will improve the readability of the paper.
To take into account the impact on the network’s loss the authors propose to use a second order approximation of the cost function of the loss. In the case of weights that originally constitute a local minimum of the loss, this leads to a formulation of the impact of the weight quantization on the loss in terms of a weighted k-means clustering objective, where the weights are derived from the hessian of the loss function at the original weights.
The hessian can be computed efficiently using a back-propagation algorithm similar to that used to compute the gradient, as shown in cited work from the literature.
The authors also propose to alternatively use a second-order moment term used by the Adam optimisation algorithm, since it can be loosely interpreted as an approximate Hessian.
In section 4.5 the authors argue that with their approach it is more natural to quantise weights across all layers together, due to the hessian weighting which takes into account the variable impact across layers of quantisation errors on the network performance.
The last statement in this section, however, was not clear to me:
“In such deep neural networks, quantising network parameters of all layers together is more efficient since optimizing layer-by-layer clustering jointly across all layers requires exponential time complexity with respect to the number of layers.”
Perhaps the authors could elaborate a bit more on this point?
In section 5 the authors develop methods to take into account the code length of the weight quantisation in the clustering process.
The first method described by the authors (based on previous work), is uniform quantisation of the weight space, which is then further optimised by their hessian-weighted clustering procedure from section 4.
For the case of nonuniform codeword lengths to encode the cluster indices, the authors develop a modification of the Hessian weighted k-means algorithm in which the code length of each cluster is also taken into account, weighted by a factor lambda. Different values of lambda give rise to different compression-accuracy trade-offs, and the authors propose to cluster weights for a variety of lambda values and then pick the most accurate solution obtained, given a certain compression budget.
In section 6 the authors report a number of experimental results that were obtained with the proposed methods, and compare these results to those obtained by the layer-wise compression technique of Han et al 2015, and to the uncompressed models.
For these experiments the authors used three datasets, MNIST, CIFAR10 and ImageNet, with data-set specific architectures taken from the literature.
These results suggest a consistent and significant advantage of the proposed method over the work of Han et al. Comparison to the work of Gong et al 2014 is not made.
The results illustrate the advantage of the hessian weighted k-means clustering criterion, and the advantages of the variable bitrate cluster encoding.
In conclusion I would say that this is quite interesting work, although the technical novelty seems limited (but I’m not a quantisation expert).
Interestingly, the proposed techniques do not seem specific to deep conv nets, but rather generically applicable to quantisation of parameters of any model with an associated cost function for which a locally quadratic approximation can be formulated. It would be useful if the authors would discuss this point in their paper.
Dear Reviewer,
We appreciate you for your kind review and suggestions. We will prepare a revision and upload it soon.
1) We will try to reduce the length of the paper as your suggestion when we prepare a revision.
2) For layer-by-layer quantization, we need to find a proper compression rate for each layer, i.e., we need to find a proper number of clusters and a binary code for each layer. We would like to note that the complexity of the joint optimization (across layers) for this can exponentially increase as the number of layers increases. It is because the total number of possible combinations for clustering and coding that we need to examine for this optimization increases exponentially as the number of layers increases.
As your suggestion, we will elaborate this point more when we revise the paper.
3) As you pointed out, the quantization methods proposed in this paper are applicable for other models where the objective function can be approximated as a quadratic function. In this paper, we identified and derived the connection of the neural network quantization problem to this generic data compression problem; we think this is one of our contributions in this paper. We would be glad to discuss this point more when we revise the paper. Thank you for your suggestion.
Best,
Yoojin
This paper proposes a network quantization method for compressing the parameters of neural networks, therefore, compressing the amount of storage needed for the parameters.  The authors assume that the network is already pruned and aim for compressing the non-pruned parameters. The problem of network compression is a well-motivated problem and of interest to the ICLR community.
The main drawback of the paper is its novelty. The paper is heavily built on the results of Han 2015 and only marginally extends Han 2015 to overcome its drawbacks.
It should be noted that the proposed method in this paper has not been proposed before.
The paper is well-structured and easy to follow.
Although it heavily builds on Han 2015, it is still much longer than Han 2015. I believe that there is still some redundancy in the paper. The experiments section starts on Page 12 whereas for Han 2015 the experiments start on page 5.
Therefore, I believe much of the introductory text is redundant and can be efficiently cut.
Experimental results in the paper show good compression performance compared to Han 2015 while losing very little accuracy. Can the authors mention why there is no comparison with Hang 2015 on ResNet in Table 1?
Some comments:
1) It is not clear whether the procedure depicted in figure 1 is the authors’ contribution or has been in the literature.
2) In section 4.1 the authors approximate the hessian matrix with a diagonal matrix. Can the authors please explain how this approximation affects the final compression?
Also how much does one lose by making such an approximation?
minor typos (These are for the revised version of the paper):
1) Page 2, Parag 3, 3rd line from the end: fined-tuned -> fine-tuned
2) Page 2, one para to the end, last line: assigned for -> assigned to
3) Page 5, line 2, same as above
4) Page 8, Section 5, Line 3: explore -> explored
Dear Reviewer,
We appreciate the reviewer for your kind review, suggestions and corrections. We are preparing a revised version of the paper while reducing the number of pages as the reviewer suggested. We will upload the revision as soon as it is ready.
1. We referred [Han 2015] and compared our work to it since it had been the state-of-the-art in network compression.
However, we would like to note that our work focuses more on the theoretical analysis and optimization of network quantization.
In particular, we formulated the network quantization problem as an optimization problem to minimize the network performance loss given the compression ratio constraint. Hessian-weighting followed from this mathematical analysis and optimization, which is one of our contributions.
We also identified that the network quantization problem can be related to the entropy-constrained scalar quantization (ECSQ) problem in information theory when optimal variable-length codes, e.g., Huffman codes, are employed.
Then, two solutions of ECSQ were proposed for network quantization. One of them is uniform quantization.
Although uniform quantization is a straightforward method, it has not been used before with Huffman coding. We provided a theoretical understanding why uniform quantization is the best quantization scheme in this paper. We would like to emphasize that uniform quantization is not always good; it is not efficient with fixed-length coding, which is also first shown in this paper.
We also suggested some novel ideas in this paper, e.g.,  (1) using second moments of gradients calculated by the Adam optimizer as an alternative of Hessian for free and (2) quantizing all layers of a network together using Hessian-weighting.
2. ResNet was not evaluated in [Han 2015] and so there is no comparison with [Han 2015] on ResNet in Table 1.
We note that [Han 2015] evaluated layer-by-layer quantization while layer-by-layer compression rate is fine-tuned heuristically. Such layer-by-layer compression rate fine-tuning is not easy for very deep networks like ResNet.
Instead, we proposed quantizing all network layers together at once and using Hessian-weighting to handle the different impact of quantization errors properly not only within layers but also across layers. We included ResNet in our experiments in order to see the gain of our methods using Hessian-weighting for very deep networks. The impact of quantization errors could vary more substantially across layers than within layers.
Thus, Hessian-weighting can have more benefit for deeper networks like ResNet. Our experiment results also show more gain of our methods in ResNet.
3. The procedure in Figure 1 is a network compression pipeline proposed by [Han 2015]. We will clarify this point.
4. We approximated the Hessian matrix as a diagonal matrix in order to simplify the optimization problem for network quantization. With this diagonal approximation, the network quantization problem reduces to a weighted k-means clustering problem. As the reviewer mentioned, this simplification comes with performance loss.
However, we conjecture that the loss due to approximation is small.
The reason is that the contributions from off-diagonal terms are not always additive. Quantization errors can be either negative or positive and therefore the summation of off-diagonal terms may cancel with each other and end up with being a small value.
However, diagonal terms are all non-negative and therefore their contributions are always additive.
Thus, we think that the contribution of diagonal terms is dominant and the loss due to diagonal approximation is small.
We could not verify this conjecture though since solving the problem without diagonal approximation is too complex; we even need to compute the whole Hessian matrix, which is also too costly.
Best,
Yoojin
Dear Reviewers,
We have uploaded a revised paper.
1) We reduced the length of the paper to 10 pages excluding references and Appendix. To this end,
- We reduced the length of Abstract.
- We curtailed the first three sections substantially while removing Figure 1 and Figure 2.
- We removed the detailed discussion on the alternative of Hessian in Section 4.4.
- We removed the detailed discussion on the iterative ECSQ algorithm in Section 5.4 while moving some to Appendix A.2.
- We removed the experiment results for LeNet.
We will try to reduce it further next week. Please let us know your opinion. We are open to reduce it further.
2) We added Remark 1 in Section 4.1 to address the reviewer's comment on the effect of diagonal approximation:
"Remark 1. The diagonal approximation for Hessian simplifies the optimization problem as well as its solution for network quantization. This simplification however comes with some performance loss. We conjecture that the loss due to this approximation is small. The reason is that the contributions from off-diagonal terms are not always additive and their summation may end up with a small value.
However, diagonal terms are all non-negative and therefore their contributions are always additive."
3) We added Remark 3 in Section 4.1 to address the reviewer's comment on the generalization of Hessian-weighted quantization methods to other models:
"Remark 3. Observe that the relation of the Hessian-weighted distortion measure to the quantization loss holds for any model for which the objective function can be approximated as a quadratic function with respect to the parameters to quantize.
Hence, the quantization methods proposed in this paper to minimize the Hessian-weighted distortion measure are not specific to neural networks but are generally applicable to quantization of parameters of any model whose objective function is locally quadratic with respect to its parameters approximately."
4) We elaborated the exponential time complexity of layer-by-layer quantization in the last paragraph of Section 4.5:
"Optimizing compression ratios jointly across all layers (to maximize the overall compression ratio for all layers) requires exponential time complexity with respect to the number of layers. This is because the total number of possible combinations of compression ratios for individual layers increases exponentially as the number of layers increases."
5) We removed the description on the "additional bits" part from Section 4.5 to reduce the length of the paper. We note that it is just a minor implementation issue as the reviewer pointed out.
6) We clarified that [Han 2015] did not evaluate ResNet by adding a row for [Han 2015] in Table 1 and put "N/A" for ResNet.
7) We fixed the typos found by reviewers.
8) We removed Appendix A.1 and Appendix A.4 to reduce the length of the paper further.
Thank you again for your valuable comments and suggestions. We look forward to further comments.
Best,
Yoojin
Dear Reviewers,
We have reduced the length of the paper further and it is now 9 pages (excluding references and Appendix). Here is a list of major changes:
- We removed equation (2) of the previous revision since it is redundant.
- We removed Section 2 of the previous revision, and added Section 3.1 instead in this revision.
- We removed Remarks 1-3 in Section 4.1 of the previous revision, and moved all of them to Appendix A.1 instead in this revision.
- We removed the second paragraph of Section 4.5 in the previous revision.
- We reduced the length of Remark 4 in Section 5.3 of the previous revision.
We are open to reduce it further if needed. Please let us know your opinion. Thank you.
Best,
Yoojin

Meta-review:
role: PC
data: comment: The paper proposes using quantization schemes to compress the weights of a neural network. The paper carries out a methodical study of first deriving the objective function for optimizing the quantization, and then uses various quantization schemes. Experiments show competitive performance in terms of compression and accuracy tradeoff.
 
 I am happy to go with the reviewers' recommendations to accept the paper.
 
 A minor comment:
 It is important to mention other frameworks that compress neural networks, e.g. 
 https://arxiv.org/abs/1509.06569
 Although the weights are not quantized above, the number of parameters is reduced.
 Similarly there are other works looking into network compression. It will be good to mention them.

Decision: Accept (Poster)