Title: Incremental Network Quantization: Towards Lossless CNNs with Low-precision Weights
Authors: Aojun Zhou
Anbang Yao
Yiwen Guo
Lin Xu
Yurong Chen
Venue: ICLR 2017
Keywords: Deep learning
Optimization

Abstract: This paper presents incremental network quantization (INQ), a novel method, targeting to efficiently convert any pre-trained full-precision convolutional neural network (CNN) model into a low-precision version whose weights are constrained to be either powers of two or zero. Unlike existing methods which are struggled in noticeable accuracy loss, our INQ has the potential to resolve this issue, as benefiting from two innovations. On one hand, we introduce three interdependent operations, namely weight partition, group-wise quantization and re-training. A well-proven measure is employed to divide the weights in each layer of a pre-trained CNN model into two disjoint groups. The weights in the first group are responsible to form a low-precision base, thus they are quantized by a variable-length encoding method. The weights in the other group are responsible to compensate for the accuracy loss from the quantization, thus they are the ones to be re-trained. On the other hand, these three operations are repeated on the latest re-trained group in an iterative manner until all the weights are converted into low-precision ones, acting as an incremental network quantization and accuracy enhancement procedure. Extensive experiments on the ImageNet classification task using almost all known deep CNN architectures including AlexNet, VGG-16, GoogleNet and ResNets well testify the efficacy of the proposed method. Specifically, at 5-bit quantization (a variable-length encoding: 1 bit for representing zero value, and the remaining 4 bits represent at most 16 different values for the powers of two), our models have improved accuracy than the 32-bit floating-point references. Taking ResNet-18 as an example, we further show that our quantized models with 4-bit, 3-bit and 2-bit ternary weights have improved or very similar accuracy against its 32-bit floating-point baseline. Besides, impressive results with the combination of network pruning and INQ are also reported. We believe that our method sheds new insights on how to make deep CNNs to be applicable on mobile or embedded devices. The code will be made publicly available.

TL;DR: This paper presents INQ, targeting to efficiently transform any pre-trained full-precision convolutional neural network (CNN) model into a low-precision version whose connection weights are constrained to be either powers of two or zero.

Review discussion:
1) Can you compare to more existing methods than just to BWN and TWN in Table 4?
For example, a very relevant one is Han et al. 2016. Looking at your Table 1, AlexNet ref 32-bit versus AlexNet 5-bit, the relative size of the networks is 5/32=16%. Comparing this to their figure 6, it seems that at 16% they get 0% accuracy loss.
So, is this method really worth it?
2) Why did you stick to weights as powers of 2, why not simply do k-means?
3) Regarding weight partitioning - you tried only random partitioning and partitioning based on the absolute value. Doesn't it make sense to also partition based on quantization error? You can just find the set of weights that are most similar to values they would be quantized to, and tune the rest.
4) I'm not completely clear about the formulation that is repeated a few times "5-bit (4 bits for powers of two plus 1 bit for zero)" ? Does it mean that out of the possible 2^5 values that can be expressed with 5 bits, one is a zero, others are +- powers of 2, and one is unused? Or are you talking about some variable length encoding (though in this case the representation wouldn't be 5 bit)?
5) I think the region between equation 1 the end of section 2.1 needs some clarifications, though I think I understand it, it required a bit of thinking to understand what is going on. Firt, "bounding the elements in P_l" below equation 1 is a bit confusing, as the elements are bounded just by [-2^n1, +2^n1] (so no dependence on n2).
Second, where does equation 2 come from, it's not explained apart from saying that it is "tricky". I would guess n1=round(log2(s)), so where does 4/3 come from? From later on (eq 4) 3/2 is also feasible, but 4/3 just seems random.
Then, first paragraph of page 4, shouldn't the last term of the equation for n2 be (2^b - 1)/2, i.e. -1 should be out of the exponent, especially since 2^(b-1)/2 is just 2^(b-2)? Again this is not explained, but my thinking is that:
2^b bits are available for representing these many values: 1 for 0, 2*(n1-n2+1) for both signs of the n1-n2+1 whole numbers that are in the range [n2, n1]
So: 2^b = 2*(n1-n2+1)+1, thus n2=n1+1-(2^b-1)/2
Below equation 4, it might be worth saying that 2*beta is the following number in the sequence alpha, beta, 2 beta (apart from when beta=0), in order to understand equation 4 easier.
Also, doesn't it make sense to perform this quantization in the log scale instead of in the linear scale? The elements in P_l are distributed uniformly in the log scale (apart from 0) so it seems more natural to quantize that way i.e. something like w_l = 2^b sign(w_l) when .. for alpha and beta being the consecutive exponents.
We really appreciate your insightful comments and suggestions. In the past 1.5 weeks, we performed comprehensive analyses and experiments accordingly.
Question1: “Can you compare to more existing methods than just to BWN and TWN in Table 4?
For example, a very relevant one is Han et al. 2016. Looking at your Table 1, AlexNet ref 32-bit versus AlexNet 5-bit, the relative size of the networks is 5/32=16%. Comparing this to their figure 6, it seems that at 16% they get 0% accuracy loss.
So, is this method really worth it?”
Our responses are in four folds: (1) following your suggestion, we shall add more existing methods for comparison.
However, we argue that it is not fair to directly compare our INQ with Han et al. 2016, so we compared them in a more fair way as described below. Note that Han et al. 2016 is a hybrid compression solution combining three different techniques, namely network pruning [Han et al. 2015], scalar quantization [Gong et al. 2014] and Huffman encoding.
First, network pruning of Han et al. 2015 can achieve about 9X compression for CNN model like AlexNet, however the compression is mainly obtained from the FC layers, actually its performance on Conv. layers is <3X (as can be seen in the Table 5 of Han et al. 2015).
Besides, Han et al.’s pruning method is run by separately performing pruning and re-training in an iterative way, which is very time-consuming, usually in several weeks or even months. We solved this problem in our recent NIPS work [Guo et al., 2016], obtaining ~7X speed-up in training.
Besides, in our NIPS work, we also improved compression performance of network pruning from 9X to 18X.
Second, in Han et al. 2016, scalar quantization [Gong et al. 2014] further improves the compression from 9X to 27X (as can be seen in the Figure 1 of Han et al. 2016).
However, scalar quantization is mainly proposed to compress parameters in the densely connected layers (i.e., FC layers) of a CNN model, and it will introduce within 1% accuracy loss when using a decent number of clustering centers.
When using it to compress the parameters in the Conv. layers with a relevant compression ratio (e.g., 4X), it will introduce >3% accuracy loss (details can be seen in our responses to following Question 2).
Finally, Huffman encoding only brings little gain (from 27X to 35X) in compression compared with the other two techniques; (2) in this work, our proposed INQ method mainly address the problem of how to efficiently (the whole procedure of our INQ usually costs <1 day for processing AlexNet, VGG-16, GoogleNet and ResNets) convert floating-point weights of all layers of an arbitrary CNN model into either powers of two or zero without accuracy loss, and its performance has been extensively evaluated on almost all existing deep CNN models including recent residual networks which have no FC layer (excluding classification layer), using 5-bit, 4-bit and 3-bit quantization.
Another advantage is the original complex floating-point multiplication operations can be replaced by binary bit shift operations on dedicated hardware like FPGA; (3) taking AlexNet as an instance, when we combing our INQ with our network pruning method [Guo et al., 2016], we achieved much better compression results compared with Han et al. 2016.
Specifically, we can achieve 53X to 89X compression with slightly improved or very similar top-5 and top-1 accuracy, yielding significant margins compared with Han et al.’s work; (4) besides, with our INQ, we also made great progress on developing lossless CNNs with low-precision weights and low-precision activations (details as can be seen in our responses to following Question 2).
Therefore, it is clear that our proposed INQ is definitively worthwhile.
More comparative results will be added to the paper.
Question2: “Why did you stick to weights as powers of 2, why not simply do k-means?”
The reasons are in three folds: (1) with k-means as did in scalar quantization [Gong et al. 2014], the final weight values are still in the floating-point. The advantage of making the weights as either powers of two or zero by our INQ is that the original complex floating-point multiplication operations can be replaced by binary bit shift operations on dedicated hardware like FPGA.
That is, promising speed up can be achieved; (2) k-means is more suitable for compressing parameters in the FC layers of a CNN model, and it will introduce ~1% accuracy loss when using a decent number of clustering centers, such as 16 in the paper of [Gong et al. 2014]. What’s more, we also did related experiments on AlexNet to compare the performance of our INQ and scalar quantization. For scalar quantization (for fair comparison, re-training was also used to enhance its performance), we set the number of cluster centers for each of 5 Conv. layers and 3 FC layers to 32. In the experiment, scalar quantization incurred >3% accuracy loss.
When we changed the number of cluster centers for Conv. layer from 32 to 128, scalar quantization got an accuracy loss of 0. 98%. Compared with our INQ which achieved improved accuracy with 5-bit, 4-bit and 3-bit quantization, we believe that our method is better than scalar quantization; (3) as we described in the conclusion section of our paper, besides low-precision weights, we also plan to make the outputs (i. e. , activations) of all layers of a CNN model into low-precision versions (i. e. , either powers of two or zero).
In this way, all complex floating-point multiplication operations can be replaced by binary bit shift operations both with software and hardware. Fortunately, we have already made great progress in this task. Taking VGG-16 as an instance, with 5-bit weights and 4-bit activations, our INQ achieved improved top-5 and top-1 recognition rates (>1% accuracy gain) compared with original VGG-16 reference with floating-point weights and floating-point activations. To the best of our knowledge, this should be the best results reported so far.
Detailed result comparisons will be added to the paper as the part of the supplementary materials.
Question3: “Regarding weight partitioning - you tried only random partitioning and partitioning based on the absolute value. Doesn't it make sense to also partition based on quantization error? You can just find the set of weights that are most similar to values they would be quantized to, and tune the rest.”
Thanks for your constructive suggestion, our responses are in two folds: (1) the reason why we tried only random partitioning and partitioning based the absolute value of the weight is mainly inspired by our recent network pruning work published in NIPS [Guo et al. 2016]. In our NIPS work, we tried a lot of different solutions on identifying weight importance, and finally found the two solutions we used in this work are the best choices compared with the others.
Thus in this paper, we directly tried random portioning and partitioning based on the absolute value of the weight; (2) we think your suggestion for partitioning with quantization error is insightful, so we made some experiments on AlexNet to explore its potential.
However, until so far we have not obtained good results as our INQ can do. We shall continue to tune it.
Question4: “I'm not completely clear about the formulation that is repeated a few times "5-bit (4 bits for powers of two plus 1 bit for zero)" ? Does it mean that out of the possible 2^5 values that can be expressed with 5 bits, one is a zero, others are +- powers of 2, and one is unused? Or are you talking about some variable length encoding (though in this case the representation wouldn't be 5 bit)?”
Our responses are in two folds: (1) in 5-bit quantization, we use 1 bit to represent zero value only (because we cannot write zero into a power of 2), and another 4 bits to represent the quantized values of powers of two (including sign). Here, the number of the quantized values of powers of two and zero is at most 17 (i.e., 16+1); (2) with 5-bit quantization, taking AlexNet as an instance, the final number of the quantized values of powers of two and zero is 15, 16, 16, 15, 14, 14, 14 and 14 for layers from 1 to 8, respectively.
Specifically, taking 1st and 2nd Conv. layers of Alexnet as examples, the quantized values fall into the sets of {-2^-8(5.04%, this is the accumulated portion), -2^-7(6.56%), -2^-6(9.22%), -2^-5(10.52%), -2^-4(9.75%), -2^-3(4.61%), -2^-2(0.67%), 0(5.51%), 2^-8(5.20%), 2^-7(6.79%), 2^-6(9.99%), 2^-5(11.15%), 2^-4(10.14%), 2^-3(4.26%), 2^-2(0.60%)} and {-2^-8(10.55%), -2^-7(12.09%), -2^-6(13.08%), -2^-5(8.73%), -2^-4(2.70%), -2^-3(0.39%), -2^-2(0.01%), 0(11.30%), 2^-8(9.70%), 2^-7(11.01%), 2^-6(11.05%), 2^-5(6.57%), 2^-4(2.26%), 2^-3(0.53%), 2^-2(0.05%), 2^-1(0.0003%)}, respectively. It is clear that the bit length for every layer of AlexNet is actually less than 5 even though each layer has at most 17 candidate values in 5-bit quantization.
Therefore, for different layers of a CNN model, there will be variable length encoding if such kind of representation is necessary.
Question5:  “I think the region between equation 1 the end of section 2.1 needs some clarifications, though I think I understand it, it required a bit of thinking to understand what is going on. Firt, "bounding the elements in P_l" below equation 1 is a bit confusing, as the elements are bounded just by [-2^n1, +2^n1] (so no dependence on n2).
Second, where does equation 2 come from, it's not explained apart from saying that it is "tricky". I would guess n1=round(log2(s)), so where does 4/3 come from? From later on (eq 4) 3/2 is also feasible, but 4/3 just seems random.
Then, first paragraph of page 4, shouldn't the last term of the equation for n2 be (2^b - 1)/2, i.e. -1 should be out of the exponent, especially since 2^(b-1)/2 is just 2^(b-2)? Again this is not explained, but my thinking is that: 2^b bits are available for representing these many values: 1 for 0, 2*(n1-n2+1) for both signs of the n1-n2+1 whole numbers that are in the range [n2, n1] So: 2^b = 2*(n1-n2+1)+1, thus n2=n1+1-(2^b-1)/2 Below equation 4, it might be worth saying that 2*beta is the following number in the sequence alpha, beta, 2 beta (apart from when beta=0), in order to understand equation 4 easier.
Also, doesn't it make sense to perform this quantization in the log scale instead of in the linear scale? The elements in P_l are distributed uniformly in the log scale (apart from 0) so it seems more natural to quantize that way i.e. something like w_l = 2^b sign(w_l) when .. for alpha and beta being the consecutive exponents.”
We would like to thank your valuable suggestions first. Here, we address your comments one by one as the follows: (1) hyper-parameters n1 and n2 help to bound P_l in the sense that its non-zero elements are constrained to be in the range of either [-2^n1, -2^n2] or [2^n2, 2^n1].
That is, network weights/connections with absolute values smaller than 2^n2 will be pruned away (i.e., setting to zero) in the final low-precision model; (2) factor 4/3 in Equation (2) is set to make sure that all the elements in P_l (including -2^n1 and 2^n1) correspond with the quantization rule defined in Equation (4).
In other words, factor 4/3 in Equation (2) is related to factor 3/2 in Equation (4). If n1 is defined as round(log2(s)), then 2^n1 may be larger than s and at least half of its rounding space will be wasted.
Besides, according to experimental results, we found round(log2(s)) will introduce larger quantization error compared with our proposed method.
For instance, if s=0.354, it will be quantized into 2^-1 by round(log2(s)) and 2^-2 by our Equation (2). Comparatively, the respective quantization error is 0.146 vs 0.104; (3) for the relationship between n1 and n2 described in the first paragraph of page 4, your understanding is correct. Definitively, we shall add more explanatory sentences for a more clear clarification; (4) we fully agree that making quantization in the log scale is worth exploring, but our concern is that it should be a little bit more difficult to implement it and may cause some extra computational overhead.
Thus, we may consider it as part of our future work and report related results after the experimental analyses.
We updated results in the paper, and more related modifications and results will also be added to the paper.
References:
Song Han, Jeff Pool, John Tran, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. ICLR, 2016.
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. NIPS, 2015.
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115, 2014.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. NIPS, 2016.
1. Could you comment on the distribution of weights after quantization?
For instance, for 5 bit quantization, how often did the weights assume the most extreme values?
2. Did you compare quantization to powers of two vs. scalar quantization (uniform spacing in some interval) at the same bit-rate?
Thanks for your valuable comments and suggestions. In the past week, we performed comprehensive analyses and experiments accordingly.
Question1:  “Could you comment on the distribution of weights after quantization?
For instance, for 5 bit quantization, how often did the weights assume the most extreme values?”
We made statistical analyses on the distribution of quantized weights accordingly. Taking AlexNet as an instance, for 5-bit quantization: (1) in the 1st Conv. layer, the quantized weights fall into the set of {-2^-8(5.04%, this is the accumulated portion), -2^-7(6.56%), -2^-6(9.22%), -2^-5(10.52%), -2^-4(9.75%), -2^-3(4.61%), -2^-2(0.67%), 0(5.51%), 2^-8(5.20%), 2^-7(6.79%), 2^-6(9.99%), 2^-5(11.15%), 2^-4(10.14%), 2^-3(4.26%), 2^-2(0.60%)}, that is, the values of {-2^-6, -2^-5, -2^-4, 2^-6, 2^-5, 2^-4} occupy over 60% of all quantized weights; (2) in the 2nd Conv. layer, the quantized weights fall into the set of {-2^-8(10.55%), -2^-7(12.09%), -2^-6(13.08%), -2^-5(8.73%), -2^-4(2.70%), -2^-3(0.39%), -2^-2(0.01%), 0(11.30%), 2^-8(9.70%), 2^-7(11.01%), 2^-6(11.05%), 2^-5(6.57%), 2^-4(2.26%), 2^-3(0.53%), 2^-2(0.05%), 2^-1(0.0003%)}, that is, the values of {-2^-8, -2^-7, -2^-6, -2^-5, 0, 2^-8, 2^-7, 2^-6, 2^-5} occupy over 94% of all quantized weights; (3) the distributions of the quantized weights in the 3rd, 4th and 5th Conv. layers are similar to that of the 2nd Conv. layer.
Besides, more weights are quantized into zero in the 2nd, 3rd, 4th and 5th Conv. layers compared with the 1st Conv. layer; (4) in the 1st FC layer, the quantized weights fall into the set of {-2^-10(8.95%), -2^-9(12.29%), -2^-8(16.48%), -2^-7(10.84%), -2^-6(0.79%), -2^-5(0.002%), 0(8.86%), 2^-10(8.30%), 2^-9(10.51%), 2^-8(12.91%), 2^-7(8.95%), 2^-6(1.12%), 2^-5(0.01%), 2^-5(0.00001%)}, that is, the values of {-2^-10, -2^-9, -2^-8, -2^-7, 0, 2^-10, 2^-9, 2^-8, 2^-7} occupy about 98% of all quantized weights; (5) the distributions of the quantized weights in the 2nd and 3rd FC layers are similar to that of the 1st FC layer. Generally, for AlexNet, the distributions of the quantized weights in the Conv. layers are usually more scattered compared with the FC. layers, this may be partially the reason why it is much easier to get good parameter compression results on FC layers in comparison to Conv. layers when using methods such as SVD, hashing [Chen et al. 2015] and scalar quantization [Gong et al. 2014].
Another thing we want to emphasize here is that, in 5-bit quantization, we use 1 bit to represent zero value only (since we cannot write zero into a power of 2), and another 4 bits to represent the quantized values of powers of two. Here, the number of the quantized values of powers of two and zero is at most 17 (i.e., 16+1), but for the most of layers of a CNN model, it is usually less than 17. Taking AlexNet as an instance, the number of the quantized values of powers of two and zero is 15, 16, 16, 15, 14, 14, 14 and 14 for layers from 1 to 8, respectively.
Detailed analyses on the distribution of weights after quantization will be added to the paper as the part of supplementary material.
Question2: “Did you compare quantization to powers of two vs. scalar quantization (uniform spacing in some interval) at the same bit-rate?”
Our responses are in two aspects: (1) in the paper, we cited scalar quantization [Gong et al. 2014], but we did not compare our proposed INQ with it. The reasons are in two folds.
First, scalar quantization is mainly proposed to compress the parameters in the densely connected layers (i.e., FC layers) of a CNN model. As reported in the paper of Gong et al. 2014, at the best performance, scalar quantization will introduce within 1% accuracy loss when setting 16 centers in clustering.
However, our INQ addresses all layers of a CNN model, and it can achieve lossless compression when using 5-bit, 4-bit and 3-bit for quantization, and it can also achieve competitive accuracy when using 2-bit for ternary quantization (we updated our results in the paper).
Second, the final weights obtained from scalar quantization are still floating-point clustered centers, but in our INQ, the final weights are in the form of either powers of two or zero. Comparatively, the direct advantage of our INQ is that the original floating-point multiplication operations can be replaced by binary bit shift operations on dedicated hardware like FPGA; (2) following your suggestion, we also did related experiments on AlexNet to compare the performance of our INQ and scalar quantization. For scalar quantization (for fair comparison, re-training was also used to enhance its performance), we set the number of cluster centers for each of 5 Conv. layers and 3 FC layers to 32. In the experiment, scalar quantization incurred >3% accuracy loss.
When we changed the number of cluster centers for Conv. layers from 32 to 128, scalar quantization got an accuracy loss of 0.98%. This is consistent with Gong et al.’s paper that scalar quantization is mainly proposed to compress the parameters in the densely connected layers of a CNN model. Compared with our INQ which achieved improved accuracy with 5-bit (also 4-bit and 3-bit) quantization, we believe that our method is better than scalar quantization.
Detailed result comparisons will be added to the paper as the part of the supplementary materials.
References:
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115, 2014.
Wenlin Chen, James T. Wilson, Stephen Tyree, Kilian Q. Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. ICML, 2015.
Nice idea but not complete, model size is not reduced by the large factors found in one of your references (Song 2016), where they go to 5 bits, but this is ontop of pruning which gives overall 49X reduction in model size of VGG (without loss of accuracy). You may achieve similar reductions with inclusion of pruning (or better since you go to 4 bits with no loss) but we should see this in the paper, so at the moment it is difficult to compare
Thanks for your recognition of the novelty of our idea. To your concerns, we could immediately provide some combined result comparison that you are interested in, please see our following responses for details.
Question1:  “Nice idea but not complete, model size is not reduced by the large factors found in one of your references (Song 2016), where they go to 5 bits, but this is ontop of pruning which gives overall 49X reduction in model size of VGG (without loss of accuracy). You may achieve similar reductions with inclusion of pruning (or better since you go to 4 bits with no loss) but we should see this in the paper, so at the moment it is difficult to compare”
Our responses are in five folds:
(1) Actually, we did some related experiments that you are interested using a combination of pruning and quantization.
However, since this paper mainly addresses the problem of how to efficiently convert any floating-point CNN model into a lossless low-precision version whose connection weights are constrained to be either powers of two or zero, we did not put this part of result comparison into the original paper. To the best of our knowledge, with the proposed INQ, we provided comprehensive quantization results on ImageNet using almost all existing deep CNN models including AlexNet, VGG-16, GoogleNet and ResNets.
Another reason why we did not compare our INQ with deep compression [Han et al. 2016] is that network pruning and scalar quantization (i.e., the 1st and 2nd modules in deep compression) do not work well on Conv. layer (details will be described later), however recent GoogleNet and ResNet architectures are fully convolutional (excluding last classification layer);
(2) Since it is not fair to directly compare our INQ with Han et al. 2016, we compared them in a more fair way just as you suggested. Note that Han et al. 2016 is a hybrid compression solution combining three different techniques, namely network pruning [Han et al. 2015], scalar quantization [Gong et al. 2014] and Huffman encoding.
First, network pruning of Han et al. 2015 can achieve 9X compression (as can be seen in the Table 4 of Han et al. 2016) for CNN model like AlexNet, however the compression is mainly obtained from the FC layers. Actually its performance on Conv. layers is <3X (as can be seen in the Table 4 of Han et al. 2016).
Besides, Han et al.’s pruning method is run by separately performing pruning and re-training in an iterative way, which is very time-consuming, it will cost at least several weeks for processing AlexNet. We solved this problem in our recent NIPS work [Guo et al., 2016], obtaining ~7X speed-up in training.
Besides, in our NIPS work, we also improved compression performance of network pruning from 9X to 18X.
Second, in Han et al. 2016, scalar quantization [Gong et al. 2014] further improves the compression from 9X to 27X.
Finally, Huffman encoding brings additional gain (from 27X to 35X) in compression. Taking AlexNet as an instance for comparison, when combing the proposed INQ with our network pruning method [Guo et al., 2016], we achieved much better compression results compared with Han et al. 2016.
Specifically, with 5-bit quantization, we can achieve 53X compression with slightly improved top-5 and top-1 accuracy (i.e., no accuracy loss), yielding 51.43%/96.30% improvement in compression performance compared with Han et al.’s full solution/fair solution (i.e., combination of pruning and scalar quantization only);
(3) We also did related experiments on AlexNet to compare the performance of our INQ and scalar quantization [Gong et al. 2014]. For scalar quantization (for fair comparison, re-training was also used to enhance its performance), we set the number of cluster centers for each of 5 Conv. layers and 3 FC layers to 32. In the experiment, scalar quantization incurred >3% accuracy loss.
When we changed the number of cluster centers for Conv. layers from 32 to 128, scalar quantization got an accuracy loss of 0.98%. This is consistent with Gong et al.’s paper that scalar quantization is mainly proposed to compress the parameters in the densely connected layers (i.e., FC layers) of a CNN model. Compared with our INQ which addresses all layers of a CNN model, and achieved improved accuracy with 5-bit and 4-bit quantization, we believe that our method is much better than scalar quantization;
4) The final weights obtained from scalar quantization [Gong et al. 2014], network pruning [Han et al. 2015] and combined solution [Han et al. 2016] are still floating-point values, but in our INQ, the final weights are in the form of either powers of two or zero. Comparatively, the direct advantage of our INQ is that the original floating-point multiplication operations can be replaced by binary bit shift operations on dedicated hardware like FPGA;
(5) As we described in the conclusion section of our original paper, besides low-precision weights, we also plan to make the outputs (i.e., activations) of all layers of a CNN model into low-precision versions (i.e., either powers of two or zero). In this way, all complex floating-point multiplication operations can be replaced by binary bit shift operations both with software and hardware. Recently, we have already made great progress in this task. Taking VGG-16 as an instance, with 5-bit weights and 4-bit activations, our INQ achieved improved top-5 and top-1 recognition rates (1.64%/1.16 gain in top-1/top-5 accuracy) compared with original VGG-16 reference with floating-point weights and floating-point activations. To the best of our knowledge, this should be the best results so far reported in the field.
Definitively, we will update our paper with more related comparisons if they are necessary.
Hope our above responses are helpful to address your concerns.
References:
Song Han, Jeff Pool, John Tran, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. ICLR, 2016.
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. NIPS, 2015.
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115, 2014.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. NIPS, 2016.
The idea of this paper is reasonable - gradually go from original weights to compressed weights by compressing a part of them and fine-tuning the rest. Everything seems fine, results look good, and my questions have been addressed.
To improve the paper:
1) It would be good to incorporate some of the answers into the paper, mainly the results with pruning + this method as that can be compared fairly to Han et al. and outperforms it.
2) It would be good to better explain the encoding method (my question 4) as it is not that clear from the paper (e.g. made me make a mistake in question 5 for the computation of n2). The "5 bits" is misleading as in fact what is used is variable length encoding (which is on average close to 5 bits) where:
- 0 is represented with 1 bit, e.g. 0
- other values are represented with 5 bits, where the first bit is needed to distinguish from 0, and the remaining 4 bits represent the 16 different values for the powers of 2.
There is a great deal of ongoing interest in compressing neural network models. One line of work has focused on using low-precision representations of the model weights, even down to 1 or 2 bits.
However, so far these approaches have been accompanied by a significant impact on accuracy. The paper proposes an iterative quantization scheme, in which the network weights are quantized in stages---the largest weights (in absolute value) are quantized and fixed, while unquantized weights can adapt to compensate for any resulting error. The experimental results show this is extremely effective, yielding models with 4 bit or 3 bit weights with essentially no reduction in accuracy.
While at 2 bits the accuracy decreases slightly, the results are substantially better than those achieved with other quantization approaches.
Overall this paper is clear, the technique is as far as I am aware novel, the experiments are thorough and the results are very compelling, so I recommend acceptance. The paper could use another second pass for writing style and grammar.
Also, the description of the pruning-inspired partitioning strategy could be clarified somewhat... e.g., the chosen splitting ratio of 50% only seems to be referenced in a figure caption and not the main text.
Thanks to all the reviewers for constructive suggestions and comments. We are really excited that the novelty of our paper has been well recognized.
In this updated version, we carefully considered all reviewers’ suggestions to improve the paper. Generally, we performed four aspects of works: (1) the result comparison of pruning + quantization between our method and Han et al.’s method [1] was incorporated into the paper (please see Section 3.4 for details); (2) the result comparison of weight quantization between our method and vector quantization [2] was also incorporated into the paper (please see Section 3.4 for details); (3) we tried our best to improve the clarifications of our encoding method for weight quantization, definition of bit-width, detailed experimental settings and so on, and several rounds of proof-reading and revising were also conducted; (4) more experimental results (including the statistical analyses on the distribution of weights after quantization and our latest progress on developing INQ for deep CNNs with low-precision weights and low-precision activations) that reviewers may be interested were added to the paper as the supplementary materials.
Moreover, to make our work fully reproducible, the code (along with an instruction manual) will be released to public as we promised in the paper submission.
(1) To reviewer 1:
Question1: “Also, the description of the pruning-inspired partitioning strategy could be clarified somewhat... e.g., the chosen splitting ratio of 50% only seems to be referenced in a figure caption and not the main text.”
Following your suggestion, we added detailed parameter settings (such as splitting ratio and etc.) to the respective sets of experiments described in Section 3 accordingly.
Question 2: “The paper could use another second pass for writing style and grammar.”
Following your suggestion, we tried our best to do a much better work on revising and proof-reading, with the helps from the native colleagues in USA.
(2) To reviewer 2:
Thanks for your recognition of the novelty of our method. We believe that our responses posted on Dec. 16, 2016 should well address your concern on the result comparison of pruning + quantization between our method and Han et al.’s method [1].
Furthermore, detailed result comparisons can be found in Section 3.4 of the paper. It can be clearly seen that our method outperforms Han et al.’s method [1] with significant margins.
(3) To reviewer 3:
Question 1: “1) It would be good to incorporate some of the answers into the paper, mainly the results with pruning + this method as that can be compared fairly to Han et al. and outperforms it.”
Following your suggestion, we incorporated related results into the paper (please see Section 3.4 for details).
Question 2: “It would be good to better explain the encoding method (my question 4) as it is not that clear from the paper (e.g. made me make a mistake in question 5 for the computation of n2).”
Following your suggestion, we revised related parts, especially the clarification of our encoding method based on our previous responses to your questions accordingly (please see Section 2.1 for details).
Question 3: "The "5 bits" is misleading as in fact what is used is variable length encoding (which is on average close to 5 bits) where: - 0 is represented with 1 bit, e.g. 0 - other values are represented with 5 bits, where the first bit is needed to distinguish from 0, and the remaining 4 bits represent the 16 different values for the powers of 2.”
Following your suggestion, we made a clear clarification on the definition of bit-width accordingly.
References:
Song Han, Jeff Pool, John Tran, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. ICLR, 2016.
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115v1, 2014.

Meta-review:
role: PC
data: comment: The paper presents a method for iterative quantization of neural networks weights to powers of 2. The technique is simple, but novel and effective, with thorough evaluation on a variety of ImageNet classification models.

Decision: Accept (Poster)