Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a novel neural network compression technique.
The goal is to compress maximally the network specification via parameter quantisation with a minimum impact on the expected loss.
It assumes pruning of the network parameters has already been performed, and only considers the quantisation of the individual scalar parameters of the network.
In contrast to previous work (Han et al. 2015a, Gong et al. 2014) the proposed approach takes into account the effect of the weight quantisation on the loss function that is used to train the network, and also takes into account the effect on a variable-length binary encoding of the cluster centers used for the quantisation. 

Unfortunately, the submitted paper is 20 pages, rather than the 8 recommended. The length of the paper seems unjustified to me, since the first three sections (first five pages) are very generic and redundant can be largely compressed or skipped (including figures 1 and 2). Although not a strict requirement by the submission guidelines, I would suggest the authors to compress their paper to 8 pages, this will improve the readability of the paper.

To take into account the impact on the network’s loss the authors propose to use a second order approximation of the cost function of the loss. In the case of weights that originally constitute a local minimum of the loss, this leads to a formulation of the impact of the weight quantization on the loss in terms of a weighted k-means clustering objective, where the weights are derived from the hessian of the loss function at the original weights.
The hessian can be computed efficiently using a back-propagation algorithm similar to that used to compute the gradient, as shown in cited work from the literature. 
The authors also propose to alternatively use a second-order moment term used by the Adam optimisation algorithm, since it can be loosely interpreted as an approximate Hessian. 

In section 4.5 the authors argue that with their approach it is more natural to quantise weights across all layers together, due to the hessian weighting which takes into account the variable impact across layers of quantisation errors on the network performance. 
The last statement in this section, however, was not clear to me: 
“In such deep neural networks, quantising network parameters of all layers together is more efficient since optimizing layer-by-layer clustering jointly across all layers requires exponential time complexity with respect to the number of layers.”
Perhaps the authors could elaborate a bit more on this point?

In section 5 the authors develop methods to take into account the code length of the weight quantisation in the clustering process. 
The first method described by the authors (based on previous work), is uniform quantisation of the weight space, which is then further optimised by their hessian-weighted clustering procedure from section 4. 
For the case of nonuniform codeword lengths to encode the cluster indices, the authors develop a modification of the Hessian weighted k-means algorithm in which the code length of each cluster is also taken into account, weighted by a factor lambda. Different values of lambda give rise to different compression-accuracy trade-offs, and the authors propose to cluster weights for a variety of lambda values and then pick the most accurate solution obtained, given a certain compression budget.  

In section 6 the authors report a number of experimental results that were obtained with the proposed methods, and compare these results to those obtained by the layer-wise compression technique of Han et al 2015, and to the uncompressed models. 
For these experiments the authors used three datasets, MNIST, CIFAR10 and ImageNet, with data-set specific architectures taken from the literature. 
These results suggest a consistent and significant advantage of the proposed method over the work of Han et al. Comparison to the work of Gong et al 2014 is not made.
The results illustrate the advantage of the hessian weighted k-means clustering criterion, and the advantages of the variable bitrate cluster encoding.  

In conclusion I would say that this is quite interesting work, although the technical novelty seems limited (but I’m not a quantisation expert).
Interestingly, the proposed techniques do not seem specific to deep conv nets, but rather generically applicable to quantisation of parameters of any model with an associated cost function for which a locally quadratic approximation can be formulated. It would be useful if the authors would discuss this point in their paper.

review_title: interesting experimental evaluation of variable bit-rate CNN weight compression scheme

final_score: rating: 7: Good paper, accept
review_content: The paper has two main contributions:

1) Shows that uniform quantization works well with variable length (Huffman) coding

2) Improves fixed-length quantization by proposing the Hessian-weighted k-means, as opposed to standardly used vanilla k-means. The Hessian weighting is well motivated, and it is also explained how to use an efficient approximation "for free" when using the Adam optimizer, which is quite neat. As opposed to vanilla k-means, one of the main benefits of this approach (apart from improved performance) is that no tuning on per-layer compression rates is required, as this is achieved for free.

To conclude, I like the paper: (1) is not really novel but it doesn't seem other papers have done this before so it's nice to know it works well, and (2) is quite neat and also works well. The paper is easy to follow, results are good. My only complaint is that it's a bit too long.

Minor note - I still don't understand the parts about storing "additional bits for each binary codeword for layer indication" when doing layer-by-layer quantization. What's the problem of just having an array of quantized weight values for each layer, i.e. q[0][:] would store all quantized weights for layer 0, q[1][:] for layer 1 etc, and for each layer you would have the codebook. So the only overhead over joint quantization is storing the codebook for each layer, which is insignificant. I don't understand the "additional bit" part. But anyway, this is really not a important as I don't think it affects the paper at all, just authors might want to additionally clarify this point (maybe I'm missing something obvious, but if I am then it's likely some other people will as well).


review_title: Effective quantization

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a network quantization method for compressing the parameters of neural networks, therefore, compressing the amount of storage needed for the parameters.  The authors assume that the network is already pruned and aim for compressing the non-pruned parameters. The problem of network compression is a well-motivated problem and of interest to the ICLR community. 

The main drawback of the paper is its novelty. The paper is heavily built on the results of Han 2015 and only marginally extends Han 2015 to overcome its drawbacks. It should be noted that the proposed method in this paper has not been proposed before. 

The paper is well-structured and easy to follow. Although it heavily builds on Han 2015, it is still much longer than Han 2015. I believe that there is still some redundancy in the paper. The experiments section starts on Page 12 whereas for Han 2015 the experiments start on page 5. Therefore, I believe much of the introductory text is redundant and can be efficiently cut. 

Experimental results in the paper show good compression performance compared to Han 2015 while losing very little accuracy. Can the authors mention why there is no comparison with Hang 2015 on ResNet in Table 1?

Some comments:
1) It is not clear whether the procedure depicted in figure 1 is the authors’ contribution or has been in the literature.
2) In section 4.1 the authors approximate the hessian matrix with a diagonal matrix. Can the authors please explain how this approximation affects the final compression? Also how much does one lose by making such an approximation?

minor typos (These are for the revised version of the paper):
1) Page 2, Parag 3, 3rd line from the end: fined-tuned -> fine-tuned
2) Page 2, one para to the end, last line: assigned for -> assigned to
3) Page 5, line 2, same as above
4) Page 8, Section 5, Line 3: explore -> explored
review_title: Review for "Towards the Limit of Network Quantization"

final_score: rating: null
review_content: The paper analyzes the impact of quantization of the NN performance. Have you considered the impact of re-training part of the networks with quantized parameters? 

I ask this because Quantization typically leads to zero gradients, but for some particular designs (which might be included in the proposed formulation, which is presented in a quite general setup), re-training after quantization part of the network is quite beneficial. See, for instance, in a relatively shallow setup, "Image classification with the fisher vector: Theory and practice", Sanchez et al., IJCV 2013. I don't see how these possible interactions between (re-)training and quantization are handled by the proposed formulation. Maybe it would be good to clarify this point.  
review_title: Impact of (re-)training on performance

final_score: rating: null
review_content: 1) For the case of Huffman coding we see that Uniform quantization works very well actually compared to the Iterative ECSQ. Clarification - end of section 5.3 "Note that one can use Hessian-weighted mean..." - so what is used in the end, Hessian weighted mean or just a non-weighted mean? I would like to see a comparison of the two. If uniform quantization with non-weighted mean works, then this part of the paper boils down to contain no contribution as this method is straight-forward and already exists.

2) I don't understand the focus on storing parameters in one gigantic array or storing them individually for each layer. Isn't this just an implementation detail and for the sake of argument we can always just pretend they are in one gigantic array?

Other comments
- I think the paper is way too long for what it is. ICLR doesn't have a page limit but strongly suggests 8 pages while this is 17 pages long. Plenty of places where things can be summarized more efficiently, e.g. just beofre 6.1 seems like duplicated text of 6.1, many places keep saying the same thing like that it is assumed that pruning is performed first, etc.
- I would emphasise more the fact that this hessian weighting allows you to not tune per-layer compression rates as it's one of the main positives of the paper, it is explained in the text but it is not mentioned neither in the abstract nor in the conclusions.

review_title: Various

Meta-review: The paper proposes using quantization schemes to compress the weights of a neural network. The paper carries out a methodical study of first deriving the objective function for optimizing the quantization, and then uses various quantization schemes. Experiments show competitive performance in terms of compression and accuracy tradeoff.
 
 I am happy to go with the reviewers' recommendations to accept the paper.
 
 A minor comment:
 It is important to mention other frameworks that compress neural networks, e.g. 
 https://arxiv.org/abs/1509.06569
 Although the weights are not quantized above, the number of parameters is reduced.
 Similarly there are other works looking into network compression. It will be good to mention them.
Decision: Accept (Poster)