Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Revised Review:

The authors have addressed most of my concerns with the revised manuscript. I now think the paper does just enough to warrant acceptance, although I remain a bit concerned that since the benefits are only achievable with customized hardware, the relevance/applicability of the work is somewhat limited.

Original Review:

The paper proposes a technique for quantizing the weights of a neural network, with bit-depth/precision varying on a per-parameter basis. The main idea is to minimize the number of bits used in the quantization while constraining the loss to remain below a specified upper bound. This is achieved by formulating an upper bound on the number of bits used via a set of "tolerances"; this upper bound is then minimized while estimating any increase in loss using a first order Taylor approximation.

I have a number of questions and concerns about the proposed approach. First, at a high level, there are many details that aren't clear from the text. Quantization has some bookkeeping associated with it: In a per-parameter quantization setup it will be necessary to store not just the quantized parameter, but also the number of bits used in the quantization (takes e.g. 4-5 extra bits), and there will be some metadata necessary to encode how the quantized value should be converted back to floating point (e.g., for 8-bit quantization of a layer of weights, usually the min and max are stored). From Algorithm 1 it appears the quantization assumes parameters in the range [0, 1]. Don't negative values require another bit? What happens to values larger than 1? How are even bit depths and associated asymmetries w.r.t. 0 handled (e.g., three bits can represent -1, 0, and 1, but 4 must choose to either not represent 0 or drop e.g. -1)? None of these details are clearly discussed in the paper, and it's not at all clear that the estimates of compression are correct if these bookkeeping matters aren't taken into account properly.

Additionally the paper implies that this style of quantization has benefits for compute in addition to memory savings. This is highly dubious, since the method will require converting all parameters to a standard bit-depth on the fly (probably back to floating point, since some parameters may have been quantized with bit depth up to 32). Alternatively custom GEMM/conv routines would be required which are impossible to make efficient for weights with varying bit depths. So there are likely not runtime compute or memory savings from such an approach.

I have a few other specific questions: Are the gradients used to compute \mu computed on the whole dataset or minibatches? How would this scale to larger datasets? I am confused by the equality in Equation 8: What happens for values shared by many different quantization bit depths (e.g., representing 0 presumably requires 1 bit, but may be associated with a much finer tolerance)? Should "minimization in equation 4" refer to equation 3?

In the end, while do like the general idea of utilizing the gradient to identify how sensitive the model might be to quantization of various parameters, there are significant clarity issues in the paper, I am a bit uneasy about some of the compression results claimed without clearer description of the bookkeeping, and I don't believe an approach of this kind has any significant practical relevance for saving runtime memory or compute resources. 
review_title: Clarity issues; concerns about practical relevance

final_score: rating: 6: Marginally above acceptance threshold
review_content: I have read the responses to the concerns raised by all reviewers. I find the clarifications and modifications satisfying, therefore I keep my rating of the paper to above acceptance threshold.

-----------------
ORIGINAL REVIEW:

The paper proposes a method for quantizing neural networks that allows weights to be quantized with different precision depending on their importance, taking into account the loss. If the weights are very relevant, it assigns more bits to them, and in the other extreme it does pruning of the weights.

This paper addresses a very relevant topic, because in limited resources there is a constrain in memory and computational power, which can be tackled by quantizing the weights of the network. The idea presented is an interesting extension to weight pruning with a close form approximate solution for computing the adaptive quantization of the weights.

The results presented in the experimental section are promising. The quantization is quite cheap to compute and the results are similar to other state-of-the-art quantization methods. 
From the tables and figures, it is difficult to grasp the decrease in accuracy when using the quantized model, compared to the full precision model, and also the relative memory compression. It would be nice to have this reference in the plots of figure 3.  Also, it is difficult to see the benefits in terms of memory/accuracy compromise since not all competing quantization techniques are compared for all the datasets.
Another observation is that it seems from figure 2 that a lot of the weights are quantized with around 10 bits, and it is not clear how the compromise accuracy/memory can be turned to less memory, if possible. It would be interesting to know an analogy, for instance, saying that this adaptive compression in memory would be equivalent to quantizing all weights with n bits.

OTHER COMMENTS:

-missing references in several points of the paper. For instance, in the second paragraph of the introduction, 1st paragraph of section 2.

- few typos:
*psi -> \psi in section 2.3
*simply -> simplify in proof of lemma 2.2
*Delta -> \Delta in last paragraph of section 2.2
*l2 -> L_2 or l_2 in section 3.1 last paragraph.
review_title: The paper proposes a method for quantizing neural networks that allows weights to be quantized with different precision depending on their importance

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors present an interesting idea to reduce the size of neural networks via adaptive compression, allowing the network to use high precision where it is crucial and low precision in other parts. The problem and the proposed solution is well motivated. However, there are some elements of the manuscript that are hard to follow and need further clarification/information. These need to definitely be addressed before this paper can be accepted.

Specific comments/questions:
- Page 1: Towards the bottom, in the 3rd to last line, reference is missing.
- Page 1: It is a little hard to follow the motivation against existing methods.
- Page 2: DenseNets and DeepCompression need citations
- Lemma 2.1 seems interesting - is this original work? This needs to be clarified.
- Lemma 2.2: Reference to Equation 17 (which has not been presented in the manuscript at this point) seems a little confusing and I am unable to following the reasoning and the subsequent proof which again refers to Equation 17.
- Alg 2: Should it be $\Delta$ or $\Delta_{k+1}$? Because in one if branch, we use $\Delta$, in the other, we use the subscripted one.
- Derivation in section 2.3 has some typographical errors.
- What is $d$ in Equation 20 (with cases)? Without this information, it is unclear how the singular points are handled.
- Page 6, first paragraph of Section 3: The evaluation is a little confusing - when is the compression being applied during the training process, and how is the training continued post-compression? What does each compression 'pass' constitute of?
- Figure 1b: what is the 'iteration' on the horizontal axis, is it the number of iterations of Alg3 or Alg2? Hoping it is Alg3 but needs to be clarified in the text.
- Section 3: What about compression results for CIFAR and SVNH? 
review_title: Interesting intuitive idea; evaluation needs more clarification

Meta-review: Given the changes to the paper, the reviewers agree that the paper meets the bar for publication at ICLR. There are some concerns regarding the practical impact on CPUs and GPUs. I ask the authors to clearly discuss the impact on different hardware. One can argue if adaptive quantization techniques are helpful, then there is a chance that future hardware will support them. All of the experiments are conducted on toy datasets. Please consider including some experiments on Imagenet as well.
Decision: Accept (Poster)