Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The motivation of the paper is to be able to train low precision networks to a high-accuracy. Quantization is a useful tool in model compression, and doing it well for very low-precision models (2-3 bit precision specifically), is challenging.

The main contribution of the paper comes from:
a) Step Size Gradient: They propose a gradient which is sensitive to the distance between the value and the transition point. This is different from other methods which have gradients dependent only on the clip point.
b) Step Size Gradient Scale: This is an interesting contribution, where they try to match the ratio of average update of the step size ‘s’ and average magnitude of ‘s’, with that of the network weights. This leads them to scale the gradient according to the precision and number of parameters. They demonstrate that this scaling actually helps improve the accuracy.

The results for 8-bit precision are not new. Several results (Quantization and Training of Neural Networks for Efficient
Integer-Arithmetic-Only Inference, Jacob et al., Quantizing deep convolutional networks for
efficient inference: A whitepaper, Krishnamoorthi et al.), show 8-bit quantization results where the accuracy matches floating point accuracy, and in some case exceeds it (low precision quantization acting as a regularizer). However, the results for lower precision are impressive.

There are a few questions:
1. In sec 2.1, you mention that ‘each layer of weights and activations has a distinct step size, represented as an fp32 value, initialized to …’. Can you explain the intuition behind the initial value of the step size, and how is it a function of v?
2. ‘Model Compression via Distillation and Quantization’ (Polino et al.) shows distillation actually helps significantly improve accuracy. I wonder if the authors have tried different weight combinations for the distillation loss, and using bigger models as teacher models.
3. I would like to get more details of the inference setup, specifically the size and inference latency improvements over full-precision networks. The practical applicability of low-precision networks, specifically 2-bit and 3-bit networks, equally depends on the inference infrastructure, as it does on the training improvements.
4. Have you evaluated your method for a non-Vision usecase?

Overall this is a good work, I would tend towards accepting this.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: 
This paper trains low-precision network with quantized weights and quantized activation. The main idea is to split the scale and quantized values. Both scales and weights are updated with backprop and SGD. The paper presents excellent experimental results on ImageNet. 

The paper is generally well written and easy to follow. However, there does exist quite some grammar errors, especially in abstract, which could be improved. 

Moreover, I would like the authors to clarify some technical details. Are the scales s, so called step size in the paper, for every weight, every convolutional kernel, or very layer? How do you deal with BatchNorm?

What is the main benefits of the proposed quantization method in general? Is it for fast inference, fast training, or just memory compression? Do the authors see the real benefits in practice besides claiming the accuracy does not drop?

I would suggest the authors discuss and compare with XNOR network in detail. The proposed method looks similar. 

I am wondering how the baseline methods are tuned. There are quite a few “tricks” like learning rate scheduler and weight decay, which I do consider them as contributions of the paper. But would baseline methods also benefit from more hyper-parameter tuning?

Minor issue, I donot get the explanation of eq (4), and it looks rather unnecessary. It sounds to me starting from a trained network and then train 90 epochs is a rather long time. Could the authors convince me this is a standard setting by providing some reference?


================ after rebuttal=========================
Thank the authors for reply. My rating does not change. The proposed does look similar to XNOR, and the only difference seems to be how the scales are updated. Since there is only one scale per layer, I will be quite surprised if the proposed method can be much better than XNOR. Moreover, since BatchNorm is not quantized and it is everywhere in a ResNet-like architecture, it surprises me how much the scale helps. Finally, I am worried about practical benefits towards the authors' claim because the networks are not fully quantized. 

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The paper is concerned with neural networks using low-precision operations. This is an important research are both for applications (e.g., deploying neural networks on embedded systems or other constrained hardware, or increasing throughput on a production system) and for theoretical reasons (e.g., potentially modelling the behavior of biological neural networks).

The paper makes two primary contributions: (1) It proposes a new approximation for the gradient of the quantized data with respect to the step size parameter (equation 3) based on the well-known straight through estimator. (2) It proposes a heuristic for scaling gradient updates based on the number of weights in a given layer and the quantization levels (clipping thresholds). Experimental results showing higher accuracy than previously reported approaches in a number of settings and for common metrics, and best-to-date results for other settings.

I recommend the paper for publication.

In terms of improving the paper further, the authors could expand their experiment section. E.g., how does their method compare to current SOTA knowledge-distillation methods; how does it deal with other architectures (recurrent models, attention, etc).

Minor notes on the manuscript:

In section 3.4, both the term “population size” as well as the symbol N_W come without any definition. While closer reading makes it clear what is meant is the number of weights per layer, a definition closer to their first usage would be good. This goes in particular for N_W which comes in several versions (N_W, N, nweights).

In section 3.7, the sentence starting with “We found that using this approach to distill” does not seem to fit within its context for this reader, as it describes “full-to-full” distillation. Perhaps it is meant to read “low precision student network”?
review_title: Official Blind Review #3

Meta-review: Main content: Paper is about training low precision networks to a high-accuracy.

Discussion:
reviewer 2: impressive results, main questions are around some clarity in the experiments tried, but sounds like authors addressed most of this in rebuttal.
reviewer 1: well written paper, but authors think some technical details could be clarified. 
reviewer 3:  well written but experimental section could be improved.
Recommendation: all reviewers are in consensus, well written paper but some experiments/technical details could be improved. i vote poster.
Decision: Accept (Poster)