Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper focuses on the quantization of ConvNets. This paper proposes a learned linear symmetric quantizer to reduce the precision of weight, bias, and activation. The proposed approach works as the following: for a pre-trained neural network, it computes the new weight and activation as a product of a quantized value with a scaling factor. The quantization is based on a simple linear, symmetric function as in equation (1). The value of the scaling factor is searched by "simulated gradient" or exponential moving average during re-training. Next, batch normalization is fused into convolution, and the scaling factor and biases are re-calculated. Last, the scaling factor on the convolution is merged with bias terms, to remove the need for multiplication in hardware implementation. Since bias terms usually have a much larger dynamic range, higher precision is used to represent biases. Experiments show that the method achieves competitive results compared with previous quantization methods, and the quantized models can be deployed on hardware more easily. 

The contribution of the paper, in my opinion, is to show that using simple methods without many bells and whistles, we can achieve competitive quantization performance.  And when performing quantization, it is important to consider the hardware implementation. Details on how to deal with scaling factors, how to deal with biases, and so on, can have significant influences on the overall performance. 

However, the main concern of the paper is that the methods adopted in the paper are too plain. The paper successfully integrated previous methods but did not propose new ideas that inspire future research. As a result, I would not recommend acceptance for publication. 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a linear symmetric quantizer for integer accelerators called LLSQ, which learns the quantization scaling factor using simulated gradient as update policy. Their main contribution is enabling inference on integer-only hardware by covering all parameters of all operators in convolutional networks, including weight, bias, activation and scaling factor. To address the quantization noise issue in bias parameters, they adopt Straight-Through Estimator and fine-tune the parameters after quantization. To improve inference efficiency, they apply BN layer fusion. They conduct experiments on public datasets for image classification and object detection to conclude that LLSQ achieves lower accuracy degradation compared to previous work. Finally, they test the quantized model on a specialized integer accelerator, showing the feasibility of the quantization on real hardware.

In conclusion, this paper generalizes linear symmetric quantization to all parameters in order to deploy the network on specialized integer neural network processors for efficient inference. In terms of algorithmic contribution, this paper introduces the scaling factor as a learnable parameter of the quantizer, but lacks enough theoretical justification. Therefore, I would consider weakly accepting the paper.

For the algorithm, the following should be addressed.
1.	In scaling factor updating policy, the simulated gradient performs better compared to EMA. However, the motivation for choosing this policy is unclear, and there is no mathematical derivation of the gradient value.
2.	In this study, pretrained network weights in full precision are used, thus the quantization procedure should take fixed weights as inputs. However, in section 3.3, the quantization scaling factor is updated in the training phase of the network, with weights being updated at the same time This seems to be a contradiction, and the experiment results where quantizer is trained with network weights fixed should also be presented.

For the experiment, the following should be addressed.
1.	The training time for different bit-width settings should be included in the results.
2.	The reason for leaving the first and last layers in full precision is unclear, and doing so may go against the objective of deploying the quantized model on specialized integer hardware according to the paper.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The submission proposes to train a linear symmetric quantizing function for integer processors.

The proposed idea makes sense at a high level, and the empirical results look somewhat compelling, but the write-up is not particularly clear (see clarity-related comments). I found it hard to extract a complete picture of how the proposed approach operates: from the leftmost diagram in Figure 1 I can infer what high-level steps are involved, but I wouldn’t know how to re-implement the approach from the textual description itself.

Clarity-related comments:

- The submission never explicitly states what x^r and p(x^r) correspond to in the network. My understanding from the context is that x^r is the scalar value of a model parameter or activation, and that p(x^r) is the empirical distribution over all model parameters and activations. Can the authors clarify?
- Some kind of pseudo-code for the re-training of the weights and activations (Section 3.3) would help clear up reader confusion. At this point, are only the weights and activations quantized in the network? How is backpropagation handled (and if the straight-through estimator is used, why is it only mentioned in Section 3.5)? How do parameter and alpha updates interleave?
- How many alpha values are there? Section 3.3 gives the impression that there is a single quantization parameter shared by all parameters and activations, but Equation 3 uses different alpha values for the weights and activations.
- Section 3.4 is difficult to parse due to bad notation. If the output of the convolution layer is fed into the batch normalization layer, it would be clearer to reuse symbols (i.e. change “x” to “o” in Equation 4).

Additional comments:

- Organization of the different sections could be improved. Related work discussion is scattered throughout three different sections, namely Introduction, Motivation, and Related Work.
- Writing quality could be improved  (e.g., “*Except that*, some designs rely [...]”, “Edge or embedded neural network accelerators *are generally having* three primary design goals [...]”) but has a relatively small negative impact on readability.
review_title: Official Blind Review #3

Meta-review: This paper considers the question of how to quantize deep neural networks, for processors operating on low-precision integers.  The authors propose a methodology and have evaluated it thoroughly. The reviewers all agree that this question is important in practice, though there was disagreement about how novel a contribution this paper is specifically, and on its clarity. The clarity questions were resolved on rebuttal, so I lean to accepting the paper.
Decision: Accept (Poster)