Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: I couldn't follow the method described in the paper. The authors are basically trying to address post-training quantization by perturbing the the weights of a trained DNN. The goal is to perturb the weights so that the quantized DNN will behave similar to the original full-precision DNN. The authors draw a link between this optimization problem and optimizing for the "reconstruction" of the output activations of a block (see Equation 7). The technique BRECQ, shown in Algorithm 1, is basically to optimize the perturbation of the weights for the right hand side of Equation 7 for each block of a DNN.

The method only seems to work on ReLU networks, so it's restricted to CNNs, but this is still extremely useful.

The experimental results are very strong. BRECQ is the first to achieve 4-bit integer post-training weight quantization that nearly matches the fp32 baseline on ImageNet. Even at 2-bit, BRECQ can often get within 5% of the baseline while other methods leave the model with inoperable loss of accuracy.

I couldn't pinpoint any major flaws in the paper, and the results are extremely impressive.
review_title: Couldn't understand paper, results seem strong

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Post-training quantization is an important problem, especially for industry. This paper leverages the basic building blocks and conducts a block-wise quantization. Nice results are obtained with the proposed method. 

The proposed method is cheap to implement and pushes the post-training quantization to 2-bit. 
The measurement problem of mixed precision literature raised in this paper is of insights. This problem may inspire the community to find a better measurement in future work. 

Extensive experiments on various methods (handcrafted and designed by NAS), various tasks (classification, detection), various configurations (different bits, latency, model size) are impressive. Various baselines are also included to make the results stronger. 


Questions:
The block-diagonal scheme is selected according to experimental results. Is it possible to visualize the real Hessian of stage-wise settings? (it may be impossible to the full Hessian matrix for the whole network). If we see a few non-zero elements at the off-diagonal for the block-wise setting, this choice can be better motivated. 

review_title: Interesting Method and Solid Results

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes BRECQ which is a new Post Training Quantization (PTQ) method. The goal of the paper is to push the limit of PTQ to low bit precision (INT2). They try to address this by considering both inter and intra-layer sensitivity to find the best update to the model parameters so that the output from a block is minimally changed/perturbed. Furthermore, the authors also consider mixed precision quantization setting.

Empirical results are shown for multiple NNs for image classification and object detection.


While the paper is trying to address an interesting problem and there are a lot of empirical results, but I had a very hard time to follow the paper's main idea and the "final" proposed algorithm. It would be great if the authors could answer the questions below and I will reconsider my score accordingly.


- What is the final algorithm? The data provided in Algorithm 1 is very vague and there are no equations. It is not clear if you are actually using any second-order information since based on Eq 11 it seems that the proposed method is only using the gradient, and not the second order information. If so are the discussion from page 2-4 necessary?

- It is not clear if the equality given in Eq 11 is correct. With the softmax layer the Hessian diagonal would not be the same as gradient^2.


- Why not use the change in loss in Eq 4 (left) instead of using a second-order Taylor series? Second-order Taylor series is an approximation and does not hold for large perturbations. Also when we have access to evaluating the change in loss, why do we even need to focus on measuring the second order term? This does not seem to be correct/necessary at least for uniform quantization.

- An ablation study is actually needed for the above.

- What is the experimental setting for the ablation study in Table 1? while not mentioned, from the accuracy it seems the results are on ImageNet but this is not mentioned. Did the authors consider other bit precision settings?


- In page 3, it is stated that using the Hessian directly is not possible due to memory and computational overhead. However, you can use matrix free methods which do not require forming the Hessian explicitly.
review_title: Paper Not Clear

final_score: rating: 7: Good paper, accept
review_content: This paper explores the post-training inference quantization. Based on second-order quantization error analysis, it proposes to reconstruct quantized model in a block level to achieve SOTA accuracy for INT2 weight quantization, which distinguish this paper from previous reported layer-wise reconstruction approach. The proposed approach is intuitive and supported by extensive experiments across a wide range of image classification and object detection tasks.

Overall, this paper is well written. The idea is straightforward but comes from a detailed theoretical analysis and supported by strong experimental results. The authors also proposed solutions to approximate pre-activation Hessian by using Fisher Information Matrix which sounds reasonable and is easy to implement. The authors did comprehensive comparison with SOTA approaches, not just PTQ, but also quantization aware training and mix-precision frameworks.  


I have some questions wish to be clarified.

1). It is not clear to me, in the reconstruction process, how does the short-cut (e.g. in ResNet) are handled.  

2). In the error analysis, batch normalization layers are not considered. Batch norm will have direct impact on the output activation, hence the quantization bias term. Plus, batch norm statistics will be changed during the reconstruction phase. Could the authors comment on the impact of batch norm layers?

3). The authors use 1024 training samples to do the reconstruction. What happens when more samples are used? What determines the number of samples needed?

4). The data presented seems to focus mainly on weight quantization, while leaving activation in relative higher precision. What happens when quantize activations in 2 bits together with the weights in this approach? What are the challenges?

5). It would have been nice to verify this approach on tasks other than vision, such as speech and NLP tasks.

6). Seems a format error on one of the reference: Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. {SINGLE} {path} {one}-{shot} {neural} {architecture} {search} {with} {uniform} {sampling}, 2020. URL https://openreview.net/forum?id=r1gPoCEKvH. 

review_title: A good paper on post-training inference quantization

Meta-review: This paper proposes a new method for post-training quantization, achieving very good results. After the author's response, all the reviewers were positive. There were some issues regarding clarity, and about explaining why the methods work better than just optimizing the loss, but I think the reviewers were eventually satisfied.  Following some info after the author's response phase, I'll just ask the authors to verify their published code works with publicly available PyTorch packages, so their method could be easily used.
Decision: Accept (Poster)