Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies the effect of quantization during training together with batch normalization in quantized deep neural networks. The compound effect of convolution and batch normalization on the dynamic range of activations has implications on the progress of training. The authors propose a protocol for training a quantized neural network combining filter pruning, fine tuning and bias correction. The experiments show that the model size is reduced significantly, while keeping or improving the accuracy.

Strengths
- The paper is clearly presented, and the effect of batch normalization in the dynamic range of a quantize layer is interesting.
- The method, in the paper settings, is effective in reducing the model size and sometimes improving accuracy.

Weaknesses
- There is not qualitative analysis of the problem of the dynamic range and how the proposed method alleviates it. 
- The novelty is limited in my opinion. The paper combines typical practices used in other works. For example, pruning filters with small magnitude is a common practice. it is also common to fine tune a network after pruning or compression, which not only reduces model size but often also improves the accuracy due to lower overfitting. And bias correction to account for the reconstruction error is also commonly used in similar cases (e.g. DFQ, Finkelstein2019, Masana2017). 
- The number of iterations in the experiments is a fixed number. It would be more convincing using a validation set with early stopping.
- The pruning+fine tuning seems to be done once (twice in the proposed workflow). In that case the comparison may not be fair without several iterations of pruning+fine tuning, to compare models when they cannot be further pruned. 

Overall, I think the novelty of the paper is limited, with concerns about the experiments.

Questions
Please address weaknesses

Finkelstein et al., Fighting Quantization Bias With Bias, ECV@CVPR 019 
Masana et al., Domain-adaptive deep network compression, ICCV2017

-- Post rebuttal

I appreciate the response by the authors and the new experiments. I also read the other reviews and responses. I think the paper has improved in the revised version. However, I'm still concerned about the novelty, which still remains relatively incremental, as also pointed by other reviewers. I update my rating to 5.
review_title: Effective idea with limited novelty

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposed to prune certain channels to improve the accuracy of quantized DNN model. The motivation comes from the observation that the channels which have small variance are actually harmful to the quantization-aware training. The authors show that these channels with small variance can be pruned without significant influence to the neural network, and the accuracy loss can be easily recovered with fine-tuning. 

Pros:

This paper makes an important observation that channels with small variance lead to a problem that the filter weights will have a wide range after fusing the BN layer into convolution. Directly performing quantization on the weight tensor will lead to a large quantization scale and large quantization error.

The authors find that the channels with small variance can actually be removed. Based on their following BN operations, these channels can be seen as constant channels approximately, and these constants can be folded into the bias of the next layer. By this way, the neural network is not only more suitable to quantize, but also has less model size and computation.

The paper gives a study showing that removing the channels with small variance improves the accuracy of quantized model. The proposed method is also compared with several other quantization methods.

Cons:

On ImageNet, the experiments are only did on MobileNetV2, and only a few methods are compared. Although the current experiments show the benefits of PfQ, it is better to have more experiments to make the validation part more solid.

Table 1 is to show the benefits of removing disturbing weights. Why use the activation quantization instead of weights quantization? It's better to analyze weight quantization here too.

There are some questions which are not clear in the paper:

1. Will per-channel quantization have the problem discussed in this paper? What about the results of using / not using PfQ for per-channel quantization? The min-max quantization scale (eq 20) is used in this paper, what if we use learnable quantization (e.g., LSQ [1])? Is it still suffer from the channels having small variance?

2. DFQ is used as the quantization method. Is this a quantization method cited in this paper? I didn't find the reference and the full name for DFQ.

3. Is the proposed method only applied to DNNs which have BN layers? Is it possible to apply the proposed method to models without BN layers?

Other comments:
It's easier for people to understand the key idea of this paper if moving Figure 1 to the first few pages.

In general I think this paper is a good work on quantization and model compression. It makes good observation of factors that influence quantization, and shows that pruning can actually help quantization in some cases. There are some issues on the experiment (See cons), it's better to validate the proposed method on more DNN architectures and compare with more quantization methods.

References

[1] Esser, S.K., McKinstry, J.L., Bablani, D., Appuswamy, R. and Modha, D.S., 2019. Learned step size quantization. arXiv preprint arXiv:1902.08153.
review_title: borderline, tend to accept

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Overall
This work present a Pruning mechanism for Quantization scenario. Duo to the low-bits effects, the quantized network is hard to train properly. Therefore, authors provide a new method call Pruning for Quantization (PfQ) and a workflow to solve the model compression problem practically. Comparing to some current quantization methods, PfQ obtains some gain from solution to the performance on benchmark datasets such as ImageNet and CIFAR100.

### Pros
Clear idea about PfQ and the mathematical results provide the reason why the authors want to do so.

### Cons
1) Generally, the joint quantization (Q) and pruning (P) framework is not novel at present. For example, the following papers have tried to solve the model compression problem with P and Q jointly:
Tung, Frederick, and Greg Mori. "Clip-q: Deep network compression learning by in-parallel pruning-quantization." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
Tung, Frederick, and Greg Mori. "Deep neural network compression by in-parallel pruning-quantization." IEEE transactions on pattern analysis and machine intelligence (2018).
Wang, Ying, Yadong Lu, and Tijmen Blankevoort. "Differentiable Joint Pruning and Quantization for Hardware Efficiency." European Conference on Computer Vision. Springer, Cham, 2020.
And compression with the BN mechanisms are also available:
Gao, Xitong, et al. "Dynamic channel pruning: Feature boosting and suppression." arXiv preprint arXiv:1810.05331 (2018).
Liu, Yuan, et al. "Local Normalization Based BN Layer Pruning." International Conference on Artificial Neural Networks. Springer, Cham, 2019.
Kang, Minsoo, and Bohyung Han. "Operation-Aware Soft Channel Pruning using Differentiable Masks." arXiv preprint arXiv:2007.03938 (2020).
To support the claim that PfQ has the SOTA performance on benchmark datasets, first, PfQ should outperforms other BN based pruning techniques under the quantization settings. Then, PfQ should be better than those Pruning and Quatization Joint optimal methods for model compression. Especially, the readers would like to know why the variance based pruning approach is better than those BN weights based methods. It is not convincible enough for readers to follow the idea without analysis into details.
2) It is lack of explanation why the workflow is needed for the model compression with PfQ. The reason why multiple PfQs are executed is missing. A more clear algorithm would be better for the presentation of the "workflow".
3) Ablation studies: since the author design multiple round of PfQs, the paper is expected to show the studies on what will happen if the workflow just contains PfQ and reasonable finetune. Why the performance could not beat the current "workflow"? How would the variance distribution be after the "workflow"?
4) More proof-reading will make the paper look better. For example,  in page 7, section 4.3.1, "a single GPU (2080ti)" => "a single GPU (2080Ti)". In the same sentence, "for the learning in cifar100" => "for the learning in CIFAR-100". 

review_title: Clear idea. More analysis and results are needed to support the idea

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a pruning for quantization (PfQ) scheme to improve the fine-tuning of the quantized network. PfQ removes the filters that disturb the fine-tuning of the DNN while retaining good performance. Some experiments are conducted to demonstrate the effectiveness of the proposed PfQ.

Pros:
- The paper is well motivated and the method seems valid.

Cons:
- The writing quality is poor. in its current form, a general readership will struggle to understand it. An expert familiar with the details of the field (not just the general area) can probably disentangle the text, but otherwise, it's too convoluted.
- This paper is over-claimed. For example, how BatchNorm disturbs the fine-tuning of the quantized network has been pointed out by [1]. The analysis presented by the authors is too incremental.

Some remarks:
- Please note that BN can not be absorbed into the previous convolution layer in the training stage. The mean and variance in Eq.(2) should be running-moving average statistics, which should be pointed out in the beginning.
- To improve the quality of this paper, more rigorous mathematical notations and some visual experiments should be provided.
- It would be better to show some ablation studies to show how your method works.

[1] Data-free quantization through weight equalization and bias correction. Markus Nagel et al.

**********After rebuttal

The revised version has a better shape. In particular, I like the analytical experiment (Fig.2), which demonstrates that the proposed scheme can improve the wide dynamic range. Overall, this paper observes that BN with small variance influence quantization and proposes a protocol for training a quantized neural network combining filter pruning. 

Some issues still prevent it from being accepted. For example, PfQ is proposed to reduce the dynamic range. However, there is even no definition of the dynamic range in the paper which may make readers hard to understand the mechanism of PfQ.  Besides, the author claim that the weight widening the dynamic range in quantization is theoretically analyzed. But the analysis of Eqn. (14-16) is less rigorous. The readers may expect to see how weights in the case of $V_{c}^{L,\tau} \approx 0$ increase the dynamic range according to its definition compared to those weights in the other cases ($V_{c}^{L,\tau} \gg 0$).

The paper proposes an effective approach of quantization, which reduces the model size and improves accuracy. I would like to increase my rating to 5.
review_title: Review of Paper 609

Meta-review: Four reviewers rate this article borderline. R3 finds the paper clearly presented and the method effective, but misses quantitative analysis of the dynamic range problem as well as novelty. Following the discussion and revision, she/he considers the paper improved and updated the score to 5, still being concerned about the novelty. R1 considers the paper makes an important observation but has concerns about experiments, rating it 6. R2 considers that the paper contributes a clear idea, but indicates that more analysis and supporting results are needed. She/he indicated a number of shortcomings in the initial review, and found the update good, hence tending to rate the paper higher after the responses, 6. R4 considers the paper well motivated and the method valid. However, he/she found the writing poor and over-claiming results, and that more rigorous mathematical notation would help. After the discussion and revision, he/she found the paper better and increased the score to 5, but still found issues preventing the paper from being accepted. In summary, the reviewers agree that the paper contains an interesting and well motivated method, but they also point at a number of shortcomings. The revision improved several of them but others persisted. Although the ratings improved after the discussion, the overall rating is borderline. This is a very competitive call, and hence I have to recommend reject at this time. 
Decision: Reject