Title: DNN Quantization with Attention
Authors: Ghouthi BOUKLI HACENE
Lukas Mauch
Shubhankar Chowdhury
Stefan Uhlich
Fabien Cardinaux
Venue: ICLR 2022
Keywords: Deep learning
Computer vision
Quantization

Abstract: Low-bit quantization of network weights and activations can drastically reduce the memory footprint, complexity, energy consumption and latency of Deep Neural Networks (DNNs). Many different quantization methods like min-max quantization, Statistics-Aware Weight Binning (SAWB) or Binary Weight Network (BWN) have been proposed in the past. However, they still cause a considerable accuracy drop, in particular when applied to complex learning tasks or lightweight DNN architectures. In this paper, we propose a novel training procedure that can be used to improve the performance of existing quantization methods. We call this procedure \textit{DNN Quantization with Attention} (DQA). It relaxes the training problem, using a learnable linear combination of high, medium and low-bit quantization at the beginning, while converging to a single low-bit quantization at the end of the training. We show empirically that this relaxation effectively smooths the loss function and therefore helps convergence. Moreover, we conduct experiments and show that our procedure improves the performance of many state-of-the-art quantization methods on various object recognition tasks. In particular, we apply DQA with min-max, SAWB and BWN to train $2$bit quantized DNNs on the CIFAR10, CIFAR100 and ImageNet ILSVRC 2012 datasets, achieving a very good accuracy comparing to other conterparts.


Review discussion:
Strength:
-	Authors shows the proposed multi-biwidth training effectively reduces quantization error and helps smooth loss landscape with some sample cases and visualizations.
Weaknesses:
-	Overall performance of the given approach is not satisfying. Most recent quantization papers mainly conduct experiments on the large-scale ImageNet dataset since the CIFAR datasets are prone to easy overfitting. Almost all papers I know doing low-bit quantization has better results than this one.
On weight-only 2-bit quantization with MobilenetV2:
SAT: 66.8 (Neural Network Quantization with Scale-Adjusted Training BMVC 2020)
DeepComp 58.1 (Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. ICLR 2016)
This work: 52.2
On both weight and activation 2-bit quantization with ResNet18:
PACT 64.4 (Pact: Parameterized clipping activation for quantized neural networks. arXiv 18)
LQNet 64.9 (Lq-nets: Learned quantization for highly accurate and compact deep neural networks. ECCV 18)
SAT 65.5 (Neural Network Quantization with Scale-Adjusted Training BMVC 2020)
This work: 60.4
-	Some technical details are not clear. The authors use a penalty term (equation 6) to regularize the attention weights of different bitwdith.
However, it is not known whether all bitwidths in the network will be converged to the lowest bit which is the target. If some bitwidths are not property converged, will there be any issue on the performance?
-	Writing needs improvement. There are a lot of grammar errors and typos:
Page 2 last paragraph: a way how to train a (delete how)
Page 3 background 4th paragraph: Note, that because (no comma)
Page 3 Background 4th paragraph: this problem is most pronounced for low bitwidhts (typo)
NO.
We thank the reviewer for his effort in reading and understanding the proposed method, and his comments that are relevant to us, and help us to improve the paper quality.
The proposed method is a way to use already existing quantization methods and improve them, and not a quantization technique by itself.
Therefore, the comparison is done with a given quantization method, and the same quantization method when considering our proposed method. We also compare with Binary Relax which is another method that considers existing quantization techniques and improves them.
More details about the proposed method and its purpose, and corrections are introduced in the final version.
Pros:
- The paper is well written and the idea is easy to follow.
- Extensive ablation studies are provided to evaluate different components of the proposed method.
Cons:
- More parameters are introduced in the training stage, such as α. This will increase the computation and storage cost. More theoretical and experimental analysis should be given to study that.
- Multiple quantizers with different bit-width are conducted in the proposed method, which will increase the storage and computation cost for quantization.
- In the experiments, the authors compare their method with the corresponding counterparts with the same bit of n1.
However, the proposed method has three quantizers with different bit-width, and n1 is the lowest bit-width.
Therefore, this comparison seems unfair. For a fair comparison, the baselines and the proposed method should be compared under the same computation and storage cost.
- The proposed method has not been compared with state-of-the-art approaches, which cannot comprehensively evaluate the proposed method.
NO.
First of all, we would like to thank the reviewer for pointing out very interesting comments that will help us to improve the paper quality.
Indeed more parameters and quantizers are introduced during training that increases the memory footprint and computational power.
However, in this paper, we aim at quantizing a given network for the inference process, and with the proposed method we end up with a quantized network without any additional parameter.
Thus the method aims at improving a quantized network performance without adding additional complexity during inference. That will be better explained in the final version.
The comparison is done when considering the inference process when the obtained network using our method is quantized on only n1 bits and thus for inference and deployment it uses the same memory and computational power as other counterparts. More details will be introduced in the Results section to avoid such confusion.
The proposed method is not a quantization technique but a way to better use and improve existing quantization methods. It has been compared to Binary Relax because as the proposed method, it aims to relax the quantization problem and improve already existing quantization techniques. Other state-of-the-art will be added in the final version of the paper to show that when applying our proposed contribution, such methods can be improved.
1) In my understanding, what the paper call attention is simply a value weighting the importance of the different quantization functions (see Eqn. 2-3).
Therefore this terminology is misleading in my opinion, as the vector of attention a only depends on the “trainable parameter alpha”, and not does not depend on the input (either or activation) as one would imagine with this name. Maybe I misunderstood something, but this is what I infer from Eqn (4). Eqn (2-3) and Figure 1.
2) The paper does not consier in the literature review techniques for quantizing neural networks [A,B,C] that, to my knowledge, are state-of-the-art for quantizing popular neural networks.
[A] Permute, Quantize, and Fine-tune: Efficient Compression of Neural Networks, Martinez et al., CVPR’2021
[B] And the bit goes down: Revisiting the quantization of neural networks, Stock et al. ICLR’2020
[C] Training with Quantization Noise for Extreme Model Compression, Fan et al. ICLR’2021
At a high level, some elements are similar to these approaches, even though the details may differ.  One noticeably similarity is the various of precision, which is already present in the work by Fan et al [C] when they consider both blocks that are not quantized (i.e., 32 bits) versus some that are quantized with low-precision, and randomly choose. In my own experience, choosing randomly a choice is better that relaxation, so I guess the paper should have included such a comparison.
In any case the paper should be better positioned against the recent literature.
3) The paper is compared to poor baselines.
As a result, the paper reports results that do not look competitive the state of the art [A,B,C] on Imagenet ILSVRC 2012, which is the most significant benchmark in the paper.
For instance, Stock et al. [B] report some results with R18  (Figure 3 left, table 3) that are as follow: compression factors of x20 for top-1 accuracy at 67.87. In the submitted paper.
Additionally, more recent papers [A] and [C] are compared to [B] on larger networks and show that they further improve results.
Therefore I conclude that the proposed approach is not competitive, while additionally requiring a more engaged scheduling that may not generalize as well to other training settings. The paper states that it could be combined with any compression method, therefore in this context it would be worth using the same or similar quantization as in [A,B,C] and show how the method compares to these methods.
4) Formally I have nothing against having in the same section the introduction and the related work. In the case of this paper, I found that the introduction is actually more a related work than a formal introduction providing the motivation and rationale of the paper content at the core of the initial discussion.  This discussion appears later in the background section.
While I know well this area and therefore the problem at stake, I would advise to re-work jointly these two sections.
5) The paper  mentions that the method could be used for quantization activation, but only addresses the case of weight quantization. I think that a lot of practical considerations would appear with activation quantization, so I would suggest either to support this claim with experiments,  or to suppress or soften this unsupported claim.
6) The paper needs some polishing. Some mistakes indicate that the paper was not analyzed by a spell-checker, for instance in the abstract: conterparts -> counterparts. The term bitwidth is not established (and occurred at least once with a typo).
NO.
We would like to thank the reviewer for his comments that help us to improve the paper quality.
Some typos in the paper make some parts a bit confusing. The final version will be better presented.
The paper aims at finding a way to improve quantization methods and not at introducing a new one. We used some well known quantization methods to show that. More quantization methods will be added in the final version to show how they can be improved using our method.
We are separating the introduction and related work to make each part more relevant and clear in the final version.
In the section Results, Tables 2 and 4 report obtained accuracy when both weights and activations are quantized.
Also in the final version, we are considering more complex tasks as object detection to more stress the proposed contribution.
Strengths:
1. According to the experiment, DOA performs better than the naive quantization strategy and Binary-Relax consistently across the experimented datasets and networks.
Weaknesses:
1. The presentation needs to be improved. There are grammar errors and typos in the paper.
2. The paper compares the performance of DOA with only one related quantization work (Binary-Relax). It is not sufficient to demonstrate the effectiveness of the proposed work. There are many quantization works, both quantization-aware training and post-training quantization. It seems that some of them may have better performance in the experimented settings.
For example, the work LQ-Nets reports an accuracy of 68% with 2-bit weights and 32 bits activation with ResNet-18 on ImageNet, but DOA proposed in this paper only achieves an accuracy of 66.9%.
Minor comments or questions:
1. How to decide what quantization method to use (e.g., min-max, SAWB, BWN, TWN) when using DAQ in practice? Appendix A only defines each quantization method but doesn't give any guidance on how to choose them.
2. For experiments on the ImageNet dataset using ResNet18, why not report R18+BR and MV2+BR results?
3. In section 4.2, the paper mentions all reported validation accuracies are the results of a single training run. It might be better to report averaged results across several runs even if the convergence of the networks is not noisy.
4. In Table 1, it seems that DQA using SWAB consistently gives better results than the FP version. Do you have any insight regarding this?
NO.
We would like to thank the reviewer for his constructive reviews. The presentation will be improved in the final version and we are checking all grammar errors.
In this paper, we are comparing to Binary-relax because this method tries to relax the quantization problem and improves already existing quantization methods as our method. The idea here is not to introduce a new quantization method but to find a way how to use the already existing methods more efficiently.
For Instance, LQ-Net is a quantization method that can be used in our method to be improved. We will reformulate the method purpose in the paper to make it more clear. We will also add more quantization methods in the final version and show how they can be improved using our method.
Due to time, we could not run more experiments. More results will be added to the final version.

Meta-review:
role: PC
data: comment: This paper proposes a new learning procedure for quantizing neural networks. Basically, DQA method proposed in this paper uses attention to obtain a linear combination of the existing network quantization techniques and uses it to pursue more efficient quantization.

Overall, it seems the submission was written in haste, so there are many typos and errors. Above all, the motivation that it can be applied to various existing techniques could not be proved experimentally at all since it only covers one somehow obsolete work. In addition, as in [1], it seems necessary to quantize not only weights but also activations, or to verify in lightweight networks such as MobileNetV2 rather than ResNet.

[1] Cluster-Promoting Quantization with Bit-Drop for Minimizing Network Quantization Loss, ICCV 2021

Decision: Reject