Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes DNN quantization with Attention (DQA), which uses a learnable linear combination of high, medium, and low-bit quantization at the beginning. It gradually converges to a single low-bit quantization at the end of training. Experiments show that DQA outperforms the naive quantization and the Binary-Relax method consistently across three datasets and two networks.
main_review: Strengths:
1. According to the experiment, DOA performs better than the naive quantization strategy and Binary-Relax consistently across the experimented datasets and networks.

Weaknesses:
1. The presentation needs to be improved. There are grammar errors and typos in the paper.
2. The paper compares the performance of DOA with only one related quantization work (Binary-Relax). It is not sufficient to demonstrate the effectiveness of the proposed work. There are many quantization works, both quantization-aware training and post-training quantization. It seems that some of them may have better performance in the experimented settings. For example, the work LQ-Nets reports an accuracy of 68% with 2-bit weights and 32 bits activation with ResNet-18 on ImageNet, but DOA proposed in this paper only achieves an accuracy of 66.9%. 

Minor comments or questions:
1. How to decide what quantization method to use (e.g., min-max, SAWB, BWN, TWN) when using DAQ in practice? Appendix A only defines each quantization method but doesn't give any guidance on how to choose them.
2. For experiments on the ImageNet dataset using ResNet18, why not report R18+BR and MV2+BR results?
3. In section 4.2, the paper mentions all reported validation accuracies are the results of a single training run. It might be better to report averaged results across several runs even if the convergence of the networks is not noisy. 
4. In Table 1, it seems that DQA using SWAB consistently gives better results than the FP version. Do you have any insight regarding this?
summary_of_the_review: Overall, I think the paper is not good enough due to aforementioned nontrivial weaknesses.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The papers addresses the problem of compression of neural networks. The paper builds upon binary-relax prior work. The precision is adapted during training with a mixture-based quantization method through temperature cooling and a set of an ``attention’’ vector a. The idea of the method, called DQA, is to progressively moves from a mixture of quantization functions (mixing high with low precision, for instance 32 bit with 2 bit training), to a single one (low precision) towards the end of the training. The paper states that the method can be used with several types of quantization methods. The evaluation is carried on computer vision architectures (ResNet18, MobileNet) for image recognition tasks (Cifar-10, Cifar-100, Imagenet ILSVRC 2012). 

main_review: 
1) In my understanding, what the paper call attention is simply a value weighting the importance of the different quantization functions (see Eqn. 2-3). Therefore this terminology is misleading in my opinion, as the vector of attention a only depends on the “trainable parameter alpha”, and not does not depend on the input (either or activation) as one would imagine with this name. Maybe I misunderstood something, but this is what I infer from Eqn (4). Eqn (2-3) and Figure 1. 

2) The paper does not consier in the literature review techniques for quantizing neural networks [A,B,C] that, to my knowledge, are state-of-the-art for quantizing popular neural networks. 
[A] Permute, Quantize, and Fine-tune: Efficient Compression of Neural Networks, Martinez et al., CVPR’2021
[B] And the bit goes down: Revisiting the quantization of neural networks, Stock et al. ICLR’2020
[C] Training with Quantization Noise for Extreme Model Compression, Fan et al. ICLR’2021
At a high level, some elements are similar to these approaches, even though the details may differ.  One noticeably similarity is the various of precision, which is already present in the work by Fan et al [C] when they consider both blocks that are not quantized (i.e., 32 bits) versus some that are quantized with low-precision, and randomly choose. In my own experience, choosing randomly a choice is better that relaxation, so I guess the paper should have included such a comparison. In any case the paper should be better positioned against the recent literature. 

3) The paper is compared to poor baselines. As a result, the paper reports results that do not look competitive the state of the art [A,B,C] on Imagenet ILSVRC 2012, which is the most significant benchmark in the paper. For instance, Stock et al. [B] report some results with R18  (Figure 3 left, table 3) that are as follow: compression factors of x20 for top-1 accuracy at 67.87. In the submitted paper. Additionally, more recent papers [A] and [C] are compared to [B] on larger networks and show that they further improve results. Therefore I conclude that the proposed approach is not competitive, while additionally requiring a more engaged scheduling that may not generalize as well to other training settings. The paper states that it could be combined with any compression method, therefore in this context it would be worth using the same or similar quantization as in [A,B,C] and show how the method compares to these methods. 

4) Formally I have nothing against having in the same section the introduction and the related work. In the case of this paper, I found that the introduction is actually more a related work than a formal introduction providing the motivation and rationale of the paper content at the core of the initial discussion.  This discussion appears later in the background section. While I know well this area and therefore the problem at stake, I would advise to re-work jointly these two sections. 

5) The paper  mentions that the method could be used for quantization activation, but only addresses the case of weight quantization. I think that a lot of practical considerations would appear with activation quantization, so I would suggest either to support this claim with experiments,  or to suppress or soften this unsupported claim. 

6) The paper needs some polishing. Some mistakes indicate that the paper was not analyzed by a spell-checker, for instance in the abstract: conterparts -> counterparts. The term bitwidth is not established (and occurred at least once with a typo). 



summary_of_the_review: The paper does not demonstrate that the method is a significant contribution to the state of the art in quantizing neural networks. The experiments are only applied on image classification task with small architectures, and in the setting that I found comparable in the literature, the results do not look great, which questions the significance of this work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper attempt to address a challenging quantization problem, i.e., low-bit quantization. This work utilizes a learnable linear combination of high, medium, and low-bit quantization at the beginning while converging to a single low-bit quantization at the end of the training. In the quantization procedure, multiple quantizers and the corresponding attention matrices are adopted to fuse the quantized weights or activations.
main_review: Pros:
- The paper is well written and the idea is easy to follow.
- Extensive ablation studies are provided to evaluate different components of the proposed method.

Cons:
- More parameters are introduced in the training stage, such as α. This will increase the computation and storage cost. More theoretical and experimental analysis should be given to study that.
- Multiple quantizers with different bit-width are conducted in the proposed method, which will increase the storage and computation cost for quantization.
- In the experiments, the authors compare their method with the corresponding counterparts with the same bit of n1. However, the proposed method has three quantizers with different bit-width, and n1 is the lowest bit-width. Therefore, this comparison seems unfair. For a fair comparison, the baselines and the proposed method should be compared under the same computation and storage cost.
- The proposed method has not been compared with state-of-the-art approaches, which cannot comprehensively evaluate the proposed method.
summary_of_the_review: This work utilizes a learnable linear combination of high, medium, and low-bit quantization at the beginning while converging to a single low-bit quantization at the end of the training. In the quantization procedure, multiple quantizers and the corresponding attention matrices are adopted to fuse the quantized weights or activations, which will increase the computation and storage cost. Some experiments are conducted to evaluate the proposed method. However, it lacks some comprehensive and fair comparison.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work presents a training method for low-bit network quantization. While training, it employs a multi-bitwidth paradigm in order to alleviate the nonsmooth optimization landscape with lower bitwidth. It uses a temperature parameter and a penalty term to force the network to gradually converge to the target low bit. Experiments are conducted on CIFAR10, CIFAR100, ImageNet Classification with ResNet 18 and MobilenetV2.
main_review: Strength:
-	Authors shows the proposed multi-biwidth training effectively reduces quantization error and helps smooth loss landscape with some sample cases and visualizations.

Weaknesses:
-	Overall performance of the given approach is not satisfying. Most recent quantization papers mainly conduct experiments on the large-scale ImageNet dataset since the CIFAR datasets are prone to easy overfitting. Almost all papers I know doing low-bit quantization has better results than this one. 

On weight-only 2-bit quantization with MobilenetV2:
SAT: 66.8 (Neural Network Quantization with Scale-Adjusted Training BMVC 2020)
DeepComp 58.1 (Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. ICLR 2016)
This work: 52.2

On both weight and activation 2-bit quantization with ResNet18:
PACT 64.4 (Pact: Parameterized clipping activation for quantized neural networks. arXiv 18)
LQNet 64.9 (Lq-nets: Learned quantization for highly accurate and compact deep neural networks. ECCV 18)
SAT 65.5 (Neural Network Quantization with Scale-Adjusted Training BMVC 2020)
This work: 60.4

-	Some technical details are not clear. The authors use a penalty term (equation 6) to regularize the attention weights of different bitwdith. However, it is not known whether all bitwidths in the network will be converged to the lowest bit which is the target. If some bitwidths are not property converged, will there be any issue on the performance?
-	Writing needs improvement. There are a lot of grammar errors and typos:

Page 2 last paragraph: a way how to train a (delete how)

Page 3 background 4th paragraph: Note, that because (no comma)

Page 3 Background 4th paragraph: this problem is most pronounced for low bitwidhts (typo)

summary_of_the_review: Overall, this work presents a new approach to help improve the training convergence of low-bit quantization for neural network. Due to the weak results on large-scale datasets and unclear technical details, I do not think it meets our bar at ICLR.
review_title: null

Meta-review: This paper proposes a new learning procedure for quantizing neural networks. Basically, DQA method proposed in this paper uses attention to obtain a linear combination of the existing network quantization techniques and uses it to pursue more efficient quantization.

Overall, it seems the submission was written in haste, so there are many typos and errors. Above all, the motivation that it can be applied to various existing techniques could not be proved experimentally at all since it only covers one somehow obsolete work. In addition, as in [1], it seems necessary to quantize not only weights but also activations, or to verify in lightweight networks such as MobileNetV2 rather than ResNet.

[1] Cluster-Promoting Quantization with Bit-Drop for Minimizing Network Quantization Loss, ICCV 2021
Decision: Reject