Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a novel and efficient quantization method for Vision Transformers. The authors first identify the bottleneck low-bit quantized Vision Transformers which comes from the information distortion of the low-bit quantized self-attention map. The authors then develop an IRM and a DGD scheme for fully quantized Vision Transformers based on the aforementioned bottleneck, which leads to a fully quantized Vision Transformers. 
strengths_and_weaknesses: 1.In Line 61, what is the actual meaning of “quantified representations”?

2.In section 4.1 “Architecture bottleneck”, how are these experiments trained? the same as the main experiments settings?

3.In Line 155, “distance”should be “distances”. 

4.Font sizes of the legend in Figure 4 are inconsistent and should be larger. 

5.Are the IRM applied before quantizing attention module, or after quantizing? Such two situations may lead to different results. 

6.In section 5.1, authors mentioned that the Q-ViT is initialized from pertained full-precision counterparts. However, how are the alpha and zero point initialized?

7.In Eq. 7, what does “q_x is the random quantized variables in Q_a(x)” mean? What are the possible values of “q_x”?

8.In Eq. 8, after introducing IRM, the module calculate the mean and variance of Q and K in each block for each forward process, will this affect the speed of inference?

questions: Please see above.
limitations: Please see above.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the quantization of the Vision Transformer model, especially in the low-bit case. For the problem of large quantization accuracy loss at low bits, two improvements, IRM and DGD, are proposed in this paper, respectively. Concretely, IRM maximizes the information entropy of the quantized representation. DGD supervises the model with other patch-based similarity pattern matrices.
strengths_and_weaknesses: Strengths:
1.	The quantization of ViT models is an important research topic. In this paper, the accuracy loss caused by ViT quantization is mitigated to some extent.
2.	The article is well written and well finished. It is easy to follow.

Weaknesses:
1.	The analysis of vit quantification could be explained in depth:
(a) this paper argues that `a direct quantization method leads to the information distortion’ in Line 45. The approach proposed in this paper does not improve this phenomenon either (e.g. 1.2268 in Fig1(b) v.s. 1.3672 in Fig5(b) for Block.3. The variance difference is even larger with the proposed approach). 
(b) The quantization of MHSA introduces a large loss of precision, which has been found in transformer quantization in the NLP (such as Q-BERT, Q8BERT, BinaryBERT, FullyBinaryBert, etc.) and is not unique to the ViT model. 
2.	Some minor problems:
(a)	In Fig2, the tilde hat of k is too small. It should be inconsistent with q’s hat. 
(b)	In Equation 9, $Q_k$ might be $Q(k)$ to be consistent with $Q(q)$.

questions: 1.	This paper alleviates the problem of accuracy loss caused by ViT quantization to some extent by two improvement measures. However, I expect that the author dig deeper into the differences between ViT and CNN quantization and the differences between transformer quantization in vision and NLP.
2.	I notice that this work applies Adam optimizer with weight decay set as 0, which is not a regular setting. I suggest the author add extra ablation studies on it to make the paper more complete.

limitations: The limitations have been well addressed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focused on Vision Transformers quantization aware training methods. In this paper, the bottleneck of QAT for ViT are firstly studied which are mainly caused by the information distortion in MHSA. This paper thus propose IRM and DGD scheme to solve such bottleneck which retains the performance of quantized ViT from full-precision counterparts. 
strengths_and_weaknesses: Strength:

1. The paper is technically sound and easy to understand.

2. The experiments show that the proposed method is effctive.

Weakness:

1. How is the “6.14x”in Abstract calculated?

2. In L155, how is the distance between each attention weight calculated?

3. In L191, learning rates of learnable gamma and beta are not clarified, are they the same as the learning rate of the whole network? 

4. In Eq. 10, why just query and key are distilled, instead of query, key and value?

5. How are alpha_x and alpha_w learned through training? Are the training settings the same as other learnable parameters?

6. In Eq. 7, after maximizing the information entropy, q_x should follow an uniform distribution, however, in the aforementioned figure 4, the query and key may follow a Gaussian distribution. This seems to be conflicting. 

7. In Eq. 8, how are the mu and sigma calculated? Are they channel-wise or layer-wise?

8. How is Eq. 9 derived? Especially the part“1/2 log2πe[gamma^2(sigma^2 + epsilon)]”?

questions: See weakness above.
limitations: -
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, Q-ViT is proposed, an effective fully quantized scheme for vision transformer. This paper first introduces the bottleneck for severe performance drop and then present the Information Rectification Module (IRM) and Distribution Guided Distillation (DGD) to bulld a high-performance fully quantized vision transformers. And the experimental results demonstrate the superiority of Q-ViT.
strengths_and_weaknesses: Strengths
1. The analysis of the bottleneck for severe performance drop is clear.
2. The experimetal results are state-of -the-art.
3. The paper is well-written and easy to understand.
4. The code and checkpoints are open sourced.

Weaknesses
1. When the deterministic quantization function is applied to quantized ViT, such objective is equivalent to maximizing the information entropy. The reason is not clearly described.
2. The experiments are sufficient but the proof of these two techniques is a bit weak.
questions: 1. Please prove why maximizing the mutual information between quantized and full-precision representations is equivalent to maximizing the information entropy $H(Q_x)$, there is nothing new in the the disussion of supplementary material.

2. The learnable parameters $\gamma_q$, $\beta_q$ and $\gamma_k$ and $\beta_k$ are updated with the total loss, they could get a better information as Figure 4, but how to prove that they could achieve the state of information maximization. The Information Rectification Module (IRM) seems a loss-aware method instead a strong constrain to keep the max information.


limitations: Yes
review_title: null

Meta-review: metareview: This paper proposes a novel method for Vision Transformers quantization. The IRM and DGD scheme is developed to solve the bottleneck of low-bit quantized Vision Transformers. All the reviewers agree that the proposed method is novel and effective. The concerns and questions are well addressed during the rebuttal period. The overall quality is clearly above the bar, and thus the paper should be accepted for publication. 
confidence: Certain
Decision: Accept