Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: summary: This paper proposed a post-training quantization algorithm for vision transformers. The algorithm introduces a ranking loss to improve the quantization performance of the model by keeping the relative order of self-attention results. Besides, this paper explores a mixed-precision quantization scheme by exploiting the nuclear norm of each attention map and output feature.  The proposed method is verified on several benchmark models and datasets.
main_review: Pros:
- The paper proposes a post-training quantization method for vision transformer and applies it to different CV tasks.
- Ablation studies on ranking loss, bias correction and mixed-precision quantization are welcome.
- The paper is clear to read.

Cons:
- The authors adopt Pearson correlation coefficient as the measurement for the similarity (L156-162). There are many other simpler choices for computing similarity (cosine, Euclidean, KL, etc). For example, the work of [22] uses cosine similarity [22], and the previous work of "Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming" uses Euclidean distance. Why do the authors choose the Pearson correlation coefficient, and what are the advantages of this metric?
- Regarding ranking-aware quantization for self-attention, the authors state that "we empirically find that the relative order of the attention map has been changed after quantization as shown in Fig 1, which could cause a significant performance degradation" (L166-167). This "observation" is crucial to the main contribution and novelty of this paper, but it lacks theoretical analysis and sufficient experimental evaluations to demonstrate that the change of order actually negatively affects the final quantization accuracy. Without such analysis or evaluations, is the proposed solution reasonable and reliable? Moreover, suppose the statement is true, it lacks sufficient experiments to show that the order can be improved by the proposed solution. 
- The alternative searching [22] and bias correction [16] methods are not first proposed in this paper. The authors should cite them in the mentioned positions in case readers misunderstand they are contributions of this paper. 
- Section 3.3: Mixed-Precision Quantization: what is the difference between [9] and this paper? I do not seem novelty in this part.
- What is the significance of the mixed-precision setting of this paper (L236-237)? Most practical hardware platforms have supported int8 inference with good hardware performance.
- It lacks hyper-parameter analysis (e.g., alpha/beta/gamma/theta).
- Figure 1 and Equation (6,7) do not match. Figure 1 shows pre-layerNorm but Equation (6,7) uses post-layerNorm.

Updated:
I've read all the reviews and author responses. The authors addressed part of my concerns, e.g., explain the differences between cosine similarity and Pearson correlation and provide results of them. But some other major concerns are not addressed.

1) For ranking strategy, Table 3 actually shows better results over the baseline but the gain is very limited. More importantly, the idea of ranking is not well explained in the paper to demonstrate the order of attention values is crucial for quantizing transformer.
2) For nuclear norm, yes, this paper lacks necessary comparison experiments with the baseline of using Hessian Coefficient. Only stating that nuclear norm is easier than Hessian Coefficient is far insufficient to support this contribution.
3) For mix-precision quantization, I think more general or more challenging settings (e.g., using 2-8 mixed bits or using lower bits of {2,3,4} as HAWQ-v2 [9]) can be evaluated to make this part of paper stronger but they are missing. More importantly, the algorithm of mix-precision quantization is almost the same as [9] except the nuclear norm (but nuclear nom is not well compared with exiting work to support its contribution). So the technical novelty is limited.

In general, I think the main strength of the paper is that it brings best practices to realize post-training quantization to vision transformer, but the novelty is very limited: 1) low-bit quantization follows existing alternative searching [22] and bias correction [16] and 2) mix-precision quantization follows HAWQ-v2 [9], and the different algorithmic parts between this paper and previous methods are not well explained and validated. Thus, I think the quality of this paper does not reach the bar of NeurIPS and still vote for rejection.

limitations_and_societal_impact: No. I did not find clear discussions on the limitations and potential negative societal impact of the work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors have proposed a post-training quantization method for vision transformer, where different modules employs the different bit-width. The quantization intervals are determined by optimizing the similarity between the quantized and original output features. In addition, the ranking-aware loss is proposed to promote the quantization performance of the self-attention layers.
main_review: Strengths:
The paper is well organized and I think it has significant contribution to the field. The proposed similarity and ranking-aware loss are rational. The experiments are well evaluated and they have demonstrated the effectiveness of the proposed method.

Weaknesses:
- What is the optimization process of the mixed-precision? I have a little confusion about the Eq.(18) and what is the meaning of the Y in Eq.(18)?
- What is the configuration of the mixed-precision in the experiments since you have used 6 MP and 8 MP?
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a post-training quantization scheme for vision transformer, which determines the bit-width of each layer according to the nuclear norm of the attention map and output feature. In order to find the optimal quantization interval, the authors develops two loss to optimize the quantization errors.

main_review: (1)	The similarity-aware loss and bias correction are reasonable and the designed ranking-aware loss designed for the self-attention layer is novel.
(2) The authors propose a mixed-precision quantization method for the MSA and MLP modules and experimental results demonstrate the effectiveness of the proposed method
(3) This paper is well organized.

limitations_and_societal_impact: (1)	What are the optimization goal of the self-attention layer? Are they the weights and inputs of the Q and K? There are some details need to be provided in Eq.(12).
(2)	Does the attention map in Eq.(13) the same as the output in Eq.(12)?
(3)	Some related works need to be discussed more concretely, such as quantization method for the transformer [1] [2].
[1] Fully Quantized Transformer for Improved Translation.
[2] Efficient 8-bit quantization of transformer neural machine language translation model.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a novel quantization algorithm for vision transformer by searching for the best quantization intervals of inputs and weights. The optimization is conducted by minimizing the similarity loss and the ranking loss. The bias correction method is employed and the nuclear norm based mixed-precision is proposed. The proposed method is evaluated on the image classification and object detection tasks.
main_review: Generally, this is a novel and effective quantization method for vision transformer, since the training and inference costs of vision transformer are vast. The proposed ranking-aware loss is specially designed for vision transformer and the mixed-precision quantization is instructive. The experimental results of the proposed quantization method outperforms the post-training quantization methods of CNN.
limitations_and_societal_impact: I have one question that the proposed method can be directly used on CNN except the proposed ranking-aware loss. Have you compared the method on CNN and does it have the same beneficial? 
Another question is what’s the meaning of Y in Eq.(18)? Is it the attention map or the output feature?
review_title: null

Meta-review: The manuscript has been reviewed by four experienced reviewers, among whom three voted for acceptance and one voted for rejection. There are several rounds of discussion post the release of the reviews, and the rejection reviewer (NVLj) raises some concerns, including the quality of 1) Fig.1, 2) comparisons to Hessian Coefficient, and 3) limited novelty. All three concerns are highly constructive, yet given other contributions of the submission acknowledged by other reviewers, the pros indeed overweight the cons. Specifically, despite reviewer NVLj believes that the submission lacks novelty, all the other three reviewers, in fact, viewed the proposed approach as being novel.

The AC agrees with the major vote and hence recommends an acceptance. However, it is strongly recommended that the authors also incorporate the comments from reviewer NVLj, especially on comparisons to Hessian Coefficient, into the final version, if possible.
Decision: Accept (Poster)