Post-Training Quantization for Vision Transformer

Abstract: Recently, transformer has achieved remarkable performance on a variety of computer vision applications. Compared with mainstream convolutional neural networks, vision transformers are often of sophisticated architectures for extracting powerful feature representations, which are more difficult to be developed on mobile devices. In this paper, we present an effective post-training quantization algorithm for reducing the memory storage and computational costs of vision transformers. Basically, the quantization task can be regarded as finding the optimal low-bit quantization intervals for weights and inputs, respectively. To preserve the functionality of the attention mechanism, we introduce a ranking loss into the conventional quantization objective that aims to keep the relative order of the self-attention results after quantization. Moreover, we thoroughly analyze the relationship between quantization loss of different layers and the feature diversity, and explore a mixed-precision quantization scheme by exploiting the nuclear norm of each attention map and output feature. The effectiveness of the proposed method is verified on several benchmark models and datasets, which outperforms the state-of-the-art post-training quantization algorithms. For instance, we can obtain an 81.29% top-1 accuracy using DeiT-B model on ImageNet dataset with about 8-bit quantization. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/VT-PTQ. 


Reviews and author responses:

 We would like to thank the reviewer for the helpful and valuable feedbacks.

$\textbf{Q1:}$ The manuscript proposes a post-training quantization for vision transformer. The task is urgently needed for deployment in industrial environments. The related work is cited adequately, but more analysis about the relationship between the quantization of BERT and ViT should be provided.

$\textbf{A1:}$ Thanks for the encouraging and positive comments. There are many training-aware quantization methods for BERT now and we think they can be transfered to the vision transformer with modest change. We will also apply the proposed method on BERT in the future although this paper focuses on the quantization of vision transformer.   

$\textbf{Q2:}$ The method is not very clarified and some equations are very confusing, especially the strategy for obtaining different bit-widths is not clear.

$\textbf{A2:}$ Sorry for the confusion.There are lots of candidate bit-width configurations in the search space, so we utilize a Pareto Frontier approach to find the bit-width configuration with the minimal $\Omega$ as described in the following equation (Eq.(18) in the paper).

$\Omega = \sum_{i=1}^L \Omega_i=\sum_{i=1}^L \sum_{j=1}^{m}\sigma_j(\textbf{Y})\cdot\|\widehat{\textbf{Y}}-\textbf{Y}\|^2_2.$

where L is the number of layers and m is the number of singular values.

$\textbf{Q3:}$ Figure 2 has no clear explanation. In addition, the motivation for introducing rank-loss is best to provide more sufficient evidence.

$\textbf{A3:}$ In Figure 1 (if I am not misunderstand), we use the similarity-aware quantization for the MLP module as it only contains linear operations and utilize ranking-aware loss combined with the similarity for the MSA module since the attention layer plays a critical role in the transformer structure. The bit-widths of the MSA and MLP modules are determined by the weight nuclear norm of the output features and attention maps. 

As for the motivation of introducing rank-loss, the attention layer plays a critical role in transformer structure and ranking order relavance is another metric different from the Pearson coefficient. We find that the rank of the attention map could be changed after quantization, where the difference of the rank before and after quantization is shown in the figure of the link (https://imgtu.com/i/ftNMkQ). And the difference can be compensated by introducing the ranking-loss as shown in the figure. The experimental results demonstrate the effectiveness of the proposed method.

 Thanks for the helpful and constructive comments.

$\textbf{Q1:}$ However the use of rank-quantization for self-attention layers, as opposed to standard correlation used for MLP here, is not tested or justified. Is quantization based on rank actually superior to other methods (e.g. similarity using the Pearson correlation coefficient?)

$\textbf{A1:}$ Thanks for the nice concern. The ranking-aware loss is combined with the similarity using the Pearson correlation coefficient for the self-attention layers. We have conducted the experiments about the ranking-aware quantization as shown in Table 3 in the manuscript. It can be seen that using ranking-aware loss outperforms than only using similarity-aware quantization.

| models  | Similarity | Ranking | Bias Correction | Mixed Precision | Top-1 Accuracy |
| :--: | :--: | :--: |  :--: | :--: | :--: |
| ViT-B | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | 76.61 |
| ViT-B | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 76.98 |
 

$\textbf{Q2:}$ Why not use the standard Spearman correlation coefficient to evaluate rank, which is identical to the Pearson used here except on ranks?

$\textbf{A2:}$ Thanks for the constructive suggestion. The Spearman correlation coefficient is a good choice and we will add the analysis in the manuscript.

$\textbf{Q3:}$ A large number of hyper-parameters are manually chosen, presumably to favor the authors method: α and β are set to 0.5 and 1.2, trade-off parameter γ and the threshold θ in Eq. (13) are set to 0.1 and 0.2, the maximum iteration is set to 20 if not mentioned specifically. For mixed-precision, we utilize {4,5,6,7,8} and {6,7,8,9,10} bits while the target bit-width are 6 bit and 8 bit. How fair is this for the three other methods used in comparison ( percentile-based method, EasyQuant, Bit-Split)? Have the authors made similar efforts to optimize other methods?

$\textbf{A3:}$ Thanks for the concerns. For the percentile-based method, we have tried the 95%, 98%, 99% and 99.9% maximum value as the clipping threshold and show the best result in the manuscript. Compared to EasyQuant, we added two parameters $\gamma$ and $\theta$ for the ranking-loss since the $\alpha$ and $\beta$ are used to determine the optimization range of quantization. For the Bit-split method, it is a hyper-parameter free scheme so we just follows the optimization process in the paper. The mixed-precision scheme is simple and the performance of the proposed method using fixed bit-width is also better than the previous methods.

$\textbf{Q4:}$ In the big picture, quantization is one possible post-processing method for the goal of reducing memory and computation, others include distillation (which the authors specifically do not use), various projection methods (e.g. fast-food transform [a]), pruning, etc. It is not clear why quantization should be used instead of these many methods for the same goal, nor why it should not be used in combination. It is not clear wether the three methods compared against (percentile-based method, EasyQuant, Bit-Split) represent a comprehensive collection of alternative quantization methods.

$\textbf{A4:}$ Thanks for the suggestion. Post-training quantization is a popular and efficient method for reducing memory and computation since it does't need re-training or fine-tuning, especially when the dataset is not easy to obtain. We will combine it with other compression method (pruning, distillation etc.) in the future, while this paper focuses on the post-training quantization of transformer. 

The compared methods (percentile-based method[1], EasyQuant[2], Bit-Split[3]) are classical or the state-of-art post-training quantization methods. To our best knowledge, we are the first to apply the post-training quantization algorithm for computer vision transformer.

[1]Fully quantized network for object detection, CVPR'2019.

[2]Easyquant: Post training quantization via scale optimization, arxiv'2020.

[3]Towards accurate post-training network quantization via bit-split and stitching, ICML'2020.

 We would like to thank for the encouraging and helpful feedbacks. Here are the responses to the questions.

$\textbf{Q1:}$ In Section 3.2, the motivation of using the Pearson correlation coefficient to measure the similarity between the original and quantized feature maps is not clear. Why not use the Euclidean distance or the cosine similarity?

$\textbf{A1:}$ Thanks for the nice question. The Person correlation coefficient for conducting the proposed method is identical to the normalized cosine similarity. The formulation can be presented as:

$Cosine(x,y)=\frac{\sum_i x_i y_i}{\sqrt{\sum_i x_i^2}\sqrt{\sum_i y_i^2}}$

$Person(x,y)=\frac{\sum_i (x_i-\overline{x}) (y_i-\overline{y}) } {\sqrt{\sum_i (x_i-\overline{x})^2} \sqrt{\sum_i (y_i-\overline{y})^2}} = Cosine(x-\overline{x},y-\overline{y})$

It can be seen that Cosine similarity is not invariant to shifts so the Person correlation coefficient is more precise for evaluating the similarity since the mean value can be corrected by the bias correction. The experimental results in the following table demonstrate the analysis and we will add the analysis and the experimental results in the manuscript.

| 8-bit ViT-B  | Euclidean distance | Cosine similarity | Pearson correlation |
| :--: | :--: | :--: |  :--: |
| Top-1 Accuracy | 75.42% | 75.57% | 75.81% |

$\textbf{Q2:}$ The effect of alternating searching is not discussed. To demonstrate the effectiveness of the proposed alternating searching, it would be better for the authors to provide more results without the alternative searching, which would strengthen the paper.

$\textbf{A2:}$ Thanks for the constructive suggestion. We have done ablation study for the alternative searching method as shown in the following table, where no alternating searching is using when iteration is zero. As we can see, the result of using alternating searching is obviously better than optimizing each layer separately and iteratively optimization can further improve the performance of the quantization model.

| Iterations  | 0 | 1 | 10 | 20 |
| :--: | :--: | :--: |  :--: | :-:|
| Top-1 Accuracy | 74.36% | 75.28% | 75.75% | 75.81% |

$\textbf{Q3:}$ In line 180, what does $\Delta_l$ denotes? What do $\alpha$ and $\beta$ denote? More explanations are required.

$\textbf{A3:}$ Sorry for the typos in Line 180. We have corrected the writing to: We linearly divide the range of [$\alpha S_l, \beta S_l$] into $C$ candidate, where $S_l$ represents the maximum value of the weights or activations in the $l$-th layer.

$\textbf{Q4:}$ Please use "alternating" instead of "alternative ". Here, the proposed method alternates between steps instead of a different optimization.

$\textbf{A4:}$ Thanks for the suggestion.

$\textbf{Q5:}$ An important metric is missing in the experiments, i.e. computational cost. It would be better for the authors to provide more results in terms of Bit-Operations (BOPs) [1][2] since BOPs is an important metric to measure the computational cost.

$\textbf{A5:}$ Thanks for the valuable suggestion. We will add the BOPs in the experimental results in the manuscript. Due to the space limitation, we only show part of the results here.

| Model | Dataset | Method | w-bit | a-bit | BOPs(G) | Top-1 Accuracy |
| :--: | :--: | :--: | :--: | :--: |  :--: | :--:|
|ViT-B| ImageNet| Baseline | 32 | 32 | --- | 77.91 |
|ViT-B| ImageNet| Percentile | 6 | 6 | 1995 | 71.58 |
|ViT-B| ImageNet| Proposed | 6 MP | 6 MP | 1996 | 75.26 |
|ViT-B| ImageNet| Percentile | 8 | 8 | 3546 | 74.10 |
|ViT-B| ImageNet| Proposed | 8 MP | 8 MP | 3549 | 76.98 |

$\textbf{Q6:}$ In Section 4.1, the authors randomly sample a subset of the dataset for calibration. However, the effect on the number of samples is unclear. It would be better for the authors to provide more results in terms of the different number of samples.

$\textbf{A6:}$ Thanks for the constructive suggestion. We have conducted experiments about the different number of samples and the results are shown in the following table. The experiments are conducted with ViT-B model on ImageNet dataset.

| Number of samples | 10 | 100 | 1000 | 2500 | 5000 |
| :--: | :--: | :--: |  :--: | :--:| :--: | 
| Top-1 Accuracy | 24.56% | 57.39% | 75.81% | 75.92% | 75.90% |

 Thanks for the encouraging and helpful feedbacks. Here are the reponses to the concerns.

$\textbf{Q1:}$ Relatively weak baselines such as ViT and DeiT are used. I understand the rebuttal time is limited, thus it is optional but better that the author conduct experiments based on a stronger baseline such as Swin Transformer [1]. 

$\textbf{A1:}$ Thanks for the positive comment and due to the limited time, we only conduct some experiments on Swin transformer. The results are shown in the following table.

| Model | Method | W-bit | A-bit | Top-1 Accuracy |
| :--: | :--: | :--: |  :--: | :--:|
| Swin-S | --- | 32 | 32 | 83.0 |
| Swin-S | Percentile | 8 | 8 | 79.56 |
| Swin-S | Proposed | 8 | 8 | 82.31 |
| Swin-B | --- | 32 | 32 | 83.3 |
| Swin-B | Percentile | 8 | 8 | 80.25 |
| Swin-B | Proposed | 8 | 8 | 83.02 |


					
[1] Swin transformer: Hierarchical vision transformer using shifted windows.

$\textbf{Q2:}$ Some typos: Line 143. The choice of … is-> the choices of … are 

$\textbf{A2:}$ Thanks for the reminding and we will correct the typos in the manuscript.


This paper proposed quantization of the output of a transformer network, to reduce memory and computation requirement.

The method proposes to quantize 
1. the MLP portion of the transformer using the Pearson correlation coefficient  as a criteria 
2. the self-attention component via a ranking loss
3. a bias correction unit
4. A PCA-based method is used to combine (1 & 2), referred to as mixed-precision (MP)

The method is tested on several popular transformer models (ViT, DeiT, DETR), and the datasets CIFAR-10, CIFAR-100 and ILSVRC-2012 ImageNet for classification, and CoCo for detection, showing the proposed quantization method reaches close to the full-precision accuracy. It also improves upon a handful of other quantization methods for the specific 8-bit and 6-bit size ( percentile-based method, EasyQuant, Bit-Split).
  
The main claim to novelty here is that previous work has not quantized transformers, and that a special rank quantization technique is required for the self-attention component. However the use of rank-quantization for self-attention layers, as opposed to standard correlation used for MLP here, is not tested or justified. Is quantization based on rank actually superior to other methods (e.g. similarity using the Pearson correlation coefficient?)

Why not use the standard Spearman correlation coefficient to evaluate rank, which is identical to the Pearson used here except on ranks?

A large number of hyper-parameters are manually chosen, presumably to favor the authors method: α and β are set to 0.5 and 1.2, trade-off parameter γ and the threshold θ in Eq. (13) are set to 0.1 and 0.2, the maximum iteration is set to 20 if not mentioned specifically. For mixed-precision, we utilize {4,5,6,7,8} and {6,7,8,9,10} bits while the target bit-width are 6 bit and 8 bit.
   How fair is this for the three other methods used in comparison ( percentile-based method, EasyQuant, Bit-Split)? Have the authors made similar efforts to optimize other methods?

In the big picture, quantization is one possible post-processing method for the goal of reducing memory and computation, others include distillation (which the authors specifically do not use), various projection methods (e.g. fast-food transform [a]), pruning, etc.
   It is not clear why quantization should be used instead of these many methods for the same goal, nor why it should not be used in combination.
   It is not clear wether the three methods compared against (percentile-based method, EasyQuant, Bit-Split) represent a comprehensive collection of alternative quantization methods.

Yang, Z., Moczulski, M., Denil, M., De Freitas, N., Smola, A., Song, L., & Wang, Z. (2015). Deep fried convnets. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1476-1483).
 See previous comments and suggestions.

This submission presents an effective post-training quantization algorithm for reducing the memory storage and computational costs of vision transformers. The main contribution is a ranking loss introduced into the conventional quantization objective to keep the relative order of the self-attention results after quantization. Moreover, it explores a mixed-precision quantization scheme by exploiting the nuclear norm of the attention and output feature maps.   Originality:
The manuscript proposes a post-training quantization for vision transformer. The task is urgently needed for deployment in industrial environments. The related work is cited adequately, but more analysis about the relationship between the quantization of BERT and ViT should be provided.

Quality:
The idea of the manuscript is somewhat innovative, but the method part is not very clear. The quantization model with mixed-precision can get considerable results but more comparison between other quantization methods used on BERT should be given.

Clarity:
The method is not very clarified and some equations are very confusing, especially the strategy for obtaining different bit-widths is not clear. Figure 2 has no clear explanation. In addition, the motivation for introducing rank-loss is best to provide more sufficient evidence. 

Significance: 
The goal of the paper is significant in practical applications and the results demonstrate the method is validation. However, the method is not clear introduced, and the followers are hard to do further study on this method.
 The limitations of the method is not provided

This paper proposes a mixed-precision post-training quantization method for vision transformer, which introduces the similarity-aware and ranking-aware quantization for the linear layer and self-attention layer, respectively. The experimental results are better than the methods for CNN quantization.  -	Basically, this is a novel and effective method for the compression for the vision transformer. 
-	The proposed ranking-aware loss for the self-attention layer benefits for the quantization and the mixed-precision quantization further promotes the performance of quantized transformer. 
-	The paper is well written and organized.
 - Relatively weak baselines such as ViT and DeiT are used. I understand the rebuttal time is limited, thus it is optional but better that the author conduct experiments based on a stronger baseline such as Swin Transformer [1].
- Some typos: Line 143. The choice of … is-> the choices of … are

[1] Swin transformer: Hierarchical vision transformer using shifted windows.



The authors propose an effective post-training quantization algorithm to reduce the model size and computational overhead of the vision transformers. Specifically, the authors formulate quantization as an optimization problem for finding the optimal quantization intervals. To this end, the authors first introduce similarity-aware loss to maximize the similarity between the quantized and original feature maps. Then, the authors propose ranking loss to keep the relative order of the self-attention results after performing quantization. Moreover, the authors further devise bias correction to reduce the accumulated quantization error. Experimental results on CIFAR-10/100 and ImageNet demonstrate the effectiveness of the proposed method. However, some important details are missing and experiments are not sufficient. If the authors can solve these issues, I would be happy to change my score. My detailed comments are as follows.   **Contribution**: 
1.	The authors propose to maximize the similarity between the quantized and original feature maps by the similarity-aware loss.

2.	The authors empirically find that the relative order of the attention maps is changed after quantization, resulting in significant performance degradation. To address this, the authors propose a ranking loss to preserve the relative order of attention values.

3.	The authors propose bias correction to reduce the accumulated error caused by quantization.

**Questions and points needed to be improved**:
1.	In Section 3.2, the motivation of using the Pearson correlation coefficient to measure the similarity between the original and quantized feature maps is not clear. Why not use the Euclidean distance or the cosine similarity?

2.	The effect of alternating searching is not discussed. To demonstrate the effectiveness of the proposed alternating searching, it would be better for the authors to provide more results without the alternative searching, which would strengthen the paper.

3.	In line 180, what does $\Delta_l$ denotes? What do $\alpha$ and $\beta$ denote? More explanations are required.

4.	Please use "alternating" instead of "alternative ". Here, the proposed method alternates between steps instead of a different optimization. 

5.	An important metric is missing in the experiments, i.e. computational cost. It would be better for the authors to provide more results in terms of Bit-Operations (BOPs) [1][2] since BOPs is an important metric to measure the computational cost.

6.	In Section 4.1, the authors randomly sample a subset of the dataset for calibration. However, the effect on the number of samples is unclear. It would be better for the authors to provide more results in terms of the different number of samples.

**Reference**:

[1] Differentiable Joint Pruning and Quantization for Hardware Efficiency. ECCV 2020.

[2] Single Path One-Shot Neural Architecture Search with Uniform Sampling. ECCV 2020.
 Please discuss the limitations of the proposed method.

Meta-review: This paper is solving an important problem for efficient vision transformer deployment in industrial environments. The goal of the paper is significant in practical applications. However, reviewers are concerned about the novelty is not enough and the method part is not clear; A large number of hyper-parameters are manually chosen, and the baselines and comparison with other approaches is weak. This paper is recommended for rejection. 

Decision: accept