Post-Training Quantization for Vision Transformer
Authors: Zhenhua Liu
Yunhe Wang
Kai Han
Wei Zhang
Siwei Ma
Wen Gao
Venue: NeurIPS 2021 Poster (2021)
Keywords: Post-training
vision transformer
ranking loss of self-attention
mixed-precision

Abstract: Recently, transformer has achieved remarkable performance on a variety of computer vision applications. Compared with mainstream convolutional neural networks, vision transformers are often of sophisticated architectures for extracting powerful feature representations, which are more difficult to be developed on mobile devices. In this paper, we present an effective post-training quantization algorithm for reducing the memory storage and computational costs of vision transformers. Basically, the quantization task can be regarded as finding the optimal low-bit quantization intervals for weights and inputs, respectively. To preserve the functionality of the attention mechanism, we introduce a ranking loss into the conventional quantization objective that aims to keep the relative order of the self-attention results after quantization. Moreover, we thoroughly analyze the relationship between quantization loss of different layers and the feature diversity, and explore a mixed-precision quantization scheme by exploiting the nuclear norm of each attention map and output feature. The effectiveness of the proposed method is verified on several benchmark models and datasets, which outperforms the state-of-the-art post-training quantization algorithms. For instance, we can obtain an 81.29% top-1 accuracy using DeiT-B model on ImageNet dataset with about 8-bit quantization. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/VT-PTQ. 

Post- Post-Trainin Quantization for Vision Transformer
Zhenhua Liu Yunhe Wang"; Kai Han', Wei Zhang', Siwei Ma 13, Wen Gao
'School of Electronic Engineering and Computer Science, Peking University Huawei Noah's Ark Lab 'Peng Cheng Laboratory 1iu Ziu-zh@pku.ede edu. cn, {yunhe. wang, kai. han,, wz. (e..zha)) com, (swma, wgao) pku..edu.cn
Abstract
Recently, transformer has achieved remarkable performance on a variety com- puter vision applications. Compared with mainstream convolutional neural net- works, vision transformers are often sophisticated architectures for extracting powerful feature representations, which are more difficult to be developed bile devices. In this paper, we present an effective post-training quantization algo- rithm for reducing the memory storage and computational costs of vision transform- ers. Basically, the quantization task can be regarded as finding the optimal low-bit quantization intervals for weights and inputs, respectively. To preserve the function- ality of the attention mechanism, we introduce ranking loss into the conventional quantization objective that aims keep the relative order of the self-attention sults after quantization. Moreover, we thoroughly analyze the relationship between guantization loss of different layers and the feature diversity, and explore a mixed- precision quantization scheme by exploiting the nuclear norm of each attention map and output feature. The effectiveness of the proposed method is verified on several benchmark models and datasets, which outperforms the state-of-the-ant post- training quantization algorithms. For instance, we can obtain an 81.29% top-1 accu- racy using DeïT-B model ImageNet dataset with about 8-bit quantization. Code will be available at https://gitee..om/t cow//innppreemoessseeeenneeeeeee PTQ.
Introduction
Following the applications in Natural Language Processing (NLP) tasks, transformer-based models have shown great power in various Computer Vision (CV) tasks, such image classification [1T 26).
object detection 4 39 and image super-resolution [5]. Pre-trained with large-scale data, these models usually have hundreds of millions of parameters. For instance, there 307M parameters and 64G FLOPs in the ViT-L model, which is both memory and computation expensive during inference. This brings great challenges for these models run on resource-constrained devices like mobile phones and intelligent cars. Besides, the real-tine pal--time computer vision applications that integrate transformer-bae based models have meet low latency requirements to achieve a high quality customer experience. Therefore, the model compression technology of transformer-basee models is urgently needed for deployment in industrial environments.
and weight decomposition 37 14 32 compresses neural network by using lower bit-width for weight values without changing the model architecture, which is particularly useful for carefully- designed network architectures like transformers. Quantizing both weights and inputs can speed up inference by tuning floating-point operations into integer or bit operations. There have been some
Among various compression methods like pruning
16
29
19
quantization method 9 38 27
"Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

training aware quantization approaches for transformer-basee models in NLP (e.g., BERT (T7) B4 However, these methods are not designed for computer vision tasks and usually need additional training or fine-tuning. Furthermore, in some scenarios, the entire training data is not available to optimize the quantization model and the training costs for edge devices are intolerable.
23 35 22.
Post-training quantization [24] is kind of efficient model compression technique, which can directly quantize neural network models without fine-tuning. Most of the existing post-training quantization methods are designed for convolutional neural networks 3 30 or recurrent neural networks [36]. These methods do not take the character of vision transformer into consideration (e.g., the attention mechanism do not exist in CNNs), which are not perfectly suitable for quantizing vision transformer. However, vision transformers are showing stronger performance in large variety of computer vision tasks. Thus, we are motivated to explore the post-tre -ossttraiiin quantization for them to reduce the costs on memory and computation.
In this paper, we study the post-training quantization method for vision transformer models with mixed-precision for higher compression and speed-up ratios. The quantized process in the transformer is formulated as an optimization problem for finding the optimal quantization intervals. Specially, our goal to maximize the similarity between the full-precision and quantized outputs in vision transformers. To better preserve the functionality of the attention mechanism, we thoroughly analyze the difference between attention layers and conventional layers such MLP. Then, ranking loss is introduced to keep the relative order of attention values. Furthermore, we propose to determine the bit-widths of each layer according the feature diversity, i.e,, the nuclear norm calculated by the attention map and output features. We alternatively search the quantization intervals of weights and inputs in all layers to obtain the best quantization results. In addition, bias correction is introduced to diminish the cumulative quantization error. Experimental results on several benchmarks demonstrate the effectiveness of our algorithm for achieving better performance over the state-of-art post-training quantization approaches.
Related Works
Here, we reviews the transformer-based models designed for computer vision tasks. And the training-aware quantization schemes proposed for BERT and post-training quantization algorithms are summarized and analyzed.
2.1 Vision Transformer
Inspired by the major success of transformer architectures in the field of NLP, researchers have recently applied transformer to computer vision (CV) tasks [13.. Chen et al. [6] trained a sequence transformer to auto-regressively predict pixels, achieving results comparable to CNNs on image classification tasks. Another vision transformer model is VIT, which applies a pure transformer directly to treat image patches as the sequences. Recently proposed by Dosovitskiy er al. [1]. it has achieved great performance multiple image recognition benchmarks. Touvron et al. (26] produce competitive convolution-fres transformers by training on ImageNet only while introducing a teacher-student strategy specific to transformers. In addition basic image classification, transformer has been utilized to address a variety of other computer vision problems, including object detection [4] (39]
semantic segmentation [5), image processing [5]. and video understanding (5] Han al. [15 proposed a -annsfoomeer----Taasffoooon
(TNT) model for modeling both patch-level and pixel-level representation. Tang al. (25) proposed augmented shortcut scheme improve the performance of vision transformers. Thanks to its exceptional performance, more and more researchers are
proposing transformer- models for wide range of computer vision tasks.
2.2 Compression of Transformer in NLP
Owing the remarkable performance of BERT in many NLP tasks, many researchers have tried to compress the model to reduce the memory and computation complexity of BERT. Wu et al. [3I proposed Short Range Attention (LSRA) to conduct transformer on edge devices, where one group of heads specializes in the local context modeling (by convolution) while another group specializes in the long-distance relationship modeling. In (22 34 8-bit quantization is successfully applied to Transformer-bast based models with comparable performance as the full- full-precision baseline. However,

---

quantizing these models to ultra low bits (e.g., or 2 bits) can be much more challenging due to significant reduction in model capacity. To avoid severe accuracy drop, more complex quantization methods, like mixed-precioe precision quantization 24 K3 and product quantization (PQ) [2] are used. In addition, Zhang et al. [55] propose TernaryBERT, which use both approximation-basee and loss- aware ternarization methods and empirically investigate the ternarization granularity of different parts of BERT. Moreover, to reduce the accuracy degradation, they also leverage the knowledge distillation technique. Bai et al. [1] further push BERT quantization to the limit with weight binarization. They propose ternary weight splitting, which initializes the binary model by equivalent splitting from a half-sized temnary network. However, these methods are not designed for computer vision tasks and need additional training or fine-tuning.
2.3 Post-Training Quantization
There are many works focusing on developing post-training quantization methods, without any training or fine-tuning. In particular, Yoni et al. [8] propose the OMSE method to optimize the Lg distance between the quantized tensor and the original tensor. Moreover, Ron et al. [ present the so-called ACIQ method to analytically compute the clipping range, as well as the per-channel bit allocation for NNs. Zhao et al. (36] propose outlier channel splitting (OCS) method to solve the outlier channel problem. Wang et al. (28) propose a Bit-Split and Stitching framework for lower-bit post-training quantization and an Error Compensated Activation Quantization method, which could lower the quantization error for activations. Nagel et al. (20) propose AdaRound, a weight-rounding mechanism for post-training quantization that adapts the data and the task loss. By approximating the task loss with a Taylor series expansion, the rounding task is posed as a quadratic unconstrained binary optimization problem. The recent work of (21) propose Data-Free Quantization, which further pushes post-traine trainini quantization to zero-shot scenarios, where neither training nor testing data are accessible during quantization. Cai et al. 3 introduce ZeroQ, which distills an input data distribution to match the statistics in the batch normalization layers of the model and utilize a Pareto Frontier method select automatically the bit- tit-precision configuration of mixed peeecdiieiiioonnn settings. These methods are designed for CNNs and do not consider the unique structure of vision transformers such as self-attention layers.
3 Methodology
In this section, we elaborate on the proposed mixed- preecd-oreiiioon post-training quantization scheme for the vision transformer. The similarity- quantization for linear layers and ranking- aware quantization for self-attention layers are presented. In addition, the bias correction method for optimization and the mixed-precision quantization based on nuclear norm of the attention map and output feature are introduced.
3.1 Preliminaries
A standard transformer receives input as 1-D sequence of token embeddings, so the vision transformers usually reshape the image E #x WX xC into sequence of flatted 2D patches IP e B=x(P2-C) Here, H and W are the height and width of the original image and (P; P) is the resolution of each image patch, n = -p vision transformers use constant widths through all of its layers, trainable linear projection maps each vectorized patch to the model dimension d. Thus, the input to the first transformer layer is:
HW
is then the effective sequence length for the transformer. Usually, the
X [Telaesi 1 F ..... ; W R| + E
where W E R (P2 c)xd E [(+1)xxd
(2)
standard transformer layer includes two main modules: Multi-Head Self Attention (MSA) and Multi-Layer Perceptron (MLP) module. For the 1-th transformer layer, suppose the input to it is X e R the attention scores computed by the dot product of queries and keys can be formulated
as:
A, Q.K. = X.W? W.W?Wk RT X..
(3)
3

---

2403
Multi Layer
Similarity
2200
Bit-width
Percectron
Interval
Output feature
Layer Norm
MSA
6550
2520
Bit width
Muiti-head Self Attention
Interval
Ranking loss
5220
1520
Layer Norm
Nuclear norm
Attention map
Palch Em sridediings
Figure 1: Diagram of the proposed mixed- precision post-training quantization method for vision transformer. The similarity-awarr and ranking- aware quantization are designed for finding the optimal quantization interval of the linear operations and self-attention layers. The bit-widths of transformer layers are determined based on the nuclear norm of the attention map and the output feature.
Then the softmax function is applied the normalized scores to get the output and the output of the multi-head self attention module is:
MSA(X:) = Softmax
Ay) WP.
(4)
The MLP module contains two linear layers parameterized by W e Rdddd, E Rd and 2 E Rd x b IRd respectively, where d is the number of neurons in the intermediate layer of MLP. Denote the input to MLP Zi Rnxd the output is then computed
MLP(Z,) GeLU(Z, +b' +
Combining Eq. and the forward propagation for the I-th transformer layer can be formulated as:
Z X; MSA(LN(X:)), X Z + MLP(LN(Z))),
(7)
where LN represents the layer normalization.
The most computational costs of vision transformer lie on the large matrix multiplication in MSA and MLP module. Following the mainstream quantization methods for CNNs 21). we quantize all the weights and inputs involved in matrix multiplication. For weight quantization, we quantize the weights Qº WK W WO W W in Eq. (4) and (5) for all transformer layers, as well as the linear embedding F in Eq. (I.. Besides these weights, we also quantize the inputs of all linear layers and matrix multiplication operations. Following the methods in [22 35). we do quantize the softmax operation and layer normalization, because the parameters contained in these operations are negligible and quantizing them may bring significant accuracy degradation.
3.2 Ranking-Aware Post-Training Quantization
For post-training quantization, we need to restrict the floating flaatin-numbbess to finite set of values. The choice of quantization intervals is critical for quantization and one popular option is to use uniform quantization function, where the data range is equally split:
A Y Clamp(Round Y.
-2 200-1,20111 1-11-0 1),
(8)

---

where A is the quantization interval, b is the quantization bit-width and Y is a tensor representing weights or inputs. Clamp denotes that elements in the tensor that exceed the ranges of the quantized domain are clipped.
For the layers in vision transformer, the original output can be computed O X, Wi The uniform quantization for the weights and inputs and the corresponding dequant operation can be described as:
O MAA (X)) (Wi) A
where O, denotes the outputs of the quantized layer. From Eq. (8) and Eq. (9 it can be seen that the quantization intervals actually control the clamping thresholds in quantization process, which affects the quantization results great extent. Therefore, the quantization intervals for both weights AW and inputs A, where inputs X₁ are generated from a given calibration dataset D with N samples. Specifically, the calibration dataset much less than the common training dataset.
motivated to focus on optimizing
The self- attentienoo layer is the critical component of the transformer since it can calculate the global relevance of the features, which makes the transformer unique from the convolutional neural networks. For the calculation of self-attention (Eq. B), we empirically find that the relative order of the attention map has been changed after quantization as shown in Fig [ which could cause a significant perfor- mance degradation. Thus, ranking loss is introduced to solve this problem during the quantization process:
h w-1 12 ΣΣΣ p=1pppppt
Lranking
@(((App - Akg) sign(AAp Akg)),
(10)
in which (mm) (0 m)+ hinge function with parameter 0, (h, w) are the size of matrix A. Given pair of examples, the loss is 0 only when the examples are in the correct order and differed by a margin.
Then we combine the ranking loss with the similarity-aware quantization, and the overall optimization goal can be described as:
N r(O; s.t. AW AX E R+
min 7. Cranking AA
(11)
N 1
where Crank denote the pairwise ranking based loss function, and y is the trade-off hyper-parameter. T(0) O denotes the similarity metric between the original and quantized output feature maps, which can be formulated as:
_,,00,-000,--
r 0)
(12)
where the Pearson correlation coefficient is adopted the measurement for the similarity since it subtracts the mean value of the data and can be more representative for the similarity between the distribution of quantized and original feature maps.
To solve the above optimization problem, we present a simple but efficient alternative searching method for the uniform quantization of transformer layers. Firstly. the quantization interval of inputs AX is fixed, and the quantization interval of weights A," optimized for adjustment. Secondly, AW is fixed, and AX is optimized to fine-tune the quantization interval of the inputs. AWW and A are alternately optimized until the target function converges or the maximum iteration is exceeded. Moreover, for fast convergence, A and A are initialized in terms of the maximum of weights or inputs respectively. For the search space of AW and AX" we linearly divide interval of [aAi, A into C' candidate options and conduct a simple search strategy on them.
To further reduce the biased error for the outputs raised by quantization, bias correction method is then introduced after each search iteration. Suppose the quantization error of weights and inputs are defined as:
Bias Correction
x VAX (X) x X, W = VAN (W) AW W.
(13)
(14)
5

---

If the expectation of the error for output is not zero, then the mean of the output will change. This shift in distribution may lead to detrimental behavior in the following layers. We can correct this change by seeing that:
[ = E + Bie X B| W EEE
(15)
Thus, subtracting the expected error on the output from the biased output ensures that the mean for each output unit is preserved. For implementation, the expected error be computed using the calibration data and subtracted from the layer's s bias parameter, since the expected error vector has the same shape as the layer's output.
3.3 Nuclear Norm Based Mixed-Precision Quantization
Different transformer layers are attending to different structures, and is expected that they exhibit different sensitivity. Thus, assigning the same number of bit-widths to all the layers is sub-optimal. As result, we explore mixeddppeeisse quantization, where more bits are assigned to more sensitive layers in order to retain performance. Considering the unique structure of transformer layer, we assign all the operations the MSA or MLP modules with the same bit-width. This will also be friendly to the hardware implementation since the weights and inputs are assigned with the same bit-width.
Singular value decomposition (SVD) is an important matrix decomposition approach in linear algebra. takes rectangular matrix of gene expression data, whose formulation can be written as
M UEV, tr(M) Σ Ei
(16)
where the diagonal entries Eg of Σ are known as the singular values of M. And the nuclear norm tr the sum of singular values, which represents the data relevance of the matrix. In this paper, we propose to estimate the sensitivity of the transformer layer with the nuclear norm of the attention map in the MSA module and the output feature in the MLP module. The nuclear norm can be used to reduce the search space of the mixed-preciste that are more sensitive and vice versa. Inspired by the method in [1O), we utilize a Pareto frontier approach to determine the bit- width. The main idea is sort each candidate bit-width configuration based on the total second-order perturbation that they cause, according to the following metric:
settings, while using higher bit-widths for layers
LABA
LMSA Σ
Σ Ω Σ tr(A:) |A
A.E
1 0
(17)
Given a target model size, the candidate bit-width configurations are sorted based on their value and choose the bit-width configuration with minimal Ω. The nuclear norm of the attention map and output feature each transformer layer are shown in Figure As we can see, they are various for different transformer layers.
4 Exprimental results
In this section, we evaluate the performance of the proposed post-training quantization scheme on vision transformer model for image classification (ViT [II and DeiT (26)) and object detection (DETR (4). To the best of our knowledge, there no published work done on post-training quantization of vision transformer at this point, methods for CNNs as described the papers by ourselves. shown that the proposed method outperforms the conventional post-training quantization methods. Moreover, extensive experiments of ablation study have shown that the proposed similarity-ave ranking-awar quantization and bias correction method are beneficial for the post-training quantization of vision transformer.
implement recent post-training quantization
4.1 Implementation details
Datasets For image classification, the CIFAR-10, 10, CIFAR-100 and IISVRR--201 ImageNet (we refer to it ImageNet in what follows) datasets are utilized to evaluate the quantization performance.

---

The CIFAR-10 dataset consists of 50K training images and 10K test images, which are labeled for 10 classes. And CIFAR- --0 dataset also contains 50K training images and 10K test images, expect that they are labeled for 100 classes. ImageNet dataset contains 1.2 million training images and 50K' validation images labeled for 1,000 categories. For object detection task, the COC02017 dataset is utilized to evaluate the quantization performance, which contains 118K training images and 5K validation images.
Experimental settings We randomly select 100 images for CIFAR-10 10 and CIFAR-100 dataset and 1000 images for ImageNet and COCO2017 dataset from the training dataset the calibration dataset. For the hyper-parameter, a and B3 are set to 0.5 and 1.2 for all the experiments. The maximum iteration is set to 20 if not mentioned specifically. For mixed-precision, we utilize (4,5,6,7,8) and (6,7,8,9,0) bits while the target bit-width are 6 bit and 8 bit, respectively.
Baseline For image classification, we evaluate our quantization method on two popular vision transformer implementation: ViT [M] and DeiT [26]. The ViT-B, ViT-L., DeiT-S, DeiT-B are adopted as the baseline model, whose top-1 accuracy ImageNet dataset are 71.58%, 71.48%, 79.8%, 81.8% respectively. For a fair comparison, we utilize the official implementation of DeiT and do not use other techniques like knowledge distillation. For object detection, the DETR model using ResNet-50 backbone is adopted, which achieves a 42.0 mAP on COCO dataset.
4.2 Results and Analysis
Image classification The experimental results are shown in Table T We firstly evaluate the proposed method on ViT-B and VIT-L model. ViT-B model is 12-layer transformer with 12 heads and 768 embedding dimension. For the similar quantized model size, the proposed method outperforms percentile-based method [18] by 3.35% and 2.07% on CIFAR-10 dataset, respectively, And it worth noting that the performance of the proposed 8-bit model is comparable to the full- precision model. The proposed method obtains the similar performance on CIFAR-100 dataset and ImageNet dataset, while the average gains are 2.95% and 3.28% respectively. Moreover, the performance of the proposed 6-bit model is even better than the 8-bit percentile-basee model, which means that the proposed method can save about 25% memory and 44% computational costs than conventional post-training quantization method.
VYT-L model is much larger network which consists of 24 transformer layer with 16 heads and 1024 embedding dimension. It contains 307M parameters, however its performance is worse than ViT-B. We also test the quantization methods on CIFAR-IO, CIFAR- and ImageNet dataset. As shown in Table [ the performance of the proposed method outperforms the percentile-based method by large margin. is worth mentioning that the 8-bit proposed model is even better than full-precision model CIFAR- 10 dataset and comparable to the call-precision model on CIFAR- 100 dataset and ImageNet model. It is supposed that there is more redundancy in the ViT-L model and the performance degradation of quantization is less than that of ViT-B model.
The architecture of DeiT network is the same as ViT, expect that DeiT utilizes the data augmentation and regularization strategies. As result, the performance of DeiT is much better than VIT. Among the models, ViT-S consists of 12 transformer layers with heads and 384 embedding dimension. As we can see, the percentile-based method largely hurts the performance while the accuracy losses of 6-bit and 8-bit models are 9.31% 31% and 5.82%. EasyQuant 30 a popular simple post-training quantization method which improves the performance loss to 6.54% and 3.21% %, respectively. Bit-Split proposes a bit splitting and stitching framework (28). while the Top-1 accuracy degradation are 5.76% and 2.74%. In comparison, the Top-1 accuracy losses of the proposed post-training quantization scheme are 5.22% and 2.33% respectively. In addition, when the mixed- -eexed-precisoo is conducted, the 8-bit quantized model can achieve 78.09% Top-1 I accuracy.
DeiT-B is much larger network than DeîT-S, which consists of 12 transformer layers with 12 heads and 768 embedding dimension. As shown in Table the Top- I accuracy of percentile-bee beesentiee-bseed are 73.99% and 75.21% % when quantized to 6-bit and 8-bit respectively. And the proposed scheme improves the performance of the quantized model to 77.47% and 81 .99%. Another point is that the aceuracy losses of DeiT-B are smaller than DeïT-S and we think that this is because DeiT-B consists of more parameters and is more representive when quantized to the same bit-width.
7

---

Table 1: Comparison on the performance of proposed mixed-precision post-training quantization method with conventional quantization method for image classification. 'MP" represents for mixed- precision.
Model
Dataset
Method
W-bit
A-bit
Model size (MB)
Top-1 Accuracy
Baseline
32
32
344
98.13
Percentile
64.5
93,48
CIFAR-I0
Ours
MP
64.6
96.83
Percentile
86.2
95.72
Ours
MP
86.0
97.79
Baseline
32
32
344
87.13
Percentile
64.5
80.56
VIT-B
CIFAR-100
Ours
6 MP
MP
64.4
83.99
Percentile
86.2
83.28
Ours
MP
8 MP
86.5
85.76
Baseline
344
77.91
Percentile
64.5
71.58
ImageNet
Ours
MP
MP
64.8
75.26
Percentile
86.2
74.10
Ours
MP
MP
86.5
76.98
Baseline
32
32
1228
97.86
Percentile
230.2
93.27
CIFAR- 10
Ours
MP
232
96.09
Percentile
307
94.19
Ours
MP
305.8
97.90
Baseline
32
32
1228
86.35
Percentile
230.2
80.54
VIT-L
CIFAR-100
Ours
6 MP
MP
231
83.69
Percentile
307
83.01
Ours
MP
MP
307.8
85.83
Baseline
1228
76.53
Percentile
230.2
71.48
ImageNet
Ours
MP
MP
231.6
75.46
Percentile
307
75.17
Ours
MP
MP
306.4
76.41
Baseline
32
88
79.8
Percentile 18
16.5
70.49
Easy
30
16.5
73.26
Bit-Split 228
16.5
74.04
Ours
16.5
74.58
DeiT-S
ImageNet
Ours
6 MP
MP
16.6
75.10
Percentile
22.0
73.98
EasyQuant 30
22.0
76.59
Bit-Split 22
22.0
77.06
Ours
22.0
77.47
Ours
MP
MP
22.2
78.09
Baseline
32
32
344
81.8
Percentile 18
64.5
73.99
EasyQuant
30
64.5
75,86
Bit-Split 28
64.5
76.39
Ours
MP
43.6
75.94
Ours
64.5
77.02
DeiT-B
ImageNet
Ours
MP
64.3
77.47
Percentile
18
86.0
75.21
30
86.0
79.36
Bit-Split
28
8
86.0
79.42
Ours
86.0
80.48
Ours
MP
8 MP
86.8
81.29

---

Table 2: Comparison on the performance of proposed mixed -exeedppreision post-training quantization
method with conventional quantization method for DETR. 'MP' represents for mixed -eeed-precisio..
Model
Dataset
Method
W-bit
A-bit
Model size (MB)
mAP
Baseline
32
32
164
42.0
Percentile
30.75
37.5
EasyQuant
130
6
30.75
39.0
Bit-Split
(28
30.75
38.9
Ours
6
30.75
40.1
DETR
COCO2017
Ours
MP
MP
30,98
40.5
Percentile 18
8
8
41.00
38.6
EasyQuant
30
41.00
40.4
Bit-Split 28
8
41.00
40.6
Ours
41.00
41.2
Ours
MP
MP
41.64
41.7
Object Detection In order to show the generalization capability of proposed method, we also evaluate our method for object detection task using DETR 4.. The experimental results are shown in T As we can see, the proposed method outperforms percentile-based method, EasyQuant, Bit- Split by 2.6, 1.1 and 1.2 mAP for 6-bit quantization, respectively. The mixed peeeeddoocisionn quantization further boost the performance of the method. For 8-bit quantization, the mAP of the proposed mixed -ixeddprecisson quantization method comparable to the full full-precision model.
4.3 Ablation study
In this section, evaluate the effect of the proposed similarity aware quantization module, ranking- aware quantization module, bias correction method and the sexeed-recciion method. The experimen- tal results shown in Table while experiments model. As we can see, the Top-1 accuracy of only using similarity- aware quantization 75.42% which is inferior the full-precision model and using ranking-aware quantization loss and bias correction method can improve the performance by 0.52% and 0.39%. It is worth noting that the nuclear norm based mixed-precision since it considers the variant sensitivity of different layers.
conducted on ImageNet dataset with VIT-B
further promote the performance of the quantized model,
It is also shown that the Top-1 accuracy of using the similarity-awar aware preccdppeeision quantization is 76.26%. And the ranking-awar quantization and bias correction can still boost the performance in this case. Besides, the performance of the 8-bit quantized model using all the proposed methods is 76.98 76,98, which comparable to the full precision model.
Table 3: Ablation study of the proposed similarity aware quantization module, ranking-aware quantization module, bias correction and mixed mixed-precision method.
Model
Similarity Ranking
Bias Correction
Mixed-Precision
Model size (MB)
Top-1 Accuracy
344
77.91
X
86.2
75.42
X
86.2
75.94
V
X
86.2
75.81
ViT-B
X
86.2
76.49
V
86.5
76.26
V
86.5
76.61
86.5
76.53
86.5
76.98
also compared the performance of the proposed method with the hessian- mixed-precision method, where the experiments are conducted with ViT-B on ImageNet dataset. As we can see in Table the proposed method achieves similar result while the consuming computation time much less than hessian-based approach.

---

Table 4: Performance comparison with hessian- mixed=precision method.
Method
W-bit
A-bit
Model size (MB)
Computation time (s)
Top-1 Accuracy
Hessian- bassean-basd
8
86.7
754.6
77.01
Ours
8 MP
8 MP
86.5
53,1
76.98
5
Conclusion
In this paper, have developed novel post-training quantization scheme for vision transformer, in which the bit-widths of each layer are variant based on the nuclear norm of the attention map and output feature in the transformer layer. To solve the optimization problem of the quantization, propose to search the optimal quantization interval for remaining the similarity between the quantized and original feature maps. In addition, and conventional layers and introduce a ranking loss to keep the relative order of the attention values. Specifically, the bias correction is employed reduce the accumulated quantization error. Last but not the least, the optimal quantization interval for each transformer layer is carefully optimized using an alternative searching strategy. Experimental results show that the proposed method outperforms the conventional post-training quantization method by large margin in terms of both network accuracy and memory costs.
thoroughly analyze the different between attention layers
Acknowledge
This work partly supported by the National Natural Science Foundation of China (61961130392) and PKU-Baidu Fund(2019BD0033 Besides, High-Performance Computing Platform of Peking University is acknowledged.
References
[1] Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization. arXiv preprint arXiv:2012.1570.
2020.
[2] Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. Post-training 4-bit guantization of convolution networks for rapid-deployment. arXiv preprint arXiv: 001..8800...... 2018.
[3] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer.
Zeroq: A novel zero shot quantization framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13169-13178. 2020.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. arXiv preprint
arXiv:2005. 12872, 2020.
[5] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma. Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. arXiv preprint arXiv:2012.00364, 2020.
[6] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691-1703. PMLR, 2020.
[7] Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint 100.vv:80000000.. 2018.
[8] Yoni Choukroun, Eli Kravehik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. In ICCV Workshops, pages 3009-3018, 2019.
[9] M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained tot I or- arXiv
preprint arXiv:1602. 02830, 2016.
[10] Zhen Dong, Zhewei Yao, Yaohui Cai, Daiyaan Arfeen, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trace-weighted quantization of neural networks.
arXiv preprint arXiv: 19t.:9991..333 2019.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.
10

---

An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint
arXiv:2010. 11929, 2020.
[12] Angela Fan, Pierre Stock. Benjamin Graham, Edouard Grave, Rémi Gribonval, Hervé Jégou, and Armand Joulin. Training with quantization noise for extreme model compression. arXiv e-prints, pages ar Xiv--200, 2020.
[13] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao. Chunjing Xu, Yixing Xu, et al. A survey on visual transformer. arXiv preprint arXiv:2012. /2556, 2020.
[14] Kai Han, Yunhe Wang, Yixing Xu, Chunjing Xu, Enhua Wu, and Chang Xu. Training binary neural networks through learning with noisy supervision. In International Conference on PMLR, 2020.
Machine Learning, pages
[15] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv preprint arXiv: 1tt::2/3:.. 103. 00 12, 2021.
[16] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.0014 2015.
[17] Jacob Devlin Ming-Wo Weeg Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186, 2019.
[18] Rundong Li, Yan Wang. Feng Liang, Hongwei Qin, Junjie Yan, and Rui Fan. Fully quantized network for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2810-12111 10-2819, 2019.
[19] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision. pages 2736-2744 2017.
[20] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort.
Up or down? adaptive rounding for post- tost-training quantization. In International Conference on Machine Learning, pages 7197-7206. PMLR. 2020.
[21] Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325-1334, 2019.
[22] Gabriele Prato, Ella Charlaix, and Mchdi Rezagholizadeh. Fully quantized transformer for improved translation. 2019.
[23] Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8815-8821,
2020.
[24] Wonyong Sung, Sungho Shin, and Kyuyeon Hwang. Resiliency of deep neural networks under quantization. arXiv preprint arXiv: 100::/511.0444 2015.
Yehui Tang, Kai Han, Chang Xu, An Xiao, Yiping Deng, Chao Xu, and Yunhe Wang. Aug- mented shortcuts for vision transformers. arXiv preprint arXiv:2106.1 /5911, 2021. [26] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint XrXiv:2012. 12877, 2020.
[25]
[27] Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quantization with mixed precision. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8612-866 8620, 2019.
[28] Peisong Wang. Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training network quantization via spt-sp and stitching. In International Conference on Machine Learning, pages 9847-9856. PMLR, 2020.
[29] Yunhe Wang, Chang Xu, Shan You, Dacheng Tao, and Chao Xu. Cnnpack: Packing convolu- tional neural networks in the frequency domain. In NIPS, volume 1, page 3, 2016.
[30] Di Wu, Qi Tang, Yongle Zhao, Ming Zhang. Ying Fu, and Debing Zhang. Easyquant: Post- training quantization via scale optimization. arXiv preprint arXiv:2006. J66,,69,0..669, 2020. [31] Zhanghao Wu, Zhijian Liu, Ji Lin. Yujun Lin, and Song Han. Lite transformer with long-short range attention. arXiv preprint arXiv:2004. / 1886, 2020.
[32] Yixing Xu, Kai Han, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Learning frequency domain approximation for binary neural networks. arXiv preprint arXiv:2
2021.
[33] Ali Hadi Zadeh, Isak Edo, Omar Mohamed Awad, and Andreas Moshovos. Gobo: Quantizing iitention-based alp models for low latency and energy efficient inference. In 2020 53dd Annual
11

---

IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 811-824 IEEE, 2020.
[34]
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert.
arXiv preprint arXiv:1910.06188 2019.
[35] Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. Ternarybert: Distillation-wware ultra-low bit bert. arXiv preprint arXiv:2009, 12812, 2020.
[36]
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural net- work quantization without retraining using outlier channel splitting. In International conference
on machine learning, pages 7543-7552. PMLR, 2019.
[37]
Zhisheng Zhong, Fangyin Wei, Zhouchen Lin, and Chao Zhang, Ada-tucker: Compressing deep neural networks via adaptive dimension adjustment tucker decomposition, Neural Networks,
110:104-115, 2019.
[38] Chenzhuo Zhu, Song Han, Huizi Mao, and William Dally. Trained ternary quantization. arXiv
preprint 16 1612.0104, 2016.
[39] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr:
Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159,
2020.
12


Reviews:
[{"review_id": "iTEDOni2-n5", "reviewer": "Reviewer_Zj6U", "summary": "This paper proposed quantization of the output of a transformer network, to reduce memory and computation requirement.\n\nThe method proposes to quantize \n1. the MLP portion of the transformer using the Pearson correlation coefficient  as a criteria \n2. the self-attention component via a ranking loss\n3. a bias correction unit\n4. A PCA-based method is used to combine (1 & 2), referred to as mixed-precision (MP)\n\nThe method is tested on several popular transformer models (ViT, DeiT, DETR), and the datasets CIFAR-10, CIFAR-100 and ILSVRC-2012 ImageNet for classification, and CoCo for detection, showing the proposed quantization method reaches close to the full-precision accuracy. It also improves upon a handful of other quantization methods for the specific 8-bit and 6-bit size ( percentile-based method, EasyQuant, Bit-Split).\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe main claim to novelty here is that previous work has not quantized transformers, and that a special rank quantization technique is required for the self-attention component. However the use of rank-quantization for self-attention layers, as opposed to standard correlation used for MLP here, is not tested or justified. Is quantization based on rank actually superior to other methods (e.g. similarity using the Pearson correlation coefficient?)\n\nWhy not use the standard Spearman correlation coefficient to evaluate rank, which is identical to the Pearson used here except on ranks?\n\nA large number of hyper-parameters are manually chosen, presumably to favor the authors method: α and β are set to 0.5 and 1.2, trade-off parameter γ and the threshold θ in Eq. (13) are set to 0.1 and 0.2, the maximum iteration is set to 20 if not mentioned specifically. For mixed-precision, we utilize {4,5,6,7,8} and {6,7,8,9,10} bits while the target bit-width are 6 bit and 8 bit.\n   How fair is this for the three other methods used in comparison ( percentile-based method, EasyQuant, Bit-Split)? Have the authors made similar efforts to optimize other methods?\n\nIn the big picture, quantization is one possible post-processing method for the goal of reducing memory and computation, others include distillation (which the authors specifically do not use), various projection methods (e.g. fast-food transform [a]), pruning, etc.\n   It is not clear why quantization should be used instead of these many methods for the same goal, nor why it should not be used in combination.\n   It is not clear wether the three methods compared against (percentile-based method, EasyQuant, Bit-Split) represent a comprehensive collection of alternative quantization methods.\n\nYang, Z., Moczulski, M., Denil, M., De Freitas, N., Smola, A., Song, L., & Wang, Z. (2015). Deep fried convnets. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1476-1483).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See previous comments and suggestions.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "iFVsYIEEZj_", "reviewer": "Reviewer_3X6o", "summary": "The authors propose an effective post-training quantization algorithm to reduce the model size and computational overhead of the vision transformers. Specifically, the authors formulate quantization as an optimization problem for finding the optimal quantization intervals. To this end, the authors first introduce similarity-aware loss to maximize the similarity between the quantized and original feature maps. Then, the authors propose ranking loss to keep the relative order of the self-attention results after performing quantization. Moreover, the authors further devise bias correction to reduce the accumulated quantization error. Experimental results on CIFAR-10/100 and ImageNet demonstrate the effectiveness of the proposed method. However, some important details are missing and experiments are not sufficient. If the authors can solve these issues, I would be happy to change my score. My detailed comments are as follows. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Contribution**: \n1.\tThe authors propose to maximize the similarity between the quantized and original feature maps by the similarity-aware loss.\n\n2.\tThe authors empirically find that the relative order of the attention maps is changed after quantization, resulting in significant performance degradation. To address this, the authors propose a ranking loss to preserve the relative order of attention values.\n\n3.\tThe authors propose bias correction to reduce the accumulated error caused by quantization.\n\n**Questions and points needed to be improved**:\n1.\tIn Section 3.2, the motivation of using the Pearson correlation coefficient to measure the similarity between the original and quantized feature maps is not clear. Why not use the Euclidean distance or the cosine similarity?\n\n2.\tThe effect of alternating searching is not discussed. To demonstrate the effectiveness of the proposed alternating searching, it would be better for the authors to provide more results without the alternative searching, which would strengthen the paper.\n\n3.\tIn line 180, what does $\\Delta_l$ denotes? What do $\\alpha$ and $\\beta$ denote? More explanations are required.\n\n4.\tPlease use \"alternating\" instead of \"alternative \". Here, the proposed method alternates between steps instead of a different optimization. \n\n5.\tAn important metric is missing in the experiments, i.e. computational cost. It would be better for the authors to provide more results in terms of Bit-Operations (BOPs) [1][2] since BOPs is an important metric to measure the computational cost.\n\n6.\tIn Section 4.1, the authors randomly sample a subset of the dataset for calibration. However, the effect on the number of samples is unclear. It would be better for the authors to provide more results in terms of the different number of samples.\n\n**Reference**:\n\n[1] Differentiable Joint Pruning and Quantization for Hardware Efficiency. ECCV 2020.\n\n[2] Single Path One-Shot Neural Architecture Search with Uniform Sampling. ECCV 2020.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please discuss the limitations of the proposed method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "15"}, "extra_text": {}}, {"review_id": "fFnX145byj6", "reviewer": "Reviewer_H7ei", "summary": "This submission presents an effective post-training quantization algorithm for reducing the memory storage and computational costs of vision transformers. The main contribution is a ranking loss introduced into the conventional quantization objective to keep the relative order of the self-attention results after quantization. Moreover, it explores a mixed-precision quantization scheme by exploiting the nuclear norm of the attention and output feature maps. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThe manuscript proposes a post-training quantization for vision transformer. The task is urgently needed for deployment in industrial environments. The related work is cited adequately, but more analysis about the relationship between the quantization of BERT and ViT should be provided.\n\nQuality:\nThe idea of the manuscript is somewhat innovative, but the method part is not very clear. The quantization model with mixed-precision can get considerable results but more comparison between other quantization methods used on BERT should be given.\n\nClarity:\nThe method is not very clarified and some equations are very confusing, especially the strategy for obtaining different bit-widths is not clear. Figure 2 has no clear explanation. In addition, the motivation for introducing rank-loss is best to provide more sufficient evidence. \n\nSignificance: \nThe goal of the paper is significant in practical applications and the results demonstrate the method is validation. However, the method is not clear introduced, and the followers are hard to do further study on this method.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the method is not provided", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "delsUQX1jNX", "reviewer": "Reviewer_gcUY", "summary": "This paper proposes a mixed-precision post-training quantization method for vision transformer, which introduces the similarity-aware and ranking-aware quantization for the linear layer and self-attention layer, respectively. The experimental results are better than the methods for CNN quantization.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-\tBasically, this is a novel and effective method for the compression for the vision transformer. \n-\tThe proposed ranking-aware loss for the self-attention layer benefits for the quantization and the mixed-precision quantization further promotes the performance of quantized transformer. \n-\tThe paper is well written and organized.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- Relatively weak baselines such as ViT and DeiT are used. I understand the rebuttal time is limited, thus it is optional but better that the author conduct experiments based on a stronger baseline such as Swin Transformer [1].\n- Some typos: Line 143. The choice of … is-> the choices of … are\n\n[1] Swin transformer: Hierarchical vision transformer using shifted windows.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is solving an important problem for efficient vision transformer deployment in industrial environments. The goal of the paper is significant in practical applications. However, reviewers are concerned about the novelty is not enough and the method part is not clear; A large number of hyper-parameters are manually chosen, and the baselines and comparison with other approaches is weak. This paper is recommended for rejection. 
Decision: Accept (Poster)