Post-Training Quantization for Vision Transformer
Authors: Zhenhua Liu
Yunhe Wang
Kai Han
Wei Zhang
Siwei Ma
Wen Gao
Venue: NeurIPS 2021 Poster (2021)
Keywords: Post-training
vision transformer
ranking loss of self-attention
mixed-precision

Abstract: Recently, transformer has achieved remarkable performance on a variety of computer vision applications. Compared with mainstream convolutional neural networks, vision transformers are often of sophisticated architectures for extracting powerful feature representations, which are more difficult to be developed on mobile devices. In this paper, we present an effective post-training quantization algorithm for reducing the memory storage and computational costs of vision transformers. Basically, the quantization task can be regarded as finding the optimal low-bit quantization intervals for weights and inputs, respectively. To preserve the functionality of the attention mechanism, we introduce a ranking loss into the conventional quantization objective that aims to keep the relative order of the self-attention results after quantization. Moreover, we thoroughly analyze the relationship between quantization loss of different layers and the feature diversity, and explore a mixed-precision quantization scheme by exploiting the nuclear norm of each attention map and output feature. The effectiveness of the proposed method is verified on several benchmark models and datasets, which outperforms the state-of-the-art post-training quantization algorithms. For instance, we can obtain an 81.29% top-1 accuracy using DeiT-B model on ImageNet dataset with about 8-bit quantization. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/VT-PTQ. 

Post- Post-Trainin Quantization for Vision Transformer
Zhenhua Liu Yunhe Wang"; Kai Han', Wei Zhang', Siwei Ma 13, Wen Gao
'School of Electronic Engineering and Computer Science, Peking University Huawei Noah's Ark Lab 'Peng Cheng Laboratory 1iu Ziu-zh@pku.ede edu. cn, {yunhe. wang, kai. han,, wz. (e..zha)) com, (swma, wgao) pku..edu.cn
Abstract
Recently, transformer has achieved remarkable performance on a variety com- puter vision applications. Compared with mainstream convolutional neural net- works, vision transformers are often sophisticated architectures for extracting powerful feature representations, which are more difficult to be developed bile devices. In this paper, we present an effective post-training quantization algo- rithm for reducing the memory storage and computational costs of vision transform- ers. Basically, the quantization task can be regarded as finding the optimal low-bit quantization intervals for weights and inputs, respectively. To preserve the function- ality of the attention mechanism, we introduce ranking loss into the conventional quantization objective that aims keep the relative order of the self-attention sults after quantization. Moreover, we thoroughly analyze the relationship between guantization loss of different layers and the feature diversity, and explore a mixed- precision quantization scheme by exploiting the nuclear norm of each attention map and output feature. The effectiveness of the proposed method is verified on several benchmark models and datasets, which outperforms the state-of-the-ant post- training quantization algorithms. For instance, we can obtain an 81.29% top-1 accu- racy using DeïT-B model ImageNet dataset with about 8-bit quantization. Code will be available at https://gitee..om/t cow//innppreemoessseeeenneeeeeee PTQ.
Introduction
Following the applications in Natural Language Processing (NLP) tasks, transformer-based models have shown great power in various Computer Vision (CV) tasks, such image classification [1T 26).
object detection 4 39 and image super-resolution [5]. Pre-trained with large-scale data, these models usually have hundreds of millions of parameters. For instance, there 307M parameters and 64G FLOPs in the ViT-L model, which is both memory and computation expensive during inference. This brings great challenges for these models run on resource-constrained devices like mobile phones and intelligent cars. Besides, the real-tine pal--time computer vision applications that integrate transformer-bae based models have meet low latency requirements to achieve a high quality customer experience. Therefore, the model compression technology of transformer-basee models is urgently needed for deployment in industrial environments.
and weight decomposition 37 14 32 compresses neural network by using lower bit-width for weight values without changing the model architecture, which is particularly useful for carefully- designed network architectures like transformers. Quantizing both weights and inputs can speed up inference by tuning floating-point operations into integer or bit operations. There have been some
Among various compression methods like pruning
16
29
19
quantization method 9 38 27
"Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

training aware quantization approaches for transformer-basee models in NLP (e.g., BERT (T7) B4 However, these methods are not designed for computer vision tasks and usually need additional training or fine-tuning. Furthermore, in some scenarios, the entire training data is not available to optimize the quantization model and the training costs for edge devices are intolerable.
23 35 22.
Post-training quantization [24] is kind of efficient model compression technique, which can directly quantize neural network models without fine-tuning. Most of the existing post-training quantization methods are designed for convolutional neural networks 3 30 or recurrent neural networks [36]. These methods do not take the character of vision transformer into consideration (e.g., the attention mechanism do not exist in CNNs), which are not perfectly suitable for quantizing vision transformer. However, vision transformers are showing stronger performance in large variety of computer vision tasks. Thus, we are motivated to explore the post-tre -ossttraiiin quantization for them to reduce the costs on memory and computation.
In this paper, we study the post-training quantization method for vision transformer models with mixed-precision for higher compression and speed-up ratios. The quantized process in the transformer is formulated as an optimization problem for finding the optimal quantization intervals. Specially, our goal to maximize the similarity between the full-precision and quantized outputs in vision transformers. To better preserve the functionality of the attention mechanism, we thoroughly analyze the difference between attention layers and conventional layers such MLP. Then, ranking loss is introduced to keep the relative order of attention values. Furthermore, we propose to determine the bit-widths of each layer according the feature diversity, i.e,, the nuclear norm calculated by the attention map and output features. We alternatively search the quantization intervals of weights and inputs in all layers to obtain the best quantization results. In addition, bias correction is introduced to diminish the cumulative quantization error. Experimental results on several benchmarks demonstrate the effectiveness of our algorithm for achieving better performance over the state-of-art post-training quantization approaches.
Related Works
Here, we reviews the transformer-based models designed for computer vision tasks. And the training-aware quantization schemes proposed for BERT and post-training quantization algorithms are summarized and analyzed.
2.1 Vision Transformer
Inspired by the major success of transformer architectures in the field of NLP, researchers have recently applied transformer to computer vision (CV) tasks [13.. Chen et al. [6] trained a sequence transformer to auto-regressively predict pixels, achieving results comparable to CNNs on image classification tasks. Another vision transformer model is VIT, which applies a pure transformer directly to treat image patches as the sequences. Recently proposed by Dosovitskiy er al. [1]. it has achieved great performance multiple image recognition benchmarks. Touvron et al. (26] produce competitive convolution-fres transformers by training on ImageNet only while introducing a teacher-student strategy specific to transformers. In addition basic image classification, transformer has been utilized to address a variety of other computer vision problems, including object detection [4] (39]
semantic segmentation [5), image processing [5]. and video understanding (5] Han al. [15 proposed a -annsfoomeer----Taasffoooon
(TNT) model for modeling both patch-level and pixel-level representation. Tang al. (25) proposed augmented shortcut scheme improve the performance of vision transformers. Thanks to its exceptional performance, more and more researchers are
proposing transformer- models for wide range of computer vision tasks.
2.2 Compression of Transformer in NLP
Owing the remarkable performance of BERT in many NLP tasks, many researchers have tried to compress the model to reduce the memory and computation complexity of BERT. Wu et al. [3I proposed Short Range Attention (LSRA) to conduct transformer on edge devices, where one group of heads specializes in the local context modeling (by convolution) while another group specializes in the long-distance relationship modeling. In (22 34 8-bit quantization is successfully applied to Transformer-bast based models with comparable performance as the full- full-precision baseline. However,

---

quantizing these models to ultra low bits (e.g., or 2 bits) can be much more challenging due to significant reduction in model capacity. To avoid severe accuracy drop, more complex quantization methods, like mixed-precioe precision quantization 24 K3 and product quantization (PQ) [2] are used. In addition, Zhang et al. [55] propose TernaryBERT, which use both approximation-basee and loss- aware ternarization methods and empirically investigate the ternarization granularity of different parts of BERT. Moreover, to reduce the accuracy degradation, they also leverage the knowledge distillation technique. Bai et al. [1] further push BERT quantization to the limit with weight binarization. They propose ternary weight splitting, which initializes the binary model by equivalent splitting from a half-sized temnary network. However, these methods are not designed for computer vision tasks and need additional training or fine-tuning.
2.3 Post-Training Quantization
There are many works focusing on developing post-training quantization methods, without any training or fine-tuning. In particular, Yoni et al. [8] propose the OMSE method to optimize the Lg distance between the quantized tensor and the original tensor. Moreover, Ron et al. [ present the so-called ACIQ method to analytically compute the clipping range, as well as the per-channel bit allocation for NNs. Zhao et al. (36] propose outlier channel splitting (OCS) method to solve the outlier channel problem. Wang et al. (28) propose a Bit-Split and Stitching framework for lower-bit post-training quantization and an Error Compensated Activation Quantization method, which could lower the quantization error for activations. Nagel et al. (20) propose AdaRound, a weight-rounding mechanism for post-training quantization that adapts the data and the task loss. By approximating the task loss with a Taylor series expansion, the rounding task is posed as a quadratic unconstrained binary optimization problem. The recent work of (21) propose Data-Free Quantization, which further pushes post-traine trainini quantization to zero-shot scenarios, where neither training nor testing data are accessible during quantization. Cai et al. 3 introduce ZeroQ, which distills an input data distribution to match the statistics in the batch normalization layers of the model and utilize a Pareto Frontier method select automatically the bit- tit-precision configuration of mixed peeecdiieiiioonnn settings. These methods are designed for CNNs and do not consider the unique structure of vision transformers such as self-attention layers.
3 Methodology
In this section, we elaborate on the proposed mixed- preecd-oreiiioon post-training quantization scheme for the vision transformer. The similarity- quantization for linear layers and ranking- aware quantization for self-attention layers are presented. In addition, the bias correction method for optimization and the mixed-precision quantization based on nuclear norm of the attention map and output feature are introduced.
3.1 Preliminaries
A standard transformer receives input as 1-D sequence of token embeddings, so the vision transformers usually reshape the image E #x WX xC into sequence of flatted 2D patches IP e B=x(P2-C) Here, H and W are the height and width of the original image and (P; P) is the resolution of each image patch, n = -p vision transformers use constant widths through all of its layers, trainable linear projection maps each vectorized patch to the model dimension d. Thus, the input to the first transformer layer is:
HW
is then the effective sequence length for the transformer. Usually, the
X [Telaesi 1 F ..... ; W R| + E
where W E R (P2 c)xd E [(+1)xxd
(2)
standard transformer layer includes two main modules: Multi-Head Self Attention (MSA) and Multi-Layer Perceptron (MLP) module. For the 1-th transformer layer, suppose the input to it is X e R the attention scores computed by the dot product of queries and keys can be formulated
as:
A, Q.K. = X.W? W.W?Wk RT X..
(3)
3

---

2403
Multi Layer
Similarity
2200
Bit-width
Percectron
Interval
Output feature
Layer Norm
MSA
6550
2520
Bit width
Muiti-head Self Attention
Interval
Ranking loss
5220
1520
Layer Norm
Nuclear norm
Attention map
Palch Em sridediings
Figure 1: Diagram of the proposed mixed- precision post-training quantization method for vision transformer. The similarity-awarr and ranking- aware quantization are designed for finding the optimal quantization interval of the linear operations and self-attention layers. The bit-widths of transformer layers are determined based on the nuclear norm of the attention map and the output feature.
Then the softmax function is applied the normalized scores to get the output and the output of the multi-head self attention module is:
MSA(X:) = Softmax
Ay) WP.
(4)
The MLP module contains two linear layers parameterized by W e Rdddd, E Rd and 2 E Rd x b IRd respectively, where d is the number of neurons in the intermediate layer of MLP. Denote the input to MLP Zi Rnxd the output is then computed
MLP(Z,) GeLU(Z, +b' +
Combining Eq. and the forward propagation for the I-th transformer layer can be formulated as:
Z X; MSA(LN(X:)), X Z + MLP(LN(Z))),
(7)
where LN represents the layer normalization.
The most computational costs of vision transformer lie on the large matrix multiplication in MSA and MLP module. Following the mainstream quantization methods for CNNs 21). we quantize all the weights and inputs involved in matrix multiplication. For weight quantization, we quantize the weights Qº WK W WO W W in Eq. (4) and (5) for all transformer layers, as well as the linear embedding F in Eq. (I.. Besides these weights, we also quantize the inputs of all linear layers and matrix multiplication operations. Following the methods in [22 35). we do quantize the softmax operation and layer normalization, because the parameters contained in these operations are negligible and quantizing them may bring significant accuracy degradation.
3.2 Ranking-Aware Post-Training Quantization
For post-training quantization, we need to restrict the floating flaatin-numbbess to finite set of values. The choice of quantization intervals is critical for quantization and one popular option is to use uniform quantization function, where the data range is equally split:
A Y Clamp(Round Y.
-2 200-1,20111 1-11-0 1),
(8)

---

where A is the quantization interval, b is the quantization bit-width and Y is a tensor representing weights or inputs. Clamp denotes that elements in the tensor that exceed the ranges of the quantized domain are clipped.
For the layers in vision transformer, the original output can be computed O X, Wi The uniform quantization for the weights and inputs and the corresponding dequant operation can be described as:
O MAA (X)) (Wi) A
where O, denotes the outputs of the quantized layer. From Eq. (8) and Eq. (9 it can be seen that the quantization intervals actually control the clamping thresholds in quantization process, which affects the quantization results great extent. Therefore, the quantization intervals for both weights AW and inputs A, where inputs X₁ are generated from a given calibration dataset D with N samples. Specifically, the calibration dataset much less than the common training dataset.
motivated to focus on optimizing
The self- attentienoo layer is the critical component of the transformer since it can calculate the global relevance of the features, which makes the transformer unique from the convolutional neural networks. For the calculation of self-attention (Eq. B), we empirically find that the relative order of the attention map has been changed after quantization as shown in Fig [ which could cause a significant perfor- mance degradation. Thus, ranking loss is introduced to solve this problem during the quantization process:
h w-1 12 ΣΣΣ p=1pppppt
Lranking
@(((App - Akg) sign(AAp Akg)),
(10)
in which (mm) (0 m)+ hinge function with parameter 0, (h, w) are the size of matrix A. Given pair of examples, the loss is 0 only when the examples are in the correct order and differed by a margin.
Then we combine the ranking loss with the similarity-aware quantization, and the overall optimization goal can be described as:
N r(O; s.t. AW AX E R+
min 7. Cranking AA
(11)
N 1
where Crank denote the pairwise ranking based loss function, and y is the trade-off hyper-parameter. T(0) O denotes the similarity metric between the original and quantized output feature maps, which can be formulated as:
_,,00,-000,--
r 0)
(12)
where the Pearson correlation coefficient is adopted the measurement for the similarity since it subtracts the mean value of the data and can be more representative for the similarity between the distribution of quantized and original feature maps.
To solve the above optimization problem, we present a simple but efficient alternative searching method for the uniform quantization of transformer layers. Firstly. the quantization interval of inputs AX is fixed, and the quantization interval of weights A," optimized for adjustment. Secondly, AW is fixed, and AX is optimized to fine-tune the quantization interval of the inputs. AWW and A are alternately optimized until the target function converges or the maximum iteration is exceeded. Moreover, for fast convergence, A and A are initialized in terms of the maximum of weights or inputs respectively. For the search space of AW and AX" we linearly divide interval of [aAi, A into C' candidate options and conduct a simple search strategy on them.
To further reduce the biased error for the outputs raised by quantization, bias correction method is then introduced after each search iteration. Suppose the quantization error of weights and inputs are defined as:
Bias Correction
x VAX (X) x X, W = VAN (W) AW W.
(13)
(14)
5

---

If the expectation of the error for output is not zero, then the mean of the output will change. This shift in distribution may lead to detrimental behavior in the following layers. We can correct this change by seeing that:
[ = E + Bie X B| W EEE
(15)
Thus, subtracting the expected error on the output from the biased output ensures that the mean for each output unit is preserved. For implementation, the expected error be computed using the calibration data and subtracted from the layer's s bias parameter, since the expected error vector has the same shape as the layer's output.
3.3 Nuclear Norm Based Mixed-Precision Quantization
Different transformer layers are attending to different structures, and is expected that they exhibit different sensitivity. Thus, assigning the same number of bit-widths to all the layers is sub-optimal. As result, we explore mixeddppeeisse quantization, where more bits are assigned to more sensitive layers in order to retain performance. Considering the unique structure of transformer layer, we assign all the operations the MSA or MLP modules with the same bit-width. This will also be friendly to the hardware implementation since the weights and inputs are assigned with the same bit-width.
Singular value decomposition (SVD) is an important matrix decomposition approach in linear algebra. takes rectangular matrix of gene expression data, whose formulation can be written as
M UEV, tr(M) Σ Ei
(16)
where the diagonal entries Eg of Σ are known as the singular values of M. And the nuclear norm tr the sum of singular values, which represents the data relevance of the matrix. In this paper, we propose to estimate the sensitivity of the transformer layer with the nuclear norm of the attention map in the MSA module and the output feature in the MLP module. The nuclear norm can be used to reduce the search space of the mixed-preciste that are more sensitive and vice versa. Inspired by the method in [1O), we utilize a Pareto frontier approach to determine the bit- width. The main idea is sort each candidate bit-width configuration based on the total second-order perturbation that they cause, according to the following metric:
settings, while using higher bit-widths for layers
LABA
LMSA Σ
Σ Ω Σ tr(A:) |A
A.E
1 0
(17)
Given a target model size, the candidate bit-width configurations are sorted based on their value and choose the bit-width configuration with minimal Ω. The nuclear norm of the attention map and output feature each transformer layer are shown in Figure As we can see, they are various for different transformer layers.
4 Exprimental results
In this section, we evaluate the performance of the proposed post-training quantization scheme on vision transformer model for image classification (ViT [II and DeiT (26)) and object detection (DETR (4). To the best of our knowledge, there no published work done on post-training quantization of vision transformer at this point, methods for CNNs as described the papers by ourselves. shown that the proposed method outperforms the conventional post-training quantization methods. Moreover, extensive experiments of ablation study have shown that the proposed similarity-ave ranking-awar quantization and bias correction method are beneficial for the post-training quantization of vision transformer.
implement recent post-training quantization
4.1 Implementation details
Datasets For image classification, the CIFAR-10, 10, CIFAR-100 and IISVRR--201 ImageNet (we refer to it ImageNet in what follows) datasets are utilized to evaluate the quantization performance.

---

The CIFAR-10 dataset consists of 50K training images and 10K test images, which are labeled for 10 classes. And CIFAR- --0 dataset also contains 50K training images and 10K test images, expect that they are labeled for 100 classes. ImageNet dataset contains 1.2 million training images and 50K' validation images labeled for 1,000 categories. For object detection task, the COC02017 dataset is utilized to evaluate the quantization performance, which contains 118K training images and 5K validation images.
Experimental settings We randomly select 100 images for CIFAR-10 10 and CIFAR-100 dataset and 1000 images for ImageNet and COCO2017 dataset from the training dataset the calibration dataset. For the hyper-parameter, a and B3 are set to 0.5 and 1.2 for all the experiments. The maximum iteration is set to 20 if not mentioned specifically. For mixed-precision, we utilize (4,5,6,7,8) and (6,7,8,9,0) bits while the target bit-width are 6 bit and 8 bit, respectively.
Baseline For image classification, we evaluate our quantization method on two popular vision transformer implementation: ViT [M] and DeiT [26]. The ViT-B, ViT-L., DeiT-S, DeiT-B are adopted as the baseline model, whose top-1 accuracy ImageNet dataset are 71.58%, 71.48%, 79.8%, 81.8% respectively. For a fair comparison, we utilize the official implementation of DeiT and do not use other techniques like knowledge distillation. For object detection, the DETR model using ResNet-50 backbone is adopted, which achieves a 42.0 mAP on COCO dataset.
4.2 Results and Analysis
Image classification The experimental results are shown in Table T We firstly evaluate the proposed method on ViT-B and VIT-L model. ViT-B model is 12-layer transformer with 12 heads and 768 embedding dimension. For the similar quantized model size, the proposed method outperforms percentile-based method [18] by 3.35% and 2.07% on CIFAR-10 dataset, respectively, And it worth noting that the performance of the proposed 8-bit model is comparable to the full- precision model. The proposed method obtains the similar performance on CIFAR-100 dataset and ImageNet dataset, while the average gains are 2.95% and 3.28% respectively. Moreover, the performance of the proposed 6-bit model is even better than the 8-bit percentile-basee model, which means that the proposed method can save about 25% memory and 44% computational costs than conventional post-training quantization method.
VYT-L model is much larger network which consists of 24 transformer layer with 16 heads and 1024 embedding dimension. It contains 307M parameters, however its performance is worse than ViT-B. We also test the quantization methods on CIFAR-IO, CIFAR- and ImageNet dataset. As shown in Table [ the performance of the proposed method outperforms the percentile-based method by large margin. is worth mentioning that the 8-bit proposed model is even better than full-precision model CIFAR- 10 dataset and comparable to the call-precision model on CIFAR- 100 dataset and ImageNet model. It is supposed that there is more redundancy in the ViT-L model and the performance degradation of quantization is less than that of ViT-B model.
The architecture of DeiT network is the same as ViT, expect that DeiT utilizes the data augmentation and regularization strategies. As result, the performance of DeiT is much better than VIT. Among the models, ViT-S consists of 12 transformer layers with heads and 384 embedding dimension. As we can see, the percentile-based method largely hurts the performance while the accuracy losses of 6-bit and 8-bit models are 9.31% 31% and 5.82%. EasyQuant 30 a popular simple post-training quantization method which improves the performance loss to 6.54% and 3.21% %, respectively. Bit-Split proposes a bit splitting and stitching framework (28). while the Top-1 accuracy degradation are 5.76% and 2.74%. In comparison, the Top-1 accuracy losses of the proposed post-training quantization scheme are 5.22% and 2.33% respectively. In addition, when the mixed- -eexed-precisoo is conducted, the 8-bit quantized model can achieve 78.09% Top-1 I accuracy.
DeiT-B is much larger network than DeîT-S, which consists of 12 transformer layers with 12 heads and 768 embedding dimension. As shown in Table the Top- I accuracy of percentile-bee beesentiee-bseed are 73.99% and 75.21% % when quantized to 6-bit and 8-bit respectively. And the proposed scheme improves the performance of the quantized model to 77.47% and 81 .99%. Another point is that the aceuracy losses of DeiT-B are smaller than DeïT-S and we think that this is because DeiT-B consists of more parameters and is more representive when quantized to the same bit-width.
7

---

Table 1: Comparison on the performance of proposed mixed-precision post-training quantization method with conventional quantization method for image classification. 'MP" represents for mixed- precision.
Model
Dataset
Method
W-bit
A-bit
Model size (MB)
Top-1 Accuracy
Baseline
32
32
344
98.13
Percentile
64.5
93,48
CIFAR-I0
Ours
MP
64.6
96.83
Percentile
86.2
95.72
Ours
MP
86.0
97.79
Baseline
32
32
344
87.13
Percentile
64.5
80.56
VIT-B
CIFAR-100
Ours
6 MP
MP
64.4
83.99
Percentile
86.2
83.28
Ours
MP
8 MP
86.5
85.76
Baseline
344
77.91
Percentile
64.5
71.58
ImageNet
Ours
MP
MP
64.8
75.26
Percentile
86.2
74.10
Ours
MP
MP
86.5
76.98
Baseline
32
32
1228
97.86
Percentile
230.2
93.27
CIFAR- 10
Ours
MP
232
96.09
Percentile
307
94.19
Ours
MP
305.8
97.90
Baseline
32
32
1228
86.35
Percentile
230.2
80.54
VIT-L
CIFAR-100
Ours
6 MP
MP
231
83.69
Percentile
307
83.01
Ours
MP
MP
307.8
85.83
Baseline
1228
76.53
Percentile
230.2
71.48
ImageNet
Ours
MP
MP
231.6
75.46
Percentile
307
75.17
Ours
MP
MP
306.4
76.41
Baseline
32
88
79.8
Percentile 18
16.5
70.49
Easy
30
16.5
73.26
Bit-Split 228
16.5
74.04
Ours
16.5
74.58
DeiT-S
ImageNet
Ours
6 MP
MP
16.6
75.10
Percentile
22.0
73.98
EasyQuant 30
22.0
76.59
Bit-Split 22
22.0
77.06
Ours
22.0
77.47
Ours
MP
MP
22.2
78.09
Baseline
32
32
344
81.8
Percentile 18
64.5
73.99
EasyQuant
30
64.5
75,86
Bit-Split 28
64.5
76.39
Ours
MP
43.6
75.94
Ours
64.5
77.02
DeiT-B
ImageNet
Ours
MP
64.3
77.47
Percentile
18
86.0
75.21
30
86.0
79.36
Bit-Split
28
8
86.0
79.42
Ours
86.0
80.48
Ours
MP
8 MP
86.8
81.29

---

Table 2: Comparison on the performance of proposed mixed -exeedppreision post-training quantization
method with conventional quantization method for DETR. 'MP' represents for mixed -eeed-precisio..
Model
Dataset
Method
W-bit
A-bit
Model size (MB)
mAP
Baseline
32
32
164
42.0
Percentile
30.75
37.5
EasyQuant
130
6
30.75
39.0
Bit-Split
(28
30.75
38.9
Ours
6
30.75
40.1
DETR
COCO2017
Ours
MP
MP
30,98
40.5
Percentile 18
8
8
41.00
38.6
EasyQuant
30
41.00
40.4
Bit-Split 28
8
41.00
40.6
Ours
41.00
41.2
Ours
MP
MP
41.64
41.7
Object Detection In order to show the generalization capability of proposed method, we also evaluate our method for object detection task using DETR 4.. The experimental results are shown in T As we can see, the proposed method outperforms percentile-based method, EasyQuant, Bit- Split by 2.6, 1.1 and 1.2 mAP for 6-bit quantization, respectively. The mixed peeeeddoocisionn quantization further boost the performance of the method. For 8-bit quantization, the mAP of the proposed mixed -ixeddprecisson quantization method comparable to the full full-precision model.
4.3 Ablation study
In this section, evaluate the effect of the proposed similarity aware quantization module, ranking- aware quantization module, bias correction method and the sexeed-recciion method. The experimen- tal results shown in Table while experiments model. As we can see, the Top-1 accuracy of only using similarity- aware quantization 75.42% which is inferior the full-precision model and using ranking-aware quantization loss and bias correction method can improve the performance by 0.52% and 0.39%. It is worth noting that the nuclear norm based mixed-precision since it considers the variant sensitivity of different layers.
conducted on ImageNet dataset with VIT-B
further promote the performance of the quantized model,
It is also shown that the Top-1 accuracy of using the similarity-awar aware preccdppeeision quantization is 76.26%. And the ranking-awar quantization and bias correction can still boost the performance in this case. Besides, the performance of the 8-bit quantized model using all the proposed methods is 76.98 76,98, which comparable to the full precision model.
Table 3: Ablation study of the proposed similarity aware quantization module, ranking-aware quantization module, bias correction and mixed mixed-precision method.
Model
Similarity Ranking
Bias Correction
Mixed-Precision
Model size (MB)
Top-1 Accuracy
344
77.91
X
86.2
75.42
X
86.2
75.94
V
X
86.2
75.81
ViT-B
X
86.2
76.49
V
86.5
76.26
V
86.5
76.61
86.5
76.53
86.5
76.98
also compared the performance of the proposed method with the hessian- mixed-precision method, where the experiments are conducted with ViT-B on ImageNet dataset. As we can see in Table the proposed method achieves similar result while the consuming computation time much less than hessian-based approach.

---

Table 4: Performance comparison with hessian- mixed=precision method.
Method
W-bit
A-bit
Model size (MB)
Computation time (s)
Top-1 Accuracy
Hessian- bassean-basd
8
86.7
754.6
77.01
Ours
8 MP
8 MP
86.5
53,1
76.98
5
Conclusion
In this paper, have developed novel post-training quantization scheme for vision transformer, in which the bit-widths of each layer are variant based on the nuclear norm of the attention map and output feature in the transformer layer. To solve the optimization problem of the quantization, propose to search the optimal quantization interval for remaining the similarity between the quantized and original feature maps. In addition, and conventional layers and introduce a ranking loss to keep the relative order of the attention values. Specifically, the bias correction is employed reduce the accumulated quantization error. Last but not the least, the optimal quantization interval for each transformer layer is carefully optimized using an alternative searching strategy. Experimental results show that the proposed method outperforms the conventional post-training quantization method by large margin in terms of both network accuracy and memory costs.
thoroughly analyze the different between attention layers
Acknowledge
This work partly supported by the National Natural Science Foundation of China (61961130392) and PKU-Baidu Fund(2019BD0033 Besides, High-Performance Computing Platform of Peking University is acknowledged.
References
[1] Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization. arXiv preprint arXiv:2012.1570.
2020.
[2] Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. Post-training 4-bit guantization of convolution networks for rapid-deployment. arXiv preprint arXiv: 001..8800...... 2018.
[3] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer.
Zeroq: A novel zero shot quantization framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13169-13178. 2020.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. arXiv preprint
arXiv:2005. 12872, 2020.
[5] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma. Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. arXiv preprint arXiv:2012.00364, 2020.
[6] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691-1703. PMLR, 2020.
[7] Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint 100.vv:80000000.. 2018.
[8] Yoni Choukroun, Eli Kravehik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. In ICCV Workshops, pages 3009-3018, 2019.
[9] M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained tot I or- arXiv
preprint arXiv:1602. 02830, 2016.
[10] Zhen Dong, Zhewei Yao, Yaohui Cai, Daiyaan Arfeen, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trace-weighted quantization of neural networks.
arXiv preprint arXiv: 19t.:9991..333 2019.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.
10

---

An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint
arXiv:2010. 11929, 2020.
[12] Angela Fan, Pierre Stock. Benjamin Graham, Edouard Grave, Rémi Gribonval, Hervé Jégou, and Armand Joulin. Training with quantization noise for extreme model compression. arXiv e-prints, pages ar Xiv--200, 2020.
[13] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao. Chunjing Xu, Yixing Xu, et al. A survey on visual transformer. arXiv preprint arXiv:2012. /2556, 2020.
[14] Kai Han, Yunhe Wang, Yixing Xu, Chunjing Xu, Enhua Wu, and Chang Xu. Training binary neural networks through learning with noisy supervision. In International Conference on PMLR, 2020.
Machine Learning, pages
[15] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv preprint arXiv: 1tt::2/3:.. 103. 00 12, 2021.
[16] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.0014 2015.
[17] Jacob Devlin Ming-Wo Weeg Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186, 2019.
[18] Rundong Li, Yan Wang. Feng Liang, Hongwei Qin, Junjie Yan, and Rui Fan. Fully quantized network for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2810-12111 10-2819, 2019.
[19] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision. pages 2736-2744 2017.
[20] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort.
Up or down? adaptive rounding for post- tost-training quantization. In International Conference on Machine Learning, pages 7197-7206. PMLR. 2020.
[21] Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325-1334, 2019.
[22] Gabriele Prato, Ella Charlaix, and Mchdi Rezagholizadeh. Fully quantized transformer for improved translation. 2019.
[23] Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8815-8821,
2020.
[24] Wonyong Sung, Sungho Shin, and Kyuyeon Hwang. Resiliency of deep neural networks under quantization. arXiv preprint arXiv: 100::/511.0444 2015.
Yehui Tang, Kai Han, Chang Xu, An Xiao, Yiping Deng, Chao Xu, and Yunhe Wang. Aug- mented shortcuts for vision transformers. arXiv preprint arXiv:2106.1 /5911, 2021. [26] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint XrXiv:2012. 12877, 2020.
[25]
[27] Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quantization with mixed precision. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8612-866 8620, 2019.
[28] Peisong Wang. Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training network quantization via spt-sp and stitching. In International Conference on Machine Learning, pages 9847-9856. PMLR, 2020.
[29] Yunhe Wang, Chang Xu, Shan You, Dacheng Tao, and Chao Xu. Cnnpack: Packing convolu- tional neural networks in the frequency domain. In NIPS, volume 1, page 3, 2016.
[30] Di Wu, Qi Tang, Yongle Zhao, Ming Zhang. Ying Fu, and Debing Zhang. Easyquant: Post- training quantization via scale optimization. arXiv preprint arXiv:2006. J66,,69,0..669, 2020. [31] Zhanghao Wu, Zhijian Liu, Ji Lin. Yujun Lin, and Song Han. Lite transformer with long-short range attention. arXiv preprint arXiv:2004. / 1886, 2020.
[32] Yixing Xu, Kai Han, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Learning frequency domain approximation for binary neural networks. arXiv preprint arXiv:2
2021.
[33] Ali Hadi Zadeh, Isak Edo, Omar Mohamed Awad, and Andreas Moshovos. Gobo: Quantizing iitention-based alp models for low latency and energy efficient inference. In 2020 53dd Annual
11

---

IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 811-824 IEEE, 2020.
[34]
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert.
arXiv preprint arXiv:1910.06188 2019.
[35] Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. Ternarybert: Distillation-wware ultra-low bit bert. arXiv preprint arXiv:2009, 12812, 2020.
[36]
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural net- work quantization without retraining using outlier channel splitting. In International conference
on machine learning, pages 7543-7552. PMLR, 2019.
[37]
Zhisheng Zhong, Fangyin Wei, Zhouchen Lin, and Chao Zhang, Ada-tucker: Compressing deep neural networks via adaptive dimension adjustment tucker decomposition, Neural Networks,
110:104-115, 2019.
[38] Chenzhuo Zhu, Song Han, Huizi Mao, and William Dally. Trained ternary quantization. arXiv
preprint 16 1612.0104, 2016.
[39] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr:
Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159,
2020.
12


Reviews:
[{"review_id": "vEo4xDLgJ1y", "reviewer": "Reviewer_NVLj", "summary": "This paper proposed a post-training quantization algorithm for vision transformers. The algorithm introduces a ranking loss to improve the quantization performance of the model by keeping the relative order of self-attention results. Besides, this paper explores a mixed-precision quantization scheme by exploiting the nuclear norm of each attention map and output feature.  The proposed method is verified on several benchmark models and datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The paper proposes a post-training quantization method for vision transformer and applies it to different CV tasks.\n- Ablation studies on ranking loss, bias correction and mixed-precision quantization are welcome.\n- The paper is clear to read.\n\nCons:\n- The authors adopt Pearson correlation coefficient as the measurement for the similarity (L156-162). There are many other simpler choices for computing similarity (cosine, Euclidean, KL, etc). For example, the work of [22] uses cosine similarity [22], and the previous work of \"Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming\" uses Euclidean distance. Why do the authors choose the Pearson correlation coefficient, and what are the advantages of this metric?\n- Regarding ranking-aware quantization for self-attention, the authors state that \"we empirically find that the relative order of the attention map has been changed after quantization as shown in Fig 1, which could cause a significant performance degradation\" (L166-167). This \"observation\" is crucial to the main contribution and novelty of this paper, but it lacks theoretical analysis and sufficient experimental evaluations to demonstrate that the change of order actually negatively affects the final quantization accuracy. Without such analysis or evaluations, is the proposed solution reasonable and reliable? Moreover, suppose the statement is true, it lacks sufficient experiments to show that the order can be improved by the proposed solution. \n- The alternative searching [22] and bias correction [16] methods are not first proposed in this paper. The authors should cite them in the mentioned positions in case readers misunderstand they are contributions of this paper. \n- Section 3.3: Mixed-Precision Quantization: what is the difference between [9] and this paper? I do not seem novelty in this part.\n- What is the significance of the mixed-precision setting of this paper (L236-237)? Most practical hardware platforms have supported int8 inference with good hardware performance.\n- It lacks hyper-parameter analysis (e.g., alpha/beta/gamma/theta).\n- Figure 1 and Equation (6,7) do not match. Figure 1 shows pre-layerNorm but Equation (6,7) uses post-layerNorm.\n\nUpdated:\nI've read all the reviews and author responses. The authors addressed part of my concerns, e.g., explain the differences between cosine similarity and Pearson correlation and provide results of them. But some other major concerns are not addressed.\n\n1) For ranking strategy, Table 3 actually shows better results over the baseline but the gain is very limited. More importantly, the idea of ranking is not well explained in the paper to demonstrate the order of attention values is crucial for quantizing transformer.\n2) For nuclear norm, yes, this paper lacks necessary comparison experiments with the baseline of using Hessian Coefficient. Only stating that nuclear norm is easier than Hessian Coefficient is far insufficient to support this contribution.\n3) For mix-precision quantization, I think more general or more challenging settings (e.g., using 2-8 mixed bits or using lower bits of {2,3,4} as HAWQ-v2 [9]) can be evaluated to make this part of paper stronger but they are missing. More importantly, the algorithm of mix-precision quantization is almost the same as [9] except the nuclear norm (but nuclear nom is not well compared with exiting work to support its contribution). So the technical novelty is limited.\n\nIn general, I think the main strength of the paper is that it brings best practices to realize post-training quantization to vision transformer, but the novelty is very limited: 1) low-bit quantization follows existing alternative searching [22] and bias correction [16] and 2) mix-precision quantization follows HAWQ-v2 [9], and the different algorithmic parts between this paper and previous methods are not well explained and validated. Thus, I think the quality of this paper does not reach the bar of NeurIPS and still vote for rejection.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No. I did not find clear discussions on the limitations and potential negative societal impact of the work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "anTo0Py03pG", "reviewer": "Reviewer_oxRc", "summary": "The authors have proposed a post-training quantization method for vision transformer, where different modules employs the different bit-width. The quantization intervals are determined by optimizing the similarity between the quantized and original output features. In addition, the ranking-aware loss is proposed to promote the quantization performance of the self-attention layers.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe paper is well organized and I think it has significant contribution to the field. The proposed similarity and ranking-aware loss are rational. The experiments are well evaluated and they have demonstrated the effectiveness of the proposed method.\n\nWeaknesses:\n- What is the optimization process of the mixed-precision? I have a little confusion about the Eq.(18) and what is the meaning of the Y in Eq.(18)?\n- What is the configuration of the mixed-precision in the experiments since you have used 6 MP and 8 MP?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "SFe-825MK17", "reviewer": "Reviewer_bDSX", "summary": "This paper proposes a post-training quantization scheme for vision transformer, which determines the bit-width of each layer according to the nuclear norm of the attention map and output feature. In order to find the optimal quantization interval, the authors develops two loss to optimize the quantization errors.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(1)\tThe similarity-aware loss and bias correction are reasonable and the designed ranking-aware loss designed for the self-attention layer is novel.\n(2) The authors propose a mixed-precision quantization method for the MSA and MLP modules and experimental results demonstrate the effectiveness of the proposed method\n(3) This paper is well organized.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "(1)\tWhat are the optimization goal of the self-attention layer? Are they the weights and inputs of the Q and K? There are some details need to be provided in Eq.(12).\n(2)\tDoes the attention map in Eq.(13) the same as the output in Eq.(12)?\n(3)\tSome related works need to be discussed more concretely, such as quantization method for the transformer [1] [2].\n[1] Fully Quantized Transformer for Improved Translation.\n[2] Efficient 8-bit quantization of transformer neural machine language translation model.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "O8S5KQC0NNZ", "reviewer": "Reviewer_YhTZ", "summary": "The authors propose a novel quantization algorithm for vision transformer by searching for the best quantization intervals of inputs and weights. The optimization is conducted by minimizing the similarity loss and the ranking loss. The bias correction method is employed and the nuclear norm based mixed-precision is proposed. The proposed method is evaluated on the image classification and object detection tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Generally, this is a novel and effective quantization method for vision transformer, since the training and inference costs of vision transformer are vast. The proposed ranking-aware loss is specially designed for vision transformer and the mixed-precision quantization is instructive. The experimental results of the proposed quantization method outperforms the post-training quantization methods of CNN.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have one question that the proposed method can be directly used on CNN except the proposed ranking-aware loss. Have you compared the method on CNN and does it have the same beneficial? \nAnother question is what’s the meaning of Y in Eq.(18)? Is it the attention map or the output feature?\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript has been reviewed by four experienced reviewers, among whom three voted for acceptance and one voted for rejection. There are several rounds of discussion post the release of the reviews, and the rejection reviewer (NVLj) raises some concerns, including the quality of 1) Fig.1, 2) comparisons to Hessian Coefficient, and 3) limited novelty. All three concerns are highly constructive, yet given other contributions of the submission acknowledged by other reviewers, the pros indeed overweight the cons. Specifically, despite reviewer NVLj believes that the submission lacks novelty, all the other three reviewers, in fact, viewed the proposed approach as being novel.

The AC agrees with the major vote and hence recommends an acceptance. However, it is strongly recommended that the authors also incorporate the comments from reviewer NVLj, especially on comparisons to Hessian Coefficient, into the final version, if possible.
Decision: Accept (Poster)