Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer
Authors: Yanjing Li
Sheng Xu
Baochang Zhang
Xianbin Cao
Peng Gao
Guodong Guo
Venue: NeurIPS 2022 Accept (2022)

Abstract: The large pre-trained vision transformers (ViTs) have demonstrated remarkable performance on various visual tasks, but suffer from expensive computational and memory cost problems when deployed on resource-constrained devices. Among the powerful compression approaches, quantization extremely reduces the computation and memory consumption by low-bit parameters and bit-wise operations. However, low-bit ViTs remain largely unexplored and usually suffer from a significant performance drop compared with the real-valued counterparts. In this work, through extensive empirical analysis, we first identify the bottleneck  for  severe performance drop comes from  the information distortion of the low-bit quantized self-attention map. We then develop an information rectification module (IRM) and a distribution guided distillation (DGD) scheme for fully quantized vision transformers (Q-ViT) to effectively eliminate such distortion, leading to a fully quantized ViTs. We evaluate our methods on popular DeiT and Swin backbones. Extensive experimental results show that our method achieves a much better performance than the prior arts. For example, our Q-ViT can theoretically accelerates the ViT-S by 6.14x and achieves about 80.9% Top-1 accuracy, even surpassing the full-precision counterpart by 1.0% on ImageNet dataset. Our codes and models are attached on https://github.com/YanjingLi0202/Q-ViT

Q- Accurate and Fully Quantized Low-bit Vision Transformer
Yanjing Li Sheng Xu't, Baochang Zhang" Xianbin Caol-, Peng Gao", Guodong Guo
'Beihang University, Beijing, P.R.China
2Zhongguancun Laboratory, Beijing, P.R.China
Shanghai Artificial Intelligence Laboratory, Shanghai, P.R.China
Institute of Deep Learning, Baidu Research, Beijing. P.R.China
National Engineering Laboratory for Deep Learning Technology and Application,
Beijing, P.R.China
(yanjingli, shengxu, bezhang, xbcao) @ buaaa adua.edu.cn
Abstract
The large pre-trained vision transformers (ViTs) have demonstrated remarkable performance various visual tasks, but suffer from expensive computational and memory cost problems when deployed on assource-constraiie the powerful compression approaches. quantization extremely reduces the compu- tation and memory consumption by low-bit parameters and bit-wise operations. However, low-bit VITs remain largely unexplored and usually suffer from a signifi- cant performance drop compared with the real-valued counterparts. In this work, through extensive empirical analysis, we first identify the bottleneck for severe performance drop comes from the information distortion of the low-bit quantized self-attention map. We then develop an information rectification module (IRM) and distribution guided distillation (DGD) scheme for fully quantized vision transformers (Q-ViT) to effectively eliminate such distortion, leading to a fully quantized ViTs. We evaluate methods on popular DeiT and Swin backbones. Extensive experimental results show that our method achieves a much better per- formance than the prior arts. For example, our Q- V-T can theoretically accelerates the ViT-S by 6. 6.14x and achieves about 80.9% Top-1 accuracy, even surpassing the full-precision counterpart 1.0% ImageNet dataset. Our codes and models are attached on https: htt///gtttuu thub. com/Yanj /ttp//a0nnnnL100022000211 http://ghhho.ooooo/oo/oggg/noo/0
devices. Among
Introduction
Inspired by the success in natural language processing (NLP), transformer-base models have shown great power in various computer vision (CV) tasks, such image classification 4 and object detection [2]. Pre-trained with large-scale data, these models usually have tremendous number of parameters, For example, there are 632M parameters taking up 2528MB memory usage and 162G FLOPs in the ViT-H model, which both memory and computation expensive during inference. This limits these models for the deployment resource-limited platforms. Therefore, compressed transformers are urgently needed for real applications.
Substantial efforts have been made to compress and accelerate neural networks for efficient online inference. Methods include compact network design 10 network pruning 99. low-rank decomposi-
tion (3. quantization [1 30 32] and knowledge distillation [24] 31. Quantization is particularly
suitable for deployment on Al chips because it reduces the bit-width of network parameters and
activations for efficient inference. Prior post-training quantization (PTQ) methods [18 [4] on ViTs
t Equal contribution. Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Block.0.query
Block.3 quuery
Block.6.query
Block Bllock..qqeery
quuery
Block.6 6.query
j 1.2124
==
#
j 0.0765
0.0323
1.2748
1.0753
1.6533
1.2268
13118
(a) Full-Precision
(b) Fully quantized VIT (baseline)
Figure I: The histogram of query values q (blue shadow) along with the PDF curve (red line) of Gaussian distribution (J,,a") (20], for selected layers DeïT-T and 4-bit fully quantized DeïT-T (baseline). a and o² are the statistical mean and variance of the values.
directly compute quantized parameters based on pre-trained full full-precision models, which constrains the model performance sub-optimized level without fine-tuning: Furthermore, quantizing these models based on PTQ methods to ultra-low bits (e.g., 4 bits or lower) ineffective and suffers from a significant performance reduction.
training (QAT) 16 methods perform quantization during back propagation and achieve much less performance drop with a higher compression rate generally. QAT shown to be effective for CNN models [17 for CV tasks. However, QAT methods remain largely unexplored for low-bit quantization of vision transformers. Therefore, ViT baseline, a straightforward yet effective solution based on common techniques. Our study discovers that the performance drop of fully quantized VIT lies in the information distortion among the attention mechanism in the forward process, and the ineffective optimization for eliminating the distribution difference through distillation in the backward propagation. First, the attention mechanism of ViT aims at modeling long-distance dependencies (7 4.. However, our analysis shows that direct quantization method leads to the information distortion, i.e., significant distribution variation for the query module between quantized VIT and full-precision counterpart. For example, as shown in Fig. 1 the variance difference is 0.4409 (1.2124 v.s. 1.6533) for the first block]] This inevitably deteriorates the representation capability of the attention module on capturing the global dependency for the input. Second, the distillation for the fully quantized ViT baseline utilizes distillation token (following (25)) to directly supervise the classification output of the quantized ViT. However, we found that such simple supervision ineffective, which is coarse-grained for the large gap between the quantized attention scores and their full -reciionn counterparts.
Differently, quantization-awee
first build fully quantized
To address the aforementioned issues, fully quantized ViT (Q-ViT) is developed by retaining the distribution of quantized attention modules as that of full-precision counterparts (see the overview in propose modify the distorted distribution over quantized attention modules through an Information Rectification Module (IRM) based on information entropy maximization, in the forward process. While in the backward process, we present Distribution Guided Distillation (DGD) scheme to eliminate the distribution variation through attention similarity loss between quantized VIT and full-precision counterpart. The contributions of our work include:
Fig.2), (). Accordingly,
We propose an Information Rectification Module (IRM) based on the information theory to address the information distortion problem. IRM applies quantized representations in the attention module with a maximized information entropy, allowing the quantized model to restore the representation of input images.
We develope a Distribution Guided Distillation (DGD) scheme to eliminate the distribution mismatch in distillation. DGD takes appropriate activations and utilizes knowledge from the similarity matrices in distillation to perform optimization accurately.
Our Q- ViT, for the first time, explores a promising way towards accurate and low-bit ViT. Extensive experiments the ImageNet benchmark show that our Q-V outperforms the baseline by a large margin, and achieves comparable performances with the full full-precision counterparts.
The Gaussian distribution hypothesis is supported by 20

---

Rectification (IRM)
Pach Dibelding
Disrribution Guided Distillation (DGD)
Teucher
query
key
valsev
MHSA
MHSA
Gg
Gg
9,00
0,dki
Add & NocE
Add & Neen
MLP
MLP
Add & Noms
Add & Norn
Alteniium
Marts Xaliplaatze
X L
Gk
GRET
(2(A)
Cbarter
Figure 2: Overview of Q-VīT, applying Information Rectification Module (IRM) for maximizing representation information and Distribution Guided Distillation (DGD) for accurate optimization.
2
Related Work
Vision transformer. Motivated by the great success of Transformer in natural language processing, researchers are trying apply Transformer architecture to Computer Vision tasks. Unlike mainstream CNN-based models, Transformer is capable of capturing long-distance visual relations by its self- attention module and provides the paradigm without image-specific inductive bias. VIT 4 views 16 x 16 image patches token sequence and predicts classification via a unique class token, which shows promising results. Subsequently, many works, such as DeiT 25 and PVT [28 achieve further improvement on ViT, making it more efficient and applicable in downstream tasks. CONTAINER 77] fully utilizes a hybrid ViT aggregate dynamic and static information, exploring new framework for visual tasks. However, these high-performing vision transformers are attributed to the large number of parameters and high computational overhead, limiting their adoption. Therefore, innovating smaller and faster vision transformer becomes new trend. DynamicViT (23] presents dynamic token sparsification framework to prune redundant tokens progressively and dynamically, achieving competitive complexity and accuracy trade-off. Evo-VIT (33 proposes slow-fast updating mechanism that guarantees information flow and spatial structure, trimming down both the training and inference complexity. While the above works focus efficient model designing, this paper boosts the compression and acceleration in the track of quantization.
Quantization. Quantizing neural networks (QNNs) often possess low-bit (1~ 4-bit) weights and activations to accelerate model inference and save the memory usage. Specifically, ternary weights are introduced to reduce the quantization error in TWN [13]. DoReFa-Net [35] exploits convolution kernels with low bit-width parameters and gradients accelerate both the training and inference. TTQ (36] uses two full-precision scaling coefficients to quantize the weights to ternary values. [37 presented a 2 ~ 4-bit quantization scheme using a two-stage approach to alternately quantize the weights and activations, which provides optimal trade-off among memory, efficiency, and performance. [1]] parameterizes the quantization intervals and obtain their optimal values by directly minimizing the task loss of the network and also the accuracy degeneration with further bit-width reduction. (29) introduces transfer learning into network quantization to obtain an accurate low-precision model by utilizing the Beilbbckk-eibble (KL) divergence. [6] enables accurate approximation for tensor values that have bell-shaped distributions with long tails and finds the entire range by minimizing the quantization error. In our Q-VTT, aim implement accurate, fully quantized vision transformer under the QAT paradigm.
3 Baseline of Fully Quantized VIT
First all, we build a baseline to study the fully quantized ViT since has never been proposed in previous works. A straightforward solution is quantize the representations (weights and activations) in VIT architecture in the forward propagation and apply distillation to the optimization in the backward propagation.
Quantized ViT architecture. briefly introduce the technology of neural network quantization. We first introduce general asymmetric activation quantization and symmetric weight quantization

---

scheme as
Qa [clip( (r (oo==///os 2)/02, - OpHl Qw(() = |clip (aww Q 2p w (w) x O'w
(1)
I = (z) x +,
Here, clipfy: 1,, r2) returns y with values below rounds y to the nearest integer. With quantizing activations to signed bits and weights to signed b bits. O 2m_1____0 OF 2 1 and Q = 2 Q2 2 1. In general, the forward and backward propagation of quantization function in quantized network is formulated as
1ı and values above 72 set r2, and ly]
Forward: Q-Tincar(z:)
w O (1Qa(a) + z/0z) @ 1e (w)),
OJ
OJ 0J 08
if f 1- Q
Backward:
06
dr
db dx
otherwise
OJ dæ Ox Ow 0
0J aJ dr dw
if if QW!
Ow De Ow 8w
otherwise
where J is loss function, Q(-) is applied in the forward propagation while the straight-through estimator (STE) [ is used to retain the derivation of gradient in backward propagation. @ denotes the matrix multiplication with efficient bit-wist wise operations.
The input images are first encoded as patches and passes through several transformer blocks. Such transformer block consists of two components: Multi-Head self-Attention (MHSA) and Multi-Layer Perceptron (MLP). The computation attention weight depends on the corresponding query q, key k and value v, and the quantized computation in attention head is
Q-Linear, (x) k Q-Linear, (I), Q-Linear, (z).
where Q-Linear, Q- Q-Linerr Q-Linear, denote the three quantized linear layers for q. k, respec- tively. Thus, the attention weight is formulated as
1
A (Qa(g) & Qk(k))
Vd
(4)
Q^ = Qa (softmax(AA)
Training for Quantized VIT. Knowledge distillation an essential supervision approach for training quantized neural networks, which bridges the performance gap between quantized models and full- precision counterparts. The usual practice is using distillation through attention described in 25
Laist = (Cc((((Zz)).... +
2 (c)()))))))
2
(5)
V11 = arg max (c).
4 Proposed Q-ViT
With aforementioned quantizing and training pipeline, a fully quantized ViT baseline is built, which however has large performance gap with full-precision counterparts. Our study shows that the baseline suffers an severe information distortion among quantized attention scores in the forward propagation and distillation direc- tion misleading the backward propagation.
Quetze
Querze Activnicaa MILSA
Faly Quazimiss
Daldinlll
Dull-Bacc
Figure 3: Analysis bottleneck from architec- ture perspective. We report the accuracy of 2-bit quantized DeiT-S and DeiT-B on ImageNet dataset about replacing full-precision structure.
4.1 Performance degeneration of Fully Quantized ViT Baseline
Intuitively, in the fully quantized VIT baseline, the information representation capability largely depends the transformer-bast based architecture,

---

such as the attention weight in MHSA module. However, the performance improvement brought by such architecture is severely limited by the quantized parameters, while the rounded and discrete quantization also significantly affect the optimization. The phenomenon identifies the bottleneck of the fully quantized ViT baseline comes from architecture and optimization for the forward and backward propagation, respectively.
Architecture bottleneck. We replace each module with the full-precision counterpart respectively
and compare the accuracy drop as shown in Fig. S These ablation experiments are conducted the same as experiments in Tab. 2 We find that quantizing query, key, value and attention weight (i.e., softmax( A) in Eq. (4) to 2 bits brings the most significant drop of accuracy among all parts of the ViT. up to 10.03%. While quantized MLP layers and quantized weights of linear layers in MHSA brings only 1.78% and 4.26% drop, respectively. And once query, key. value and attention weight are quantized, even keep all weights of linear layers in MHSA module full- full-precision, the performance drops (10.57%) are still significant. Thus, improving the attention structure is pivotal to solve the performance drop problem of quantized VIT.
Optimization bottleneck. We calculate 12-norm distances between each attention weight among different blocks in DeiT-S architecture shown in Fig. The MHSA modules in full-precision ViT with different depth learn different representations from images. As mentioned in [22). lower ViT layers attend both locally and globally while higher ViT layers pay most attention to global representations. However, the fully quantized ViT (blue lines in Fig. 4 fails to learn accurate attention map distances. Thus, requires a new design that could utilize the full-precision teacher's 's information better.
4.2 Information Rectification in Q-Attention
To address the information distortion of quantized representations in the forward propagation, we propose an efficient Q-Attention structure based information theory, which statistically maximizes the entropy of representation and revives the attention mechanism in the fully quantized VIT. Since the representations with extremely compressed bit-width fully quantized ViT have limited capabilities, the ideal quantized representation should preserve the given full-precision counterparts much as possible, which means the mutual information between quantized and full-precision representations should be maximized as mentioned in (20).
We further show the statistical results that the distribution of query and key values in ViT architectures intending to follow Gaussian distributions under the distilling supervision, whose histograms are in bell-shape (20). For example, in Fig. D and Fig. 5 we have shown the query and key distributions and their corresponding Probability Density Function (PDF) using the calculated mean and standard deviation for each MHSA layer. Therefore, the distributions of query and key in the MHSA modules of full- full-precision counterparts are formulated as
((((g)) (9)), k~ *~N(n(k),o(k))
Since the weight and the activation with extremely compressed bit-width in fully quantized VIT have limited capabilities, the ideal quantization process should preserve the corresponding full-precision counterparts much as possible, thus the mutual information between quantized and full-precision representations should he maximized 20) As shown in [19), for Gaussian distribution, the quantizers with maximum output entropy (MOE) and minimum average error (MAE) are approximately the same within a multiplicative constant. Thus the process of minimizing the error between full- full-precision values and quantized values is equivalent maximizing the information entropy of the quantized values. Thus, when the deterministic quantization function is applied to quantized ViT, such objective equivalent to maximizing the information entropy H(Qx) of quantized representation Qx [19 in Eq. (4), which is defined as
H(Qa(x)) = Σ log = log 2Tee
9x
(7)
max (x) n nln2
when p(ax)
2
2
where 9x the random quantized variables in Qu (which is (a(q) or Qa(k) in different con- ditions) with probability mass function p(-). For better retaining the information contained in the
5

---

DeT
OVT
DUT
OAT
QUP
OUT
OAT
Figure 4: iventtin-ddsstane comparison for full-precision DeiT-Small (green lines). fully quantized DeiT-Small baseline (blue lines), and Q-ViT (yellow lines) for same input. Q-ViT shows similar behavior with the full- precision model, while baseline suffers indistinguishable attention distance for information degradation.
MHSA modules from the full-precision counterparts, the information entropy in the quantization process should be maximized.
However, direct application of quantization function converting the values into finite fixed points brings irreversible disturbance to the distributions and the information entropy H(Q((q) and (k)a(k)) degenerates to much lower level than the full- precision counterparts. To mitigate the information degradation from the quantization process in the attention mechanism, a Information Rectification Module (IRM) is proposed for effectively maximizing the information entropy of quantized attention weights
Qo(9)= QQ( 9 ((q) + 3g
k 1 u(k k- 3k SKk
Qa(k) = Qa(
TaV (2( +
7k o²Vo=(() ck
where 7as 8q and Tk. S& learnble parameters for modifying the distribution of q. while eq and ek are constants preventing the denominator being 0. The learning rates of learnable 7a: 8q and 7h. Sk
same as the whole network. Thus after IRM, the information entropy H(Q. and (k))(k)) are formulated as
H(Q(O)) - log 2 20e + eg),. H(Q(K)) log 2.re 2 + Ek]]
(9)
Then to revive the attention mechanism to capture critic elements by information entropy maximiza- tion, the learnable parameters 7q. Bg and 7k, 3y reshape the distributions of the query and key values achieve the state of information maximization. In nutshell, in our IRM-Attention structure, the information entropy of quantized attention weight is maximized to alleviate its severe information distortion and revive the attention mechanism.
4.3 Distribution Guided Distillation through Attention
To address the attention distribution mismatch occurred in fully quantized ViT baseline in the backward propagation, we further propose Distribution Guided Distillation (DGD) scheme with apposite distilled activations and the well-designed similarity matrices to effectively utilize knowledge from the teacher, which optimizes the fully quantized ViT more accurately.
As an optimization technique based on element-leve comparison of activation, distillation allows the quantized ViT to mimic the full- full-precision teacher model about output logits. However, we find that the

---

Bluer.0..uery
Bluury
gluer.
Block.0.query
Block.3.query
chock.6.ques
0.0841
-0.0361
M -0.0176
j 0.0847
-0.05988
-0.0011
1.2124
1,27448
14755
1.3346
1.3672
1.3038
(a) Full-Precisionn
Q-VIT
Figure 5: The histogram of query and key values q, (blue shadow) along with the PDF curve (red line) of Gaussian distribution (1, o²) [20), for 3 selected layers in DeiT-T and 4-bit Q-ViT. μ and o² are the statistical mean and variance of the values.
distillation procedure used in previous ViT and fully quantized VIT baseline (Sec. 3) unable to deliver meticulous supervision to attention weights (shown Fig. 4). leading to insufficient optimization. To solve the optimization insufficiency in the distillation of the fully quantized VIT, we propose the Serriibtton-GGuidde Distillation (DGD) method Q-V.. We first build patch-based similarity pattern matrices for distilling the upstream query and key instead of attention following [26]. which formulated
C². q (9%)
G / 11
(10)
Gk, = K c G
where || llz denotes l2 normalization, and L, h are layer index and head index. Previous work shows that matrices constructed in this way regarded as the specific patterns reflecting the semantic comprehension network [26]. And the patches encoded from input images containing high-level understanding of parts, objects, and scenes [8]. Thus such semantic-level distillation target provide guided and meticulous supervision to the quantized VIT. The corresponding G :T and G k. constructed in the same way by the teacher's activation. Thus combining the original distillation loss Eq. (5), the final distillation loss is then formulated as
LDGD Σ
Σ
C
G Gra 112
||G
G 112-
LE(1.L] HE
(11)
Ldistillation Laist + LDGD,
where L and H denote the number of ViT layers and heads. With the proposed Distribution Guided Distillation, the Q-ViT retains the distribution over query and key from the full full-precision counterparts (as shown in Fig.5)
Our DGD scheme first provides the distribution- aware optimization direction together with processing appropriate distilled parameters and then constructs similarity matrices to climinate scale differences and numerical instability, thereby improves fully quantized ViT by accurate optimization.
Experiments
In this section, we evaluate the performance of the proposed Q-VIT model for image classification task using popular DeiT [25) and Swin [15] backbones. To the best of our knowledge, there is no training of vision transformer at this point, so we implement the baseline and LSQ [S] methods by ourselves.
publicly available source code on quantization-awan
5.1 Datasets and Implementation Details
The experiments are carried out on the ILSVRCI2 ImageNet classification dataset [12]. The ImageNet dataset is more challenging due to its large scale and greater diversity. There are 1000 classes and 1.2 million training images, and 50k validation images in it. In our experiments, we use the classic data augmentation method described in (25).
Datasets.
Experimental settings. our experiments, we initialize the weights of quantized model with the corresponding pretrained full-precision model. The quantized model is trained for 300 epochs with batch-size 512 and the base learning rate 2 4. We do not use warm-up scheme. For all the experiments, we apply LAMB (34] optimizer with weight decay set O. Other training settings

---

follow DeiT [25] or Swin Transformer [15]. Note that we use 8-bit for the patch embedding (first) layer and the classification (last) layer following [5].
Bakchone. We evaluate our quantization method on two popular vision transformer implementation: DeiT (25] and Swin Transformer [13]. The DeïT-S, DeiT-B, Swin-T and Swin-S are adopted as the backbone models, whose Top-1 accuracy on ImageNet dataset are 79.9%, 81.8%, 81.2%, and 83.2% respectively. For fair comparison, we utilize the official implementation of DeiT and Swin Transformer.
5.2 Ablation Study
We give quantitative results of the proposed IRM and DGD in Tab. T As shown in Tab. [ the fully quantized VIT baseline suffers severe performance drop on classification task (0.2%, 2. 1% and 1 11.7% with 2/3/4-bit, respectively). IRM and DGD improve the performance when used alone, and the two techniques further boost the performance considerably when combined together. For example, the IRM improve the 2-bit Baseline by 1.7% and the DGD achieves 2.3% performance improvement. While combining the IRM and DGD together, the performance improvement achieves 3.8%.
Table 1: Evaluating the components of Q-ViT
ViT based on ViT-S backbone.
Method
#Bits
Top-1
#Bits
Top-I
#Bits
Top-1
Full-precision
32-32
79.9
Baseline
4-4
79.7
3-3
77.8
2-2
68.2
+IRM
4-4
80.2
3-3
78.2
2-2
69.9
+DGD
4-4
80.4
3-3
78.5
2-2
70.5
IRM-DGD (Q-ViT))
4.
80.9
333
799.0
2-2
72.0"
To conclude, the two techniques can promote each other to improve Q-ViT and close the performance gap between fully quantized ViT and full-precision counterpart.
5.3 Main Results
The experimental results are shown in Tab. 2
We compare our method with 2/3/4-bit baseline and LSQ [5] based on the same frameworks for the task of image classification with the ImageNet dataset. We also report the classification performance of the 8-bit post-training quantization networks percentile VT-PTQ [8]. We firstly evaluate the proposed method on DeiT-S and DeiT-B models.
For DeiT-S backbone, compared with 8-bit VT-PTQ method, our 4bit Q-T achieves much larger compression ratio than 8-bit VT-PTQ. but with significant performance improvement (78.1 vs. 80.9%), And it is worth noting that the proposed 2-bit model significantly compresses the DeiT-S by 21.5x on FLOPs. The proposed method boosts the performance of 2/3/4-bit Baseline by 3.9%, 1.5% and 1.2% with the same architecture and bit-width, which is significant on the ImageNet dataset. For larger DeiT-B, as shown in Tab. 2 the performance of the proposed method outperforms the 2/3/4-bit Baseline by 3.8%, 1.7% and 1.9%, a large margin. Also note that the proposed b/t/4-bit model significantly compresses the DeiT-B by 21 x, 12x and 7.6x on FLOPs. Compared with 8-bit post-training quantization methods, our method achieves significantly higher compression rate, and the performance improvement is significant.
Also, our method generates convincing results on Swin-transformers. As shown in Tab. 2 the performance of the proposed method with Swin-T outperforms the 2/3/4-bit Baseline method by 4.1% 2.1% and 2.0%, large margin. Compared with 8-bit post-training quantization methods, our method achieves significantly higher compression rate, and comparable performance. Note that our 4-bit QVT surpasses the full-precision by 1.3% counterpart using Swin-T, which demonstrates the significance of our Q- Q-V.T. For larger Swin-S, the performance of the proposed method outperforms the 2/3/4-bit Baseline by 4.3%, 1.8% and 1.5%. Also note that our 4-bit Q-ViT surpasses the full- precision by 1.1% .11% counterpart using Swin-S and significantly compresses the Swin-S by 7.9x. which demonstrates the effectiveness and efficiency of our Q-VIT.
8

---

Table 2: Quantization results on ImageNet dataset. "#Bits" (W-A) is the bit width for weights and activation.
Network
Method
#Bits
SiZ)
GL,OPs(G)
Top-1
Top-5
Full-precision
32-32
88.2
4.3
79.9
95.0
VT-PTQ
8MP-8MP
22.2
78.1
LSQ
4-4
11.4
0.7
79.6
94.6
Baseline
4-4
11.4
0.7
79.7
94.5
Q-ViT
11.4
0.7
80.9
94.9
DeïT-S
LSQ
3-3
8.7
0..4
77.3
93.0
Baseline
8.7
0.4
77.5
93.3
Q-VIT
3-3
8.7
0.4
79.0
94.2
LSQ
2-2
6.0
0.2
68.0
86.4
Baseline
2-2
6.0
0.2
68.2
86.5
Q-ViT
2-2
6.0
0.2
72.1
90.3
Full-precision
32-32
346.2
16.8
81.8
95.6
VT-PTQ
8MP-8MP
86.8
81.3
LSQ
4-4
2.2
80.9
95.1
Baseline
4-4
44.1
2.2
81.1
95.3
Q-ViT
4-4
44.1
2.2
83.0
96.1
DeiT-B
1.QQ
3-3
33.4
1.4
79.0
94.5
Baseline
3-3
33.4
1.4
79.3
94.9
Q-VIT
3-3
33.4
1.4
81.0
95.1
LSQ
2-2
22.7
0.8
70.3
88.6
Baseline
2-2
22.7
0.8
70.4
88.8
Q-ViT
2-2
22.7
0.8
74.2
92.2
Full-precision
32-32
114.2
4.5
81.2
95.5
LSQ
4-4
14.6
0.6
80.2
95.2
Baseline
4-4
14.6
0.6
80.5
95.4
Q-ViT
4-4
14.6
0.6
82.5
97.3
LSQ
3-3
11.2
0.3
79.7
94.9
Swin-T
Baseline
3-3
11.2
0.3
79.8
95.1
Q-VIT
3-3
11.2
0.3
80.9
96.1
LSQ
2-2
7.7
0.2
70.4
88.8
Baseline
2-2
7.7
0.2
70.6
89.0
Q-VIT
2-2
7.7
0.2
74.7
92.5
Full Full-precision
32-32
199,8
8.7
83.2
96.2
LSQ
4-4
7.0
82.5
97.1
Baseline
4-4
7.0
1.1
82.9
97.3
Q-ViT
4-4
7.0
1.1
84.4
98.3
L3Q
3-3
3.3
0.6
80.6
95.7
Swin-S
Baseline
3.3
5.5.
0.6
80.9
95.9
Q-ViT
3-3
5.5
0.6
82.7
97.5
LSQ
2-2
3.9
0.3
72.4
90.2
Baseline
2-2
3.9
0.3
72.7
90.6
Q-ViT
2-2
3.9.
0.3
76.9
94.9
Conclusion
In this paper, we introduce Q- Q-VIT to improve the fully quantized ViTs with high compression ratio and competitive performance. We first build a theoretical framework of fully quantized ViT and analysis the bottlenecks of the fully quantized VIT baseline. Then we introduce Information Rectification Module and Distribution Guided Distillation Q-ViT performance improvement. Our proposed Q Q-ViT' achieve comparable performance with full-precision counterparts with ultra-low bit weights and activations. Our work gives an insightful analysis and effective solution about the crucial issues in VIT full guantization, which blazes promising path for the extreme compression of VIT.

---

7 Acknowledgement
This work was supported in part by the National Natural Science Foundation of China under Grant 62076016, under Grant 62206272 and 61827901, Beijing Natural Science Foundation-Xiaom Innovation Joint Fund L223024, Foundation of China Energy Project GINY-19-90.
References
[1] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: arXiv: 3432, 2013.
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. Proc. of ECCV, pages 213-229, 2020.
[3] Misha Denil, Babak Shakibi, Laurent Dinh, Marc' Aurelio Ranzato, and Nando De Freitas. Predicting parameters in deep learning. In Proc. of NeurlPS, pages 2148-2156, 2013.
[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[5] Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar- mendra S Modha. Learned step size quantization. arXiv preprint arXivv1902.088110 2019.
[6] Jun Fang, Ali Shafice, Hamzah Abdel-Aziz, David Thorsley, Georgios Georgiadis, and Joseph H Hassoun. Post-training piecewise linear quantization for deep neural networks. In Proc. of ECCV, pages 69-86, 2020.
[7] Peng Gao, Jiasen Lu, Hongsheng Li, Roozbeh Mottaghi, and Aniruddha Kembhavi. Container: Context aggregation network. In Proc. of NeurlPS, pages 19171, 2021.
[8] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 2021.
(9] Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. In Proc. of IJCAI, pages 2234-2240, 2018.
[10] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 04461, 2017.
[11] Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In Proc. of CVPR, pages 4350-4359, 2019.
[12] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Proc. of NeurlPS, pages 1097-1105, 2012.
[13] Fengfu Li, Zhang, and Bin Liu. Temary weight networks. arXiv preprint ArXiv:1605.047 04 2016.
[14] Yang Lin, Tianyu Zhang, Peiqin Sun, Zheng Li, and Shuchang Zhou. Fii Fully quantized vision transformer without retraining. arXiv preprint arXi::2111.1338 2021.
[15) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. Proc. of ICCV, pages 10012-10022, 2021.
[16] Zechun Liu, Zhiqiang Shen, Marios Savvides, and Swang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In Proc. of ECCV, pages 143-159, 2020.
10

---

[17] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit emns with improved representational capability and advanced training algorithm. In Proc. of ECCV, 2018.
[18] Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. Proc. of NewrIPS, 2021.
[19] D Messerschmitt. Quantizing for maximum output entropy (corresp.). IEEE Transactions on Information Theory, 1766):612-12,, 1971.
[20] Haotong Qin, Yifu Ding, Mingyuan Zhang, Qinghua Yan, Aishan Liu, Qingqing Dang, Ziwei Liu, and Xianglong Liu. Bibert: Accurate fully binarized bert. In Proc. of ICLR, pages 1-24, 2022.
[21] Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. In Proc. of CVPR, pages 2250-2259, 2020.
[22] Maithra Raghu, Thomas Unterthiner, Simon Komblith, Chiyuan Zhang, and Alexey Dosovitskiy, Do vision transformers see like convolutional neural networks? Proc. of NeurlPS, pages 1-13, 2021.
[23] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamievit: Efficient vision transformers with dynamic token sparsification. Proc. of NeurlPS, pages 1-14, 2021.
[24] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In Proc. of ICLR, pages 1-13, 2015.
[25] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Proc. of ICML, pages 10347-10357, 2021.
[26] Frederick Tung and Greg Mori. pages 1365-1374, 2019.
knowledge distillation. In Proc. of ICCV,
[27] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Proc. of NeurlPS, 30, 2017.
[28] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proc. of ICCV, pages 568-578, 2021.
[29] Zheng Xie, Zhiquan Wen, Jing Liu, Zhiqiang Liu, Xixian Wu, and Mingkui Tan. transferring quantization. In Proc. of ECCV, pages 625-642, 2020.
Deep
[30] Sheng Xu, Yanjing Li, Tiancheng Wang, Teli Ma, Baochang Zhang, Peng Gao, Yu Qiao, Jinhu Lu, and Guodong Guo. Recurrent bilinear optimization for binary neural networks. arXiv preprint 2xXiv:2209.0142 2022.
[31] Sheng Xu, Yanjing Li, Bohan Zeng, Teli Ma, Baochang Zhang, Xianbin Cao, Peng Gao, and Jinhu Lu. Ida-det: An information discrepancy-awar distillation for 1-bit detectors. arXiv preprint arXiv:2210.0347, 2022.
[32] Sheng Xu, Junhe Zhao, Jinhu Lu, Baochang Zhang, Shumin Han, and David Doermann. Layer-wise searching for 1-bit detectors. In Proc. of CVPR, pages 5682-5691, 2021.
[33] Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng. Ke Li. Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision trans- former. arXiv preprint 11X0v:2108.01 01 390, 2021.
[34] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. Proc. of ICLR, pages 1-37, 2020.
11

---

[35] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net, Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv: 0666000660, 2016.
[36] Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. In Proc. of ICLR, pages 1-10, 2017.
[37] Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and Ian Reid. Towards effective low-bitwidth convolutional neural networks, In Proc. of CVPR, pages 7920-7928, 2018.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes]
See Section 5
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Section 4
(b) Did you describe the limitations of your work? [Yes] Q-ViT' is effective and efficient as shown in our extensive experiments, but the extension bit is still unexplored. We believe that the knowledge distillation, data augmentation can be used to to further enhance the performance of 1-bit Q-ViT, which will be further explored our future work.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] This work does not have a direct negative social impact. However, we should be aware of the power of neural networks and prevent them from being used for harmful applications.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
12

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowd sourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "nXSVVIc-GgW", "reviewer": "Reviewer_RZGP", "summary": "In this paper, Q-ViT is proposed, an effective fully quantized scheme for vision transformer. This paper first introduces the bottleneck for severe performance drop and then present the Information Rectification Module (IRM) and Distribution Guided Distillation (DGD) to bulld a high-performance fully quantized vision transformers. And the experimental results demonstrate the superiority of Q-ViT.", "questions": "1. Please prove why maximizing the mutual information between quantized and full-precision representations is equivalent to maximizing the information entropy $H(Q_x)$, there is nothing new in the the disussion of supplementary material.\n\n2. The learnable parameters $\\gamma_q$, $\\beta_q$ and $\\gamma_k$ and $\\beta_k$ are updated with the total loss, they could get a better information as Figure 4, but how to prove that they could achieve the state of information maximization. The Information Rectification Module (IRM) seems a loss-aware method instead a strong constrain to keep the max information.\n\n", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The analysis of the bottleneck for severe performance drop is clear.\n2. The experimetal results are state-of -the-art.\n3. The paper is well-written and easy to understand.\n4. The code and checkpoints are open sourced.\n\nWeaknesses\n1. When the deterministic quantization function is applied to quantized ViT, such objective is equivalent to maximizing the information entropy. The reason is not clearly described.\n2. The experiments are sufficient but the proof of these two techniques is a bit weak.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "g7epMkrpr86", "reviewer": "Reviewer_BbJC", "summary": "This paper proposed a novel and efficient quantization method for Vision Transformers. The authors first identify the bottleneck low-bit quantized Vision Transformers which comes from the information distortion of the low-bit quantized self-attention map. The authors then develop an IRM and a DGD scheme for fully quantized Vision Transformers based on the aforementioned bottleneck, which leads to a fully quantized Vision Transformers. ", "questions": "Please see above.", "limitations": "Please see above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1.In Line 61, what is the actual meaning of “quantified representations”?\n\n2.In section 4.1 “Architecture bottleneck”, how are these experiments trained? the same as the main experiments settings?\n\n3.In Line 155, “distance”should be “distances”. \n\n4.Font sizes of the legend in Figure 4 are inconsistent and should be larger. \n\n5.Are the IRM applied before quantizing attention module, or after quantizing? Such two situations may lead to different results. \n\n6.In section 5.1, authors mentioned that the Q-ViT is initialized from pertained full-precision counterparts. However, how are the alpha and zero point initialized?\n\n7.In Eq. 7, what does “q_x is the random quantized variables in Q_a(x)” mean? What are the possible values of “q_x”?\n\n8.In Eq. 8, after introducing IRM, the module calculate the mean and variance of Q and K in each block for each forward process, will this affect the speed of inference?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dsvR1RjL0i0", "reviewer": "Reviewer_HQzf", "summary": "This paper studies the quantization of the Vision Transformer model, especially in the low-bit case. For the problem of large quantization accuracy loss at low bits, two improvements, IRM and DGD, are proposed in this paper, respectively. Concretely, IRM maximizes the information entropy of the quantized representation. DGD supervises the model with other patch-based similarity pattern matrices.", "questions": "1.\tThis paper alleviates the problem of accuracy loss caused by ViT quantization to some extent by two improvement measures. However, I expect that the author dig deeper into the differences between ViT and CNN quantization and the differences between transformer quantization in vision and NLP.\n2.\tI notice that this work applies Adam optimizer with weight decay set as 0, which is not a regular setting. I suggest the author add extra ablation studies on it to make the paper more complete.\n", "limitations": "The limitations have been well addressed.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThe quantization of ViT models is an important research topic. In this paper, the accuracy loss caused by ViT quantization is mitigated to some extent.\n2.\tThe article is well written and well finished. It is easy to follow.\n\nWeaknesses:\n1.\tThe analysis of vit quantification could be explained in depth:\n(a) this paper argues that `a direct quantization method leads to the information distortion’ in Line 45. The approach proposed in this paper does not improve this phenomenon either (e.g. 1.2268 in Fig1(b) v.s. 1.3672 in Fig5(b) for Block.3. The variance difference is even larger with the proposed approach). \n(b) The quantization of MHSA introduces a large loss of precision, which has been found in transformer quantization in the NLP (such as Q-BERT, Q8BERT, BinaryBERT, FullyBinaryBert, etc.) and is not unique to the ViT model. \n2.\tSome minor problems:\n(a)\tIn Fig2, the tilde hat of k is too small. It should be inconsistent with q’s hat. \n(b)\tIn Equation 9, $Q_k$ might be $Q(k)$ to be consistent with $Q(q)$.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Vnz70NEln7j", "reviewer": "Reviewer_RZMx", "summary": "This paper focused on Vision Transformers quantization aware training methods. In this paper, the bottleneck of QAT for ViT are firstly studied which are mainly caused by the information distortion in MHSA. This paper thus propose IRM and DGD scheme to solve such bottleneck which retains the performance of quantized ViT from full-precision counterparts. ", "questions": "See weakness above.", "limitations": "-", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The paper is technically sound and easy to understand.\n\n2. The experiments show that the proposed method is effctive.\n\nWeakness:\n\n1. How is the “6.14x”in Abstract calculated?\n\n2. In L155, how is the distance between each attention weight calculated?\n\n3. In L191, learning rates of learnable gamma and beta are not clarified, are they the same as the learning rate of the whole network? \n\n4. In Eq. 10, why just query and key are distilled, instead of query, key and value?\n\n5. How are alpha_x and alpha_w learned through training? Are the training settings the same as other learnable parameters?\n\n6. In Eq. 7, after maximizing the information entropy, q_x should follow an uniform distribution, however, in the aforementioned figure 4, the query and key may follow a Gaussian distribution. This seems to be conflicting. \n\n7. In Eq. 8, how are the mu and sigma calculated? Are they channel-wise or layer-wise?\n\n8. How is Eq. 9 derived? Especially the part“1/2 log2πe[gamma^2(sigma^2 + epsilon)]”?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept