PackQViT: Faster Sub-8-bit Vision Transformers via Full and Packed Quantization on the Mobile
Authors: Peiyan Dong
LEI LU
Chao Wu
Cheng Lyu
Geng Yuan
Hao Tang
Yanzhi Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Vision Transformers
Quantization
Real-time on mobile
Sub-8-bit

Abstract: While Vision Transformers (ViTs) have undoubtedly made impressive strides in computer vision (CV), their intricate network structures necessitate substantial computation and memory resources. 
A decision-making process for CV tasks typically entails performing computations with low latency, which is a tricky problem for ViT models.
Model quantization is a widely-used technique to optimize the hardware efficiency of deep neural networks.
Full quantization under Sub-8-bit precision, in particular, is a promising solution to reduce inference latency significantly. 
Unfortunately, current commodity hardware, such as CPUs and GPUs, still struggles to efficiently execute these sub-8-bit quantized networks, as their SIMD instructions only support a granularity of 8 bits or wider.
Also, there is a scarcity of literature that presents a full quantization paradigm for ViTs.
In this paper, we propose an activation-aware fully sub-8-bit quantization-aware training (QAT) framework called PackQViT for efficient yet accurate ViT acceleration on mobile devices to facilitate real-time AI-powered decision-making.
Specifically, in revisiting data activation within the ViT dataflow, two characteristics are relevant to quantization strategy and precision: the long-tailed distribution and systematic channel-wise outliers.
In response, we employ either log2 quantization or clipping to address the long-tailed distribution and incorporate outlier-aware training for residual link quantization to regulate the various channel-wise outliers more consistently.
Notably, due to the systematic fixed pattern, outlier-aware training approach can predict the channel indices and regularized scales of outliers in advance, thus avoiding the runtime data-adaptive selection during inference.
Furthermore, we employ Int-$2^{n}$-Softmax, Int-LayerNorm, and Integer GELU to enable integer-only computation flow. Finally, we develop a SIMD-based 4-bit packed multiplier to achieve end-to-end ViT acceleration on mobile phones.
Compared to prior studies on ViT quantization using 8-bit precision, PackQViT surpasses other works by an improved accuracy ranging from 0.4\% to 17.9\% for various widely used ViTs on ImageNet dataset; under 4-bit precision, PackQViT demonstrates 0.4%$\sim$2.8% higher accuracy. Compared to the baseline multiplier, our implementations on the Realme GT Android smartphone with Snapdragon 870 SoC CPU achieve 2.6x$\sim$3.7x speedup under 8-bit scenario and 3.8x$\sim$5.9x speedup under 4-bit which ensures practical real-time performance.

PackQViT: Faster Sub-8-bit Vision Transformers via Full and Packed Quantization on the Mobile
Peiyan Dong'; Lei Lu Chao Wu'n, Cheng Lyu', Geng Yuan?, Hao Tang" Yanzhi Wang' Northeastern University, University of Georgia, Carnegie Mellon University (dong.pe, pe, lu. 1ei1, cha.wu, yanz. wang) wnorthesstern.ccmmmm edu
Abstract
While Vision Transformers (VITs) have undoubtedly made impressive strides computer vision (CV), their intricate network structures necessitate substantial computation and memory resources. decision-making process for CV tasks typi- cally entails performing computations with low latency, which is tricky problem for VIT models. Model quantization is a widely-used technique to optimize the hardware efficiency of deep neural networks. Full quantization under Sub-8-bit precision, in particular, is promising solution to reduce inference latency signif- icantly. Unfortunately, current commodity hardware, such CPUs and GPUs, still struggles to efficiently execute these sub-8-bit quantized networks, as their SIMD instructions only support granularity of 8 bits wider. Also, there is scarcity of literature that presents full quantization paradigm for ViTs. In this paper, propose an activation-aware fully sub-8-bit quantization-awar training (QAT) framework called PackQViT for efficient yet accurate ViT acceleration on mobile devices to facilitate real-time AI- Al-powered revisiting data activation within the ViT dataflow, two characteristics are relevant quantization strategy and precision: the long-tailed distribution and systematic channel-wise outliers. In response, we employ either log2 quantization clipping address the long-tailed distribution and incorporate outlier-aware training for residual link quantization to regulate the various channel-wise outliers more consis- tently. Notably, due to the systematic fixed pattern, outler--waaee training approach predict the channel indices and regularized scales of outliers in advance, thus avoiding the runtime data-adaptive selection during inference. Furthermore, we employ Int-2"-Softmax Int-LayerNorm, and Integer GELU to enable integer-only computation flow. Finally, we develop a SIMD 4-bit packed multiplier to achieve end-to-end ViT acceleration mobile phones. Compared to prior studies on ViT quantization using 8-bit precision, PackQViT surpasses other works by improved accuracy ranging from 0.4% to 17.9% for various widely used ViT's on ImageNet dataset; under 4-bit precision, PackQViT demonstrates 0.4% higher accuracy. Compared the baseline multiplier, the Realme GT Android smartphone with Snapdragon 870 SoC CPU achieve 2.6 <..77 speedup under 8-bit scenario and 3.8 ~5.9 speedup under 4-bit which ensures practical real-time performance. Codes available at PackQViT.
Specifically,
implementations
Introduction
Transformers [3, 35] have experienced resurgence in recent times, with ViTs [16] demonstrating remarkable versatility across broad range domains, including computer vision (CV), e.g., image
"Equal Contribution
CCoresponding Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

The Speedup Ratio of PackQViT on Mulliple VITS.
Full Preeison LSQ
LSQ.8 Q-V-
Q PackQVIT4
PaISQVII 8
Model
INT8 w/o FullQ
INT8
INT4 mio FullQ
INT4
82
DeiT-T
2.5x
3.1x
3.6x
4,9
Tap- Aecurey) Imnggeer
DefI-8
2.7%
3.3x
3,9x
5.4x
79
3.7%
Swin-T
2.2x
2.6x
2.8x
3.8x
Swin-S
2.1x
2.6%
2.9%
3.8x
76
Dell's
Swin
Figure 1: PackQviT performance on the Realme GT Android smartphone with Snapdragon 870 SoC CPU. Left: Top-1 accuracy (%) ImageNet compared to SOTA QAT methods on 8-hit/4-bit precision. Right: The speedup ratio (x) compared to FP32 inference.
classification [16], object detection [9, 53], semantic segmentation [51], image processing [10], and video understanding [52], as well as in complex scenarios that involve multi-modal data. Moreover, ViTs have the potential to unify diverse application domains through shared architectures, addressing two of the most pressing challenges in deep learning: the reliance on limited domain somain-specific data; the need for constant model refinement to meet evolving demands. Given these strengths, ViTs are poised to become dominant force in the field of deep learning.
On the downside, ViTs are usually times slower than competitive convolutional neural networks (CNNs). Many factors limit the inference speed of VIT, including the massive number of pa- rameters and quadratic-increasing computation complexity with respect to token length. Model quantization is a promising approach to address the above-mentioned issues. Still, it faces the following challenges before ViTs become an indispensable staple of real-world applications on resource-constrainee hardware (e.g., augmented or virtual reality applications on mobile devices). (i) Researchers [23, 27, 11, 44, 34, 50, 36] have developed quantization techniques to reduce both computation and communication requirements for Transformers. However, most of these techniques are based on training-fres Post-Quaae dest-Quattization (PTQ), which was originally proposed for natural lan- guage processing (NLP) oriented Transformers [49, 14, 38, 6, 48, 46) and may lead accuracy drops when directly applied to ViTs. NLP-oriented transformers tend to have a larger model size than VIT [14], making it difficult utilize QAT on limited computing resources. In contrast, ViT can adapt QAT to minimize the quantization error incurred in PTQ. Although QAT has been applied to ViT in [29], it lacked an analysis of data distribution within networks and was not fully quantized, which constrained task accuracy and practical implementation. (ii) To enhance accuracy, ViTs use more hardware- -nnfeeree-- computations than CNNs (c.g.. GELU, LayerNorm. and Softmax). Previ- ous methods [34, 29] do not quantize Softmax/LayerNer GELU because quantization may cause significant accuracy degradation. However, data moving between different data domains, c.g., dequan- tizing and requantizing, and data movement between floating-poii and integer domains, will cause more hardware overhead. (iii) CPUs utilize SIMD units perform multiple operations in parallel efficiently. SIMD instructions can effectively exploit byte-level data (8-bit bit integers) parallelism and are well supported mainstream ISAS and DNN processing frameworks, e.g., GEMMLOWP [24] in TensorFlow-Lite and QNNPACK [18] in PyTorch. However, these low-precision libraries are ineffective for running sub-8-bit (bit- width<<8 quantized networks, as the SIMD-based units only support data parallel execution for 8 bits or wider. Consequently, this poses challenges in imple- menting sub-8-bit ViTs on mobile with both acceptable accuracy and lower runtime speed. In short, should address the hardware implementation issue while enjoying the additional optimization dimension provided by multi-head self-attention.
In this paper, we present novel framework called PackQViT for efficient and accurate end-to-end ViT inference on mobile devices using activation-avw aware full sub-8-bit quantization. Our approach involves conducting hardware profiling of ViTs on commercial mobile CPUs to identify an end-to-en -nd acceleration scheme and model full quantization. To decrease data precision without compromising model accuracy, we revisit the data distribution in ViTs, Weight data has a typical normal distribution, while activation data has the long-tailed distribution (attention maps and activation after GELU) and systematic channel-wel -ise outliers in the addition of residual links. Thus, we implement uniform

---

quantization on weight, log2 quantization on attention maps, and uniform quantization on activation values after GELU with the clipped range [-1, 10]. To address the issue of channel-wise outliers, we introduce an outlier-aware training for the addition of residual links. The outlier-aware training can predict the channel indices and regularize scales of outliers with a power-of-twe ratio, avoiding the complex hardware control logic for dynamic data selection during runtime inference. We modify the inference formulas Softmax, LayerNorm, and GELU, enabling integer-only inference. Finally, we develop a SIMD-based 4-bit packed multiplier to accelerate ViT inference on mobile devices.
Compared to state-of-the-an art (SOTA) ViT quantization studies [23, 27, 11, 44, 34] as shown in Fig- ure 1, Pac achieves better accuracy: Under 8-bit precision, PackQ ViT can achieve 0.4%~5.88 higher accuracy than other works and 0.9%~~2.3% better accuracy than the full precision models; PackQViT-Swin can achieve 1.4%~17.99% higher accuracy than other works and 0.9% better accuracy than the full precision models. Under 4-bit precision, PackQViT can achieve 0.4%~~2.8 8% higher accuracy than others and <0.4% accuracy drops than full precision models. Compared to the baseline multiplier on the Realme GT Android smartphone, our inference acceleration PackQ achieves 2.6 x ~3.7x speedup (8-bit) and 3.8 x 5.99% speedup (4-bit). According to our knowledge, Pack- QViT reaches the real-time performance (34.8 8 ms DeiT-T) of classic ViTs on the mobile phone for the first time. Overall, our contributions are summarized as follows:
We propose an activation-awer sub-8-bil full QAT framework for the efficient and effective inference of ViTs.
We propose an end-to-end integer framework equipped with a SIMD-based 4-bit packed multiplier for real-time VIT inference on mobile devices.
We conduct experiments to showcase the superior inference accuracy of PackQVIT under the sub-8-bit scenario compared to state-of-the-arn ViT quantization studies, while also highlighting its significant hardware efficiency.
2 Background and Related Work
2.1 Vision Transformers
Transformers which consist of multi-head self- attention (MSA) module and an MLP (FFN)
Satraax
GAL
Layesenn
module, both with LN the beginning, were initially designed to tackle long sequence learn- ing in NLP tasks. However, after the impres- sive success of pure transformer architec- ture for image classification [17], the interest in transformers for CV surged. This univer- sality of transformer architectures from NL.P to CV is attributed to the more uniform rep- resentations across all layers than CNNs, self- attention mechanism enabling early aggregation of global information, and VIT residual connec- tions that propagate features strongly from lower to higher layers [37]. Consequently, several ViTs [4, 40, 1, 21, 39, 47, 45, 28] have been proposed for various CV tasks, including object detection, semantic segmentation, and image retrieval, achieving competitive performance against CNN counterparts.
INT4
FELLINT
DeiT-T Latency (ms)
Masiietaliikiatio Laycneen
CUUU Olfore
Wisdonlpprviins
1P32
Fulk)
PULL INT
Fut
00:00
50:00
100
$00.00
313000
2009
Swin-T Latency (ms)
Figure 2: DeiT-T/Swin-T breakdown on Realme GT Android smartphone CPU.
2.2 End-to-End Acceleration Methods
Hardware Profiling Analysis. To better understand the runtime breakdown for ViTs, we profile DeiT- T and Swin-T, popular ViT models, on the Snapdragon 870 on-board CPU. We present the model in FP32 precision, INT4 w/o full quantization, and INT4 w/ full quantization. And non-full quantization means LayerNorm/Sofmma/GGE.
are still in the floating-pon paating-point&oo domain. In Figure 2, we observe that for the full precision De on an Android smartphone with Snapdragon 870 SoC CPU, matrix multiplications, i.e., Linear Transform, Q**T Attention Linear Projection, FCI, FC2, occupy the latency distribution close to 88.51%; Nonlincar operations, i.c., GELU, LayerNorm, SoftMax, merely
3

---

2
Value afher GELL in FTN 12-th block Figure 3: Left: A representative normal distribution of the weight of the 12-th FC1 layer in DeïT-T. Middle and Right: Long-tail distribution for attention map and activation after GELU. take the 10.16% of latency distribution. However, under the 4-bit w/o full quantization, nonlinear operations take 44.61% of latency distribution. Suppose we do the full 4-bit quantization on the model. In that case, nonlinear operations will only take about 24.21% of latency distribution, and the model speedup ratio will be improved from 3.6x to 4.9x as Table in Feature Similar to DeiT-T, the speedup ratio of Swin-T will be increased from 2.8x to 3.8x. Hence, research on full quantization for end-to-end acceleration imperative.
Weight Distribation of 12-th PCI DeTT-T
0.2
0.C
0.2
Attention Map with Quantization Lend
Quantization on Transformers. Quantization is one of the most powerful ways to decrease neu- ral networks' computational time and memory consumption. uses low-bit representations for weight and activation tensors. Low-bit fixed-point representations, such INT8 and INT4, further reduce energy consumption as the fixed-point operations are more efficient than their floating point counterparts [22]. Current quantization methods can be divided into two categories: QAT and PTQ. NLP-oriented Transformers mainly employ PTQ for three reasons [49, 14, 48]: too large model size (usually over 350M). not accessible dataset. limited academic computational resources to support the training of large language models. However, VIT's small model size and availability of public datasets make it suitable for QAT, which avoids the issues associated with PTQ that can lead to sub-optimal model performance or significant accuracy reduction without fine-tuning. To illustrate, previous PTQ approaches [34, 31] quantized VIT models to bits, resulting in only 1.2%~~1.8% accuracy decrease. [29] proposes a QAT method for ViTs with information-rectife and similarity-aware strategies. However, this work does not quantize Softmax, LayerNorm, and GELU modules, which hold significant execution time, e.g., 44.61* % of latency distribution for 4-bit DeÎT-T execution and 42.74% for 4-bit Swin-T execution mobile CPU (Figure 2). In PackQViT, we aim to implement an accurate, fully quantized VIT under QAT.
2.3 Low-Precision Linear Algebra Kernels
Low- now-precision linear algebra kernels aim to maximize computing throughput on low-precision operands by extending existing wider bit-width linear algebra kernels. Using lower-precision operands has been shown to improve performance in two ways. caches to fit more data and ii) lower-precision SIMD instructions to be utilized (e.g., vmlaq s80 in ARMv8 ISA) to process more elements in parallel than higher-prectte precision instructions (c.g., vmlaq f320). SOTA low- low-precision linear algebra kernels, such as Google's 's GEMMLOWP [24] and Facebook's 's QNNPACK [18], have been developed to maximize the throughput of low-precision operands, which are highly effective improving the DNN inference efficiency under the 8-bit quantization (W8A8) scenario, e.g., 3x end-to-end speedup compared to the FP32 baseline when running on PyTorch with a 64-bit ARM Cortex- A7tte CPU [2]. However, for more aggressive sub-8-bi bit quantization, it does not provide additional performance benefits because commodity CPUs only support Single Instruction Multiple Data (SIMD) operations of 8-bit greater precision. As result, low-precision kernels merely zero-extend the sub-8-bit operands to align them with byte boundaries, treating them as 8-bit operands.
3 Revisiting Data Distribution within ViTs
Unlike [32] which merely analyzes the activation value in the attention map and LayerNorm, we conduct a comprehensive analysis of the data distribution (weight& activation) of ViTs. Weight. The data has a standard normal distribution as shown in Figure 3. Activation. Two characteristics affect the quantization strategy: long-tail distribution attention maps activation after GELJ and channel-wil outliers in the addition of the residual link.

---

Chandd wbe Vorteten
Channe boiiee Variadien
2013
1973
Logel Dinner
Legnl (op)
BiStunt
Channd Index
Ruduans Rodurt Robaon Babt bul DelR See Seles
Jane
Figure 4: Left: Channel-wise minimum and maximum values of the second residual link addition the 9th block of Swin-T. Middle: Channel-wiss ranges of the last residual link addition in representative models. Right: Comparison of common Int-Softmax [32] and "n=2"--Softmx in quantized MSA inference.
Long-Taie Tail Distribution. Attention map. As model sizes increase, the storage and computation required for the Softmax with an attention map each head create a bottleneck that substantially impacts the throughput and latency of inference (see Section 2.2). To address this issue, we also investigated attention maps, which reveal a long-tail distribution (see Figure 3). Based
token pruning technigues [26, 15], the image-level redundancy causes most of the attention values to be centered around a small value and the rest of the values are discrete and converge to I as Figure Two cases are put here to show the attention sparsity. Compared to the uniform quantization which assigns only one bin to such large number of values, the log2 method (only with 4-bit) has more resolution to cover this data range. Activation after GELU, The GELU activation function in FFN leads to a truncation effect that heavily concentrates the resulting values around zero, showcasing clear one-sided stacking pattern (see Figure 3(b)).
A serious inter-channel variation in the addition of residual link. Fig- ure 4 depicts the channel-wise outliers in the last residual link. To provide a comparison, we display the activations of ResNets in the last outputs of the 4th step. Notably, the channel-wise ranges in ViTs exhibit more significant fluctuations than in ResNets. This is because the data flow of nonlinear functions, such Softmax and GELU, existing in the residual link of transformer structures, results in greater disparity between the input and output of the residual blocks compared CNN models. These results suggest that layer-wise quantization, with the same quantization parameters for all channels, would cause an unacceptable quantization error. Systematic outliers. Although outliers basically appear every sequence, they are concentrated < 6% fixed feature dimensions of the fixed encoder blocks, shown in Figure 4 (Left) (see Appendix 8). For instance, when testing 1024 images, outliers always occurred the 21 89th 146th and 189th channel of the 9th block in Swin-T. Furthermore, setting these outlier features to zero in DeiT-T causes the top-1 accuracy drop by 30% on ImageNet classification. Since the outlier locations follow a fixed pattern, anticipate them in advance, obviating the hardware design with complex control logic to support online adaa-iee selection.
Channel-Wise Outliers.
4 Activation-Aware Fully-Quantized ViTs
In detail, Section 4.1 presents the preliminary of quantization. In Sections 4.2-4.3, we propose two novel quantization methods to fit the distinct activation distribution inside ViTs, long-tail-aware guantization, and outlier-aware training for systematic channel-wise outliers quantization. Section 4.4 shows the SIMD-based 4-bit packed multiplier to support the 4-bit computation on mobiles practically.
4.1 Preliminary
Given the quantization bit-width is b, the quantizer ((X can be formulated function mapping a flotinng-poin number X to the nearest quantization bin. Among various quantizers, uniform [23] and log2 [7] are generally used. Apart from the special data distribution mentioned in Section 3, we apply symmetric layer-wise uniform quantization on weights and asymmetric channel-wise uniform quantization activations. For QAT training, the forward and backpropagation of the quantization
5

---

function in a quantized network is formulated as
Forward: Q Linear(x) = = =x:1=xx5,0xx(( (Qx(x) + _ Qw((w)
8x
25
05
if [4, otherwise
Backward:
di
dx
dx dx
0
(1)
a5 25 dx dw
ac as
if w E [u,,
dx di
dw
dx dw dw
otherwise
is loss function, straight-through estimator [5] is used obtain the derivation of the gradient in backpropagation for ((-). To achieve a fully quantized ViT. we leverage the analysis in Section 3 and quantize all modules, including Conv, Linear, MatMul, LayerNorm, Softmax, GELU, etc.
4.2 Long-Tail-Awerr Quantization
Two activation feature maps in dataflow exhibit long-tail distribution: the attention map and the activation after GELU. GELU truncates activation values be concentrated around 0, while the rest keep the original normal distribution. To allocate enough precision bits, we clip the activation value to [-1, 10] after GELU with uniform quantization. Moreover, we adapt GELU to support integer-only inference and regularize the quantization enror as [15]. In this context, we focus on the attention map.
Log2 Quantization for Attention Map. Since the output range (0, 1) Softmax makes the log2 quantization [7] calibration-free and log2 can convert the MatMul to more hardware-efficient BitShift, we propose:
Autno Q(Aun b) = clipp -oloo-loogooo (09g2(Auu)) (Aut) 00,2 1),
(2)
Attno -V 2-Att V/2 = Vi2 >> Attno 2M (Vo << (M Attmo)),
(3)
where 2 Note that directly right shifting Ve with the Attng may result in severe truncation error. M - Attno with scaling 2M can prevent the error through a left-shift operation.
Inference with Int-2"-Softmax. Replacing the natural constant inside the Softmax with [8] as shown in Figure 3, we propose Int-2"-Softmax with Log2 inference:
2
exp(s- XØ) R 5 220, Int-Softmax(s Xo) = M - log2
(4)
2%o
BitShift is implemented for the 2 and also for the integer log2 function. Experiments show that the replacement does not cause an accuracy drop under the same training receipt [42] (see Appendix 8).
We illustrate the difference between normal Int-Softmax [25] and method in Figure 4. On the left, we present the common Int-Softmax with log2 quantization. On the right, our Int-2"-Softmax replaces the foating-point exponential calculation with BitShift. Notably, i-exp is second-order polynomial that still involves the multiplication and addition in the floating-point domain, while Int-2"-Softmax employs 4-bit representation attention maps and replaces multiplication with BitShift operations, thereby enabling the inter-only dataflow, and reducing both computation and memory footprint.
4.3
Outlier-Aware Training for Systematic Channel- Wise Outliers Quantization
The addition of residual links contains many systematic and channel-wist outliers. To mitigate this, we propose the outlir-aware training approach that predicts the precise channel indices of the addition of residual links and regulated scales of the outliers with the power-of-twe ratio (PTR). Thanks to the systematic outliers, outlier-aware training can obviate the need for complex control logic during inference to support addatva selection.
Power-of- Twwo Ratio for Residual Link Quantization. Given the input activation BxXXC and the PTR rENC, then the quantized activation Xę can be formulated as:
X 2/5
max (X) min(X)
min(X) 2R 1) == =ccppp max(x)
Xe = Q(x(b) = clip(1
+, -
1,0,2- (5)

---

Algorithm 1: Outlier-Aweeee Training
Given the full-precision VIT Model, the test subdatasct of target Datasct D, the number of encoder blocks L, Epoch, for searching, Epoch, for fine-tuning, and quantized low-bit b;
// Step1: Initialize the PTC with the outlier estimated from the full sull-precision Model. foreach E (0..,111111111 do
To. Fo Caeckhhekkkap Outiers:;(MMddel D);
Quantize Model; by Eq. (5) with lo, Fo, into the QModely;
end
// Step2: Search for the channel index of outliers and determine the PTR by the regulerization.
Yo. i=to:
foreach eps E (0.1,...,Epochl
1] do // These tuo operations are gradient-fres.
1, Check Check_hutliernee (autlienssoMMddel))
argmin ||;; (R.
255 s22
// Following (2), Eq. (4) and (5).
taskioss- quantization/oss QModei b); Backward(task quantization/as):
12 end
// Step3: Finatune the b-bit quantizatoon
Fix and for outliers and quantize QModel with fine-tune Epochy;
14 Finalize the quantized VIT Model.
Hyperparameter R=max(r)=4 satisfies different inter-channel variations across models. We predict each outlier's channel index and PTR by the following outlier-aware training algorithm.
Outreer-Aware Training. This algorithm divided into three steps: Initialize the PTR by detecting the outlier of the full-precision model with 3o method [12]. Search for the channel index and the PTR r of each outlier with the /2 minimization. Fix the index and r obtained step 2 and fine-tune the model (see Algorithm 1).
Inference with Int-LayerNorm. During inference, with the gathered and r of systematic outliers, we extract the layer-wise parameters and and compute μt, a in the integer domain. Mcanwhile, thanks to the BitShift, PTR r can be calculated with the quantized Layernor efficiently (also refer to [25] and see Appendix 8).
4.4 SIMD-Based 4-Bit Packed Multiplier
In the 8-bit multiplier of the SIMD CPU, INT4 data is represented as INT8 with the high 4 bits set to which means there no benefit for the efficiency improvement of INT4 computation. We develop SIMD-based 4-bit packed multiplier (Figure 6): 1) Dot-multiplication In the SIMD kernel, concatenate two weights Wi,j and Wi+1,; from adjacent rows and multiply them with their shared activation value. The output is INT16 data type, with the first bits representing the result of the multiplication with Wi.j- and the last bits corresponding to W/+1J-- Following the SIMD memory mechanism where 16-bit of memory footprint will be freed up for the result of each dot-multiplication, we can avoid potential overflow issues. 2) Addition. Combined with the Bitshift operator, expand the 16-bit output in Step into 32-bit, where the s 8 bits are O, the 2nd 8 bits containing Output the 3rd being 0, and 4th containing Output;+1.j- Next, we perform row-by-row summation. Since 32-bit bit of memory footprint will be supplied for each addition, this process can handle up 2 times additions without overflow. which is sufficient for multi- attention (head-dimension = 32/64). Finally, we split the output into two INT16 values and quantize them back to INT4 value-level. which can be fused into the GeMM kernel.
Our method is different from per-channel quantization for outliers. We perform layer-wise quantiza- tion on the activation matrix; all the channels share the same quantization parameters, i.c., scaling factors and zero-point. However, the predicted power-of-two coefficients will refine the outlier channels' scaling factors. In the practical implementation, those power-of-two coefficients will be equivalently mathematically transformed over the corresponding weights, as shown in Figure 5.
7

---

X1 Xus Xu
Pa
01
[stn,
Quantice Carmpa QUNTS
Theoretioally 60% pultipicctions 62%
33
d
Figure 6: The paradigm of SIMD-basedr INT4 packed multiplier.
Power beo cooficerta for actierien chemreds
Our framework is compatible with per-channel quantization. However, common inference en- gines usually provide support only for layer- wise quantization on activation and per-channel quantization weights, such as the Gemm and Convolution operator configuration. For example, ArmComputel.ibray [41] only sup- ports channel-wise quantization configuration for weight matrix instead of the input activation.
1913
1914
W1d
A11
A31
A13
814
X
W31
W02
1953
1924
x8
A31
3co
A30
W11
1010
"Win
Win
Presaed www.:eeeeeedddeeeeeeooooooo
Figure 5: Matrix Multiplication Implementation for Power-of-Two Ratio.
5 Evaluation
Our experiments include activation-aware quantization and hardware implementation for multiple ViTs. To the best of our knowledge, there is no full quantization-aware training of ViTs at this point, so we reproduce and implement the ViT [29] and LSQ [19] methods by ourselves. Based on the proposed hardware design logic of the 4-bit multiplier, our team is committed to enhancing the support for lower-bit precision, such as 3-bit and 2-bit, which will be major area of our future development. Currently, we mainly present our model performance 8-bit and 4-bit precision.
5.1 Experiment Setup
Training Setup for PackQVIT Framework. The baseline models with 32-bit floating-pos ppint precision are from the Torch library [20]. Our experiments are conducted on the ImageNet-IK 1K [13] with two popular vision transformer implementations, including DeiT [43] and Swin [33]. Apart from special notes, we apply symmetric layer-wise quantization on weights and asymmetric channel-wise quantization activations. The hyperparameter R in Power-of-Twe Coefficients is set to 4. The training settings follow Q-ViT [29] with its DGD distillation, and the training process is executed on 4 NVIDIA V V000 GPUs. Because
execute the outiie--aware training in fine-tuned way, i.e., 70 epochs for step 2, 30 epochs for step 1, the training hours of the entire pipeline can be 37.5% -5%% of f of full precision models (see Appendix 8).
Hardware Platform. To validate the practical deployment efficiency of the proposed PackQVIT, we choose the Realme GT Master Android smartphone as the mobile platform, which is equipped with a Snapdragon 870 SoC consisting the onboard Octa-core Kryo 585 CPU. We control the testing process to single thread and let the 3.2GHz Cortex-A77 Main Core perform the computation to obtain the most stable acceleration ratio with the proposed framework. The latency has been reported via 100 iterations for each test. Our inference engine modified based on the ARM Compute Library. For the operators that are not yet supported in DeiT and Swin (such as Int-LayerNorm, Integer GELU, Int-2"-Softmax), we implement them based on the design hierarchy of 'kernel' to 'operator" to 'runtime' function, using parallel optimization based on ARM processor's SIMD support.
5.2 Model Accuracy and Speed Performance
This paper employs several popular post-training quantization methods, including MinMax, EMA [23], Percentile [27], OMSE [11], Bit-Split [44], PTQ for VIT [34], and FQ-ViT [32]. Also, we compare with the SOTA quantization- training method, Q-ViT [29]. For the sake of fairness, reproduced the results of Q- ViT with guantized LayerNorm and Softmax.

---

Table 1: Comparison of the Top-1 (%) accuracy with state-of of-t---rrtt methods on ImageNet dataset. We report bit width in the order of weighh/cttivationnattennoo
Method
#Bits
DeiT-T [43]
DeiT-S [43]
DeiT-B [43]
Swin-T [33]
Swin-S [33]
Full Precision
32/32/32
72.21
79.85
81.85
81.35
83.2
PTQ
MinMax
8/8/8
70.94
75.05
78.02
64.38
74.37
EMA
8/8/8
71.17
75.71
78.82
70.81
75.05
Percentile
8/8/8
71.47
76.57
78.37
78.78
78. 12
OMSE
8/8/8
71.3
75.03
79.57
79.3
78.96
Bit-Sphi
8/8/8
77.06
79.42
PTQ VIT
8/8/8
77.47
80.48
FQ-VIT
8/8/8
71.61
79.17
81.2
80.51
82.71
QAT
Q-ViT
8/8/8
73.6
80,2
82.4
81.8
83.6
PackOVIT
8/8/8
74.6
80.8
82.9
82.4
84,1
PackQVIT
8/8/4
74.5
80.8
82.9
82.3
84.1
LSQ
4/4/4
70.5
77.8
79
78.7
80.5
Q-VIT
4/4/4
72.1
79.1
81.1
82.4
PackQVIT
4/4/4
72.7
79.6
81.5
81.5
82.8
Table 2: Comparision of Objection Detec-
Table 3: Evaluating the components on DeiT-T.
tion using DETR-50 on COCO val2017.
Method
Top-1
#Bits
Top-1
Methods
#Bits
mAP
AP50
AP75
Full Procision
32-32
72.2
32-32
72.2
Baseline (29)
8-8
73.6
4.4
72.1
FP32
32
59.5
83.3
64.7
Lag2-Atten (Sec. 4.2)
8-8
74.1
444
72.4
VT-PTQ
8
57.6
82.3
63.1
Outlier aware Train (Sec. 4.3)
8-8
74.2
4-4
72.5
PackQViT
8
58.3
82.9
63.9
-Sotmmxx 4.2)
73.5
PackQViT
4
55.6
82.2
60
Lat-LayerNorm (Sec. 4.3)
8-8
73.6
4.4
72.1
PackQVVT
8-8
74.5
4-4
72.7
Image Classification on ImageNet. In the 8-bit scenario, SOTA methods for PTQ have suffered significant accuracy drop, ranging from 0.5% to 16.9% compared to the full precision baseline. However, QAT has shown promise in enhancing task accuracy by 0.5% to 2.4% over the baseline, thanks to its ability to minimize quantization errors and eliminate model redundancy. This suggests that redundancy in the original model hinders from converging to the optimum, as confirmed by the quantized DeiT-T model achieving 1.4% 2.4% higher accuracy than the baseline. While the current SOTA QAT method, Q-ViT, has made significant strides in correcting information distribution and knowledge transfer within ViTs, it falls short analyzing the internal data flow, leaving room for improvement in quantization accuracy. Additionally, Q-V still relies on floating-pons for Softmax, LayerNorm, and GELU, which makes challenging for efficient hardware deployment. In contrast, our proposed PackQViT leverages activation flow fitting to achieve an additional accuracy boost 0.6% to 1% over VTT. Moreover, when switching the data precision the attention matrix, the accuracy only reduces by less than 0.1 %, indicating that 4-bit log2 data type is sufficient to cover the data representation.
computations
In the 4-bit scenario, PickQViT can effectively prevent accuracy degradation by less than 0.4%. Compared to other QAT methods, PackQViT still achieves better accuracy performance of 0.4% to 2.8%. Furthermore, our method can be practically implemented on mobile devices using the ARM 8-bit multiplier and our proposed SIMD-based 4-bit Packed Multiplier.
Object Detection on MSCOCO. We compare object detection with transformers (DETR) [9], an end-to-end detector via transformer encoder-decoder. We perform our quantization on DETR-RSI We compare the large-scale COCO dataset [30] as shown in Table 2. We compare our method under the 4-bit precision. We report the detection performance of the 8-bit PTQ method, VT-PTQ [34].
For this more complex task, our method has a negligible detection performance degradation by 1.2% mAP, which is better than the VT-PTQ method by 0.7 mAP. For more aggressive quantization under bit the model has 3.9 mAP performance drops. For more complex tasks, quantization tends to lead to degradation of task performance.
Lateney Analysis. Based on Table 4, we can draw the following conclusions: 8-bit quantization can bring an overall acceleration ratio of 2.5x to 3x depending the size of the model, as the high computational workload on mobile devices can benefit from the higher utilization efficiency of limited memory resources under quantization algorithms. Specifically, INT8 quantization can achieve approximately 3.5x acceleration compared to FP32 in GeMM, which very beneficial for models like Transformers, where most mathematical operations are matrix operations (>80%

---

Table 4: Latency (ms) on mainstream processors on edge platforms (1-thread).
Model
Method
#Bits
Size(MB)
Android CPU (8)
RaspherryPii
RISC-V (s)
Full-precision
32
0.172
23.97
DeiT-T
PuckQVIT
8
5.2
0.056
7.73
1.11
PuckQVIT
4
2.7
0.037
5.15
0.74
Full-precision
32
88.2
0.363
39.75
7.2
DeiT-S
PackQViT
0.11
17.5
2.2
PackQViT
0.073
Full-pracision
32
346.2
0.858
143.4
17
DeiT-B
PuckQViT
8
86,8
0.233
38.24
4.6
PackQVIT
44.1
0.156
26.12
3.16
Full-precision
32
114.2
0,276
44.9
5.5
Swin-T
PackQVIT
0.099
16.4
2.1
PackQVTT
14.6
0.085
13.76
1.69
Full-preciston
32
199.8
0,55
89.78
10.9
Swin-S
PackQVIT
8
50.2
0.213
34.4
4.2
PackQViT
0.164
25.93
3.29
computation). Hence, ViT can obtaín more speedup gain than CNN, whose memory movement, such as weight reshaping, image2column, and column2image under different data layouts in each convolution operation, can dilute the efficient quantization speedup on GeMM. The 4-bit compression and concatenation technique can further improve this advantage, achieving approximately 1. 1.75x acceleration compared INT8 multiplication. This is because, while the theoretical computational workload is halved, overhead is introduced due to internal shifts of concatenated weights and the recovery of stored results in INT8 format.
Our 4-bit packed multiplier can be executed on mainstream processors on edge platforms (e.g.. mobile phones, Raspberry Pis, and RISC-V V 1oT processors), which face challenges in processing low-bit data since their SIMD instructions only support 8-bit or wider data granularity. Hence, we also test the speedup gain on other edge devices. Note that INT8 matrix multiplication within PackQViT utilizes the original byte-level quantized GeMM kernel. For the 8-bit configuration, models can be speedup by up to 3.7x and up to 2.7x for DeiT and Swin models, respectively. For the 4-bit configuration, models can be speedup by up to 5.5x and up to 3.5x for DeiT and Swin models through our proposed packed 4-bit multiplier.
5.3 Ablation Study
Breakdown of Task Accuracy. We give quantitative results of the proposed log2 quantization, outlier-aware training, and integer activation functions Table We Q-ViT the quantization baseline, which can improve the accuracy by 1.4%, which illustrates the redundancy inside the full precision model and suffers a performance drop by 0.1% in 4-bit settings. Log2 Attention and outlier- aware Training improve the performance when used alone, and integer activation functions almost maintain the original precision with only 0.1 % accuracy drops when setting ==4 iit-2"-Softmax When combining all the components to enable integer-only flow, the performance can be boosted considerably, e.g., log2 quantization improves the 8-bit baseline by 0.5% and outlier-aware training by 0.6%. In contrast, the combination with integer-only functions can reach 0.9%.
Conclusion and Limitations
This paper proposes an activation-aware fully sub-8-bit QAT framework called PackQViT for ViT inference acceleration the mobile. We first arrived at several critical observations regarding the data distribution in ViT' and pointed out two distinct characteristics in activations influencing the quantization strategy: long-tailes distribution, and systematic channel-wise outliers. To address the long-tailed distribution, we utilized 4-bit log2 quantization and clipping. For the systematic channel-wise outleer we designed outlier-awaee training to predict outliers' indexes and scale and regularize them with PTR in advance. We also develop SIMD-based 4-bit packed multiplier to support ViT and achieve end-to-end ViT acceleration on mobile phones. Experiments show that PackQVIT achieves superior task accuracy compared to SOTA quantization studies with significant hardware efficiency. Notably, we primarily validate our QAT methods with 4-bit precision. However, the PackQViT also holds the potential for application in lower precision scenarios. We are currently developing an implementation of lower-bit. SIMD-based multiplier to achieve enhanced acceleration.
7 Acknowledgment
This work is supported in part by the National Science Foundation CCF-1937500.
10

---

References
[1] Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, and Herve Jegou. Xcit: Cross-covariance image transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[2] Saad Ashfaq, MohammadHossein AskariHemmat, Sudhakar Sah, Ehsan Saboori, Olivier Mastropietro, and Alexander Hoffman. Accelerating deep learning model inference on arm cpus with ultra-low bit quantization and runtime. arXiv preprint arXiv:2207.08880) 2022.
[3] Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR), 2015.
[4] Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.0822 08254, 2021.
[5] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint T0t.:1308.444 2013.
[6] Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. EMNLP, 2021.
[7] Jingyong Cai, Masashi Takemoto, and Hironori Nakajo. A deep look into logarithmic quan- tization of model parameters in neural networks. In Proceedings of the 10th International Conference on Advances in Information Technology, pages 1-8, 2018.
[8] Gian Carlo Cardarilli, Luca Di Nunzio, Rocco Fazzolari, Daniele Giardino, Alberto Nannarelli, Marco Re, and Sergio Spanò. A pseudo-softmax function for hardware-based high speed image classification. Scientific reports, 11(1):15307, 2021.
[9] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to to-elttoddddddd object detection with transformers. In European Conference Computer Vision (ECCV), pages 213-229 Springer, 2020.
[10] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma. Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12299- 12310,, 2021.
[11] Yoni Choukroun, Eli Kravehik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 3009-3018, IEEE, 2019.
[12] Wikipedia contributors. 68-95-22 99.99.7 rule. In Wikipedia, The Free Encyclopedia, pages 0-1, 2022.
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255. leee, 2009.
[14] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 O): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 33:28089-22 1010089-28115 28154, 2021.
[15] Peiyan Dong, Mengshu Sun, Alec Lu, Yanyue Xie, Kenneth Liu, Zhenglun Kong, Xin Meng, Zhengang Li, Xue Lin, Zhenman Fang, et al. Heatvit: hardwaare-fffcient adaptive token pruning for vision transformers. In 2023 IEEE International Symposium on PerfoorPrfo Computer Architecture (HPCA), pages 442-455. IEEE, 2023.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlshy. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.

---

[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICL.R, 2021.
[18] Marat Dukhan, Yiming Wu, and Hao Lu. Qnnpack: Open source library for optimized mobile deep learning. 2018.
[19] Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar- mendra S Modha. Learned step size quantization. arXiv preprint arXiv:1902.088/333 2019.
[20] Facebook. Torchvision, 2021. Last accessed Sept 12, 2021.
[21] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. In Advances in Neural Information Processing Systems, 2021.
[22] Mark Horowitz. 1.1 computing's ss energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pages 10-14. IEEE, 2014.
[23] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-rrihmeeic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704-2713, 2018.
[24] Benoit Jacob and Pete Warden. gemmlowp: A small self-contained low-precision gemm library. Retrieved June, 14:2018, 2017.
[25] Schoon Kim, Amir Gholami. Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. I-bert: Integer-only bert quantization. In International conference on machine learning, pages 5506- 5518. PMLR, 2021.
[26] Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Bin Ren, Minghai Qin, Hao Tang, and Yanzhi Wang. Spvit: Enabling faster vision transformers via soft token pruning. arXiv preprint arXiv:2112. 38900, 2021.
[27] Rundong Li, Yan Wang, Feng Liang. Hongwei Qin, Junjie Yan, and Rui Fan. Fully quantized network for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 281 2810-2819, 2019.
[28] Wenhao Li, Hong Liu, Hao Tang, Pichao Wang, and Lue Van Gool. Mhformer: Multi-hypothesis transformer for 3d human pose estimation. In CVPR, 2022.
[29] Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Peng Gao, and Guodong Guo. Q-vit: Accurate and fully quantized low-bit vision transformer. Advances in Neural Information Processing Systems, 34:28092-288 28103, 2022.
[30] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[31] Yang Lin, Tianyu Zhang, Peiqin Sun, Zheng Li, and Shuchang Zhou. Fq-vit: Fully quantized vision transformer without retraining. International Joint Conferences on Artificial Intelligence Organization, 34:28092-28103. 2022.
[32] Yang Lin, Tianyu Zhang, Peiqin Sun, Zheng Li, and Shuchang Zhou. Fq-vit: Post-training quan- tization for fully quantized vision transformer. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 1173-1179, 2022.
[33] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. International Conference Computer Vision (ICCV), 2021.
12

---

[34] Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 28092-28100 2021.
[35] Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention model for natural language inference. In Proceedings of the 2016 Conference on Empirical Methods Natural Language Processing, pages 2249-2255, 2016.
[36] Gabriele Prato, Ella Charlaix, and Mehdi Rezagholizadeh. Fully quantized transformer for machine translation. In Findings of the Association for Computational Linguisties: EMNLP 2020, pages 1-14, 2020.
[37] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Doso- vitskiy. Do vision transformers see like convolutional neural networks? arXiv preprint arXiv:2108..m 08810, 2021.
[38] Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8815-882) 2020.
[39] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers, arXiv preprint arXiv:2106. 10270, 2021.
[40] Hao Tang, Zhenyu Zhang, Humphrey Shi, Bo Li, Ling Shao, Nicu Sebe, Radu Timofte, and Luc Van Gool. Graph transformer gans for graph-constrained house generation. In CVPR, 2023.
[41] ARM team. Arm computelibrary simd. 19(21), 2023.
[42] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Deit: Training data-efficient image transformers amp; distillation through attention. In International Conference on Machine Learning, volume 139, pages 10347-10357,
July 2021.
[43] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training cata-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357, 2021.
[44] Peisong Wang, Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training network quantization via bit-split and stitching. In International Conference on Machine Learning, pages 9847-9856. PMLR, 2020.
[45] Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative position encoding for vision transformer. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pages 10033-10041, 2021.
[46] Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. Advances in Neural Information Processing Systems, 34:27318-27311 2022.
[47] Guanglei Yang, Hao Tang, Mingli Ding, Nicu Sebe, and Elisa Ricci. Transformer-based attention networks for continuous pixel-wise prediction. In ICCV, 2021.
[48] Zhewei Yao, Reza Yazdani Aminabadi. Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168 05:27168-277 271888,, 2022.
[49] Ali Hadi Zadeh, Isak Edo, Omar Mohamed Awad, and Andreas Moshovos. Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference. In 2020 53rd Annual IEEF/ACM International Symposium on Microarchitecture (MICRO), pages 81 1 1--224 IEFE, 2020.
13

---

[50] Zihan Zhao, Yuncong Liu, Lu Chen, Qi Liu, Rao Ma, and Kai Yu. An investigation on different underlying quantization schemes for pre-trained language models. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 359-371. Springer, 2020.
[51] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to -euuuueeee----suuenee perspective with transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 6881-6890, 2021.
[52] Luowei Zhou, Yingbo Zhou, Jason J Corso, Richard Socher, and Caiming Xiong. End-to-end dense video captioning with masked transformer. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages
2018.
[53] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations (ICLR), 2020.
14


Reviews:
[{"review_id": "jfnPwyL0Vn", "reviewer": "Reviewer_yEDd", "summary": "In this paper, the authors proposed PackQViT, a fast and accurate sub-8-bit quantization for running ViT on mobile devices. To maintain accuracy, the authors proposed different quantization schemes for activation and weights, and an outlier-aware training stage. For efficiency, they proposed fully integer softmax and layernorm, and a SIMD-based 4-bit packed multiplier. Overall, the method can accelerate the inference of ViT models on mobile devices at a small accuracy degradation. \n", "questions": "Please see above 1 and 2 in the weakness part. ", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Firstly, the paper is generally well written and easy to follow. \n2. The authors present a carefully designed algorithm & system co-design solution to improve the accuracy and efficiency of the ViT models. The observation on the different distribution of models weights and activations are quite insightful.\n3. The results are very solid, with both high accuracy numbers and actual speed up under 8-bit/4-bit cases.", "weaknesses": "1. The accuracy numbers in Table 1 and 3 are a bit confusing: in many cases, the 8-bit accuracy is higher than the original FP32 model, which makes the ablation study hard to compare. Does it indicate the original model is not well trained? It is more proper to use a model trained with a good schedule, instead of undertrained ones. \n2. The paper lacks a detailed discussion on the kernel implementation. There are a lot of different workloads like integer quantization, log quantization, packed 4-bit multiplication, etc. It would be very helpful to give more discussion on how to implement the kernel so that it runs efficiently on smartphone SoC.  \n3, It would be more convincing if the authors can also add some results on other tasks apart from image classification, like detection, for example.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**ReW1**: We appreciate your valuable comments since this allows us to clarify this phenomenon.\nThe accuracy of the full precision model in Table 1 of our paper is sourced from the original paper of DeiT and Swin, both of which do not perform the DGD distillation techniques. Q-ViT proposes DGD distillation, and our training setting uses this distillation technique, as mentioned in Line 270. This is why we can get better accuracy results than the full precision models. It is worth mentioning that we can achieve significantly better task accuracy with the same training setting than Q-ViT.\n\n**Table A.** Comparison of Full Precision models w/ & w/o DGD Distillation\n|Model| DGD Distillation | #Bits | Top-1 Acc.(%)|\n|------------|-----|-----|-----|\n|DeiT-T|N    |32  |72.2\t|\n|DeiT-T|Y    |32 |74.4\t|\n|Swin-T|N    |32  |81.35\t|\t\t\t\n|Swin-T|Y    |32. |82.2\t|\t\n\nFollowing your valuable suggestions, we also train the full precision model from scratch with the DGD distillation techniques, as shown in Table A. It shows that the model accuracy can be further increased by 2.2% and 0.85% for DeiT-T and Swin-T, respectively. For more analysis of the distillation techniques, please refer to Reviewer 7eJH: W2.\n\n**ReW2**: We appreciate your valuable comments since this allows us to describe our hardware deployment in detail.\nThe implementation below has been programmed based on ArmISAs and tested on the maincore 3.2GHz Cortex A77 of Snapdragon 870 onboard CPU. The kernels have been implemented within ArmComputeLibrary v22.05 [1] Inference framework.\n\n(1) Integer Quantization Computation Flow in current inference engines:\n\tQuantizeLayer(Scale, Zeropoint) -> Quantized Operators -> DequantizeLayer (Scale, Zeropoint) with the data type conversion sequence of: FP32 -> INT -> FP32 -> …, thus the inter-operator data transmission is still in floating-point format, ensuring the mathematical correctness with scale & zeropoint of the current quantization inference.\n\t\n(2) Softmax Implementation based on Log2Quantization:\n\n\tLog2Softmax(in_ptr, tmp_ptr, length):\n    sum = 0\n    vec_max = Globalmax\n    /* 2^n & Reduced Sum */\n    for i = 0 to length, step 16:\n        in = load(in_ptr + i)\n        in = vec_max - in\n        in = 1 << in;\n        sum = sum + sum_vec(in)\n        store(tmp_ptr + i, in)\n    /* Normalize Log2SFM */\n    for i = 0 to length, step 16:\n        temp = load(tmp_ptr + i)\n        res = log2(Approximate_nearest_POW2(sum / temp))\n        res = M - res\n\nCompared with the computation of exp(x) in common softmax inference, where the approximation is usually calculated via 3 sequential steps: Range Reduction, Polynomial Approximation, and Reconstruction, which requires more than 13 floating-point operation instructions for a single function implementation (e.g., sfm_impl.cpp on ArmComputeLibrary), our flow removes the costly floating-point calculations together with reducing memory allocation for temporal result storage. Via integer-based LeftShift operations and parallel processing for 16 operands, efficient Softmax with preserved accuracy and computational savings could be achieved.\n\n(3) Packed 4-bit Multiplier:\n\tThe breakdown of MLA (multiplication & Addition) with MUL and ADD does not introduce any obvious inference latency differences according to our benchmarks on both Snapdragon 870 onboard-CPU and RaspberryPi 4B, opening up the possibility of inserting result-adjustment auxiliary operations between the MUL and ADD to achieve SMMW (single-multiplication-multiple-weight).\nThe 4-bit Multiplier has been implemented based on ARM ISAs following: \n\tMUL (Multiplication), LSL (LeftShift), ORR (BitwiseOR), AND (BitwiseAND) the same process as Figure 5 in our paper.\n\t\n\t4-bit_GeMM_4x4(i, s1, s2):\n    s1 = [ 4x4 matrix of src1 ]\n    s2 = [ 1x4 vector of src2 ]\n    c = [ 4x4 matrix of zeros ]\n    mask = [ 1x4 vector of 0x00FF00FF ]\n    p = [ 1x4 vector with zeros ]\n    /* Inner loop for 16 elements\n       4 units per loop */\n    for j in range(4):\n        /* Lane-wise Multiplication */\n        p = [multiply the j-th row of s1 with the i-th element of s2]\n        /* Product Rearrangement */\n        t = [left shift elements of p by 8 bits]\n        p = [bitwise OR between p, t]\n        p = [bitwise AND between p, mask]\n        /* Accumulation */\n        c.row[j] = [add elements of p to the corresponding j-th row of c]\n    return c\n**ReW3**:  We further compare object detection with transformers (DETR) [2], an end-to-end detector via a transformer encoder-decoder. We perform our quantization on DETR-R50. We compare the large-scale COCO [3] dataset as shown in Table B. We compare our method under the 4-bit precision. We also report the detection performance of the 8-bit PTQ method, VT-PTQ [4].\n\n**Table B.** Evaluation Results with our proposed method using DETR-50 on COCO val2017. #Bits denotes the bit-width of weights, activations, and attention activations are quantized into #Bits precision.\n|Methods| #Bits |mAP | AP50| AP75| \n|------------|-----|-----|-----|-----|\n|full-precision|32  |\t59.5|\t83.3|\t64.7|\n|VT-PTQ       |8    |\t57.6|\t82.3|\t63.1|\n|Ours             |8    |\t58.3|\t82.9|\t63.9|\n|Ours\t\t|4    |\t55.6|\t82.2|\t60|\n\nFor this more complex task, our method has a negligible detection performance degradation by 1.2% mAP, which is better than the VT-PTQ method by 0.7 mAP. For more aggressive quantization under 4-bit, the model has 3.9 mAP performance drops. For more complex tasks, quantization tends to lead to a degradation of task performance.\n\nThanks again for your suggestions to improve our paper presentation. We will add these details in our final version.\n\n[1] ARM, Available: https://arm-software.github.io/ComputeLibrary/v22.05.\n\n[2] Carion,  et al. \"End-to-end object detection with transformers\" ECCV, 2020.\n\n[3] Lin, et al. \"Microsoft coco: Common objects in context\", ECCV, 2014.\n\n[4] Liu,  et al. \"Post-training quantization for vision transformer\", Neurips, 2021.\n"}, {"review_id": "oZgWCJsX5Y", "reviewer": "Reviewer_7eJH", "summary": "This work started from analyzing the distribution of weights and activations of ViT models, identified several characteristics of critical layers, and applied Integer and hardware friendly techniques to address the issues. Furthermore, the author demonstrated a fully quantized ViT (including LayerNorm, Softmax, and GELU) and did a nice profiling works on mobile CPU to verify the benefits of INT8 and INT4 models. Specifically, INT4 instructions are not natively supported on this HW platform hence required a custom kernel based on INT8 instruction sets. Eventually, the author achieved 3-6x speed-up over FP32 model with comparable or slightly higher accuracy on ImageNet-1k.    ", "questions": "1. The choice of log2-attention seems to be a bit counter-intuitive. As the attention map is generally understood as \"highlighting the importance of the content\", why would adding more granularities to the \"unimportant part\", i.e. close to 0 attention score, help the model accuracy? In fact, if this technique helps as the author claims, we would expect it to be more apparent on INT4 rather than INT8 on Table 3, because effective utilization of the limited bins should be more critical for INT4. \n\n2. Around **Line 186**, author compares the way channel-wise outliers are handled to \"online data-adaptive selection\", (which is sometimes called dynamic quantization). But this technique as described in **Sec 4.3** would be closer to a \"per-channel quantization for activations.\" And the per-channel option might be more generalizable than the technique described here. Maybe the author could comment on the differences and requirement for HW/SW implementation between the reported technique vs per-channel quantization for activations.  ", "limitations": "No societal impact.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\nMarginal\nMost of the techniques employed in this work seems to have been reported in Ref 31, FQ-ViT, already. The new components are distribution analysis, outlier-aware training, the implementation of specialized HW kernels, and using profiling to demo the importance of quantizing Softmax/LayerNorm/GELU in addition to matmuls on mobile CPU. The reviewer would consider this as \"application to a new domain\" compared to Ref 31.\n\nQuality:\nGood\n\nClarity:\nGood\n\nSignificance:\nCould be a good reference for quantization on mobile devices.", "weaknesses": "1. Novelty, as mentioned above.\n\n2. A few claims/explanations may not be well-supported, for example:\n**Line 292**, *\"(QAT having higher accuracy than FP baseline) suggests that redundancy in the original model hinders it from converging to the optimum\"* \n\nBecause the QAT in this work is a 70-epochs fine-tuning on top of the baseline FP32, the accuracy improvement could have been derived from this fine-tuning instead of the quantization itself. In fact, this behavior is not uncommon in QAT/fine-tuning. On the other hand, to support the \"explanation\", the author would need to \"fine-tune\" the FP32 with the same settings and compare it with INT8 results.  \n\nIn **section 5.3**, take INT4 cases as an example, the author seems to compare accuracy differences within a small range based on a single run or best record. It would be more reasonable if the author could report a 5-run average of the baseline or FP32 model. After taking run-to-run variations into account, author could have claimed some options \"comparable\" instead of \"suffer a performance drop by 0.1%\" as on **Line 323.** \n\n3. Although it's obvious for readers with background knowledge, it would be better to define variables like z and s in Eq 1, as these are used in later equations as well. Also a couple typos, like Eq 5 and third row in the blue block on Fig. 5, . ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**ReW1**: Our difference between Ref [31] can be categorized into four aspects, including motivations, optimization of task accuracy, optimization of hardware efficiency, and evaluation results.\n\nMotivation: We analyze data distribution within ViTs (long-tail distribution and outlier analysis), which provides insights on quantizing model weights and activations effectively to enhance task accuracy and on-device efficiency. Meanwhile, we conduct a hardware latency profiling to figure out the runtime bottleneck of the whole models, which guides us to optimize operator kernels for efficiency.\n\nOptimization of task accuracy: Different from Ref [31] which uses an input-adaptive way to runtime detect and optimize outliers during model inference. We present one structured pattern of channel-wise outliers: most outliers (over 91.5%) appear in the fixed channels of the fixed block. So we predict outliers' channel index and power-of-two coefficients and mitigate the adverse effects of outliers through outlier-aware training, not dynamic inference. To validate this structured/fixed pattern, we run multiple models from small sizes (DeiT-T/Swin-T) to large models (DeiT-B/Swin-B) on ImageNet-1k, MS COCO, and Cifar-100.\n\nOptimization of hardware efficiency: (1) We figure out the nonlinear operators as the speed bottleneck. Then, we adapt them into hardware-friendly ones differently from Ref [31]. Int-2n-Softmax: We replace the exponent e with a faster operation,  power-of-2 to bring more efficiency benefits without accuracy degradation;  GeLU: Based on [5], we adapt it with efficient hardware implementation (detailed implementation in Reviewer yEDd: ReW2 and Reviewer ysuz: ReW1). We have mentioned these in our appendix and will add more details in our final version. (2) Ref [31] uses a runtime dynamic selection method to optimize outliers, while we leverage structured/fixed pattern of data outliers to predict their channel index and power-of-two coefficients in advance through training. Then we can preload these indexes and coefficients with the model weight and compute them simultaneously with the matrix multiplication between weights and activations. Thus, our design can avoid dynamic overhead which is significantly huge in the practical implementation. Also, the dynamic processing on activations is not well-supported in the common inference engine, such as ArmComputeLibrary. (3) We quantize models into lower bits 4-bit. To achieve practical acceleration, we propose the packed 4-bit multiplier to support this sub-8-bit matrix multiplication, as mentioned in  Reviewer yEDd: W2.\n\nEvaluation results: We not only can get better accuracy results than Ref [31], but also accelerate end-to-end on-device latency performance of the quantized ViTs on multiple edge devices (Results in Reviewer QVMx: ReW1).\n\n**ReW2**: (1) Following your suggestion, we add a group of experiments that only fine-tune models under the same setting (100 epochs & distillation) without quantization, as shown in Table C.\n\n**Table C.**\n|Model   | Fine Tuning   | #Bits | Top-1 Acc.(%)|\n|-----------|------------------|-----|-----|\n| DeiT-T | N (baseline)  | 32 | 72.2   |\n| DeiT-T | Y                   | 32 | 74.1   |\n| DeiT-T | Y                   | 8   | 74.5   |\n| Swin-T | N (baseline)  | 32 | 81.35 |\n| Swin-T | Y                   | 32 | 82      |\n| Swin-T | Y                   | 8   | 82.3   |\n\nWith pure finetuning, the accuracy of full-precision models can be increased by 1.9% and 0.65% for DeiT-T and Swin-T, respectively. Combined with quantization, the accuracy can be further increased by 0.4% and 0.3% for DeiT-T and Swin-T. This illustrates that quantization-aware training to 8-bit can contribute to redundancy reduction to enhance task accuracy (the same as [6]). Even though the finetuning can help the accuracy, our method deploys the same distillation as Q-ViT and ends with better task accuracy than it.\n\n(2) We have run the Q-ViT baseline 5 more times these days. We get the average result of 0.15% with a standard deviation = 1.0724. We will add these details in Line 323. \n\n**ReW3**: We'll fix these typos in our final version.\n\n**ReQ1**: 1. Remaining unimportant tokens' information still helps improve task accuracy [7]. We cannot use the token packing technique mentioned in [7] which averages the unimportant tokens into one package token to recover the information within unimportant tokens. Still, we can mitigate the degree of aggressive quantization. 2. In Table 3 of our paper, the accuracy increase is 0.5% for INT8 and 0.3% for INT4. Information degradation is so much in 4-bit, which will mitigate the recoverability of 4-bit. \n\n**ReQ2**: Our method is different from per-channel quantization for outliers. We perform layer-wise quantization on the activation matrix; all the channels share the same quantization parameters, i.e., scaling factors and zero-point. However, the predicted power-of-two coefficients will refine the outlier channels' scaling factors. In the practical implementation, those power-of-two coefficients will be equivalently mathematically transformed over the corresponding weights, as shown in Figure A.\n\nOur framework is compatible with per-channel quantization. However, common inference engines usually provide support only for layer-wise quantization on activation and per-channel quantization on weights, such as the Gemm and Convolution operator configuration. For example, ArmComputeLibrary[1] only supports channel-wise quantization configuration for weight matrix, instead of the input activation. \n\nThese questions can help us enrich the content of the paper. We will add these details to our final version.\n\n[5] Sehoon, et al. \"I-bert: Integer-only bert quantization\", ICML, 2021.\n\n[6] Liang, et al. \"Pruning and quantization for deep neural network acceleration: A survey.\" Neurocomputing, 2021.\n\n[7] Kong, et al. \"Spvit: Enabling faster vision transformers via latency-aware soft token prunin,\" ECCV, 2022.\n\n\n\n\n"}, {"review_id": "9OuWCXQxep", "reviewer": "Reviewer_QVMx", "summary": "This paper introduces an innovative framework to optimize the implementation of Vision Transformers (ViTs) on mobile devices using sub-8-bit quantization. This framework identifies and addresses key challenges such as long-tailed distribution and systematic channel-wise outliers within ViT dataflow. Through the use of a three-step outlier-aware training process and an integer-only computation flow, PackQViT enhances the performance of ViTs. Additionally, a SIMD-based 4-bit packed multiplier was developed for efficient ViT acceleration. Compared to previous studies, PackQViT shows significant improvements in accuracy on the ImageNet dataset and speedups on a mobile implementation, marking a substantial advancement in the deployment of ViTs in resource-constrained environments.\n", "questions": "The Outlier-Aware Training methodology merits a more comprehensive explanation to enhance clarity. Can various inputs X yield diverse outliers, as well as predicted channel indices i and Power-of-Two Ratios r? What strategies are employed to identify and manage these outliers for differing inputs? How to fix the index i and r obtained in step 2 if the i and r is input-dependent? Furthermore, what is the protocol if multiple outliers are detected within the same channel?\n", "limitations": "The evaluation of PackQViT solely on the Snapdragon 870 SoC CPU limits the generalizability of the findings to a specific hardware environment. To establish a stronger foundation, it would indeed be beneficial to test PackQViT on a broader range of hardware devices representing diverse hardware and software environments commonly encountered in real-world applications.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This work directly addresses the numerous challenges faced by the application of ViTs on mobile devices and provides very practical solutions. More importantly, the authors have deployed and tested the algorithm mentioned in this paper on real hardware devices, which makes the paper more practical. Although they don't tell a very fancy story, this pragmatic style is exactly what the field of model compression and quantization needs today.\n\n- The paper identifies and tackles key challenges within the ViT dataflow, including long-tailed distribution and systematic channel-wise outliers, making it effective for real-world deployments.\n- The paper demonstrates a practical implementation of full sub-8-bit quantization for ViTs, a promising yet challenging technique for improving hardware efficiency. It also employs an innovative SIMD-based 4-bit packed multiplier for better end-to-end acceleration.\n- The proposed three-step outlier-aware training approach can predict and regularize outliers, reducing complex hardware control logic during runtime inference.", "weaknesses": "The effectiveness of PackQViT is indeed reliant on specific hardware configurations, particularly SIMD-based units. The evaluation of PackQViT was conducted on a Snapdragon 870 SoC CPU, and its performance may vary on different hardware architectures. The results were presented primarily for ImageNet dataset and might not generalize across all kinds of data and diverse application domains.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**ReW1**: For more results on another task, e.g., object detection, please refer to Reviewer yEDd: ReW3 which can validate that our proposed method can be generalized to other image domains. 2. Our 4-bit packed multiplier can be executed on mainstream processors on edge platforms (e.g., mobile phones, Raspberry Pis, and RISC-V IoT processor), which face challenges in processing low-bit data since their SIMD instructions only support 8-bit or wider data granularity. Hence, we test the speedup gain on multiple edge devices as shown in Table D. Note that INT8 matrix multiplication within PackQViT utilizes the original byte-level quantized GeMM kernel. For the 8-bit configuration, models can be speedup by up to 3.7x and up to 2.7x for DeiT and Swin models, respectively. For the 4-bit configuration,  models can be speedup by up to 5.5x and up to 3.5x for DeiT and Swin models, through our proposed packed 4-bit multiplier.  Thanks for your valuable suggestions. This one is constructive in enhancing our paper's generalization. We will add these results to the appendix of our final paper version.\n\n**Table D.** Hardware Results under Different Precision for Various ViTs on Multiple Edge Devices\n| Model | Method           | #Bits | Size(MB) | Android CPU (s) | RaspberryPi (s) | RISC-V (s)|\n|-----------|-------------------|-----|-----|-----|------|-----|\n| DeiT-T | Full-precision | 32 | 20.1 | 0.172 | 23.97 | 3.4   |\n| DeiT-T |PackQViT         | 8   | 5.2   | 0.056 | 7.73   | 1.11 |\n| DeiT-T | PackQViT      | 4   | 2.7   | 0.037 | 5.15   | 0.74 |\n| DeiT-S | Full-precision | 32 | 88.2 | 0.363 | 59.75 | 7.2   |\n| DeiT-S | PackQViT        | 8   | 22.2 | 0.110 | 17.5   | 2.2   |\n| DeiT-S | PackQViT      | 4   | 11.4 | 0.073 | 11.97 | 1.46 |\n| DeiT-B | Full-precision | 32 | 346.2 | 0.858 | 143.4 | 17   |\n| DeiT-B | PackQViT       | 8   | 86.8 | 0.233 | 38.24   | 4.6   |\n| DeiT-B | PackQViT      | 4   | 44.1 | 0.156 | 26.12 | 3.16 |\n| Swin-T | Full-precision | 32 | 114.2 | 0.276 | 44.9 | 5.5  |\n| Swin-T | PackQViT        | 8   | 28.6 | 0.099 | 16.4   | 2.1  |\n| Swin-T | PackQViT      | 4   | 14.6 | 0.085 | 13.76 | 1.69 |\n| Swin-S | Full-precision | 32 | 199.8 | 0.55 | 89.78 | 10.9  |\n| Swin-S | PackQViT       | 8   | 50.2 | 0.213 | 34.4   | 4.2  |\n| Swin-S | PackQViT      | 4   | 25.5 | 0.164 | 25.93 | 3.29 |\n\n**ReQ1**: We show one structured pattern of channel-wise outliers: most of the outliers (over 91.5%) appear in the fixed channels of the fixed block. To validate this structured/fixed pattern, we run multiple models from small sizes (DeiT-T/Swin-T) to large models (DeiT-B/Swin-B) on ImageNet-1k, MS COCO, and Cifar-100. This similar phenomenon is found in [8] for Bert models. We performed a similar analysis on ViTs and verified this law's applicability to ViTs. Even though some outliers are not in the predicted channels, the related quantization error can be eliminated by our training process. Furthermore, suppose multiple outliers are detected within the same channel. In that case, we still use the power-of-two coefficient to refine the scaling factor of that channel, which regularizes the data distribution into a smaller range that is similar to others. After that, we can perform layer-wise quantization on the whole activation matrix. Please refer to Reviewer 7eJH: Q2 for more hardware implementation details for this trick.\n\n**ReLimitation.**: Thanks for your valuable suggestions which can validate the generalization of the proposed packed 4-bit multiplier. We also test the inference implementation on various edge devices, including the RealmeGT Android Phone with Snapdragon 870 SoC, Raspberry4 B with Quad-core CPU and the simulator for RISC-V ultra-low power IoT processor. Please refer to ReW1 for detailed results. \n\n[8] Dettmers, Tim, et al. \"Llm. int8 (): 8-bit matrix multiplication for transformers at scale.\" Neurips, 2022.\n"}, {"review_id": "IkTYWg1xQB", "reviewer": "Reviewer_ktdW", "summary": "This paper introduces a quantisation framework for end-to-end ViT inference on CPU devices. The main idea is to exploit the byte-level data parallelism present in the SIMD units of CPUs, and present a comprehensive set of techniques to enable low-latency ViT inference. The proposed framework includes two parts: quantization tricks , and a data-parallel strategy using SIMD instructions. To prove the efficacy of the proposed framework, the paper presents an accuracy evaluation (Table 1) and a performance for the ViT inference on CPUs (Table 2).", "questions": "- How do you see the proposed approach being used as PTQ? If it cannot be used as a PTQ method, why? If it can, how does it compare to other methods in Table 1?\n- Why would this only work on CPUs? What about the INT8 support on GPUs?\n- What performance gain does each optimisation provide? For example, can you isolate each one and explain how much performance gain it delivers?", "limitations": "I do not think this is applicable to this paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper has successfully demonstrated improved latency on mobile devices, verifying that these results are not just theoretical. Furthermore, the paper highlights the great performance achieved in the QAT setup (see Table 1). The major benefit of this approach is the reduced engineering effort when packing items into SIMD units.", "weaknesses": "The two major issues with this paper are firstly, its evaluation, and secondly, its originality. Regarding its evaluation, the paper is a QAT method but it is only compared to various PTQ methods in Table1. This is not a fair comparison unless you showcase the PTQ performance under the same precision setup. Furthermore, I am uncertain how much accuracy each method is providing (e.g. int-layernorm, log2 quan...).\n\nThis paper lacks novelty, as the proposed quantisation is merely an amalgamation of existing methods. Specifically, treating channel-wise outliers is widely discussed in LLM quantisation work (e.g. LLM.int8()) and various quantisation methods in the CNN domain. Furthermore, the int-layernorm is also from existing work. The only notably new contribution appears to be log2 quantisation on activations and softmax in my opinion, although this is not focused on in the paper, limiting its novelty claim.\n\nThe presentation is far from its best state. The color schemes of this paper are suboptimal (e.g., Figures 1 and 2). Generally, all fonts used in the figures are too small and this really impairs readability. Furthermore, names such as \"Int8 No full\" in Figure 1 are quite confusing. Additionally, Figure 4 is too small and the rightmost figure is overlapped with the one in the middle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**ReW1**: Thanks for your questions helping us clarify the paper. (1) We have reported our results with QAT in the lower part of Table 1 in our paper. We also provide our analysis of each method in the ablation study, as shown in Table 3 of our paper. (2) Our approach is compatible with PTQ. We report our method only under the PTQ setting as shown in Table E. Since the accuracy optimization of outlier-aware training cannot be leveraged, our results are comparable to the SOTA PTQ methods.\n\n**Table E.** Comparison of the Top-1 (%) accuracy with the SOTA PTQ method on the ImageNet dataset.\n| Method | #Bits | DeiT-T | DeiT-S | Swin-T | Swin-S |\n|-----------|-------------------|-----|-----|-----|------|\n| FQ-VIT | 8/8/8 | 71.6 | 77.1 | 81.2 | 80.5 | 82.7 |\n| ours      | 8/8/8 | 71.4 | 77.4 | 81.4 | 80.1 | 82.7 |\n\n**ReW2**: (1) LLM.int8() is a literature on a large language model, Bert. Thanks to its inspiration, we conducted a series of further explorations on ViT models.\nLLM.int8() utilizes per-channel quantization with mixed precision to reduce the adverse outlier effect of activations. However, per-channel quantization is not a hardware-friendly way and is not supported in current main computing frameworks, as mentioned in Reviewer 7eJH: Q2. In contrast, we deploy outlier-aware training and layer-wise quantization with power-of-two coefficients on the outlier channel, which enhances the task accuracy with negligible hardware overhead.\n\n(2) Please refer to Reviewer 7eJH: ReW1 which we clarify our novelties.\n\n**ReW3**: “INT8 No full” means we only quantize weights and activations in matrix multiplication, while leaving the nonlinear operations, e.g., GeLU, Softmax, and LayerNorm still in floating-point domain computation. We will change “int8 No full” in Figure 1 into “INT8 w/o FullQ” with detailed notes. We draw Figure 2 again to make our hardware profiling clearer.  Also, we make Figures 1 and 4 clearer and upload these figures in our appended PDF. We apologize for our ambiguity. And we will make it clearer in our final paper version. \n\n**ReQ1**: Please refer to W1.\n\n**ReQ2**: Currently, integer-based SIMD support is widely adopted at the CPU level. Many mainstream inference engines utilize integer-based SIMD acceleration for quantized linear operators, leading to significant reductions in latency. However, current mobile GPUs offer varying levels of support for INT8 instructions. For example, Arm has started supporting 8-bit sdot from Mali G71, G52, etc., while still remaining unsupported for other series. Thus, current quantized inference on GPU still lacks a standardized design framework. To our knowledge, no inference engine utilizes genuine 8-bit integer instructions to construct the GPU inference kernel. SUMMIN KIM and his team pointed out that, ‘there are currently few frameworks that support INT8 quantization for mobile GPUs’ in “Performance Evaluation of INT8 Quantized Inference on Mobile GPUs,” published on IEEE Access. As claimed in the official doc of TensorflowLite, they claimed that ‘we run quantized models by giving it a ‘floating-point view’ of the original model’ based on floating-point execution backend. Besides, the supporting document of ArmComputeLibrary illustrates the requirement of INT8 conversion to INT32 before actual execution. Thus, our computation flow has not yet been promoted onto quantized inference on mobile GPUs.\n\nThe framework we propose is a versatile solution that, in theory, can be adapted to various backends that support true integer-based fixed-point inference. However, we have not yet extended its application to the mobile GPU domain due to the lack of sufficient hardware integer-processing capability or slow integration of computation libraries on mobile GPUs.\n\nBut that shall be the next stage of our research. We will first try to establish an applicable and generalizable quantization framework for mobile GPU, including both a new quantization algorithm design and int8-oriented kernel implementation. Then, we would expand the low-bit inference onto the mobile GPU field.\n\n**ReQ3**: Thanks for helping us to break down our optimization gain in detail both from the accuracy and latency part.\n\n(1) For the breakdown analysis of our accuracy, please refer to the ablation study and Table 3 in our paper, where we break down our accuracy gain for each sub-method.\n(2) For the optimization of hardware efficiency, you can refer to our latency profiling Figure in Q3. Also, Table F shows the actual speedup on Arm-based platforms of our hardware-friendly design of nonlinear operations.\n\nTable F. Operator speedUp compared with the original implementation.\n| Operator                      | Snapdragon 870 onboard CPU |\n|------------------------------ | ------------------------- |\n| Softmax                       | 3.7x                      |\n| GeLU                          | 3.9x                      |\n| Layernorm                     | 1.4x                      |\n\n Table G. Evaluating non-linear operator optimization gain on DeiT-T.\n| Method                      | Latency(ms) |\n|------------------------------ | ------------------------- |\n| Int4-w/o FullQuantization                       | 46.7                       |\n| +  Int-$2^{4}$-Softmax                          | 41.2                      |\n| + I-GeLU                     | 36.6                      |\n| + Int-Layernorm               | 34.8        |\n\nAs shown in Table G, the integer-based optimization on 3 non-linear operators would provide an overall latency decrease of approximately 12ms, more than 25% of the original non-optimal method. The elimination of non-linear operators built on floating-point instructions from the overall computation flow has led to significant efficiency improvements. This optimization becomes even more meaningful when applied alongside 4-bit quantization, where non-linear operations share a higher latency proportion with further acceleration on matrix multiplication.\n\n"}, {"review_id": "YzJttnmyZ6", "reviewer": "Reviewer_R5sB", "summary": "This paper is about developing a quantization method for ViT, including DeiT and Swin Transformer. The developed algorithm can be run on mobile phones. Experiments were conducted on several datasets, with comparisons to other quantization methods. ", "questions": "no other questions. See the weakness part for comments. ", "limitations": "it may reduce the accuracy for some application scenarios, using quantization. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "a new strategy for quantization of ViT, mainly for inference;\n\nsaving computation cost and memory usage in inference. ", "weaknesses": "The presentation is not clear in some places. The authors should re-organize the parts of introduction and method, making it clearer and easy understanding for readers. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**ReW1**: Thank you so much for improving our paper presentation. We will rewrite the final paragraph of our introduction as follows:\n\nOur contribution can be categorized into four aspects, including motivations, optimization of task accuracy, optimization of hardware efficiency, and evaluation results.\n\nMotivation: We first perform the analysis of data distribution within ViTs (long-tail distribution and outlier analysis), which provides insights on quantizing model weights and activations effectively in order to enhance task accuracy and on-device efficiency. Meanwhile, we conduct a hardware latency profiling to figure out the runtime bottleneck of the whole models, which guides us to optimize operator kernels for efficiency.\n\nOptimization of task accuracy: We apply the log2 on attention to deal with the long-tail distribution of the attention matrix and deploy the outlier-aware training to eliminate accuracy degradation caused by outlier quantization. \n\nOptimization of hardware efficiency: (1) We figure out the nonlinear operators as the speed bottleneck. Then, we adapt them into hardware-friendly ones. Int-2n-Softmax, Int-GeLU, Int-layerNorm; (2) We leverage structured/fixed patterns of data outliers to predict their channel index and power-of-two ratio in advance through training. Then we can preload these indexes and ratios with the model weight and compute them at the same time with the matrix multiplication between weights and activations. Thus, our design can avoid dynamic hardware overhead; (3) We quantize models into 4-bit and propose the packed 4-bit multiplier to support this sub-8-bit matrix multiplication.\n\nEvaluation results: We not only can get superior accuracy results than existing works but also accelerate end-to-end on-device latency performance of the quantized ViTs on multiple edge devices.\n\nMeanwhile, following these organizational logic motivations, optimization of task accuracy, and optimization of hardware efficiency, we will restructure the presentation of our methodology.\n\nThank you for supplying this chance for us to make our paper clearer.\n\n**ReLimitation**: Thanks for your kind reminder. Please refer to Reviewer yEDd: ReW3 for the detailed results on object detection. Indeed, it shows that for more complex tasks, quantization tends to lead to a degradation of task performance.\nThanks again for your valuable questions. In our final version, we will add one discussion part to clarify this point.\n"}, {"review_id": "fVvABRik4E", "reviewer": "Reviewer_ysuz", "summary": "The paper proposes a 4-bit vision Transformer and claims its efficient deployment on mobile CPUs. The attention module is quantized with mixed quantization schemes, i.e., inputs are quantized using different methods, logarithmic and uniform linear. To run 4-bit inference on CPUs, the paper also presents a manually designed packed matmul kernel that leverages the SIMD instructions.", "questions": "The main concerns have already included several questions. More questions that can improve the paper quality if addressed are listed as follows:\n\n1. Line 176 claims that Softmax and GELU lead to the disparity of channel-wise outliers between ViTs and CNNs. This is questionable because (1) softmax is bounded, so it would not cause outliers as shown in Figure 4 (left); (2) GeLU is very similar to ReLU in terms of forward propagation. It does not explain the disparity.\n\n2. Line 203 claims that \"GELU truncates activation values to be centered around 0\". This is questionable because again GELU is very similar to ReLU. It has no centering effect.\n\n3. What is the model size in the experiments?\n\n4. In Section 5.3 ablation study, is there any explanation on why combing all int quantization will have quality improvement compared to used alone?", "limitations": "The paper discusses limitations in Section 6. The reviewer is not aware of any social impact of this paper.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This work managed to quantize a vision Transformer to 4-bit. Most importantly, it runs efficient inference on CPUs, which only has supports for up to 8-bit instructions. The demonstration of the potential of the 4-bit integer quantization for mobile platforms is nice to the community. The latency breakdown of a vision Transformer provides good insights.", "weaknesses": "My main concern is the lack of details on the quantization methodology. I will list them as follows:\n\n1. The paper claims integer-only computation flow, but does not explain how this is achieved. For example, how exactly is the GELU function quantized? To achieve a good model quality, quantization functions usually introduce a scaling factor for pre-quantization and post-quantization steps, which includes floating-point operations. Are there such scaling operations in GELU and shortcut quantization here? What is the overhead then? In Figure 4 (right), why are outputs INT8? The outputs correspond to Equation 3 and the accumulation should be high-precision. Are they quantized again? These details are missing.\n\n2. In Section 4.2, the BitShift operation is introduced to compute the matmul between attention scores and value matrices. This is a new operation other than matmul. It is unclear how this is accelerated on CPUs. If another kernel is designed specifically for this operation, how does its performance compare to INT8 SIMD matmul?\n\n3. It is unclear how much benefit quantizing shortcuts can bring since its overhead may be overwhelming for elementwise operations. There needs to be some experimental analysis on this. If the benefits are more on the memory side, then the paper needs to explain the memory layout of the shortcuts, e.g., are the integer shortcuts packed on the fly when transferring to the DRAM?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**ReW1.**: (1) For efficient implementation of GeLU, we revisit I-GeLU [5] segmentation function with a=-0.2888, b=-1.769, x$\\in$Z as Eq. 1 in our appended PDF.\n\nWithin the (-2,2) data range, the value of  I-GeLU is {-0.0257, -0.1152, 0, 0.5919, 1.3885} under the integer domain.\nSo we directly use their quantized value, which is equal to adapting Eq. 1 into a segmentation function with linear kernels and lookup table under INT scenarios as Eq. 2. Before Int-GeLU$_{imp}$, we do the quantization and just send the integer value directly into these modified nonlinear functions. We can also minimize the quantization error of this modification through our QAT training.\n\nImplementation flow for I-GeLU$_{imp}$ kernel based on ArmISAs:\n\n\tI_GeLU_imp(in_ptr, length):\n    threshold = 3\n    oneVec  = 1\n    zeroVec = 0\n    for i = 0 to length, step 16:\n        /* 16 operands  per loop */\n        vec = load(in_ptr + i) \n        /*  set as 1 when  input = 2 */\n        mask1 = check_equal(vec, 2)\n        vec = select(mask1, oneVec, vec)\n        /* Mask-based BitwiseZeroSetting below threshold */\n        mask2 = compare(vec, threshold)  \n        vec = select(mask2, zeroVec, vec)\n        /* Result Storage */\n        store(in_ptr + i, vec) \n, where the actual execution only utilizes efficient integer-based comparison and bitwise selection intrinsics like vceqq_s8 and  vbslq_s8 on ArmISAs instead of complicated floating-point-level raw GeLU calculations.\n\n(2)For the proceeding on the shortcut/residual link: The integer-only means we conduct the inner-operator computation based on the integer computation flow. We currently cannot avoid the floating-point based inter-operator data transmission due to the unavoidable ‘Dequantize’ process of common inference frameworks that would require a predefined scale & zeropoint for ‘integer to floating’ conversion after each operator execution. Also, the latency occupied by residual link addition is trivial, and making it into an integer domain won’t lead to obvious high hardware benefits. In our paper, we do the quantization on the addition results of the residual link, but the addition itself is still computed in FP32. We will revise the ‘integer-only data flow’ into ‘integer-only inner operation inside each computing kernel, which makes the main computation flow keep integer computing.”\n\n(3) For Figure 4, you are right and that value should be INT12. However, we clip the overfitting number into the maximum of INT8, 127. \n\n**ReW2.**: Benchmark on Multiplication Versus BitShifting: We have benchmarked the instruction-level of integer-based multiplication versus BitShifting on on-board Snapdragon 870 SoC.\nFirstly, naive instructions have been deployed: where a * b (b = 4) versus a << 2, each operation has been conducted for 100k iterations for stable latency measurement, based on 10 separate tests we’ve done, Bitshifting generally achieves 1.71x speedup over multiplication. Then, with NEON SIMD support on 128-bit processing registers where ‘vmulq_s8’ (vector_multiplication) and ‘vshlq_n_s8’ (vector_left_shift_lanewise) have been utilized, based on 10 separate tests we’ve done, Bitshifting generally achieves 1.62x speedup over multiplication. Thus, Matmul based on Bitshift operation is friendly to CPUs instead of calling another GeMM kernel, which aligns with the idea of computation efficiency of our article.\n\n**ReW3.**: Just as we mentioned in ReW1, we keep residual links in the floating-point domain and just quantize the addition value of residual links into the integer domain. Sorry for the confusing points again!\n\n**ReQ1.**: In Line 177, we mentioned, “This is because the data flow of nonlinear functions, such as Softmax and GELU, existing in the residual link of transformer structures, results in a greater disparity between the input and output of the residual blocks compared to CNN models. ” We should change ‘such as Softmax and GELU’ into “i.e., LayerNorm, Softmax and GELU.’ According to our visualization (Figure B), these three operators can generate different dynamic ranges of their input/output activation tensors, especially for LayerNorm and GELU. The cumulative difference will result in a considerable mismatch between the different dynamic ranges of activation tensors in the residual connections. And this potentially excites significant outliers in the data flow [9]. We will clarify it in our final version.\n\n**ReQ2.**: We want to mention the long-tail distribution caused by GeLU as Eq. 1, which makes much value turn into 0, as shown in Figure C. We will make this clearer in our final version. Thank you for such a thorough review. You help us a lot to improve our paper presentation.\n\n**ReQ3.**: Please refer to Reviewer QVMx: ReW1.\n\n**ReQ4.**: Our baseline is another QAT method, Q-ViT. Each ‘+’ in Table 3 of Section 5.3 means we correct one disadvantage of Q-ViT. After each added technique, the accuracy compared with baseline, Q-ViT, can be improved. So after combining them all together, the improved accuracy can be better than one trick used alone. Thanks again for your careful and responsible review.\n\n Your comments really help us improve our paper and also give us deeper thinking on ViT structures and on-device implementation. Because of the limited pages, we are sorry that we cannot elaborate on the details.  We will add more in the appendix in our final version.\n\n[9] Bondarenko,  et al. \"Understanding and overcoming the challenges of efficient transformer quantization.\" arXiv preprint arXiv:2109.12948 (2021).\n\n\n\n"}]

Author rebuttal:
We first sincerely thank every reviewer for your insightful and constructive feedback. Then, we will answer the specific questions from each reviewer. We upload a pdf file with figures (Figure A,B,C) and equations (Equation 1,2) that we will present in our rebuttal.



Meta-review: This paper presents a framework for quantization vision transformer (ViT) models on both linear and non-linear operations allowing their deployment on edge devices. The paper applies various quantization schemes based on the data-distribution analysis to preserve model accuracy. Additionally, custom kernels are implemented, showing speedup on edge CPUs.  

The paper is in general well-written. The insights provided through data distribution analysis and the latency breakdown are particularly enlightening. The introduced framework is practical. The evaluation is comprehensive, demonstrating model accuracy across diverse tasks and acceleration on various edge devices.

During the rebuttal phase, the authors conscientiously responded to queries and concerns, providing additional data and detailed explanation on kernel implementation, object detection task results, and latency on diverse edge devices. The effort has led to improvement on reviewer ratings. Those additional materials should be integrated into the final version of the paper.

The main weakness of the paper is its relatively limited novelty, particularly regarding the quantization techniques which seems to draw from existing methods. However, considering this framework to be valuable in practical as a reference for the community seeking to deploy ViT models on edge devices, the paper is recommended for acceptance.
Decision: Accept (poster)