SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Peiyan Dong $^{1*}$ , Lei Lu $^{1*}$ , Chao Wu $^{1*}$ , Cheng Lyu $^{1}$ , Geng Yuan $^{2}$ , Hao Tang $^{3\dagger}$ , Yanzhi Wang $^{1}$
<sup>1</sup>Northeastern University, <sup>2</sup>University of Georgia, <sup>3</sup>Carnegie Mellon University
{dong.pe, lu.lei1, cha.wu, yanz.wang}@northeastern.edu
While Vision Transformers (ViTs) have undoubtedly made impressive strides in computer vision (CV), their intricate network structures necessitate substantial computation and memory resources. A decision-making process for CV tasks typically entails performing computations with low latency, which is a tricky problem for ViT models. Model quantization is a widely-used technique to optimize the hardware efficiency of deep neural networks. Full quantization under Sub-8-bit precision, in particular, is a promising solution to reduce inference latency significantly. Unfortunately, current commodity hardware, such as CPUs and GPUs, still struggles to efficiently execute these sub-8-bit quantized networks, as their SIMD instructions only support a granularity of 8 bits or wider. Also, there is a scarcity of literature that presents a full quantization paradigm for ViTs. In this paper, we propose an activation-aware fully sub-8-bit quantization-aware training (QAT) framework called PackQViT for efficient yet accurate ViT acceleration on mobile devices to facilitate real-time AI-powered decision-making. Specifically, in revisiting data activation within the ViT dataflow, two characteristics are relevant to quantization strategy and precision: the long-tailed distribution and systematic channel-wise outliers. In response, we employ either log2 quantization or clipping to address the long-tailed distribution and incorporate outlier-aware training for residual link quantization to regulate the various channel-wise outliers more consistently. Notably, due to the systematic fixed pattern, outlier-aware training approach can predict the channel indices and regularized scales of outliers in advance, thus avoiding the runtime data-adaptive selection during inference. Furthermore, we employ Int-2 $^n$ -Softmax, Int-LayerNorm, and Integer GELU to enable integer-only computation flow. Finally, we develop a SIMD-based 4-bit packed multiplier to achieve end-to-end ViT acceleration on mobile phones. Compared to prior studies on ViT quantization using 8-bit precision, PackQViT surpasses other works by an improved accuracy ranging from  $0.4\%$  to  $17.9\%$  for various widely used ViTs on ImageNet dataset; under 4-bit precision, PackQViT demonstrates  $0.4\% \sim 2.8\%$  higher accuracy. Compared to the baseline multiplier, our implementations on the Realme GT Android smartphone with Snapdragon 870 SoC CPU achieve  $2.6 \times \sim 3.7 \times$  speedup under 8-bit scenario and  $3.8 \times \sim 5.9 \times$  speedup under 4-bit which ensures practical real-time performance. Codes available at PackQViT.
Transformers [3, 35] have experienced a resurgence in recent times, with ViTs [16] demonstrating remarkable versatility across a broad range of domains, including computer vision (CV), e.g., image
Figure 1: PackQViT performance on the Realme GT Android smartphone with Snapdragon 870 SoC CPU. Left: Top-1 accuracy (\%) on ImageNet compared to SOTA QAT methods on 8-bit/4-bit precision. Right: The speedup ratio (x) compared to FP32 inference.
The Speedup Ratio of PackQViT on Multiple ViTs.
Model	INT8 w/o FullQ	INT8	INT4 w/o FullQ	INT4
DeiT-T	2.5x	3.1x	3.6x	4.9x
DeiT-S	2.7x	3.3x	3.9x	5.4x
DeiT-B	3.0x	3.7x	4.4x	5.8x
Swin-T	2.2x	2.6x	2.8x	3.8x
Swin-S	2.1x	2.6x	2.9x	3.8x
classification [16], object detection [9, 53], semantic segmentation [51], image processing [10], and video understanding [52], as well as in complex scenarios that involve multi-modal data. Moreover, ViTs have the potential to unify diverse application domains through shared architectures, addressing two of the most pressing challenges in deep learning:  $\bullet$  the reliance on limited domain-specific data;  $\bullet$  the need for constant model refinement to meet evolving demands. Given these strengths, ViTs are poised to become a dominant force in the field of deep learning.
On the downside, ViTs are usually times slower than competitive convolutional neural networks (CNNs). Many factors limit the inference speed of ViT, including the massive number of parameters and quadratic-increasing computation complexity with respect to token length. Model quantization is a promising approach to address the above-mentioned issues. Still, it faces the following challenges before ViTs become an indispensable staple of real-world applications on resource-constrained hardware (e.g., augmented or virtual reality applications on mobile devices). (i) Researchers [23, 27, 11, 44, 34, 50, 36] have developed quantization techniques to reduce both computation and communication requirements for Transformers. However, most of these techniques are based on training-free Post-Quantization (PTQ), which was originally proposed for natural language processing (NLP) oriented Transformers [49, 14, 38, 6, 48, 46] and may lead to accuracy drops when directly applied to ViTs. NLP-oriented transformers tend to have a larger model size than ViT [14], making it difficult to utilize QAT on limited computing resources. In contrast, ViT can adapt QAT to minimize the quantization error incurred in PTQ. Although QAT has been applied to ViT in [29], it lacked an analysis of data distribution within networks and was not fully quantized, which constrained task accuracy and practical implementation. (ii) To enhance accuracy, ViTs use more hardware-unfriendly computations than CNNs (e.g., GELU, LayerNorm, and Softmax). Previous methods [34, 29] do not quantize Softmax/LayerNorm/GELU because quantization may cause significant accuracy degradation. However, data moving between different data domains, e.g., dequantizing and requantizing, and data movement between floating-point and integer domains, will cause more hardware overhead. (iii) CPUs utilize SIMD units to perform multiple operations in parallel efficiently. SIMD instructions can effectively exploit byte-level data (8-bit integers) parallelism and are well supported in mainstream ISAs and DNN processing frameworks, e.g., GEMMLOWP [24] in TensorFlow-Lite, and QNNPACK [18] in PyTorch. However, these low-precision libraries are ineffective for running sub-8-bit (bit-width  $< 8$ ) quantized networks, as the SIMD-based units only support data parallel execution for 8 bits or wider. Consequently, this poses challenges in implementing sub-8-bit ViTs on mobile with both acceptable accuracy and lower runtime speed. In short, we should address the hardware implementation issue while enjoying the additional optimization dimension provided by multi-head self-attention.
In this paper, we present a novel framework called PackQViT for efficient and accurate end-to-end ViT inference on mobile devices using activation-aware full sub-8-bit quantization. Our approach involves conducting hardware profiling of ViTs on commercial mobile CPUs to identify an end-to-end acceleration scheme and model full quantization. To decrease data precision without compromising model accuracy, we revisit the data distribution in ViTs. Weight data has a typical normal distribution, while activation data has the long-tailed distribution (attention maps and activation after GELU) and systematic channel-wise outliers in the addition of residual links. Thus, we implement uniform
quantization on weight, log2 quantization on attention maps, and uniform quantization on activation values after GELU with the clipped range [-1, 10]. To address the issue of channel-wise outliers, we introduce an outlier-aware training for the addition of residual links. The outlier-aware training can predict the channel indices and regularize scales of outliers with a power-of-two ratio, avoiding the complex hardware control logic for dynamic data selection during runtime inference. We modify the inference formulas of Softmax, LayerNorm, and GELU, enabling integer-only inference. Finally, we develop a SIMD-based 4-bit packed multiplier to accelerate ViT inference on mobile devices.
Compared to state-of-the-art (SOTA) ViT quantization studies [23, 27, 11, 44, 34] as shown in Figure 1, PackQViT achieves better accuracy: Under 8-bit precision, PackQViT can achieve  $0.4\% \sim 5.8\%$  higher accuracy than other works and  $0.9\% \sim 2.3\%$  better accuracy than the full precision models; PackQViT-Swin can achieve  $1.4\% \sim 17.9\%$  higher accuracy than other works and  $0.9\%$  better accuracy than the full precision models. Under 4-bit precision, PackQViT can achieve  $0.4\% \sim 2.8\%$  higher accuracy than others and  $< 0.4\%$  accuracy drops than full precision models. Compared to the baseline multiplier on the Realme GT Android smartphone, our inference acceleration PackQViT achieves  $2.6 \times \sim 3.7 \times$  speedup (8-bit) and  $3.8 \times \sim 5.9 \times$  speedup (4-bit). According to our knowledge, PackQViT reaches the real-time performance (34.8 ms in DeiT-T) of classic ViTs on the mobile phone for the first time. Overall, our contributions are summarized as follows:
- We propose an activation-aware sub-8-bit full QAT framework for the efficient and effective inference of ViTs.
- We propose an end-to-end integer framework equipped with a SIMD-based 4-bit packed multiplier for real-time ViT inference on mobile devices.
- We conduct experiments to showcase the superior inference accuracy of PackQViT under the sub-8-bit scenario compared to state-of-the-art ViT quantization studies, while also highlighting its significant hardware efficiency.
2.1 Vision Transformers
Transformers which consist of a multi-head self-attention (MSA) module and an MLP (FFN) module, both with LN at the beginning, were initially designed to tackle long sequence learning in NLP tasks. However, after the impressive success of a pure transformer architecture for image classification [17], the interest in transformers for CV surged. This universality of transformer architectures from NLP to CV is attributed to the more uniform representations across all layers than CNNs, self-attention mechanism enabling early aggregation of global information, and ViT residual connections that propagate features strongly from lower to higher layers [37]. Consequently, several ViTs [4, 40, 1, 21, 39, 47, 45, 28] have been proposed for various CV tasks, including object
detection, semantic segmentation, and image retrieval, achieving competitive performance against CNN counterparts.
Figure 2: DeiT-T/Swin-T breakdown on Realme GT Android smartphone CPU.
2.2 End-to-End Acceleration Methods
Hardware Profiling Analysis. To better understand the runtime breakdown for ViTs, we profile DeiT and Swin-T, popular ViT models, on the Snapdragon 870 on-board CPU. We present the model in FP32 precision, INT4 w/o full quantization, and INT4 w/ full quantization. And non-full quantization means LayerNorm/Softmax/GELU are still in the floating-point domain. In Figure 2, we observe that for the full precision DeiT-T on an Android smartphone with a Snapdragon 870 SoC CPU, matrix multiplications, i.e., Linear Transform,  $Q^{*}K^{T}$ , Attention*V, Linear Projection, FC1, FC2, occupy the latency distribution close to 88.51%; Nonlinear operations, i.e., GELU, LayerNorm, SoftMax, merely
Figure 3: Left: A representative normal distribution of the weight of the 12-th FC1 layer in DeiT-T. Middle and Right: Long-tail distribution for attention map and activation after GELU.
take the  $10.16\%$  of latency distribution. However, under the 4-bit w/o full quantization, nonlinear operations take  $44.61\%$  of latency distribution. Suppose we do the full 4-bit quantization on the model. In that case, nonlinear operations will only take about  $24.21\%$  of latency distribution, and the model speedup ratio will be improved from  $3.6x$  to  $4.9x$  as Table in Feature 1. Similar to DeiT-T, the speedup ratio of Swin-T will be increased from  $2.8x$  to  $3.8x$ . Hence, research on full quantization for end-to-end acceleration is imperative.
Quantization on Transformers. Quantization is one of the most powerful ways to decrease neural networks' computational time and memory consumption. It uses low-bit representations for weight and activation tensors. Low-bit fixed-point representations, such as INT8 and INT4, further reduce energy consumption as the fixed-point operations are more efficient than their floating-point counterparts [22]. Current quantization methods can be divided into two categories: QAT and PTQ. NLP-oriented Transformers mainly employ PTQ for three reasons [49, 14, 48]:  $\bullet$  too large model size (usually over  $350\mathrm{M}$ ).  $\bullet$  not accessible dataset.  $\bullet$  limited academic computational resources to support the training of large language models. However, ViT's small model size and availability of public datasets make it suitable for QAT, which avoids the issues associated with PTQ that can lead to sub-optimal model performance or significant accuracy reduction without fine-tuning. To illustrate, previous PTQ approaches [34, 31] quantized ViT models to 8 bits, resulting in only a  $1.2\% \sim 1.8\%$  accuracy decrease. [29] proposes a QAT method for ViTs with information-rectified and similarity-aware strategies. However, this work does not quantize Softmax, LayerNorm, and GELU modules, which hold significant execution time, e.g.,  $44.61\%$  of latency distribution for 4-bit DeiT-T execution and  $42.74\%$  for 4-bit Swin-T execution on mobile CPU (Figure 2). In PackQViT, we aim to implement an accurate, fully quantized ViT under QAT.
2.3 Low-Precision Linear Algebra Kernels
Low-precision linear algebra kernels aim to maximize computing throughput on low-precision operands by extending existing wider bit-width linear algebra kernels. Using lower-precision operands has been shown to improve performance in two ways. i) caches to fit more data and ii) lower-precision SIMD instructions to be utilized (e.g., vmlaq s8() in ARMv8 ISA) to process more elements in parallel than higher-precision instructions (e.g., vmlaq f32)). SOTA low-precision linear algebra kernels, such as Google's GEMMLOWP [24] and Facebook's QNNPACK [18], have been developed to maximize the throughput of low-precision operands, which are highly effective in improving the DNN inference efficiency under the 8-bit quantization (W8A8) scenario, e.g.,  $3 \times$  end-to-end speedup compared to the FP32 baseline when running on PyTorch with a 64-bit ARM Cortex-A72 CPU [2]. However, for more aggressive sub-8-bit quantization, it does not provide additional performance benefits because commodity CPUs only support Single Instruction Multiple Data (SIMD) operations of 8-bit or greater precision. As a result, low-precision kernels merely zero-extend the sub-8-bit operands to align them with byte boundaries, treating them as 8-bit operands.
3 Revisiting Data Distribution within ViTs
Unlike [32] which merely analyzes the activation value in the attention map and LayerNorm, we conduct a comprehensive analysis of the data distribution (weight & activation) of ViTs. • Weight. The data has a standard normal distribution as shown in Figure 3. • Activation. Two characteristics affect the quantization strategy: long-tail distribution – attention maps & activation after GELU and channel-wise outliers – in the addition of the residual link.
Figure 4: Left: Channel-wise minimum and maximum values of the second residual link addition in the  $9^{th}$  block of Swin-T. Middle: Channel-wise ranges of the last residual link addition in representative models. Right: Comparison of common Int-Softmax [32] and Int- $2^{n}$ -Softmax in quantized MSA inference.
Long-Tail Distribution.  $\bullet$  Attention map. As model sizes increase, the storage and computation required for the Softmax with an attention map in each head create a bottleneck that substantially impacts the throughput and latency of inference (see Section 2.2). To address this issue, we also investigated attention maps, which reveal a long-tail distribution (see Figure 3). Based on token pruning techniques [26, 15], the image-level redundancy causes most of the attention values to be centered around a small value and the rest of the values are discrete and converge to 1 as Figure 3. Two cases are put here to show the attention sparsity. Compared to the uniform quantization which assigns only one bin to such a large number of values, the log2 method (only with 4-bit) has more resolution to cover this data range.  $\bullet$  Activation after GELU. The GELU activation function in FFN leads to a truncation effect that heavily concentrates the resulting values around zero, showcasing a clear one-sided stacking pattern (see Figure 3(b)).
Channel-Wise Outliers. A serious inter-channel variation in the addition of residual link. Figure 4 depicts the channel-wise outliers in the last residual link. To provide a comparison, we display the activations of ResNets in the last outputs of the 4th step. Notably, the channel-wise ranges in ViTs exhibit more significant fluctuations than in ResNets. This is because the data flow of nonlinear functions, such as Softmax and GELU, existing in the residual link of transformer structures, results in a greater disparity between the input and output of the residual blocks compared to CNN models. These results suggest that layer-wise quantization, with the same quantization parameters for all channels, would cause an unacceptable quantization error. Systematic outliers. Although outliers basically appear in every sequence, they are concentrated in  $< 6\%$  fixed feature dimensions of the fixed encoder blocks, as shown in Figure 4 (Left) (see Appendix 8). For instance, when testing 1024 images, outliers always occurred in the  $21^{st}$ ,  $89^{th}$ ,  $146^{th}$ , and  $189^{th}$  channel of the  $9^{th}$  block in Swin-T. Furthermore, setting these outlier features to zero in DeiT-T causes the top-1 accuracy drop by  $30\%$  on ImageNet classification. Since the outlier locations follow a fixed pattern, we can anticipate them in advance, obviating the hardware design with complex control logic to support online data-adaptive selection.
4 Activation-Aware Fully-Quantized ViTs
In detail, Section 4.1 presents the preliminary of quantization. In Sections 4.2-4.3, we propose two novel quantization methods to fit the distinct activation distribution inside ViTs, long-tail-aware quantization, and outlier-aware training for systematic channel-wise outliers quantization. Section 4.4 shows the SIMD-based 4-bit packed multiplier to support the 4-bit computation on mobiles practically.
4.1 Preliminary
Given the quantization bit-width is  $b$ , the quantizer  $Q(X|b)$  can be formulated as a function mapping a floating-point number  $X \in R$  to the nearest quantization bin. Among various quantizers, uniform [23] and log2 [7] are generally used. Apart from the special data distribution mentioned in Section 3, we apply symmetric layer-wise uniform quantization on weights and asymmetric channel-wise uniform quantization on activations. For QAT training, the forward and backpropagation of the quantization
function in a quantized network is formulated as
$$
\text {F o r w a r d}: Q - \operatorname {L i n e a r} (x) = \widehat {x} \cdot \widehat {w} = s _ {x} s _ {w} \left(Q _ {X} (x) + \frac {z}{s _ {x}}\right) \cdot Q _ {w} (w),
$$
$$
\text {B a c k w a r d}: \frac {\partial \zeta}{\partial x} = \frac {\partial \zeta}{\partial \widehat {x}} \frac {\partial \widehat {x}}{\partial x} = \left\{ \begin{array}{l l} \frac {\partial \zeta}{\partial x} & \text {i f} x \in [ l _ {x}, u _ {x} ] \\ 0 & \text {o t h e r w i s e} \end{array} , \right. \tag {1}
$$
$$
\frac {\partial \zeta}{\partial w} = \frac {\partial \zeta}{\partial x} \frac {\partial x}{\partial \widehat {w}} \frac {\partial \widehat {w}}{\partial w} = \left\{ \begin{array}{l l} \frac {\partial \zeta}{\partial x} \frac {\partial x}{\partial \widehat {w}} & \text {i f} w \in [ l _ {w}, u _ {w} ] \\ 0 & \text {o t h e r w i s e} \end{array} \right.,
$$
$\zeta$  is loss function, straight-through estimator [5] is used to obtain the derivation of the gradient in backpropagation for  $Q(\cdot)$ . To achieve a fully quantized ViT, we leverage the analysis in Section 3 and quantize all modules, including Conv, Linear, MatMul, LayerNorm, Softmax, GELU, etc.
4.2 Long-Tail-Aware Quantization
Two activation feature maps in dataflow exhibit a long-tail distribution: the attention map and the activation after GELU. GELU truncates activation values to be concentrated around 0, while the rest keep the original normal distribution. To allocate enough precision bits, we clip the activation value to [-1, 10] after GELU with uniform quantization. Moreover, we adapt GELU to support integer-only inference and regularize the quantization error as [15]. In this context, we focus on the attention map.
Log2 Quantization for Attention Map. Since the output range  $(0,1)$  of Softmax makes the log2 quantization [7] calibration-free and log2 can convert the MatMul to more hardware-efficient BitShift, we propose:
$$
\operatorname {A t t n} _ {Q} = Q (\operatorname {A t t n} | b) = \operatorname {c l i p} \left(\left\lfloor - \log_ {2} (\operatorname {A t t n}) \right], 0, 2 ^ {b} - 1\right), \tag {2}
$$
$$
\operatorname {A t t n} _ {Q} \cdot V _ {Q} = 2 ^ {- \operatorname {A t t n} _ {Q}} \cdot V _ {Q} = V _ {Q} > > \operatorname {A t t n} _ {Q} = \frac {1}{2 ^ {M}} \cdot \left(V _ {Q} <   <   (M - \operatorname {A t t n} _ {Q})\right), \tag {3}
$$
where  $M = 2^{b} - 1$ . Note that directly right shifting  $V_{Q}$  with the Attn  $Q$  may result in severe truncation error.  $M - \mathrm{Attn}_{Q}$  with scaling  $\frac{1}{2^M}$  can prevent the error through a left-shift operation.
Inference with Int-  $2^{n}$ -Softmax. Replacing the natural constant  $e$  inside the Softmax with 2 [8] as shown in Figure 3, we propose Int-  $2^{n}$ -Softmax with Log2 inference:
$$
\exp (s \cdot X _ {Q}) \approx s ^ {\prime} \cdot 2 ^ {X _ {Q}}, \quad \text {I n t - S o f t m a x} (s \cdot X _ {Q}) = M - \log_ {2} \left\lfloor \frac {\sum 2 ^ {X _ {Q}}}{2 ^ {X _ {Q}}} \right\rfloor . \tag {4}
$$
BitShift is implemented for the  $2^{XQ}$  and also for the integer log2 function. Experiments show that the replacement does not cause an accuracy drop under the same training receipt [42] (see Appendix 8).
We illustrate the difference between normal Int-Softmax [25] and our method in Figure 4. On the left, we present the common Int-Softmax with log2 quantization. On the right, our Int- $2^n$ -Softmax replaces the floating-point exponential calculation with BitShift. Notably, i-exp is a second-order polynomial that still involves the multiplication and addition in the floating-point domain, while Int- $2^n$ -Softmax employs a 4-bit representation on attention maps and replaces multiplication with BitShift operations, thereby enabling the inter-only dataflow, and reducing both computation and memory footprint.
4.3 Outlier-Aware Training for Systematic Channel-Wise Outliers Quantization
The addition of residual links contains many systematic and channel-wise outliers. To mitigate this, we propose the outlier-aware training approach that predicts the precise channel indices of the addition of residual links and regulated scales of the outliers with the power-of-two ratio (PTR). Thanks to the systematic outliers, outlier-aware training can obviate the need for complex control logic during inference to support data-adaptive selection.
Power-of-Two Ratio for Residual Link Quantization. Given the input activation  $X \in B \times L \times C$  and the PTR  $r \in N^C$ , then the quantized activation  $X_Q$  can be formulated as:
$$
X _ {Q} = Q (X \mid b) = \operatorname {c l i p} \left(\left\lfloor \frac {X}{2 ^ {r} s} \right\rfloor + z, 0, 2 ^ {b} - 1\right), \quad s = \frac {\max  (X) - \min  (X)}{2 ^ {R} \left(2 ^ {b} - 1\right)}, \quad z = \operatorname {c l i p} \left(\left\lfloor - \frac {\min  (X)}{\max  (X)} \right\rfloor , 0, 2 ^ {b} - 1\right). \tag {5}
$$