SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1: Outlier-Aware Training
Given the full-precision ViT Model, the test subdataset of target Dataset  $D$ , the number of encoder blocks  $L$ , Epochs for searching,  $\mathrm{Epoch}_f$  for fine-tuning, and quantized low-bit  $b$ ; // Step1: Initialize the PTC  $r$  with the outlier estimated from the full-precision Model.  
foreach  $l \in [0,1,\dots,L-1]$  do  
 $i_o, r_o = \text{Check\_Outliers}_{3\sigma}(\text{Model}_l, D)$ ;  
Quantize Model by Eq. (5) with  $i_o, r_o, b$  into the QModel;  
end  
// Step2: Search for the channel index of outliers and determine the PTR  $r$  by the 12 regularization.  
 $r = r_o, i = i_o$ ;  
foreach eps  $\in [0,1,\dots, \text{Epoch}_s - 1]$  do  
// These two operations are gradient-free.  
 $i, r = \text{Check\_Outliers}_{3\sigma}(\text{Model}_l)$ ;  
 $r_i = \text{argmin} \left| |X_i - \left[ \frac{X_i}{2^{r_i s}} \right] \cdot 2^{r_i s} \right|_2$ ;  
// Following Eq. (2), Eq. (4) and Eq. (5).  
taskloss, quantizationloss = Quantize(QModel, b);  
Backward(taskloss, quantizationloss);  
end  
// Step3: Finetune the  $b$ -bit quantizaton.  
Fix  $r$  and  $i$  for outliers and quantize QModel with fine-tune Epochf;  
Finalize the quantized ViT Model.
Hyperparameter  $R = \max(r) = 4$  satisfies different inter-channel variations across models. We predict each outlier's channel index  $i$  and PTR  $r$  by the following outlier-aware training algorithm.
Outlier-Aware Training. This algorithm is divided into three steps:  $\bullet$  Initialize the PTR by detecting the outlier of the full-precision model with  $3\sigma$  method [12].  $\bullet$  Search for the channel index  $i$  and the PTR  $r$  of each outlier with the  $l_{2}$  minimization.  $\bullet$  Fix the index  $i$  and  $r$  obtained in step 2 and fine-tune the model (see Algorithm 1).
Inference with Int-LayerNorm. During inference, with the gathered  $i$  and  $r$  of systematic outliers, we extract the layer-wise parameters  $s$  and  $z$ , and compute  $\mu$ ,  $\sigma$  in the integer domain. Meanwhile, thanks to the BitShift, PTR  $r$  can be calculated with the quantized Layernorm efficiently (also refer to [25] and see Appendix 8).
4.4 SIMD-Based 4-Bit Packed Multiplier
In the 8-bit multiplier of the SIMD CPU, INT4 data is represented as INT8 with the high 4 bits set to 0, which means there is no benefit for the efficiency improvement of INT4 computation. We develop our SIMD-based 4-bit packed multiplier (Figure 6): 1) Dot-multiplication. In the SIMD kernel, we concatenate two weights  $W_{i,j}$  and  $W_{i+1,j}$  from adjacent rows and multiply them with their shared activation value. The output is an INT16 data type, with the first 8 bits representing the result of the multiplication with  $W_{i,j}$ , and the last 8 bits corresponding to  $W_{i+1,j}$ . Following the SIMD memory mechanism where 16-bit of memory footprint will be freed up for the result of each dot-multiplication, we can avoid potential overflow issues. 2) Addition. Combined with the Bitshift operator, expand the 16-bit output in Step 1 into 32-bit, where the  $1^{st}$  8 bits are 0, the  $2^{nd}$  8 bits containing  $Output_{i,j}$ , the  $3^{rd}$  being 0, and 4th containing  $Output_{i+1,j}$ . Next, we perform a row-by-row summation. Since 32-bit of memory footprint will be supplied for each addition, this process can handle up to  $2^{8}$  times additions without overflow, which is sufficient for multi-head attention (head-dimension = 32/64). Finally, we split the output into two INT16 values and quantize them back to INT4 in value-level, which can be fused into the GeMM kernel.
Our method is different from per-channel quantization for outliers. We perform layer-wise quantization on the activation matrix; all the channels share the same quantization parameters, i.e., scaling factors and zero-point. However, the predicted power-of-two coefficients will refine the outlier channels' scaling factors. In the practical implementation, those power-of-two coefficients will be equivalently mathematically transformed over the corresponding weights, as shown in Figure 5.
Figure 6: The paradigm of SIMD-based INT4 packed multiplier.
Our framework is compatible with per-channel quantization. However, common inference engines usually provide support only for layerwise quantization on activation and per-channel quantization on weights, such as the Gemm and Convolution operator configuration. For example, ArmComputeLibrary [41] only supports channel-wise quantization configuration for weight matrix instead of the input activation.
Figure 5: Matrix Multiplication Implementation for Power-of-Two Ratio.
Our experiments include activation-aware quantization and hardware implementation for multiple ViTs. To the best of our knowledge, there is no full quantization-aware training of ViTs at this point, so we reproduce and implement the Q-ViT [29] and LSQ [19] methods by ourselves. Based on the proposed hardware design logic of the 4-bit multiplier, our team is committed to enhancing the support for lower-bit precision, such as 3-bit and 2-bit, which will be a major area of our future development. Currently, we mainly present our model performance on 8-bit and 4-bit precision.
Training Setup for PackQViT Framework. The baseline models with 32-bit floating-point precision are from the TorchVision library [20]. Our experiments are conducted on the ImageNet-1K [13] with two popular vision transformer implementations, including DeiT [43] and Swin [33]. Apart from special notes, we apply symmetric layer-wise quantization on weights and asymmetric channel-wise quantization on activations. The hyperparameter  $R$  in Power-of-Two Coefficients is set to 4. The training settings follow Q-ViT [29] with its DGD distillation, and the training process is executed on 4 NVIDIA V100 GPUs. Because we execute the outlier-aware training in a fine-tuned way, i.e., 70 epochs for step 2, 30 epochs for step 1, the training hours of the entire pipeline can be  $37.5\% \sim 54\%$  of train-from-scratch of full precision models (see Appendix 8).
Hardware Platform. To validate the practical deployment efficiency of the proposed PackQViT, we choose the Realme GT Master Android smartphone as the mobile platform, which is equipped with a Snapdragon 870 SoC consisting of the onboard Octa-core Kryo 585 CPU. We control the testing process to use a single thread and let the  $3.2\mathrm{GHz}$  Cortex-A77 Main Core perform the computation to obtain the most stable acceleration ratio with the proposed framework. The latency has been reported via 100 iterations for each test. Our inference engine is modified based on the ARM Compute Library. For the operators that are not yet supported in DeiT and Swin (such as Int-LayerNorm, Integer GELU, Int- $2^n$ -Softmax), we implement them based on the design hierarchy of 'kernel' to 'operator' to 'runtime' function, using parallel optimization based on ARM processor's SIMD support.
5.2 Model Accuracy and Speed Performance
This paper employs several popular post-training quantization methods, including MinMax, EMA [23], Percentile [27], OMSE [11], Bit-Split [44], PTQ for ViT [34], and FQ-ViT [32]. Also, we compare with the SOTA quantization-aware training method, Q-ViT [29]. For the sake of fairness, we reproduced the results of Q-ViT with quantized LayerNorm and Softmax.
Table 1: Comparison of the Top-1 (%) accuracy with state-of-the-art methods on ImageNet dataset. We report bit width in the order of weight/activation/attention.
Method	#Bits	DeiT-T [43]	DeiT-S [43]	DeiT-B [43]	Swin-T [33]	Swin-S [33]
Full Precision	32/32/32	72.21	79.85	81.85	81.35	83.2
PTQ
MinMax	8/8/8	70.94	75.05	78.02	64.38	74.37
EMA	8/8/8	71.17	75.71	78.82	70.81	75.05
Percentile	8/8/8	71.47	76.57	78.37	78.78	78.12
OMSE	8/8/8	71.3	75.03	79.57	79.3	78.96
Bit-Split	8/8/8	-	77.06	79.42	-	-
PTQ for ViT	8/8/8	-	77.47	80.48	-	-
FQ-ViT	8/8/8	71.61	79.17	81.2	80.51	82.71
QAT
Q-ViT	8/8/8	73.6	80.2	82.4	81.8	83.6
PackQViT	8/8/8	74.6	80.8	82.9	82.4	84.1
PackQViT	8/8/4	74.5	80.8	82.9	82.3	84.1
LSQ	4/4/4	70.5	77.8	79	78.7	80.5
Q-ViT	4/4/4	72.1	79.1	81.1	81	82.4
PackQViT	4/4/4	72.7	79.6	81.5	81.5	82.8
Table 2: Comparison of Objection Detection using DETR-50 on COCO val2017.
Methods	#Bits	mAP	AP50	AP75
FP32	32	59.5	83.3	64.7
VT-PTQ	8	57.6	82.3	63.1
PackQViT	8	58.3	82.9	63.9
PackQViT	4	55.6	82.2	60
Table 3: Evaluating the components on DeiT-T.
Method	#Bits	Top-1	#Bits	Top-1
Full Precision	32-32	72.2	32-32	72.2
Baseline [29]	8-8	73.6	4-4	72.1
+ Log2-Atten (Sec. 4.2)	8-8	74.1	4-4	72.4
+ Outlier-aware Train (Sec. 4.3)	8-8	74.2	4-4	72.5
+ Int-24-Softmax (Sec. 4.2)	8-8	73.5	4-4	72.0
+ Int-LayerNorm (Sec. 4.3)	8-8	73.6	4-4	72.1
PackQViT	8-8	74.5	4-4	72.7
Image Classification on ImageNet. In the 8-bit scenario, SOTA methods for PTQ have suffered a significant accuracy drop, ranging from  $0.5\%$  to  $16.9\%$  compared to the full precision baseline. However, QAT has shown promise in enhancing task accuracy by  $0.5\%$  to  $2.4\%$  over the baseline, thanks to its ability to minimize quantization errors and eliminate model redundancy. This suggests that redundancy in the original model hinders it from converging to the optimum, as confirmed by the quantized DeiT-T model achieving  $1.4\%$  to  $2.4\%$  higher accuracy than the baseline. While the current SOTA QAT method, Q-ViT, has made significant strides in correcting information distribution and knowledge transfer within ViTs, it falls short in analyzing the internal data flow, leaving room for improvement in quantization accuracy. Additionally, Q-ViT still relies on floating-point computations for Softmax, LayerNorm, and GELU, which makes it challenging for efficient hardware deployment. In contrast, our proposed PackQViT leverages activation flow fitting to achieve an additional accuracy boost of  $0.6\%$  to  $1\%$  over Q-ViT. Moreover, when switching the data precision of the attention matrix, the accuracy only reduces by less than  $0.1\%$ , indicating that a 4-bit log2 data type is sufficient to cover the data representation.
In the 4-bit scenario, PackQViT can effectively prevent accuracy degradation by less than  $0.4\%$ . Compared to other QAT methods, PackQViT still achieves a better accuracy performance of  $0.4\%$  to  $2.8\%$ . Furthermore, our method can be practically implemented on mobile devices using the ARM 8-bit multiplier and our proposed SIMD-based 4-bit Packed Multiplier.
Object Detection on MSCOCO. We compare object detection with transformers (DETR) [9], an end-to-end detector via a transformer encoder-decoder. We perform our quantization on DETR-R50. We compare the large-scale COCO dataset [30] as shown in Table 2. We compare our method under the 4-bit precision. We report the detection performance of the 8-bit PTQ method, VT-PTQ [34].
For this more complex task, our method has a negligible detection performance degradation by  $1.2\%$  mAP, which is better than the VT-PTQ method by 0.7 mAP. For more aggressive quantization under 4-bit, the model has 3.9 mAP performance drops. For more complex tasks, quantization tends to lead to a degradation of task performance.
Latency Analysis. Based on Table 4, we can draw the following conclusions: 8-bit quantization can bring an overall acceleration ratio of  $2.5\mathrm{x}$  to  $3\mathrm{x}$  depending on the size of the model, as the high computational workload on mobile devices can benefit from the higher utilization efficiency of limited memory resources under quantization algorithms. Specifically, INT8 quantization can achieve approximately  $3.5\mathrm{x}$  acceleration compared to FP32 in GeMM, which is very beneficial for models like Transformers, where most mathematical operations are matrix operations ( $>80\%$
Table 4: Latency (ms) on mainstream processors on edge platforms (1-thread).
Model	Method	#Bits	Size(MB)	Android CPU (s)	RaspberryPi (s)	RISC-V (s)
DeiT-T	Full-precision	32	20.1	0.172	23.97	3.4
PackQViT	8	5.2	0.056	7.73	1.11
PackQViT	4	2.7	0.037	5.15	0.74
DeiT-S	Full-precision	32	88.2	0.363	59.75	7.2
PackQViT	8	22.2	0.11	17.5	2.2
PackQViT	4	11.4	0.073	11.97	1.46
DeiT-B	Full-precision	32	346.2	0.858	143.4	17
PackQViT	8	86.8	0.233	38.24	4.6
PackQViT	4	44.1	0.156	26.12	3.16
Swin-T	Full-precision	32	114.2	0.276	44.9	5.5
PackQViT	8	28.6	0.099	16.4	2.1
PackQViT	4	14.6	0.085	13.76	1.69
Swin-S	Full-precision	32	199.8	0.55	89.78	10.9
PackQViT	8	50.2	0.213	34.4	4.2
PackQViT	4	25.5	0.164	25.93	3.29
computation). Hence, ViT can obtain more speedup gain than CNN, whose memory movement, such as weight reshaping, image2column, and column2image under different data layouts in each convolution operation, can dilute the efficient quantization speedup on GeMM. The 4-bit compression and concatenation technique can further improve this advantage, achieving approximately  $1.75\mathrm{x}$  acceleration compared to INT8 multiplication. This is because, while the theoretical computational workload is halved, overhead is introduced due to internal shifts of concatenated weights and the recovery of stored results in INT8 format.
Our 4-bit packed multiplier can be executed on mainstream processors on edge platforms (e.g., mobile phones, Raspberry Pis, and RISC-V IoT processors), which face challenges in processing low-bit data since their SIMD instructions only support 8-bit or wider data granularity. Hence, we also test the speedup gain on other edge devices. Note that INT8 matrix multiplication within PackQViT utilizes the original byte-level quantized GeMM kernel. For the 8-bit configuration, models can be speedup by up to  $3.7\mathrm{x}$  and up to  $2.7\mathrm{x}$  for DeiT and Swin models, respectively. For the 4-bit configuration, models can be speedup by up to  $5.5\mathrm{x}$  and up to  $3.5\mathrm{x}$  for DeiT and Swin models through our proposed packed 4-bit multiplier.
Breakdown of Task Accuracy. We give quantitative results of the proposed log2 quantization, outlier-aware training, and integer activation functions in Table 3. We set Q-ViT as the quantization baseline, which can improve the accuracy by  $1.4\%$ , which illustrates the redundancy inside the full precision model and suffers a performance drop by  $0.1\%$  in 4-bit settings. Log2 Attention and outlier-aware Training improve the performance when used alone, and integer activation functions almost maintain the original precision with only  $0.1\%$  accuracy drops when setting  $n = 4$  in  $\mathrm{Int - }2^{n}$ -Softmax. When combining all the components to enable integer-only flow, the performance can be boosted considerably, e.g., log2 quantization improves the 8-bit baseline by  $0.5\%$  and outlier-aware training by  $0.6\%$ . In contrast, the combination with integer-only functions can reach  $0.9\%$ .
This paper proposes an activation-aware fully sub-8-bit QAT framework called PackQViT for ViT inference acceleration on the mobile. We first arrived at several critical observations regarding the data distribution in ViTs and pointed out two distinct characteristics in activations influencing the quantization strategy: a long-tailed distribution, and systematic channel-wise outliers. To address the long-tailed distribution, we utilized 4-bit log2 quantization and clipping. For the systematic channel-wise outlier, we designed outlier-aware training to predict outliers' indexes and scale and regularize them with PTR in advance. We also develop a SIMD-based 4-bit packed multiplier to support PackQViT and achieve end-to-end ViT acceleration on mobile phones. Experiments show that PackQViT achieves superior task accuracy compared to SOTA quantization studies with significant hardware efficiency. Notably, we primarily validate our QAT methods with 4-bit precision. However, the PackQViT also holds the potential for application in lower precision scenarios. We are currently developing an implementation of a lower-bit SIMD-based multiplier to achieve enhanced acceleration.
This work is supported in part by the National Science Foundation CCF-1937500.
[50] Zihan Zhao, Yuncong Liu, Lu Chen, Qi Liu, Rao Ma, and Kai Yu. An investigation on different underlying quantization schemes for pre-trained language models. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 359-371. Springer, 2020.
[51] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6881-6890, 2021.
[52] Luowei Zhou, Yingbo Zhou, Jason J Corso, Richard Socher, and Caiming Xiong. End-to-end dense video captioning with masked transformer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8739-8748, 2018.
[53] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations (ICLR), 2020.