SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Table 2, we perform a similar experiment on NLP using Bert [4]. We can observe the generalization of our results from ConvNets to Transformers, REx can find better accuracy per bits trade-offs as compared to four references including non-uniform quantization [41]. Bert is a pre-trained model with 86 million parameters which is now considered a medium-sized model. Both the full-precision and quantized models can fit on a single middle range GPU. However, recent state-of-the-art models, such as OPT [42], are so large that they need multiple gpus just to be loaded on memory. These models ought to be compressed for sustainable usage. In the following section, we generalize the performance of REx to extreme model sizes.
4.3.3 Application to Handling Outliers in LLMs
A known pitfall [43] for quantization on LLMs, comes from the presence of extreme outliers among their weight values. These outliers stretch out the weight values range and increase the scaling factor in Equation 1, which, in turn, causes smaller weights to be rounded abruptly to zero. Worse, as suggested in [43], this phenomenon seems to occur more as the model size increases and might appear as a major problem for future work in large DNN quantization. In order to overcome this challenge, we adapt REx to only quantize the outliers in a residue using binary values (W1/A16) while the remaining weights are quantized in int4 (W4/A16). As a result, the overhead from REx is limited to a binary expansion with over  $99.8\%$  sparsity. As listed in Table 3, our evaluation on common sense reasoning tasks demonstrates that REx provides a significant improvement over other
Table 2: GLUE task quantized in W4/A8. We consider the BERT transformer architecture [4] and provide the original performance from the article (original) of BERT on GLUE as well as our reproduced results (reproduced). REx is applied to the weights with 3 bits + 33% sparse expansion.
task	original	reproduced	uniform [5]	log [41]	SQuant [19]	SPIQ [26]	REx
CoLA	49.23	47.90	45.60	45.67	46.88	46.23	47.02
SST-2	91.97	92.32	91.81	91.53	91.09	91.01	91.88
MRPC	89.47/85.29	89.32/85.41	88.24/84.49	86.54/82.69	88.78/85.24	88.78/85.06	88.71/85.12
STS-B	83.95/83.70	84.01/83.87	83.89/83.85	84.01/83.81	83.80/83.65	83.49/83.47	83.92/83.85
QQP	88.40/84.31	90.77/84.65	89.56/83.65	90.30/84.04	90.34/84.32	90.30/84.21	90.50/84.35
MNLI	80.61/81.08	80.54/80.71	78.96/79.13	78.96/79.71	78.35/79.56	78.52/79.86	79.03/79.96
QNLI	87.46	91.47	89.36	89.52	90.08	89.64	90.08
RTE	61.73	61.82	60.96	60.46	60.21	60.21	61.20
WNLI	45.07	43.76	39.06	42.19	42.56	42.12	42.63
FP
Use REx	-
HellaSwag	52.43
OpenBookQA	27.20
ARC-E	61.91
ARC-C	32.94
Winogrande	65.04
PiQA	76.88
BoolQ	65.90
Average Score	54.61
Table 3: Evaluation on Common sense reasoning benchmarks for OPT-13B [42] LLM quantized in W4/A16. For each quantization operator DFQ [14], SQuant [19] and PowerQuant [44], we share performance with and without REx (noted with check marks). We also provide the original full-precision (FP) performance.
DFQ [14]
X	✓
49.25	50.14
25.80	25.40
59.93	61.91
30.2	32.42
64.56	64.72
75.84	76.17
54.71	65.54
51.47	53.76
SQuant [19]
X	✓
49.23	50.21
25.40	26.20
59.97	61.95
30.12	32.34
64.48	64.88
75.84	76.30
54.28	65.38
51.33	53.91
PowerQuant [44]
X	✓
51.29	50.98
25.80	27.80
60.82	60.52
31.57	32.94
64.88	65.04
75.90	76.93
70.43	69.45
54.38	54.81
Table 4: Upper bound  $U$  (see theorem B.1 and B.2) over the maximum error as compared to the corresponding empirical measurement  $U_{\mathrm{empirical}}$  of that error for a VGG 16 [45] trained on ImageNet. The closer the upper bound  $U$  to the value  $U_{\mathrm{empirical}}$  the better.
weights bit-width	expansion order K	sparsity	U	Uempirical
8	1	X	0.12	0.05
8	4	X	1.99 ×10-7	1.78 ×10-7
8	2	50%	0.06	0.05
8	4	50%	1.17 ×10-7	0.65 ×10-7
quantization operators at virtually no cost. Hence, REx appears as a ready-to-use solution to the outlier problem for quantization of LLMs.
4.4 Empirical Validation of the Theoretical Bounds
Having shown the interest of REx for quantizing various architectures for computer vision and NLP tasks, we now empirically confirm its mathematical guarantees. In Table 4, we validate the proposed upper bound  $U$  in Equation 4 on the maximum error on the predictions on a VGG-16 [45] trained on ImageNet. The tightness of the provided theoretical results can be estimated from the gap between our estimation and the empirical maximum error  $U_{\mathrm{empirical}}$  from quantization on the predictions, which is measured as the infinite norm between the full-precision and quantized logits. We observe that a naive 8-bits quantization (i.e. no expansion) leads to an upper bound  $U = 0.12$ , while we observe  $U_{\mathrm{empirical}} = 0.05$ . The norms of the logits is equal to 0.3423. Therefore, the proposed upper bound is relatively tight and significantly lower than the logits magnitude: in such a case, due to overconfidence, the error shouldn't affect the classification. The proposed upper bound is even tighter for larger values of  $K$ , and becomes lower and lower (for both the theoretical and corresponding empirical maximum errors) when introducing sparsity. This further demonstrates the good properties of the proposed expansion approximation in REx in addition to the relevance of its theoretical guarantees, which are critical in data-free quantization.
Table 5: We report the different trade-offs achieved with REx expanding over different proposed quantization operators in W4/A4 as compared to their performance in W8/A8, on a MobileNet V2.
method	W4/A4	W4+25%/A4	W4+50%/A4	W4+75%/A4	W6/A6	W8/A8
naive [5]	0.1	53.11	64.20	71.61	51.47	70.92
SQuant [19]	4.23	58.64	67.43	71.74	60.19	71.68
SPIQ [26]	5.81	59.37	68.82	71.79	63.24	71.79
AdaRound [46]	56.17	61.30	69.80	71.77	68.71	71.75
BrecQ [47]	66.57	70.94	71.28	71.76	70.45	71.76
4.5 Flexibility with respect to the Quantization Operator
Most recent approaches for data-free quantization focus on designing better quantization operators. Interestingly, as we already hinted on large language models, our approach is agnostic to the choice of the quantization operator and can thus be combined with these approaches without bells and whistles. In Table 5, we report the possible trade-offs achievable with REx combined with recent approaches focusing on the quantization operator on MobileNet V2. The different trade-offs are sorted in ascending order in terms of added overhead operations, e.g.  $\mathrm{W4}_{+25\%}$  leads to less operations than  $\mathrm{W4}_{+50\%}$ . First, when used with SQuant [19], REx achieves full-precision accuracy in W4/A4 with only 75% overhead, even outperforming W8/A8 quantization. SPIQ [26], can also be adapted with REx in order to achieve good accuracy using only 4 bits representation as it benefits from finer weight quantization. This explains the slightly higher accuracies than SQuant using 25% and 50% sparsity. Finally, with AdaRound [46] and BrecQ [47], two PTQ techniques, we observe similar results as expected. In particular, BrecQ which already achieves decent accuracy in W4/A4 with a 5.23 points accuracy drop gets closer to the original accuracy (0.86 point accuracy drop) using a quarter of the expansion. Those results demonstrate REx versatility.
In this work, we proposed a novel data-free quantization method, dubbed REx, that consists in an expansion of residual quantization errors. Furthermore, we proposed a group-sparse version of the residual expansion that allows to find the best accuracy v.s. speed trade-offs. We demonstrated the exponential convergence of the quantized weights obtained through the different expansion methods towards the full-precision model. These theoretical guarantees are crucial in the context of data-free quantization where we cannot empirically measure the accuracy degradation in an industrial application context. As such, REx allows to find superior trade-offs for several bit-width representations, which allows better flexibility and adaptability to specific hardwares. In particular, we showed the added value of REx through extensive empirical validation. It appears that REx significantly outperforms recent data-free quantization methods on a wide range of ConvNet architectures applied to image classification, object detection, semantic segmentation as well as transformers architectures on GLUE text classification. Furthermore, we showed that REx allows to efficiently handle outliers within the weight distributions, a well-known pitfall when attempting to quantize LLMs, using a single binary residual to account for outliers. Lastly, the ideas presented in this paper are orthogonal to most recent approaches focusing on improving the quantization operator, and hence can straightforwardly be combined with those approaches.
5.1 Limitations:
The residual expansion method introduced in this paper does not adapt to the inter-layer importance and runtime cost discrepancies. An interesting future work would thus consist in applying more expansion orders on the most important layers w.r.t. the model accuracy, as well as using fewer orders for the most computationally expensive layers.
Acknowledgments: This work has been supported by the French National Association for Research and Technology (ANRT), the company Datakalab (CIFRE convention C20/1396) and by the French National Agency (ANR) (FacIL, project ANR-17-CE33-0002). This work was granted access to the HPC resources of IDRIS under the allocation 2022-AD011013384 made by GENCI.
[49] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In AAAI, volume 34, pages 7432-7439, 2020.
[50] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019.
[51] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021.
[52] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.
[53] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018.
[54] Mark Sandler, Andrew Howard, et al. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, pages 4510-4520, 2018.
[55] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, pages 801-818, 2018.
[56] Erica Klarreich. Multiplication hits the speed limit. Communications of the ACM, 63(1):11-13, 2019.