REx: Data-Free Residual Quantization Error Expansion
Authors: Edouard YVINEC
Arnaud Dapogny
Matthieu Cord
Kevin Bailly
Venue: NeurIPS 2023 poster (2023)
Keywords: deep learning
quantization
compression
acceleration
data-free

Abstract: Deep neural networks (DNNs) are ubiquitous in computer vision and natural language processing, but suffer from high inference cost. This problem can be addressed by quantization, which consists in converting floating point operations into a lower bit-width format. With the growing concerns on privacy rights, we focus our efforts on data-free methods. However, such techniques suffer from their lack of adaptability to the target devices, as a hardware typically only supports specific bit widths. Thus, to adapt to a variety of devices, a quantization method shall be flexible enough to find good accuracy v.s. speed trade-offs for every bit width and target device. To achieve this, we propose REx, a quantization method that leverages residual error expansion, along with group sparsity. 
We show experimentally that REx enables better trade-offs (in terms of accuracy given any target bit-width) on both convnets and transformers for computer vision, as well as NLP models. In particular, when applied to large language models, we show that REx elegantly solves the outlier problem that hinders state-of-the-art quantization methods.
In addition, REx is backed off by strong theoretical guarantees on the preservation of the predictive function of the original model. Lastly, we show that REx is agnostic to the quantization operator and can be used in combination with previous quantization work.

REx: Data-Free Residual Quantization Error Expansion
Edouard Yvinec Arnaud Dapogny², Matthieu Cord Kevin Bailly Sorbonne Université CNRS, ISIR, f-75005, 4 Place Jussieu 75005 Paris, France Datakalab², 114 boulevard Malesherbes, 75017 Paris, France eyčdatakale caatttaaalb.cco ab.
Abstract
Deep neural networks (DNNs) are ubiquitous in computer vision and natural language processing, but suffer from high inference cost. This problem can be addressed by quantization, which consists converting floating point operations into a lower bit-width format. With the growing concerns on privacy rights, we focus our efforts on data-free methods. However, such techniques suffer from their lack of adaptability to the target devices, specific bit widths. Thus, to adapt variety of devices, quantization method shall be flexible enough to find good accuracy v.s. speed trade-offs for every bit width and target device. To achieve this, we propose REx, a quantization method that leverages residual error expansion, along with group sparsity. We show experimentally that REx enables better trade-offs (in terms of accuracy given any target bit-width) on both convnets and transformers for computer vision, well NLP models. particular, when applied to large language models, we show that REx elegantly solves the outlier problem that hinders sate-of-the-ane quantization methods. addition, REx is backed off by strong theoretical guarantees on the preservation of the predictive function of the original model, Lastly, show that REx is agnostic to the quantization operator and can be used in combination with previous quantization work.
hardware typically only supports
1 Introduction
Deep neural networks (DNNs) achieve outstanding performance several challenging computer vision tasks such as image classification [1], object detection [2] and semantic segmentation [3] as well as natural language processing benchmarks such as text classification [4]. However, their accuracy comes at a high computational inference cost which limits their deployment, moreso on edge devices when real-time treatment as well energy consumption tackled via DNN quantization, í.e. by reducing the bit-width representation of the computations from floating point operations (FP) e.g. int8 (8-bits integer representation), int4, int3 even lower-bit representation such as ternary (where weights values are either -1, 0 or +1) quantization. Because DNN inference principally relies matrix multiplication, such quantization dramatically diminishes the number of bit-wise operations (as defined by (5]), thus limiting the DNN latency and energy consumption. However, DNN quantization usually comes the expense of the network accuracy. As a consequence, DNN quantization is an active field of research |6, 7, 8, 9, 10, 11, 12, 13] that aims at limiting this accuracy drop while reducing the number of bit-wise operations.
concern. This problem can be
All the aforementioned methods are data-driven as they either involve training a network from scratch or fine-tune already trained and quantized one. However, while such approaches usually allow lower quantization errors using low bit-wise representations, due to the growing concerns on privacy rights and data privacy, there ever-increasing number of real-case scenarios (e.g. health and
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

military services) where data may not be available for quantization purpose. Furthermore, the bloom of large langage models (LLMs) that are very expensive train further motivates the use post-hoc data-free quantization methods. Motivated by these observations, several data-free quantization algorithms were published in recent years [14, 15, 16, 17, 18, 19), which focus on the quantization operator, Le. the transformation which maps the floating point weights to their low-bit, fixed point, values. However, these approaches still struggle to offer an interesting alternative to data-driven techniques in terms of accuracy.
Furthermore, when considering a specific target device for deployment, traditional quantization methods, usually focusing on the quantization operator, offer limited options: given a supported bit width (given by the device, as most hardware usually support only a few representation formats [20]) they either achieve satisfactory accuracy or not. To address this concern, we wish to design flexible quantization method, i.e. one that can provide several accuracy vs. speed trade-off points for each bit width. Drawing inspiration from wavelets-basee methods for image compression [21, 22], we tackle this limitation by considering the successive residual quantization errors between the quantized and original model. Increasing the number of residuals in the expansion (i.e. the expansion order) increases the fidelity to the original, non-quantized model at the expense of additional computations. In addition, we propose a group-sparse expansion which allows us to maintain the accuracy using significantly less bit operations. Hence, given a target device, our approach allows finding the best accuracy speed trade-off. Our contributions are thus four-fold:
REx, data-free quantization method that is both efficient and flexible. REx lever- ages residual quantization, along with -rousssparsity, to enable finding suitable tradc-offs depending on a target bit-width.
Theoretical guarantees both the exponential convergence of the quantized model towards the full-precision model and the maximum error with respect to the predictive function. This is of paramount importance in data-free context, where cannot easily measure the accuracy degradation.
Extensive empirical validation we show through thorough empirical validation that, as standalone method, REx significantly outperforms every state-of-ther offthh-art data-free quan- tization technique, allowing to find better trade-offs on a variety of benchmarks involving ConvNet for classification, object detection or semantic segmentation as well transformers on GLUE text classification.
A ready-to-use solution that uses single binary residual to handle outliers within the weight distributions, which is well-known pitfall when attempting quantize L.MSs.
2 Related Work
2.1
Quantization
In this section, we review existing methods for DNN quantization, with an emphasis on approaches geared towards run-time acceleration. The vast majority of DNN quantization techniques rely on data usage (Quantization Aware Training). Furthermore, methods such as [7, 8, 9, 10, 11, 23, 24] rely on variants of straight through estimation alleviate the rounding operation gradients. Among these methods, [25] bears the most resemblance with the proposed REx method. It minimizes the residual error during training, using weight decay over the residue. The similarity with REx comes from the use of second order expansion of the quantization errors. However, it discards the quantization error after training while we propose to keep the extra operations in order to ensure a high fidelity the provided pre-trained model.
2.2 Data-Free Quantization
Nagel et al. [14] discuss the necessity have data available so as to successfully design quantization pipeline. They proposed method that consists in balancing the weight ranges over the different layers of a model, using scale invariance properties that are specific to piece-wise affine (e.g. ReLU) activation functions, and relying on a traditional, naive quantization operator [5]. In the current state of data-free quantization research, we see two major trends: methods that focus on the rounding operator itself [19, 26] and methods that generate synthetic data [27, 12, 13]. With REx, aim

---

R |==
R 2=
R 3-
R Am4)
R k-
A
R 3
A
0.3
0
[==1)
2=22)
R [x=3)
((==
R Rk-a
R
[x]
F [x]]
residual expansion
(b) group-sparee expansion
Figure I: Illustration the proposed method for a two-layers neural network. (a) residual expansion at order the intensity of the colormap indicates the magnitude of the residual error. (b) group-sparse expansion for orders
k 2 1 (7 = 50% sparsity).
at enabling hardware flexibility for these methods by allowing to find better trade-ofTs in terms of accuracy and compression rate given a fixed bit-width.
2.3 Flexibility in Quantization
In practice, the existing data-free quantization methods only offer a single possible quantized model given a supported bit-width. Nevertheless, most hardwares do support wide range of bit-width. For instance, Turing [28] and Untether [29] architectures support int4 and int8 quantization while the Nvidia A100 [20] supports int8, int4 and binary (intl) quantization. Conversely, REx circumvents this limitation by offering several trade-off' given a bit-width representation.
Methodology
Let's consider a trained network with L layers and trained weights Wi. Given target integer representation in bits, e.g. int8 or int4, we consider quantization operator Q. Formally, Q maps [min(W?]; }; max [W]]]= C I to the quantized interval [-20-1, 2 1]nz. The most straightforward way to do so is to apply scaling sw, and round [-] the scaled tensor, i.e.:
W
Q(Wi) =
(I)
.8w,
With Sw, the quantization scale for Wt computed as in [5], without loss of generality. Following the standard formulation [30], a quantization operator Q, comes with de-quantization operator Q-1, For the simple quantization operator Q in Equation (1), natural choice is Q (Q(Wi) sw; (W:). Note that, despite the notation, Q is not true inverse, as by definition of the quantized space, there some loss of information. This loss, called the quantization error, is defined as: Wi Q (Q(Wi)) In data-free quantization, we want to minimize this error in order to achieve the highest possible fidelity to the original model. In the following section, we describe how we can efficiently reduce the quantization error for fixed target bit-width b.
3.1 Residual Expansion
We propose to quantize the residual errors introduced by the quantization process. Although the proposed method can be applied any tensor, let's consider weight tensor W. the full-precision space (R), its first approximation is R 1-'-22111000 (Q(W)). To reduce the quantization error, we define R2 the quantized residual error
R2 Q- (Q (W-R+))
(2)
Consequently, during the quantized inference, we compute R R WX which provides finer approximation than the simple evaluation R X. The process can be generalized to any

---

expansion order K, leading to the following:
K-1 Σ
R = Q-
W
R*
The resulting expanded layer is illustrated in Figure 1 (a) in the case K = 4. Intuitively, expansion (R' ..., R*) provides the approximation Σx= R of W and this approximation converges exponen- tially fast to the original full full-precision weights with respect to K. As the support of the quantization error space smaller than one quantization step, the error decreases by factor larger than 2 with each expansion term (more details in Appendix A). Furthermore, the quantization error decreases, it is expected that the prediction of the quantized model would achieve a closer match to the original one. This is especially important in the context of data-free quantization not only do we not have the option to perform fine-tuning to recover aceuracy, but also we cannot evaluate the degradation set. Nonetheless, we can estimate an upper bound on the maximum error Emax introduced by quantization on the predictions as
of the model on a calibration/validatioo
K-1
Emax S U =
II
Σ
SR 2
+
where sn is the scaling factor from equation I applied to each residue. The detailed derivations are provided in Appendix B. This implies that, practice and regardless on the quantization operator, network can be quantized with high fidelity with only few expansion orders to fit given bit-width. Furthermore, this process can also be applied to the activations.
3.2 Input Expansion
Quantizing the weights of a DNN with the aforementioned method already leads to significant memory footprint reduction. However, significantly decrease the inference runtime. the inputs and activations of each layer also have to be quantized so that each the computations can be processed in the quantized bit-width. For that matter, let the imput tensor of layer 1. We define the expansion of I in quantized residual errors similarly to the weights expansion. Using the generic quantization operator Q, we get (()) = Q(T) and define the th order of quantization as
(
K-1 (1-Ee'(((())
7(k) T(k)= 1 k=1
In order to efficiently exploit the resulting expansions, of the weights and inputs. In other words, if we note kı the expansion order of a residue from the inputs and ko a residue from the weights, then only perform the computations for orders such that kn + 12 < (the rest being negligible in comparison). As a result, the quantized layer / computes:
propose bound the accumulated order
k Σ
I:I
((a) ((2)
(6)
where & is the base operation of the layer, e.g. a convolution for convolutional layer or matrix multiplication for a culle-connected layer. Similarly to the weights, the error between the full- precision inputs and the proposed expansion of the inputs converges exponentially fast to 0 with respect to the order of the expansion. However, with formulations from equations (3) and (5), the overhead computations induced by the expansion is non-negligible. In the following section, we provide a solution to tackle this issue.
3.3 Sparse Expansion
The residual expansion defined in equation 3 is based upon the assumption that the quantization error is equally important for every neuron. Thus, we propose to reduce the overhead cost by only expanding the most important neurons. However, in data-free compression we do not have access to activations or gradients: hence, we measure the relative importance of a neuron in layer by the norm of its weights [31]. The resulting expanded layer is illustrated in Figure I (b). Given target

---

budget (in %) of overhead computations, we only expand the K-1 % most important neurons. The sparse residue is defined as:
(R( = (H) (kk)
(7)
where 1 indicates the indices of the most important neurons. Similarly to what precedes, each expansion order derived sequentially from previous orders and we can bound the quantization error for the sparse expansion (see Appendix A). The method for computing the weights of the expanded model summarized in Algorithm
Algorithm 1 Expansion Algorithm
Require: trained DNN with L layers, hyper-parameters K and y, operator Q initialize 7 and initialize f(k) clone of f with K per-layer kernels
for / & (1...., L.} do
W base kernel of layer in f
Wase t accumulated quantization error
for k E [I....,K] do
R + Q(W - Wace) g1
equation
set hth kernel of layer of f(K) with R()
W < Wax + (R
end for end for
Also note that in the sparse expansion, allow higher expansion orders to re-consider neurons that were previously considered unimportant. Consequently, on top of improving the exponential convergence well as lowering the upper bound on the maximum error with respect to the overhead computations, this method systematically outperforms the standard residual expansion in practice. be set
Proof of this result can be found in Appendix C. The budget of overhead computations as not to introduce computational overhead, depending on the bit-width b. For example, let's consider device supporting only and bit (binary) quantization. If we want to achieve the same latency as 8 bit quantization using only 1bit quantization we will have a budget lower than 700% overhead w.r.t. naive 1 bit quantization. Consequently, for full expansions, we get 7 5 budget is then split across layers using a simple linear repartition. This strategy gives more emphasis the layers closest to the prediction head which also correspond to the largest layers, and empirically provides the best results [32]. As result, given a number bit operations (BOPS), the expanded model can better fit the inference device while preserving the full- cull-precision accuracy. Furthermore, all the added computations are performed in parallel which reduces their cost in practice. It allows better trade-off's in terms of accuracy and quantization compression rate, as will be shown in the upcoming experiments.
700%. This
4 Quantization Experiments
In the following sections, we first go through the implementation requirements and efficient strategies fully leverage the proposed expansions. Second, perform comparison of each expansion methods in order to show the flexibility of REx with respect to the bit-width. Third, we compare REx other quantization schemes under the constraint of equal bit operations. Finally, we validate for each expansion their respective upper bound on the maximum error with respect to the original predictions.
4.1 Implementation Details and Benchmarks
We ran our tests on 6 different backbones, including ConvNets and transformers and 5 tasks from both computer vision and natural language processing. We used ImageNet [33], Pascal VOC 2012 [34], CityScapes dataset [35] and GLUE [36] and common sense reasoning benchmarks (details Appendix D). Unless stated otherwise, we apply symmetric, static, per-channel quantization as defined in [30] and perform batch normaaiziaii folding prior to any processing using the optimal
5

---

100
W21A8 sparss
W2/A8 REX
W4/A0 REx- sporse
accuracy of FP
K-88
Wi/A8 REx
K-4
W3/AB
WoVNS
0.20
0.30
0.40
0.50
0.70
of the bit operations with respect W8/A8
Figure 2: Accuracy vs. inference time, for EfficientNet BO. The higher (accuracy) and the further the left (inference cost) the better. The circles show the baseline results with W3/A8, W4/A8, WS/A8 and W6/A8 quantization. The dashed lines show the trade-off's performance of REx in W4/A8 and ternary quantization (W2/A8). Finally, the plain lines show REx (with sparsity at 10%) also in W4/A4 and temary quantization. The numbers in the symbols stands for the expansion order. REx, and a fortiori the sparse version, enables better trade-offs.
method from [37]. In order to leverage the existing efficient implementations of the convolutional layers and fully-connected layers in CUDA, we propose to implement the expanded layer using single kernel rather than K kernels. This is achieved by concatenating the kernels along the output dimension. Consequently, the challenge of efficiently splitting the computations to fully leverage the target device computational power is left to the inference engine. In practice, this results both better performance and less work in order to adapt the method to existing engines such as Open Vino [38] and TensorRT [39]. We detail the implementation and overhead of the addition of the residual computations in Appendix E. Furthermore, we evaluate the latency overhead of REx in Appendix F. In the following section, we demonstrate the ability of REx to find good accuracy vs speed tradc-offs.
4.2 Flexible Quantization
Figure 2 shows different trade-offs enabled by REx on different bit-widths for an FfficcinnNeet-00 on ImageNet. First, the baseline quantization with the baseline quantization operator from [5] (as depicted by the circles of different colors, one for each bit width) offers no trade-off possibility given specific bit-width and usually performs poorly below int8 quantization (e.g. barely reaching 20. 29. topl accuracy in W6/A8 quantization). REx, however, in the same setup, offers several trade-offs for each specific bit-width (e.g. int4 and ternary on Figure 2) and supporting hardware. Furthermore, the sparse expansion enables finding more potential trade-offs (by varying the budget and expansion order) for every bit-width. Those trade-offs are generally more interesting than comparable ones obtained using the baseline method, which empirically confirms the theoretical results (Appendix C). Furthermore, Figure shows that using higher order, sparse residues allows to find even better trade-offs, as, in this case, e.g. in W2/A8 we reach full-precision accuracy at order 10 with 10% sparse residues. This shows that the process converges fast with respect to the sparsity rates. All in all, these results demonstrate the flexibility of REx to find good accuracy v.s. speed trade-offs, given a budget total bit operations (BOPs) to fit. the following section, we evaluate the ability of REx to outperform existing quantization methods in terms of equal bops.
4.3 Main Results
4.3.1
Experiments on Computer Vision Models
In order to highlight the benefits of residual quantization errors expansions as stand alone improve- ment upon existing methods with equal BOPs, we compare REx using the naive quantization operator from [5] on a variety of reference benchmarks. First, in Table 1, we report the performance on three different computer vision networks between state-of of tatee--f-theaaae the-art methods in W6/A6 quantization and REx using a sparse expansion at order K using 50% of a 4 bit representation in order to get similar total number of bit operations (150% of 4 bits 2 6 bits). For all networks, REx significantly outperforms recent state-of-the-ar data-free quantization methods at equal BOPs. Furthermore, we confirm these results object detection and image segmentation shown in Figure 3. We can

---

Table 1: Comparison equal BOPs with existing methods in W6/A6 and REx with W4/A6 +50% of one 4 bit residue.
DNN
method
year
bits
Accuracy
DFQ [14]
ICCV'19 19
W6/A6
71.36
ZeroQ [17]
CVPR'20
W6/A6
72.93
DSG [18]
CVPR'21
W6/A6
74.07
ResNet 50 (76.15)
GDFQ [40]
ECCV'20
W6/A6
74.59
SQuant [19]
ICLR'22
W6/A6
75.95
SPIQ [26]
WACV'23
W6/A6
75.98
REx
150% X W4/A6
76.01
DFQ [14]
ICCV'19
W6/A6
45.84
SQuant [19]
ICLR'22
W6/A6
61.87
MobNet v2 (71.80)
SPIQ [26]
WACV'23
W6/A6
63.24
REx
150%
W4/A6
64.20
DFQ [14]
ICCV
W6/A6
43.08
EffNet BO (77.10)
SPIQ [26]
ICLR'22
W6/A6
54.51
REx
150%
W4/A6
57.63
% mlou of FP model
100
K-3
K-3
K-5
% of of EP model
100
KF8
KF4
Kot
Wovkd RDt opons
WDVM
WeulS Hox 32a50
K=6
K-2
K-5
50
25
28
K=28
K--2
0.00
30.00
0.00
10.00
20.00
30.00
40.00
% bit operations with respect to FP model
% of bit operations with respect to FP model
Figure 3: (left) Mean intersection over union (mloU) of Deeplab V3+ with MobileNet V2 backhone CityScapes for semantic segmentation. (right) Mean average precision (mAP) of SSD with MobileNet V2 backhone on Pascal VOC for object detection. the performance of data-free quantization solution, DFQ [14] for comparison.
observe that REx can maintain the full precision accuracy while dividing by 3.23 the number of bit operations required to an inference.
4.3.2 Experiments on NLP
In Table perform similar experiment NLP using Bert [4]. We observe the generalization of our results from ConvNets to Transformers, REx can find better accuracy per bits trade-offs as compared four references including non-uniform quantization [41]. Bert is pre-trained model with 86 million parameters which now considered a medium sized model. Both the full-precision and quantized models can fit on single middle range GPU. However, recent state-of-the-art models, such as OPT [42], are so large that they need multiple gpus just to be loaded on memory. These models ought be compressed for sustainable usage. In the following section, generalize the performance of REx to extreme model sizes.
4.3.3 Application to Handling Outliers LLMs
A known pitfall [43] for quantization on LLMs, comes from the presence of extreme outliers among their weight values. These outliers stretch out the weight values range and increase the scaling factor in Equation 1, which, in turn, causes smaller weights to be rounded abruptly to zero. Worse, as suggested in [43], this phenomenon seems to occur more as the model size increases and might appear major problem for future work in large DNN quantization. In order to overcome this challenge, we adapt REx to only quantize the outliers
residue using binary values (W1/A16) while the remaining weights are quantized in int4 (W4/A16). As result, the overhead from REx limited to a binary expansion with over 99.8% sparsity. As listed in Table 3, our evaluation on common sense reasoning tasks demonstrates that REx provides significant improvement over other

---

Table 2: GLUE task quantized in W4/A8. We consider the BERT transformer architecture [4] and provide the original performance from the article (original) of BERT on GLUE well reproduced results (reproduced).. REx is applied the weights with bits + 33% sparse expansion.
task
original
reproduced
uniform [5]
log [41]
SQuant [19]
SPIQ [26]
REx
CoLA 49.23
47.90
45.60
45.67
46.88
46.23
47.02
SST-2
91.97
92.32
91.81
91.53
91.09
91.01
91.88
MRPC
89.47/85 29
89.32/85.4)
88 38.224/84499
86.54/82.69
88.78/85.24
88. 71/87/88800
088.71885.12
STS-B 83.95/83 70
84.01/83.87
83.89/83.88
84.01/83.8
83.80/83.66
83 43.44/83.47
83.92/83.85
QQP
88.40/84.31
90. 30.77/84.65
89. $9.56683.0 65
90.30/84.04
90.34/84.33
90.30/84.21
90.50/84.35
MNLI 80.61/81.00
80. 50.54/80.71
78.96/79.11
78.96/79.71
78.35/79.56
78. 78.52/79.86
79.03/79.96
QNLI 87.46
91.47
89.36
89.52
90.08
89.64
90.08
RTE 61.73
61.82
60.96
60.46
60.21
60.21
61.20
WNLI 45.07
43.76
39.06
42.19
42.56
42.12
42.63
Table Evaluation on Common sense reasoning benchmarks for OPT-13B [42] LLM quantized in W4/A16.
For each quantization operator DFQ [14], SQuant [19] and PowerQuant [44), we share performance with and without REx (noted with check marks). We also provide the original full prec-icciiooon (FP) performance.
FP
DFQ [14]
SQuant [19]
PowerQuant [44]
Use REx
X
/
HellaSwag
52.43
49.25
50.14
49.23
50.21
51.29
50.98
OpenBookQA
27.20
25.80
25.40
25.40
26.20
25.80
27.80
ARC-E
61.91
59.93
61.91
59.97
61.95
60.82
60.52
ARC-C
32.94
30.2
32.42
30.12
32.34
31.57
32.94
Winogrande
65.04
64.56
64.72
64.48
64.88
64.88
65.04
PiQA
76.88
75.84
76.17
75.84
76.30
75.90
76.93
BoolQ 65.90
54.71
65.54
54.28
65.38
70.43
69.45
Average Score
54.61
51.47
53.76
51.33
53.91
54.38
54.81
Table 4: Upper bound U (sce thcorem B.1 and B.2) over the maximum error compared to the corresponding empirical mcasurement Uempiniall that error for VGG 16 [45] trained on ImageNet. The closer the upper bound to the value Ummpunaa the better. weights bit-width expansion order K sparsity
U
U Vmmprrical
8
X
0.12
0.05
8
1.99 x10
1.78
x10
8
2
50%
0.06
0.05
50%
1.17 x
0.65 x10-7
quantization operators at virtually cost. Hence, REx appears ready-to-use solution to the outlier problem for quantization of LLMss
4.4 Empirical Validation of the Theoretical Bounds
Having shown the interest of REx for quantizing various architectures for computer vision and NLP tasks, we now empirically confirm its mathematical guarantees. In Table 4, we validate the proposed upper bound U in Equation 4 the maximum error on the predictions [45] trained on ImageNet. The tightness the provided theoretical results can be estimated from the gap between our estimation and the empirical maximum error Uempirical from quantization on the predictions, which is measured as the infinite norm between the full-precision and quantized logits. We observe that naïve 8-bits quantization (i.e. no expansion) leads to an upper bound U =00 0.12, while we observe Uemparical 0.05. The norms of the logits is equal to 0.3423. Therefore, the proposed upper bound is relatively tight and significantly lower than the logits magnitude: such case, due to overconfidence, the eтror shouldn' it affect the classification. The proposed upper bound even tighter for larger values of K, and becomes lower and lower (for both the theoretical and corresponding empirical maximum errors) when introducing sparsity. This further demonstrates the good properties of the proposed expansion approximation REx in addition to the relevance of its theoretical guarantees, which are critical in data-free quantization.
VGG-16

---

Table 5: We report the different trade-off's achieved with REx expanding different proposed quantization operators in W4/A4 as compared to their performance in W8/A8, on a MobileNet V2.
method
W4/A4
W4, 25%
W AA
W4, A5%
W6/A6
W8/A8
naive [5]
0.1
53.11
64.20
71.61
51.47
70.92
SQuant [19]
4.23
58.64
67 67.43
71.74
60.19
71.68
SPIQ [26]
5.81
59.37
68.82
71.79
63.24
71.79
AdaRound [46]
56.17
61.30
69.80
71.77
68.71
71.75
BrecQ [47]
66.57
70.94
71.28
71.76
70.45
71.76
4.5 Flexibility with respect to the Quantization Operator
Most recent approaches for data-free quantization focus on designing better quantization operators. Interestingly, already hinted on large language models, our approach is agnostic to the choice of the quantization operator and can thus be combined with these approaches without bells and whistles. In Table 5, we report the possible trade-offs achievable with REx combined with recent approaches focusing on the quantization operator on MobileNet V2. The different trade-off's are sorted in ascending order in terms of added overhead operations, e.g. W4, 25% leads to less operations than W4, so%. First, when used with SQuant [19], REx achieves full-precision accuracy in W4/A4 with only 75% overhead, even outperforming W8/A8 quantization. SPIQ [26], can also be adapted with REx in order to achieve good accuracy using only 4 bits representation as it benefits from finer weight quantization. This explains the slightly higher accuracies than SQuant using 25% and 50% sparsity. Finally, with AdaRound [46] and BrecQ [47], two PTQ techniques, we observe similar results as expected. In particular, BreeQ which already achieves decent accuracy in W4/A4 with 5.23 points accuracy drop gets closer to the original accuracy (0.86 point accuracy drop) using quarter of the expansion. Those results demonstrate REx versatility.
5 Conclusion
In this work, we proposed novel data-free quantization method, dubbed REx, that consists in an expansion of residual quantization errors. Furthermore, we proposed group-sparse version of the residual expansion that allows to find the best accuracy v.s. speed trade-offs. We demonstrated the exponential convergence of the quantized weights obtained through the different expansion methods towards the full-precision model. These theoretical guarantees are crucial in the context of data-free quantization where we cannot empirically measure the accuracy degradation in an industrial application context. As such, REx allows to find superior trade-off' for several bit- width representations, which allows better flexibility and adaptability to specific hardwares. In particular, we showed the added value of REx through extensive empirical validation. appears that REx significantly outperforms recent fata-free quantization methods on wide range of ConvNet architectures applied to image classification, object detection, semantic segmentation as well as transformers architectures GLUE text classification. Furthermore, showed that REx allows to efficiently handle outliers within the weight distributions, a well-known pitfall when attempting to quantize LLMs, using single binary residual to account for outliers. Lastly, the ideas presented in this paper are orthogonal to most recent approaches focusing on improving the quantization operator, and hence can straightforwardly be combined with those approaches.
5.1 Limitations:
The residual expansion method introduced in this paper does not adapt to the inter-layer importance and runtime cost discrepancies. An interesting future work would thus consist in applying more expansion orders on the most important layers w.r.t the model accuracy, as well as using fewer orders for the most computationally expensive layers.
Acknowledgments: This work has been supported by the french National Association for Research and Technology (ANRT), the company Datakalab (CIFRE convention C20/1396) and by the French National Agency (ANR) (Facll., project ANR-I7-CCE3--001 This work was granted access the HPC resources of IDRIS under the allocation 2022-ADO1 1013384 made by GENCL

---

References
[1] Kaiming He, Xiangyu Zhang, al. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[2] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scout Reed, Cheng- Yang Fu, and Alexander Berg. Ssd: Single shot multibox detector. ECCV, pages 21-37. Springer, 2016.
[3] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706. 05587, 2017.
[4] Jacob Devlin, Ming-Wes Meing-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.0480) 2018.
[5] Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv: /806. 08342, 2018.
[6] Matthicu Courbariaux, Itay Hubara, et al. Binarized neural networks: Training deep neural networks with weights and activations constrained I or-1. NeurIPS, 2016.
[7] Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. ICLR, 2018
[8] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam. and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer- arithmetic-oni inference. In CVPR, pages 2704 2014,2713,220 2018
[9] Jan Achterhold, Jan Mathias Koehler, Anke Schmeink, and Tim Genewein. Variational network quantiza- tion. ICLR, 2018
[10] Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling. Relaxed quantization for discretized neural networks. ICLR, 2018.
[11] Tao Sheng, Chen Feng, Shaojic Zhuo, Xiaopeng Zhang, Liang Shen, and Mickey Alcksic. A quantization- friendly separable convolution for mobilenets. In 2018 1st Workshop Energy Efficiint Machine Learning and Cognitive Computing for Embedded Applications (EMC2), pages 14-18. IEEE, 2018
[12] Kanghyun Choi, Hye Yoon Lee, Deokki Hong, Joonsang Yu, Noseong Park, Youngsok Kim, and Jinho Lee. It's all in the teacher: Zero-shot quantization brought closer to the teacher. CVPR, pages 831 1-8321, 2022.
[13] Yumshan Zhong, Mingbao Lin, Gongrui Nan, Jianzhuang Liu, Baochang Zhang, Yonghong Tian, and Rongrong Ji. Intraq: Learning synthetic images with intra-class heterogeneity for zero-shot network quantization. In CVPR, pages 12339-1222 12348, 2022.
[14] Markus Nagel, Mart van Baalen, et al. Data-free quantization through weight equalization and bias correction. In ICCV, pages 3925-1334, 2019.
[15] Eldad Meller, Alexander Finkelstein, Uri Almog, and Mark Grobman. Same, same but different: Re- covering neural network quantization error through weight factorization. In ICML, pages 4486-4495, 2019.
[16] Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting. In ICML, pages 7543-7552, 2019.
[17] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In CVPR, pages 13169-13178, 2020.
[18] Xiangguo Zhang, Haotong Qin, Yifu Ding, Ruihao Gong, Qinghua Yan, Renshuai Tao, Yuhang Li, Fengwei Yu, and Xianglong Liu. Diversifying sample generation for accurate data-free quantization. In CVPR, pages 15658 15661, 2021.
[19] Guo Cong et al. Squant: On-the-fly data-free quantization via diagonal hessian approximation. ICLR, 2022.
[20] Nvidia. Nvidia a100 tensor core gpu architecture. weh tech report (https::/imags..nn nvidia. com/ com/aam-dame damem-dam/eeen en z2. ittoluttios//eee center/ntenneee coanttrr/viii di a - whitepaper. pdf), 2021.
[21] Majid Rabbani. Jpeg2000: Image compression fundamentals, standards and practice. Journal of Electronic Imaging, 11(2):286, 2002.
[22] Stephane G Mallat. A theory for multiresolution signal decomposition: the wavelet representation. In Fundamental Papers in Wavelet Theory, pages 494 513. Princeton University Press, 2009.
[23] Karen UlIrich, Edward Meeds, and Max Welling. Soft weight-sharing for neural network compression. ICLR, 2017.
10

---

[24] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint wrxvv:1006.06160 2016.
[25] Sangyun Oh, Hyeonuk Sim, Sugil Lee, and Jongeun Lee. Automated log-scale quantization for low-cost deep neural networks. In CVPR. pages 742 142-751, 2021.
[26] Edouard Yvinec, Arnaud Dapogny, Matthicu Cord, and Kevin Bailly. Spiq: Data-free per-channel static input quantization. WACV, 2023.
[27] Yuhang Li, Feng Zhu, Ruihao Gong, Mingzhu Shen, Xin Dong, Fengwei Yu, Shaoqing Lu, and Shi Gu. Mixmix: All you need for data-free compression are feature and data mixing. In ICCV, pages 4410-4419, 2021.
[28] Boyuan Feng. Yuke Wang, Tong Geng, Ang Li, and Yufei Ding. Apnn-tc: Accelerating arbitrary precision neural networks on ampere gpu tensor cores. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-13, 2021.
[29] Cliff Robinson. Untether.ai boqueria 1458 risc-v core ai accelerator, Aug 2022.
[30] Amir Gholami, Schoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103. 13630, 2021.
[31] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint urXiv:1611.064400 2016.
[32] Edouard Yvinec, Arnaud Dapogny, Matthieu Cord, and Kevin Bailly, Red: Looking for redundancies for data-free structured compression of deep neural networks. NeurIPS, 2021.
[33] Deng, W. Dong. et al. ImageNet: A Large-Scale Scale Hierarchical Image Database. In CVPR, 2009.
[34] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.paccal- 2012.
network. wwtp://eeeeeeeooooo2222ooooooo
[35] Marius Cordts, Mohamed Omran, Scbastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bemt Schiele. The cityscapes dataset for semantic urban scene understanding.
In CVPR, pages 3213-3223, 2016.
[36] Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE:
A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyingg and Interpreting Neural Networks for NLP, pages
353- 353. Association for Computational Linguistics, 2018.
[37] Edouard Yvinee, Arnaud Dapogny, and Kevin Bailly. To fold or not fold: necessary and sufficient. condition on batch noneaanormalizationn layers folding. IJCAL, 2022.
[38] Intel. Intel@ distribution of openvino toolkit. Intel, 2022.
[39] Nvidia. Nvidia distribution of tensorrt toolkit. Nvidia, 2022.
[40] Shoukai Xu, Haokun Li, Bohan Zhuang, Jing Liu, Jiezhang Cao, Chuangrun Liang, and Mingkui Tan. Generative low- bitwiit data free quantization. In ECCV, pages 1-17. Springer. 2020.
[41] Daisuke Miyashita. Edward HI Lee. and Boris Murmann. Convolutional neural networks using logarithmic data representation. arXiv preprint arXiv: arXiv:1603.000 1e...00000..55 2016.
[42] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint 2rXiv:2205.01068 2022.
[43] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm, int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208. 07339, 2022.
[44] Edouard Yvinec, Arnaud Dapogny, Matthieu Cord, and Kevin Bailly, Powerquant: Automorphism search for non-uniform quantization. In ICLR, 2023.
[45] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. BMVC 2014, 2014.
[46] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up down? adaptive rounding for post-training quantization. In ICML., pages 7197-7206. PMLR, 2020.
[47] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. NewrIPS, 2021.
[48] Christopher Clark, Kenton Lee, Ming-
Ming-Wei Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 100. 2019.

---

[49]
Yonatan Bisk, Rowan Zellers. Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical common- sense in natural language. In AAAI, volume 34, pages 7432-7439, 2020.
[50] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint 1905, 07830, 2019.
[51] Keisuke Sakaguchi. Ronan Bras, Chandra Bhagavatula. and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106 2021.
[52] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 0833,05557, 2018.
[53] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? dataset for open book question answering. arXiv preprint arXiv: (809002789, 2018.
[54] Mark Sandler, Andrew Howard, ct al. Mobilenetv2: Inverted residuals and lincar bottlenecks. In CVPR, pages 4510-4520, 2018.
[55] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, pages 801-818, 2018.
[56] Erica Klarreich. Multiplication hits the speed limit. Communications of the ACM, 63( 11-33,,, 2019.
12


Reviews:
[{"review_id": "9hvq4aXbpe", "reviewer": "Reviewer_E57B", "summary": "This paper presents a low-bitrate post-hoc quantization method with three major techniques: residual expansion, input expansion, and sparse expansion. The techniques are theoretically guaranteed to improve the accuracy of the quantization, and experimental results show that they outperform competing approaches.", "questions": "What is the (machine) runtime on CPU and/or GPU?", "limitations": "This paper uses public dataset and benchmark, so it would not have potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* Provides theoretical bound for accuracy-bitrate trade-off\n* Evaluation that compares the theoretical bound and empirical benchmarks.", "weaknesses": "Lack of (machine) runtime evaluation. The project propose to use quantization for better inference runtime, yet none of the benchmark demonstrates the improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer again. We agree that the proposed study would benefit from an empirical analysis of the actual latency induced by REx. Hopefully the new results provided in the general comment addresses your concern.\n"}, {"review_id": "Ij78c8vZ2v", "reviewer": "Reviewer_d6mY", "summary": "In this work, authors proposed a novel quantization method called REx, that leverages residual error expansion. The idea is to repeatedly apply quantization to the quantized residual error, which provides an increasingly better approximation of the original floating-point tensor as the number of steps grows. To combat the computational overhead from additional quantization steps, authors combine the residual expansion with the group sparsity. Two hyper-parameters of the method (a number of steps K, and the sparsity rate $\\gamma$) are used to control the accuracy vs. speed trade-off. The method is agnostic to the quantization operator itself and can be combined with many existing methods.", "questions": "* The method is advertised as data-free, however, I don't see how it is the case in general. As far as I understood, it's data-free if the quantization operator Q itself is data-free (which is the case of DFQ, for instance). However, in most experiments, the underlying Q is not data-free (e.g., AdaRound, BRECQ, or even naive quantization with static range estimation). Could you elaborate?\n\n* Related to the previous comment, in the group sparsity, the importance of a neuron is defined by the norm of its weights (L151: assuming the data-free setting). Assuming we have access to activations and gradients, have you experimented with other importance metrics (e.g., gradient-based, FIT)?\n\n* I Would like to know a bit more details on the outlier quantization in LLMs. How are outliers detected/defined? Will the suggested approach work with outliers that are both positive and negative? In Table 3, what gives the most improvement - the method itself or the special treat of the outliers (would be nice to have an ablation study and/or have some additional data on e.g. quantization error for tensors with outliers with W1 vs. W4).\n\n* (Sorry if I missed it) Do you quantize embeddings, LayerNorm weights/biases, and LLM head, are there any specific assumptions (e.g., first and or last layer in 8-bit)?", "limitations": "Authors mentioned one limitation that the method does not adapt to the per-layer importance and runtime cost discrepancies.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Experiments on both CNNs (including some of the more difficult-to-quantize models like MobileNets and EfficientNets) and LLMs across various tasks + combination with several quantization operators from the prior work.\n* Theoretical derivations on the upper bound of the quantization error + comparison with the empirical values.\n* Some attention to efficient hardware implementation (Section 4.1), e.g. instead of using K separate CUDA kernels for the error expansion step, use a single one with concatenated output channels, etc.", "weaknesses": "* Two hyper-parameters (K and $\\gamma$) that have to be set. Even though $\\gamma$ can be set to match a certain BOP target (as it is done in the paper), the user still has to select a value for K.\n* I have some concerns related to the data-free nature of the method, please see and elaborate on the below question.\n* I would like to see error bars / standard deviations in some of the results, e.g. in Table 2. I wonder if the improvement is statistically significant (it is well known that some of the datasets from GLUE are quite small which might lead to high standard deviations in the results).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your interest in the method and for pointing out several elements that will improve the clarity of the paper, shall it be accepted for publication.\n\n**set the hyper-parameters** As stated by the reviewer REx uses two hyper-parameters $K$ and $\\gamma$. In our main results where we compare REx to other quantization methods, we show that using $K=2$ is almost systemically enough in order to achieve better performance. Aside from LLMs where we use a special setting (unstructured sparsity, removing 99\\% of the parameters in the expansion to only account for outliers in said expansion), we use s$50\\% tructured sparsity (easier to leverage practically speaking). These set of default hyper-parameters already outperform existing data-free methods. Furthermore, our theory provides that the higher $K$ (and lower $\\gamma$ to keep the global budget) the better the performance which indicates that further improvements can be expected when tuning these parameters.\n\n**error bars and significance** We agree on the fact that GLUE benchmark does have variance in the observed accuracies. To answer your concern, we provide here updated versions of our results (we also include other Tables in order to be exhaustive). These changes were added to the revised paper.\n\n| table | Uniform | SQuant | SPIQ | REx |\n| :---: | :---: | :---: | :---: | :---: |\n| updated table 2 (average scores) | 74,16 $\\pm$ 0.08 | 74,68 $\\pm$ 0.19 | 74,48 $\\pm$ 0.35 | 75,00 $\\pm$ 0.16 |\n\nThus, the provided results are significant. On a side note: we did not update the Table 1 and 3 as the quantization process is deterministic and we only have a single pre-trained version of the models.\n\n**is the method data-free:** As stated by the reviewer REx is data-free as long as the quantization operator also is. **Peut etre ajouter qu'on clarifie ça et ajouter 1 ligne ds le papier pour lui donner raison** Aside from Table 5 where we showcase the ability of the proposed method to also work outside of the boundaries of data-free quantization, all of our tests where fully data-free. Thus, we believe that it is appropriate to call our approach data-free as REx in itself do not use data and as such constitute a data-free step in the quantization process. \n\n**other criterion for sparsity** To answer your concern, we tested other critera on ResNet-50 such as _gradients_ and _weights $\\times$ gradients_. The results are provided below:\n\n| sparsity method | weights norm | gradients | weights $\\times$ gradients |\n| :---: | :---: | :---: | :---: |\n| accuracy (W$4_{+ 25%}$/A4) | 53.11 $\\pm$ 0.26 | 52.89 $\\pm$ 0.62 | 53.45 $\\pm$ 0.37 |\n\nThese methods provided small to no improvement, which we attribute to the fact that the sparsity goal here is not challenging enough to justify the use of more complex and costly criterion. We however acknowledge that this is an interesting future research direction.\n\n**how do we identify outliers** We identify outliers among weight values only (contrary to LLM.int8 [2] for example). The reason behind this choice lies in the fact that we do not want to slice tensors at inference and use fine-grained mixed precision, as this generally leads to significant overhead [3]. As we quantize in 4 bits, we define outliers as any value that is more than 6 standard deviations away from the average weight values (Note that this definition is equivalent to the one used in LLM.int8, where the authors define an outlier as any value larger than 6.0 - or $6 \\times 1.0$ as features are reduced by the layer norms). Furthermore, in quantization binary values are either $-1$ or $1$, which enables us to quantize both negative and positive outliers.\n\n**LLM specific quantization** To specifically answer your question, we fold the weights and biases from the layer norms, we quantize all the fully-connected layers (including the head) and apply the same quantize to all layers (nothing specific to the first and last layers). We do assume that the softmax and reduction steps are performed in higher precision, as it was done e.g. in I-Bert [1]. **We added those elements in the revised manuscript.**\n\n## references\n\n[1] Kim, Sehoon, et al. \"I-bert: Integer-only bert quantization.\" International conference on machine learning. PMLR, 2021.\n\n[2] Dettmers, Tim, et al. \"Llm. int8 (): 8-bit matrix multiplication for transformers at scale.\" arXiv preprint arXiv:2208.07339 (2022).\n\n[3] Xiao, Guangxuan, et al. \"Smoothquant: Accurate and efficient post-training quantization for large language models.\" International Conference on Machine Learning. PMLR, 2023."}, {"review_id": "Ekk72D0Zvx", "reviewer": "Reviewer_mdRP", "summary": "In this paper, the authors propose a fixable quantization method called REx, which utilizes residual error expansion to further improve quantization error. Additionally, they suggest applying group-sparsity to reduce the computational cost associated with residual error expansion.\n\nExisting DFQ methods have a limitation in terms of flexibility when it comes to quantizing based on the representation format supported by hardware, considering the trade-off between accuracy and speed-up. However, the REx method overcomes this limitation and provides a better trade-off. Furthermore, by calculating the theoretical upper bound of quantization error during the process of residual error expansion and demonstrating it through actual quantization error, the REx method shows that it can have less quantization error than or similar to existing methods at fewer BOPs.\n\nThe proposed REx method can be combined with recently developed quantization methods, leading to quantized models with improved accuracy.", "questions": "1. While Figure 2 represents the inference time in terms of BOPs, is there any analysis regarding the actual latency? I am curious about the analysis of the additional computational cost introduced by residual expansion and how much it can be reduced by group-sparsity expansion.\n2. In Figure 2 and Figure 3, when W2A8 is not supported by the hardware, additional packing and unpacking operations are required, which may incur overhead. It would be interesting to see how the graph's trends change when considering this overhead.\n3. In Table 5, there is a tendency for higher group sparsity to result in better performance. I am curious about the reasons behind this observation.", "limitations": "The paper provided valuable insights by demonstrating the ability of the proposed method to find a better trade-off between compression ratio and accuracy. However, it is challenging to agree with the paper's emphasis on whether it truly offers a superior approach in terms of the accuracy vs. speed trade-off. Since INT GEMM relies on the formats supported by hardware, it is necessary to analyze the speed gain achievable when applying REx + group-sparsity in practice.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* It demonstrates that the proposed REx method can find a better trade-off point between compression ratio and accuracy compared to existing methods through residual error expansion and group-sparsity expansion.\n* Theoretical and experimental evidence is provided for the upper bound of quantization error that the proposed REx method can achieve, showing a theoretical basis for achieving better quantization error at the same BOPs.\n* The proposed method can be applied in a composite manner to various quantization methods previously proposed, and experiments on different models demonstrate improvements in accuracy.", "weaknesses": "* It appears necessary to compare the results in Table 3 with other papers on LLM quantization, such as OPTQ, LLM.int8, and SmoothQuant.\n* Throughout the paper, different bit-widths for weights and activations were used, which may present challenges in utilizing the formats supported by hardware. However, there is a lack of discussion regarding this aspect.\n* There is a need for discussion on the criteria for dividing clusters and the benefits of higher sparsity in achieving better accuracy.\n* In order to discuss the trade-off between accuracy and speed, there should be a discussion on speed. The paper lacks discussion on this aspect. It would be beneficial to provide unit test results comparing the latency when applying group-sparsity expansion along with INT GEMM + SpMM compared to conventional INT GEMM, as well as the throughput on the GPU.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**comparison to other methods on LLMs** In the table below, we provide a comparison between REx (with the PowerQuant method) and other post-training quantization methods designed for LLMs. We report performance in either W4/A16 or W8/A8 based on the available data points excerpted from the papers as well as from our own experiments (Note that, as arguably the biggest bottleneck when quantizing LLMs is memory bandwidth, W4/A16 should be the favored format).\n\n| method | data-usage | average score |\n| :---: | :---: | :---: |\n| DFQ (W4/A16) | data-free | 51.47 |\n| REx + DFQ (W4/A16) | data-free | 53.76 |\n| REx + PowerQuant (W4/A16) | data-free | 54.81 |\n| OPTQ (W4/A16) | data-driven | 54.37 |\n| SmoothQuant (W8/A8) | data-driven | 53.81 |\n| LLM.int8() | data-driven | 54,11 |\n\n\nOur observations are three fold. First, contrary to OPTQ, we do not use group-wise quantization which prevents the quantization of the activations to anything lower than 16 bits. Also, contrary to LLM.int8 we do not need mixed precision between float16 and int8 among the same tensor. And contrary to all of these methods, we do not use any data. Second, OPTQ offers the best performance overall which can be attributed to both the use of group-wise quantization and the weight optimization step. Third, we believe that better performance could be achieved if we were to combine REx and SmoothQuant. Nevertheless, as such, REx+DFQ achieves very close performance to the other methods without suffering from the aforementioned limitations. Furthermore, if we use a recent non-uniform method as the quantization operator rather than DFQ, REx + PowerQuant [4] achieves the highest score.\n\n**actual speed** Thank you for your comment. We used the BOPS metric to provide results that are hardware-agnostic and easier to reproduce and compare with other work. However, we do agree that the method on its own would greatly benefit from actual latency measurements. Please see the general comments (Table 1 and 2) for the latency results.\n\n**hardware bit-width support** As our results suggest in the general comment, when the hardware does not properly support the provided bit-width, packing and un-packing does introduce overhead, which impacts REx performance. In that example, we consider a CPU that only supports int8 and fed it with int4 weight values. The resulting REx model is 0.658\\% slower than it DFQ counterpart with 4.65 higher accuracy. All in all, if the hardware offers limited support REx will only improve the accuracy of the final model. However, as our results on the gpu show, if the hardware does support more bit-widths (e.g. int1, int4 and int8), then REx both significantly improves the accuracy.\n\n**higher sparsity in table 5 leading to higher accuracy** There seems to be a misunderstanding here, due to a lack of clarity on our part. The sparsity indicated with respect to the proposed expansion such as W$4_{+ 25\\%}$/A4, means that we keep 25\\% of the expansion values. In other words, the results provided in Table 5 are indeed intuitive, i.e. the larger the expansion the higher the accuracy. This was stated in l154 where $\\gamma$ refers to the kept overhead or in other words the kept values.\n\n## references\n\n[4] Yvinec, Edouard, et al. \"PowerQuant: Automorphism Search for Non-Uniform Quantization.\" ICLR, 2023."}, {"review_id": "4p7UtM36kV", "reviewer": "Reviewer_m1LK", "summary": "The paper proposed REx which allows the flexibility to find the PTQ recipe given a speed/accuracy tradeoff by computing the residual expansion of weights and activations in a data-free manner. The method minimizes error between the FP16 and INT quantizations by computing and adding the quantized residual errors. The authors further propose to reduce the overhead with selective error computations by using the parameter magnitude as a proxy for parameter importance. Additionally, the authors also provide the theoretical error upper bound for a given expansion and show the tightness of the bound.\n\n### Post Rebuttal Update\nI have increased my score to 5 -- Borderline accept.", "questions": "I am not convinced about BOPS and how comparing existing methods at equivalent BOPS is fair.\n\nI would like to see some results on latency using this strategy. Latency (and size) of models are the true real-world metrics and I think BOPS is a poor proxy for Latency and limits the applicability of this method.\n\nI would like to understand how the authors propose to deal with unstructured sparsity that the method introduces.\n\n", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The proposed PTQ method is data-free and the idea is quite interesting.\n\nThe authors provide theoretical backing for the error upper bound for a given expansion which is greatly appreciated\n\nThe paper compares with a fair number of previous approaches\n", "weaknesses": "Figure 2 is extremely difficult to read due to the color scheme (especially for someone who might be colorblind). I would sincerely request to authors to improve the color scheme.\n\nThe authors claim that the residual expansions can be made sparse using the norm of the parameters as a selection criterion which might lead to unstructured sparsity. Especially in Table 1, where the sparsity is 50%, this would lead to high inference latencies unless the target device supports unstructured sparsities (which current GPUs do not). Why would this method be preferred over existing methods?\n\nI am not sure I understand the BOPS metric. Hardware targets only support certain bit widths which have a constant cost of operations i.e INT4 cores would take the same amount of time even if you use W3/A3 recipe. I don't think comparing existing methods at equivalent BOPS is fair.\n\nFor LLMs, the authors claim that the residue with W1/A16 has virtually no cost. I am not sure I understand I follow this, even with 1bit residue -- Assuming someone has optimized the weight loading with cache-line optimizations that have minimal impact on weight loading times, performing a large number of FP16 operations will add significant computational overhead when the models are compute-bound. \n\nIn Line 54, the authors claim a budget of $\\gamma$ for the computational overhead but instead use a budget of $\\gamma$ for the weight overhead. I would like to point out that a given weight over-head budget does not translate to an equal computational overhead [1], which is usually larger.\n\nIn general, there is no discussion on the impact of inference latencies in using this strategy which *significantly* diminishes the impact of this work.\n\nThe writing of the paper can be improved. Some of the errors I noticed were: shall -> should in line 8, in -> of in line 83\n\n\n[1] - https://dl.acm.org/doi/abs/10.1145/3400302.3415679", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**figure 2** We updated Figure 2 to make more readable (please see the pdf attached to our general comment)\n\n**bops evaluation and latency** In order to address your major concern regarding performance, we proposed a latency evaluation in the general comment in order to show the good performance of the proposed REx method on both CPU (with limited bit-width support: int8 only) and GPU (supports for multiple bit-widths: int1, int4 and int8). In our original work, we focused on BOPS due to the simpler reproducibility for future research. However, we acknowledge the fact that the proposed REx method does benefit from these new results.\n\n**sparsity: granularity and latency** We think that there were a misunderstanding regarding the sparse expansion and its support. In Table 1 and every other results (aside from LLMs), the introduced sparsity is applied at the neuron level (l154) . Thus, the sparsity is structured and is rather straightforward to leverage on hardware devices. This is confirmed by empirical evidence of latency evaluation (please see Table 1 and 2 in the general comment). On the other hand, regarding the sparse (unstructured) expansion for LLMs, this method does need support for sparse matrix multiplications. While nvidia gpus only support semi-structured pruning we do believe that this format can offer greater benefits as we evaluate it CPU. We show that the sparse binary expansion only adds a 0.26\\% overhead in terms of latency which, in our opinion, further supports our initial claim on the negligibility of this operation.\n\nWe hope that these clarifications convince you on the interest of the proposed ideas.\n"}]

Author rebuttal:
# General Comment

We thank the reviewers for their interest and critics. The theoretical guarantees and thorough empirical validation offered by REx were highlighted by the reviewers as a strength of this paper. In the initial submission, we measured the accuracy with respect to the number of BOPS (Bitwise Operation Per Second) for a simple and fair comparison. From our understanding, It appears that a major and common concern among some of the reviewers is that we did not provide comparisons in terms of direct on-device latency. We propose a common response to this matter here, and will address the reviewers' other concerns individually.

The choice of the evaluation metric in deep neural network acceleration is not trivial. The BOPS metric bare the advantage of being easily reproducible and comparable as it is (hardware and software agnostic). This is the reason why we opted for this metric. However, we see the point raised by the reviewers and agree that the study would greatly benefit from actual runtime measurements. Consequently, we propose the following extra experiments: 
 1. our main result and comparison is reported in Table 1 below, in which we report the latency of the referenced methods as well as the result from REx. Please bear in mind that in Table 1 the sparsity is structured (which we did not clearly mention in the paper).
 2. our second main result is reported in Table 3 with LLMs, in which case the sparsity is unstructured in order to further limit the memory overhead as it is a major concern with LLM quantization. Thus, in the case of LLM only, we will measure the performance obtained using unstructured expansions.

 ## Latency with structured sparse expansions
50\% structured sparsity is leveraged by all hardware devices. Thus, we could conduct our experiments on multiple hardware devices. We considered a CPU (intel xeon) and a GPU (A100) for their difference in bit-width support. The latency is reported as the average over 1000 runs in milliseconds. We decide to compare to DFQ as it is the method that offers the lowest latency due to its use of per-tensor uniform quantization. Our results are listed in the Table below (for a ResNet-50).

| method | CPU | GPU | accuracy |
|:---:|:---:|:---:|:---:|
| original model | 7.108 | 0.861 | 76.15 |
| DFQ W6/A6 | **3.039** | 0.606 | 71.36 |
| REx 150% W4/A6 | 3.059 ($+0.658$) %| **0.564** ($-6.930$) %| 76.01 |

We can observe that although the bops are equivalent REx offers a lower latency than DFQ on GPU by $6.930$%. This result can be explained by the fact that in the GPU do provide support in int4 and int8. On the other hand on the CPU, we can clearly see the lack of support for this bit-width which leads to the measurement of the expansion overhead only and an overhead of $0.658$%. Still this overhead is fairly limited due to the good parallelization capabilities of the hardware. If measured the throughput instead, the lack of support for the int4 format would hinder the performance of REx on CPU.

Overall, if the hardware supports multiple quantization format, then REx offers the highest accuracy at the lowest inference latency which supports our initial claim that REx offers better trade-offs in terms of accuracy *v.s.* speed.

 ## Latency with unstructured sparse expansions
Regarding unstructured sparsity efficiency, we only considered the CPU benchmark as it is the only support for such inference format. For our own sake, we measured the latency of a single fully-connected layer from the MLP block of the OPT-13B model on which we conducted our initial experiments. Similarly to the previous test, we measure 1000 runs using a naive implementation based on scipy and report the average latency in the table down below

| method | CPU |
|:---:|:---:|
| original layer         | 0.104799 |
| DFQ W4/A16             | 0.055230 |
| REx W4/A16 + 1% A1/A16 | 0.055376 $(+0.2643)$% |

Our results highlight the marginal overhead of $0.2643$% introduced by the sparse binary expansion on fully-connected layers.

All in all, we believe that these results (Table 1 and 2 in this response for latency, as well as e.g. Table 3 in the paper for accuracy on OPT-13B), and the fact that REx significantly improve the accuracy (and outlier handling in LLMs) of existing quantization methods at the price of, in the worst case scenario (when the considered bit-width is not supported), very little latency overhead, and with adequate hardware support, significant speed boost. This further shows the interest of the proposed method.


Meta-review: All reviewers voted for "weak/borderline accept". The main reasons were

1. The paper is clearly written.
2. The method is novel, interesting, and with theoretical guarantees.
3. There are extensive experiments, on many models and benchmarks.
4. There are improvements over previous SOTA methods.
5. Some attention is given to efficient hardware implementation.

Though 4+5 could be improved further, I feel these current results are sufficient to warrant acceptence.
Decision: Accept (poster)