SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Edouard Yvinec $^{1,2}$ , Arnaud Dapogny $^{2}$ , Matthieu Cord $^{1}$ , Kevin Bailly $^{1,2}$   
Sorbonne Université $^{1}$ , CNRS, ISIR, f-75005, 4 Place Jussieu 75005 Paris, France  
Datakalab $^{2}$ , 114 boulevard Malesherbes, 75017 Paris, France  
ey@atakalab.com
Deep neural networks (DNNs) are ubiquitous in computer vision and natural language processing, but suffer from high inference cost. This problem can be addressed by quantization, which consists in converting floating point operations into a lower bit-width format. With the growing concerns on privacy rights, we focus our efforts on data-free methods. However, such techniques suffer from their lack of adaptability to the target devices, as a hardware typically only supports specific bit widths. Thus, to adapt to a variety of devices, a quantization method shall be flexible enough to find good accuracy v.s. speed trade-offs for every bit width and target device. To achieve this, we propose REx, a quantization method that leverages residual error expansion, along with group sparsity. We show experimentally that REx enables better trade-offs (in terms of accuracy given any target bit-width) on both convnets and transformers for computer vision, as well as NLP models. In particular, when applied to large language models, we show that REx elegantly solves the outlier problem that hinders state-of-the-art quantization methods. In addition, REx is backed off by strong theoretical guarantees on the preservation of the predictive function of the original model. Lastly, we show that REx is agnostic to the quantization operator and can be used in combination with previous quantization work.
Deep neural networks (DNNs) achieve outstanding performance on several challenging computer vision tasks such as image classification [1], object detection [2] and semantic segmentation [3] as well as natural language processing benchmarks such as text classification [4]. However, their accuracy comes at a high computational inference cost which limits their deployment, moreso on edge devices when real-time treatment as well as energy consumption are a concern. This problem can be tackled via DNN quantization, i.e. by reducing the bit-width representation of the computations from floating point operations (FP) to e.g. int8 (8-bits integer representation), int4, int3 or even lower-bit representation such as ternary (where weights values are either  $-1$ , 0 or  $+1$ ) quantization. Because DNN inference principally relies on matrix multiplication, such quantization dramatically diminishes the number of bit-wise operations (as defined by [5]), thus limiting the DNN latency and energy consumption. However, DNN quantization usually comes at the expense of the network accuracy. As a consequence, DNN quantization is an active field of research [6, 7, 8, 9, 10, 11, 12, 13] that aims at limiting this accuracy drop while reducing the number of bit-wise operations.
All the aforementioned methods are data-driven as they either involve training a network from scratch or fine-tune an already trained and quantized one. However, while such approaches usually allow lower quantization errors using low bit-wise representations, due to the growing concerns on privacy rights and data privacy, there is an ever-increasing number of real-case scenarios (e.g. health and
military services) where data may not be available for quantization purpose. Furthermore, the bloom of large language models (LLMs) that are very expensive to train further motivates the use of post-hoc data-free quantization methods. Motivated by these observations, several data-free quantization algorithms were published in recent years [14, 15, 16, 17, 18, 19], which focus on the quantization operator, i.e. the transformation which maps the floating point weights to their low-bit, fixed point, values. However, these approaches still struggle to offer an interesting alternative to data-driven techniques in terms of accuracy.
Furthermore, when considering a specific target device for deployment, traditional quantization methods, usually focusing on the quantization operator, offer limited options: given a supported bit width (given by the device, as most hardware usually support only a few representation formats [20]) they either achieve satisfactory accuracy or not. To address this concern, we wish to design a flexible quantization method, i.e. one that can provide several accuracy vs. speed trade-off points for each bit width. Drawing inspiration from wavelets-based methods for image compression [21, 22], we tackle this limitation by considering the successive residual quantization errors between the quantized and original model. Increasing the number of residuals in the expansion (i.e. the expansion order) increases the fidelity to the original, non-quantized model at the expense of additional computations. In addition, we propose a group-sparse expansion which allows us to maintain the accuracy using significantly less bit operations. Hence, given a target device, our approach allows finding the best accuracy vs. speed trade-off. Our contributions are thus four-fold:
- REx, a data-free quantization method that is both efficient and flexible. REx leverages residual quantization, along with group-sparsity, to enable finding suitable trade-offs depending on a target bit-width.
- Theoretical guarantees on both the exponential convergence of the quantized model towards the full-precision model and the maximum error with respect to the predictive function. This is of paramount importance in a data-free context, where we cannot easily measure the accuracy degradation.
- Extensive empirical validation we show through a thorough empirical validation that, as a standalone method, REx significantly outperforms every state-of-the-art data-free quantization technique, allowing to find better trade-offs on a variety of benchmarks involving ConvNet for classification, object detection or semantic segmentation as well as transformers on GLUE text classification.
- A ready-to-use solution that uses a single binary residual to handle outliers within the weight distributions, which is a well-known pitfall when attempting to quantize LLMs.
2.1 Quantization
In this section, we review existing methods for DNN quantization, with an emphasis on approaches geared towards run-time acceleration. The vast majority of DNN quantization techniques rely on data usage (Quantization Aware Training). Furthermore, methods such as [7, 8, 9, 10, 11, 23, 24] rely on variants of straight through estimation to alleviate the rounding operation gradients. Among these methods, [25] bears the most resemblance with the proposed REx method. It minimizes the residual error during training, using weight decay over the residue. The similarity with REx comes from the use of a second order expansion of the quantization errors. However, it discards the quantization error after training while we propose to keep the extra operations in order to ensure a high fidelity to the provided pre-trained model.
2.2 Data-Free Quantization
Nagel et al. [14] discuss the necessity to have data available so as to successfully design a quantization pipeline. They proposed a method that consists in balancing the weight ranges over the different layers of a model, using scale invariance properties that are specific to piece-wise affine (e.g. ReLU) activation functions, and relying on a traditional, naive quantization operator [5]. In the current state of data-free quantization research, we see two major trends: methods that focus on the rounding operator itself [19, 26] and methods that generate synthetic data [27, 12, 13]. With REx, we aim
(a) residual expansion
(b) group-sparse expansion
Figure 1: Illustration of the proposed method for a two-layers neural network. (a) residual expansion at order 4: the intensity of the colormap indicates the magnitude of the residual error. (b) group-sparse expansion for orders  $k \geq 1$  ( $\gamma = 50\%$  sparsity).
at enabling hardware flexibility for these methods by allowing to find better trade-offs in terms of accuracy and compression rate given a fixed bit-width.
2.3 Flexibility in Quantization
In practice, the existing data-free quantization methods only offer a single possible quantized model given a supported bit-width. Nevertheless, most hardwares do not support a wide range of bit-width. For instance, Turing [28] and Untether [29] architectures support int4 and int8 quantization while the Nvidia A100 [20] supports int8, int4 and binary (int1) quantization. Conversely, REx circumvents this limitation by offering several trade-offs given a bit-width representation.
Let's consider  $F$ , a trained network with  $L$  layers and trained weights  $W_{l}$ . Given a target integer representation in  $b$  bits, e.g. int8 or int4, we consider a quantization operator  $Q$ . Formally,  $Q$  maps  $\left[\min \{W_l\} ;\max \{W_l\} \right] \subset \mathbb{R}$  to the quantized interval  $[-2^{b - 1};2^{b - 1} - 1]\cap \mathbb{Z}$ . The most straightforward way to do so is to apply a scaling  $s_{W_l}$  and round  $\lfloor \cdot \rfloor$  the scaled tensor, i.e.:
$$
Q \left(W _ {l}\right) = \left\lfloor \frac {W _ {l}}{s _ {W _ {l}}} \right\rceil \tag {1}
$$
With  $s_{W_l}$  the quantization scale for  $W_l$  computed as in [5], without loss of generality. Following the standard formulation [30], a quantization operator  $Q$ , comes with a de-quantization operator  $Q^{-1}$ . For the simple quantization operator  $Q$  in Equation (1), a natural choice is  $Q^{-1}(Q(W_l)) = s_{W_l} \times Q(W_l)$ . Note that, despite the notation,  $Q^{-1}$  is not a true inverse, as by definition of the quantized space, there is some loss of information. This loss, called the quantization error, is defined as:  $W_l - Q^{-1}(Q(W_l))$ . In data-free quantization, we want to minimize this error in order to achieve the highest possible fidelity to the original model. In the following section, we describe how we can efficiently reduce the quantization error for a fixed target bit-width  $b$ .
3.1 Residual Expansion
We propose to quantize the residual errors introduced by the quantization process. Although the proposed method can be applied to any tensor, let's consider a weight tensor  $\bar{W}$ . In the full-precision space  $(\mathbb{R})$ , its first approximation is  $R^1 = Q^{-1}(Q(W))$ . To reduce the quantization error, we define  $R^2$  as the quantized residual error
$$
R ^ {2} = Q ^ {- 1} \left(Q \left(W - R ^ {1}\right)\right) \tag {2}
$$
Consequently, during the quantized inference, we compute  $R^1 X + R^2 X \approx WX$  which provides a finer approximation than the simple evaluation  $R^1 X$ . The process can be generalized to any
expansion order  $K$ , leading to the following:
$$
R ^ {K} = Q ^ {- 1} \left(Q \left(W - \sum_ {k = 1} ^ {K - 1} R ^ {k}\right)\right) \tag {3}
$$
The resulting expanded layer is illustrated in Figure 1 (a) in the case  $K = 4$ . Intuitively, an expansion  $(R^1, \dots, R^K)$  provides the approximation  $\sum_{k=1}^{K} R^k$  of  $W$  and this approximation converges exponentially fast to the original full-precision weights with respect to  $K$ . As the support of the quantization error space is smaller than one quantization step, the error decreases by a factor larger than  $2^b$  with each expansion term (more details in Appendix A). Furthermore, as the quantization error decreases, it is expected that the prediction of the quantized model would achieve a closer match to the original one. This is especially important in the context of data-free quantization as not only do we not have the option to perform fine-tuning to recover accuracy, but also we cannot evaluate the degradation of the model on a calibration/Validation set. Nonetheless, we can estimate an upper bound on the maximum error  $\epsilon_{\mathrm{max}}$  introduced by quantization on the predictions as
$$
\epsilon_ {\max } \leq U = \prod_ {l = 1} ^ {L} \left(\sum_ {i = 1} ^ {l} \left(\frac {1}{2 ^ {b - 1} - 1}\right) ^ {K - 1} \frac {s _ {R ^ {i}}}{2} + 1\right) - 1 \tag {4}
$$
where  $s_{R^i}$  is the scaling factor from equation 1 applied to each residue. The detailed derivations are provided in Appendix B. This implies that, in practice and regardless on the quantization operator, a network can be quantized with high fidelity with only a few expansion orders to fit a given bit-width. Furthermore, this process can also be applied to the activations.
3.2 Input Expansion
Quantizing the weights of a DNN with the aforementioned method already leads to significant memory footprint reduction. However, to significantly decrease the inference runtime, the inputs and activations of each layer also have to be quantized so that each the computations can be processed in the quantized bit-width. For that matter, let  $I$  be the input tensor of a layer  $l$ . We define the expansion of  $I$  in quantized residual errors similarly to the weights expansion. Using the generic quantization operator  $Q$ , we get  $I^{(1)} = Q(I)$  and define the  $K^{\mathrm{th}}$  order of quantization as
$$
I ^ {(K)} = Q ^ {- 1} \left(Q \left(I - \sum_ {k = 1} ^ {K - 1} Q ^ {- 1} \left(I ^ {(k)}\right)\right)\right) \tag {5}
$$
In order to efficiently exploit the resulting expansions, we propose to bound the accumulated order of the weights and inputs. In other words, if we note  $k_{1}$  the expansion order of a residue from the inputs and  $k_{2}$  a residue from the weights, then we only perform the computations for orders such that  $k_{1} + k_{2} < K$  (the rest being negligible in comparison). As a result, the quantized layer  $l$  computes:
$$
f: I \mapsto \sum_ {k _ {1}, k _ {2} \in \{1, \dots , K \} ^ {2}} ^ {k _ {1} + k _ {2} \leq K + 1} I ^ {\left(k _ {1}\right)} \otimes R ^ {\left(k _ {2}\right)} \tag {6}
$$
where  $\otimes$  is the base operation of the layer, e.g. a convolution for a convolutional layer or a matrix multiplication for a fully-connected layer. Similarly to the weights, the error between the full-precision inputs and the proposed expansion of the inputs converges exponentially fast to 0 with respect to the order  $K$  of the expansion. However, with formulations from equations (3) and (5), the overhead computations induced by the expansion is non-negligible. In the following section, we provide a solution to tackle this issue.
3.3 Sparse Expansion
The residual expansion as defined in equation 3 is based upon the assumption that the quantization error is equally important for every neuron. Thus, we propose to reduce the overhead cost by only expanding the most important neurons. However, in data-free compression we do not have access to activations or gradients: hence, we measure the relative importance of a neuron in a layer by the norm of its weights [31]. The resulting expanded layer is illustrated in Figure 1 (b). Given a target
budget  $\gamma$  (in  $\%$  ) of overhead computations, we only expand the  $\frac{\gamma}{K - 1}\%$  most important neurons. The sparse residue is defined as:
$$
\left(R _ {\gamma} ^ {(k)}\right) _ {i} = \left(R ^ {(k)}\right) _ {i} \cdot \mathbb {1} _ {\gamma} ^ {(k)} \tag {7}
$$
where  $\mathbb{1}_{\gamma}^{(k)}$  indicates the indices of the most important neurons. Similarly to what precedes, each expansion order is derived sequentially from previous orders and we can bound the quantization error for the sparse expansion (see Appendix A). The method for computing the weights of the expanded model is summarized in Algorithm 1.
Algorithm 1 Expansion Algorithm
Require: trained DNN  $f$  with  $L$  layers, hyper-parameters:  $K$  and  $\gamma$ , operator  $Q$  initialize  $\gamma^l$  and initialize  $f^{(K)}$  as a clone of  $f$  with  $K$  per-layer kernels for  $l \in \{1, \dots, L\}$  do  
 $W \leftarrow$  base kernel of layer  $l$  in  $f$ $W_{\mathrm{acc}} \gets 0$  accumulated quantization error  
for  $k \in \{1, \dots, K\}$  do  
 $R_{\gamma^l}^{(k)} \gets Q(W - W_{\mathrm{acc}}) \mathbb{1}_{\gamma}^{(k)}$   
set  $k^{\mathrm{th}}$  kernel of layer  $l$  of  $f^{(K)}$  with  $R_{\gamma^l}^{(k)}$ $W_{\mathrm{acc}} \gets W_{\mathrm{acc}} + Q^{-1}(R_{\gamma^l}^{(k)})$   
end for  
end for
Also note that in the sparse expansion, we allow higher expansion orders to re-consider neurons that were previously considered unimportant. Consequently, on top of improving the exponential convergence as well as lowering the upper bound on the maximum error with respect to the overhead computations, this method systematically outperforms the standard residual expansion in practice. Proof of this result can be found in Appendix C. The budget  $\gamma$  of overhead computations can be set so as not to introduce computational overhead, depending on the bit-width  $b$ . For example, let's consider a device supporting only 8 and 1 bit (binary) quantization. If we want to achieve the same latency as 8 bit quantization using only 1bit quantization we will have a budget lower than  $700\%$  overhead w.r.t. a naive 1 bit quantization. Consequently, for full expansions, we get  $\gamma \leq \frac{8}{1} - 1 = 700\%$ . This budget is then split across layers using a simple linear repartition. This strategy gives more emphasis to the layers closest to the prediction head which also correspond to the largest layers, and empirically provides the best results [32]. As a result, given a number bit operations (BOPS), the expanded model can better fit the inference device while preserving the full-precision accuracy. Furthermore, all the added computations are performed in parallel which reduces their cost in practice. It allows better trade-offs in terms of accuracy and quantization compression rate, as will be shown in the upcoming experiments.
4 Quantization Experiments
In the following sections, we first go through the implementation requirements and efficient strategies to fully leverage the proposed expansions. Second, we perform a comparison of each expansion methods in order to show the flexibility of REx with respect to the bit-width. Third, we compare REx to other quantization schemes under the constraint of equal bit operations. Finally, we validate for each expansion their respective upper bound on the maximum error with respect to the original predictions.
4.1 Implementation Details and Benchmarks
We ran our tests on 6 different backbones, including ConvNets and transformers and 5 tasks from both computer vision and natural language processing. We used ImageNet [33], Pascal VOC 2012 [34], CityScapes dataset [35] and GLUE [36] and common sense reasoning benchmarks (details in Appendix D). Unless stated otherwise, we apply symmetric, static, per-channel quantization as defined in [30] and perform batch-normalization folding prior to any processing using the optimal
Figure 2: Accuracy vs. inference time, for EfficientNet B0. The higher (accuracy) and the further to the left (inference cost) the better. The circles show the baseline results with W3/A8, W4/A8, W5/A8 and W6/A8 quantization. The dashed lines show the trade-offs performance of REx in W4/A8 and ternary quantization (W2/A8). Finally, the plain lines show REx (with sparsity at  $10\%$ ) also in W4/A4 and ternary quantization. The numbers in the symbols stand for the expansion order. REx, and a fortiori the sparse version, enables better trade-offs.
method from [37]. In order to leverage the existing efficient implementations of the convolutional layers and fully-connected layers in CUDA, we propose to implement the expanded layer using a single kernel rather than  $K$  kernels. This is achieved by concatenating the kernels along the output dimension. Consequently, the challenge of efficiently splitting the computations to fully leverage the target device computational power is left to the inference engine. In practice, this results in both better performance and less work in order to adapt the method to existing engines such as OpenVino [38] and TensorRT [39]. We detail the implementation and overhead of the addition of the residual computations in Appendix E. Furthermore, we evaluate the latency overhead of REx in Appendix F. In the following section, we demonstrate the ability of REx to find good accuracy vs speed trade-offs.
4.2 Flexible Quantization
Figure 2 shows different trade-offs enabled by REx on different bit-widths for an EfficientNet-B0 on ImageNet. First, the baseline quantization with the baseline quantization operator from [5] (as depicted by the circles of different colors, one for each bit width) offers no trade-off possibility given a specific bit-width and usually performs poorly below int8 quantization (e.g. barely reaching  $20.29\%$  top1 accuracy in W6/A8 quantization). REx, however, in the same setup, offers several trade-offs for each specific bit-width (e.g. int4 and ternary on Figure 2) and supporting hardware. Furthermore, the sparse expansion enables finding more potential trade-offs (by varying the budget and expansion order) for every bit-width. Those trade-offs are generally more interesting than comparable ones obtained using the baseline method, which empirically confirms the theoretical results (Appendix C). Furthermore, Figure 2 shows that using higher order, sparse residues allows to find even better trade-offs, as, in this case, e.g. in W2/A8 we reach full-precision accuracy at order 10 with  $10\%$  sparse residues. This shows that the process converges fast with respect to the sparsity rates. All in all, these results demonstrate the flexibility of REx to find good accuracy v.s. speed trade-offs, given a budget of total bit operations (BOPs) to fit. In the following section, we evaluate the ability of REx to outperform existing quantization methods in terms of equal bops.
4.3.1 Experiments on Computer Vision Models
In order to highlight the benefits of residual quantization errors expansions as a stand alone improvement upon existing methods with equal BOPs, we compare REx using the naive quantization operator from [5] on a variety of reference benchmarks. First, in Table 1, we report the performance on three different computer vision networks between state-of-the-art methods in W6/A6 quantization and REx using a sparse expansion at order  $K = 2$  using  $50\%$  of a 4 bit representation in order to get a similar total number of bit operations (150% of 4 bits ≈ 6 bits). For all networks, REx significantly outperforms recent state-of-the-art data-free quantization methods at equal BOPs. Furthermore, we confirm these results on object detection and image segmentation as shown in Figure 3. We can
Table 1: Comparison at equal BOPs with existing methods in W6/A6 and REx with W4/A6 +50% of one 4 bit residue.
DNN	method	year	bits	Accuracy
ResNet 50 (76.15)	DFQ [14]	ICCV'19	W6/A6	71.36
ZeroQ [17]	CVPR'20	W6/A6	72.93
DSG [18]	CVPR'21	W6/A6	74.07
GDFQ [40]	ECCV'20	W6/A6	74.59
SQuant [19]	ICLR'22	W6/A6	75.95
SPIQ [26]	WACV'23	W6/A6	75.98
REx	-	150% × W4/A6	76.01
MobNet v2 (71.80)	DFQ [14]	ICCV'19	W6/A6	45.84
SQuant [19]	ICLR'22	W6/A6	61.87
SPIQ [26]	WACV'23	W6/A6	63.24
REx	-	150% × W4/A6	64.20
EffNet B0 (77.10)	DFQ [14]	ICCV'19	W6/A6	43.08
SPIQ [26]	ICLR'22	W6/A6	54.51
REx	-	150% × W4/A6	57.63
Figure 3: (left) Mean intersection over union (mIoU) of a Deeplab V3+ with MobileNet V2 backbone on CityScapes for semantic segmentation. (right) Mean average precision (mAP) of a SSD with MobileNet V2 backbone on Pascal VOC for object detection. We add the performance of a data-free quantization solution, DFQ [14] for comparison.
observe that REx can maintain the full precision accuracy while dividing by 3.23 the number of bit operations required to run an inference.
4.3.2 Experiments on NLP