SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tianqi Chen $^{1,2}$ , Weixiang Xu $^{1}$ , Weihan Chen $^{1}$ , Peisong Wang $^{1,2,3\dagger}$ , Jian Cheng $^{1,2,3,4\dagger}$
$^{3}$ AIRIA  $^{4}$ Maicro.ai
{chentianqi2023, xuweixiang2018,chenweihan2018}@ia.ac.cn,
{peisong.wang,jcheng}@nlpr.ia.ac.cn
The Winograd algorithm is an efficient convolution implementation, which performs calculations in the transformed domain. To further improve the computation efficiency, recent works propose to combine it with model quantization. Although Post-Training Quantization has the advantage of low computational cost and has been successfully applied in many other scenarios, a severe accuracy drop exists when utilizing it in Winograd convolution. Besides, despite the Winograd algorithm consisting of four stages, most existing methods only quantize the element-wise multiplication stage, leaving a considerable portion of calculations in full precision. In this paper, observing the inconsistency among different transformation procedures, we present PTQ-Aware Winograd (PAW) to optimize them collaboratively under a unified objective function. Moreover, we explore the full quantization of faster Winograd (tile size  $\geq 4$ ) for the first time. We further propose a hardware-friendly method called Factorized Scale Quantization (FSQ), which can effectively balance the significant range differences in the Winograd domain. Experiments demonstrate the effectiveness of our method, e.g., with 8-bit quantization and a tile size of 6, our method outperforms the previous Winograd PTQ method by  $8.27\%$  and  $5.38\%$  in terms of the top-1 accuracy on ResNet-18 and ResNet-34, respectively.
Recently, Convolution Neural Networks (CNNs) have demonstrated state-of-the-art performance in various computer vision tasks [1; 2; 3; 4]. However, the high computation and storage cost hinders their deployment on resource-limited devices. To address this problem, various solutions have been proposed in the literature, including network pruning [5], low-rank decomposition [6], network quantization [7; 8; 9; 10; 11] and faster convolution implementation [12; 13]. In this paper, we focus on network quantization, faster convolution implementation, and their combination.
Model quantization converts the floating-point weights and activations to low-bit integers. According to whether data with labels are required for training, quantization can be divided into two categories: Quantization-Aware Training (QAT) [14; 15; 16; 17] and Post-Training Quantization (PTQ) [18; 19; 20; 21]. Although QAT can achieve promising performance, training the network requires many GPU resources and a full training dataset. Therefore, it is not always practical when either the training dataset is unavailable (e.g., privacy and commercial training data) or rapid deployment is required. On the contrary, as PTQ only needs few-shot, unlabeled calibration data and fewer computation resources, it is widely applicable in the industry.
An alternative method for enhancing the speed of CNNs involves the development of faster convolution implementations, such as FFT [13] and Winograd [12]. Among them, the Winograd algorithm is the most popular fast convolution operator [22]. Recent research [23; 24; 25] has focused on combining the Winograd algorithm with model quantization to further utilize the benefits of both techniques. However, two significant problems remain: On the one hand, while Winograd convolution with larger tile sizes (tile size  $\geq 4$ ) can provide more significant acceleration, existing PTQ methods for such convolution still suffer from drastic accuracy degradation. On the other hand, most existing works [24; 26] only quantize the element-wise multiplications, while leaving the domain transformation implemented by full-precision matrix multiplications, which can comprise a significant portion of the computation ( $40.1\%$  when the input and output channels are 32 [24]). However, according to our experiments (Table 2 and Figure 2), these matrix multiplications are challenging to quantize because of the significant distribution imbalance in the Winograd domain.
In this paper, we first identify that quantization makes the transformation procedures in the Winograd algorithm inconsistent. Based on this observation, we propose PTQ-Aware Winograd (PAW), which optimizes transformation matrices collaboratively under a unified objective function. We then explore the fully quantized Winograd convolution, which is the first time for large tile sizes. We empirically find it is a non-trivial task because of the significant range differences of pixels in the Winograd domain. Through theoretical analysis of this phenomenon, we discover that it is possible to achieve comparable performance to per-pixel quantization without sacrificing computational efficiency via Factorized Scale Quantization (FSQ), which factorizes the tile size scales into vector size scales. Experiments are conducted to demonstrate the effectiveness of our method, e.g., with 8-bit quantization and a tile size of 6, our method outperforms the previous Winograd PTQ method by  $8.27\%$  and  $5.38\%$  in terms of the top-1 accuracy on ResNet-18 and ResNet-34, respectively.
Overall, our contributions in this work are threefold:
- We experimentally observe that quantization will disrupt the consistency between different transformation procedures and propose PTQ-Aware Winograd, which utilizes a unified optimization procedure to make the Winograd algorithm more robust to quantization.
- Through extensive experiments and theoretical analysis, we propose Factorized Scale Quantization, a hardware-friendly method suitable for the distribution characteristic of the Winograd domain tensors.
- To the best of our knowledge, we are the first to achieve full quantization of Winograd convolution with large tile sizes (4 and 6). Experiments prove that our proposed method shows significant improvements over previous PTQ methods, even under fully-quantization settings.
Quantization-aware training (QAT) [8; 27; 28; 29; 30] is a technique that simulates quantization noise during end-to-end training of neural networks. It uses discretized weights during both forward and backward propagation and updates original full-precision weights. Since the gradient of the quantization function is either zero or undefined everywhere, this procedure is done via a gradient approximation method called straight-through estimator [31]. Recently, some methods also add quantization related parameters to this training procedure, such as clipping range [32] and step size [14]. Although QAT methods have promising performance, they need the whole dataset and huge GPU resources.
Post-training quantization (PTQ) [7; 21; 9; 18; 33] is a more lightweight method that does not require retraining the network end-to-end. It only needs a small number of samples to estimate activation distribution. Despite its efficiency, PTQ suffers from a more significant accuracy degradation than QAT. The research community has actively proposed various methods to alleviate this problem. For example, [21] observes the scaling equivariance of activation functions and proposes to balance weights in consecutive layers. They also propose bias correction, which absorbs high biases into the next layer. Recently, some methods have found that the change in feature maps is a practical proxy signal for final task loss and use a layer-wise feature map reconstruction procedure to boost performance. Bit-Split [7] turns the low-bit discrete optimization into multiple single-bit optimization problems to find the optimal quantization scales and integer weights. AdaRound [18]
Table 1: Percentage of the computational burden of different operations in F(6,3) Winograd convolution. Experiments are conducted on different blocks of ResNet-20. "Offline" means this operation can be preprocessed before inference.
Operation	Block 0	Block 1	Block 2	Total
BTXB	22%	16%	10%	16%
U ⊙ V	39%	56%	72%	56%
ATOA	39%	28%	18%	28%
GWGT	Offline	Offline	Offline	Offline
proposes optimizing the round function with a soft relaxation, which makes huge progress than the original round-to-nearest function.
Winograd convolution quantization Winograd is an fast convolution implementation first applied by [12]. To further improve computational efficiency, many works [23; 24; 34; 35; 36] focus on combining it with model quantization. For Winograd convolution networks with tile size  $\geq 4$ , most works can only achieve an acceptable accuracy drop when using QAT, where the strong effectiveness of retraining the whole network hides the difficulty of quantization [23; 34; 36]. For instance, BQW [24] observes the heterogeneity of channel ranges in transformed tensors and equalizes them via migrating the scale variance from activations to weights. However, compared to the significant performance improvements achieved in QAT, their method only achieves limited progress in the case of PTQ. One of the reasons, we think, is that their method ignores the quantization impact on the Winograd algorithm, which is more notable without fine-tuning the whole network. In this paper, we will solve this problem and explore a more challenging scenario, fully quantized Winograd convolution using PTQ.
3.1 Winograd
Winograd proposes the minimum filtering algorithm of finite impulse response (FIR) filtering in [37]. For  $r \times r$  standard convolutions with filter size  $r$ , the algorithm transforms the convolution operations to the Winograd domain and generates  $m \times m$  (spatial) outputs at a time, which is denoted as  $F(m, r)$ . The parameter  $m$  is called tile size, which is used to balance the speedup and numerical precision. The Winograd algorithm can be divided into four stages:
Input transformation: Firstly, a patch  $X \in \mathbb{R}^{C_i \times a \times a}$  is extracted from the input data, with patch size  $a = m + r - 1$ . Then the c-th channel sub-tensor of  $X$ , denoted as  $X_c$ , is transformed into the Winograd domain using input transformation matrix  $B \in \mathbb{R}^{a \times a}$ . In this paper, the transformed inputs are denoted as  $U \in \mathbb{R}^{C_i \times a \times a}$ :
$$
U _ {c} = B ^ {T} X _ {c} B, \quad c = 1, \dots , C _ {i} \tag {1}
$$
Weight transformation: Similarly, weights  $W \in \mathbb{R}^{C_o \times C_i \times r \times r}$  are transformed into the Winograd domain to get transformed weights  $V \in \mathbb{R}^{C_o \times C_i \times a \times a}$ . This process can be performed offline before inference because model weights are frozen after training.
$$
V _ {f, c} = G W _ {f, c} G ^ {T}, \quad G \in \mathbb {R} ^ {a \times r} \quad \text {w h e r e} \quad f = 1, \dots , C _ {o}, \quad c = 1, \dots , C _ {i} \tag {2}
$$
Element-wise multiplication: In the Winograd domain, the convolution operation is performed by element-wise multiplications between  $U$  and  $V$ :
$$
O _ {f} = \Sigma_ {c} ^ {C _ {i}} U _ {c} \odot V _ {f, c} \tag {3}
$$
where  $\odot$  denotes element-wise multiplication.
Output transformation: Finally,  $O \in \mathbb{R}^{C_o \times a \times a}$  are transformed back to the feature map domain, and then, we get final outputs  $Y \in \mathbb{R}^{C_o \times m \times m}$ :
$$
Y _ {f} = A ^ {T} O _ {f} A, \quad A \in \mathbb {R} ^ {a \times m} \quad \text {a n d} \quad f = 1, \dots , C _ {o} \tag {4}
$$
Depending on the particular choice of Winograd domain (i.e., polynomial domain), these transformation matrices  $A$ ,  $B$ , and  $G$  in Eq. (1)-(4) can be different. For  $F(2,3)$ , the most common choice of the Winograd domain is  $f(x) = (x - 1)(x + 1)x$ , then  $A$ ,  $B$  and  $G$  can be constructed as follows[12]. More details are provided in the supplemental materials.
$$
A ^ {T} = \left[ \begin{array}{c c c c} 1 & 1 & 1 & 0 \\ 0 & 1 & - 1 & - 1 \end{array} \right], \quad B ^ {T} = \left[ \begin{array}{c c c c} 1 & 0 & - 1 & 0 \\ 0 & 1 & 1 & 0 \\ 0 & - 1 & 1 & 0 \\ 0 & 1 & 0 & - 1 \end{array} \right], \quad G = \left[ \begin{array}{c c c} 1 & 0 & 0 \\ \frac {1}{2} & \frac {1}{2} & \frac {1}{2} \\ \frac {1}{2} & - \frac {1}{2} & \frac {1}{2} \\ 0 & 0 & 1 \end{array} \right] \tag {5}
$$
Although the Winograd algorithm reduces the computational complexity of convolution operations by performing them in the Winograd domain, these transformation steps also bring additional overheads. An example is shown in Table 1. When the channel numbers  $C_o$  and  $C_i$  are not large enough, these transformation processes will incur considerable computational costs.
Quantization converts a full-precision tensor to an integer one. To improve hardware simplicity and efficiency, we utilize symmetric uniform quantization without zero-point. For a tensor  $V$ , quantization maps it to the integer values  $\widetilde{V}$  and de-quantization remaps it to float-point values  $Q(V)$ :
$$
\widetilde {V} = \left\lfloor \operatorname {c l i p} \left(\frac {V}{s}, - q _ {\min }, q _ {\max }\right) \right] \tag {6}
$$
$$
Q (V) = \widetilde {V} \cdot s \tag {7}
$$
where  $s$  represents a full-precision scalar called quantization scale,  $\lfloor z\rfloor$  rounds  $z$  to the nearest integer and  $clip(z,r_1,r_2)$  clamps  $z$  into the range  $[r_1,r_2]$ . For symmetric quantization with  $B$  bits,  $q_{min} = -2^{B - 1}$  and  $q_{max} = 2^{B - 1}$ .
In Eq. (6), we utilize a scalar  $s$  to quantize the entire tensor, called per-tensor quantization. Previous studies [24; 26] have shown that using per-pixel quantization, which provides independent scales for different pixels of transformed tensors, can significantly improve performance:
$$
\tilde {U} _ {c} = \left\lfloor U _ {c} \oslash S _ {U} \right] \tag {8}
$$
$$
\widetilde {V} _ {f, c} = \left\lfloor V _ {f, c} \oslash S _ {V} \right] \tag {9}
$$
Here  $\varnothing$  is the element-wise division and we emit the clip function.  $S_U$  and  $S_V\in \mathbb{R}^{a\times a}$  are tile size scales and can be factored out of the summation of Eq. (3). Thus, we can perform MAC operations in fixed-point format:
$$
O _ {f} = \underbrace {\left(\Sigma_ {c} ^ {C _ {i}} \widetilde {U} _ {c} \odot \widetilde {V} _ {f , c}\right)} _ {\text {i n t 8}} \odot \left(S _ {U} \odot S _ {V}\right) \tag {10}
$$
4.1 PTQ-aware Winograd algorithm
While current state-of-the-art PTQ methods [18; 19; 9; 33] can achieve near-lossless 6-bit quantization on popular non-Winograd CNNs, recent research has shown that 8-bit quantization of Winograd convolutions still suffers from a significant accuracy drop (e.g.,  $9.71\%$  for F(6,3) on ResNet-18 [24]). We argue that one key reason why quantization and the Winograd algorithm are not well-compatible is that quantization alters input and weight transformation procedures, making the whole algorithm inconsistent.
To support our argument, we experiment on the first layer of ResNet-18 and take weight transformation as an example. After quantization, the weight transformation procedure is changed from  $GWG^T$  to  $Q(GWG^T)$ . Consequently, the other two transformation procedures,  $A^TOA$  and  $B^TXB$ , will not be consistent with this new weight transformation. To illustrate it, we randomly perturb  $A$  and  $B$  and compute the reconstruction loss  $\mathcal{L}_{\text{quant}}$  and  $\mathcal{L}_{\text{origin}}$  with or without quantization, respectively:
$$
\mathcal {L} _ {\text {o r i g i n}} = \Sigma_ {f} ^ {C _ {o}} \left\| A ^ {T} \left(\Sigma_ {c} ^ {C _ {i}} \left(B ^ {T} X _ {c} B\right) \odot \left(G W _ {f, c} G ^ {T}\right)\right) A - Y _ {f} \right\| ^ {2} \tag {11}
$$
(a) Random  $A,B$
(b) Random  $A,G$
Figure 1: Reconstruction loss points obtained by randomly perturbing  $A$ ,  $B$  (a) or  $A$ ,  $G$  (b). The bottom points indicate a minor loss without quantization. And the left points indicate a minor loss with quantization.
$$
\mathcal {L} _ {\text {q u a n t}} = \Sigma_ {f} ^ {C _ {o}} \| A ^ {T} \left(\Sigma_ {c} ^ {C _ {i}} \left(B ^ {T} X _ {c} B\right) \odot Q \left(G W _ {f, c} G ^ {T}\right)\right) A - Y _ {f} \| ^ {2} \tag {12}
$$
The results are shown in Figure 1(a). Although original  $A$  and  $B$  can produce desired results with floating-point  $G W G^{T}$  (the yellow triangle on the bottom), they are no longer optimal when changing  $G W G^{T}$  to  $Q(GW_{G}^{T})$ . Concretely, about half of the random perturbed matrices lead to smaller errors (the blue dots on the upper left of the yellow triangle). A similar phenomenon also exists for input transformation, as shown in Figure 1(b). These experiments indicate that the inconsistency between domain transformations caused by quantization renders the original matrices unsuitable for quantization scenarios, which is the primary reason why Winograd convolutions are difficult to quantize. In order to align these transformation procedures after quantization, we propose to adjust transformation matrices via an optimization procedure as follows:
$$
\underset {A, B, G} {\operatorname {a r g m i n}} \mathbb {E} _ {X \sim \mathcal {D}} \left[ \Sigma_ {f} ^ {C _ {o}} \left\| A ^ {T} \left(\Sigma_ {c} ^ {C _ {i}} Q \left(B ^ {T} X _ {c} B\right) \odot Q \left(G W _ {f, c} G ^ {T}\right)\right) A - Y _ {f} \right\| ^ {2} \right] \tag {13}
$$
Since the transformation procedures in the Winograd algorithm are adopted considering the effect of PTQ, we name this method as PTQ-Aware Winograd (PAW). By using the straight-through estimator [31] to approximate the gradient through the round function as a pass-through operation, we can obtain the derivative of  $G$  (derivatives of  $A$  and  $B$  are shown in supplemental materials) and update it using SGD:
$$
\frac {\partial \mathcal {L}}{\partial G} = \Sigma_ {f} ^ {C _ {o}} \Sigma_ {c} ^ {C _ {i}} \left(\frac {\partial L}{\partial O _ {f}} \odot Q (U _ {c})\right) G W _ {f, c} ^ {T} + \left(\frac {\partial L}{\partial O _ {f}} \odot Q (U _ {c})\right) ^ {T} G W _ {f, c} \tag {14}
$$
$$
\frac {\partial L}{\partial O _ {f}} = 2 A \left(A ^ {T} O _ {f} A - Y\right) A ^ {T} \tag {15}
$$
4.2 Fully-quantized Winograd convolution
4.2.1 Motivation
Winograd convolution involves a sequence of computational steps from Eq. (1) to Eq. (4). To further improve computation efficiency via full quantization, all these operations need to be quantized except
Table 2: Exploration on 8-bit quantization of different components of Winograd convolution on ResNet-18. Output tensor in the Winograd domain  $(O)$  is the primary cause of accuracy degeneration.
FP	A	B	X	U	V	O	BTX	ATO
Accuracy	69.76	69.74	69.66	69.67	69.40	69.40	0.20	69.22	66.56
Figure 2: The ranges of  $O$  vary widely between different pixels. In both layers, the std of  $O_{5,5}$  is ten times larger than that of  $O_{4,4}$ .
Table 3: The performance of different quantization schemes for  $O$ . Although per-pixel quantization is excellent in maintaining model accuracy, it is not supported in hardware.
Granularity	Accuracy	Hard-friendly
Quantize O (per-tensor)	0.216	✓
Quantize O (per-pixel)	68.740	✗
Quantize O (ours)	67.866	✓
Eq. (2), which can be done offline. However, we empirically find that it is non-trivial to achieve full quantization because that accuracy will drop to zero due to the quantization of  $O$  (i.e., the output tensor in the Winograd domain), as shown in Table 2. After further investigation, we find that the ranges of  $O$  vary widely between different pixels. For example, as shown in Figure 2, the standard deviation (std) of  $O$  distributed at pixel (4,4) is ten times larger than that at pixel (5,5). A shared quantization scale for them is difficult to achieve both small round error and clamping error. A possible solution is to provide independent scales for them, i.e., to use per-pixel quantization (Table 3). However, since  $O$  will take part in matrix multiplication (Eq. 4) instead of element-wise multiplication, per-pixel quantization will lead to different scales in the summation dimension, which makes it not feasible in general hardware.
4.2.2 Factorized-scale quantization
Motivated by the strong range difference, we present the following theorem, which shows the cause of this distribution characterization and proves the standard deviation of  $O$  can be factorized into two vectors.
Proposition 1. Assume all elements of  $W \in \mathbb{R}^{C_i \times r \times r}$  and  $X \in \mathbb{R}^{C_i \times a \times a}$  are independently and identically distributed variables with zero mean (e.g.,  $X \sim \mathcal{N}(0, \sigma_X)$ ,  $W \sim \mathcal{N}(0, \sigma_W)$ ). If denote  $B^T = (b_1^T, b_2^T, \ldots, b_a^T)^T$  and  $G = (g_1^T, g_2^T, \ldots, g_a^T)^T$ , we have:
$$
V a r \left[ O _ {i j} \right] = V a r \left[ \Sigma_ {c} ^ {C _ {i}} \left(B ^ {T} X _ {c} B \odot G W G ^ {T}\right) _ {i j} \right] = u _ {i} v _ {j} \tag {16}
$$
where  $u_{i} = \sqrt{C_{i}\sigma_{X}\sigma_{W}} ||b_{i}||^{2}||g_{i}||^{2}$  and  $v_{j} = \sqrt{C_{i}\sigma_{X}\sigma_{W}} ||b_{j}||^{2}||g_{j}^{2}||$
Proof. Firstly, since  $B^T X_c B$  and  $GW_c G^T$  are the linear combinations of independent and identical variables with zero mean, we can calculate their mean and variance:
$$
E \left[ \left(B ^ {T} X _ {c} B\right) _ {i j} \right] = 0 \quad a n d \quad E \left[ \left(G W _ {c} G ^ {T}\right) _ {i j} \right] = 0, \tag {17}
$$
$$
\operatorname {V a r} \left[ \left(B ^ {T} X _ {c} B\right) _ {i j} \right] = \operatorname {V a r} \left[ b _ {i} ^ {T} X _ {c} b _ {j} \right] = \left\| b _ {i} \right\| ^ {2} \| b _ {j} ^ {2} \| \sigma_ {X}, \tag {18}
$$
$$
V a r \left[ \left(G W _ {c} G ^ {T}\right) _ {i j} \right] = V a r \left[ g _ {i} ^ {T} W _ {c} g _ {j} \right] = \left\| g _ {i} \right\| ^ {2} \| g _ {j} ^ {2} \| \sigma_ {W} \tag {19}
$$
Then we can derive the variance of  $O$ :
$$
\begin{array}{l} V a r \left[ O _ {i j} \right] \stackrel {(a)} {=} \Sigma_ {c} ^ {C _ {i}} V a r \left[ (B ^ {T} X _ {c} B) _ {i j} \cdot (G W _ {c} G ^ {T}) _ {i j} \right] \\ \stackrel {(b)} {=} \Sigma_ {c} ^ {C _ {i}} \operatorname {V a r} \left[ \left(B ^ {T} X _ {c} B\right) _ {i j} \right] \operatorname {V a r} \left[ \left(G W _ {c} G ^ {T}\right) _ {i j} \right] = C _ {i} \| b _ {i} \| ^ {2} \| b _ {j} \| ^ {2} \| g _ {i} \| ^ {2} \| g _ {j} ^ {2} \| \sigma_ {X} \sigma_ {W} \tag {20} \\ \end{array}
$$
The equation (a) holds because the variables in different channels are independent. The equation (b) holds because the random variables  $U$  and  $V$  are independent of each other and have zero mean.
After finding that the standard deviation of  $O$  can be decoupled along rows and columns separately, an intuitive thought is that the per-pixel scales may also satisfy similar rules. A more theoretical proof is provided as Theorem 1.
Figure 3: An illustration of Factorized Scale Quantization. Since the standard deviation of  $O$  can be decoupled along rows and columns separately, using two vector scales,  $\alpha$  and  $\beta$ , is enough for balancing  $O$  to a similar distribution.
Theorem 1. Assuming  $X \sim \mathcal{N}(0, \sigma_X)$ ,  $W \sim \mathcal{N}(0, \sigma_W)$ , the optimal per-pixel scales  $S^*$  which minimize quantization error in Eq. (21) can be factorized as:  $S^* = \alpha * \beta^T$ .
$$
\underset {S} {\operatorname {a r g m i n}} \quad E _ {X \sim \mathcal {N} (0, \sigma_ {X}), W \sim \mathcal {N} (0, \sigma_ {W})} \left[ \| O - Q _ {S} (O) \| ^ {2} \right] \tag {21}
$$
Proof. In Proposition 1, we have proven that:
$$
V a r \left[ O _ {i j} \right] = \sqrt {C _ {i} \sigma_ {X} \sigma_ {W}} \left\| b _ {i} \right\| ^ {2} \left\| g _ {i} \right\| ^ {2} \cdot \sqrt {C _ {i} \sigma_ {X} \sigma_ {W}} \left\| b _ {j} \right\| ^ {2} \left\| g _ {j} ^ {2} \right\| \tag {22}
$$
Observing that  $O_{ij}$  is the summation of  $C_i$  independent variables, where  $C_i$  is the number of input channels and is usually large (e.g., 128). According to the Central Limit Theorem [38], we can assume  $O_{ij}$  follows Gaussian distribution. In supplementary material, we will show that the optimal scale  $s$  to minimize the mean-square error of quantizing Gaussian variables  $z \sim \mathcal{N}(0, \sigma^2)$  is proportional to  $\sigma$ , i.e.,  $s = K\sigma$ , where  $K$  is a constant. Thus, we have: