Towards Efficient and Accurate Winograd Convolution via Full Quantization
Authors: Chen Tianqi
Weixiang Xu
Weihan Chen
Peisong Wang
Jian Cheng
Venue: NeurIPS 2023 poster (2023)
Keywords: Winograd Convolution
Quantization

Abstract: The Winograd algorithm is an efficient convolution implementation, which performs calculations in the transformed domain. To further improve the computation efficiency, recent works propose to combine it with model quantization. Although Post-Training Quantization has the advantage of low computational cost and has been successfully applied in many other scenarios, a severe accuracy drop exists when utilizing it in Winograd convolution. Besides, despite the Winograd algorithm consisting of four stages, most existing methods only quantize the element-wise multiplication stage, leaving a considerable portion of calculations in full precision.
In this paper, observing the inconsistency among different transformation procedures, we present PTQ-Aware Winograd (PAW) to optimize them collaboratively under a unified objective function. Moreover, we explore the full quantization of faster Winograd (tile size $\geq4$) for the first time. We further propose a hardware-friendly method called Factorized Scale Quantization (FSQ), which can effectively balance the significant range differences in the Winograd domain. Experiments demonstrate the effectiveness of our method, e.g., with 8-bit quantization and a tile size of 6, our method outperforms the previous Winograd PTQ method by 8.27\% and 5.38\% in terms of the top-1 accuracy on ResNet-18 and ResNet-34, respectively.

Towards Efficient and Accurate Winograd Convolution via Full Quantization
Tianqi Chen Weixiang Xu', Weihan Chen', Peisong Wangl.2.3 Jian Chengl.220 1233,4
Institute of Automation, Chinese Academy of Sciences Sccooo of Artificial Intelligence, University of Chinese Academy of Sciences AIRIA 'aaccro.ai
fchentiang12023, xuweixi ang2018, chenveihan2018)00... ac, cn, {peisong, wang, (eceennnnnlprr.e.eeeeee ia. ac.ac.c
Abstract
The Winograd algorithm is an efficient convolution implementation, which per- forms calculations the transformed domain. To further improve the computation efficiency, recent works propose to combine it with model quantization. Although Post- rest-Trainig Quantization has the advantage of low computational cost and has been successfully applied in many other scenarios, severe accuracy drop exists when utilizing in Winograd convolution. Besides, despite the Winograd algorithm consisting of four stages, most existing methods only quantize the element- wise multiplication stage, leaving considerable portion of calculations in full precision. In this paper, observing the inconsistency among different transformation proce- dures, we present PTQ-Aware Winograd (PAW) to optimize them collaboratively under a unified objective function. Moreover, explore the full quantization of faster Winograd (tile size 2 4) for the first time. We further propose hardware- friendly method called Factorized Scale Quantization (FSQ), which effectively balance the significant range differences in the Winograd domain. Experiments demonstrate the effectiveness of method, e.g., with 8-bit quantization and tile size of 6, our method outperforms the previous Winograd PTQ method by 8.27% and 5.38% in terms of the top-1 accuracy on ResNet-18 and ResNer-34, respectively.
Introduction
Recently, Convolution Neural Networks (CNNs) have demonstrated state-of-the-art performance in various computer vision tasks [T] 2 4. However. the high computation and storage cost hinders their deployment on resource-limited devices. To address this problem, various solutions have been proposed in the literature, including network pruning (5), low-rank decomposition [6]. network quantization
10 11 and faster convolution implementation 13) In this paper, we focus on network quantization, faster convolution implementation, and their combination.
Model quantization converts the floating-poir point weights and activations low-bit integers. According to whether data with labels are required for training, quantization can be divided into two categories: Quantization-Aware Training (QAT) [4 [5] [7] and Post-Training Quantization (PTQ) [18] 19
20 21
Although QAT can achieve promising performance, training the network requires many GPU resources and a full training dataset. Therefore, it is not always practical when either the training dataset is unavailable (c.g., privacy and commercial training data) or rapid deployment is required. On the contrary, as PTQ only needs few-shot, unlabeled calibration data and fewer computation resources, it is widely applicable in the industry.
'Correspondng authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

An alternative method for enhancing the speed of CNNs involves the development of faster convolu- tion implementations. such as FFT [33 and Winograd [2]. Among them, the Winograd algorithm is the most popular fast convolution operator [22] Recent research [23 24, 25 has focused on combin- ing the Winograd algorithm with model quantization to further utilize the benefits of both techniques. However, two significant problems remain: On the one hand, while Winograd convolution with larger tile sizes (tile size24) can provide more significant acceleration, existing PTQ methods for such convolution still suffer from drastic accuracy degradation. On the other hand, most existing works 24 26 only quantize the element-wise multiplications, while leaving the domain transformation implemented by full-precision matrix multiplications, which can comprise significant portion of the computation (40.1% when the input and output channels are 32 (24)). However, according to our experiments (Table 2 and Figure 2). these matrix multiplications are challenging to quantize because of the significant distribution imbalance in the Winograd domain.
In this paper, we first identify that quantization makes the transformation procedures in the Winograd algorithm inconsistent. Based on this observation, we propose PTQ-Aware Winograd (PAW), which optimizes transformation matrices collaboratively under a unified objective function. We then explore the fully quantized Winograd convolution, which is the first time for large tile sizes. We empirically find it is non-trivial task because of the significant range differences of pixels in the Winograd domain. Through theoretical analysis of this phenomenon, we discover that it is possible to achieve comparable performance to per-pixel quantization without sacrificing computational efficiency via Factorized Scale Quantization (FSQ), which factorizes the tile size scales into vector size scales. Experiments 8-bit quantization and a tile size of 6, our method outperforms the previous Winograd PTQ method by 8.27% and 5.38% in terms of the top-1 accuracy on ResNet-18 18 and ResNet-34, respectively.
conducted to demonstrate the effectiveness of our method, c.g., with
Overall, our contributions in this work are threefold:
We experimentally observe that quantization will disrupt the consistency between different transformation procedures and propose PTQ-Aware Winograd, which utilizes unified optimization procedure to make the Winograd algorithm more robust to quantization.
Through extensive experiments and theoretical analysis, we propose Factorized Scale Quan- tization, a hardware-friendly method suitable for the distribution characteristic of the Wino- grad domain tensors.
To the best of our knowledge, we are the first achieve full quantization of Winograd convolution with large tile sizes (4 and 6). Experiments prove that our proposed method shows significant improvements over previous PTQ methods, even under dully-quantization settings.
2 Related Works
Quantization-wware training (QAT) 8 27 28 29 30 is a technique that simulates quantization noise during end-to-end training of neural networks. It uses discretized weights during both forward and backward propagation and updates original full-precision weights. Since the gradient of the quantization function is either zero or undefined everywhere, this procedure is done via a gradient approximation method called trraihh-thhough estimator [I. Recently, some methods also add quantization related parameters to this training procedure, such as clipping range (32) and step size 14]. Although QAT methods have promising performance, they need the whole dataset and huge GPU resources.
9 18 33 is more lightweight method that does not require retraining the network end-to-end. It only needs a small number of samples to estimate activation distribution. Despite its efficiency, PTQ suffers from more significant accuracy degra- dation than QAT. The research community has actively proposed various methods to alleviate this problem. For example, (21) observes the scaling equivariance of activation functions and proposes to balance weights in consecutive layers. They also propose bias correction, which absorbs high biases into the next layer. Recently, some methods have found that the change in feature maps is a practical proxy signal for final task loss and use a layer-wise feature map reconstruction procedure to boost performance. Bit-Split 7 turns the low-bit discrete optimization into multiple single-bit bit optimization problems to find the optimal quantization scales and integer weights. AdaRound [18]
Post-training quantization (PTQ)
2

---

Table 1: Percentage of the computational burden of different operations in F(6.3) Winograd convolu- tion. Experiments are conducted on different blocks of ResNet-20. "Offline" means this operation can be preprocessed before inference.
Operation
Block
Block
Block 2
Total
B X
22%
16%
10%
16%
O V
39%
56%
72%
56%
A'
39%
28%
18%
28%
GWG
Offline
Offline
Offline
Offline
proposes optimizing the round function with soft relaxation, which makes huge progress than the original round-to-nearest function.
Winograd convolution quantization Winograd is an fast convolution implementation first applied by [[2]. To further improve computational efficiency, many works [23 24, 34 35| 36) focus on combining it with model quantization. For Winograd convolution networks with tile size 24, most works can only achieve an acceptable accuracy drop when using QAT, where the strong effectiveness of retraining the whole network hides the difficulty of quantization 23 34 36. For instance, BQW [4]] observes the heterogeneity of channel ranges in transformed tensors and equalizes them via migrating the scale variance from activations to weights. However, compared to the significant performance improvements achieved in QAT, their method only achieves limited progress in the case of PTQ. One of the reasons, think, that their method ignores the quantization impact the Winograd algorithm, which is more notable without fine-tuning the whole network. In this paper, will solve this problem and explore more challenging scenario, fully quantized Winograd convolution using PTQ.
3 Background
3.1 Winograd
Winograd proposes the minimum filtering algorithm of finite impulse response (FIR) filtering in 37.. For r x r standard convolutions with filter size r, the algorithm transforms the convolution operations to the Winograd domain and generates m x (spatial) outputs at time, which is denoted as ((m,r). The parameter m called tile size, which is used to balance the speedup and numerical precision. The Winograd algorithm can be divided into four stages:
Firstly, a patch X C.xaaa is extracted from the input data, with patch size a = m + r 1. Then the c-th channel sub-tensor of X, denoted as X., is transformed into the Winograd domain using input transformation matrix paxa, In this paper, the transformed inputs are denoted as U E p0xxxxx xaxa.
Input transformation:
U. U.=B"X,B, c c= 1=,,..,C
(I)
Weight transformation: Similarly, weights W E Coxxxr are transformed into the Winograd domain to get transformed weights V RCxx0xxaxa This process can be performed offline before inference because model weights are frozen after training.
Vf,- = G",, G Raxr where
1= Co. c= 1..,.., C.
(2)
Element-wise multiplication: In the Winograd domain, the convolution operation is performed by element-wise multiplications between U and V:
0, ④ V/.
where c denotes element-wiss multiplication.
Output transformation:
Finally, O E RCxxxx are transformed back to the feature map domain,
and then, we get final outputs Y RC xxxxm
y AE paxm and f == L,... Co
(4)
3

---

Depending on the particular choice of Winograd domain (i.e., polynomial domain), these transforma- tion matrices A, B, and in Eq. 4 can be different. For F(2.3), the most common choice the Winograd domain is f'(x) = (x ( + J),, then A, B and G can be constructed as follows More details are provided the supplemental materials.
1
0
-1
0
0
07
0 -1 -1
0
AT
BT
G
2
(5)
0
0
-1
1
0
i]
0
-1
Although the Winograd algorithm reduces the computational complexity of convolution operations by performing them in the Winograd domain, these transformation steps also bring additional overheads. An example is shown in [ When the channel numbers Co and C₁ are not large enough, these transformation processes will incur considerable computational costs.
3.2 Quantization
Quantization converts a full- preccisen tensor to an integer one. To improve hardware simplicity and efficiency, utilize symmetric uniform quantization without prro-poin.. For tensor V, quantization maps it to the integer values V and de-quantization remaps it to float-point values Q(V):
V = clip( V
Imin 9mox)
(6)
Q(V)= = s
(7)
where s represents a full-precision scalar called quantization scale, [>] rounds z to the nearest integer and (dipp(,,T1,22 clamps into the range [2], Qmin = 22 B- and Qmax = n 1-
r2]. For symmetric quantization with B bits,
In Eq. 66. we utilize scalar s to quantize the entire tensor, called per-tensor quantization. Previous studies 24 26 have shown that using per-pixel quantization, which provides independent scales for different pixels of transformed tensors, can significantly improve performance:
Ü. = LUA @ St
(8)
and
V/. = [V, 2 Sy]
(9)
Here 2 is the element-wist division and emit the clip function. Sy and Sy E Raxo tile size scales and can be factored out of the summation of Eq. 43. Thus, we can perform MAC operations in fixed-point format:
O,
(ICr U. Vn,e) 0
(10)
int8
4 Method
4.1
PTQ-aware Winograd algorithm
While current state-of-the-arr PTQ methods 18
19 33
achieve near-lossless 6-bit quantization
on popular non- Winograd CNNs, recent research has shown that 8-bit quantization of Winograd convolutions still suffers from a significant accuracy drop (e.g.. 9.71% % for F(6.3) on ResNet-18 (24). We argue that one key reason why quantization and the Winograd algorithm are not well-compatible that quantization alters input and weight transformation procedures, making the whole algorithm inconsistent.
To support argument, experiment on the first layer of ResNet- 18 and take weight transformation as an example. After quantization, the weight transformation procedure is changed from GW G" to QGW GT) Consequently, the other two transformation procedures, A. and 8" XB, will not be consistent with this new weight transformation. To illustrate it, we randomly perturb A and B and compute the reconstruction loss Lquant and Corigin with without quantization, respectively:
Corigin = 1 (EC: (BTXx X.B) @ (CW,.GGT) Y
(11)

---

0.00
Racoretructuu Loua without Quantisation
Quantisatton
2.00
1.00
6.000
2.000
Reconstraction Less with Quantrzation
Recon Reconutrution Loss with Quantization
00.35
Original
(a) Random A,
(b) Random A, G
Figure 1: Reconstruction loss points obtained by randomly perturbing A, B (a) A, G (b). The bottom points indicate a minor loss without quantization. And the left points indicate a minor loss with quantization.
Lquant
(B"X,B) ""
(12)
The results are shown in Figure I(a] Although original A and B can produce desired results with floating-point GW G (the yellow triangle on the bottom), they
longer optimal when changing GWG to (GGWG") Concretely, about half of the random perturbed matrices lead to smaller errors (the blue dots on the upper left of the yellow triangle). A similar phenomenon also exists for input transformation, as shown in Figure I(b) These experiments indicate that the inconsistency between domain transformations caused by quantization renders the original matrices unsuitable for quantization scenarios, which is the primary reason why Winograd convolutions are difficult to quantize. order align these transformation procedures after quantization, we propose adjust transformation matrices via an optimization procedure as follows:
argmin Ex-D 2 IIA" (2. (EBQQ(B" X.B) o Q(GWW,.G"))A X/| A.B,G
(13)
Since the transformation procedures in the Winograd algorithm are adopted considering the effect of PTQ, we name this method as PTQ-Aware Winograd (PAW). By using the straight-through estimator 31 to approximate the gradient through the round function as pass-through operation, we can obtain the derivative of G (derivatives of A and are shown in supplemental materials) and update
using SGD:
0C
AL o Q(U.))WW), + o (( 80,
OL
C. C 1 DC
(14)
0G
80,
where
aL 00,
-
(15)
4.2
Fully-quantized Winograd convolution
4.2.1
Motivation
Winograd convolution involves sequence of computational steps from Eq. 11 to Eq. (4. To further improve computation efficiency via full quantization, all these operations need to be quantized except
Table 2: Exploration on 8-bit quantization of different components of Winograd convolution on 18.Net-18. Output tensor in the Winograd domain (O) is the primary cause of accuracy degeneration.
FP
A
B
X
U
V
TT
ATO
Accuracy 69.76
69.74
69.66
69.67
69.40
69.40
0.20
69.22
66.56
5

---

6.0
6444 #668.J8
itdo.33 05,5) idd3.43
300
055-51 18811.6
Table 3: The performance of different quantization
200
schemes for O. Although per-pixel quantization is ex- cellent maintaining model accuracy, it is not supported in hardware.
2200
100
500
360
-20
Layer2-1.coml
-30
10
-10
Lccerr
10
Granularity
Accuracy
Hard- hard-friendly
Quantize O (per-tensor)
0.216
Figure 2: The ranges of O vary widely between different pixels. In both layers, the std of O5.5 is ten times larger than that of O4.4-
Quantize O (per-pixel)
68.740
X
Quantize O (ours)
67.866
Eq. (2). which be done offline. However, empirically find that it is non-trivial to achieve full quantization because that accuracy will drop to zero due to the quantization of O (i.e., the output tensor in the Winograd domain), as shown in Table 2] After further investigation, that the ranges of O vary widely between different pixels. For example, as shown in Figure [ the standard deviation (std) of O distributed pixel (4,4) is times larger than that at pixel (5,5). A shared quantization scale for them is difficult to achieve both small round error and clamping error. possible solution
find
provide independent scales for them, i.c., use per-pixel quantization (Table 3). However, since O will take part in matrix multiplication (Eq. 4 instead of element- multiplication, per-pixel quantization will lead different scales in the summation dimension, which makes it not feasible in general hardware.
4.2.2 Factorized quantization
Motivated by the strong range difference, we present the following theorem, which shows the cause of this distribution characterization and proves the standard deviation of can be factorized into two vectors.
Proposition 1. Assume all elements of W E Rixxrr and X E CRCxxxax are independently and identically distributed variables with zero mean (e.g, ~x(, ~N(0, )) If denote
B =(bf ,0f,b7] b/. bb)) and G (of 992 ...,gh) we have:
Var Var[Oig] = Var [>c (B" GWG yul = 4,y
(16)
where u, VCaxow |b, 13 19 and vj VC 16,111199811
Proof. Firstly, since Bd X.B and GW.G" are the linear combinations of independent and identical variables with zero mean, we can calculate their mean and variance:
E [(B" X)B
and E [[GW Diel = O,
(17)
Var [
x. B)y = Var [b X.b,] = ||b |
(18)
Var [GWW.G) 1y] = Var (oTWe]) = | low
(19)
Then we can derive the variance of 0:
(a) >c Var [(B x. (GW.G"))l
Var [Ong]
bb) CCv [(B x.B) Var [GW,G")) = G.16b./P161111 11 1111
(20)
The equation (a) holds because the variables in different channels are independent. The equation (b) holds because the random variables and V are independent of each other and have zero mean.
After finding that the standard deviation of O can be decoupled along rows and columns separately, an intuitive thought is that the per-pixel scales may also satisfy similar rules. more theoretical proof is provided as Theorem

---

Diag (c)
(t) Diag(
Std
Std
Std
Const
Figure 3: An illustration of Factorized Scale Quantization. Since the standard deviation of O can be decoupled along rows and columns separately, using two vector scales, or and 8, is enough for balancing O a similar distribution.
Theorem 1. Assuming X NO, xx). W ~ ~N(0,ow) ow), the optimal per-pixel scales S which minimize quantization error Eq. (1) can be factorized as: a 1
argmin Ex-Nioxx)) -10--s(0)11?
(21)
Proof. In Proposition 1. we have proven that:
Var [Oty] VCaxow 119111 /Caxowllbb 11,,1
(22)
Observing that Oig is the summation C, independent variables, where C. the number of input channels and is usually large (e.g.. 128). According to the Central Limit Theorem (38), we can assume Og follows Gaussian distribution. In supplementary material, we will show that the optimal scale to minimize the mean-square error of quantizing Gaussian variables NNO0, proportional to a, i.e., Ko, where K is a constant. Thus, we have:
Sig /CCo / [KCill,
= a4. -.
(23)
The benefit of factorizing per-pixel scales into two vectors is that we can move both scales into transformation matrices. we can facilitate per-tensor matrix multiplication implementation when per-pixel quantization is utilized:
ATOA AT (So 2 σ 0)-4 AA
= -A .(aa 3²) Õ) A
(24)
AAŌAR
where:
A1 AT Diag(a), Ar = Diag(3) A and Ō 100 Sol
(25)
Note that at this stage, matrix A is quantized. After rescaling, we can further quantize these transformation matrices and middle results to obtain fully-quantized Winograd convolution. In this paper, this method is called Factorized Scale Quantization (FSQ) and illustration of it is shown in Figure
4.2.3 Optimization procedure
In this section, we focus on how to determine the optimal a and 3. Given samples {0,,02, .... ON3, we can minimize the quantization error via the following non-linear least square regression problems:
argminnZ 10-10 (a 87) o 0/12 a.3
(26)
7

---

The optimization of Eq. (26 is non-trivial due to the integer constraint of Ô. Thus we propose an iteration method solve
Solving o when fixed & and Ó. When 3 and Ô are fixed, 26) is quadratic function with respect to a. Then the closed-form solution for be easily obtained
EN 07))
(27)
(0) (
Solving When fixed and O. The clsed-form solution for ,3 is similar that of a:
1 (O7 3 O4) -a
8r ENSC. (O) o 07)² (a
(28)
Solving Ô when fixed and B. When and 8 are fixed, Ö is simply the integer values by rounding to the nearest:
07+ L0 0 (a" [9]
(29)
5 Experiments
5.1 Experimental settings
We full- full-precision pre-trained models and replace all 3x3 convolutions (stride 1) with Winograd convolutions. All convolutions, including the first and last layers, are quantized using symmetric quantization. Following BQW244, use per-pixel quantization for transformed inputs U and weights V. In order to verify the effectiveness of our method, we first construct a strong baseline on Post-training Quantization Winograd. Concretely, quantize U and V, respectively. Following BRECQ [19], we use 1024 unlabeled images and Adam 33) optimizer with 20k iterations and batch size of 32. Experiments show that strong baseline has surpassed previous state-of-the-ar PTQ Winograd work. Based on it, we further verify our method on the strong baseline. The learning rates of A, B, and G are set to (le-4, le-4, 5e-4) by default, and the reason will be shown in Section 5.4 All our experiments are conducted on NVIDIA GeForce RTX 3090 24GB GPU servers and last for several hours.
resort to Adaround [18] and LSQ [4] to
5.2 PTQ-Aware Winograd
In this section, we compare our PTQ-Aware Winograd method to the previous work BQW 24 with comprehensive experiments settings, including various bitwidths, tile sizes, datasets and models.
Table 4: PTQ results of ResNet-20 on CIFAR-10 with different tile sizes (4,6) and different bitwidths (4,6,8).
Winograd
Quantization Accuracy
Model
Bits
Algorithm
Strong Baseline
BQW 244
PAW
4
22.46
17.33
62.50 (+45.17)
F(4.3)
80.63
79.36
90.25 (+12.65)
ResNet-20
8
89.55
90.31
92.02 (+1.71)
(91.76%)
4
13.51
10.00
21.71 (+11.71)
F(6,3)
46.62
39.75
85.29 (+45.54)
81.44
81.44
91.10 (+9.66)
CIFAR-10. The results of different PTQ methods on ResNet-20 are shown in Table Although our strong baseline has been able to outperform previous method, our PTQ-Aware Winograd (PAW) can even improve it further, especially with lower bitwidth or larger tile size. For example, our methods surpass BQW 24 by 12.65 % and 45.54% when using 6-bit quantization on F(4,3) and F(6.3), respectively, which means our method makes deploying CNNs with more limited computation resources possible.

---

ImageNet. We also conduct experiments on ImageNet, more challenging image classification dataset than CIFAR-IO 10. In this experiment, replace the convolutions in the last block with F(4.3) Winograd convolution since F(6,3) Winograd convolution needs more padding for x 7 input size. Noting that BQW [24] only conducts PTQ with 8-bit because of severe accuracy degeneration on lower bitwidth, we compare our method with our strong baseline. The results presented in Table demonstrate two breakthroughs of our method. First, we are the first to achieve negligible accuracy drop with 8-bit post-training quantization Winograd convolution. Second, we are the first to achieve an acceptable accuracy drop with 6-bit post-training quantization on Winograd convolution.
5.3 Factorized Scale Quantization
In this section, we use Factorized Scale Quantization (FSQ) for O and per-pixel quantization for U and V. Other components are conducted with per-tensor quantization. We conduct experiments on ImageNet with bitwidths 6 and In 6-bit quantization, we use 6-bit to quantize U, V, and 8-bit to quantize other components. The results shown in Table 5 indicate that proposed FSQ the crucial step to fully quantizing Winograd convolutions, and it is well-compatible with PTQ- Aware Winograd (PAW). Even with uully-quatizatioon settings, methods can achieve comparable accuracy compared to BQW
Table 5: PTQ results of ResNets on ImageNet. Previous method BQW 24 does not quantize matrix multiplications in Winograd transformation. 'SB' indicates our Strong Baseline.
Partial Quantization
Full Quantization
Model
Tile
Bits
SB
BQW 24
PAW
PAW
FSQ
FSQ+PAW
50.86
N/A
65.15
0.26
44.21
64.34
F(4,3)
ResNet-18
8
68.08
67.54
69.06
0.21
64.46
68.16
(69.76%)
24.58
N/A
59.58
0.13
18.68
59.30
F(6.3)
8
65.84
60.09
68.36
0.13
58.48
66.89
6
58.85
N/A
69.71
0.88
52.10
68.80
F(4.3)
ResNer-34
8
72.11
71.86
72.67
0.96
69.51
71.75
(73.30%
27.91
N/A
63.56
0.19
22.06
62.11
F(6.3)
8
69.68
66.52
71.90
0.29
63.60
69.72
6
73.25
N/A
74.94
22.44
72.21
74.75
F(4.3)
ResNet-50
75.89
75.84
76.01
17.80
75.90
75.74
(76.15%)
6
69.20
N/A
73.99
5.97
66.85
73.84
F(6.3)
75.34
74.47
75.80
3.20
73.66
75.36
5.4 Ablation studies
To evaluate the effectiveness of our quantization-aware Winograd, conduct an ablation study on ResNet- 18 by optimizing different combinations of A, B, and G. Empirically, we find that the learning rate G more sensitive than those of A and B. Thus we fix the learning rates of and B to le-4 for all experiments and perform grid search on the learning rate of G separately, which is varied the interval [le-7, le-3). The best results are shown in Table When optimizing them, the optimal learning rate for G is 5e-4, and the performance improvement reaches 1.22%.
Table 6: Impact of different design choices for optimization of transformation matrices, on the ImageNet validation accuracy (%) for ResNet-18.
Optimized Matrices
None
B
G
A,
B
B, G
A,
A, B,B,C G
Accuracy
68.04
68.07
68.24
68.03
68.87
68.28
68.03
69.26

---

6
Conclusions
This paper focuses accelerating deep convolution neural networks by combining the Winograd algorithm and model quantization. We propose a unified optimization procedure to improve the compatibility between quantization and the Winograd algorithm. To achieve a more significant speed-up via full quantization, we propose a theoretically supported and hardware-friendlly method called Factorized-Scall Quantization, which is suitable for the distribution characteristies of tensors in the Winograd domain. Experiments are conducted to show that our method can boost performance by a large margin, especially with lower bitwidth or larger tile size, which means our method makes deploying CNNs with more limited computation resources possible.
7
Acknowledgements
This work supported in part by the STI 2030-Major Projects (No.2021200201504)) the NSFC- General Technology Collaborative Fund for Basic Research (No.U1936204), the Jiangsu Key Re- search and Development Plan No.BEE023006, No. 2002022220......
References
[1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016, pages 770-778, IEEE Computer Society, 2016.
[2] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Commun. ACM, 60(6):84-90, 2017.
[3] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 2980-2988. IEEE Computer Society, 2017.
[4] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 2472-248 Computer Vision Foundation IEBE Computer Society, 2018
[5] Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. 2016.
[6] Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE Trans. Pattern Anal. Mach. Intell., 38(10): 1955, 2016.
[7] Peisong Wang, Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training network quantization via bit-split and stitching. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Muchine Learning Reseurch, pages 9847-9856. PMLR, 2020.
[8] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew G. Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient aringer-rithmmttccoonyy inference. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages Computer Vision Foundation/ IEEE Computer Society, 2018
[9] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In 2020 IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 13166-13173 13175. Computer Vision Foundation / IEEE, 2020.
[10] Weihan Chen, Peisong Wang, and Jian Cheng. Towards mixed-precision quantization of neural networks via constrained optimization. In 2021 IEEF/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10- 17, 2021, pages 5330-5339. IEEE, 2021.
10

---

[11] Peisong Wang. Xiangyu He, Gang Li, Tianli Zhao, and Jian Cheng. -inssit--inucingg binarized neural networks. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Seeood Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 12192-12199. AAAI Press, 2020.
[12) Andrew Lavin and Scott Gray. Fast algorithms for convolutional neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 4013-4021. IEEE Computer Society, 2016.
[13] Michaël Mathieu, Mikael Henaff, and Yann LeCun. Fast training of convolutional networks through ffts. 2014.
[14] Steven K. Esser. Jeffrey L. McKinstry. Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S. Modha. Learned step size quantization. 2020.
[15] Huixia Li, Chenqian Yan, Shaohui Lin, Xiawu Zheng, Baochang Zhang, Fan Yang, and Rongrong Ji. PAMS: quantized super-resolution via parameterized max scale. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and LanMiichael Frahm, editors, Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV, volume 12370 of Lecture Notes in Computer Science, pages 580. Springer, 2020.
[16] Xiangyu He, Zitao Mo, Ke Cheng, Weixiang Xu, Qinghao Hu, Peisong Wang, Qingshan Liu, and Jian Cheng. Proxybnn: Learning binarized neural networks via proxy matrices. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020- 16th European Conference, Glasgow; UK, August 23-28, 2020, Proceedings, Part III, volume 12348 of Lecture Notes in Computer Science, pages 221-241. Springer, 2020.
[17] Weixiang Xu, Fanrong Li, Yingying Jiang, Yong A, Xiangyu He, Peisong Wang, and Jian Cheng. Improving extreme low-bit quantization with soft threshold. IEEE Trans. Circuits Syst. Video Technol., 33(4): 1549-1563, 2023.
[18] Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In Proceedings of the 37th International Conference on Machine Learning, ICML. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 7197-7206. PMLR, 2020.
[19] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. BRECQ: pushing the limit of post-training quantization by block reconstruction. 2021.
[20] Peisong Wang, Weihan Chen, Xiangyu He, Qiang Chen, Qingshan Liu, and Jian Cheng. Optimization-based post-training quantization with bit-split and stitching. IEEE Trans. Pattern Anal. Mach. Intell., 199-2135,, 2023.
[21] Markus Nagel. Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In 2019 IEEE/CVE International Conference on Computer Vision, ICCV 2019. Scoul, Korea (South). October 27 November 2, 2019, pages 1325-1334. IEBE, 2019.
[22] NVIDIA TensorRT. sttpps///dot www.nniiiia.o... www.aaa..o//pppppaarrao/////ooee
[23] Javier Frrmanddz-Marqqeee Paul N. Whatmough. Andrew Mundy, and Matthew Mattina. Search- ing for winograd- -aware aware quantized networks. 2020.
[24] Vladimir Chikin and Vladimir Kryzhanovskiy. Channel balancing for accurate quantization of winograd convolutions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 12497-12506. IEEE, 2022.
[25] Lingchuan Meng and John Brothers. Efficient winograd convolution via integer arithmetic. CORR, Abss/190.0199 2019.
[26] Renzo Andri, Beatrice Bussolino, Antonio Cipolletta, Lukas Cavigelli, and Zhe Wang. Going further with winograd convolutions: Tap-wise quantization for efficient inference on 4x4 tiles. In 55th IEFF/ACM International Symposium on Micrarchitecture, MICRO 2022, Chicago, II, USA, October 1-5, 2022, pages 582-598. IEEE, 2022.
11

---

[27] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pages 525-542 Springer, 2016.
[28] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. pages 3123-3131, 2015.
[29] Zeyu Zhu, Fanrong Li, Zitao Mo, Qinghao Hu, Gang Li, Zejian Liu, Xiaoyao Liang, and Jian Cheng. Srm a 2q Aggregation-avan quantization for graph neural networks. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[30] Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. HAQ: hardware-aware automated quantization with mixed precision. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 8612-8620. Computer Vision Foundation / IEEE, 2019.
[31] Yoshua Bengio. Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, 0obs//308.300 3332, 2013.
[32] Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. PACT: parameterized clipping activation for quantized neural networks. CoRR, abs/ MaW/180050608. 2018.
[33] Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. Qdrop: Randomly dropping quantization for extremely low-bit post-training quantization. 2022.
[34] Lingchuan Meng and John Brothers. Efficient winograd convolution via integer arithmetic. CoRR, htss/1901.0196. 2019.
[35] Guangli Li, Lei Liu, Xueying Wang. Xiu Ma, and Xiaobing Feng. Lance: efficient cow-precision quantized winograd convolution for neural networks based on graphics processing units. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020, pages 3842-3846. IEEE, 2020.
[36] Renzo Andri, Beatrice Bussolino, Antonio Cipolletta, Lukas Cavigelli, and Zhe Wang. Going further with winograd convolutions: Tap-wise quantization for efficient inference 4x4 tiles. In 55th IEEE/ACM International Symposium on Microarchitecture, MICRO 2022, Chicago, IL, USA, October 1-5, 2022. pages 582-598. IEEE, 2022.
[37] Shmuel Winograd. Arithmetic complexity of computations, volume 33. Siam, 1980.
[38] Robert V. Brill. Applied Statistics and Probability for Engineers, volume 46. 2004.
[39] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2015.
12

---

A Winograd transformation matrices
Depending on the particular choice of Winograd domain (i.e., polynomial domain), transformation matrices A, B, and the Winograd algorithm can be different. In the paper, we present that the most popular interpolation points for F(2,3) are [0, +1, -1 -]] and then these transformation matrices can be constructed follows:
T
-1
0
0
0
A
0 -1 -1
B
1
0
G
2
(30)
-1
0
2
0
0
-1
0
For F(4,3) and F(6,3), we choose the same transformation matrices as BQW Winograd transformation matrices are as follows:
[24]
For F(4,3), the
1
1
1
1
1
07
AT
2
2
0
(31)
0
1
4
4
0
8
-8
1
-5
0
-4
-4
D
(32)
-2
-1
2
-2
4
0
G
(33)
24
13
For F(6,3), the Winograd transformation matrices are as follows:
1
1
1
07
AT
1
2
2
0
(34)
0
4
4
0
0
8
-8
1
0
21
0
21
0
-1
37
1
-1
1
1
14 1144
17
BT
(35)
2
0
2
0
-2
4
0
-1
G
(36)
13

---

B Derivatives of transformation matrices
In the paper, in order to align these transformation procedures after quantization, we propose to adjust transformation matrices via an optimization procedure follows:
orgmin Ex-D A,B,G
E
O Q(GW) Yr|l"]
(37)
By using the straight-through estimator 31 to approximate the gradient through the round function as pass-through operation, we can obtain the derivatives of A, B and G. In this paper, we directly present the derivative of B. Here, a more comprehensive derivation is provided follows:
Oc aBy
0C 80,
= tr
(38)
000
0Bg
OC
Co tr
X.br,) [ ( (1xx)) @(Vval))
(39)
007
OL
807 [[B" Xcdes) [[B"Xx00s) awendl
0C
Q(V,,))+ + [ [00
(40)
d0g
xC xC
(bxiX.B)
Oc の @(Vr.a + (("r))
e Q(V
[0,
(41)
x DC=
OL X.B
20, (()
OL + +XT B ao,
Q(V
(42)
ij
ij
We have obtained the derivative of Bij, and now we can provide the expression for the derivative of B:
Oc
EC. x, OL 00, o Q(v,,)) x² B OL
80, Q(Vx.)))
(43)
dB
The derivatives of A, G and Oy can be computed in similar manner:
OC
C. 2
OJA(A"O;A- Yn) + + O;A-))
(44)
0A
OC
0L Q(U.))
åL 00,
Co
Da.
@ Q(U.))
(45)
0G
00,
OC -Y)
(46)
a0,
C Optimal quantization scale for Guassion varibles
Theorem I, in order to demonstrate that the optimal per-pixel scale can be factorized into vectors, we rely on the conclusion that the optimal scale s* to minimize the mean-square error of quantization of Gaussian variables 2 ~ (0, o²) is proportional to o, i.e s* Ko, where K a constant. Here, will provide a proof of it. (0, o²), the optimal scale minimize the mean-square error of quantization of is proportional to the standard deviation o, i.e., s" = Ko, where K is a constant.
Theorem 2. Assuming
Proof. Because NN00, o²), z can be reparameterized
u. where u (0.
14

---

E [Q(=) - - (=(=)(((=) - 2)2
(47)
Pu(u)(Q(ou) o du
(48)
Pu(u)(clip(
Qminn qmax)
ou) du
(49)
o²
Pu(u)(dipp
Qmin amaw) - 2)
(50)
8/0
(-)(-) 8
(51)
Eq. (47 can be treated function of s/o when solving for with known value. Assuming
K minimizes function (1xx), i.e., K = argmin 1(xr), we have:
x
argmin E [(Q(z) 2) = argmin o² = K
(52)
D Experiments on other architectures
In Section 5, we compare our methods to previous work BQW on the ResNet model family with comprehensive experiment settings, including various bit widths, tile sizes, and datasets. Here, we present a similar analysis for two other popular architectures VGG and Squeezenet using the Cifar- 10 dataset. The results are shown Table and Table These results align with our analysis in Section 5. Our PTQ-Aware Winograd (PAW) method outperforms the strong baseline introduced in Section and our FSQ method is well-compatille with PAW.
Table 1: PTQ results of VGG11 CIFAR-10.
Partial Quantization
Full Quantization
Model
Tile
Bits
Baseline
PAW
FSQ
FSQ+PAW
89.13
91.56
86.59
91.55
F(4.3)
VGG-11 (92.02%)
92.02
92.28
90.82
91.83
75.10
89.94
68.98
90.34
F(6.3)
91.27
91.88
88.44
91.63
Table 2: PTQ results SqueezeNet on CIFAR-10.
Partial Quantization
Full Quantization
Model
Tile
Bits
Baseline
PAW
FSQ
FSQ+PAW
6
89,69
91.98
88.66
91.78
F(4.3)
SqueezeNet
8
92.61
92.68
92.01
92.80
(92.62%)
6
80.50
90.67
76.48
91.26
F(6.3)
8
92.37
92.61
90.54
92.42
15


Reviews:
[{"review_id": "aohvg5dOX0", "reviewer": "Reviewer_Rvu6", "summary": "This paper presents a Post-training-quantization-aware Winograd (PAW) method to optimize all transformation procedures required by the Winograd algorithm to achieve post-training quantization of pre-trained ResNet models. A useful factorized scale quantization (FSQ) method is also proposed to balance the differences in the range of values in the Winograd domain. Obvious increases in terms of classification accuracy values on CIFAR-10 and ImageNet datasets are obtained with different ResNet-style model structures.", "questions": "1. From Table 4, the cases in which PAW resulted in the most considerable improvements are 6-bit and 8-bit for F(6,3). Could you please explain in what type of hardware, 6-bit could be a more efficient setting than 8-bit?\n\n2. From Table 5, F(4,3) results are often better than the F(6,3) results, which matches my understanding. In that sense, could you please explain why improving the performance with tile size 6 matters? Following that logic, could you please provide more evidence of the advantage of using a large tile size (>=4)?", "limitations": "Only ResNet-style model architectures are tested, which is limited. The authors also did not compare the results and efficiency against those with the standard tile size.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper achieved good improvements by solving problems observed in practice in cases with large tile sizes. It is useful to know the problems the authors observed and their solutions.", "weaknesses": "1. The presentation of the paper was not satisfactory and is quite difficult to follow. A lot of polishing work needs to be made before the paper could reach a state that I could recommend accepting. \n\n2. Some motivation for the work needs to be justified: why checking and solving the issue happened when quantizing ResNet-style models with large (>=4) tile sizes? Some justification and performance differences need to be made when comparing to those with smaller tile sizes.  \n\nMany minor issues are listed below:\n1. Many issues in the use of English:\na. \"Another approach to accelerate CNNs is faster implementations of convolution, such as FFT [11] and Winograd [10]. Among them,\"  ...\nb. \"for the first time. And we further propose a \" ... \nand many others (too many to list)...\n2. \"surpasses the previous state-of-the-art method by 8.27% and 5.38% on ResNet-18 and ResNet-34, respectively.\" -> Better to be clear surpass in terms of what?\n3. Different numbers of significant figures are used in Table 2.\n4. Abbreviation std is not defined in the paper.\n5. \"does not require retraining the network end-to-end.\" -> It seems the authors may mean both updating the entire model and training from scratch by using the wording \"end-to-end\" here. Many ambiguities like this exist throughout the paper. \n6. Would be more friendly to the readers to define tile size in the paper (I guess you mean kernel size or filter size here). Could also provide the definitions of F(4,3) and F(6,3) before using them. \n7. Many issues in the references. For instance, conference references [7], [19], and [30] are in different formats. Some references, such as [33], are not cited in their official publication sources. \n8. In table 5, perhaps better to use the wording \"baseline\" than \"strong baseline\". We believe all baseline systems authors try to quote are strong enough. \n9. Since non-standard tile sizes are used in this paper, it may be better to explain more carefully about the model architectures rather than simply referring to them using the standard terms (ResNet-34 etc.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful reading of our article and providing helpful feedback. We have scrutinized the manuscript and made corresponding modifications, e.g., correcting some issues in tables and references. We have also polished the paper per your recommendations and explained some specific terms (e.g., tile size) more clearly. **We apologize for the confusion between tile size and filter size.** The final version would be more reader-friendly, even for those unfamiliar with the Winograd algorithm. **We also analyze the relationship between speedup ratio and tile sizes and formally demonstrate the necessity of adopting large tile sizes to accelerate CNNs.** \n\n**Q1:** Some motivation for the work needs to be justified: why checking and solving the issue happened when quantizing ResNet-style models with large (>=4) tile sizes?\n\n**A1:**   We apologize for the confusion. The tile size $m$ in Winograd convolution differs from the filter size $r$ in standard convolution. **For $r \\times r$ standard convolutions with filter size $r$, the algorithm transforms the convolution operations to the Winograd domain and generates $m \\times m$ (spatial) outputs at a time, which is denoted as $F(m,r)$.** The parameter m is called tile size, which is used to balance the speedup and numerical precision. \n\nCompared with standard convolution, the Winograd algorithm requires fewer multiplications. The speedup ratio depends on tile size $m$. For a $m \\times m$ output patch, $F(m,r)$ Winograd convolution requires $C_{in}C_{out}(m+r-1)^2+2C_{in}(m+r-1)^2+2C_{out}(m+r-1)^2m$ multiplications. In contrast, standard convolution requires $C_{in}C_{out}(mr)^2$ multiplications. Usually, filter numbers $C_{out}$ and channel numbers $C_{in}$ are much larger than $m$ and $r$ in modern CNNs, and the speedup ratio is about $\\frac{(mr)^2}{(m+r-1)^2}$. Therefore, **the larger the tile size $m$, the more significant the speedup becomes (Table R1-1).**\n\nHowever, Winograd convolution suffers from numeric accuracy issues: the floating point (FP) error increases **exponentially** with the tile size [1] (Table R1-1). Therefore, considering the trade-off between accuracy and efficiency, most papers [5,7] focus on F(4,3) and F(6,3).\n\n **Table R1-1 FP Error and Speedup** \n\n| |2|4|6|8|10|\n|-|:-:|-|-|-|-|\n|ResNet-20 FP error| --  |$1.19×10^{−3}$|$3.23×10^{−3}$|$7.46× 10^{−3}$|$7.88×10^{−2}$|\n|SqueezeNet FP error| --- |$7.31×10^{-5}$|$1.32×10^{−4}$|$2.17×10^{−4}$|$1.25×10^{−3}$|\n|Speedup|2.25|4|5.0625|5.76 |6.25|\n\n**Q2:** Many issues in the use of English: a. \"Another approach to accelerate CNNs is faster implementations of convolution, such as FFT [11] and Winograd [10]. Among them,\" ... b. \"for the first time. And we further propose a \" \n\n**A2:** We have addressed the issues you highlighted as follows:\n\n(a) \"An alternative method for enhancing the speed of CNNs involves the development of faster convolution implementations\n\n(b) \"...for the first time. We further propose a...\"\n\n(c) \"...our proposed method outperforms ... in terms of the **top-1 accuracy** \".\n\n(d) \" the standard deviation (**std**) of O distributed....\"\n\n**Q3:** \"Post-training quantization is a more lightweight method that does not require retraining the network end-to-end\" -> It seems the authors may mean both updating the entire model and training from scratch by using the wording \"end-to-end\" here. \n\n**A3:** You are right. **QAT updates the entire model with end-to-end training, while PTQ only needs to optimize quantization parameters layer by layer.** Due to page constraints, we use the term \"end-to-end\" to convey this distinction in the related works section. Many other papers use the exact phrase to distinguish between PTQ and QAT, e.g., \"A White Paper on Neural Network Quantization\"[2].  In the final revision, we have modified the sentence as the bolded sentence. \n\n**Q4:** Since non-standard tile sizes are used in this paper, it may be better to explain more carefully about the model architectures rather than simply referring to them using the standard terms (ResNet-34 etc.).\n\n**A4:** As introduced in A2, the tile size is a parameter of the Winograd algorithm. The model architectures (e.g., filter size and input/output sizes of each layer) remain unchanged. \n\n**Q5:** Could you please explain in what type of hardware, 6-bit could be a more efficient setting than 8-bit?\n\n**A5:**  Although current general-purpose GPUs may not fully support 6-bit operations, many domain-specific accelerators can support precision-scalable computation. For example, Stripes[3] and BitFusion[4], both built on the bit-serial approach, can at run-time tune to the desired precision mode (e.g., 6-bit). In this type of hardware, the advantages of utilizing 6-bit computation over 8-bit computation are prominent and can be categorized into two key aspects:\n\n1. **Reduced Storage Access and Energy Consumption:** \n2. **Enhanced Computation Latency in Bit-Serial Scenarios:** In bit-serial architectures, 6-bit computation exhibits lower latency than 8-bit computation, achieving an acceleration effect of 1.3x [3].\n\n**Q6:** Only ResNet-style model architectures are tested, which is limited. \n\n**A6:** Because of the length limitation, we answer this question in the **global rebuttal.** \n\n[1] Alam et al., \"Winograd Convolution for Deep Neural Networks : Efficient Point Selection\", 2022\n\n[2] Nagel et al., \"A White Paper on Neural Network Quantization\", 2021\n\n[3] Judd et al., \"Stripes**:** Bit-serial deep neural network computing\", 2016\n\n[4] Sharma et al., \"Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Network\", 2018\n\n[5] Chikin et al., \"Channel Balancing for Accurate Quantization of Winograd Convolutions\", 2022\n\n[6] Li et al., \"BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction\", 2021\n\n[7]  Fernández-Marqués et al., \"Searching for Winograd-aware Quantized Networks\", 2020"}, {"review_id": "RQL9B5bbmG", "reviewer": "Reviewer_JmCw", "summary": "This paper proposes a post-training quantization algorithm for Winograd convolution, which overcomes the inconsistency in domain transformation by adjusting the transformation matrices together (PTQ-Aware) via a unified optimization procedure, and achieves full quantization by a new factorized scale quantization (FSQ) method. Experiments on CIFAR-10 and ImageNet show the effectiveness of the proposed algorithm. It surpasses the previous state-of-the-art Winograd post-training quantization algorithm significantly in terms of accuracy.\n ", "questions": "* Could you briefly compare your proposed algorithm with some QAT methods that also achieve full-quantization (e.g. the two papers I listed in the Weakness section) and summarize the advantage of your algorithm?\n* Could you provide some evaluation on the inference speed of the proposed algorithm compared with previous algorithms? If not possible, some simple analysis is also OK. I'd like to know whether there's some hidden computation overhead or complexity introduced by your algorithm. \n", "limitations": "I hope the authors can better differentiate their method with previous Winograd quantization algorithms and justify their method's advantages. If it simply adapts QAT full-quantization to the PTQ scenario, the novelty and contribution of the paper would be somewhat undermined.  ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The proposed Winograd post-training quantization algorithm is considerably more accurate than previous methods, and is hardware friendly.\n* The paper is well presented. I am able to understand the whole paper as a reader whose research focus is not on model quantization.  ", "weaknesses": "* The paper failed to mention or compare with some quantization-aware training (QAT) algorithms that achieve full-quantization, e.g \"Searching for Winograd-aware Quantized Networks\" [1] and \"Going Further With Winograd Convolutions: Tap-Wise Quantization for Efficient Inference on 4x4 Tiles\" [2]. These QAT algorithms share quite similar ideas with this paper and the accuracy of their final quantized models are much higher.  \n  * [1] https://arxiv.org/abs/2002.10711  (2020)\n  * [2] https://arxiv.org/abs/2209.12982 (2022)\n* If possible, some quantitive evaluation on how faster the proposed algorithm is (compared to previous algorithms) is desired.\n* Two typos: \n  * line 188: v_i -> v_j\n  * line 239: long -> learn\n\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Could you briefly compare your proposed algorithm with some QAT methods that also achieve full-quantization (e.g. the two papers I listed in the Weakness section) and summarize the advantage of your algorithm?\n\n**A1**: Thank you for your suggestion. This response includes three parts: **the difference between QAT and PTQ**, **a comparison with QAT algorithms**, and a **results comparison**.\n\n**In the quantization research community, QAT and PTQ are two distinct research areas.** Although QAT can achieve more promising performance, training the network requires great many GPU resources and a full training dataset. Therefore, it is not always practical when either the training dataset is unavailable (e.g., privacy and commercial training data) or rapid deployment is required. Therefore, much effort has been spent in the industry and research community toward improving PTQ performance. \n\n**The drawbacks of QAT will be amplified when it is applied to the Winograd algorithm.** Training large Winograd models can be slow and use a significant amount of memory. This is a direct consequence of implementing all the stages involved in the Winograd convolution and retaining the inputs to each operation in memory to backpropagate through the entire process. According to the open source code of [1], the Winograd-aware QAT method costs about 10 hours on ResNet18 using the ImageNet dataset with **4 GPUs** (NVIDIA RTX 3090) for **one training epoch**. In contrast, **the whole optimization procedure** of the proposed PTQ method can be accomplished in around 5 hours on **a single GPU**\n\n\"Searching for Winograd-aware Quantized Networks\" [1]  and \"Going Further With Winograd Convolutions: Tap-Wise Quantization for Efficient Inference on 4x4 Tiles\" are two classical papers about QAT for Winograd convolutions. **Compared to them,  we have the following difference and advantages**:\n\n-   [1] quantizes all the transformation matrices $A$,$B$, and $G$, but the intermediate results such as $B^TX$ and $A^TO$ are not quantized. So [1] needs higher precision to hold these results and carry out the next operation.  In addition, their methods, as mentioned above, require a full dataset and huge training resources, which are sometimes unavailable.\n-   [2] proposes tap-wise quantization for $U$ and $V$ which is used in many following works. They don't re-quantize the high-precision immediate results $O$ and utilize the shift-and-add operations to accelerate the computation of $A^TOA$.\n\n**Moreover, the strong effectiveness of retraining the whole network hides the difficulty of full quantization, especially for quantizing output tensors in the Winograd domain $O$**.  According to our analysis in Section 4.2.1,  $O$ suffers from huge distribution differences among taps and deserves per-pixel quantization (or called tap-wise quantization in [2]). However, since O will take part in matrix multiplication (Eq. 4) instead of element-wise multiplication, per-pixel quantization will lead to different scales in the summation dimension, which makes it not feasible in general hardware. Thus, we proposed a more hardware-friendly method FSQ, which factorizes per-pixel scales into two vectors and we can move both scales into transformation matrices. We also **theoretically** demonstrate that, under the assumption of the identical and independent distribution of weights and activations, our method is equally optimal to per-pixel quantization in minimizing quantization errors.\n\nIt's also worth noting that although our methods only require few-shot, unlabeled calibrations, and fewer computation resources, **we can achieve comparable performance with previous QAT works.** The difference and the results of ResNet-20 using Cifar-10 dataset are shown in Table R2-1.\n\n**Table R2-1 Comparision with Other Methods**\n\n|                   | Quantization Type | Optimziation of Transformation Matrices             | Full Quantization                                       | Quantiation  graunlarity of $O$ | Acc drop    |\n| ----------------- | ----------------- | --------------------------------------------------- | ------------------------------------------------------- | ------------------------------- | ----------- |\n| BQW[3]            | QAT/PTQ           | Fixed                                               | Not quantize  Winograd transformation                   | Not Quantize                    | -0.02/-1.42 |\n| Winograd-Aware[1] | QAT               | End-to-end Training(Multiple GPUs and full dataset) | Not re-quantize the immediate results $A^TO$ and $B^TX$ | Per-tensor                      | -0.7        |\n| Tap-wise quant[2] | QAT               | Fixed                                               | Not re-quantize the immediate results $O$               | Not Quantize                    | -0.6        |\n| Ours              | PTQ               | Reconstruction Loss (Single GPU and unlabeled data) | Quantize all components                                 | FSQ                             | -0.47       |\n\n**Q2:** If possible, some quantitive evaluation on how faster the proposed algorithm  (compared to previous algorithms) is desired.\n\n**A2:** Because of the length limitation, we answer this question in the **global rebuttal**. \n\n**Q3:** Two typos: line 188: $v_i$ -> $v_j$ line 239: long -> learn\n\n**A3:** Thank you for your careful reading of the paper. These issues will be corrected in the updated manuscript.\n\n[1] Fernández-Marqués et al., \"Searching for Winograd-aware Quantized Networks\", 2020\n\n[2] Andir et al., \"Going Further With Winograd Convolutions:Tap-Wise Quantization for Efficient Inference on 4x4 Tiles\", 2022\n\n[3] Chikin et al.,  \"Channel Balancing for Accurate Quantization of Winograd Convolutions\", 2022\n\n"}, {"review_id": "mXZvgk2zfz", "reviewer": "Reviewer_2v14", "summary": "The paper proposes a PTQ-Aware Winograd (PAW) method to improve the performance of deep learning inference with quantized parameters and Winograd Convolution. In particular, all steps of the Winograd operation are combined and optimized with a unified objective to reduce the domino effect of quantization in different parts of the Winograd sequence. Moreover, to mitigate the range difference of the Winograd output transformation, factorized-scale quantization is proposed. This involves balancing the distribution of the output transformation using two factorized scaling parameters. This approach outperforms previous work and the cifar-10 dataset.", "questions": "Most of the proofs in this paper assume the parameters follow a Gaussian distribution. Previous studies have shown that the distribution of parameters is similar to Laplace distribution. Does the proof in the paper also hold valid with a near-Laplace distribution of parameters?", "limitations": "The author does not discuss whether the new approach works for size 6 and size 8 of the Winograd operation.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The analysis of quantization for all parts of the Winograd operation is novel and interesting. The paper is well-written, and the proofs are explained well.", "weaknesses": "1- The overhead of proposed quantization algorithm needs to be compared with the QAT approach, since both methods perform training during the quantization process.\n\n2- The new approach has only been tested on the ResNet model. It is suggested that experimental results for other networks be added to the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** The overhead of proposed quantization algorithm needs to be compared with the QAT approach, since both methods perform training during the quantization process.\n\n**A1:**  Thank you for your suggestion. QAT methods require much more GPU resources and training data than PTQ. **When applying it to the Winograd algorithm,  these drawbacks will be amplified.** Training large Winograd models can be slow and use a significant amount of memory. This is a direct consequence of implementing all the stages involved in the Winograd convolution and retaining the inputs to each operation in memory to backpropagate through the entire process. According to the open source code of [1], the Winograd-aware QAT method costs about 10 hours on ResNet18 using the ImageNet dataset with **4 GPUs** (NVIDIA RTX 3090) for **one training epoch**. In contrast, the whole optimization procedure of the proposed PTQ method can be accomplished in around 5 hours on **a single GPU**.\n\n**The optimization procedures (13) and (26) happen before inference.**  During the inference process, as introduced in Section 4.2.2, the benefit of our proposed FSQ is that we can move scales $\\alpha$ and $\\beta$ into transformation matrices. So we can facilitate per-tensor matrix multiplication implementation when per-pixel quantization is utilized. The overhead of our proposed algorithm only includes re-quantization operation, which is negligible compared to the original Winograd transformation.\n\n**Q2:** The new approach has only been tested on the ResNet model. It is suggested that experimental results for other networks be added to the paper.\n\n**A2:**  Thank you for your suggestion. In the paper, we show the results of ResNet models to compare our method with another PTQ work [2]. Here, we add experiments on VGG and SqueezeNet using the Cifar-10 dataset. The results align with our expectations. The detailed results are presented in Table R3-1 and Table R3-2.\n\n**Table R3-1. Accuracy (\\%) on VGG11 (92.02%).**\n\n| Tile Size | BRECQ[4] | PAW   | FSQ   | FSQ+PAW |\n| --------- | -------- | ----- | ----- | ------- |\n| F4        | 89.13    | 91.56 | 86.59 | 91.55   |\n|           | 92.02    | 92.28 | 90.82 | 91.83   |\n| F6        | 75.10    | 89.94 | 68.98 | 90.34   |\n|           | 91.27    | 91.88 | 88.44 | 91.63   |\n\n**Table R3-2. Accuracy (\\%) on SqueezeNet (92.62%).**\n\n| Tile Size | BRECQ[4] | PAW   | FSQ   | FSQ+PAW |\n| --------- | -------- | ----- | ----- | ------- |\n| F4        | 89.69    | 91.98 | 88.66 | 91.78   |\n|           | 92.61    | 92.68 | 92.01 | 92.80   |\n| F6        | 80.50    | 90.67 | 76.48 | 91.26   |\n|           | 92.37    | 92.61 | 90.54 | 92.42   |\n\n**Q3:** Most of the proofs in this paper assume the parameters follow a Gaussian distribution. Previous studies have shown that the distribution of parameters is similar to Laplace distribution. Does the proof in the paper also hold valid with a near-Laplace distribution of parameters?\n\n**A3:**  **Yes, our proof is also valid for the near-Laplace distribution of parameters**. Actually, in Proposition 1, we only use the condition that $X$ and $W$ are zero-mean independent and identically distributed to calculate the mean and standard deviation of $O$. Therefore, the assumption that the parameters follow a Gaussian distribution is not necessary, and the proof also applies to the case of a Laplace distribution with mean zero (even other zero-mean distributions). We have modified the theorem in the revision.\n\n**Q4:** The author does not discuss whether the new approach works for size 6 and size 8 of the Winograd operation.\n\n**A4:** Thank you for your suggestion. **The experiments on tile size 6 have been shown in Table 4 and Table 5.** We don't expand our methods for tile size >=8 in the paper because of two reasons:  On the one hand, The speedup ratio of Winograd convolution,  represented by $\\frac{(mr)^2}{(m+r-1)^2}$, increases more slowly as the tile size becomes larger. On the other hand, Winograd convolution suffers from numeric accuracy issues: the floating point (FP) error increases exponentially with the tile size [4]. According to [4], the normalized L1 norm of floating point error for various deep convolution neural networks is shown in Table R3-3. Therefore, considering the trade-off between accuracy and efficiency, most papers [2,3] focus on F(4,3) and F(6,3).\n\n **Table R3-3 Numerical Error and Speedup** \n\n|                     |   2    | 4                     | 6                | 8                | 10               |\n| ------------------- | :----: | --------------------- | ---------------- | ---------------- | ---------------- |\n| ResNet-20 FP error  | -----  | $1.19 \\times 10^{−3}$ | $3.23 × 10^{−3}$ | $7.46 × 10^{−3}$ | $7.88 × 10^{−2}$ |\n| SqueezeNet FP error | ------ | $7.31\\times 10^{-5}$  | $1.32 × 10^{−4}$ | $2.17 × 10^{−4}$ | $1.25 × 10^{−3}$ |\n| Speedup             |  2.25  | 4                     | 5.0625           | 5.76             | 6.25             |\n\n[1] Fernández-Marqués et al,.\"Searching for Winograd-aware Quantized Networks\", 2020\n\n[2] Chikin et al.,  \"Channel Balancing for Accurate Quantization of Winograd Convolutions\", 2022\n\n[3] Li et al., \"BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction\", 2021\n\n[4] Alam et al., \"Winograd Convolution for Deep Neural Networks : Efficient Point Selection\", 2022"}, {"review_id": "VNVa4CKbQa", "reviewer": "Reviewer_BWC3", "summary": "This paper proposes PTQ-Aware Winograd (PAW), Factorized-scale quantization and a iterative optimization algorithm to solve the problem of quantization on Winograd domain. These methods not only fully quantize the whole Winograd Convolution, but also surpass the existing Winograd quantization methods in terms of effect and hardware friendliness.", "questions": "See Weakness. ", "limitations": "See Weakness. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This is the first work to perform full-quantization for Winograd Convolution\n\n2. Factorized-scalequantization makes the quantization for Winograd more hardware-friendly compared to per-pixel quantization in previous work\n\n3. This paper focuses on the properties of the Winograd domain that make quantification difficult and gives a full analysis\n\n4. The effect of the method is obviously beyond the existing methods, and the theoretical derivation is complete and the experiment is fully credible\n\n", "weaknesses": "1. In the explanation in Section 4.1:\n\n   - Is the motivation of this section: In the original Winograd method, A, B and G satisfy some strict mathematical relationship with each other, but the perturbation brought by quantization destroys the strict relationship between them, and the small perturbation brought by such quantization will bring much greater overall loss. Therefore, the authors simulated the perturbation by artificially adding the perturbation and tested the Reconstruction Loss?\n\n   - How was the range of perturbation added determined, and was the actual quantization error measured and used as a reference? Have you tested how much the Reconstruction Loss will decrease after optimization?\n\n2. This work achieves an obvious improvement over previous work, but what are the challenges if we continue to expand the tile size? Why?\n\nI would like to hear the authors’ feedback during rebuttal, and fixing these issues would further improve the quality of this paper.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** In the explanation in Section 4.1:\n\nIs the motivation of this section: In the original Winograd method, A, B, and G satisfy some strict mathematical relationship with each other, but the perturbation brought by quantization destroys the strict relationship between them, and the small perturbation brought by such quantization will bring much greater overall loss. Therefore, the authors simulated the perturbation by artificially adding the perturbation and tested the Reconstruction Loss?\n\nHow was the range of perturbation added determined, and was the actual quantization error measured and used as a reference? Have you tested how much the Reconstruction Loss will decrease after optimization?\n\n**A1:** (a) Your understanding of the motivation is right, but  your analysis is a little different from ours.\n\nIn the original Winograd method, $A$, $B$, and $G$ satisfy some strict mathematical relationships with each other to ensure these Winograd transformations happen in the same Winograd domain. However, quantization noise destroys the relationship between them. For example, the weight transformation after quantization becomes $Q(GWG^T)$ , which may mismatch with original transformations $AOA^T$ and $B^TXB$.\n\nTo demonstrate this, we change transformation matrices $A$ and $B$ towards different directions (by adding random noise) and test the reconstruction loss using $Q(GWG^T)$ and $GWG^T$, respectively. In Figure 1,  we discover that although original A and B can produce desired results with floating-point $GWGT$, they are no longer optimal when changing $GWG^T$ to $Q(GWG^T)$. About half of the random matrices A and B lead to more minor errors (the blue dots on the upper left of the yellow triangle). Thus, it is necessary to align these transformation procedures after quantization.\n\n(b) We test the reconstruction loss decreasing before and after optimization. Typically, the reconstruction loss will decrease to about 1/3 of the original loss. We show the results of different layers on ResNet-18 in Table R4-1.\n\n**Table R4-1. Reconstruction loss **\n\n|        | layer1.0.conv2 | layer2.0.conv2 | layer3.0.conv2 | layer4.0.conv2 |\n| ------ | -------------- | -------------- | -------------- | -------------- |\n| Before | $7.21e-02$     | $1.56e-01$     | $4.71e-01$     | $2.65e-01$     |\n| After  | $1.73e-02$     | $6.17e-02$     | $1.27e-01$     | $6.92e-02$     |\n\n**Q2:** This work achieves an obvious improvement over previous work, but what are the challenges if we continue to expand the tile size? Why?\n\n**A2:** We don't expand our methods for tile size >=8 in the paper because of two reasons: \n\nOn the one hand, The speedup ratio of Winograd convolution,  represented by $\\frac{(mr)^2}{(m+r-1)^2}$, increases more slowly as the tile size becomes larger. As an illustration, the speedup ratios for $F(2,3)$, $F(4,3)$, $F(6,3)$, and $F(8,3)$ are 2.25, 4, 5.0625, and 5.76, respectively (Table R4-2). On the other hand, Winograd convolution suffers from numeric accuracy issues: the floating point (FP) error increases exponentially with the tile size [1]. According to [1], the normalized L1 norm of floating point error for various deep convolution neural networks is shown in Table R4-2. Therefore, considering the trade-off between accuracy and efficiency, most papers [2,3,4] focus on F(4,3) and F(6,3).\n\n **Table R4-2 Numerical Error and Speedup** \n\n|                     |   2    | 4                     | 6                | 8                | 10               |\n| ------------------- | :----: | --------------------- | ---------------- | ---------------- | ---------------- |\n| ResNet-20 FP error  | -----  | $1.19 \\times 10^{−3}$ | $3.23 × 10^{−3}$ | $7.46 × 10^{−3}$ | $7.88 × 10^{−2}$ |\n| SqueezeNet FP error | ------ | $7.31\\times 10^{-5}$  | $1.32 × 10^{−4}$ | $2.17 × 10^{−4}$ | $1.25 × 10^{−3}$ |\n| Speedup             |  2.25  | 4                     | 5.0625           | 5.76             | 6.25             |\n\n[1] Alam et al., \"Winograd Convolution for Deep Neural Networks : Efficient Point Selection\", 2022\n\n[2] Chikin et al.,  \"Channel Balancing for Accurate Quantization of Winograd Convolutions\", 2022\n\n[3] Andir et al., \"Going Further With Winograd Convolutions:Tap-Wise Quantization for Efficient Inference on 4x4 Tiles\", 2022\n\n[4] Fernández-Marqués et al., \"Searching for Winograd-aware Quantized Networks\", 2020"}, {"review_id": "ym1x1As8Uz", "reviewer": "Reviewer_M3Gq", "summary": "This paper proposed to fully quantize the Winograd convolution post-training under the observation of disruption of consistency between different transformation procedures, the new proposed Factorized Scale Quantization is suitable in the Winograd domain. The experiments demonstrate significant improvements compared with previous post-training-quantization methods.", "questions": "This paper emphasizes the hardware-friendly deployment of this proposed quantization method, will the optimization procedure including solving $\\alpha$, $\\beta$, and $\\tilde{O}$ cost much more additional resources?", "limitations": "This paper hasn't mentioned or discussed its limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is organized well, the exploration of quantization of different components of Winograd convolution clearly shows the primary cause of accuracy degeneration, and the following proposed method precisely targets this problem.\n2. The proposed method is interesting, which utilizes two factor vectors to replace the per-pixel scales, and these two vectors can be merged into the transformation matrices.\n3. Some experimental results are even higher than the baseline.\n", "weaknesses": "The experiments only demonstrate the quantization bit, there is no computation cost or inference time comparison which are very important to this work, it is difficult to know how much the proposed improve the efficiency other than accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** The experiments only demonstrate the quantization bit, there is no computation cost or inference time comparison which are very important to this work, it is difficult to know how much the proposed improve the efficiency other than accuracy.\n\n**A1:** Thank you for your suggestion.  Because of the time limitation, we can't implement  Winograd convolution on GPUs or FPGAs, necessitating low-level optimizations. Instead, we opt for BOPs as an alternative metric for measuring computation cost. This metric is widely used in various fields such as Neural Architecture Search (NAS), pruning, and quantization research(\\[4,5,6])\n\nBOPs is defined as $BOPs=b_1 \\cdot b_2 \\cdot MAC$, where $b_1$ and $b_2$ represent the bit-width of two operators, respectively. **Please notes that the optimization procedures (13) and (26) happen before inference.** During the inference process, the computation cost of quantized Winograd convolution comprises three components: **element-wise multiplications** $U\\odot V$, **Winograd transformations ($BXB^T$ and $AOA^T$)**, and **quantization overhead**. \n\nFor element-wise multiplication, like [1, 3], our methods use the per-pixel quantization, with the memory overhead of  $(m+r-1)*(m+r-1)$ to store the quantization scales and $N\\times C_{out}\\times (m+r-1)\\times (m+r-1)+ C_{in}\\times C_{out}\\times (m+r-1)\\times(m+r-1)$ times flops to re-quantize $U$ and $V$.\n\nFor Winograd transformations ($BXB^T$ and $AOA^T$), as introduced in Section 4.2.2, the benefit of our proposed FSQ is that we can move scales $\\alpha$ and $\\beta$ into transformation matrices. **So we can facilitate per-tensor matrix multiplication implementation when per-pixel quantization is utilized**. Therefore, quantizing $X$, $B^TX$, $O$, and $A^TO$ results in a constant memory overhead and a computation overhead of $2 \\times N \\times (C_{out} + C_{in}) \\times (m + r - 1) \\times (m + r - 1)$. Because these quantization and re-quantization operations need the same times flops as the tensor size, the overhead is negligible compared to Winograd transformations which involve twice matrix multiplications.\n\nThe BOPs of different methods to quantize F(6,3) ResNet-20 are shown in Table R5-1. [1] quantizes all the transformation matrices $A$,$B$, and $G$, but the intermediate results such as $B^TX$ and $A^TO$ are not quantized. So [1] needs higher precision to hold these results and carry out the next operation. [2] don't quantize these Winograd transformations to maintain accuracy. Compared to them, our full-quantization of Winograd will achieve less computation cost.\n\n**Table R5-1 BOPs**\n\n|                    | Im2col(FP) | Winograd(FP) | BQW[2]  | Winograd-AwareNet[1] | Ours(PAW) | Ours(PAW+FSQ) |\n| ------------------ | ---------- | ------------ | ------- | -------------------- | --------- | ------------- |\n| Low-precision BOPs | 0          | 0            | 464.56M | 1282.45M             | 464.56M   | 791.71M       |\n| Flops              | 40.81M     | 12.37M       | 5.27M   | 0.48M                | 5.27M     | 0.80M         |\n| Total BOPs         | 10.44G     | 3.16G        | 1.81G   | 1.41G                | 1.81G     | 0.99G         |\n\n**Q2:** This paper emphasizes the hardware-friendly deployment of this proposed quantization method, will the optimization procedure including solving $\\alpha$, $\\beta$, and $O$ cost much more additional resources?\n\n**A2:** The overhead of the optimization procedure to solve $\\alpha$, $\\beta$, and $O$ is negligible. There are two reasons: (1) The optimization procedure (13) and (26) happens before inference.  (2) According to our experiments, this optimization procedure (26) converges fast. In most cases, it will converge in several to dozens of iterations, which takes approximately several minutes for one layer.\n\n\n\n[1] Fernández-Marqués et al., \"Searching for Winograd-aware Quantized Networks\", 2020\n\n[2] Chikin et al.,  \"Channel Balancing for Accurate Quantization of Winograd Convolutions\", 2022\n\n[3] Andir et al., \"Going Further With Winograd Convolutions:Tap-Wise Quantization for Efficient Inference on 4x4 Tiles\", 2022\n\n[4] Wang et al.,  \"Differentiable Joint Pruning and Quantization for Hardware Efficiency\", 2020\n\n[5] Guo et al.,  \"Single path oneshot neural architecture search with uniform sampling\", 2020\n\n[6] Liu et al.,  \"Towards precise binary neural network with generalized activation functions\", 2020"}]

Author rebuttal:
Thank you reviewers for your helpful feedback and constructive advice. Based on the reviewers' questions, comments, and recommendations, we have made many revisions that may significantly improve the quality of the paper.

**Here, we explain the common concern of reviewers on the computation cost of our algorithm.**
 Because of the time limitation, we can't implement  Winograd convolution on GPUs or FPGAs, necessitating low-level optimizations. Instead, we opt for **BOPs** as an alternative metric for measuring computation cost. This metric is widely used in various fields such as Neural Architecture Search (NAS), pruning, and quantization research(\[1,2,3])

BOPs is defined as $BOPs=b_1 \cdot b_2 \cdot MAC$, where $b_1$ and $b_2$ represent the bit-width of two operators, respectively. **Please notes that the optimization procedures (13) and (26) happen before inference.** During the inference process, the computation cost of quantized Winograd convolution comprises three components: **element-wise multiplications** $U\odot V$, **Winograd transformations ($BXB^T$ and $AOA^T$)**, and **quantization overhead**. 

For **element-wise multiplication**, like [1, 3], our methods use the per-pixel quantization, with the memory overhead of  $(m+r-1)*(m+r-1)$ to store the quantization scales and $N \times C_{out}\times (m+r-1)\times (m+r-1)+ C_{in}\times C_{out}\times (m+r-1)\times(m+r-1)$ times flops to re-quantize $U$ and $V$.

For **Winograd transformations ($BXB^T$ and $AOA^T$)**, as introduced in Section 4.2.2, the benefit of our proposed FSQ is that we can move scales $\alpha$ and $\beta$ into transformation matrices. **So we can facilitate per-tensor matrix multiplication implementation when per-pixel quantization is utilized**. Therefore, quantizing $X$, $B^TX$, $O$, and $A^TO$ results in a constant memory overhead and a computation overhead of $2 \times N \times (C_{in} + C_{out}) \times (m + r - 1) \times (m + r - 1)$. Because these quantization and re-quantization operations need the same times flops as the tensor size, the overhead is negligible compared to Winograd transformations which involve twice matrix multiplications.

The BOPs of different methods to quantize F(6,3) ResNet-20 are shown in Table R5-1. [1] quantizes all the transformation matrices $A$,$B$, and $G$, but the intermediate results such as $B^TX$ and $A^TO$ are not quantized. So [1] needs higher precision to hold these results and carry out the next operation. [2] don't quantize these Winograd transformations to maintain accuracy. Compared to them, our full-quantization of Winograd will achieve less computation cost.

**Table R1**

|                    | Im2col(FP) | Winograd(FP) | BQW[5]  | Winograd-AwareNet[4] | Ours(PAW) | Ours(PAW+FSQ) |
| ------------------ | ---------- | ------------ | ------- | -------------------- | --------- | ------------- |
| Low-precision BOPs | 0          | 0            | 464.56M | 1282.45M             | 464.56M   | 791.71M       |
| Flops              | 40.81M     | 12.37M       | 5.27M   | 0.48M                | 5.27M     | 0.80M         |
| Total BOPs         | 10.44G     | 3.16G        | 1.81G   | 1.41G                | 1.81G     | 0.99G         |


**Another common concern of reviewers is the performance of our algorithm on other networks.** In the paper, we show the results of ResNet-style models to compare our method with another PTQ work [5]. Here, we add experiments on VGG and Squeezenet using the Cifar-10 dataset. The results align with our expectations. The detailed results are presented in Table R1-2 and Table R1-3. We also add these experiments as Sec. 5 in supplementary materials.

**Table R2. Accuracy (\%) on VGG11 (92.02%).**

| Tile Size | BRECQ [6] | PAW   | FSQ   | FSQ+PAW |
| --------- | --------- | ----- | ----- | ------- |
| F4        | 89.13     | 91.56 | 86.59 | 91.55   |
|           | 92.02     | 92.28 | 90.82 | 91.83   |
| F6        | 75.10     | 89.94 | 68.98 | 90.34   |
|           | 91.27     | 91.88 | 88.44 | 91.63   |

**Table R3. Accuracy (\%) on SqueezeNet (92.62%).**

| Tile Size | BRECQ [6] | PAW   | FSQ   | FSQ+PAW |
| --------- | --------- | ----- | ----- | ------- |
| F4        | 89.69     | 91.98 | 88.66 | 91.78   |
|           | 92.61     | 92.68 | 92.01 | 92.80   |
| F6        | 80.50     | 90.67 | 76.48 | 91.26   |
|           | 92.37     | 92.61 | 90.54 | 92.42   |

[1] Wang et al.,  "Differentiable Joint Pruning and Quantization for Hardware Efficiency", 2020

[2] Guo et al.,  "Single path oneshot neural architecture search with uniform sampling", 2020

[3] Liu et al.,  "Towards precise binary neural network with generalized activation functions", 2020

[4] Fernández-Marqués et al., "Searching for Winograd-aware Quantized Networks", 2020

[5] Chikin et al.,  "Channel Balancing for Accurate Quantization of Winograd Convolutions", 2022

[6] Li et al., "BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction", 2021


Meta-review: The paper introduces a Post-training-quantization-aware Winograd (PAW) method, aiming to optimize the transformation procedures required by the Winograd algorithm for post-training quantization of pre-trained ResNet models. The proposed method also includes a factorized scale quantization (FSQ) technique to address the range of values in the Winograd domain. The paper's findings are supported by experiments on CIFAR-10 and ImageNet datasets using various ResNet architectures.

The paper presents a novel approach to post-training quantization for Winograd convolution, which has been acknowledged by reviewers as a significant contribution. The proposed method demonstrates improvements over existing methods, especially in terms of classification accuracy. The paper's organization and structure have been appreciated, with some reviewers noting the clear presentation of the challenges and solutions in the Winograd domain.

The reviewers are all positive about the paper. Although some concerns are rasied, they are addressed in rebuttal. The AC agrees with the reviewers to accept the paper. The authors should focus on improving the paper's presentation and clarity, providing a more comprehensive experimental evaluation, and addressing the technical issues highlighted by the reviewers.
Decision: Accept (poster)