Title: Towards Efficient and Accurate Winograd Convolution via Full Quantization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Winograd Convolution

Quantization
Abstract: The Winograd algorithm is an efficient convolution implementation, which performs calculations in the transformed domain. To further improve the computation efficiency, recent works propose to combine it with model quantization. Although Post-Training Quantization has the advantage of low computational cost and has been successfully applied in many other scenarios, a severe accuracy drop exists when utilizing it in Winograd convolution. Besides, despite the Winograd algorithm consisting of four stages, most existing methods only quantize the element-wise multiplication stage, leaving a considerable portion of calculations in full precision.
In this paper, observing the inconsistency among different transformation procedures, we present PTQ-Aware Winograd (PAW) to optimize them collaboratively under a unified objective function. Moreover, we explore the full quantization of faster Winograd (tile size $\geq4$) for the first time. We further propose a hardware-friendly method called Factorized Scale Quantization (FSQ), which can effectively balance the significant range differences in the Winograd domain. Experiments demonstrate the effectiveness of our method, e.g., with 8-bit quantization and a tile size of 6, our method outperforms the previous Winograd PTQ method by 8.27\% and 5.38\% in terms of the top-1 accuracy on ResNet-18 and ResNet-34, respectively.

Towards Efficient and Accurate Winograd Convolution via Full Quantization

Abstract

The Winograd algorithm is an efficient convolution implementation, which performs calculations in the transformed domain. To further improve the computation efficiency, recent works propose to combine it with model quantization. Although Post-Training Quantization has the advantage of low computational cost and has been successfully applied in many other scenarios, a severe accuracy drop exists when utilizing it in Winograd convolution. Besides, despite the Winograd algorithm consisting of four stages, most existing methods only quantize the element-wise multiplication stage, leaving a considerable portion of calculations in full precision. In this paper, observing the inconsistency among different transformation procedures, we present PTQ-Aware Winograd (PAW) to optimize them collaboratively under a unified objective function. Moreover, we explore the full quantization of faster Winograd (tile size ≥ 4 ) for the first time. We further propose a hardwarefriendly method called Factorized Scale Quantization (FSQ), which can effectively balance the significant range differences in the Winograd domain. Experiments demonstrate the effectiveness of our method, e.g., with 8-bit quantization and a tile size of 6, our method outperforms the previous Winograd PTQ method by 8.27% and 5.38% in terms of the top-1 accuracy on ResNet-18 and ResNet-34, respectively.

1 Introduction

Recently, Convolution Neural Networks (CNNs) have demonstrated state-of-the-art performance in various computer vision tasks [1; 2; 3; 4]. However, the high computation and storage cost hinders their deployment on resource-limited devices. To address this problem, various solutions have been proposed in the literature, including network pruning [5], low-rank decomposition [6], network quantization [7; 8; 9; 10; 11] and faster convolution implementation [12; 13]. In this paper, we focus on network quantization, faster convolution implementation, and their combination.

Model quantization converts the floating-point weights and activations to low-bit integers. According to whether data with labels are required for training, quantization can be divided into two categories: Quantization-Aware Training (QAT) [14; 15; 16; 17] and Post-Training Quantization (PTQ) [18; 19; 20; 21]. Although QAT can achieve promising performance, training the network requires many GPU resources and a full training dataset. Therefore, it is not always practical when either the training dataset is unavailable (e.g., privacy and commercial training data) or rapid deployment is required. On the contrary, as PTQ only needs few-shot, unlabeled calibration data and fewer computation resources, it is widely applicable in the industry.

An alternative method for enhancing the speed of CNNs involves the development of faster convolution implementations, such as FFT [13] and Winograd [12]. Among them, the Winograd algorithm is the most popular fast convolution operator [22]. Recent research [23; 24; 25] has focused on combining the Winograd algorithm with model quantization to further utilize the benefits of both techniques. However, two significant problems remain: On the one hand, while Winograd convolution with larger tile sizes (tile size ≥ 4) can provide more significant acceleration, existing PTQ methods for such convolution still suffer from drastic accuracy degradation. On the other hand, most existing works [24; 26] only quantize the element-wise multiplications, while leaving the domain transformation implemented by full-precision matrix multiplications, which can comprise a significant portion of the computation (40.1% when the input and output channels are 32 [24]). However, according to our experiments (Table 2 and Figure 2), these matrix multiplications are challenging to quantize because of the significant distribution imbalance in the Winograd domain.

In this paper, we first identify that quantization makes the transformation procedures in the Winograd algorithm inconsistent. Based on this observation, we propose P TQA ware W inograd (PAW), which optimizes transformation matrices collaboratively under a unified objective function. We then explore the fully quantized Winograd convolution, which is the first time for large tile sizes. We empirically find it is a non-trivial task because of the significant range differences of pixels in the Winograd domain. Through theoretical analysis of this phenomenon, we discover that it is possible to achieve comparable performance to per-pixel quantization without sacrificing computational efficiency via Factorized Scale Quantization (FSQ), which factorizes the tile size scales into vector size scales. Experiments are conducted to demonstrate the effectiveness of our method, e.g., with 8-bit quantization and a tile size of 6, our method outperforms the previous Winograd PTQ method by 8.27% and 5.38% in terms of the top-1 accuracy on ResNet-18 and ResNet-34, respectively.

Overall, our contributions in this work are threefold:

We experimentally observe that quantization will disrupt the consistency between different transformation procedures and propose PTQ-Aware Winograd, which utilizes a unified optimization procedure to make the Winograd algorithm more robust to quantization.

Through extensive experiments and theoretical analysis, we propose Factorized Scale Quantization, a hardware-friendly method suitable for the distribution characteristic of the Winograd domain tensors.

To the best of our knowledge, we are the first to achieve full quantization of Winograd convolution with large tile sizes (4 and 6). Experiments prove that our proposed method shows significant improvements over previous PTQ methods, even under fully-quantization settings.

2 Related Works

Quantization-aware training (QAT) [8; 27; 28; 29; 30] is a technique that simulates quantization noise during end-to-end training of neural networks. It uses discretized weights during both forward and backward propagation and updates original full-precision weights. Since the gradient of the quantization function is either zero or undefined everywhere, this procedure is done via a gradient approximation method called straight-through estimator [31]. Recently, some methods also add quantization related parameters to this training procedure, such as clipping range [32] and step size [14]. Although QAT methods have promising performance, they need the whole dataset and huge GPU resources.

Post-training quantization (PTQ) [7; 21; 9; 18; 33] is a more lightweight method that does not require retraining the network end-to-end. It only needs a small number of samples to estimate activation distribution. Despite its efficiency, PTQ suffers from a more significant accuracy degradation than QAT. The research community has actively proposed various methods to alleviate this problem. For example, [21] observes the scaling equivariance of activation functions and proposes to balance weights in consecutive layers. They also propose bias correction, which absorbs high biases into the next layer. Recently, some methods have found that the change in feature maps is a practical proxy signal for final task loss and use a layer-wise feature map reconstruction procedure to boost performance. Bit-Split [7] turns the low-bit discrete optimization into multiple single-bit optimization problems to find the optimal quantization scales and integer weights. AdaRound [18]

proposes optimizing the round function with a soft relaxation, which makes huge progress than the original round-to-nearest function.

Winograd convolution quantization Winograd is an fast convolution implementation first applied by [12]. To further improve computational efficiency, many works [23; 24; 34; 35; 36] focus on combining it with model quantization. For Winograd convolution networks with tile size ≥ 4, most works can only achieve an acceptable accuracy drop when using QAT, where the strong effectiveness of retraining the whole network hides the difficulty of quantization [23; 34; 36]. For instance, BQW [24] observes the heterogeneity of channel ranges in transformed tensors and equalizes them via migrating the scale variance from activations to weights. However, compared to the significant performance improvements achieved in QAT, their method only achieves limited progress in the case of PTQ. One of the reasons, we think, is that their method ignores the quantization impact on the Winograd algorithm, which is more notable without fine-tuning the whole network. In this paper, we will solve this problem and explore a more challenging scenario, fully quantized Winograd convolution using PTQ .

3 Background

3.1 Winograd

Winograd proposes the minimum filtering algorithm of finite impulse response (FIR) filtering in [37]. For r × r standard convolutions with filter size r , the algorithm transforms the convolution operations to the Winograd domain and generates m × m (spatial) outputs at a time, which is denoted as F ( m,r ) . The parameter m is called tile size, which is used to balance the speedup and numerical precision. The Winograd algorithm can be divided into four stages:

Input transformation : Firstly, a patch X ∈ R C i × a × a is extracted from the input data, with patch size a = m + r -1 . Then the c-th channel sub-tensor of X , denoted as X c , is transformed into the Winograd domain using input transformation matrix B ∈ R a × a . In this paper, the transformed inputs are denoted as U ∈ R C i × a × a :

Weight transformation : Similarly, weights W ∈ R C o × C i × r × r are transformed into the Winograd domain to get transformed weights V ∈ R C o × C i × a × a . This process can be performed offline before inference because model weights are frozen after training.

Element-wise multiplication : In the Winograd domain, the convolution operation is performed by element-wise multiplications between U and V :

where ⊙ denotes element-wise multiplication.

Output transformation : Finally, O ∈ R C o × a × a are transformed back to the feature map domain, and then, we get final outputs Y ∈ R C o × m × m :

Depending on the particular choice of Winograd domain (i.e., polynomial domain), these transformation matrices A , B , and G in Eq. (1)-(4) can be different. For F(2,3), the most common choice of the Winograd domain is f ( x ) = ( x -1)( x +1) x , then A , B and G can be constructed as follows[12]. More details are provided in the supplemental materials.

Although the Winograd algorithm reduces the computational complexity of convolution operations by performing them in the Winograd domain, these transformation steps also bring additional overheads. An example is shown in Table 1. When the channel numbers C o and C i are not large enough, these transformation processes will incur considerable computational costs.

3.2 Quantization

Quantization converts a full-precision tensor to an integer one. To improve hardware simplicity and efficiency, we utilize symmetric uniform quantization without zero-point. For a tensor V , quantization maps it to the integer values ˜ V and de-quantization remaps it to float-point values Q ( V ) :

where s represents a full-precision scalar called quantization scale, ⌊ z ⌉ rounds z to the nearest integer and clip ( z, r 1 , r 2 ) clamps z into the range [ r 1 , r 2 ] . For symmetric quantization with B bits, q min = -2 B -1 and q max = 2 B -1 .

In Eq. (6), we utilize a scalar s to quantize the entire tensor, called per-tensor quantization. Previous studies [24; 26] have shown that using per-pixel quantization, which provides independent scales for different pixels of transformed tensors, can significantly improve performance:

Here ⊘ is the element-wise division and we emit the clip function. S U and S V ∈ R a × a are tile size scales and can be factored out of the summation of Eq. (3). Thus, we can perform MAC operations in fixed-point format:

and

4 Method

4.1 PTQ-aware Winograd algorithm

While current state-of-the-art PTQ methods [18; 19; 9; 33] can achieve near-lossless 6-bit quantization on popular non-Winograd CNNs, recent research has shown that 8-bit quantization of Winograd convolutions still suffers from a significant accuracy drop (e.g., 9.71% for F(6,3) on ResNet-18 [24]). We argue that one key reason why quantization and the Winograd algorithm are not well-compatible is that quantization alters input and weight transformation procedures, making the whole algorithm inconsistent.

To support our argument, we experiment on the first layer of ResNet-18 and take weight transformation as an example. After quantization, the weight transformation procedure is changed from GWG T to Q ( GWG T ) . Consequently, the other two transformation procedures, A T OA and B T XB , will not be consistent with this new weight transformation. To illustrate it, we randomly perturb A and B and compute the reconstruction loss L quant and L origin with or without quantization, respectively:

738.90

738.95

739.00

739.05

739.10

739.15

Reconstruction Loss with Quantization

0.0000

0.0005

0.0010

0.0015

0.0020

0.0025

Reconstruction Loss without Quantization

Random A,B

Original A,B

(a) Random

A

,

B

491.55

491.60

491.65

491.70

491.75

491.80

491.85

Reconstruction Loss with Quantization

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Reconstruction Loss without Quantization

Random A,G

Original A,G

(b) Random

A

,

G

The results are shown in Figure 1(a). Although original A and B can produce desired results with floating-point GWG T (the yellow triangle on the bottom), they are no longer optimal when changing GWG T to Q ( GWG T ) . Concretely, about half of the random perturbed matrices lead to smaller errors (the blue dots on the upper left of the yellow triangle). A similar phenomenon also exists for input transformation, as shown in Figure 1(b). These experiments indicate that the inconsistency between domain transformations caused by quantization renders the original matrices unsuitable for quantization scenarios, which is the primary reason why Winograd convolutions are difficult to quantize. In order to align these transformation procedures after quantization, we propose to adjust transformation matrices via an optimization procedure as follows:

Since the transformation procedures in the Winograd algorithm are adopted considering the effect of PTQ, we name this method as P TQA ware W inograd (PAW). By using the straight-through estimator [31] to approximate the gradient through the round function as a pass-through operation, we can obtain the derivative of G (derivatives of A and B are shown in supplemental materials) and update it using SGD:

where

4.2 Fully-quantized Winograd convolution

4.2.1 Motivation

Winograd convolution involves a sequence of computational steps from Eq. (1) to Eq. (4). To further improve computation efficiency via full quantization, all these operations need to be quantized except

20

10

0

10

20

0

500

1000

1500

2000

2500

3000

3500

Layer2.1.conv1

O[4,4] std:0.28

O[5,5] std:3.76

10

0

10

0

500

1000

1500

2000

2500

Layer3.1.conv1

O[4,4] std:0.13

O[5,5] std:3.61

Eq. (2), which can be done offline. However, we empirically find that it is non-trivial to achieve full quantization because that accuracy will drop to zero due to the quantization of O (i.e., the output tensor in the Winograd domain), as shown in Table 2. After further investigation, we find that the ranges of O vary widely between different pixels. For example, as shown in Figure 2, the standard deviation (std) of O distributed at pixel (4,4) is ten times larger than that at pixel (5,5). A shared quantization scale for them is difficult to achieve both small round error and clamping error. A possible solution is to provide independent scales for them, i.e., to use per-pixel quantization (Table 3). However, since O will take part in matrix multiplication (Eq. 4) instead of element-wise multiplication, per-pixel quantization will lead to different scales in the summation dimension, which makes it not feasible in general hardware.

4.2.2 Factorized-scale quantization

Motivated by the strong range difference, we present the following theorem, which shows the cause of this distribution characterization and proves the standard deviation of O can be factorized into two vectors.

Proposition 1. Assume all elements of W ∈ R C i × r × r and X ∈ R C i × a × a are independently and identically distributed variables with zero mean (e.g., X ∼ N (0 , σ X ) , W ∼ N (0 , σ W ) ). If denote B T = ( b T 1 , b T 2 , ..., b T a ) T and G = ( g T 1 , g T 2 , ..., g T a ) T , we have:

Proof. Firstly, since B T X c B and GW c G T are the linear combinations of independent and identical variables with zero mean, we can calculate their mean and variance:

Then we can derive the variance of O :

The equation (a) holds because the variables in different channels are independent. The equation (b) holds because the random variables U and V are independent of each other and have zero mean.

After finding that the standard deviation of O can be decoupled along rows and columns separately, an intuitive thought is that the per-pixel scales may also satisfy similar rules. A more theoretical proof is provided as Theorem 1.

𝑶 ′

∙ 𝐎

𝑶 ′′

∙ 𝐎 ∙ 𝐃𝐢𝐚𝐠 𝜷

Row

Column

J

0

6

#

Theorem 1. Assuming X ∼ N (0 , σ X ) , W ∼ N (0 , σ W ) , the optimal per-pixel scales S ∗ which minimize quantization error in Eq. (21) can be factorized as: S ∗ = α ∗ β T .

Proof. In Proposition 1, we have proven that:

Observing that O ij is the summation of C i independent variables, where C i is the number of input channels and is usually large (e.g., 128). According to the Central Limit Theorem [38], we can assume O ij follows Gaussian distribution. In supplementary material, we will show that the optimal scale s to minimize the mean-square error of quantizing Gaussian variables z ∼ N (0 , σ 2 ) is proportional to σ , i.e., s = Kσ , where K is a constant. Thus, we have:

The benefit of factorizing per-pixel scales into two vectors is that we can move both scales into transformation matrices. So we can facilitate per-tensor matrix multiplication implementation when per-pixel quantization is utilized:

where:

Note that at this stage, matrix A is not quantized. After rescaling, we can further quantize these transformation matrices and middle results to obtain fully-quantized Winograd convolution. In this paper, this method is called F actorized S cale Q uantization (FSQ) and an illustration of it is shown in Figure 3.

4.2.3 Optimization procedure

In this section, we focus on how to determine the optimal α and β . Given N samples { O 1 , O 2 , ..., O N } , we can minimize the quantization error via the following non-linear least square regression problems:

The optimization of Eq. (26) is non-trivial due to the integer constraint of ˜ O . Thus we propose an iteration method to solve it.

Solving α when fixed β and ˜ O . When β and ˜ O are fixed, (26) is a quadratic function with respect to α . Then the closed-form solution for α can be easily obtained as:

Solving β When fixed α and ˜ O . The closed-form solution for β is similar to that of α :

Solving ˜ O when fixed α and β . When α and β are fixed, ˜ O is simply the integer values by rounding to the nearest:

5 Experiments

5.1 Experimental settings

We use full-precision pre-trained models and replace all 3x3 convolutions (stride 1) with Winograd convolutions. All convolutions, including the first and last layers, are quantized using symmetric quantization. Following BQW[24], we use per-pixel quantization for transformed inputs U and weights V . In order to verify the effectiveness of our method, we first construct a strong baseline on Post-training Quantization Winograd. Concretely, we resort to Adaround [18] and LSQ [14] to quantize U and V , respectively. Following BRECQ [19], we use 1024 unlabeled images and Adam [39] optimizer with 20k iterations and a batch size of 32. Experiments show that our strong baseline has surpassed previous state-of-the-art PTQ Winograd work. Based on it, we further verify our method on the strong baseline. The learning rates of A , B , and G are set to (1e-4, 1e-4, 5e-4) by default, and the reason will be shown in Section 5.4. All our experiments are conducted on NVIDIA GeForce RTX 3090 24GB GPU servers and last for several hours.

5.2 PTQ-Aware Winograd

In this section, we compare our PTQ-Aware Winograd method to the previous work BQW [24] with comprehensive experiments settings, including various bitwidths, tile sizes, datasets and models.

ImageNet . We also conduct experiments on ImageNet, a more challenging image classification dataset than CIFAR-10. In this experiment, we replace the convolutions in the last block with F(4,3) Winograd convolution since F(6,3) Winograd convolution needs more padding for 7 × 7 input size. Noting that BQW [24] only conducts PTQ with 8-bit because of severe accuracy degeneration on lower bitwidth, we compare our method with our strong baseline. The results presented in Table 5 demonstrate two breakthroughs of our method. First, we are the first to achieve negligible accuracy drop with 8-bit post-training quantization on Winograd convolution. Second, we are the first to achieve an acceptable accuracy drop with 6-bit post-training quantization on Winograd convolution.

5.3 Factorized Scale Quantization

In this section, we use Factorized Scale Quantization (FSQ) for O and per-pixel quantization for U and V . Other components are conducted with per-tensor quantization. We conduct experiments on ImageNet with bitwidths 6 and 8. In 6-bit quantization, we use 6-bit to quantize U , V , and 8-bit to quantize other components. The results shown in Table 5 indicate that our proposed FSQ is the crucial step to fully quantizing Winograd convolutions, and it is well-compatible with PTQAware Winograd (PAW). Even with fully-quantization settings, our methods can achieve comparable accuracy compared to BQW[24].

5.4 Ablation studies

To evaluate the effectiveness of our quantization-aware Winograd, we conduct an ablation study on ResNet-18 by optimizing different combinations of A , B , and G . Empirically, we find that the learning rate of G is more sensitive than those of A and B . Thus we fix the learning rates of A and B to 1e-4 for all experiments and perform a grid search on the learning rate of G separately, which is varied in the interval [1e-7, 1e-3]. The best results are shown in Table 6. When optimizing them, the optimal learning rate for G is 5e-4, and the performance improvement reaches 1.22%.

6 Conclusions

This paper focuses on accelerating deep convolution neural networks by combining the Winograd algorithm and model quantization. We propose a unified optimization procedure to improve the compatibility between quantization and the Winograd algorithm. To achieve a more significant speed-up via full quantization, we propose a theoretically supported and hardware-friendly method called Factorized-Scale Quantization, which is suitable for the distribution characteristics of tensors in the Winograd domain. Experiments are conducted to show that our method can boost performance by a large margin, especially with lower bitwidth or larger tile size, which means our method makes deploying CNNs with more limited computation resources possible.

References

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 770-778. IEEE Computer Society, 2016.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Commun. ACM , 60(6):84-90, 2017.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017 , pages 2980-2988. IEEE Computer Society, 2017.

Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , pages 2472-2481. Computer Vision Foundation / IEEE Computer Society, 2018.

Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. 2016.

Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE Trans. Pattern Anal. Mach. Intell. , 38(10):19431955, 2016.

Peisong Wang, Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training network quantization via bit-split and stitching. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 9847-9856. PMLR, 2020.

Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew G. Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , pages 2704-2713. Computer Vision Foundation / IEEE Computer Society, 2018.

Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020 , pages 13166-13175. Computer Vision Foundation / IEEE, 2020.

Weihan Chen, Peisong Wang, and Jian Cheng. Towards mixed-precision quantization of neural networks via constrained optimization. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pages 5330-5339. IEEE, 2021.

Peisong Wang, Xiangyu He, Gang Li, Tianli Zhao, and Jian Cheng. Sparsity-inducing binarized neural networks. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020 , pages 12192-12199. AAAI Press, 2020.

Andrew Lavin and Scott Gray. Fast algorithms for convolutional neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 4013-4021. IEEE Computer Society, 2016.

Michaël Mathieu, Mikael Henaff, and Yann LeCun. Fast training of convolutional networks through ffts. 2014.

Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S. Modha. Learned step size quantization. 2020.

Huixia Li, Chenqian Yan, Shaohui Lin, Xiawu Zheng, Baochang Zhang, Fan Yang, and Rongrong Ji. PAMS: quantized super-resolution via parameterized max scale. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV , volume 12370 of Lecture Notes in Computer Science , pages 564-580. Springer, 2020.

Xiangyu He, Zitao Mo, Ke Cheng, Weixiang Xu, Qinghao Hu, Peisong Wang, Qingshan Liu, and Jian Cheng. Proxybnn: Learning binarized neural networks via proxy matrices. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III , volume 12348 of Lecture Notes in Computer Science , pages 223-241. Springer, 2020.

Weixiang Xu, Fanrong Li, Yingying Jiang, Yong A, Xiangyu He, Peisong Wang, and Jian Cheng. Improving extreme low-bit quantization with soft threshold. IEEE Trans. Circuits Syst. Video Technol. , 33(4):1549-1563, 2023.

Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 7197-7206. PMLR, 2020.

Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. BRECQ: pushing the limit of post-training quantization by block reconstruction. 2021.

Peisong Wang, Weihan Chen, Xiangyu He, Qiang Chen, Qingshan Liu, and Jian Cheng. Optimization-based post-training quantization with bit-split and stitching. IEEE Trans. Pattern Anal. Mach. Intell. , 45(2):2119-2135, 2023.

Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019 , pages 1325-1334. IEEE, 2019.

NVIDIA TensorRT. https://docs.nvidia.com/deeplearning/tensorrt/index.html .

Javier Fernández-Marqués, Paul N. Whatmough, Andrew Mundy, and Matthew Mattina. Searching for winograd-aware quantized networks. 2020.

Vladimir Chikin and Vladimir Kryzhanovskiy. Channel balancing for accurate quantization of winograd convolutions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 12497-12506. IEEE, 2022.

Lingchuan Meng and John Brothers. Efficient winograd convolution via integer arithmetic. CoRR , abs/1901.01965, 2019.

Renzo Andri, Beatrice Bussolino, Antonio Cipolletta, Lukas Cavigelli, and Zhe Wang. Going further with winograd convolutions: Tap-wise quantization for efficient inference on 4x4 tiles. In 55th IEEE/ACM International Symposium on Microarchitecture, MICRO 2022, Chicago, IL, USA, October 1-5, 2022 , pages 582-598. IEEE, 2022.

Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV , volume 9908 of Lecture Notes in Computer Science , pages 525-542. Springer, 2016.

Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. pages 3123-3131, 2015.

Zeyu Zhu, Fanrong Li, Zitao Mo, Qinghao Hu, Gang Li, Zejian Liu, Xiaoyao Liang, and Jian Cheng. $\rm aˆ2q$: Aggregation-aware quantization for graph neural networks. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023.

Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. HAQ: hardware-aware automated quantization with mixed precision. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019 , pages 8612-8620. Computer Vision Foundation / IEEE, 2019.

Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR , abs/1308.3432, 2013.

Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. PACT: parameterized clipping activation for quantized neural networks. CoRR , abs/1805.06085, 2018.

Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. Qdrop: Randomly dropping quantization for extremely low-bit post-training quantization. 2022.

Lingchuan Meng and John Brothers. Efficient winograd convolution via integer arithmetic. CoRR , abs/1901.01965, 2019.

Guangli Li, Lei Liu, Xueying Wang, Xiu Ma, and Xiaobing Feng. Lance: efficient low-precision quantized winograd convolution for neural networks based on graphics processing units. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020 , pages 3842-3846. IEEE, 2020.

Renzo Andri, Beatrice Bussolino, Antonio Cipolletta, Lukas Cavigelli, and Zhe Wang. Going further with winograd convolutions: Tap-wise quantization for efficient inference on 4x4 tiles. In 55th IEEE/ACM International Symposium on Microarchitecture, MICRO 2022, Chicago, IL, USA, October 1-5, 2022 , pages 582-598. IEEE, 2022.

Shmuel Winograd. Arithmetic complexity of computations , volume 33. Siam, 1980.

Robert V. Brill. Applied Statistics and Probability for Engineers , volume 46. 2004.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2015.

A Winograd transformation matrices

Depending on the particular choice of Winograd domain (i.e., polynomial domain), transformation matrices A , B , and G in the Winograd algorithm can be different. In the paper, we present that the most popular interpolation points for F(2,3) are [0 , +1 , -1] and then these transformation matrices can be constructed as follows:

For F(4,3) and F(6,3), we choose the same transformation matrices as BQW [24]. For F(4,3), the Winograd transformation matrices are as follows:

For F(6,3), the Winograd transformation matrices are as follows:

B Derivatives of transformation matrices

In the paper, in order to align these transformation procedures after quantization, we propose to adjust transformation matrices via an optimization procedure as follows:

By using the straight-through estimator [31] to approximate the gradient through the round function as a pass-through operation, we can obtain the derivatives of A , B and G . In this paper, we directly present the derivative of B . Here, a more comprehensive derivation is provided as follows:

We have obtained the derivative of B ij , and now we can provide the expression for the derivative of B :

The derivatives of A , G and O f can be computed in a similar manner:

C Optimal quantization scale for Guassion varibles

In Theorem 1, in order to demonstrate that the optimal per-pixel scale S can be factorized into vectors, we rely on the conclusion that the optimal scale s ∗ to minimize the mean-square error of quantization of Gaussian variables z ∼ N (0 , σ 2 ) is proportional to σ , i.e., s ∗ = Kσ , where K is a constant. Here, we will provide a proof of it.

Theorem 2. Assuming z ∼ N (0 , σ 2 ) , the optimal scale s ∗ to minimize the mean-square error of quantization of z is proportional to the standard deviation σ , i.e., s ∗ = Kσ , where K is a constant.

Proof. Because z ∼ N (0 , σ 2 ) , z can be reparameterized as z = σ · u , where u ∼ N (0 , 1) .

Eq. (47) can be treated as a function of s/σ when solving for s with σ as a known value. Assuming K minimizes function h ( x ) , i.e., K = argmin h ( x ) , we have:

x

D Experiments on other architectures

In Section 5, we compare our methods to previous work BQW[24] on the ResNet model family with comprehensive experiment settings, including various bit widths, tile sizes, and datasets. Here, we present a similar analysis for two other popular architectures VGG and Squeezenet using the Cifar-10 dataset. The results are shown in Table 1 and Table 2. These results align with our analysis in Section 5. Our PTQ-Aware Winograd (PAW) method outperforms the strong baseline introduced in Section 5 and our FSQ method is well-compatible with PAW.

Peer reviews:
comments: SUMMARY:
This paper proposed to fully quantize the Winograd convolution post-training under the observation of disruption of consistency between different transformation procedures, the new proposed Factorized Scale Quantization is suitable in the Winograd domain. The experiments demonstrate significant improvements compared with previous post-training-quantization methods.

STRENGTHS:
1. This paper is organized well, the exploration of quantization of different components of Winograd convolution clearly shows the primary cause of accuracy degeneration, and the following proposed method precisely targets this problem.
2. The proposed method is interesting, which utilizes two factor vectors to replace the per-pixel scales, and these two vectors can be merged into the transformation matrices.
3. Some experimental results are even higher than the baseline.

WEAKNESSES:
The experiments only demonstrate the quantization bit, there is no computation cost or inference time comparison which are very important to this work, it is difficult to know how much the proposed improve the efficiency other than accuracy.

QUESTIONS:
This paper emphasizes the hardware-friendly deployment of this proposed quantization method, will the optimization procedure including solving $\alpha$, $\beta$, and $\tilde{O}$ cost much more additional resources?

LIMITATIONS:
This paper hasn't mentioned or discussed its limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PTQ-Aware Winograd (PAW), Factorized-scale quantization and a iterative optimization algorithm to solve the problem of quantization on Winograd domain. These methods not only fully quantize the whole Winograd Convolution, but also surpass the existing Winograd quantization methods in terms of effect and hardware friendliness.\n\nSTRENGTHS:\n1. This is the first work to perform full-quantization for Winograd Convolution\n\n2. Factorized-scalequantization makes the quantization for Winograd more hardware-friendly compared to per-pixel quantization in previous work\n\n3. This paper focuses on the properties of the Winograd domain that make quantification difficult and gives a full analysis\n\n4. The effect of the method is obviously beyond the existing methods, and the theoretical derivation is complete and the experiment is fully credible\n\nWEAKNESSES:\n1. In the explanation in Section 4.1:\n\n   - Is the motivation of this section: In the original Winograd method, A, B and G satisfy some strict mathematical relationship with each other, but the perturbation brought by quantization destroys the strict relationship between them, and the small perturbation brought by such quantization will bring much greater overall loss. Therefore, the authors simulated the perturbation by artificially adding the perturbation and tested the Reconstruction Loss?\n\n   - How was the range of perturbation added determined, and was the actual quantization error measured and used as a reference? Have you tested how much the Reconstruction Loss will decrease after optimization?\n\n2. This work achieves an obvious improvement over previous work, but what are the challenges if we continue to expand the tile size? Why?\n\nI would like to hear the authors’ feedback during rebuttal, and fixing these issues would further improve the quality of this paper.\n\nQUESTIONS:\nSee Weakness.\n\nLIMITATIONS:\nSee Weakness.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper proposes a PTQ-Aware Winograd (PAW) method to improve the performance of deep learning inference with quantized parameters and Winograd Convolution. In particular, all steps of the Winograd operation are combined and optimized with a unified objective to reduce the domino effect of quantization in different parts of the Winograd sequence. Moreover, to mitigate the range difference of the Winograd output transformation, factorized-scale quantization is proposed. This involves balancing the distribution of the output transformation using two factorized scaling parameters. This approach outperforms previous work and the cifar-10 dataset.

STRENGTHS:
The analysis of quantization for all parts of the Winograd operation is novel and interesting. The paper is well-written, and the proofs are explained well.

WEAKNESSES:
1- The overhead of proposed quantization algorithm needs to be compared with the QAT approach, since both methods perform training during the quantization process.

2- The new approach has only been tested on the ResNet model. It is suggested that experimental results for other networks be added to the paper.

QUESTIONS:
Most of the proofs in this paper assume the parameters follow a Gaussian distribution. Previous studies have shown that the distribution of parameters is similar to Laplace distribution. Does the proof in the paper also hold valid with a near-Laplace distribution of parameters?

LIMITATIONS:
The author does not discuss whether the new approach works for size 6 and size 8 of the Winograd operation.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a post-training quantization algorithm for Winograd convolution, which overcomes the inconsistency in domain transformation by adjusting the transformation matrices together (PTQ-Aware) via a unified optimization procedure, and achieves full quantization by a new factorized scale quantization (FSQ) method. Experiments on CIFAR-10 and ImageNet show the effectiveness of the proposed algorithm. It surpasses the previous state-of-the-art Winograd post-training quantization algorithm significantly in terms of accuracy.\n\nSTRENGTHS:\n* The proposed Winograd post-training quantization algorithm is considerably more accurate than previous methods, and is hardware friendly.\n* The paper is well presented. I am able to understand the whole paper as a reader whose research focus is not on model quantization.\n\nWEAKNESSES:\n* The paper failed to mention or compare with some quantization-aware training (QAT) algorithms that achieve full-quantization, e.g \"Searching for Winograd-aware Quantized Networks\" [1] and \"Going Further With Winograd Convolutions: Tap-Wise Quantization for Efficient Inference on 4x4 Tiles\" [2]. These QAT algorithms share quite similar ideas with this paper and the accuracy of their final quantized models are much higher.  \n  * [1] https://arxiv.org/abs/2002.10711  (2020)\n  * [2] https://arxiv.org/abs/2209.12982 (2022)\n* If possible, some quantitive evaluation on how faster the proposed algorithm is (compared to previous algorithms) is desired.\n* Two typos: \n  * line 188: v_i -> v_j\n  * line 239: long -> learn\n\nQUESTIONS:\n* Could you briefly compare your proposed algorithm with some QAT methods that also achieve full-quantization (e.g. the two papers I listed in the Weakness section) and summarize the advantage of your algorithm?\n* Could you provide some evaluation on the inference speed of the proposed algorithm compared with previous algorithms? If not possible, some simple analysis is also OK. I'd like to know whether there's some hidden computation overhead or complexity introduced by your algorithm.\n\nLIMITATIONS:\nI hope the authors can better differentiate their method with previous Winograd quantization algorithms and justify their method's advantages. If it simply adapts QAT full-quantization to the PTQ scenario, the novelty and contribution of the paper would be somewhat undermined.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a Post-training-quantization-aware Winograd (PAW) method to optimize all transformation procedures required by the Winograd algorithm to achieve post-training quantization of pre-trained ResNet models. A useful factorized scale quantization (FSQ) method is also proposed to balance the differences in the range of values in the Winograd domain. Obvious increases in terms of classification accuracy values on CIFAR-10 and ImageNet datasets are obtained with different ResNet-style model structures.\n\nSTRENGTHS:\nThe paper achieved good improvements by solving problems observed in practice in cases with large tile sizes. It is useful to know the problems the authors observed and their solutions.\n\nWEAKNESSES:\n1. The presentation of the paper was not satisfactory and is quite difficult to follow. A lot of polishing work needs to be made before the paper could reach a state that I could recommend accepting. \n\n2. Some motivation for the work needs to be justified: why checking and solving the issue happened when quantizing ResNet-style models with large (>=4) tile sizes? Some justification and performance differences need to be made when comparing to those with smaller tile sizes.  \n\nMany minor issues are listed below:\n1. Many issues in the use of English:\na. \"Another approach to accelerate CNNs is faster implementations of convolution, such as FFT [11] and Winograd [10]. Among them,\"  ...\nb. \"for the first time. And we further propose a \" ... \nand many others (too many to list)...\n2. \"surpasses the previous state-of-the-art method by 8.27% and 5.38% on ResNet-18 and ResNet-34, respectively.\" -> Better to be clear surpass in terms of what?\n3. Different numbers of significant figures are used in Table 2.\n4. Abbreviation std is not defined in the paper.\n5. \"does not require retraining the network end-to-end.\" -> It seems the authors may mean both updating the entire model and training from scratch by using the wording \"end-to-end\" here. Many ambiguities like this exist throughout the paper. \n6. Would be more friendly to the readers to define tile size in the paper (I guess you mean kernel size or filter size here). Could also provide the definitions of F(4,3) and F(6,3) before using them. \n7. Many issues in the references. For instance, conference references [7], [19], and [30] are in different formats. Some references, such as [33], are not cited in their official publication sources. \n8. In table 5, perhaps better to use the wording \"baseline\" than \"strong baseline\". We believe all baseline systems authors try to quote are strong enough. \n9. Since non-standard tile sizes are used in this paper, it may be better to explain more carefully about the model architectures rather than simply referring to them using the standard terms (ResNet-34 etc.).\n\nQUESTIONS:\n1. From Table 4, the cases in which PAW resulted in the most considerable improvements are 6-bit and 8-bit for F(6,3). Could you please explain in what type of hardware, 6-bit could be a more efficient setting than 8-bit?\n\n2. From Table 5, F(4,3) results are often better than the F(6,3) results, which matches my understanding. In that sense, could you please explain why improving the performance with tile size 6 matters? Following that logic, could you please provide more evidence of the advantage of using a large tile size (>=4)?\n\nLIMITATIONS:\nOnly ResNet-style model architectures are tested, which is limited. The authors also did not compare the results and efficiency against those with the standard tile size.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}