SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
S _ {i j} = \sqrt {K C _ {i} | | b _ {i} | | ^ {2} | | g _ {i} | | ^ {2} \sigma_ {X} \sigma_ {W}} \cdot \sqrt {K C _ {i} | | b _ {j} | | ^ {2} | | g _ {j} ^ {2} | | \sigma_ {X} \sigma_ {W}} = \alpha_ {i} \cdot \beta_ {j} \tag {23}
$$
The benefit of factorizing per-pixel scales into two vectors is that we can move both scales into transformation matrices. So we can facilitate per-tensor matrix multiplication implementation when per-pixel quantization is utilized:
$$
\begin{array}{l} A ^ {T} O A \approx A ^ {T} \cdot \left(S _ {O} \odot \tilde {O}\right) \cdot A \\ = A ^ {T} \cdot \left(\left(\alpha * \beta^ {T}\right) \odot \widetilde {O}\right) \cdot A \tag {24} \\ = A _ {L} \widetilde {O} A _ {R} \\ \end{array}
$$
$$
A _ {L} = A ^ {T} \cdot \operatorname {D i a g} (\alpha), \quad A _ {R} = \operatorname {D i a g} (\beta) \cdot A \quad \text {a n d} \quad \widetilde {O} = \left\lfloor O \oslash S _ {O} \right\rceil \tag {25}
$$
Note that at this stage, matrix  $A$  is not quantized. After rescaling, we can further quantize these transformation matrices and middle results to obtain fully-quantized Winograd convolution. In this paper, this method is called Factorized Scale Quantization (FSQ) and an illustration of it is shown in Figure 3.
4.2.3 Optimization procedure
In this section, we focus on how to determine the optimal  $\alpha$  and  $\beta$ . Given  $N$  samples  $\{O^1,O^2,\dots ,O^N\}$ , we can minimize the quantization error via the following non-linear least square regression problems:
$$
\underset {\alpha , \beta} {\operatorname {a r g m i n}} \Sigma_ {n} ^ {N} \Sigma_ {f} ^ {C _ {o}} \left\| O _ {f} ^ {n} - \left(\alpha * \beta^ {T}\right) \odot \widetilde {O _ {f} ^ {n}} \right\| ^ {2} \tag {26}
$$
The optimization of Eq. (26) is non-trivial due to the integer constraint of  $\widetilde{O}$ . Thus we propose an iteration method to solve it.
Solving  $\alpha$  when fixed  $\beta$  and  $\widetilde{O}$ . When  $\beta$  and  $\widetilde{O}$  are fixed, (26) is a quadratic function with respect to  $\alpha$ . Then the closed-form solution for  $\alpha$  can be easily obtained as:
$$
\alpha^ {*} = \frac {\Sigma_ {n} ^ {N} \Sigma_ {f} ^ {C _ {o}} \left(O _ {f} ^ {n} \odot \widetilde {O} _ {f} ^ {n}\right) \cdot \beta}{\Sigma_ {n} ^ {N} \Sigma_ {f} ^ {C _ {o}} \left(\widetilde {O} _ {f} ^ {n} \odot \widetilde {O} _ {f} ^ {n}\right) \cdot (\beta \odot \beta)} \tag {27}
$$
Solving  $\beta$  When fixed  $\alpha$  and  $\tilde{O}$ . The closed-form solution for  $\beta$  is similar to that of  $\alpha$ :
$$
\beta^ {*} = \frac {\sum_ {n} ^ {N} \sum_ {f} ^ {C _ {o}} \left(O _ {f} ^ {n} \odot \widetilde {O} _ {f} ^ {n}\right) ^ {T} \cdot \alpha}{\sum_ {n} ^ {N} \sum_ {f} ^ {C _ {o}} \left(\widetilde {O} _ {f} ^ {n} \odot \widetilde {O} _ {f} ^ {n}\right) ^ {T} \cdot (\alpha \odot \alpha)} \tag {28}
$$
Solving  $\widetilde{O}$  when fixed  $\alpha$  and  $\beta$ . When  $\alpha$  and  $\beta$  are fixed,  $\widetilde{O}$  is simply the integer values by rounding to the nearest:
$$
\widetilde {O} _ {f} ^ {n} * = \left\lfloor O _ {f} ^ {n} \oslash (\alpha^ {T} \cdot \beta) \right] \tag {29}
$$
We use full-precision pre-trained models and replace all 3x3 convolutions (stride 1) with Winograd convolutions. All convolutions, including the first and last layers, are quantized using symmetric quantization. Following BQW[24], we use per-pixel quantization for transformed inputs  $U$  and weights  $V$ . In order to verify the effectiveness of our method, we first construct a strong baseline on Post-training Quantization Winograd. Concretely, we resort to Adaround [18] and LSQ [14] to quantize  $U$  and  $V$ , respectively. Following BRECQ [19], we use 1024 unlabeled images and Adam [39] optimizer with 20k iterations and a batch size of 32. Experiments show that our strong baseline has surpassed previous state-of-the-art PTQ Winograd work. Based on it, we further verify our method on the strong baseline. The learning rates of  $A$ ,  $B$ , and  $G$  are set to (1e-4, 1e-4, 5e-4) by default, and the reason will be shown in Section 5.4. All our experiments are conducted on NVIDIA GeForce RTX 3090 24GB GPU servers and last for several hours.
5.2 PTQ-Aware Winograd
In this section, we compare our PTQ-Aware Winograd method to the previous work BQW [24] with comprehensive experiments settings, including various bitwidths, tile sizes, datasets and models.
Table 4: PTQ results of ResNet-20 on CIFAR-10 with different tile sizes (4,6) and different bitwidths (4,6,8).
Model	Winograd Algorithm	Bits	Quantization Accuracy
Strong Baseline	BQW [24]	PAW
ResNet-20 (91.76%)	F(4,3)	4	22.46	17.33	62.50 (+45.17)
6	80.63	79.36	90.25 (+12.65)
8	89.55	90.31	92.02 (+1.71)
F(6,3)	4	13.51	10.00	21.71 (+11.71)
6	46.62	39.75	85.29 (+45.54)
8	81.44	81.44	91.10 (+9.66)
CIFAR-10. The results of different PTQ methods on ResNet-20 are shown in Table 4. Although our strong baseline has been able to outperform previous method, our PTQ-Aware Winograd (PAW) can even improve it further, especially with lower bitwidth or larger tile size. For example, our methods surpass BQW [24] by  $12.65\%$  and  $45.54\%$  when using 6-bit quantization on F(4,3) and F(6,3), respectively, which means our method makes deploying CNNs with more limited computation resources possible.
ImageNet. We also conduct experiments on ImageNet, a more challenging image classification dataset than CIFAR-10. In this experiment, we replace the convolutions in the last block with F(4,3) Winograd convolution since F(6,3) Winograd convolution needs more padding for  $7 \times 7$  input size. Noting that BQW [24] only conducts PTQ with 8-bit because of severe accuracy degeneration on lower bitwidth, we compare our method with our strong baseline. The results presented in Table 5 demonstrate two breakthroughs of our method. First, we are the first to achieve negligible accuracy drop with 8-bit post-training quantization on Winograd convolution. Second, we are the first to achieve an acceptable accuracy drop with 6-bit post-training quantization on Winograd convolution.
5.3 Factorized Scale Quantization
In this section, we use Factorized Scale Quantization (FSQ) for  $O$  and per-pixel quantization for  $U$  and  $V$ . Other components are conducted with per-tensor quantization. We conduct experiments on ImageNet with bitwidths 6 and 8. In 6-bit quantization, we use 6-bit to quantize  $U$ ,  $V$ , and 8-bit to quantize other components. The results shown in Table 5 indicate that our proposed FSQ is the crucial step to fully quantizing Winograd convolutions, and it is well-compatible with PTQ-Aware Winograd (PAW). Even with fully-quantization settings, our methods can achieve comparable accuracy compared to BQW[24].
Table 5: PTQ results of ResNets on ImageNet. Previous method BQW [24] does not quantize matrix multiplications in Winograd transformation. 'SB' indicates our Strong Baseline.
Model	Tile	Bits	Partial Quantization	Full Quantization
SB	BQW [24]	PAW	PAW	FSQ	FSQ+PAW
ResNet-18 (69.76%)	F(4,3)	6	50.86	N/A	65.15	0.26	44.21	64.34
8	68.08	67.54	69.06	0.21	64.46	68.16
F(6,3)	6	24.58	N/A	59.58	0.13	18.68	59.30
8	65.84	60.09	68.36	0.13	58.48	66.89
ResNet-34 (73.30%)	F(4,3)	6	58.85	N/A	69.71	0.88	52.10	68.80
8	72.11	71.86	72.67	0.96	69.51	71.75
F(6,3)	6	27.91	N/A	63.56	0.19	22.06	62.11
8	69.68	66.52	71.90	0.29	63.60	69.72
ResNet-50 (76.15%)	F(4,3)	6	73.25	N/A	74.94	22.44	72.21	74.75
8	75.89	75.84	76.01	17.80	75.90	75.74
F(6,3)	6	69.20	N/A	73.99	5.97	66.85	73.84
8	75.34	74.47	75.80	3.20	73.66	75.36
To evaluate the effectiveness of our quantization-aware Winograd, we conduct an ablation study on ResNet-18 by optimizing different combinations of  $A$ ,  $B$ , and  $G$ . Empirically, we find that the learning rate of  $G$  is more sensitive than those of  $A$  and  $B$ . Thus we fix the learning rates of  $A$  and  $B$  to 1e-4 for all experiments and perform a grid search on the learning rate of  $G$  separately, which is varied in the interval [1e-7, 1e-3]. The best results are shown in Table 6. When optimizing them, the optimal learning rate for  $G$  is 5e-4, and the performance improvement reaches  $1.22\%$ .
Table 6: Impact of different design choices for optimization of transformation matrices, on the ImageNet validation accuracy (\%) for ResNet-18.
Optimized Matrices	None	A	B	G	A, B	B, G	A, G	A, B, G
Accuracy	68.04	68.07	68.24	68.03	68.87	68.28	68.03	69.26
This paper focuses on accelerating deep convolution neural networks by combining the Winograd algorithm and model quantization. We propose a unified optimization procedure to improve the compatibility between quantization and the Winograd algorithm. To achieve a more significant speed-up via full quantization, we propose a theoretically supported and hardware-friendly method called Factorized-Scale Quantization, which is suitable for the distribution characteristics of tensors in the Winograd domain. Experiments are conducted to show that our method can boost performance by a large margin, especially with lower bitwidth or larger tile size, which means our method makes deploying CNNs with more limited computation resources possible.
This work was supported in part by the STI 2030-Major Projects (No.2021ZD0201504), the NSFC-General Technology Collaborative Fund for Basic Research (No.U1936204), the Jiangsu Key Research and Development Plan (No.BE2023016, No.BE2021012-2).
[1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society, 2016.
[2] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Commun. ACM, 60(6):84-90, 2017.
[3] Kaiming He, Georgia Gkioxari, Piotr Dólar, and Ross B. Girshick. Mask R-CNN. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 2980-2988. IEEE Computer Society, 2017.
[4] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 2472-2481. Computer Vision Foundation / IEEE Computer Society, 2018.
[5] Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. 2016.
[6] Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. IEEE Trans. Pattern Anal. Mach. Intell., 38(10):1943-1955, 2016.
[7] Peisong Wang, Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training network quantization via bit-split and stitching. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 9847-9856. PMLR, 2020.
[8] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew G. Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 2704-2713. Computer Vision Foundation / IEEE Computer Society, 2018.
[9] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 13166-13175. Computer Vision Foundation / IEEE, 2020.
[10] Weihan Chen, Peisong Wang, and Jian Cheng. Towards mixed-precision quantization of neural networks via constrained optimization. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 5330-5339. IEEE, 2021.
[27] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pages 525-542. Springer, 2016.
[28] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. pages 3123-3131, 2015.
[29] Zeyu Zhu, Fanrong Li, Zitao Mo, Qinghao Hu, Gang Li, Zejian Liu, Xiaoyao Liang, and Jian Cheng.  $\S\mathrm{rm} \hat{\mathbf{a}}^{\prime} 2 \mathbf{q} \S$ : Aggregation-aware quantization for graph neural networks. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[30] Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. HAQ: hardware-aware automated quantization with mixed precision. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 8612-8620. Computer Vision Foundation / IEEE, 2019.
[31] Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432, 2013.
[32] Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. PACT: parameterized clipping activation for quantized neural networks. CoRR, abs/1805.06085, 2018.
[33] Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. Qdrop: Randomly dropping quantization for extremely low-bit post-training quantization. 2022.
[34] Lingchuan Meng and John Brothers. Efficient winograd convolution via integer arithmetic. CoRR, abs/1901.01965, 2019.
[35] Guangli Li, Lei Liu, Xueying Wang, Xiu Ma, and Xiaobing Feng. Lance: efficient low-precision quantized winograd convolution for neural networks based on graphics processing units. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020, pages 3842-3846. IEEE, 2020.
[36] Renzo Andri, Beatrice Bussolino, Antonio Cipolletta, Lukas Cavigelli, and Zhe Wang. Going further with winograd convolutions: Tap-wise quantization for efficient inference on 4x4 tiles. In 55th IEEE/ACM International Symposium on Microarchitecture, MICRO 2022, Chicago, IL, USA, October 1-5, 2022, pages 582-598. IEEE, 2022.
[37] Shmuel Winograd. Arithmetic complexity of computations, volume 33. Siam, 1980.
[38] Robert V. Brill. Applied Statistics and Probability for Engineers, volume 46. 2004.
[39] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2015.
A Winograd transformation matrices
Depending on the particular choice of Winograd domain (i.e., polynomial domain), transformation matrices  $A$ ,  $B$ , and  $G$  in the Winograd algorithm can be different. In the paper, we present that the most popular interpolation points for  $\mathrm{F}(2,3)$  are  $[0, +1, -1]$  and then these transformation matrices can be constructed as follows:
$$
A ^ {T} = \left[ \begin{array}{c c c c} 1 & 1 & 1 & 0 \\ 0 & 1 & - 1 & - 1 \end{array} \right], \quad B ^ {T} = \left[ \begin{array}{c c c c} 1 & 0 & - 1 & 0 \\ 0 & 1 & 1 & 0 \\ 0 & - 1 & 1 & 0 \\ 0 & 1 & 0 & - 1 \end{array} \right], \quad G = \left[ \begin{array}{c c c} 1 & 0 & 0 \\ \frac {1}{2} & \frac {1}{2} & \frac {1}{2} \\ \frac {1}{2} & - \frac {1}{2} & \frac {1}{2} \\ 0 & 0 & 1 \end{array} \right] \tag {30}
$$
For  $\mathrm{F}(4,3)$  and  $\mathrm{F}(6,3)$ , we choose the same transformation matrices as BQW [24]. For  $\mathrm{F}(4,3)$ , the Winograd transformation matrices are as follows:
$$
A ^ {T} = \left[ \begin{array}{c c c c c c} 1 & 1 & 1 & 1 & 1 & 0 \\ 0 & 1 & - 1 & 2 & - 2 & 0 \\ 0 & 1 & 1 & 4 & 4 & 0 \\ 0 & 1 & - 1 & 8 & - 8 & 1 \end{array} \right], \tag {31}
$$
$$
B ^ {T} = \left[ \begin{array}{r r r r r r} 4 & 0 & - 5 & 0 & 1 & 0 \\ 0 & - 4 & - 4 & 1 & 1 & 0 \\ 0 & 4 & - 4 & - 1 & 1 & 0 \\ 0 & - 2 & - 1 & 2 & 1 & 0 \\ 0 & 2 & - 1 & - 2 & 1 & 0 \\ 0 & 4 & 0 & - 5 & 0 & 1 \end{array} \right], \tag {32}
$$
$$
G = \left[ \begin{array}{c c c} \frac {1}{4} & 0 & 0 \\ - \frac {1}{6} & - \frac {1}{6} & - \frac {1}{6} \\ - \frac {1}{6} & \frac {1}{6} & - \frac {1}{6} \\ \frac {1}{2 4} & \frac {1}{1 2} & - \frac {1}{6} \\ \frac {1}{2 4} & - \frac {1}{1 2} & - \frac {1}{6} \\ 0 & 0 & 1 \end{array} \right] \tag {33}
$$
For  $\mathrm{F}(6,3)$ , the Winograd transformation matrices are as follows:
$$
A ^ {T} = \left[ \begin{array}{c c c c c c} 1 & 1 & 1 & 1 & 1 & 0 \\ 0 & 1 & - 1 & 2 & - 2 & 0 \\ 0 & 1 & 4 & 4 & 0 \\ 0 & 1 & - 1 & 8 & - 8 & 1 \end{array} \right], \tag {34}
$$
$$
B ^ {T} = \left[ \begin{array}{c c c c c c c c} 1 & 0 & - \frac {2 1}{4} & 0 & \frac {2 1}{4} & 0 & - 1 & 0 \\ 0 & 1 & 1 & - \frac {1 7}{4} & - \frac {1 7}{4} & 1 & 1 & 0 \\ 0 & - 1 & 1 & \frac {1 7}{4} & - \frac {1 7}{4} & - 1 & 1 & 0 \\ 0 & \frac {1}{2} & \frac {1}{4} & - \frac {5}{2} & - \frac {5}{4} & 2 & 1 & 0 \\ 0 & - \frac {1}{2} & \frac {1}{4} & \frac {5}{2} & - \frac {5}{4} & - 2 & 1 & 0 \\ 0 & 2 & 4 & - \frac {5}{2} & - 5 & \frac {1}{2} & 1 & 0 \\ 0 & - 2 & 4 & \frac {5}{2} & - 5 & - \frac {1}{2} & 1 & 0 \\ 0 & - 1 & 0 & \frac {2 1}{4} & 0 & - \frac {2 1}{4} & 0 & 1 \end{array} \right], \tag {35}
$$
$$
G = \left[ \begin{array}{c c c} 1 & 0 & 0 \\ - \frac {2}{9} & - \frac {2}{9} & - \frac {2}{9} \\ - \frac {2}{9} & \frac {2}{9} & - \frac {2}{9} \\ \frac {1}{9 0} & \frac {1}{4 5} & - \frac {2}{4 5} \\ \frac {1}{9 0} & - \frac {1}{4 5} & \frac {2}{4 5} \\ \frac {3 2}{4 5} & \frac {1 6}{4 5} & \frac {8}{4 5} \\ \frac {3 2}{4 5} & - \frac {1 6}{4 5} & \frac {8}{4 5} \\ 0 & 0 & 1 \end{array} \right] \tag {36}
$$
B Derivatives of transformation matrices
In the paper, in order to align these transformation procedures after quantization, we propose to adjust transformation matrices via an optimization procedure as follows:
$$
\underset {A, B, G} {\operatorname {a r g m i n}} \quad \mathbb {E} _ {X \sim \mathcal {D}} \left[ \Sigma_ {f} ^ {C _ {o}} \| A ^ {T} \left(\Sigma_ {c} ^ {C _ {i}} Q \left(B ^ {T} X _ {c} B\right) \odot Q \left(G W _ {f, c} G ^ {T}\right)\right) A - Y _ {f} \| ^ {2} \right] \tag {37}
$$
By using the straight-through estimator [31] to approximate the gradient through the round function as a pass-through operation, we can obtain the derivatives of  $A$ ,  $B$  and  $G$ . In this paper, we directly present the derivative of  $B$ . Here, a more comprehensive derivation is provided as follows:
$$
\begin{array}{l} \frac {\partial \mathcal {L}}{\partial B _ {i j}} = \Sigma_ {f} ^ {C _ {o}} \operatorname {t r} \left\{\frac {\partial \mathcal {L}}{\partial O _ {f} ^ {T}} \cdot \frac {\partial O _ {f}}{\partial B _ {i j}} \right\} (38) \\ = \Sigma_ {f} ^ {C _ {o}} \operatorname {t r} \left\{\frac {\partial \mathcal {L}}{\partial O _ {f} ^ {T}} \cdot \left[ \Sigma_ {c} ^ {C _ {i}} \left(\delta_ {j i} X _ {c} B\right) \odot Q \left(V _ {f, c}\right) + \left(B ^ {T} X _ {c} \delta_ {i, j}\right) \odot Q \left(V _ {f, c}\right) \right] \right\} (39) \\ = \Sigma_ {f} ^ {C _ {o}} \Sigma_ {c} ^ {C _ {i}} \operatorname {t r} \left\{\frac {\partial \mathcal {L}}{\partial O _ {f} ^ {T}} \cdot \left[ \left(\delta_ {j i} X _ {c} B\right) \odot Q \left(V _ {f, c}\right) \right] + \frac {\partial \mathcal {L}}{\partial O _ {f} ^ {T}} \cdot \left[ \left(B ^ {T} X _ {c} \delta_ {i j}\right) \odot Q \left(V _ {f, c}\right) \right] \right\} (40) \\ = \Sigma_ {f} ^ {C _ {o}} \Sigma_ {c} ^ {C _ {i}} \operatorname {t r} \left\{\left(\delta_ {j i} X _ {c} B\right) ^ {T} \cdot \left[ \frac {\partial \mathcal {L}}{\partial O _ {f}} \odot Q \left(V _ {f, c}\right) \right] + \left(B ^ {T} X _ {c} \delta_ {i j}\right) ^ {T} \cdot \left[ \frac {\partial \mathcal {L}}{\partial O _ {f}} \odot Q \left(V _ {f, c}\right) \right] \right\} (41) \\ = \Sigma_ {f} ^ {C _ {o}} \Sigma_ {c} ^ {C _ {i}} \left[ X _ {c} B \cdot \left(\frac {\partial L}{\partial O _ {f}} \odot Q \left(V _ {f, c}\right)\right) ^ {T} \right] _ {i j} + \left[ X _ {c} ^ {T} B \cdot \left(\frac {\partial L}{\partial O _ {f}} \odot Q \left(V _ {f, c}\right)\right) \right] _ {i j} (42) \\ \end{array}
$$
We have obtained the derivative of  $B_{ij}$ , and now we can provide the expression for the derivative of  $B$ :
$$
\frac {\partial \mathcal {L}}{\partial B} = \Sigma_ {f} ^ {C _ {o}} \Sigma_ {c} ^ {C _ {i}} X _ {c} B \left(\frac {\partial L}{\partial O _ {f}} \odot Q \left(V _ {f, c}\right)\right) ^ {T} + X _ {c} ^ {T} B \left(\frac {\partial L}{\partial O _ {f}} \odot Q \left(V _ {f, c}\right)\right) \tag {43}
$$
The derivatives of  $A$ ,  $G$  and  $O_{f}$  can be computed in a similar manner:
$$
\frac {\partial \mathcal {L}}{\partial A} = \Sigma_ {f} ^ {C _ {\circ}} O _ {f} ^ {T} A \left(A ^ {T} O _ {f} A - Y _ {f}\right) + O _ {f} A \left(A ^ {T} O _ {f} A - Y _ {f}\right) ^ {T} \tag {44}
$$
$$
\frac {\partial \mathcal {L}}{\partial G} = \Sigma_ {f} ^ {C _ {o}} \Sigma_ {c} ^ {C _ {i}} \left(\frac {\partial L}{\partial O _ {f}} \odot Q (U _ {c})\right) G W _ {f, c} ^ {T} + \left(\frac {\partial L}{\partial O _ {f}} \odot Q (U _ {c})\right) ^ {T} G W _ {f, c} \tag {45}
$$
$$
\frac {\partial \mathcal {L}}{\partial O _ {f}} = 2 A \left(A ^ {T} O _ {f} A - Y\right) A ^ {T} \tag {46}
$$
C Optimal quantization scale for Guassion variables
In Theorem 1, in order to demonstrate that the optimal per-pixel scale  $S$  can be factorized into vectors, we rely on the conclusion that the optimal scale  $s^*$  to minimize the mean-square error of quantization of Gaussian variables  $z \sim \mathcal{N}(0, \sigma^2)$  is proportional to  $\sigma$ , i.e.,  $s^* = K\sigma$ , where  $K$  is a constant. Here, we will provide a proof of it.
Theorem 2. Assuming  $z \sim \mathcal{N}(0, \sigma^2)$ , the optimal scale  $s^*$  to minimize the mean-square error of quantization of  $z$  is proportional to the standard deviation  $\sigma$ , i.e.,  $s^* = K\sigma$ , where  $K$  is a constant.
Proof. Because  $z \sim \mathcal{N}(0, \sigma^2)$ ,  $z$  can be reparameterized as  $z = \sigma \cdot u$ , where  $u \sim \mathcal{N}(0, 1)$ .
$$
\begin{array}{l} \mathbf {E} \left[ (Q (z) - z) ^ {2} \right] = \int_ {- \infty} ^ {\infty} p _ {z} (z) (Q (z) - z) ^ {2} d z (47) \\ = \int_ {- \infty} ^ {\infty} p _ {u} (u) \left(Q (\sigma u) - \sigma u\right) ^ {2} d u (48) \\ = \int_ {- \infty} ^ {\infty} p _ {u} (u) \left(\operatorname {c l i p} \left(\left\lfloor \frac {\sigma u}{s} \right\rfloor , - q _ {\min }, q _ {\max }\right) \cdot s - \sigma u\right) ^ {2} d u (49) \\ = \sigma^ {2} \int_ {- \infty} ^ {\infty} p _ {u} (u) \left(\operatorname {c l i p} \left(\left\lfloor \frac {u}{s / \sigma} \right\rfloor , - q _ {\min }, q _ {\max }\right) \cdot \frac {s}{\sigma} - u\right) ^ {2} d u (50) \\ = \sigma^ {2} h \left(\frac {s}{\sigma}\right) (51) \\ \end{array}
$$
Eq. (47) can be treated as a function of  $s / \sigma$  when solving for  $s$  with  $\sigma$  as a known value. Assuming  $K$  minimizes function  $h(x)$ , i.e.,  $K = \text{argmin} h(x)$ , we have:
$$
s ^ {*} = \underset {s} {\operatorname {a r g m i n}} \mathbf {E} \left[ (Q (z) - z) ^ {2} \right] = \underset {s} {\operatorname {a r g m i n}} \sigma^ {2} h \left(\frac {s}{\sigma}\right) = K \cdot \sigma \tag {52}
$$
D Experiments on other architectures
In Section 5, we compare our methods to previous work BQW[24] on the ResNet model family with comprehensive experiment settings, including various bit widths, tile sizes, and datasets. Here, we present a similar analysis for two other popular architectures VGG and SqueezeNet using the Cifar-10 dataset. The results are shown in Table 1 and Table 2. These results align with our analysis in Section 5. Our PTQ-Aware Winograd (PAW) method outperforms the strong baseline introduced in Section 5 and our FSQ method is well-compatible with PAW.
Table 1: PTQ results of VGG11 on CIFAR-10.
Model	Tile	Bits	Partial Quantization	Full Quantization
Baseline	PAW	FSQ	FSQ+PAW
VGG-11	F(4,3)	6	89.13	91.56	86.59	91.55
8	92.02	92.28	90.82	91.83
(92.02%)	F(6,3)	6	75.10	89.94	68.98	90.34
8	91.27	91.88	88.44	91.63