SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To demonstrate the efficacy of our TexQ, we conduct experiment in 3/4-bit case, since high accuracy can be easily achieved with a larger bit width. For instance, the advanced AdaDFQ [47] trails full precision with a  $0.08\%$  top-1 accuracy in 5-bit ResNet-20 case. In this section, WBAB indicates the weights and activations are quantized to B-bit. Best results in boldface.
4.2.1 CIFAR-10/100
We compare the performance against the advanced ZSQ methods on CIFAR-10/100. As presented in Table 1, our TexQ is observed to achieve state-of-the-art among the competitors, improving the top-1 accuracy by  $3.13\% / 1.58\%$  in 3-bit CIFAR-10/100 case compared to the advanced AdaDFQ. Similar
results can be observed in the 4-bit case. In particular, the top-1 accuracy of our TexQ exceeds that of the same framework with real data in 4-bit case, demonstrating that the proposed TexQ can fully extract the feature distribution in simple datasets.
Table 1: Results of ResNet-20 on CIFAR-10/100.
Bit width	Method	Top-1 Accuracy(%)
CIFAR-10	CIFAR-100
W4A4	Full-precision	93.89	70.33
Real data	91.52	66.80
GDFQ [24] (ECCV 2020)	90.11	63.75
ARC [58] (IJCAI 2021)	88.55	62.76
Qimera [48] (NeurIPS 2021)	91.26	65.10
IntraQ [29] (CVPR 2022)	91.49	64.98
ARC+AIT [59] (CVPR 2022)	90.49	61.05
AdaSG [46] (AAAI 2023)	92.10	66.42
AdaDFQ [47] (CVPR 2023)	92.31	66.81
TexQ (Ours)	92.68	67.18
W3A3	GDFQ [24, 47] (ECCV 2020)	75.11	47.61
ARC [58] (IJCAI 2021)	-	40.15
Qimera [48, 47] (NeurIPS 2021)	74.43	46.13
IntraQ [29] (CVPR 2022)	77.07	48.25
ARC+AIT [59] (CVPR 2022)	-	41.34
AdaSG [46] (AAAI 2023)	84.14	52.76
AdaDFQ [47] (CVPR 2023)	84.89	52.74
TexQ (Ours)	86.47	55.87
4.2.2 ImageNet
We further compare with competitors on challenging ImageNet, the results are presented in Table 2.
Table 2: Results of ResNet-18, MobileNetV2 and ResNet-50 on ImageNet.
Bit width	Method	Top-1 Accuracy(%)
ResNet-18	MobileNetV2	ResNet-50
Full-precision	71.47	72.49	77.73
W4A4	GDFQ [24] (ECCV 2020)	60.60	59.43	54.16
ZAQ [25] (CVPR 2021)	52.64	0.10	53.02
ARC [58] (IJCAI 2021)	61.32	60.13	64.37
Qimera [48] (NeurIPS 2021)	63.84	61.62	66.25
IntraQ [29] (CVPR 2022)	66.47	65.10	-
ARC+AIT [59] (CVPR 2022)	65.73	66.47	68.27
AdaSG [46] (AAAI 2023)	66.50	65.15	68.58
AdaDFQ [47] (CVPR 2023)	66.53	65.41	68.38
TexQ (Ours)	67.73	67.07	70.72
W3A3	GDFQ [24, 47] (ECCV 2020)	20.23	1.46	0.31
ARC [58] (IJCAI 2021)	23.37	14.30	1.63
Qimera [48, 47] (NeurIPS 2021)	1.17	-	-
AdaSG [46] (AAAI 2023)	37.04	26.90	16.98
AdaDFQ [47] (CVPR 2023)	38.10	28.99	17.63
TexQ (Ours)	50.28	32.80	25.27
Bit width For low bit width case, conventional ZSQ methods suffer from a huge accuracy loss, while our TexQ performs good generalization capability. Specifically, for 4-bit ResNet-18 case,
TexQ achieving  $67.73\%$  accuracy, outperforming AdaDFQ by  $1.2\%$ . In 3-bit case, TexQ reaches  $50.28\%$  outstanding accuracy, outperforming AdaDFQ by  $12.18\%$ , with a standard deviation of 0.34 in 5 repeated experiments. Similar results were obtained in MobileNetV2.
Network Size Larger models are likely lead to poor performance with existing ZSQ methods, especially in low bit width case. For example, in 3-bit case, Qimera [48, 47] performs well on the small ResNet-20, while achieves only  $1.17\%$  accuracy on the larger ResNet-18. In contrast, our TexQ achieved  $55.87\%$  ultra-low when quantizing ResNet-18 to 3-bit, and leads AdaDFQ with  $7.64\%$  on 3-bit ResNet50 case. These results again demonstrate the effectiveness of our method.
Hyperparameters After empirical initialization, the hyperparameters  $\mathrm{k}$ ,  $\theta_U$ ,  $\theta_L$ , and  $\varepsilon$  are searched for their optimal value with grid search by quantizing ResNet-18 to 3-bit on ImageNet, as displayed in Figure 7. The optimal configurations are  $\mathrm{k} = 9$ ,  $\theta_U = 0.3$ ,  $\theta_L = 0.5$ , and  $\varepsilon = 0.015$ . The hyperparameter  $\mathrm{k}$  represents the number of dominant texture elements, whose optimal value of  $\mathrm{k} = 9$  is consistent with what we observe in the real samples. The sum of the optimal values of  $\theta_U$  and  $\theta_L$  is 0.8, which fits our assumptions about the proportion of dominant texture elements. To avoid complex searches, these parameters were used for all experiments. While this may not be optimal for all networks, it is sufficient to exhibit the advanced performance of TexQ.
Figure 7: Influence of the hyperparameters.
Modules Ablation on key modules including  $\mathcal{L}^{C}_{\text{LAWS}}$  (Eq. 6),  $\mathcal{L}^{C}_{\text{L-BNS}}$  (Eq. 7) and mixup knowledge distillation (MKD, Eq. 9) is conducted. As presented in Table 3, dropping one or two of them results in an accuracy loss. The largest accuracy loss  $(8.37\%)$  is observed when both  $\mathcal{L}^{C}_{\text{LAWS}}$  and  $\mathcal{L}^{C}_{\text{L-BNS}}$  are removed, which indicates their cooperative relationship:  $\mathcal{L}^{C}_{\text{L-BNS}}$  loosens the shallow layers constraint, facilitating  $\mathcal{L}^{C}_{\text{LAWS}}$  to calibrate.
Table 3: Ablations on modules. We report the top-1 accuracy of 3-bit ResNet-18 on ImageNet.
\( \mathcal{L}^{C}_{\text{LAWS}} \)	\( \mathcal{L}^{C}_{\text{L-BNS}} \)	MKD	Top-1	Accuracy(%)
✓	✓	✓	50.94
✓	✓	49.03
✓	✓	43.31
✓	✓	49.63
✓	42.65
✓	49.00
✓	42.57
5.1 Why does TexQ work?
Mitigating the domain gap Our synthetic samples with calibration match the real distribution better, as presented in Figure 1 and Figure 6. Further, Figure 5 shows our synthetic sample has a more detailed texture distribution similar to the real one.
(a) ImageNet (IntraQ)
(b) ImageNet (Ours)
(c) CIFAR10 (IntraQ)
(d) CIFAR10 (Ours)
(e) CIFAR100 (IntraQ)
(f) CIFAR100 (Ours)
Figure 8: Visualization of synthetic samples. For ImageNet and CIFAR10/100, samples are obtained with ResNet-18 and ResNet-20, respectively. The advanced IntraQ was selected as a competitor.
(a) Without Calibration
Figure 9: Visualization of the running mean of BNS in ResNet-20.
(b) Introduce Calibration
(a) ImageNet
Figure 10: Cosine similarity confusion matrix of samples. Synthetic samples are obtained with ResNet-18.
(c) IntraQ
Cosineresimilarity
Increasing inter-class distance As displayed in Figure 9a, the class centers of samples from conventional methods is unclear, limiting the inter-class distance. The situation is improved by introducing calibration centers, as shown in Figure 9b.
Enhancing sample diversity Figure 1 shows our synthetic samples scatter a lot compared with IntraQ, implying the enhancement of sample diversity. To put it bluntly, synthetic images from different methods are visualized in Figure 8, showing the improvement in color and texture diversity of our samples. Further, the intra-class diversity is observed through cosine similarity confusion matrix of samples in Figure 10, showing that our samples reach a similar diversity to the real ones.
5.2 Limitations and future work
This work is built upon the concept of "texture", and the texture filters adopted are designed manually. Thus, the application to other tasks with different modality would be limited. We envision adding training process to the filters, which would generalize the calibration method proposed to other modality. Low-bit quantization remains a challenge. For instance, TexQ achieve  $70.72\%$  accuracy by quantizing ResNet-50 to 4-bit, while it drops to  $25.27\%$  in 3-bit case. The introduction of advanced distillation methods holds promise for more better results.
In this paper, we observe a non-negligible detailed texture distribution in the real samples. To retain this property in synthetic samples, we introduce synthetic calibration images and centers to calibrate the generator. To diversify the samples, mixup knowledge distillation module is introduce to create diversified samples for fine-tuning. Extensive experiments show our state-of-the-art performance on mainstream networks and datasets, especially for low bit width quantization.
This work was supported by the Science and Technology Research Program of Shenzhen City (No. KCXFZ20201221173207022, No.WDZC2020200821141349001) and the Jilin Fuyuan Guan Food Group Co., Ltd.
[58] Baozhou Zhu, Peter Hofstee, Johan Peltenburg, Jinho Lee, and Zaid Alars. Autorecon: Neural architecture search-based reconstruction for data-free. In International Joint Conference on Artificial Intelligence, 2021.
[59] Kanghyun Choi, Hye Yoon Lee, Deokki Hong, Joonsang Yu, Noseong Park, Youngsok Kim, and Jinho Lee. It's all in the teacher: Zero-shot quantization brought closer to the teacher. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8311-8321, 2022.