SynQ: Accurate Zero-shot Quantization by Synthesis-aware Fine-tuning
Authors: Minjun Kim
Jongjin Kim
U Kang
Venue: ICLR 2025 Poster (2025)
Keywords: Network Quantization
Zero-shot Quantization
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: How can we accurately quantize a pre-trained model without any data?
Quantization algorithms are widely used for deploying neural networks on resource-constrained edge devices.
Zero-shot Quantization (ZSQ) addresses the crucial and practical scenario where training data are inaccessible for privacy or security reasons.
However, three significant challenges hinder the performance of existing ZSQ methods: 1) noise in the synthetic dataset, 2) predictions based on off-target patterns, and the 3) misguidance by erroneous hard labels.
In this paper, we propose SynQ (Synthesis-aware Fine-tuning for Zero-shot Quantization),
a carefully designed ZSQ framework to overcome the limitations of existing methods.
SynQ minimizes the noise from the generated samples by exploiting a low-pass filter.
Then, SynQ trains the quantized model to improve accuracy by aligning its class activation map with the pre-trained model.
Furthermore, SynQ mitigates misguidance from the pre-trained model's error by leveraging only soft labels for difficult samples.
Extensive experiments show that SynQ provides the state-of-the-art accuracy, over existing ZSQ methods.

Published as a conference paper at ICLR 2025
SYNQ: ACCURATE ZERO-SHOT QUANTIZATION BY SYNTHESIS- AWARE
Minjun Kim, Jongjin Kim & U Kang Scoul National University, Scoul, South Korea (min [minjun. kimm j2kim99, ukang) @snu. ac. kr
ABSTRACT
How can we accurately quantize pre- treetrained model without any data? Quantization algorithms are widely used for deploying neural networks on resource-constrained edge devices. Zero-shot Quantization (ZSQ) addresses the crucial and practical scenario where training data are inaccessible for privacy or security reasons. How- ever, three significant challenges hinder the performance of existing ZSQ methods: 1) noise the synthetic dataset, 2) predictions based on off-target patterns, and the 3) misguidance by erroneous hard labels. In this paper, propose SYNQ (Synthesis-awaad Fine-tuning for Zero-shot Quantization), a carefully designed ZSQ framework overcome the limitations of existing methods. SYNQ minimizes the noise from the generated samples by exploiting a low-pass filter. Then, SYNQ trains the quantized model improve accuracy by aligning its class activation map with the pre-trained model. Furthermore, SYNQ mitigates misguidance from the pre-trained model's error by leveraging only soft labels for difficult samples. Extensive experiments show that SYNQ provides the state-of-the-ie art accuracy, over existing ZSQ methods.
INTRODUCTION
How can we accurately quantize a pre-trained model without any data? Despite the success of deep neural networks in various domains, deploying them
cesstrce-oonttraineed edge devices remains challenging due to the limited computing capabilities. Addressing this challenge involves network compression (Cheng et al.
2018 Deng et al. [2200] Park al. 2024b), where quantization Piao et al.
methods (Li et al. Piao
2022 Gholami et al. (022) represent the full-precision model
with low-bit numbers, achieving high compression rate and accelerated inference with minimal
performance degradation compared to other methods such as pruning WWang et al. 2022 Park et al.
2024a; He & Xiao 2024). knowledge distillation (Kim al. [02T]] Tran | [0.2]2222c Cho & Kang
2022]Jeor et al. 2023a Xie et al. 2023), and low-rank approximation (Jang et al. 2023). Zero-shot
Quantization (ZSQ) (Nagel et al. (2019) further advances this field by permitting quantization without
any training data. The importance of this approach is evident in real-world contexts where the training data are unavailable for privacy and security reasons (Sharma et al. 2021).
Among the various existing works (Yoo et al. 2019: Zhang et al. 202 Guo al. 2022). methods
that fine-tune the quantized model with the synthetic dataset exhibit outstanding performance (Liu et al. 2021aa] Zhong al. 2022b Fan et al.) (2024). Specifically, recent methods generate synthetic samples resembling the real sample distribution by leveraging key aspects from the pre-trained model, such
statistics (Cai al., 2020), latent embeddings (Choi al.) [021), or texture feature distribution (Chen et al. 2023). However, we observe that three major limitations still hinder the performance when utilizing synthetic datasets (see Section 3p.
Noise in the synthetic dataset. Synthetic datasets have distinct high-frequency noise unlike real images that concentrate on low frequencies (see Figures and 55) This discrepancy results in inefficient fine-tuning of quantized models, thereby directly reducing model performance.
Predictions based on off-target patterns. Quantized model from existing methods rely on incorrect image patterns for predictions (see Figure 2 Such tfr-target reliance limits the quantized model in identifying key areas necessary for accurate classification.
"Corresponding Author.

---

Published as a conference paper at ICLR 2025
Centralized in low frequency
Distributed all over the frequency domain
8
6
Bota
Samples
Magnitude spectrum (log scall)
Samples
Magnitude spectrum (log scale)
(a) ImageNet dataset
(b) Synthetic dataset (TexQ)
Figure 1: Comparison between (a) real images in ImageNet dataset and (b) generated samples in the synthetic dataset from TexQ (Chen et al., 2023). Each displays samples labeled timber wolf, tobacco shop. aircraft carrier, and beaker. We present the average magnitude spectrum for a randomly selected batch of 256 images from each dataset, highlighting their distinct differences.
(a) Input
ResNet:
(c) TexQ
(d) SYNQ (Ours)
Figure 2: Grad-CAM (Selvaraju et al. 2017) plot of the (a) input by the (b) pre-trained ResNet- 18 model on ImageNet dataset, the (c) 3bit quantized model by TexQ. and the (d) 3bit quantized model by SYNQ. While TexQ fails to capture the correct image region, SYNQ captures the region closely matching the pre-trained model.
Misguidance by erroneous hard labels. Hard labels of difficult samples are often incorrect in synthetic dataset, leading to misguided fine-tuning and ultimately harming the model (see Figure
We propose SYNQ (Synthesis-awwi Fine-tuning for Zero-shot Quantization), accurate ZSQ fine-tuning method to overcome the limitations of the existing methods that fine-tune with synthetic datasets. SYNQ clears noise from the generated samples within the synthetic dataset by applying a low-pass filter. Then, SYNQ ensures that the quantized model predicts from the correct image region by optimizing the class activation map (CAM) alignment loss distill object localization knowledge. Furthermore, SYNQ mitigates misguidance from errors of the pre-trained model by using only soft labels for difficult samples. Experimental results show that SYNQ achieves the state-of-the-arr performance, improving the image classification accuracy of the quantized model up to 1.74%p compared existing methods (see Table I.. SYNQ both powerful and versatile, seamlessly integrating into any ZSQ methods that fine-tune with synthetic datasets, regardless of Appendices C.6 and
model type, quantization bits, or dataset (see Sections 5.2 5.3
Our contributions are summarized as follows:
Observation. Our observations clearly outline three significant challenges faced by existing ZSQ methods utilizing synthetic datasets: 1) noise synthetic datasets. 2) predictions based on off ofr-target patterns, and 3) misguidance by erroneous hard labels (see Figures
5)d
Algorithm. We propose SYNQ, accurate ZSQ method to overcome the limitations of fine- tuning with synthetic datasets. SYNQ exploits a low-pass filter to minimize noise, aligns the class activation map to ensure prediction from the correct image region, and leverages soft labels for difficult samples to prevent misguidance from erroneous hard labels (see Section (4).
Experiments. We experimentally show that SYNQ consistently outperforms existing ZSQ methods on various models and datasets, achieving classification accuracy improvement of up 1.74%p (see Section and Appendix C
Reproducibility. All our implementation and datasets are available at snudm- at.a r albb Syn@.
com/
2

---

Published as a conference paper at ICLR 2025
2 PRELIMINARIES AND PROBLEM DEFINITION
We introduce the ZSQ (Zero-shot Quantization) problem and describe the preliminaries. Appendix A contains the detailed descriptions of frequently used notations in this paper.
2.1 ZERO-SHOT QUANTIZATION
In this work, we follow the typical two-step scheme (Choi et al. 2021 Li et al. 2023a to quantize a pretrained model. First, we generate the synthetic dataset that resembles the original dataset using the pre-trained model. Second, we fine-tune the quantized model with generated samples.
The goal of the first step is to produce synthetic dataset {x, of length N with corresponding labels (y: using the pre-trained model with parameters 0. We utilize noise optimization (Cai et al. 2020; Zhong et al.) 2022b), where we initialize the synthetic dataset and labels as random Gaussian noises and randomly assigned classes, respectively; then iteratively update the synthetic dataset {x, J Specifically, we minimize Batch Normalization Statistics (BNS) loss LaNS and Inception Loss (IL) LIL in Equation ( with hyperparameter a balancing them.
N
min LIL + aLBNS, where LIL = N (x
().
111
1 -1 LANS T
(0, (|o'(0) -0((0, (x)))=))
where the Ith batch normalization (BN) layer of the pre-trained model with parameters O (out of BN layers) stores the running mean (0) and standard deviation (0) of the training dataset. The mean (0, {x,) and standard deviation o' (0, (x; ) | calculated on {x,) using f. ((; 0) denotes the probability distribution by parameters and CE(.,)) stands for cross-entropy loss.
The goal of the second step is obtain the quantized model with parameters 09, using the pre-trained model with parameters and synthetic dataset { xx } with labels { We first quantize the pre-trained model with Rounding-To-Neant model with parameters d with the synthetic dataset from the previous step. For strong performance, we train the quantized model by minimizing two losses, cross-entropy loss C.E(.,)) with hard label y; and KL divergence loss (-(-ll-) which transfers knowledge from the pre-trained model. Note that we directly update the quantized model, while inferencing with its dequantized parameters. Equation (2) incorporates the two loss functions with balancing hyperparameter ACE.
(RTN) (Gupta et al. (2015). then fine-tune the quantized
N min Lzsq min 6u
N
2.2 DIFFICULTY OF AN IMAGE
Difficulty of image represents how easily the model 0 can misclassify the image Among various methods (Ribeiro et al. 2016: Lin et al.) 2017: Kishida & Nakayama, 2019 Scheidegger et al. 2021) to evaluate the difficulty of the model in correctly classifying image, follow the probability- approach (Li et al. 2019) which is used in previous ZSQ methods (Li et al., 2003a).. The difficulty 5(x;, () is determined by how low the model's predicted probability is for the correct label as described in Equation (3).
8(x, 0) = - Qy. (x;; 8),
(3)
where Qy:(x;)) (x;; 8) the probability of label y: predicted by the model with parameters 6. This definition employs the true label to specifically highlight the model's 's ambiguity toward the image. Models display an error rate of 0 for difficulties below 0.5 and an increasing error rate for higher difficulties shown in Figure indicating either incorrectness uncertainty in model predictions.
3

---

Published as a conference paper at ICLR 2025
2.3 PROBLEM DEFINITION
Given a pre-trained image classification model and quantization bits, Zero-shot Quantization (ZSQ) targets to optimize the quantized model to maintain performance without any real images. We give the formal definition as in Problem
Problem 1 (Zero-shot Quantization). We have a pre-trained model with parameters 0 and quan- rization bits B. Zero-shot quantization is to optimize the quantized model with parameters 80 for maximum accuracy within the Bbit limit without the use of real data.
3 OBSERVATION
We present the observations that highlight the three major challenges posed to existing methods.
Noise in the synthetic dataset. The synthetic dataset is noisy, it is produced by noise optimization that starts with a Gaussian noise. In rigure[] we compare (a) real images from the ImageNet dataset with (b) generated samples from the synthetic dataset following TexQ (Chen al. 2023). Generated samples display distinct grainy noise that leads to an evenly distributed frequency magnitude spectrum, in contrast to the real images whose magnitude is primarily concentrated in the low-frequency area. Note that we investigate the frequency magnitude spectrum by applying Fourier transform (Cooley & Tukey 1965 Park al. 2021] 2024c) on images. Moreover, Figure shows the severe differences in amplitude distributions between (a) real images and (b) generated samples (refer to Appendix CC3 for results on other baselines and datasets). This frequency domain discrepancy challenges the quantized model to restore classification performance during fine-tuning.
Predictions based on off-target patterns. Fine-tuning with synthetic dataset leads the quantized model to rely incorrect image patterns for predictions. Figure 2 shows the discriminative regions that Grad-CAM (Selvaraju et al. 2017) identifies for the ground-truth class across three models: (a) pre-trained ResNet- 18 model on the ImageNet dataset, (b) 3bit quantized model by TexQ (Chen et al. (023), and (c) 3bit quantized model by our SYNQ. Note that TexQ predicts based on wrong regions, unlike the pre-trained model which accurately captures critical regions (refer to Appendix C.4 for further analysis). This mismatch definitely harms the quantization performance.
CIFAR-10
Eirol rapidly grows
CIFAR- 100
ImageNett
Rate
0.6
0.4
0.2-
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Difficulty
Misguidance by erroncous hard labels. Reliance on erro- ncous hard labels in the synthetic dataset leads to misguided fine-tuning outcomes. Figure 3 shows the growing error rates for pre-trained ResNet (He all 2016) models CIFAR-10, 10, CIFAR-100, and ImageNet datasets as image difficulty increases. Difficulty of image is defined Equation (3), detailed in Section 2.2 Consequently, the pre-trained model often mislabels samples with difficulty level over 0.5. These erroneous hard labels of difficult samples damage quantization performance.
Figure 3: Error rates of pre-trained ResNet-20 on CIFAR-10 (yel- low) and CIFAR- 100 (green), and ResNet- 18 on ImageNet (purple) by difficulty. Error rate rapidly grows as the difficulty exceeds 0.5.
4 PROPOSED METHOD
4.1 OVERVIEW
We propose SYNQ (Synthesis-awarc aware Fine-tuning for Zero-shot Quantization), an accurate Zero-shot Quantization (ZSQ) method addressing the following three major challenges of existing methods that fine-tune with the synthetic dataset. These are the three main challenges that must be tackled:
C1. Noise in the synthetic dataset. Previous methods fine-tune the quantized model with a noisy synthetic dataset, which exhibits a distribution discrepancy of frequency domain compared to real images. How can we minimize the effect of the noise within the generated samples?
C2. Prediction based on off-target patterns. The quantized model predicts based on incorrect image regions that are unlike those observed in the pre-trained model. How can we optimize the quantized model to more accurately utilize on-target patterns?
C3. Misguidance from erroneous hard labels. Despite the high error rate of difficult samples, existing works trust erroneous hard labels. How can we address the misguidance by hard labels?
4

---

Published as a conference paper at ICLR 2025
Idea Low-pass Iuter
Synlhetic dalasel
Pre-trained model
Label
Backpropegation
Pre- Pre-trained model
Probabilities
Irlea 3
vix:0)
8 S
Dilcuuly
Idea 2
Soft label
True (Easy sample)
(xc: 0)
Activation map
CAM
KL
CE
Hard label
(x: 0')
Quantized model
Figure 4: Overall architecture of SYNQ. Our main ideas are 1) low-pass filter, 2) alignment of class activation map, and 3) soft labels for difficult samples. See Section 4 for details.
We address these challenges with the following main ideas:
11. Low-pass filter (Section 4.2) We directly reduce the noise from the dataset by exploiting Gaussian low-pass filter in the frequency domain.
12.
Alignment of class activation map (Section 4.3). We align the class activation map between the pre-trained and quantized models, directly distilling knowledge identify the correct image region from the pre-trained model to the quantized model.
13. Soft labels for difficult samples (Section 4.4) For difficult samples, we fine-tune only with soft labels or predictions from the pre-trained model to reduce ambiguity.
Figure + illustrates the overall process of SYNQ. SYNQ first generates a synthetic dataset from arbitrary labels. Then, SYNQ exploits a Gaussian low-pass filter refine the samples by removing noise. With this filtered dataset, SYNQ fine-tunes the quantized model with KL divergence and cross-entropy losses, following the standard ZSQ framework. SYNQ also optimizes CAM alignment loss AMAM to enhance activation map alignment for better critical region detection. SYNQ exploits the threshold r to decide on the application of cross-entropy loss based on the difficulty of sample.
4.2 LOW-PASS FILTER
The first step of Zero-shot Quantization (ZSQ) is to produce the synthetic dataset which effectively mimics the real dataset. Existing methods leverage the prediction and batch normalization statistics of the pre-trained model to generate samples. However, their limitation is the noise in the synthetic dataset, as discussed Figure [ We investigate the intensity of this noise, by performing the Fourier transform on the datasets. Figure 5 iillustrasss the amplitude distribution of (a) ImageNet dataset, (b) the synthetic dataset by TexQ (Chen et al. 2023), and (c) Gaussian-filtered samples based on the distance from the center. The dark solid line indicates the mean distribution, and the surrounding colored region shows the standard deviation within a batch of 256 randomly chosen images. While the ImageNet dataset primarily exhibits lower frequency components, the synthetic dataset contains more high-frequency components, clearly indicating a higher level of sharpness and noise. This noise observed in various ZSQ methods, regardless of the setting (see Appendix C..]
To mitigate this noise, we exploit Gaussian low-pass filter on the generated samples. Given a sample x; with width W. height H, and filtering hyperparameter Do which related to cut-off frequency, we compute the filtered sample x shown in Equation (4).
(D(u, v))² 2D
W "
H y2, (4)
X = (G G.. exp
), D,D(u,))
(v
(v
where (u, v) denotes the distance from the coordinate (u, v) to the center the frequency domain and element-wise multiplication. This Gaussian low-pass filter G works in the frequency
5

---

Published as a conference paper at ICLR 2025
Amplitude distribution
0.025
0.020
0.05
Intense high
Noise is removed
Mainly low
0.020
0.04
0.015
frequency
0.03
frequency
0.015
0.010
0.02
0.010
0.01
0.005
0.005
0.00
25
75
100
25
75
100
Frequency
Frequency
Frequency
(a) ImageNet
(b) Synthetic (TexQ)
(c) Filtered (SY
Figure 5: Comparison of amplitude distribution among (a) ImageNet dataset, (b) synthetic dataset by
TexQ, and (c) filtered samples. After filtering, the distribution closely aligns with that of real images.
domain from conducting Fourier transform F, we then apply inverse Fourier transform F to obtain the filtered sample xf. Figure c) clearly shows the positive effect of the filter: removal of noise in the high-frequency region resulting in an amplitude distribution aligning with that of real images. We further investigate the robustness of low-pass filter towards different types of noise in Appendix C.8
4.3 ALIGNMENT OF CLASS ACTIVATION MAP
The next step is to fine-tune the quantized model achieve high accuracy. The first challenge of this step is to ensure that the quantized model makes predictions using on on-target image patterns. Existing methods fine-tune the model with classification and knowledge distillation from the pre-trained model, using the synthetic dataset. However, the quantized model from these methods fail to properly localize the object as depicted in Figure To ensure the quantized model make prediction based on correct image regions, we directly align the class activation map between pre-trained and quantized models. We optimize the class activation map (CAM) alignment loss CCAM by minimizing the mean square eтror between the saliency maps S (x, and S
(x,) ) of the pre-trained and quantized models, respectively. Among various techniques (Zhou al. 2016 Selvaraju et al. 2017 7, Zagoruyko & Komodakis 2017: Chattopadhay et al.) 2018) to highlight the important region of the image, we select Grad-CAM (Selvaraju al. 2017) due to its simplicity and superiority, which we discuss further in Section 5.4 Grad-CAM generates the saliency map (x,)) by weighting the activations with their gradients, emphasizing the regions in the input image that have the greatest impact on the model's 's prediction. We formulate CAM alignment loss Equation
(x;00 09)
|| (x:) W.
S (x:
H
Oy
(5)
S (x,x) = ReLU Σ
ΣΣ w=
(x:)
Willi
OA
k:0 wh (x:)
where A k:0 (x.) denotes the activations of the last layer at channel i with Wi and H3 representing its width and height, respectively. The gradient of the predicted score y for the true class y, with respect to the activation A &i (x:) at spatial location (w, h) indicates the contribution of each activation to the model's 's prediction. Figure
[ clearly shows that LC enables SYNQ accurately target the correct image regions as the pre-trained model does (see Appendix C.4 for further analysis).
4.4 SOFT LABELS FOR DIFFICULT SAMPLES
The second challenge of the fine-tuning step is the misguidance from possibly mislabeled samples. Existing works assign random classes as labels for generated samples, then minimize the Inception Loss (II.) Lrr in Equation T to optimize the image so that the pre-trained model predicts the assigned labels. However, the pre-trained model frequently mislabels difficult samples in this approach, as the higher difficulty indicates that the pre-trained model assigns lower probabilities to the true label, following the definition in Equation
To avoid this misguidance, we exclude the cross- entropy loss with the hard labels for difficult samples. We classify samples easy difficult based on difficulty threshold For easy samples, opti- mize both the cross-entropy loss with hard labels and the KL divergence with soft labels. In contrast, for difficult samples, we exclusively optimize the KI. divergence with soft labels, completely omitting

---

Published as a conference paper at ICLR 2025
the cross-entropy loss. This approach focuses on replicating the pre-trained model's responses to ambiguous images, minimizing performance degradation caused by overconfidence in hard labels. Note that previous methods apply both soft and hard labels irrespective of sample difficulty.
4.5 OBJECTIVE FUNCTION
Combining all three ideas of SYNQ, we modify the loss function for the fine-tuning phase from LzsQ in Equation (2 to LsYNQ in Equation
SSYNQ - Σ [KL(g(x] (o)(((xf:))ll(((() :0)ll((x :0)) 11 tooe .ess) Aee ,.))..) CoaM (xf ;0, ;0,0")) (6)
where AGE and ACAM are balancing hyperparameters for cross-entropy loss and CAM alignment loss, respectively. 1.) is the indicator function which returns 1 if the inner statement is true and O otherwise. We train with the filtered samples xf" (Section 4.2 to remove the noise in the synthetic dataset. Then, we align the class activation map between the pre- pre-trained and quantized models by optimizing LCAM (Section 4.3) to transfer the knowledge of finding object the image. We also exclude cross-entropy loss for difficult samples with a threshold 7 (Section 4.4 to mitigate the impact of misguidance from hard labels.
SYNQ is compatible with any ZSQ method utilizing synthetic datasets (Zhong et al. 2022b Qian
al. 2023b Jeon et 2023b). We adopt calibration center synthesis (Chen al. 2023), difficult sample generation, and sample difficulty promotion (Li et al. 2023a) because we observe they generally perform better in ZSQ (refer to Appendix D for details). We visualize the generated
images within synthetic dataset in Figure 12 The adaptability of SYNQ is clearly demonstrated
through further experiments on other Zero-shot QAT and PTQ methods Appendices C.6 and respectively. We formulate the overall algorithm of SYNQ in Algorithm [
Complexity Analysis. We analyze the time complexity of SYNQ, where N and L represent the numbers of training samples and layers, respectively.
Theorem 1 (Time Complexity of SYNQ). Given a model with an inference complexity of OTTo), the time complexity for the quantization procedure (Algorithm 7 of SYNQ is ONNT.)
Proof. See Appendix C.1
Theorem demonstrates that SYNQ is an efficient approach, with time complexity scaling linearly with the number of training samples N and model layers L. Furthermore, SYNQ generates only 5,120 samples, making significantly faster than senerator-basee methods such AdaSG (Qian et al. 2023b) and AdaDFQ (Qian et al. 2023a), which produce over million samples (see Appendix C.9 for experiments with different sizes of dataset). We perform runtime analysis of SYNQ in Appendix to analyze the computational overhead of SYNQ.
EXPERIMENTS
We perform experiments to answer the following questions about SYNQ. Further discussions and experiments on SYNQ are discussed in Appendix C
Q1. Accuracy in Convolutional Neural Network (CNN) Quantization (Section 5.2 How accurate is the quantized CNN model from SYNQ compared to those from existing ZSQ methods?
Q2. Accuracy in Vision Transformer (VIT) Quantization (Section
5.3) How effective is SYNQ
in enhancing VIT Quantization performance?
Q3. Analysis on Class Activation Map Techniques (Section 5.4 Which CAM technique demon-
strates the highest performance?
Q4. Ablation Study (Section (55).. Are all components of SYNQ effective for enhancing the classifi- cation accuracy of the quantized model?
Q5. Hyperparameter Analysis (Section (.6).. How robust are the performance gains by SYNQ in hyperparameters AcEE ACAM. Do. and 7?
5.1 EXPERIMENTAL. SETUP
We briefly introduce the experimental setup. Further setups detailed in Appendix D
7

---

Published as a conference paper at ICLR 2025
Table 1: Zero-shot Quantization accuracy [%] of ResNet-20 (R-20) on CIFAR-IO and CIFAR-I00, and ResNet-18 18 (R-20), ResNet-50 (R-50), and MobileNetV2 (MV2) on ImageNet. WBAB indicates that both weights and activations are quantized to Bbit. Note that SYNQ achieves the highest accuracy.
R-20 (CIFAR-1O)
R-20 (CIFAR-100)
R-18 (ImageNet)
R-50 (ImageNet)
MV2 (ImagoNet)
Method
W4A4
W3A3
W444
W3A3
W444
W3A3
W444
W3A3
W4A4
W3A3
Full Procision (W32A32)
93.89
70.33
71.47
77.73
73,03
GDFQ
Xu ct al
2020
90.11
75.11
63.75
47.61
60.60
20.23
54.16
0.31
59.43
1,46
ARC
huu
2021
88.55
62.76
40.15
61.32
23.37
64.37
1.63
60.13
14.30
Qimera
Choi al.
2021
91.26
74.43
65.10
46.13
63.84
1.17
66.25
61.62
ARC
Choi al.
2022
90.49
61.05
41.34
65.73
68.27
IntraQ
Zhong. et al.
20226
91.49
77.07
64.98
48.25
66.47
45.51
65.10
AdaSCi
QQin
202.3b
92.10
66.42
52.76
66.50
37.04
68.58
16.98
26,90
AdaDPQ QQian et al
2023a
92.31
84.89
66.81
52.74
66.53
38.10
68.38
17.63
65.41
28.99
HAST
Li
2023a
92.36
86.34
66.68
55.67
66.91
42.58
65.60
Tex Chen el
2023
92.68
86.47
67.18
55.87
67.73
50.28
70.72
25.27
67.07
32.80
PLF Fan sl.
2024
92.47
88.04
66.94
$7.03
67.02
68.97
SyNQ (Proposed)
92.76
67.34
57.28
67.90
52.02
71.05
26.89
67.27
34.21
Standard Deviation
+ 0.10
0.15
+ 0.15
+ 0.29
=
0.19
+ 0.34
+ 0.17
+ 0.24
0.21
+ 0.27
Setup. We evaluate our method across three datasets by reporting the top-1 accuracy for the validation sets CIFAR-10, 10, CIFAR- 100 (Krizhevsky et al. 2009) and ImageNet (ILSVRC 2012) (Deng et al. 2009) datasets. We select ResNet-20 (He et al. (216)) model for CIFAR- 10 and CIFAR- 100, and ResNet- 18, ResNet-50 (He et al. 2016). and MobileNetV2 (Sandler et al. 2018)) model for ImageNet.
We follow this prevalent experimental setup from existing works (Chen et al., 2023 Qian al.
2023a to correctly compare the performance of SYNQ.
Competitors. We compare SYNQ with existing ZSQ methods utilizing synthetic dataset, including
GDFQ (Xu et al., 2020), ARC (Zhu et al. 2022)) Qimera (Choi et al. (2021). ARC + AIT (Choi
al. 2022), IntraQ (Zhong al. 2022bb) AdaSG (Qian et al. 2023b). ) AdaDFQ QQia al. 2023a),
HAST (Li et al. 2023a), TexQ (Chen et al. 2023). , and PLF (Fan et al. 2024). Both model weights
and activation are quantized identically for all layers.
Implementation Details. We follow the settings from IntraQ (Zhong et al. 2022b) and HAST Li al. 2023a) for equal comparison. We generate 5,120 images with a batch size of 256. The batch size for fine-tuning is 256 for CIFAR- 10/100 and 16 for ImageNet with epochs uniformly set to 100. We search T, Do, Acg, and ACAM within the ranges (0.5, 0.55, 0.6, 0.65, 0.7), [20, 40, 60, 80, 100), {0.005, 0.05, 0.5, 5), and (20, 50, 100, 200, 300, 500, 2000]. respectively, All of our experiments were done at a workstation with Intel Xeon Silver 4214 and RTX 3090.
5.2 ACCURACY IN CNN QUANTIZATION (Q1)
We evaluate the quantization accuracy of SYNQ against existing ZSQ methods using CIFAR 10, CIFAR-100, and ImageNet datasets. Our method significantly enhances quantized model accuracy on all settings with 3bit and 4bit quantization as summarized in T We report the mean and standard deviation of iterations, each using different random seed values. We have two observations from the result. First, S SYNQ benefits the fine-tuning of quantized models consistently across diverse quantization bits. models, and datasets. Compared to state-of state-of-thhearr art methods TexQ (Chen al. 2023) and PLF (Fan et al.) 2024), SYNQ achieves higher accuracies of up to 1.74%p (ResNet- 18 on ImageNet dataset). Second, SYNQ demonstrates increasing effectiveness as bit-width decreases. Considering that lower-bit quantization is inherently more challenging, our results clearly showcase the robustness of SYNQ due to its effective fine- tne-tuning that overcomes the aforementioned limitations.
5.3 ACCURACY IN VIT QUANTIZATION (Q2)
We investigate the effectiveness of SYNQ in enhancing ZSQ performance for Vision Transformers (ViTs). Table 2 shows the ZSQ precision of four ViT models, DeiT-Tiny, Deil-Small (Touvron) iet al. 2).1). Swin-Timg Tiny, and Swin-Small (Liu et al. (2021b) pre-trained on ImageNet dataset. SYNQ enhances the quantization precision across various models, achieving up to 0.58%p increase in
8

---

Published as a conference paper at ICLR 2025
Table 2: Zero-shot Quantization accuracy [%] of ViT models on ImageNet dataset. WBAB indicates that both weights and activations are quantized to Bbit. Note that SYNQ shows consistent improve- ments in quantization performance across various models.
Bits
Method
Deny-Tiny
DeiT-Small
Swin-Tiny
Swin-Small
Average
Full Precision
72.21
79.85
81.35
83,20
79.15
PSAQ-VA di
2022
65.57 0.10
72.04
1.67
75.03
0.63
70.61
W4A8
SYNQ (Proposed)
65.90 0,07
72.28
0.34
70.76 = 1.61
75.82 +: 0,54
71.19
PSAQ-Vi VTT etietal al. 2022
71.56 + 0.03
75.97
0.20
73.54 + 1.61
76.68 + 0.53
74.44
WSA8
SYNQ (Propesed)
71.74 0.03
76.16
0.29
74.11 :: 1.82
77.32 0.59
74.83
average precision when applied to the recent method PSAQ-ViT (Li et 2022).
The results show
that SYNQ is am accurate ZSQ method not only tailored for CNN but also is effective in ViTs.
5.4 ANALYSIS ON CLASS ACTIVATION MAP TECHINQUESS (Q3)
$2.5
52.0-
We compare the quantization accuracy of SYNQ when utilizing different techniques to output the class activation map. We show the 3bit quantization accuracy of ResNet-18 model in Figure 6 Grad-CAM (Selvaraju et al.
$1.5
51.0
2017) demonstrates higher performance over CAM (Zhou et al.] (016) and Grad-CAM++ (Chattopadhay et al., 2018). This attributed to Grad- CAM+ being specialized in localizing multple objects. whereas Grad- CAM focuses on a single object. Additionally, note that Grad-CAM also takes advantage over CAM in that it is a direct generalization of CAM which is applicable only to models with a global pooling layer. Thus, we utilize Grad-CAM to generate the saliency map for the CAM alignment
50.5
50.0
CAM
GC GC *GC: Grad-CAM
Figure 6: ZSQ accuracy comparison on different CAM techniques. See Section 5.3 for details.
loss LCAM, as described in Section 4.4
5.5 ABLATION STUDY (Q4)
Table Ablation study on the main ideas of SYNQ. Q All ideas contribute to the im- proved performance.
We perform ablation study to show that each main idea of SYNQ. such as low-pass filter (II) in Section 4.2 alignment of class activation map (12) in Section 4.3 and soft labels for difficult samples (I3) in Section improves the classification accuracy of the compressed model. We summarize the 3bit quantization results of ResNet-18 model on ImageNet dataset in Table 3 Note that the baseline denotes HAST (Li et al.. (2033) with layer-wise batch normalization loss from TexQ (Chen et al.) (0233) as detailed in Appendix [ Our analysis shows that all proposed ideas contribute to improved performance, with low-pass filter (II) having the strongest impact of 5.80%p.
11
12
13 Accuracy [%]
Bascline
43.63
49.43
48.26
46.42
51.24
50.81
50.06
52,02
5.6 HYPERPARAMETEL ANALYSIS (Q5)
We analyze the robustness of SYNQ concerning the newly introduced hyperparameters ACE. ACAM- Du, and in Figure We report the 3bit quantization accuracy for the ResNet-18 18 model trained on the ImageNet dataset. We have three observations from the result. First, as shown in Figure a the classification accuracy remains robust across a range of λςE and ACAM values. This robustness indicates that SYNQ remains effective even when these hyperparameters are not precisely tuned. Second, Figure (b)) illustrates the effect of varying the difficulty threshold 7. Note that the classifica- tion accuracy increases as increases from 0 to 0.5, since too low excludes many useful samples for cross-entropy training. However, the classification accuracy starts to decrease т becomes greater than 0.5, since it allows to use difficult and ambiguous samples for cross-entropy training. We observe that the value of 0.5 gives the best trade-off between using more samples and not using ambiguous samples. We further conduct deeper analysis on 7 in Appendix C.I1I verifying its impact on different settings. Third, Figure ())) shows that an appropriate balance in Do is necessary to maintain performance. Extremely low Do values result in significant performance degradation due to excessive filtering, which oversmooths the images and results the loss of crucial information. Overall, SYNQ consistently outperforms baselines across diverse range of hyperparameter values.

---

Published as a conference paper at ICLR 2025
-3.0
50.26
51.23
51.16
51.58
50.75
50.68
12.63
Classfiicatoo. Aatonnoouaaa
52-
50.0
50.5
51-
51.12
52.02
51.01
50.48
50.85
50.93
43.85
47.5
Clessitcaiion
48.0
Acs
0.08
51.35
51.83
50.58
50.30
30.58
49.39
42.31
45.0
50-
St NQ
45.5
SYNQ
0.005
50.35
50.52
40.59
42.5
49
T
TexQ
43.0
50
100
200
300
500
2000
0.0
0.2
0.4
0.6
20
40
60
80
100
ACAM
Dg
(a) Balancing hyperparameters
(b) Difficulty threshold
(c) Filtering hyperparameter
Figure 7: Hyperparameter analysis on (a) balancing hyperparameters λ and XCAM, (b) difficulty
threshold T. and (c) filtering hyperparameter Dn- See Section for details.
6 RELATED WORK
Network Quantization. Network quantization reduces the computational complexity and memory footprint of deep neural networks by converting the weights, activations, or both from full precision to lower precision formats (Deng al. 2020; Ada Guo et al., 2022. Shang et al. 2023 Park et al..
2024b). Quantization significantly speeds up inference and reduces power consumption, enabling deployment on edge devices with limited resources. Recent advancements in network quantization include Quantization-Awar Training (QAT) (Jacob et al. [2018] Lee et al. [022 Dettmers et al.
2023 Xu et al. 2024) and Post-Training Training Quantization (PTQ) (Li al. [0021] Frantar et al.J [202] Zhong et al. 2022a Jeon et al.] (022). QAT integrates quantization into training, enabling the model to learn weights robust to quantization noise, thereby maintaining higher accuracy. On the other hand, PTQ quantizes pre-trained models using calibration to minimize accuracy loss without original training data. Furthermore, advanced strategies such as sexeedprreision quantization (Koryakovskiy et al. 2023), knowledge distillation (Boo et al., 2021), adaptive quantization (Zhou et al. (0188), weight sharing (Ullrich et al. 2016). parameter reparameterization (Li ct al.) (003c) and hardware- awareness (Wang et al. 2019) have shown promising results in achieving balance between model efficiency and performance. However, existing works require real data to directly train or calibrate the quantized model. In contrast, SYNQ focuses on QAT scenarios where there is access to real data.
Zero-shot Quantization. Zero-shot Quantization (ZSQ) (Cai et al. 2020), also called as data-free quantization (Nagel et al., 2019)] Chen et al. 2019 Choi et al. 2020), performs quantization without the need for accessing the training data of full full-precision models. s. Earlier methods focused on calibrat- ing model parameters solely based on model properties without acquiring any data (Banner et al.) 2019; Guo et al. 2022). Unfortunately, these methods resulted in significant performance drops at lower bit widths such as 3bit or 4bit quantization (Xu et al. 2020 Zhong al. 2022b). Recent studies generate synthetic datasets and fine-tune tune the quantized model to enhance performance (Haroush et al.) 2020; Choi al., 2021 Liu al., 2021a Zhong et al. 2022b). GDFQ (Xu et al. 2020) first employs generative methods leveraging batch normalization statistics and extra category label information. Numerous variants have developed the field by introducing techniques such as advanced genera- tors (Zhu et al. 202II, boundary supporting samples (Choi et al.) 2021). noise optimization (Cail ct al. 2020). diversified samples (Zhang et al. (22TT) intra-class heterogeneity (Cai et al., 2020). hard sample generation (Li et al. 2023a), texture feature calibration (Chen et al. 2023), and pseudo-label filtering (Fan et al.) 2024). Recently, several works further advances ZSQ into Vision Transformers (Li
ctt al. 2022 2023b Ramachandran al. 2024). However, existing methods continue to struggle with the three primary challenges (see Section 4.1). In contrast, SYNQ tackles these challenges with three main ideas: low-pass filter, class activation map alignment, and soft labels for difficult samples.
7 CONCLUSION
We propose SYNQ (Synthesis-awd aware Fine-tuning for Zero-shot Quantization), an accurate Zero-shot Quantization (ZSQ) method that effectively addresses the three major limitations of fine-tuning with synthetic datasets: 1) noise in the synthetic dataset, 2) predictions based on off-target patterns, and the 3) misguidance by erroneous hard labels. We exploit low-pass filter to minimize noise, align the class activation map to ensure prediction from correct image region, and leverage soft labels on difficult samples avoid misguidance by erroneous hard labels. SYNQ consistently outperforms existing ZSQ methods across diverse models, quantization bits, and datasets. Future works include extending our method into settings such as object detection and diffusion models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [No.RS-2020011200899,, Flexible and Efficient Model Compression Method for Various Applications and Environments], [No.RS- 2021-1121 1343, Artificial Intelligence Graduate School Program (Seoul National University)], and 2068, Artificial Intelligence Innovation Hub (Artificial Intelligence Institute, Seoul National University)]. This work was supported by Youlchon Foundation. The Institute of Engineering Research Scoul National University provided research facilities for this work. The ICT at Scoul National University provides research facilities for this study. Kang is the corresponding author.
REFERENCES
Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment. In NeurlPS, 2019.
Yoonho Boo, Sungho Shin, Jungwook Choi, and Wonyong Sung. Stochastic precision ensemble: self-knowledge distillation for quantized deep neural networks. AAAI, 2021.
Alan C Bovik. Handbook of image and video processing. Academic press, 2010.
Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami. Michael W Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In CVPR, 2020.
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- cam++: Generalized gradient-base -based visual explanations for deep convolutional networks. WACV, 2018.
Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang. Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. Data-free learning of student networks. In CVPR, 2019.
Xinrui Chen, Yizhi Wang, Renao Yan, Yiqing Liu, Tian Guan, and Yonghong He. Texq: Zero-shot network quantization with texture feature distribution calibration. In NeurlPS, 2023.
Yu Cheng, Duo Wang. Pan Zhou, and Tao Zhang. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. JEEE Signal Processing Magazine, 35 (1):126-136, 2018.
Ikhyun Cho and Kang. Pea-kd: Parameter-efficien and accurate knowledge distillation on bert. PLOS ONE, 17(2):1-1,, 02 2022.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang. Vijayalakshmi Srinivasan. and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv: 1805 1805.06085 2018.
Kanghyun Choi, Deokki Hong, Noseong Park, Youngsok Kim, and Jinho Lee. Qimera: Data-free quantization with synthetic boundary supporting samples. NeurIPS, 2021.
Kanghyun Choi, Hye Yoon Lee, Deokki Hong, Joonsang Yu, Noseong Park, Youngsok Kim, and Jinho Lec. It's all in the teacher: Zero-shot quantization brought closer to the teacher. In CVPR, 2022.
Yoojin Choi, Jihwan Choi, Mostafa Fl-Khamy, and Jungwon Lee. Data-free network quantization with adversarial knowledge distillation. In CVPRW, 2020.
James W Cooley and John W Tukey. An algorithm for the machine calculation of complex fourier series. Mathematics of computation, 19900), 1965.
Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
11

---

Published as a conference paper at ICLR 2025
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4): 485-532, 2020.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lims. In NeurIPS, 2023.
Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra Modha. Learned step size quantization. In ICLR, 2020.
Chunxiao Fan, Ziqi Wang, Dan Guo, and Meng Wang. Data-free quantization via pseudo-label filtering. In CVPR, 2024.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: accurate quantization for generative pre-trained transformers. In ICLR, 2023.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. In Low-Power Computer Vision, pp. 291-326. Chapman and Hall/CRC, 2022.
Cong Guo, Yuxian Qiu, Jingwen Leng, Xiaotian Gao, Chen Zhang, Yunxin Liu, Fan Yang, Yuhao Zhu, and Minyi Guo. Squant: On-the-fly data-free quantization via diagonal hessian approximation. In ICLR, 2022.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In ICML, 2015.
Matan Haroush, Itay Hubara, Elad Hoffer, and Daniel Soudry. The knowledge within: Methods for data-free model compression. In CVPR, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Yang He and Lingao Xiao. Structured pruning for deep convolutional neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5):2900-29) 2919, 2024.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient origerr-rithmmtic-onll inference. In CVPR, 2018.
Jun-Gi Jang, Chun Quan, Hyun Dong Lee, and U Kang. Falcon: lightweight and accurate convolution based on depthwise separable convolution. Knowl. Inf. Syst., 65(5):2225-2249, 2023.
Hyojin Jeon, Seungcheol Park, Jin-Gee Kim, and U Kang. Pet: Parameter-effiieet knowledge distillation on transformer. PLOS ONE, 18(7):1-21, 07 2023a.
Yongkweon Jeon, Chungman Lee, Eulrang Cho, and Yeonju Ro. Mr. biq: Post-training non-uniform quantization based on minimizing the reconstruction error. CVPR, 2022.
Yongkweon Jeon, Chungman Lee, and Ho-young Kim. Genie: show me the data for quantization. In CVPR, 2023b.
Junghun Kim, Jinhong Jung, and U. Kang. Compressing deep graph convolution network with multi-staged knowledge distillation. PLOS ONE, 16(8):1-18, 08 2021.
Ikki Kishida and Hideki Nakayama. Empirical study of easy and hard examples in CNN training. In ICONIP, 2019.
Ivan Koryakovskiy, Alexandra Yakovleva, Valentin Buchnev, Temur Isaev, and Gleb Odinokikh. One-shot model for mixed-preeision quantization. CVPR, 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical Report, 2009.
12

---

Published as a conference paper at ICLR 2025
Sunwoo Lee, Jeongwoo Park, and Dongsuk Jeon. Toward efficient low-precision training: Data format optimization and hysteresis quantization. In ICLR, 2022.
Buyu Li, Yu Liu, and Xiaogang Wang. Gradient harmonized single-stage detector. In AAAI, 2019.
Huantong Li, Xiangmiao Wu, Fanbing Lv, Daihai Liao, Thomas H Li, Yonggang Zhang, Bo Han, and Mingkui Tan. Hard sample matters a lot in zero-shot quantization. In CVPR, 2023a.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit post-training quantization by block reconstruction. In ICLR, 2021.
Zhikai Li, Liping Ma, Mengjuan Chen, Jumrui Xiao, and Qingyi Gu. Patch similarity aware data-free quantization for vision transformers. In ECCV, 2022.
Zhikai Li, Mengjuan Chen, Junrui Xiao, and Qingyi Gu. Psaq-vit v2: Toward accurate and general data-free quantization for vision transformers. IEEE Transactions Neural Networks and Learning Systems, pp. 1-12, 2023b.
Zhikai Li, Junrui Xiao, Lianwei Yang, and Qingyi Gu. Repq-vit: Scale reparameterizztion for post-training quantization of vision transformers. ICCV, 2023c.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
Yuang Liu, Wei Zhang, and Jun Wang. Zero-shot adversarial quantization. In CVPR, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In CVPR, 2021b.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In ICCV, 2019.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In ICML, 2020.
Seungcheol Park, Hojun Choi, and U Kang. Accurate retraining- pruning for pretrained encoder- based language models. In ICLR, 2024a.
Seungcheol Park, Jachyeon Choi, Sojin Lee, and U Kang. A comprehensive survey of compression algorithms for language models. arXiv preprint arXiv:2401.1534 2024b.
Yongchan Park, Jun-Gi Jang, and U Kang. Fast and accurate partial fourier transform for time series data. In KDD, 2021.
Yongchan Park, Jongjin Kim, and U Kang. Fast multidimensional partial fourier transform with automatic hyperparameter selection. In KDD, 2024c.
Tairen Piao, Ikhyun Cho, and U. Kang. Sensimix: Sensitivity-aware 8-bit index & 1-bit value mixed precision quantization for bert compression. PLOS ONE, 17(4):1-22, 2022.
Biao Qian, Yang Wang, Richang Hong, and Meng Wang. Adaptive data-free quantization. In CVPR, 2023a.
Biao Qian, Yang Wang, Richang Hong, and Meng Wang. Rethinking data-free quantization zero-sum game. In AAAI, 2023b.
Akshat Ramachandran, Souvik Kundu, and Tushar Krishna. Clamp-vit: Contrastive data-free learning for adaptive post-training quantization of vits. In ECCV, 2024.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?" explaining the predictions of any classifier. In KDD, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
13

---

Published as a conference paper at ICLR 2025
Florian Scheidegger, Roxana Istrate, Giovanni Mariani, Luca Benini, Costas Bekas, and A. Cristiano I. Malossi. Efficient image dataset classification difficulty estimation for predicting deep-learning accuracy. Vis. Comput., 57(6):1593-1610 2021.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. In ICCV,, 2017.
Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. CVPR, 2023.
Prasen Kumar Sharma, Arun Abraham, and Vikram Nelvoy Rajendiran. A generalized zero-shot quan- tization of deep convolutional neural networks via learned weights statistics. IEEE Transactions on Multimedia, 25:953-9655 96555-965--00 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
Cuong Tran, Ferdinando Fioretto, Jung- Eun Kim, and Rakshit Naidu. Pruning has disparate impact on model accuracy. In NeurlPS, 2022.
Karen Ullrich, Edward Meeds, and Max Welling. Soft weight-sharing for neural network compression. In ICLR, 2016.
Huanyu Wang, Junjie Liu, Xin Ma, Yang Yong, Zhenhua Chai, and Jianxin Wu. Compressing models with few samples: Mimicking then replacing. CVPR, 2022.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quantization with mixed precision. In CVPR, 2019.
Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. Qdrop: Randomly dropping quantization for extremely low-bit post-training quantization. In ICLR, 2022.
Ross Wightman. Pytorch image models.
https://github. com/ htt.://ttaiicooo//..gggtmaaaama/ ghtman//
pytorch- iwwgiiiaaeaaaa laimaag--eeelesee 2019.
Yi Xie, Huaidong Zhang, Xuemiao Xu, Jianqing Zhu, and Shengfeng He. Towards a smaller student: Capacity dynamic distillation for efficient image retrieval. In CVPR, 2023.
Shoukai Xu, Haokun Li, Bohan Zhuang, Jing Liu, Jiezhang Cao, Chuangrun Liang, and Mingkui Tan. Generative low-bitwidth data free quantization. In ECCV, 2020.
Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantization-awe awwae low-rank adaptation of large language models. ICLR, 2024.
Jaemin Yoo, Minyong Cho, Taebum Kim, and U Kang. Knowledge extraction with no observable data. In NeurlPS, 2019.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. In ICLR, 2017.
Xiangguo Zhang, Haotong Qin, Yifu Ding, Ruihao Gong, Qinghua Yan, Renshuai Tao, Yuhang Li, Fengwei Yu, and Xianglong Liu. Diversifying sample generation for accurate data-free quantization. In CVPR, 2021.
Yunshan Zhong, Mingbao Lin, Mengzhao Chen, Ke Li, Yunhang Shen, Fei Chao, Yongjian Wu, and Rongrong Ji. Fine-grained data distribution alignment for post-training quantization. In ECCV, 2022a.
Yunshan Zhong, Mingbao Lin, Gongrui Nan, Jianzhuang Liu, Baochang Zhang, Yonghong Tian, and Rongrong Ji. Intraq: Learning synthetic images with intra-class heterogeneity for zero-shot network quantization. In CVPR, 2022b.
14

---

Published as a conference paper at ICLR 2025
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, 2016.
Yiren Zhou, Seyed-Mohsen Moosavi-Dezfooli, Ngai-Man Cheung, and Pascal Frossard. Adaptive quantization for deep neural network. In AAAI, 2018.
Baozhou Zhu, Peter Hofstee, Johan Peltenburg, Jinho Lee, and Zaid Alars. Autorecon: Neural architecture search- based reconstruction for data-free In IJCA,, 2021.
15

---

Published as a conference paper at ICLR 2025
A
NOTATION
We summarize the frequently used notations the paper Table
Table 4: Frequently used notations.
Symbol
Description
A pre-trained model
The quantized model
{x,
Synthetic samples
{y.
One-hot encoded labels of synthetic samples
g(x;0 0)
Probability distribution of a sample x; predicted by parameters
0(x;;)
Difficulty of sample x; predicted by parameters /
KL(II-
KL. divergence loss Cross-entropy loss
CE(-)
Fourier transform function
G
Gaussian low-pass filter Filtering hyperparameter
Do
ACAM
Balancing hyperparameter of CAM loss
ACE
Balancing hyperparameter of CE loss
Threshold of difficulty for cross-entropy loss
B ALGORITHM
We describe the quantization procedure of SYNQ in Algorithm Note that any technique for generating synthetic datasets is applicable.
Algorithm 1 Quantization procedure of SYNQ
Input: the pre-trained model with parameters 8, hyperparameters neps Do. ACAM, ACE, and Output:
the parameters 8 of the quantized model.
fa Step 1: Generate synthetic dataset **/
1: Initialize the synthetic dataset )x, with Gaussian noise.
2: Randomly assign labels {y: N for synthetic dataset.
3: Optimize (x; [ to minimize LII. + aLBNNS
>Compatible with any synthetic dataset
1e* Step 2: Fine-tune quantized model #*/
4: Initialize ( following the cound-to-nearee scheme.
5: Apply low-pass Gaussian filter with a cut-off frequency Do
to synthetic samples and obtain {x. ) Tim1-
Idea 1: Low-pass filter
6: for each epoch in [1...., neppl do
7: Initialize the total loss L
8: for in [1,..., -N do
9:
Perform forward pass of and fa with synthetic sample x Compare gradients and calculate the CAM loss LCAM. L L + [Tggg(x;; ())((x
10:
b Idea 2: CAM alignment
+ ACAMCCAM
12:
Calculate 6(x;;)) 0)
13:
8(x;; 0) < then
Idea 3: Soft labels difficult samples
14:
C + L + (9)
15: end
16: end for
17: Update dat to minimize
18:
end for
19:
return 89
16

---

Published as a conference paper at ICLR 2025
C FURTHER DISCUSSION AND EXPERIMENTS
C.I. PROOF OF THEOREM I
We provide the proof of Theorem 1 as below:
Proof. We investigate the time complexity of SYNQ in three steps: synthetic dataset generation, low- pass filter application, and quantized model fine-tuning. First, synthetic dataset generation involves calculating Inception Loss LI1. and Batch Normalization Statistics Loss LBNS for N samples, each requiring a forward pass through the model, resulting in complexity of ONTo))
Second, applying the low-pass filter G involves a Fourier transform F()), an element- wise multipli- cation 3 and an inverse Fourier transform (-) (see Equation 4p. Implementing with Fast Fourier Transform(FFT), the time complexity for single input with size of Z x Z is O Z log 2). O
and (Z log Z)) is for F(x), @ F(x), and F(G @ F(x)), respectively (Cooley & Tukey 1965 Therefore, the time complexity of this step is ( log Z) for N samples.
Lastly, the fine-tuning step involves calculating the loss for N filtered samples, resulting in complexity of (T, + L T=)). Here, T3 represents the complexity of computing the cross-entropy and KL divergence losses, and Tg represents the complexity of computing the Grad-CAM loss LCAM across L layers. Generating saliency maps 8 (x) for single input for all L layers using Grad-CAM requires one forward pass and L backward passes, thereby requiring LT)) for N samples (Selvaraju et al. 2017). Thus, the complexity of computing LCAM (Equation 5 ON)
because aligning the saliency maps S (x:) and 8 (x,) is negligible compared model inference time Tg. In summary, this step is simplified to (NN.TT))
Combining these complexities, we get:
O(NT, NZ log Z NLTo) ONITo) (:: To Zlog Z).
C.2 RUNTIME ANALYSIS
Baseline
Overhead
We perform runtime analysis to investigate the computational over- head introduced by SYNQ. For this, we measure the difference in fine-tuning time between the baseline methods with and without SYNQ, For fair comparison, we compare only with noise optimization methods since they do not train generator model while fine-tuning. Figure X depicts the relative contribution of baseline methods to the per-epoch fine-tuning time compared to the approach where SYNQ is added to the baseline method for three baselines, IntraQ (Zhong jet al.) (0226), HAST (Li et al. 2023a), and TexQ (Chen et al. 2023). Note that the overhead from SYNQ marginal, i.e., the added time takes only 17.81% % of the total time in average. Thus, SYNQ improves adopted models with minimal sacrifice of quantization time.
of Fine-tume tueettunngg/// Time
80
60
40
20
IntraQ HAST TexQ Methods
Figure 8: Runtime analy- sis of SYNQ. The overhead from SYNQ is marginal.
C.3 PREVALENT NOISE IN THE SYNTHETIC DATASET
In Section 4.2 and Figure 5 we empirically analyze the limitation of the exiting ZSQ approaches, i.e., their synthetic datasets contain more high-frequency components compared to real image datasets, clearly indicating a higher level of noise. We investigate this limitation across various ZSQ methods and datasets to demonstrate that a widespread issue, not confined specific scenarios. Figures and I0 show the amplitude distribution of real and synthetic datasets, respectively. We compare three real datasets CIFAR-10. CIFAR- 100, and ImageNet with the corresponding synthetic dataset produced by three baseline methods, IntraQ (Zhong et al. 2022b) (first row), HAST (Li et al.) 2023a) (third row), and TexQ (Chen et al.) 2023) (fitth row). We then apply a low-pass filter (Do = 50) to mitigate the observed noise. As shown in the figures, the discrepancy in amplitude distribution is observed regardless of the baseline method or dataset. This is effectively mitigated by exploiting the filter that removes lighfrreuency noise, thereby leading to enhanced quantization performance. In summary, both the limitation of noise in synthetic dataset and effect low- pass filter (Section 4.2 evident in large variety of settings.
17

---

Published as a conference paper at ICLR 2025
Amplitude Distibbuton
0.15
Amplitudo Distribution
0.15
Amplitude Distribution
0.05
0.12
0.12
0.09
0.09
0.03
0.06
0.06
0.02
0.08
0.03
0.01
0.00
0.00
0.00
l'rqqeecyy
Freequency
Frequency
(a) CIFAR-10
(b) CIFAR-100
(c) ImageNet
Figure Amplitude distribution of various real image datasets. See Appendix for details.
Table 5: The average per-batch KL.-divergence [x 10-2 between the saliency maps of the pre-trained
and quantized models trained with different methods and datasets. See Section C.4 for details.
Synthetic dataset
Method
Real dataset
Baseline
SYNQ
IntraQ Zhong
2022b
3.1973
4.0251
3.2891 (-18.29 %)
HAST
Li et al.
2023a
3.0976
3.9867
3.3133 (-16.89 %)
TexQ Chen et al.
2023
2.9952
3.8436
3.1542 (-17.94%)
C.4 FURTHER ANALYSIS ON CAM PATTERN DISCREPANCY
The observation of "predictions based on off- patterns" from Figure Section 3 intuitive and persuasive, but is analyzed under limited conditions. We explore the distance between saliency maps derived from Grad-CAM (Selvaraju et al. 2017) to 1) validate this challenge across diverse methods and 2) demonstrate that it applies not only to few selected images but also to the entire synthetic dataset on average. Table presents the average distance between the saliency maps of the pre-trained model (target) and the quantized models (prediction) trained with various methods and datasets. We compute the KI. divergence between the saliency maps of 3-bit quantized ResNet- 18 models pre-trained on the ImageNet dataset, treating each saliency map as distribution, and report the average distance across batches with size of 32. From the result, have three observations. First, all three baseline methods demonstrate notable CAM discrepancies, emphasizing the generality of this challenge in the ZSQ domain. Second, this challenge is significantly mitigated when using real datasets, with a reduction in distance exceeding 20% compared to synthetic datasets. This highlights that training with synthetic datasets exacerbates this problem. Last, adopting SYNQ significantly lowers the CAM discrepancy for all baseline methods, achieving a reduction of approximately 16- 18% compared the baseline. The resulting discrepancy is comparable to that of training with real datasets. Overall, the challenge of CAM pattern discrepancy 1) is evident across multiple methods and 2) notably reduced by CAM alignment of SYNQ.
C.5 COMPARISON BETWEEN CAM ALIGNMENT AND FEATURE ALIGNMENT
We compare CAM (Class Activation Map) alignment of proposed SYNQ and feature alignment of HAST (Li et al. 2023a) mitigate possible misunderstandings and highlight the novelty of the proposed idea. The main difference between CAM alignment and feature alignment lies in their focus on different aspects of the model's 's behavior. Compared to acti- vation maps that show the response the model the given input, CAM emphasizes the region of the model related to the model's prediction, highlighting the most relevant features that contribute the final decision. This is because CAM is defined based on the magnitude of the gradient with respect to the cross-entropy between the prediction and the label. Con-
Table Ablation study of two align- ment techniques. CAM alignment shows superior performance.
FA 12
1l & 13 Accuracy [%]
Baseline
43.63
46.77 0.30
48.26 +: 0.29
/
51.20 0.30
1
52.02 0.34
18

---

Published as a conference paper at ICLR 2025
Amplitude Disoritttion
0.15
Amplitudo Distribution
0.15-
Amplitude Distribuation
0.020
0.12
0.12
0.09
0.09
0.010
0.06
0.06
0.03
0.005
0.00
0
12
15
25
50
75
100
l'requency
Freqqency
Frequency
(a) IntraQ + CIFAR-10
(b) IntraQ + CIFAR-100
(c) IntraQ + ImageNet
Amplinude Diceribation
0.15
Amplinue Distribation
0.15
Amplitude Distribation
0.020
0.12
0.12
0.015
0.09
0.09
0.010
0.06
0.06
0.03
0.03
0.005
0.00
0.000
12
Frequency
Frequency
Frequency
(d) IntraQ CIFAR-10 (filtered)
(c) IntraQ + CIFAR-100 (filtered)
(f) IntraQ + ImageNet (filtered)
Amplitude Distributon
Amplitude Distribuiinn
Amplitude Distr Distribution
0.020
0.12
0.12
0.015
0.09
0.09
0.010
0.06
0.06
0.03
0.03
0.005
0.00
15
0.00
15
0.000
25
50
100
Frequency
Frequency
Proquency
(g) HAST CIFAR-10
(h) HAST + CIFAR-100
(i) HAST + ImageNet
0.15
Amplitale Distriburion
0.15
Amplitude Distribution
0.020
0.12
0.015
Amplitude
0.06
0.010
0.03
0.03
0.00
0.00
0.000
12
15
12
25
100
Frequency
Froquency
Frequency
(i) HAST CIFAR-10 (filtered)
HAST CIFAR-100 (filtered)
(I) HAST ImageNet (filtered)
Ampliuie Distrihutton
0.15
Amplitude Distribution
0.15
Amplinude Distribution
0.020
0.12
0.12
0.015
0.09
0.010
0.03
0.005
0.00
0.00
0
9
12
15
6
9
12
15
100
Frequency
Frequenay
Prequency
(m) TexQ + CIFAR-10
(n) TexQ + CIFAR-100
(0) TexQ ImageNet
Amplinue Distriburion
0.15
Ampliaue Distribunion
0.15
Amplinde Distribution
0.12
0.12
0.020
0.09
0.09
0.015
0.06
0.06
0.010
0.03
0.03
0.005
12
15
12
20
40
60
80
100
Frequency
Frequency
Frequency
(p) TexQ + CIFAR-10 (filtered)
(q) TexQ + CIFAR-100 (filtered)
(r) TexQ ImageNet (filtered)
Figure 10: Amplitude distribution of various synthetic datasets. See Appendix
C.3 for details.
19

---

Published as a conference paper at ICLR 2025
Table 7: Comparison of ZSQ accuracy [%] of the ResNet-18 model pre-trained on the ImageNet dataset, before and after applying SYNQ. Regardless of the baseline method and quantization bits, SYNQ consistently improves the ZSQ accuracy.
W3A3
W444
Type
Method
Baseline
SYNQ
Imp.
Bascline
+YNNQ
Imp.
GDFQ
Xu et
2020
20.23
25.57 0.28
5.34
60.60
61.23
0.21
0.63
Generator hased
Qimera Choi
2021
1.17
32.34 0.30
31,17
63.84
64.28 0.18
0.44
AdaDFQ Qian
2023a
38.10
40.56
2.46
66.53
66.79 0.22
0.26
IntraQ Zhong et
2022b
4,93
66.47
Noise Optimization
HAST
al.
2023a
42.58
50.69
0.38
8.11
66.91
67.19
0.21
0.28
TexQ
Chen et
2023
50.28
51.58
0.30
1.30
67.73
67.85
0.16
0.12
sidering that simple fine-tuning methods unintentionally lead the quantized model to rely on incorrect image patterns for predictions, CAM alignment shows clear advantage over feature alignment. By aligning the saliency maps between the original and quantized models, we ensure that the critical predictive regions remain consistent, thereby preserving the interpretability and accuracy of the model's 's decisions, which is more effective than merely matching activation maps.
We conduct an ablation study to compare the performance. Teports the 3bit ZSQ accuracy of ResNet-18 88 model on ImageNet dataset when applying CAM alignment (12) and feature alignment (FA). CAM alignment shows clear advantage in performance over feature alignment both with and without other ideas (II & (3) of SYNQ.
Furthermore, we compare the computational overhead of two alignments. They have the same time complexity since both maps are obtained by applying backpropagation through the network. In practice, the average training time (in seconds) of the ResNet-18 model per epoch 113.40 + 2.28 seconds and 113.27 + 2.34 seconds for CAM alignment and feature alignment, respectively. Regarding training time, the gap between two methods is negligible. Overall, CAM alignment directly targets the second challenge (prediction based off-target patterns), thereby showing notable performance enhancement with similar training time compared to the feature alignment of HAST.
C.6 APPLICATION ON DIFFERENT BASELINES
We evaluate the ZSQ performance when applying SYNQ on different baselines investigate the adapt- ability of the proposed method. Specifically, we select three generator-based baselines (GDFQ (Xu al. 2020). Qimera (Choi al. (02T), and AdaDFQ (Qian al. (023aa) and three noise opti- mization baselines (IntraQ (Zhong et al.) 2022b), HAST (Li et al. 2023a). and TexQ (Chen et al.) 2023)). While generator-ban based methods simultaneously train the generator and quantized model, noise optimization methods generate the synthetic dataset first and fine-tune with it afterwards. (reports the accuracy of quantized models and the percent point improvement (" ("Imp. "). SYNQ consistently enhances the ZSQ performance of for all baseline methods, specifically up 31.17%p. The increasing effectiveness in lower bit-width experiments highlights the superiority of SYNQ. In overall, SYNQ is powerful and versatile since it is easily integrated with any ZSQ method utilizing synthetic dataset, enhancing their performance across various settings.
C.7 ANALYSIS ON ZERO-SHOT POST-TRAINING QUANTIZATION SETTING
In this paper, we mainly discover ZSQ under settings that additional fine-tuning of the quantized model is performed, namely Quantization-Aware Training (QAT) setting. However, a recent work, Genie (Jeon et al.) 2023b) has explored ZSQ under Post-Training Quantization (PTQ) setting, where no additional fine-tuning is needed. In this section, we first brietly discuss the preliminaries on uniform quantization. Then, we compare the settings of SYNQ and Genie to mitigate possible misunderstandings and explain why Genie is neglected from competitors in the main experiments (Table [[]. Lastly, we integrate SYNQ with Genie and evaluate its quantization performance to highlight the superior adaptability and broad applicability of SYNQ.
20

---

Published as a conference paper at ICLR 2025
Table 8: Zero-shot Quantization accuracy [%] of ResNet-18 model on ImageNet quantized with Genie (Jeon et al. 2023b) as baseline. WPAQ indicates that weights and activations are quantized each into Pbit and Qbit, respectively. SYNQ shows consistent improvements quantization performance when applied to Genie, across various quantization bits.
Method
W2A2
W2A4
W3A3
W4A4
Average
Genie Jeon et al.
2023b
54.01
65.10
66.84
69.66
63,90
+ +SYNQ (Proposed)
54.97 + 0.35
65.88
0.27
67.42 + 0.21
69.88 + 0.19
64.54
Preliminaries on Uniform Quantization. We describe the preliminaries on the uniform quantization scheme. Uniform quantization is to represent the weight and activation of a higher- bit given network, within lower bit integers. To perform Bbit uniform quantization, we first linearly scale the distribution 1
of weight matrix W within a range of [-20-1 2 1 1]. then map weight values into equally
scheme (Gupta al., 2013). Given a matrix W with the size of quantization granularity, the Bbit quantized matrix W by uniform quantization is calculated as shown in Equation
divided integers following the rounding-to-neatt
7
W
3 3--a
W
- 2- where s
2 - 2
+
and [ox, (8] is the clipping range corresponding to 1-2B-1,2 choosing the clipping range [a, 3] for W is essential, as defines the scaling factor s and zero-point required for accurate quantization. A commonly used approach, known Min-max Quantization, involves setting a and 3 to the minimum and maximum values of W, respectively. The key advantage of min-max quantization is its simplicity and effectiveness, requiring calibration to define the clipping range. This makes it the most essential and unbiased baseline for fair comparison among diverse quantization methods, especially under QAT setting.
1] in integer scale. Properly
However, min-max quantization is vulnerable to outliers, especially when quantizing activation. Outliers significantly expand the range [a. (8), leading lower precision for the majority of values in W during quantization. Parameterized clipping (Choi et al. 2018) mitigates this outlier effects by allowing the range to adapt based on the data distribution. This method leverages calibration dataset to identify clipping thresholds a and /3 that best represent the data distribution for improved quantization precision. Building on this approach, advanced techniques such as adaptive round- ing (Nagel et al. 2020), learned step size (Esser et al., (020), random dropping (Wei et al., (022). block reconstruction (Li et al.) 2021). and scale reparameterization (Li et al.) 2023c) have been developed under PTQ settings, further enabling improved quantization without fine-tuning.
Direct Comparison with Genie (Jeon et 20233). We compare the settings between SYNQ and Genie (Jeon et al., 2023b). Whereas SYNQ optimizes the parameters 09 of the quantized model in QAT, Genie follows a PTQ scheme, focusing on the scale factor and zero-point z. To achieve this, Genie combines joint optimization framework for PTQ (Genie-M) for and soft-bit V (refer to AdaRound (Nagel et al. 2020) and Genie (Jeon et al., 20236) for details) with advanced techniques such LSQ (Esser al. (0200) QDrop (Wei et al.) (022), and BRECQ (Li et al. 2021). Moreover, Genie fixes the quantization bits of the first layer's weights and activation, as well as the last layer's activation, to 8 bits across all experiments. On the other hand, SYNQ and other QAT approaches that are listed in Table [ Jnniformly assign bits across all layers, using min-max quantization as the baseline. Due the differences in quantization strategies and experimental conditions, evaluating Genie alongside zero-shot QAT methods is challenging.
Accuracy in Zero-shot Post- Tost-Training Quantization. While Genie's 's PTQ framework does not sup- port experiments under min-max quantization, our proposed method SYNQ enables synthesis- aware fine-tuning for any ZSQ method that generates and utilizes synthetic datasets. Consequently, we evaluate the ZSQ accuracy under Genie's 's setting both with and without SYNQ in Table We follow Genie for the experimental settings (see Table 3 and Appendix A of the Genie paper) and carry out implementation using their official code. Note that the size of synthetic dataset is 1,024 for this experiment. Applying SYNQ leads to consistent gains in ZSQ accuracy for Genie across various bit settings, showing average enhancement of 0.66%p. These results clearly demonstrate the superiority of SYNQ. showcasing its compatibility with diverse quantization techniques other than min-max quantization.
21

---

Published as a conference paper at ICLR 2025
Table 3bit ZSQ accuracy [%]] of a ResNet-18 model pre-trained on ImageNet dataset when four different types of noise is injected into its synthetic dataset. See Section C.8 for details.
Noise
Method
Baseline
Gaussian
Speckle
S && P
Uniform
33.01
29.80
40.82
39.85
TexQ Chen al. 2023
50.28
(-34.35%)
(-40.74%)
(-18.81 1%)
(-20.74%)
43.29
35,62
46.81 (-10.01%) %)
45.11
SynQ (Propesed)
52.02
(-16.78%) %
(-31.53%%
(-13.28%)
C.8 ANALYSIS ON THE ROBUSTNESS TOWARDS NOISE
We validate the robustness of SYNQ towards different types of noise. Table compares how TexQ and SYNQ perform quantization when four distinct noise types are introduced into their synthetic datasets. Specifically, we report the 3bit ZSQ accuracy of a ResNet- 18 model pre-trained on ImageNet dataset with four types of noise: Gaussian, speckle, Salt-and-Pepper (S & P), and uniform (Bovik) 2010). We have two observations from the result. First, the low-pass filter effectively minimizes accuracy degradation across various noise types, surpassing the baseline in capacity. Second, the effect of low-pass filter and the influence of noise both vary significantly depending on the noise type. Thus, our future work involves tailoring the noise filtering approach to better handle specific noise types in synthetic datasets.
C.9 PERFORMANCE REGARDING THE SIZE OF SYNTHETIC DATASET
We analyze the performance variation of SYNQ according to the size of synthetic dataset. Figure IT shows the 3bit ZSQ accuracy of ResNet-18 18 model pre-trained the performance while doubling the size of synthetic dataset used for training from 80 to 5,120 images. Note that compare the model performance trained with 5,120 samples for the main ex- periments (see Appendix D). We have observations from the result. First, SYNQ achieves higher performance when trained with greater number of images. Although the incremental gains begin to drop near 1,000 images and onward, we expect that gen- erating more than 5,120 synthetic images could achieve superior accuracy than the results reported in Table [ Second, SYNQ outperforms TexQ even when training with only half the dataset, demonstrating the effectiveness of synthesis-aware fine-tuning in- troduced by SYNQ. In overall, SYNQ shows better performance compared to the baselines, with performance improving as the synthetic dataset size increases.
SynQ
TexQ
ImageNet dataset. We report
Aecuracy (%)
50
46
2500
5000
Size of Dataset
Figure 11: ZSQ accuracy regard- ing the size of synthetic dataset. See Appendix Cor details.
C.10 VISUALIZATION OF SYNTHETIC DATASET
Although SYNQ applicable any ZSQ methods that generate synthetic dataset, investigating 1) the training set utilized for the highest performance and 2) the effect of the low-pass filter (Idea 1) is essential understanding SYNQ. Figure 12 presents a visualization of images from three synthetic datasets before and after the low-pass filter. These datasets are generated following the baseline method which is detailed in Appendix [ by three different models: ResNet-20 model pre-trained on CIFAR-10 0 dataset (Figures a2d and 12b), ResNet-20 model pre-trained on CIFAR- 100 dataset (Figures 12c and 12d and ResNet- 18 model pre-trained on ImageNet dataset (Figures [2e [2e]and and [2f). In order to effectively visualize the effect of the low-pass filter, we set the filtering hyerpaaraeeter Du to 8, 8, and 40 for CIFAR-10, CIFAR-100, and ImageNet datasets, respectively. We have two observations from Figure 12 First, the visualized images show distinct patterns and differences across various classes. Second, the low-pass filter removes noise effectively while preserving essential features in the generated images, which are noticeable especially in lower resolution samples.
22

---

Published as a conference paper at ICLR 2025
(a) CIFAR-10
(h) CIFAR- 10 (filtered)
(c) CIFAR-100
(d) CIFAR-100 (filtered)
(c) ImageNet
(f) ImageNet (filtered)
Figure 12: Visualization of samples within the synthetic dataset before (left) and after (right) the low-pass filter, generated by (a, b) a ResNet-20 model pre-trained on CIFAR-10 dataset, (c, d) ResNet-20 model pre-trained CIFAR-100 dataset, and (e, ResNet- 18 model pre-trained on ImageNet dataset. See Appendix C.10 for details.
23

---

Published as a conference paper at ICLR 2025
SynQ
PLF
SynQ
PLF
SynQ
ToxQ
88.2
57.2
34
87.8
56.9
33
56.6
32
87.4
31
56.3
G
87.0
0.0
0.2 0.4
0.6 0.8
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
(a) R-20 CIFAR-10
(b) R-20 + CIFAR-100
(c) MV2 ImageNet
Figure Hyperparameter analysis difficulty threshold r with ResNer-20 (R-20) model pre-trained on CIFAR-10 10 and CIFAR-100 datasets, and MobileNetV2 (MV2) model pre-trained on ImageNet dataset. See Appendix C.11 for details.
C.I1 FURTHER ANALYSIS ON THE DIFFICULTY THRESHOLD
In Section and and Figure we conduct a hyperparameter analysis to analyze the robustness of SYNQ towards newly introduced hyperparameters. We investigate this aspect across various models and datasets to ensure SYNQ reflects similar tendencies across multiple settings. Figure shows the ZSQ accuracy of (a) ResNet-20 (R-20) model pre-trained on CIFAR-10 dataset. (b) a ResNet-20 (R-20) model pre-trained on CIFAR-100 dataset, and (c) a MobileNet-V (MV2) model pre-trained on ImageNet dataset, with different values. Note that Figure Tb) introduces the result of ResNet-18 18 model on ImageNet dataset. We also depict the performance of the state-of-the-at competitor as a red line for all figures. Note that SYNQ shows similar tendency across different settings, while (a) R-20 + CIFAR-10 10 maximizes with the value of 0.7. This is because the eтгоr rate of pre-trained models (see Figure 3 begins increase
higher difficulty level of approximately 0.65 for CIFAR- 10, compared to 0.5 for the others. In summary, the optimal 7 should provide nice trade-off between containing sufficient samples and not using wrong samples.
D DETAILS ON THE EXPERIMENTAL SETUP
We describe the details on the experimental setup, including datasets, competitors, hyperparameters, implementation, and training.
Datasets. We utilize three benchmark datasets, CIFAR- 10, CIFAR- (Krizhevsky al. 2009). and ImageNet (ILSVRC 2012) (Deng et al. (2009) to evaluate the classification accuracy of the quantized model obtained by SYNQ. directly use both CIFAR-I0 and CIFAR-I00 100 datasets Torch Vision package. Note that we utilize real datasets only for evaluation purposes.
Competitors. We briefly summarize the details of the competitors of SYNQ as follows:
GDFQ (X al.) 2020) is the first method to utilize a knwwledge-mateeeeeoooe generator to produce synthetic data which guided by both batch normalization statistics loss and cross-entropy loss.
ARC (Zhu et al. method.
2021 or AutoReCon is neural architecture search-based image reconstruction
Qimera (Choi al. 2021) superposed latent embeddings to generate synthetic boundary supporting samples.
AIT (Choi et al., 2022) improves the loss function and gradients for ARC to generate better samples, which we denote it as AIT ARC.
IntraQ (Zhong et al. 20226) highlights the intra-class heterogeneity and retains this property in the synthetic dataset for better performance.
24

---

Published as a conference paper at ICLR 2025
AdaSG (Qian et al.)
2023b) plots the ZSQ problem zero-sum game between two players, the generator and the quantized network to generate adaptive samples for the synthetic dataset.
AdaDFQ (Qian et al.
2023a further generalizes AdaSG to adaptively regulate the adaptability
of the synthetic samples.
2023a) pays more attention difficult samples by generating difficult samples and further promoting the sample difficulty when training the quantized model.
HAST (Li et al.
TexQ (Chen al.
2023) retains the texture feature distributions within the synthetic dataset by using synthetic calibration centers to calibrate samples.
PLF (Fan et al. 2024) evaluates synthetic data to assign pseudo-labels with different reliability to avoid misleading training.
Additionally, we compare with PSAQ- (L Lt al. 2022 and Genie (Jeon et al. 2023b) for the ViT (see Section 5.3) and PTQ (see Appendix C.7) experiments, respectively.
Baseline. We introduce the baseline method to produce the synthetic dataset for the main results and observations (e.g. Tables We adopt calibration center synthesis (Chen et al. 2023). difficult sample generation, and sample difficulty promotion (Li et al., 2023a). Producing the synthetic dataset consists of three stages. First, we produce calibration centers following Chen al. (2023). one center each for all possible classes. Second, we produce the synthetic dataset with two additional losses, harr-sample--nhanme
inception loss LHIL from HAST and layered batch normalization statistics alignment loss CL N: added on top of Equation Lastly, we attach perturbation to each image following sample difficulty promotion from HAST, to make generated samples more difficult for the quantized model. We select this baseline that combines only the synthetic dataset production part of the two papers HAST and TexQ, order to intentionally set baselines only for the first step and replace the existing fine-tuning process with the proposed synthesis-awer aware fine-tuning. Refer to the original papers for further details.
Hlyperparameters. We conduct a grid search to validate hyperparameters, and select the set with the best performance. Table [0 reports the searched hyperparameter ranges of SYNQ for the ImageNet dataset experiment. For competitors, search within the range described in each paper. We conduct 5 iteration for each experiments and report the mean and standard deviation of the results.
Table 10: Hyperparameter ranges for SYNQ.
Hyperparameter
Range
a1
[0.2, 0.4, 0.6, 0.8, 1]
02
[0.01, 0.04, 0.1]
a
[0.4, 2.5] [0.25, 0.5, 0.75, 1, 1.5, 2]
Ap
ACE
[5, 5e-1, 5e-2, 5e-3]
Do
[20, 40, 60, 80, 100]
T
[0.5, 0.55, 0.6, 0.65, 0.7]
ACAM
[20, 50, 100, 200, 300, 500, 2000]
CAM Technique
[CAM, Grad-CAM,
Implementation and Machine. We implement SYNQ with PyTorch and Torch Vision libraries in Python. For the other methods, we reproduce the result using their open-source code if possible and implement them otherwise. All of our experiments were done at a workstation with Intel Xeon Silver 4214 and RTX 3090.
Training Details. We first generate the calibration centers with constant learning rate of 0.05, following TexQ (Chen et al., 2023). Then, we optimize samples using the loss function described in Equation D with the Adam optimizer generate the synthetic dataset. This optimizer has momentum 0.9 and an initial learning rate of 0.5. The synthetic images are updated over 1,000 iterations, with the learning rate decaying by factor of 0.1 whenever the loss does not decrease for
25

---

Published as a conference paper at ICLR 2025
50 consecutive iterations. For all datasets, batch size of 256 is used, resulting in the generation of total of 5,120 images. For the fine-tuning of the quantized model, the procedure follows Equation 6 employing SGD with momentum of 0.9 and a weight decay of le-4. The batch size is set to 256 for CIFAR-10/100 and 16 for ImageNet. Initial learning rate is searched within the range of (1e-4, 1e-5, 1e-6) and is decayed by factor of 0.1 over training epochs nep 100.
VIT Quantization Experiment. We compare the ZSQ precision of PSAQ-VIT (Li et al. 2022) with that of SYNQ applied on it. PSAQ-ViT generates the synthetic dataset based the patch similarity, substituting the batch normalization statistics in CNN models. The pre-trained DeiT- Tiny, DeiT-Small (Touvron al., (02TT. Swin-Ting Tiny, and Swin-Small (Liu et al. 2021b) models on
ImageNet dataset is obtained from timm (Wightman, 2019) library. We follow Li et al. (2022) for the experimental setup, where only 32 images are used from the synthetic dataset.
26


Reviews:
[{"review_id": "dXXIEutePC", "reviewer": "Reviewer_L8UP", "summary": "The paper presents SYNQ (Synthesis-aware Fine-tuning for Zero-shot Quantization), a novel framework designed to address the challenges associated with zero-shot quantization (ZSQ) of pre-trained models, particularly in scenarios where training data is inaccessible due to privacy or security concerns. SYNQ tackles three main issues: noise in synthetic datasets, off-target pattern predictions, and misguidance from erroneous hard labels. The proposed method employs a low-pass filter to reduce noise, optimizes class activation map (CAM) alignment to ensure correct image region prediction, and uses soft labels for difficult samples to prevent misguidance. The authors show that SYNQ achieves state-of-the-art accuracy in image classification tasks compared to existing ZSQ methods.", "questions": "1. How does SYNQ handle different types of noise, and is its performance consistent across various noise levels? Before and after the low-pass filter, what is the changes of generated images?\n2. There are more related papers should be included, such as 'Data-Free Learning of Student Networks', ‘Data-free network quantization with adversarial knowledge distillation’ and others.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. SYNQ offers a unique solution to the problem of quantizing models without access to training data, which is a significant contribution to deploying neural networks on edge devices.\n2. Addressing Key Challenges: The paper clearly identifies and addresses three major challenges in ZSQ, providing a comprehensive approach to improving the accuracy of quantized models.\n3. Empirical Validation: Extensive experiments demonstrate SYNQ's effectiveness, showing improvements in classification accuracy over existing methods.", "weaknesses": "1. While the paper focuses on image classification, it's unclear how SYNQ would perform in other tasks such as object detection or segmentation.\n2. The paper could provide more details on the computational overhead introduced by SYNQ, especially the impact of the low-pass filter and CAM alignment.\n3. The paper could benefit from a deeper analysis of SYNQ's robustness to different types and levels of noise in synthetic datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "49QbPhmk9M", "reviewer": "Reviewer_bwpy", "summary": "The paper proposed a synthesis-aware fine-tuning method, SYNQ, to improve zero-shot quantization (ZSQ) performance. SYNQ defines the issues of ZSQ as follows: 1) high-frequency noise in the generated synthetic dataset, 2) predictions based on off-target patterns, and 3) misguidance by hard labels. SYNQ effectively addresses these issues to improve ZSQ performance through the use of a low-pass filter, CAM alignment, and hard label filtering.", "questions": "1. While SYNQ has been evaluated on W3 and W4, how does it perform under extremely low-bit (e.g., 2-bit) conditions? For example, GENIE [1], one of the ZSQ methods, demonstrated performance not only on W3 and W4 but also on W2. It would be beneficial to add it as a baseline and show performance in low-bit settings as well.\n2. What is the performance variation according to the size of the generated synthetic dataset?\n\n[1] Jeon et al., \"GENIE: Show Me the Data for Quantization. \", CVPR 2023.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The observations regarding the three limitations of ZSQ are interesting, and the proposed method appears feasible.\n2. The performance is validated through a variety of experiments. Specifically, experiments were conducted to verify the performance of SYNQ by comparing it with various ZDQ baselines on not only CNN-based models but also ViT-based models.\n3. The detailed analyses of the three components of SYNQ enhance the persuasiveness of the methodology.\n4. This paper is well-written and easy to follow.", "weaknesses": "1. Although the observations presented in the paper are interesting, most of the experimental evidence provided was gathered under limited conditions. For instance, in Figure 5, experiments were shown only for TexQ among various baseline models, and the analysis for CIFAR-10 and CIFAR-100 used as benchmarks in Table 1 was omitted.\n2. In Figure 2, the heat map is shown only one sample image.\n\nFor these reasons, it is difficult to be certain whether the presented observations are phenomena that can be observed only in limited baselines and datasets or are generally seen across ZSQ methods. Therefore, the authors should provide experimental evidence across various baselines and datasets beyond the limited settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oiYbjkGiQw", "reviewer": "Reviewer_jAz6", "summary": "This work points out several problems that prior works of Data-free quantization (DFQ) have.\nFirst, synthesized images are noisy compared to their real counterparts.\nSecond, models quantized with synthesized images tend to predict based on incorrect image patterns.\nIn addition, the paper claims that using hard labels on hard samples can cause misguidance.\n\nTo resolve these problems, the paper proposes three methods.\n- The paper revisits classical signal processing and removes the noise of generated images with a low-pass filter.\n- To align activation maps between a full precision model and a quantized model, the paper proposes to use grad CAM as a loss function.\n- By considering model outputs as the difficulty of the sample, the paper proposes to omit CE loss if the input is a difficult sample.", "questions": "- The reviewer thinks that off-target prediction problems may occur under the DFQ scenario, not general quantization scenarios. Nevertheless, did the authors examine whether the problem occurred with real data?\n- What is the experimental setting of the baseline in Table 3 such as the quantization algorithm? (the same result is not in the Table 1)\n- In Figure 7,\n    - Does this tendency still hold with other models and datasets? For instance, the distribution of model outputs that are used as a measurement of difficulty can be different if the number of classes differs. With various models (e.g., ResNet, MobileNet, etc) and datasets (e.g., CIFAR10, CIFAR100, ImageNet), is the optimal $\\tau$ always 0.5 or values close to it?\n    - The magnitude of $\\lambda_{CAM}$ in (a) are much larger than those of $\\lambda_{CE}$. Is the magnitude of CAM loss on average much smaller than that of $\\lambda_{CE}$ loss?\n- In Table 5 of the appendix,\n    - Those experiments are based on 3 works that use a generator. However, SynQ adopts noise optimization for generating images. Why aren’t other works that adopt noise optimization addressed?\n    - Those 3 works are improved with SynQ. However, they are worse than SynQ itself. Can the authors’ opinions about this provided?\n    - How about applying SynQ to other works based on noise optimization?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper tackles the limitations of previous works well.\n- The paper tries to denoise synthesized images with a loss-pass filter. This idea is a good point that highlights the importance of classical techniques and theories in the recent AI era.\n- The paper identifies the off-target prediction problem that occurs only in the data-free quantization scenario. It is a good suggestion that analyzes the performance of a quantized model with grad GAM and uses it as a loss function.\n- The paper executes various experiments and ablation studies for validating proposals.", "weaknesses": "- The paper refers to the limitations of previous works too much. The same content is repeated over 3 times.\n- The investigation and analysis of prior works is insufficient.\n    - The paper notes that using hard labels can be harmful to the performance of quantized models, pointing out that previous works used both hard labels (CE loss) and soft labels (KL divergence loss). It can be a novelty that determines the usage of CE loss according to difficulty. However, there already exist several works that use a soft label instead of a hard label. For instance, Qimera proposed to use the coefficient of superposed latent embedding as soft labels. AIT also pointed out the importance of soft labels and used soft labels only for the loss function.\n    - The results of current state-of-the-art works are omitted. In the CNN domain, GENIE shows better performance than this work. Also, in transformer variants, PSAQ-ViT V2 shows better results. Those works should be addressed.\n- Generated images with SynQ can help understand the superiority of the proposal.  Please attach generated images with SynQ (before and after applying the low-pass filter)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "htcHTkatMY", "reviewer": "Reviewer_7rcv", "summary": "They propose SYNQ that targets to overcome the following limitations of current ZSQs:\n1. noise in the synthetic dataset; 2. off-target patterns; 3. misguidance by erroneous hard labels.", "questions": "The authors could refer to the Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The manuscript exhibits a coherent structure and is straightforward to navigate. Figures 1 through 3 effectively illustrate the key observations and the rationale behind our approach. Notably, the visualization of the Magnitude spectrum in Figure 1 is particularly engaging. To the best of my knowledge, this method is the first zsq to complete experiments on both cnn and vit, which is appreciated.", "weaknesses": "1.line382:The header method of Table 1 is incorrectly written as CIFAR dataset\n2. line237: The Low-pass filter (Section 4.2) directly modifies the image's impact on the model without using artificial visual features to judge whether it is good or bad. Does Low-pass filters have advantages over the existing ZSQ? \n3. Is Fig.2 different at different bit widths/networks? Is this a general situation in ZSQ?\n4. Lack of computational cost analysis comparison with state of the art methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)