Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples
Authors: Kanghyun Choi
Deokki Hong
Noseong Park
Youngsok Kim
Jinho Lee
Venue: NeurIPS 2021 Poster (2021)
Keywords: data-free quantization
generative method
boundary supporting samples

Abstract: Model quantization is known as a promising method to compress deep neural networks, especially for inferences on lightweight mobile or edge devices. 
However, model quantization usually requires access to the original training data to maintain the accuracy of the full-precision models, which is often infeasible in real-world scenarios for security and privacy issues.
A popular approach to perform quantization without access to the original data is to use synthetically generated samples, based on batch-normalization statistics or adversarial learning.
However, the drawback of such approaches is that they primarily rely on random noise input to the generator to attain diversity of the synthetic samples. 
We find that this is often insufficient to capture the distribution of the original data, especially around the decision boundaries.
To this end, we propose Qimera, a method that uses superposed latent embeddings to generate synthetic boundary supporting samples.
For the superposed embeddings to better reflect the original distribution, we also propose using an additional disentanglement mapping layer and extracting information from the full-precision model.
The experimental results show that Qimera achieves state-of-the-art performances for various settings on data-free quantization. 
Code is available at https://github.com/iamkanghyunchoi/qimera.

Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples
Kanghyun Choi
Deokki Hong*
Noseong Park 1.2
Youngsok Kim
Jinho Lee
Department of Computer Science, Yonsei University Department of Artificial Intelligence, Yonsei University (kanghyun. choi, uk.hong, noseong, youngsok, 1eejinho)@yonseei.ce kr
Abstract
Model quantization is known as a promising method to compress deep neural networks, especially for inferences on lightweight mobile edge devices. How- ever, model quantization usually requires access to the original training data maintain the accuracy of the full-precision models, which is often infeasible in real-world scenarios for security and privacy issues. A popular approach to perform quantization without access to the original data is to synthetically generated samples, based on batch-normalizatioo statistics or adversarial learning. However, the drawback of such approaches is that they primarily rely random noise input to the generator to attain diversity of the synthetic samples. We find that this often insufficient to capture the distribution of the original data, especially around the de- cision boundaries. To this end, we propose Qimera, a method that uses superposed latent embeddings generate synthetic boundary supporting samples. For the superposed embeddings to better reflect the original distribution, we also propose using an additional disentanglement mapping layer and extracting information from the full-precision model. The experimental results show that Qimera achieves sate-of-theeart performances for various settings data-free quantization. Code is available at atthts:// htt.//gittt thub. com/ wwt./coomoiccaeeeee mmmmiimmkaggyuuchhoi.. /qmmrra
Introduction
Among many neural network compression methodologies, quantization is considered promising direction because it can be easily supported by accelerator hardwares [T] than pruning (2) and is more lightweight than knowledge distillation (3) However, quantization generally requires some form of adjustment (c.g., fine- (ie-tuning) using the original training data [4,5] 7 8) to restore the accuracy drop due to the quantization errors. Unfortunately, access to the original training data is not always possible, especially for deployment the field, for many reasons such as privacy and security. For example, the data could be medical images of patients, photos of confidential products, pictures of military assets.
Therefore, data-free quantization is natural direction to achieve highly accurate quantized model without accessing any training data. Among many excellent prior studies (9] 1O 11 [2], generative methods [13] [4 15] have recently been drawing much attention due to their superior performance. Generative methods successfully generate synthetic samples that resemble the distribution of
original dataset and achieve high accuracy using information from the pretrained full full-precision network, such as match-normalizatoon statistics [15
[3) or intermediate features [1]
"Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

However, a significant gap still exists between data-free quantized models and quantized models fine- tuned with original data. What is missing from the current generative data-free quantization schemes? We hypothesize that the synthetic samples of conventional methods lack boundary supporting samples [T6], which lie on or near the decision boundary of the full-precision model and directly affect the model performance. The generator designs are often based on conditional generative adversarial networks (CGANs) [T7 [8] that take class embeddings representing class-specific latent features. Based these embeddings the centroid of each class distribution, generators rely on the input of random Gaussian noise vectors to gain diverse samples. However, one can easily deduce that random noises have difficulty reflecting the complex class boundaries. In addition, the weights and embeddings of the generators are trained with cross-entropy (CE) loss, further ensuring that these samples are well-separated from each other.
In this work, we propose Qimera, method for data-free quantization employing superposed latent embeddings to create boundary supporting samples. First, we conduct motivational experiment to confirm our hypothesis that samples near the boundary can improve the quantized model performance. Then, we propose novel method based on inputting superposed latent embeddings into the generator to produce synthetic boundary supporting samples. In addition, we provide two auxiliary schemes for flattening the latent embedding space so that superposed embeddings could contain adequate features.
Qimera achieves significant performance improvement over the existing techniques. The experimental results indicate that Qimera sets new state-of- of-the-art performance for various datasets and model settings for the data-free quantization problem. Our contributions are summarized as the following:
We identify that boundary supporting samples form important missing piece of the current state-of-the-ee data-free compression.
We propose using superposed latent embeddings, which enables a generator to synthesize boundary supporting samples of the full-precision model.
We propose disentanglement mapping and extracted embedding initialization that help train a better set of embeddings for the generator.
We conduct an extensive set of experiments, showing that the proposed scheme outperforms the existing methods.
2
Related Work
2.1 Data-free Compression
Early work on data-free compression has been led by knowledge distillation [3] which usually involves pseudo-data created from teacher network statistics [19 20). Lopes et al. 19 suggested generating pseudo-data from metadata collected from the teacher the form of activation records. Nayak et al. (20) proposed a similar scheme but with a zero-shot approach by modeling the output space of the teacher model as Dirichlet distribution, which is taken from model weights. More recent studies have employed generator architectures similar to GAN (2T] to generate synthetic In the absence of the original data for training, DAFL (22) used the teacher model to replace the discriminator by encouraging the outputs to be close hot distribution and by maximizing the activation counts. KegNet (26) adopted a similar idea and used low-rank decomposition to aid the compression. Adversarial belief matching (23) and data-free adversarial distillation (24) methods suggested adversarially training the generator, such that the generated samples become harder to train. One other variant is to modify samples directly using logit maximization as in DeepInversion [25). While this approach can generate images that appear natural to human, it has the drawback of skyrocketing computational costs because each image must be modificd using backpropagation.
samples replacing the original data (22 23 24 25
Data-free quantization is similar to data- free knowledge distillation but is a more complex problem because quantization errors must be recovered. The quantized model has the same architecture as the full-precision model; thus, the early methods of cost-training quantization were focused on how
to convert the full-precision weights into quantized weights by limiting the range of activations (9)
12], correcting biases [9] 10). and equalizing the weights [10 12 ZeroQ [15] pioneered using
synthetic data for data-free quantization employing statisties stored in natch-normalization layers.
GDFQ 13 added a generator close to the ACGAN [8 to generate better synthetic samples, and

---

Data
Accuracy
Data-free
Fine-tuned with Original data
68.28%
Symthetic samples
63.39%
Synthetic + unconfusing real samples
63.91% (+0.52)
X
Synthetic + confusing real samples
65.75% (+2.36)
X
Qimera (This work)
65.10% (+1.71)
Table 1: Motivational Experiment
AutoReCon [27 suggests better performing generator found by neural architecture search. In addition, DSG [28] further suggests relaxing the batch-normalization statistics alignment to generate more diverse samples. ZAQ (14) adopted adversarial training of the generator on the quantization problem and introduced intermediate feature matching between the full-precision and quantized model. However, none of these considered aiming to synthesize boundary supporting samples of the full-precision model. Adversarial training of generators have a similar purpose, but it is fundamentally different from generating boundary supporting samples. In addition, adversarial training can be sensitive to hyperparameters and risks generating samples outside of the original distributions (29]. To the extent of our knowledge, this work is the first to propose using superposed latent embeddings to generate boundary supporting samples explicitly for data-free quantizations.
2,2 Boundary Supporting Samples
In the context of knowledge distillation, boundary supporting samples [16] are defined as samples that lie near the decision boundary of the teacher models. As these samples contain classification information about the teacher models, they can help the student model correctly mimic the teacher's behavior. Heo al. [16] applied adversarial attack (30) to generate boundary supporting samples and successfully demonstrated that they improve knowledge distillation performance. In AMKD [1], triplet loss used to aid the student in drawing a better boundary. Later, DeepDig 32 devised a refined method for generating boundary supporting samples and analyzed their characteristics by defining new metrics. Although boundary supporting samples have been successful for many problems, such as domain adaptation [33] and open-set recognition [34]. they have not yet been considered for data-free compression.
3 Motivational Experiment
To explain the discrepancy between the accuracy of the model fine-tuned with the original training data and the data-free quantized models, we hypothesized that the synthetic samples from generative data-free quantization methods lack samples near the decision boundary. To validate this hypothesis, designed experiment using the CIFAR-100 (35) dataset with the ResNet-20 network (36).
First, we forwarded images in the dataset into the pre-trained full- full-precision ResNet-20. Among these, we selected 1,500 samples (3% of the dataset, 15 samples per class) from samples where the highest confidence value was lower than 0.25, forming a group of 'confusing" samples. Then, we combined the confusing samples with synthetic samples generated from previous study (13) to fine-tune the
quantized model with 4-bit weights and 4-bit activations. We also selected as a control group an equal number of random samples from the images classified unconfusing and fine-tuned the quantized model using the same method.
The results are presented in Table The quantized model with synthetic + unconfusing real samples exhibited only 0.52%p increase the accuracy from the baseline. In contrast, adding confusing samples provided 2.36%bp improvement, filling almost half the gap towards the quantized model fine-tuned with the original data. These results indirectly validate that the synthetic samples suffer from a lack of confusing samples (i.e., boundary supporting samples). We aim to address the issue in this paper. As we indicate in Section 5 Qimera achieves 1.71%p %op performance gain for the same model in data-free setting, close to that of the addition of confusing synthetic samples.
3

---

Quantiized model boundary
Synthetic boundary supporting samples
Full precision model boundary
Synthetic Images generated
(b) Synthetic boundary supporting samples with su- perposed latent embeddings (proposed).
with Gaussian noise.
Diagram of synthetic samples in the feature space of the full-precision network. The black curves represent the decision boundary of the full-precision model, which are considered ideal for the quantized model (orange dotted curves) to mimic. When synthetic images are generated with per-class embeddings and noises as in (a), their features do not support the decision boundary, whereas the proposed approach (b) generates the boundary supporting samples, helping the quantized model to set the near-ideal decision boundary.
Figure 1:
4 Generating Boundary Supporting Samples with Superposed Latent Embeddings
4.1 Baseline Generative Data-free Quantization
Recent generative data-free quantization schemes 13 14] employ GAN-like generator to create synthetic samples. In the absence of the original training samples, the generator G attempts to generate synthetic samples so that the quantized model Q can mimic the behavior of the full-precisiog model P. For example, GDFQ (13), the loss function LGDFQ
Lanre(G) = (G)(G) + ac
where the first term Lce guides the generator output clearly classifiable samples, and the second term LaNS aligns the suach-oommllization statistics of the synthetic samples with those of the batch-normalization layers in the full- full-precision model. In another previous work ZAQ [14).
(G) (G) + 3C + (G)
where the first term L. separates the prediction outputs of P and Q, and the second term LG separates the feature maps produced by P and Q. These two losses let the generator be adversarially trained and allow it to determine samples where P cannot mimic Q adequately. Lastly, the third term Lo maximizes the activation map values of P so that the created samples do not drift too far away from the original dataset.
The quantized model Q is usually jointly trained with G, such that
LGDFQ(Q) = + 8C +Ckk())
(3)
for GDFQ, and
(Z)Q(Q) -CP -CC
(4)
for ZAQ, respectively, where LKD from Eq. 3 is the usual KD loss with Kullback-Leibler divergence.
While these two methods exhibit great performance, they both model the distribution of per-class
embeddings in the latent input space Gaussian distribution, and generate diverse samples using random Gaussian noise inputs. However, based the Gaussian distribution, it difficult correctly reflect the boundary between the two different classes, especially when all samples have one-hot simplified wooddimensional space. With samples generated from gaussian noise (Figure Tab. the two per-class distributions are far away, and a void exists between the two classes (see Figure 3b for plots from experimental data). This can cause the decision boundaries to be formed at nonideal regions. One alternative is to noise with higher
labels. Figure [ visualizes such problems

---

Extracted Embedding Initialization
Disentanglement Mapping
%6
Eyo
Conu,
PC
S
Xx--1
Eyg--1
CoRvy
FCg
LGe
N(0.1)
Embedding Superposer
9
Figure An overview of the proposed method. Proposed components are denoted as colored shapes.
variance, but the samples would overlap on a large region, resulting in too many samples with incorrect labels.
Furthermore, in the above approaches, the loss terms Eqs. and anddasum such as LCE and C. encourage the class distributions to be separated from each other. While this is beneficial for generating clean, well-classified samples, also prevents generating samples near the decision boundary, which is
necessary for training a good model [16 31.. Therefore, in this paper, we focus on methods to generate synthetic samples near the boundary from the full-precision model P (Figure lb).
4.2 Superposed Latent Embeddings
The overview of Qimera is presented in Figure 2
Often, generators use learned embeddings to create samples of multiple classes [18] 37 Given input one-hot label y representing one of C' classes and random noise vector generator G uses an embedding layer E E R to create synthetic sample é:
G(z + Ey).
Z 1).
(5)
To create boundary supporting samples, we superpose the class embeddings so that the generated samples have features lying near the decision boundaries of P. With two embeddings superposed, new synthetic sample á' becomes
f' G(z + (AE,, + ()-)))E,))
2 ~ (0, 1), 0 A
(6)
To avoid too many confusing samples from complicating the feature space, we also apply soft labels in the same manner regularizer:
Ay + A)y2,
0 OSASI.
Generalizing to K embeddings, Eqs. and become
K
K
(2', 8') = (G(S(E)),
AbRK) S(e) =
+
ABER
2 2~ N(0,1)
(8)
A; = SoSSooftal(lf = exp(p:) (ezp(pu)),
P N(0,
99)
where e E IR Dx has K embeddings from as the column vectors (i.e., e= E.g...... E and S RD + RD is a superposer function. Applying this to existing methods is straightforward and incurs only small amount of computational overhead. Similar to knowledge distillation with boundary supporting samples [16 31), the superposed embeddings are supposed to help transfer the decision boundary of the full precision (teacher) model to the quantized (student) model.
Although the superposed embedding scheme alone produces substantial amount of performance gains, the generator embedding space is often not flat enough [38]; therefore, linearly interpolating them can result in unnatural samples [39]. To mitigate this, the embedding space used in Qimera should possess two characteristics. First, the embedding space should be as flat as possible so that the samples generated from Eq. reffect the intermediate points in the feature space. Second, individual embeddings should still be sufficiently distinct from each other, correctly representing the distance between each class distribution. In the remaining two subsections, we describe our strategies for enforcing the embeddings contain the above characteristics.
5

---

4.3 Disentanglement Mapping
To perform superposing in flatter manifold, we added a learnable mapping function M : R - R before the embeddings are superposed, where D is the embedding dimension and d is the dimension of the target space. Thus, s' from Eq. 8 becomes S' G(S(m)), where mn = M (ex). Although we do not add any specific loss that guides the output space of M to be flat, training to match the output of the frel-precision model (i.e., LCF) using the superposed j' encourages M to map the input to flatter space. In practice, we modeled M as single-layer perceptron, which we call the disentanglement mapping layer. The experimental results from Section [ demonstrate that the disentanglement mapping provides a considerable amount of performance gain.
4.4 Extracted Embedding Initialization
For the embeddings to be flat, we want the distributions of the embeddings fused with noise to be similar to the feature space. For this purpose, we utilize the fully connected layer of the full precision model P. Given f. the output features of the full-precision model before the last fully connected layer, we minimize Ey dist(P(fly) P(glv)), where C' is the number of classes, is the input of the generator, and dist is some distance metric. We do not have knowledge of the distribution of f: thus, we modeled it as Gaussian distribution, such that fly ~ M 1). Therefore, solving it against Eq. simply yields E'y) = Py. In practice, we use the corresponding column from the weight of the last fully connected layer of the full-precise precision model because its weights represent the centroids of the activations. If the weights of the last fully connected layer W = |W1. W2...., wel, set My Wy. Our experiments reveal that extracting the weights from the full- cill-recciionn model and freezngg them already works well (see Appendix B.2). However, using them as initializations and jointly training them empirically works better. We believe this is because fully connected layers have bias parameters in addition to weight parameters. Because we do not extract these biases into the embeddings, slight tuning is needed by training them. This outcome aligns with the findings from the class prototype scheme used in self-supervised learning approaches (40 33
5 Experimental Results
5.1 Experiment Implementation
Our method is evaluated on CIFAR-10, CIFAR-100 100 [35] and ImageNet (ILSVRC2012 (41) datasets, which are well-known datasets for evaluating the performance of model on the image classification task. 1IFAR10/100 datasets consist of 50k training sets and 10k evaluation sets with 10 classes and 100 classes, respectively, and is used for small-scale experiments our evaluation. ImageNet dataset is used for large-scale experiments. It has 1.2 million training sets and 50k evaluation sets. To keep the data-free environment, only the evaluation sets were used for test purposes in all experiments.
For the experiments, we chose ResNet-20 (36) for CIFAR-10/100 and ResNet-18, ResNet-50, and MobileNetV2 (42) for ImageNet. We implemented all our techniques using PyTorch (43) and ran the experiments using RTX3090 GPUs All the model implementations and pre-trained weights
before quantization are from pytorchev library (44). For quantization, we quantized all the layers and activations using re-it linear quantization, described by (7), as below:
Omin 2n interval(n)
=
(10)
where f is the full- full-precision value, 6' is the quantized value, interval(n) is calculated as 2man Omin Omnn and maa are per-channel minimum and maximum value of 0.
2
To generate synthetic samples, we built a generator using the architecture of ACGAN 18] and added a disentanglement mapping layer after class embeddings followed by superposing layer. Among all batches, the superposing layer chooses between superposed embeddings and regular embeddings in p: p ratio. The dimension of latent embedding and random noise was set to be the same with the channel of the last fully connected layer of the target network. For CIFAR, the intermediate embedding dimension after the disentanglement mapping layer was set as 64. The generator was trained using Adam [45] with a learning rate of 0.001. For ImageNet, the intermediate embedding dimension was set to be 100. To maintain label information among all layers of the generator,

---

(a) Original data
(b) GDFQ
(c) Qimera
Figure 3: PCA plots of the features before the last layer of the full-precising model. In the plots of Qimera-generated samples (c), the black dots represent synthetic boundary supporting samples generated with the superposed latent embeddings, which fill the space between the clusters. GDFQ- generated samples (b) form clusters that are smaller than that of the original data (a) and lack samples the mid-ground. Best viewed in color.
apply conditional batch normalization (46) rather than regular batch normalization layer, following SN-GAN [47]. The optimizer and learning rate were the same that of CIFAR.
To fine-tune the quantized model Q, we used SGD with Nesterov (48) as an optimizer for Q with a learning rate of 0.0001 while momentum and weight decay terms as 0.9 and 0.0001 respectively. The generator and the quantized model Q were jointly trained with 200 iterations for 400 epochs while decaying the learning rate by 0.1 per every 100 epochs. The batch size was 64 and 16 for CIFAR and ImageNet respectively. While Qimera can be applied almost all generator-base chose adopt baseline loss funetions for training from GDFQ [13), because was stable and showed better results for large scale experiments. Thus, loss functions L(G) and L(Q) are equal to Eq. and Eq. 3 with
approaches,
0.1 and 6 = 1.0, following the baseline.
5.2 Visualizations of
Samples on Feature Space
To ensure that the superposed latent embedding creates boundary supporting samples, we conducted an experiment to compare generated synthetic samples feature space visually. The experimental results are based on the generators trained with ResNet-20 for CIFAR-10 dataset. The features were extracted from the intermediate activation before the last fully connected layer of the full-precision model. For the Qimera- based generator, we set K=2 during the sample generations for clarity.
After extracting the features, we projected the features into wwo-dimensional space using Principal Component Analysis (PCA) [49]. The results are presented Figure Compared with the original training set data (Figure Bap. the samples from the GDFQ-based generator (Figure 3b) show lack of boundary supporting samples and the class distributions are confined to small regions around the centroid of each class. A generator trained with Qimera, however, exhibits different characteristics (Figure 3cb. Samples that are generated from superposed latent embeddings are displayed as black dots. The black dots are mostly located on the sparse regions between the class clusters. This experimental result shows that our method, Qimera, successfully generates samples near the decision boundaries. In other words, superposed latent embeddings are not only superposed on embedding space but also the feature space, serving as synthetic boundary supporting samples.
5.3 Quantization Results Comparison
Table 2 displays the classification accuracy on various datasets, target models, and bit-width settings. Note that nwma means n-bit quantization for weights and m-bit quantization for activations. As baselines, we selected ZeroQ [13), ZAQ [14), and GDFQ [13] as the important previous works on generative data-free quantization. In addition, we implemented Mixup (50) and Cutmix (51) on top of GDFQ, which are data augmentation schemes that mix input images. To implement these schemes, we created synthetic images from GDFQ, and applied the augmentations to build training images and labels. All baseline results are from official code of the authors, where small amount of
7

---

Model (FP32 Acc.)
GDFQ +Cutmix
GDFQ +Mixup
Qimera (%p improvement)
Datasct
Bits
ZeroQ
ZAQ
GDFQ
ResNet-20 (93.89)
4w4a
79.30
92.13'
90.25
89.58
88.69 (-3,44)
91.26 ::0.49 (-0.87)
Cifar-10
5w5a
91.34
93.36
93.38'
92.75
92.79 (-0.59)
93.46 1.003 (+0.08)
ResNet-20 (70.33)
4w4a
47.45
60.42
63.39'
62.74
62.99 (-0.40)
65.10 -0.33 (+1.711
Cifar-100
5w5a
65.61
68.70'
66.12
67.51
67.78 (-0.92)
69.02 0022 (+0.32)
ResNet-18
4w4a
22.58
52.64
60.60"
58.90
61.72 (+1.12)
63.84 0033 (+3,24)
(71.47)
5w5a
59.26
64,54
68.40"
68.05
68.67 (+0.27)
69.29 00.16 (+0.89)
ImageNet
ResNct-50
4w4a
8.38
53.02"
52.12
51.80
59.25 (+6.24)
66.25 +0.90 (+13.23)
(77.73)
5w5a
48.12
73.38"
71.89
70.99
(-1.81)
75.32 0.09 (+1.94)
Mob
4w4a
10,96
0.107
59.43"
57.23
59.99 (+0.56)
61.62 0.39 (+2.19)
(73.03)
5w5a
59.88
62.35
68.11"
67.61
68.83 (+0.72)
70.45 0.07 (+2.34)
Highest among the baselines
Did not converge
Table 2: Comparison on data-free quantization schemes
Model
Qimera GDFQ
Qimera ZAAQ
Qimera
Datasel
GDFQ
ZAQ
AutoReCon
(FP32 Acc.)
AutoReCon
ResNet-20 (93,89)
4w4a
90.25
91.26 (+1)
92.13
93.91 (+ L.78)
88.55
91.16 (+2.61)
Cifar-10
5wSa
93,38
93,46 (4 0+08
93.36
93.84 ( 0.48)
92.88
93.42 (+ 0,54)
Cifar-100
ResNet-20 (70.33)
4w4a 5w5a
63.39
65.10 (+
60.42
69.30
8.88)
62.76
65.33 (.57)
66.12
69.02 (+2.90)
68.70
69.58 (- 0.88)
68.40
68.80 (+0.40)
Table 3: Performance of Qimera implemented on top GDFQ, ZAQ and AutoReCon
modifications have been made on the GDFQ baseline for applying Mixup and Cutmix. We report top-1 accuracy for each experiment. The numbers inside the parentheses of Qimera results improvements over the highest baseline.
The results demonstrate that Qimera outperforms the baselines at almost all settings. Its performance improvement is especially large for bitwidthh (4w4a) cases. For 5w5a setting, the gain still significant, and its performance reaches near that of the full-precision model, which represents the upper bound. In addition, Qimera is not limited to small datasets having a low spatial dimension. The ImageNet results prove that Qimera performs beyond other baselines with considerable gaps, large-scale dataset with many classes. Interestingly, the result of GDFQ+Mixup and GDFQ+Cutmix did not produce much noticeable improvement except for GDFQ+Mixup in 4w4a ResNet-18 18 and -50. This implies that a mixture of generated synthetic samples of different classes in the sample space not sufficient represent boundary supporting samples. In summary, Qimera achieves superior accuracy on various environments regardless of dataset or model scale.
5.4 Qimera on Top of Various Algorithms
In this paper, we have applied GDFQ 13 as baseline. However, design does not particularly depend on certain method, and can be adopted by many schemes. To demonstrate this, we implemented Qimera top of ZAQ [4 and AutoReCon (27). The results are shown in Table
AutoReCon (27) strengthens the generator architecture using a neural architecture search, and therefore applying this is no different from that of the original Qimera implemented on top of GDFQ. However, original ZAQ does not use per-class sample generation. To attach the techniques from Qimera, we extended the generator with an embedding layer, and added cross-entropy loss into the loss function as the following:
((G) = (G) (C0 + (G)
(11)
((Q) = -C) ( P (
(12)
where p was set to 0.1, and
are calculated based Eq. 8
For all cases, applying the techniques of Qimera improves the performance by a significant amount. Especially for Qimera + ZAQ, the performances obtained was better than those of our primary
implementation Qimera + GDFQ Unfortunately, did not converge with ImageNet, and thus we did not consider this implementation as the primary version of Qimera.
8

---

K
2
10
25
100
Datasct
Method
Best
p
0.4
0.7
0.4
0.7
0.7
0.4
0.7
Cifar-100 (ResNet-20)
Baseline
N/A
63.39
+SE'
63.77
64.55 (+1.16)
SF, DMP
64,73
64.76
64.01
64,22
63.60
64,32
63.82
64.63
64.76 (+1.37)
+SE EEI"
64.64
64.54
64.89
64,83
65.23
64.33
64,88
64.70
65.23 (+1.8)
SE, DM, EEI
64,90
64.76
65.10
64,52
64.72
64.4
64.63
64.79
65.10 (+1.71)
Method
K
100
250
500
1000
Best
p
0.4
0.7
0.7
0.7
ImageNot (ResNet-50)
Baseline +SE
N/A
N/A
NIA
N/A
N/A
N/A
N/A
NIA
52.12
57.34
60.93
59.27
62.91
63.12
64,09
61.04
60.89
64.09 (+11.98)
+SE DM
59.59
64.12
63.97
64.87
62.16
64.72
65.43
66.06
66.06 (+13.94)
+SE,
64.44
(+12.32)
SE, DM. BEI
61.43
63.87
62.16
64.5
59.82
66.25
59.18
65.12
66.25 (+14.13)
Superppseed Emboddings Disenianlemmnt Mapping
Eximated Embeddinng Initiilizaaion
Table 4: Ablation study and Sensitivity analysis
Cifar- 10
Cifar-100
Method
Dist. Ratio
Intrusion
Dist. Ratio
Intrusion
Mixup
2.44
0,800
3.14
0,400
SE Only
1.58
0.00260
1.64
0.053
SE+DM
1.67
0.00073
1.59
0.044
SE+DM+EEI
1.57
0.00013
1.52
0.029
Table 5: Embedding Distance Ratio and Intrusion Score
5.5 Further Experiments
We discuss some aspects of our method in this section, which are effect of cach scheme introduced in Section 4 on the accuracy of Q, a sensitivity study upon various p and settings, and a closer investigation into the effect of disentanglement mapping (DM) and extracted embedding initialization (EEI) on the embedding space. All experiments in this subsection are under 4w4a setting.
Ablation study. We conducted an ablation study by adding the proposed components one by one on the GDFQ baseline. The results are presented in Table As we can see, the superposed embedding (SE) alone brings a substantial amount of performance improvement for both Cifar- (1. (1.16%p) and ImageNet (11. .)98))). On Cifar- 100, addition of EEI provides much additional gain of 0.68% while that of DM is marginal. ImageNet, much larger than that of EEI. When both of DM and EEI are used together with SE, the additional improvement over the best among (+SE, DM) and (+SE, EEI) is relatively small. We believe this is because DM and EEI serve for the similar purposes. However, applying both techniques can achieve near-best performance regardless of the dataset.
the other hand, DM provides 1.97%p additional gain,
Sensitivity Analysis. Table also provides a sensitivity analysis for K (number of classes for superposed embeddings) and p (ratio of synthetic boundary supporting samples within the dataset). As shown the table, both parameter clearly have an impact on the performance. For Cifar-100, the sweet spot values for K and p are both small, while those of the ImageNet were both large. We believe because ImageNet has more classes. Because the number of class pairs (and the decision boundaries) is quadratic function of number of classes, there needs more boundary supporting samples with more embeddings superposed to draw the correct border.
Investigation into the Embedding Space. To investigate how DM and EEI help shaping the embedding space friendly to SE, we conducted two additional experiments with ResNet-20 in Table First, setting K=2 (between two classes), we measured the ratio of the perceptual distance and Euclidean distance between the two embeddings in the classifier's feature space. The perceptual distance is measured by sweeping X from Eqs. 6 and 7 from 0 to I by 0.01 and adding all the piecewise distances between points in the feature space (See Figure 5 in Appendix). We want the distance ratio to be close to 1.0 for the embedding space distribution to be similar to that of the feature

---

(a) Original Cifar-10
(h) Samples from GDFQ
(c) Samples from 7.AQ (d) Samples from Qimera
Figure 4: Synthetic samples generated for Cifar- 10 dataset. Each row represents one of the 10 classes, except for Z.AQ which generates samples without labels.
space. Second, we defined 'intrusion score', the sum of logits that are outside the chosen pair of classes. If the non-related classes. Therefore, lower scores are desired. For comparison, we have also included the Mixup as naive method and measured both metrics. It shows that the DM and EEI are effective for in terms of both the distance ratio and the intrusion score, explaining how DM and EEI achieves better performances.
is large, that means the synthetic boundary samples are regarded samples of
6 Discussion
6.1 Does it Cause Invasion of Privacy?
The motivation for data-free quantization that the private original data might not be available. Using a generator reconstruct samples that follow the distribution of original data might indicate the invasion of privacy. However, as displayed in Figure 4 at least with current technologies, there is no sign of privacy invasion. The generated data are far from being human interpretable, which was See Appendix (Figure 6) for more generated images.
also the case for previous works
15
14 13
Geirhos al. [52) reveal interesting property that CNNs heavily biased by local textures, global shapes. BagNet (53) supports this claim by confirming that images with distorted shapes but preserved textures can still mostly be correctly classified by CNNs. In such regard, is wonder that the generated samples are non-interpretable, because there only few combinations that maintains the global shape of all possibilities that preserve the textures. Nonetheless, not being observed does not guarantee the privacy protection. We believe it is subject for further investigations.
6.2 Limitations
Even though Qimera achieves superior performance, one drawback of this approach is the fact that it utilizes embeddings multiple classes to generate the boundary supporting samples. This restricts its application classification tasks and its variants. For example, generation of datasets such as image segmentation or object detection do not take class embeddings as the input. However, generators for such data not extensively studied yet, and envision adding diversity to them would require inputting some form of labels or embeddings similar to paint-to-image (54), which would allow Qimera be easily applied.
7 Conclusion
In this paper, we have proposed Qimera, simple yet powerful approach for data-free quantization by generating synthetic boundary supporting samples with superposed latent embeddings. We show in our motivational experiment that current state-of-th oftee-of-theart generative data-free quantization can be greatly improved by small set of boundary supporting samples. Then, we show that superposing latent embeddings can close much of the gap. Extensive experimental results on various environments shows that Qimera achieves state-of-the-art performance for many networks and datasets, especially for large-scale datasets.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by National Research Foundation of Korea (NRF) grant funded by the Korea goverament(MSITT) (No.2021R1 FIIIRIFAATRR F1A1IIII111666700 No.2020R1FII IIIA I and Institute of Information communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (No.2020-0-01361, Artificial Intelligence Graduate School Program (Yonsei University))
References
[1] Norman P Jouppi et al. "In -a-ddatcenttrr performance analysis of tensor processing unit". In: Proceedings of the International Symposium on Computer Architecture. 2017.
[2] Song Han, Huizi Mao, and William Dally. "Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding". In: International Conference on Learning Representations. 2016.
[3] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. "Distilling the knowledge in a neural network". In: Advances in Neural Information Processing Systems Workshops. 2014.
[4] Shuchang Zhou al. "Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients". In: arXiv preprint arXiv: 1606.06066666 06160 (2016).
[5] Bohan Zhuang al. "Towards effective low- bitwidth convolutional neural networks". In: Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition. 2018.
[6] Chuang Gan Lin Ji and Song Han. "Defensive quantization: When efficiency meets robustness". In: International Conference on Learning Representations. 2018.
[7] Benoit Jacob et al. "Quantization and training of neural networks for efficient integer-arilmmetic-only inference". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. 2018.
[8] Sheng Shen et al. "Q-BERT: Hessian based ultra low precision quantization BERT", In: Proceedings of the AAAI Conference on Artificial Intelligence. 2020.
[9] Ron Banner et al. Prost-training 4-bit quantization of convolution networks for rapid-deployment" In: arXiv preprint arXiv:
(2018).
[10] Markus Nagel et al. "Data-free quantization through weight equalization and bias correction". In: Proceedings of the JEEE/CVF International Conference on Computer Vision. 2019.
[11] Yoojin Choi et al. "Data-free network quantization with adversarial knowledge distillation". In: Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops. 2020.
[12] Ritchie Zhao al. "Improving neural network quantization without retraining using outlier channel splitting". In: International Conference Machine Learning. 2019.
[13] Shoukai Xu et al. "Generative low-bitwidth data free quantization". In: European Conference on Com-
puter Vision. 2020.
[14] Yuang Liu, Wei Zhang, and Jun Wang. "Zero-shot adversarial quantization". In: arXiv preprint arXiv:2103. 15263 (2021).
[15] Yaohui Cai al. "ZeroQ: A novel zero shot quantization framework". In: Proceedings the Conference on Computer Vision and Pattern Recognition. 2020.
the IEEE/CVF
[16] Byeongho Heo al. "Knowledge distillation with adversarial samples supporting decision boundary". Proceedings of the AAAI Conference on Artificial Intelligence. 2019.
[17] Mchdi Mirza and Simon Osindero. "Conditional generative adversarial nets". In: Advances Neural Information Processing Systems Workshops (2014).
[18] Augustus Odena, Christopher Olah, and Jonathon Shlens. "Conditional image synthesis with auxiliary classifier gans". In: International Conference on Machine Learning. 2017.
[19] Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. "Data-free knowledge distillation for deep neural networks". In: arXiv preprint arXiv:1710.0755 (2017).
[20] Gaurav Kumar Nayak et al. "Zero-shot knowledge distillation in deep networks". In: International Conference on Machine Learning. 2019.
[21] Ian J Goodfellow et al. "Generative adversarial nets". In: Advances in Neural Information Processing Systems. 2014.
[22] Hanting Chen et al. "Data-free learning of student networks". In: Proceedings of the IEEE/CVF Interna-
tional Conference on Computer Vision. 2019.
[23] Paul Micaelli and Amos Storkey. "Zero-shot knowledge transfer via adversarial belief matching". In: arXiv preprint
(2019).
[24] Gongfan Fang et al. "Data-free adversarial distillation". In: arXiv preprint arXiv: 1912. (2019).
11

---

[25] Hongxu Yin et al. "Dreaming to distill: Data-free knowledge transfer via deepinversion". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
[26]
Jaemin Yoo et al. "Knowledge extraction with observable data". In: Advances in Neural Information Processing Systems. 2019.
[27] Baozhou Zhu al. "AutoReCon: Neural Architecture Search-based Reconstruction for Data-free Com-
pression". In: arXiv preprint arXiv:2105.1211 (2021).
[28]
Xiangguo Zhang al. "Diversifying Sample Generation for Accurate Data-Free Quantization". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, pp. 15658-
15667.
[29] Hongyu Guo, Yongyi Mao, and Richong Zhang. "Mixup as locally linear out-of-manifold regularization". In: Proceedings of the AAAI Conference on Artificial Intelligence. 2019.
[30] Christian Szegedy ct al. "Intriguing properties of neural networks". In: International Conference on
Learning Representations. 2014.
[31] Zihe Dong et al. "Adversarial metric knowledge distillation". In: International Conference on Communi- cation and Information Processing. 2020.
[32] Hamid Karimi, Tyler Derr, and Jiliang Tang. "Characterizing the decision boundary of deep neural networks". In: arXiv preprint arXiv: 19... (2019).
[33] Kuniaki Saito et al. "Universal Domain Adaptation through Self Supervision". In: Advances in Neural
Information Processing Systems. 2020.
[34] Yujie Xu et al. "Open-set interference signal recognition using boundary samples: A hybrid approach". In: International Conference Wireless Communications and Signal Processing. 2020.
[35] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
URL: http:://ttast cs. utoronto. ca/ "krri/ieeearti features- 200-e----00--... Te.--TR.pfff
[36]
Kaiming He et al. "Deep residual learning for image recognition". In: Proceedings of the IEEE conference Computer Vision and Pottern Recognition. 2016.
[37] Tero Karras, Samuli Laine, and Timo Aila. "A style-based generator architecture for generative adversarial networks". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
[38] Georgios Arvanitidis, Lars Kai Hansen, and Spren Hauberg. "Latent space oddity: the curvature of deep generative models". In: International Conference on Learning Reprexentations. 2018.
[39] Samuli Laine. "Feature-based metries for exploring the latent space generative models". In: Interna- tional Conference on Learning Representations Workshops. 2018.
[40] Kuniaki Saito et al. "SSemissueervie
domain adaptation via minimax entropy". In: Proceedings of the IEFF/CVF International Conference on Computer Vision. 2019.
[41] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. "Imagenet classification with deep convolutional neural networks". In: Advances in Neural Information Processing Systems. 2012.
[42] Mark Sandler et al. "Mobilenetv2: Inverted residuals and lincar bottlenecks". In: Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition. 2018.
[43] Adam Paszke et al. "Automatic diffcrentiation in pytorch". In: Advances in Neural Information Processing Systems Workshops. 2017.
[44]
Computer vision models on PyTorch. URL:
[45]
Diederik P. Kingma and Jimmy Ba. "Adam: A Method for Stochastic Optimization". In: International Conference for Learning Representations. 2015.
[46] Harm de Vries et al. "Modulating carly visual processing by language". In: Advances in Neural Informa- tion Processing Systems. 2017.
[47] Takeru Miyato et al. "Spectral Normalization for Generative Adversarial Networks". In: arXiv preprint arXiv:1802.0 05957 (2018).
[48] Y. F. Nesterov. "A method for solving the convex programming problem with convergence rate O(1/k3y". In: Dokl. Akad. Nauk SSSR (1983).
[49] Karl Pearson F.R.S. "LIII. On lines and planes of closest fit to systems of points in space". In: The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science (1901).
[50] Hongyi Zhang al. "Mixup: Beyond empirical risk minimization". In: Internation Conference on Learning Representations. 2018
[51] Sangdoo Yun et al. "Cutmix: Regularization strategy to train strong classifiers with localizable features". In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
[52] Robert Geirhos et al. "ImageNet-traiee trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness" " In: International Conference on Learning Representations. 2018.
[53] Wieland Brendel and Matthias Bethge. "Approximating CNNs with Bag-of-locall lag-oo-local-Feattres models works surprisingly well on ImageNet". In: International Conference on Learning Representations. 2018.
12

---

[54] Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. "SinGAN: Learning generative model from single natural image". In: Proceedings the IEEE/CVF International Conference Computer Vision. 2019.
13


Reviews:
[{"review_id": "vgpMVazJqYh", "reviewer": "Reviewer_84sZ", "summary": "This paper addresses a popular data-free post-training quantization problem using a faux dataset of synthetically generated samples without depending on the BN layer statistics of the FP32 model. It proposes a superposed latent embeddings technique generating synthetic boundary samples to capture the distribution of the original data around the decision boundaries. The authors also introduce two auxiliary techniques to achieve additional performance gain.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-written, and its goal is well-oriented by providing a good motivational experiment. The proposed Qimera approach provides a valuable insight that the synthetic samples of conventional data-free compression methods lack samples near the decision boundary of the FP32 model. The paper is well-organized and provides a unifying view of several generative data-free quantization techniques pioneered by ZeroQ.\n\nThe novelty is moderate but not strong enough because of the following reasons: Specifically, (1) a generative data-free quantization framework to generate a synthetic dataset has been studied in recent literature, say ZeroQ(CVPR'2020), GDFQ(ECCV'2020), ZAQ(CVPR'2021), and DSG(CVPR'2021). The proposed work seems to extend more like incremental modifications to the existing methods by combining already known concepts and ideas. Their philosophy and overall approaches are quite similar by following a generative data-free quantization framework originated from ZeroQ. Moreover, (2) the significance of its contribution appears limited in scope, as the boundary supporting samples have already been successful for knowledge distillation, especially in data-free network quantization with adversarial knowledge distillation(CVPR'2020) and knowledge distillation with adversarial samples supporting decision boundary(AAAI'2019). Therefore, the authors employed the main idea of generating boundary supporting samples to apply data-free quantization instead of knowledge distillation. \n\nExperimental results look good. Still, its application is quite restricted to the classification task because of the inherent drawback of superposed latent embeddings to create samples of multiple classes, which is inferior to the other approaches like ZAQ and DSG extending the application to image segmentation and object detection beyond image classification. \n\nHere are a few concerns and questions:\n1. Please include the recently published DSG(Diversifying Sample Generation for Accurate Data-Free Quantization, CVPR'2021) paper as a reference. Comparison between the DSG and the proposed Qimera might be required. An additional evaluation comparing with DSG would be preferred. \n2. Both ZAQ and DSG have improved the previous GDFQ without using BN statistics. Adding their PCA plots would be better to understand Qimera's superiority over them. \n3. Is the amount of synthetic data generated by each benchmarking method all equal in the experiments conducted in this paper? It would be better to specify the amount of synthetic data used for fine-tuning in all methods.\n4. For experiments with Cutmix and Mixup applied to GDFQ in Table 2, is it reasonable to apply these augmentation techniques to data focused on local textures as mentioned in the Discussion? Such augmentation techniques are thought to be efficient when global shapes are distinct. Therefore, it is inappropriate to conclude the performance degradation by applying Cutmix and Mixup to these synthetic data with a lack of boundary supporting samples. If appropriate, it would be better to visualize it or organize a separate experiment to prove it.\n5. This paper experimentally demonstrates that synthetic samples generated near the decision boundary help the data-free quantized model's performance improve. The ablation study in Table 3 shows the performance changes of the model by Qimera's SE, DM, and EEI. The conceptual goal of SE seems to be on the opposite side of DM and EEI. Their combination would be quite strange because disentangling generally used in GAN is the opposite of superposing. The presented experimental results show the highest performance improvement is mainly due to the superposed latent embeddings. Does the ablation study setting still use the same amount of synthetic samples for SE, SE+DM, SE+EEI, and SE+DM+EEI? It would be good to provide valuable insights by carefully analyzing their performance difference because creating more synthetic samples could improve. Is there any other way to theoretically explain how the two auxiliary techniques improve using SE only? Or, if you visualize how DM and EEI can increase data near the decision boundary, you could see how boundary supporting samples improve the model's performance.\n\n\n=================== post-rebuttal comment ===================\n\nI thank the authors for their detailed response. The newly added experimental results address most of my concerns. Accordingly, I am happy to raise my rating. \n\n===========================================================", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and no sign of privacy invasion. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "40 hours"}, "extra_text": {}}, {"review_id": "o5Aeo6vdTiD", "reviewer": "Reviewer_syC3", "summary": "In the manuscript, the authors propose Qimera, which introduces superposed latent embeddings, disentanglement mapping, and extracted embedding initialization to generate boundary supporting samples.\nThese synthetic samples are useful for post-training data-free quantization.\nThe main contributions of the manuscript are:\n1) Pioneer at considering boundary supporting samples for data-free quantization,\n2) Propose Qimera and its components,\n3) Achieve decent experimental results, together with meaningful analysis.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper is generally interesting, and I may change my rate according to the rebuttal.\nStrength:\n1) The writing of this manuscript is good, with sufficient background knowledge, clear equations and explanations. Although the Qimera method has lots of components, the manuscript is still easy to follow.\n2) The related work section is thorough.\n3) Illustrations such as Figure 3 are helpful to validate the effectiveness of proposed methods in generating boundary supporting samples.\n4) The experimental results are quite good comparing with previous methods targeting using synthetic data on post-training quantization (PTQ).\n\nSuggestion:\nThe supplementary materials contain lots of additional experiments and ablation study, which are helpful to address a large portion of my concerns (for example, the sensitivity to other hyperparameters, comparing with more recent methods, etc). I suggest merging some of them into the main texts if possible.\n\nWeakness and Question:\n1) Regarding the disentanglement mapping, M is implicitly encouraged to map the inputs to a flatter space because we suppose unnatural samples can lead to a large loss. Some ablation study or statistics are required to validate the point.\n2) Qimera is good at generating synthetic images for PTQ. But will a better decision boundary be learned when the network has better effective capability (when applied with advanced quantization methods such as better quantizer or mixed-precision quantization)? Can Qimera improve on top of SOTA quantization methods? (ZAQ + Qimera experiments in supplementary material is a start, but ZAQ is not specifically targeting quantizers/precisions, and the ZAQ + Qimera results are not super promising).\n3) Why the distributions of the embeddings should be similar to the feature space in order to be flat?\n4) In addition to PTQ, it would be interesting to see if Qimera can help quantization-aware training (QAT) with synthetic data?\n5) From the ablation, solely applying SE can have a good improvement, but applying mixup on inputs doesn't work well. It would be good to see some experimental results on the source of contributions. Is embedding the only key factor? \n\n=========Post Rebuttal=========\n\nThanks the authors for the detailed rebuttal, which addressed many of my concerns. Based on the rebuttal and reviews from other reviewers, I will keep my rating and would like the paper to be accepted to NeurIPS 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The potential negative societal impact of this work is regarding data privacy. \nData-free quantization aims to quantize pretrained neural networks without access to the data they are trained on.\nThis generally requires an approximation of the distribution of training data derived/distilled from the neural networks.\nThe manuscript explicitly discuss this impact and show many results sampled in the approximated distribution of training data.\nSince most samples are not interpretable by humans, the proposed method is good to use (suppose that the texture of training data is not private and interpretable).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "RCBlNUXQ2W", "reviewer": "Reviewer_11Bt", "summary": "In this paper, the authors proposed Qimera to improve data-free quantization by generating samples around the decision boundary. Qimera uses superposed latent embeddings for the generator to synthesize boundary supporting samples. To improve the superposition quality, the authors proposed a disentanglement mapping layer and extracted embedding initialization from the original fully connected layer to \"flatten\" the latent space. Experimental results show that Qimera outperforms existing work on data-free quantization.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. Firstly, the paper is generally well written and easy to follow. The motivation experiments clearly show the necessity for boundary supporting samples.\n2. The proposed techniques to flatten the latent space further improve the performance of latent space superposition.\n3. Experimental results are solid. The ablation study on applying mixup to the baseline methods is very informative. It shows that interpolating in the latent space is a more effective regularization compared to interpolating in the image space.\n\nComments:\n1. I am not very clear why the disentanglement mapping alone can help make the latent space more \"flatten\". Is it similar to the style mapping in the StyleGAN family? If so, does it help to use an MLP for the mapping?\n2. Regarding the data leaking issue, in the Dreaming to Distill paper [25], the synthesized images are actually quite visually interpretable. Could you please also show the images on the ImageNet dataset? What is the difference that makes the synthesized images less interpretable in this work?\n3. Regarding \"use noise with higher variance\" (L148), is it possible to just use KD loss for the generated samples with large noises and get rid of the incorrect label issue?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses post-training data-free quantization to compress neural networks, an important domain of NN research. The reviewers found the approach, generating samples around the NN's decision boundaries (boundary supporting samples), and associated methods well motivated (though there were some clarification requests in the reviews, which the authors fulfilled). There were some concerns about similarities to prior work, both in motivations and methods: the authors agreed to include these and I encourage the authors to be generous in giving motivational credit to these works. The authors also provide additional experimental results in the rebuttal, which further demonstrated the effectiveness in the method. Therefore I recommend acceptance as a poster.
Decision: Accept (Poster)