Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples

1 Introduction . Among many neural network compression methodologies , quantization is considered a promising direction because it can be easily supported by accelerator hardwares [ 1 ] than pruning [ 2 ] and is more lightweight than knowledge distillation [ 3 ] . However , quantization generally requires some form of adjustment ( e.g. , fine-tuning ) using the original training data [ 4 , 5 , 6 , 7 , 8 ] to restore the accuracy drop due to the quantization errors . Unfortunately , access to the original training data is not always possible , especially for deployment in the field , for many reasons such as privacy and security . For example , the data could be medical images of patients , photos of confidential products , or pictures of military assets . Therefore , data-free quantization is a natural direction to achieve a highly accurate quantized model without accessing any training data . Among many excellent prior studies [ 9 , 10 , 11 , 12 ] , generative methods [ 13 , 14 , 15 ] have recently been drawing much attention due to their superior performance . Generative methods successfully generate synthetic samples that resemble the distribution of the original dataset and achieve high accuracy using information from the pretrained full-precision network , such as batch-normalization statistics [ 15 , 13 ] or intermediate features [ 14 ] . ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , a significant gap still exists between data-free quantized models and quantized models finetuned with original data . What is missing from the current generative data-free quantization schemes ? We hypothesize that the synthetic samples of conventional methods lack boundary supporting samples [ 16 ] , which lie on or near the decision boundary of the full-precision model and directly affect the model performance . The generator designs are often based on conditional generative adversarial networks ( CGANs ) [ 17 , 18 ] that take class embeddings representing class-specific latent features . Based on these embeddings as the centroid of each class distribution , generators rely on the input of random Gaussian noise vectors to gain diverse samples . However , one can easily deduce that random noises have difficulty reflecting the complex class boundaries . In addition , the weights and embeddings of the generators are trained with cross-entropy ( CE ) loss , further ensuring that these samples are well-separated from each other . In this work , we propose Qimera , a method for data-free quantization employing superposed latent embeddings to create boundary supporting samples . First , we conduct a motivational experiment to confirm our hypothesis that samples near the boundary can improve the quantized model performance . Then , we propose a novel method based on inputting superposed latent embeddings into the generator to produce synthetic boundary supporting samples . In addition , we provide two auxiliary schemes for flattening the latent embedding space so that superposed embeddings could contain adequate features . Qimera achieves significant performance improvement over the existing techniques . The experimental results indicate that Qimera sets new state-of-the-art performance for various datasets and model settings for the data-free quantization problem . Our contributions are summarized as the following : • We identify that boundary supporting samples form an important missing piece of the current state-of-the-art data-free compression . • We propose using superposed latent embeddings , which enables a generator to synthesize boundary supporting samples of the full-precision model . • We propose disentanglement mapping and extracted embedding initialization that help train a better set of embeddings for the generator . • We conduct an extensive set of experiments , showing that the proposed scheme outperforms the existing methods . 2 Related Work . 2.1 Data-free Compression . Early work on data-free compression has been led by knowledge distillation [ 3 ] , which usually involves pseudo-data created from teacher network statistics [ 19 , 20 ] . Lopes et al . [ 19 ] suggested generating pseudo-data from metadata collected from the teacher in the form of activation records . Nayak et al . [ 20 ] proposed a similar scheme but with a zero-shot approach by modeling the output space of the teacher model as a Dirichlet distribution , which is taken from model weights . More recent studies have employed generator architectures similar to GAN [ 21 ] to generate synthetic samples replacing the original data [ 22 , 23 , 24 , 25 ] . In the absence of the original data for training , DAFL [ 22 ] used the teacher model to replace the discriminator by encouraging the outputs to be close to a one-hot distribution and by maximizing the activation counts . KegNet [ 26 ] adopted a similar idea and used a low-rank decomposition to aid the compression . Adversarial belief matching [ 23 ] and data-free adversarial distillation [ 24 ] methods suggested adversarially training the generator , such that the generated samples become harder to train . One other variant is to modify samples directly using logit maximization as in DeepInversion [ 25 ] . While this approach can generate images that appear natural to a human , it has the drawback of skyrocketing computational costs because each image must be modified using backpropagation . Data-free quantization is similar to data-free knowledge distillation but is a more complex problem because quantization errors must be recovered . The quantized model has the same architecture as the full-precision model ; thus , the early methods of post-training quantization were focused on how to convert the full-precision weights into quantized weights by limiting the range of activations [ 9 , 12 ] , correcting biases [ 9 , 10 ] , and equalizing the weights [ 10 , 12 ] . ZeroQ [ 15 ] pioneered using synthetic data for data-free quantization employing statistics stored in batch-normalization layers . GDFQ [ 13 ] added a generator close to the ACGAN [ 18 ] to generate better synthetic samples , and AutoReCon [ 27 ] suggests a better performing generator found by neural architecture search . In addition , DSG [ 28 ] further suggests relaxing the batch-normalization statistics alignment to generate more diverse samples . ZAQ [ 14 ] adopted adversarial training of the generator on the quantization problem and introduced intermediate feature matching between the full-precision and quantized model . However , none of these considered aiming to synthesize boundary supporting samples of the full-precision model . Adversarial training of generators have a similar purpose , but it is fundamentally different from generating boundary supporting samples . In addition , adversarial training can be sensitive to hyperparameters and risks generating samples outside of the original distributions [ 29 ] . To the extent of our knowledge , this work is the first to propose using superposed latent embeddings to generate boundary supporting samples explicitly for data-free quantizations . 2.2 Boundary Supporting Samples . In the context of knowledge distillation , boundary supporting samples [ 16 ] are defined as samples that lie near the decision boundary of the teacher models . As these samples contain classification information about the teacher models , they can help the student model correctly mimic the teacher ’ s behavior . Heo et al . [ 16 ] applied an adversarial attack [ 30 ] to generate boundary supporting samples and successfully demonstrated that they improve knowledge distillation performance . In AMKD [ 31 ] , triplet loss was used to aid the student in drawing a better boundary . Later , DeepDig [ 32 ] devised a refined method for generating boundary supporting samples and analyzed their characteristics by defining new metrics . Although boundary supporting samples have been successful for many problems , such as domain adaptation [ 33 ] and open-set recognition [ 34 ] , they have not yet been considered for data-free compression . 3 Motivational Experiment . To explain the discrepancy between the accuracy of the model fine-tuned with the original training data and the data-free quantized models , we hypothesized that the synthetic samples from generative data-free quantization methods lack samples near the decision boundary . To validate this hypothesis , we designed an experiment using the CIFAR-100 [ 35 ] dataset with the ResNet-20 network [ 36 ] . First , we forwarded images in the dataset into the pre-trained full-precision ResNet-20 . Among these , we selected 1500 samples ( 3 % of the dataset , 15 samples per class ) from samples where the highest confidence value was lower than 0.25 , forming a group of ‘ confusing ’ samples . Then , we combined the confusing samples with synthetic samples generated from a previous study [ 13 ] to fine-tune the quantized model with 4-bit weights and 4-bit activations . We also selected as a control group an equal number of random samples from the images classified as unconfusing and fine-tuned the quantized model using the same method . The results are presented in Table 1 . The quantized model with synthetic + unconfusing real samples exhibited only 0.52 % p increase in the accuracy from the baseline . In contrast , adding confusing samples provided 2.36 % p improvement , filling almost half the gap towards the quantized model fine-tuned with the original data . These results indirectly validate that the synthetic samples suffer from a lack of confusing samples ( i.e. , boundary supporting samples ) . We aim to address the issue in this paper . As we indicate in Section 5 , Qimera achieves a 1.71 % p performance gain for the same model in a data-free setting , close to that of the addition of confusing synthetic samples . 4 Generating Boundary Supporting Samples with Superposed Latent Embeddings . 4.1 Baseline Generative Data-free Quantization . Recent generative data-free quantization schemes [ 13 , 14 ] employ a GAN-like generator to create synthetic samples . In the absence of the original training samples , the generator G attempts to generate synthetic samples so that the quantized model Q can mimic the behavior of the full-precision model P . For example , in GDFQ [ 13 ] , the loss function LGDFQ is LGDFQ ( G ) = LPCE ( G ) + αLPBNS ( G ) , ( 1 ) where the first term LCE guides the generator to output clearly classifiable samples , and the second term LBNS aligns the batch-normalization statistics of the synthetic samples with those of the batch-normalization layers in the full-precision model . In another previous work ZAQ [ 14 ] , LZAQ ( G ) = LP , Qo ( G ) + βL P , Q f ( G ) + γL P a ( G ) , ( 2 ) where the first term Lo separates the prediction outputs of P and Q , and the second term Lf separates the feature maps produced by P and Q . These two losses let the generator be adversarially trained and allow it to determine samples where P can not mimic Q adequately . Lastly , the third term La maximizes the activation map values of P so that the created samples do not drift too far away from the original dataset . The quantized model Q is usually jointly trained with G , such that LGDFQ ( Q ) = LQCE ( Q ) + δL P KD ( Q ) , ( 3 ) for GDFQ , and LZAQ ( Q ) = −LP , Qo ( G ) − βL P , Q f ( G ) , ( 4 ) for ZAQ , respectively , where LKD from Eq . 3 is the usual KD loss with Kullback-Leibler divergence . While these two methods exhibit great performance , they both model the distribution of per-class embeddings in the latent input space as a Gaussian distribution , and generate diverse samples using random Gaussian noise inputs . However , based on the Gaussian distribution , it is difficult to correctly reflect the boundary between the two different classes , especially when all samples have one-hot labels . Figure 1 visualizes such problems in a simplified two-dimensional space . With samples generated from gaussian noise ( Figure 1a ) , the two per-class distributions are far away , and a void exists between the two classes ( see Figure 3b for plots from experimental data ) . This can cause the decision boundaries to be formed at nonideal regions . One alternative is to use noise with higher variance , but the samples would overlap on a large region , resulting in too many samples with incorrect labels . Furthermore , in the above approaches , the loss terms of Eqs . 1 and 2 such as LCE and Lo encourage the class distributions to be separated from each other . While this is beneficial for generating clean , well-classified samples , it also prevents generating samples near the decision boundary , which is necessary for training a good model [ 16 , 31 ] . Therefore , in this paper , we focus on methods to generate synthetic samples near the boundary from the full-precision model P ( Figure 1b ) .

Summary: In the manuscript, the authors propose Qimera, which introduces superposed latent embeddings, disentanglement mapping, and extracted embedding initialization to generate boundary supporting samples. These synthetic samples are useful for post-training data-free quantization. The main contributions of the manuscript are: 1) Pioneer at considering boundary supporting samples for data-free quantization, 2) Propose Qimera and its components, 3) Achieve decent experimental results, together with meaningful analysis.