Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples

1 Introduction . Among many neural network compression methodologies , quantization is considered a promising direction because it can be easily supported by accelerator hardwares [ 1 ] than pruning [ 2 ] and is more lightweight than knowledge distillation [ 3 ] . However , quantization generally requires some form of adjustment ( e.g. , fine-tuning ) using the original training data [ 4 , 5 , 6 , 7 , 8 ] to restore the accuracy drop due to the quantization errors . Unfortunately , access to the original training data is not always possible , especially for deployment in the field , for many reasons such as privacy and security . For example , the data could be medical images of patients , photos of confidential products , or pictures of military assets . Therefore , data-free quantization is a natural direction to achieve a highly accurate quantized model without accessing any training data . Among many excellent prior studies [ 9 , 10 , 11 , 12 ] , generative methods [ 13 , 14 , 15 ] have recently been drawing much attention due to their superior performance . Generative methods successfully generate synthetic samples that resemble the distribution of the original dataset and achieve high accuracy using information from the pretrained full-precision network , such as batch-normalization statistics [ 15 , 13 ] or intermediate features [ 14 ] . ∗Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , a significant gap still exists between data-free quantized models and quantized models finetuned with original data . What is missing from the current generative data-free quantization schemes ? We hypothesize that the synthetic samples of conventional methods lack boundary supporting samples [ 16 ] , which lie on or near the decision boundary of the full-precision model and directly affect the model performance . The generator designs are often based on conditional generative adversarial networks ( CGANs ) [ 17 , 18 ] that take class embeddings representing class-specific latent features . Based on these embeddings as the centroid of each class distribution , generators rely on the input of random Gaussian noise vectors to gain diverse samples . However , one can easily deduce that random noises have difficulty reflecting the complex class boundaries . In addition , the weights and embeddings of the generators are trained with cross-entropy ( CE ) loss , further ensuring that these samples are well-separated from each other . In this work , we propose Qimera , a method for data-free quantization employing superposed latent embeddings to create boundary supporting samples . First , we conduct a motivational experiment to confirm our hypothesis that samples near the boundary can improve the quantized model performance . Then , we propose a novel method based on inputting superposed latent embeddings into the generator to produce synthetic boundary supporting samples . In addition , we provide two auxiliary schemes for flattening the latent embedding space so that superposed embeddings could contain adequate features . Qimera achieves significant performance improvement over the existing techniques . The experimental results indicate that Qimera sets new state-of-the-art performance for various datasets and model settings for the data-free quantization problem . Our contributions are summarized as the following : • We identify that boundary supporting samples form an important missing piece of the current state-of-the-art data-free compression . • We propose using superposed latent embeddings , which enables a generator to synthesize boundary supporting samples of the full-precision model . • We propose disentanglement mapping and extracted embedding initialization that help train a better set of embeddings for the generator . • We conduct an extensive set of experiments , showing that the proposed scheme outperforms the existing methods . 2 Related Work . 2.1 Data-free Compression . Early work on data-free compression has been led by knowledge distillation [ 3 ] , which usually involves pseudo-data created from teacher network statistics [ 19 , 20 ] . Lopes et al . [ 19 ] suggested generating pseudo-data from metadata collected from the teacher in the form of activation records . Nayak et al . [ 20 ] proposed a similar scheme but with a zero-shot approach by modeling the output space of the teacher model as a Dirichlet distribution , which is taken from model weights . More recent studies have employed generator architectures similar to GAN [ 21 ] to generate synthetic samples replacing the original data [ 22 , 23 , 24 , 25 ] . In the absence of the original data for training , DAFL [ 22 ] used the teacher model to replace the discriminator by encouraging the outputs to be close to a one-hot distribution and by maximizing the activation counts . KegNet [ 26 ] adopted a similar idea and used a low-rank decomposition to aid the compression . Adversarial belief matching [ 23 ] and data-free adversarial distillation [ 24 ] methods suggested adversarially training the generator , such that the generated samples become harder to train . One other variant is to modify samples directly using logit maximization as in DeepInversion [ 25 ] . While this approach can generate images that appear natural to a human , it has the drawback of skyrocketing computational costs because each image must be modified using backpropagation . Data-free quantization is similar to data-free knowledge distillation but is a more complex problem because quantization errors must be recovered . The quantized model has the same architecture as the full-precision model ; thus , the early methods of post-training quantization were focused on how to convert the full-precision weights into quantized weights by limiting the range of activations [ 9 , 12 ] , correcting biases [ 9 , 10 ] , and equalizing the weights [ 10 , 12 ] . ZeroQ [ 15 ] pioneered using synthetic data for data-free quantization employing statistics stored in batch-normalization layers . GDFQ [ 13 ] added a generator close to the ACGAN [ 18 ] to generate better synthetic samples , and AutoReCon [ 27 ] suggests a better performing generator found by neural architecture search . In addition , DSG [ 28 ] further suggests relaxing the batch-normalization statistics alignment to generate more diverse samples . ZAQ [ 14 ] adopted adversarial training of the generator on the quantization problem and introduced intermediate feature matching between the full-precision and quantized model . However , none of these considered aiming to synthesize boundary supporting samples of the full-precision model . Adversarial training of generators have a similar purpose , but it is fundamentally different from generating boundary supporting samples . In addition , adversarial training can be sensitive to hyperparameters and risks generating samples outside of the original distributions [ 29 ] . To the extent of our knowledge , this work is the first to propose using superposed latent embeddings to generate boundary supporting samples explicitly for data-free quantizations . 2.2 Boundary Supporting Samples . In the context of knowledge distillation , boundary supporting samples [ 16 ] are defined as samples that lie near the decision boundary of the teacher models . As these samples contain classification information about the teacher models , they can help the student model correctly mimic the teacher ’ s behavior . Heo et al . [ 16 ] applied an adversarial attack [ 30 ] to generate boundary supporting samples and successfully demonstrated that they improve knowledge distillation performance . In AMKD [ 31 ] , triplet loss was used to aid the student in drawing a better boundary . Later , DeepDig [ 32 ] devised a refined method for generating boundary supporting samples and analyzed their characteristics by defining new metrics . Although boundary supporting samples have been successful for many problems , such as domain adaptation [ 33 ] and open-set recognition [ 34 ] , they have not yet been considered for data-free compression . 3 Motivational Experiment . To explain the discrepancy between the accuracy of the model fine-tuned with the original training data and the data-free quantized models , we hypothesized that the synthetic samples from generative data-free quantization methods lack samples near the decision boundary . To validate this hypothesis , we designed an experiment using the CIFAR-100 [ 35 ] dataset with the ResNet-20 network [ 36 ] . First , we forwarded images in the dataset into the pre-trained full-precision ResNet-20 . Among these , we selected 1500 samples ( 3 % of the dataset , 15 samples per class ) from samples where the highest confidence value was lower than 0.25 , forming a group of ‘ confusing ’ samples . Then , we combined the confusing samples with synthetic samples generated from a previous study [ 13 ] to fine-tune the quantized model with 4-bit weights and 4-bit activations . We also selected as a control group an equal number of random samples from the images classified as unconfusing and fine-tuned the quantized model using the same method . The results are presented in Table 1 . The quantized model with synthetic + unconfusing real samples exhibited only 0.52 % p increase in the accuracy from the baseline . In contrast , adding confusing samples provided 2.36 % p improvement , filling almost half the gap towards the quantized model fine-tuned with the original data . These results indirectly validate that the synthetic samples suffer from a lack of confusing samples ( i.e. , boundary supporting samples ) . We aim to address the issue in this paper . As we indicate in Section 5 , Qimera achieves a 1.71 % p performance gain for the same model in a data-free setting , close to that of the addition of confusing synthetic samples . 4 Generating Boundary Supporting Samples with Superposed Latent Embeddings . 4.1 Baseline Generative Data-free Quantization . Recent generative data-free quantization schemes [ 13 , 14 ] employ a GAN-like generator to create synthetic samples . In the absence of the original training samples , the generator G attempts to generate synthetic samples so that the quantized model Q can mimic the behavior of the full-precision model P . For example , in GDFQ [ 13 ] , the loss function LGDFQ is LGDFQ ( G ) = LPCE ( G ) + αLPBNS ( G ) , ( 1 ) where the first term LCE guides the generator to output clearly classifiable samples , and the second term LBNS aligns the batch-normalization statistics of the synthetic samples with those of the batch-normalization layers in the full-precision model . In another previous work ZAQ [ 14 ] , LZAQ ( G ) = LP , Qo ( G ) + βL P , Q f ( G ) + γL P a ( G ) , ( 2 ) where the first term Lo separates the prediction outputs of P and Q , and the second term Lf separates the feature maps produced by P and Q . These two losses let the generator be adversarially trained and allow it to determine samples where P can not mimic Q adequately . Lastly , the third term La maximizes the activation map values of P so that the created samples do not drift too far away from the original dataset . The quantized model Q is usually jointly trained with G , such that LGDFQ ( Q ) = LQCE ( Q ) + δL P KD ( Q ) , ( 3 ) for GDFQ , and LZAQ ( Q ) = −LP , Qo ( G ) − βL P , Q f ( G ) , ( 4 ) for ZAQ , respectively , where LKD from Eq . 3 is the usual KD loss with Kullback-Leibler divergence . While these two methods exhibit great performance , they both model the distribution of per-class embeddings in the latent input space as a Gaussian distribution , and generate diverse samples using random Gaussian noise inputs . However , based on the Gaussian distribution , it is difficult to correctly reflect the boundary between the two different classes , especially when all samples have one-hot labels . Figure 1 visualizes such problems in a simplified two-dimensional space . With samples generated from gaussian noise ( Figure 1a ) , the two per-class distributions are far away , and a void exists between the two classes ( see Figure 3b for plots from experimental data ) . This can cause the decision boundaries to be formed at nonideal regions . One alternative is to use noise with higher variance , but the samples would overlap on a large region , resulting in too many samples with incorrect labels . Furthermore , in the above approaches , the loss terms of Eqs . 1 and 2 such as LCE and Lo encourage the class distributions to be separated from each other . While this is beneficial for generating clean , well-classified samples , it also prevents generating samples near the decision boundary , which is necessary for training a good model [ 16 , 31 ] . Therefore , in this paper , we focus on methods to generate synthetic samples near the boundary from the full-precision model P ( Figure 1b ) .

Summary: This paper addresses a popular data-free post-training quantization problem using a faux dataset of synthetically generated samples without depending on the BN layer statistics of the FP32 model. It proposes a superposed latent embeddings technique generating synthetic boundary samples to capture the distribution of the original data around the decision boundaries. The authors also introduce two auxiliary techniques to achieve additional performance gain.