Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In the manuscript, the authors propose Qimera, which introduces superposed latent embeddings, disentanglement mapping, and extracted embedding initialization to generate boundary supporting samples.
These synthetic samples are useful for post-training data-free quantization.
The main contributions of the manuscript are:
1) Pioneer at considering boundary supporting samples for data-free quantization,
2) Propose Qimera and its components,
3) Achieve decent experimental results, together with meaningful analysis.

main_review: I think the paper is generally interesting, and I may change my rate according to the rebuttal.
Strength:
1) The writing of this manuscript is good, with sufficient background knowledge, clear equations and explanations. Although the Qimera method has lots of components, the manuscript is still easy to follow.
2) The related work section is thorough.
3) Illustrations such as Figure 3 are helpful to validate the effectiveness of proposed methods in generating boundary supporting samples.
4) The experimental results are quite good comparing with previous methods targeting using synthetic data on post-training quantization (PTQ).

Suggestion:
The supplementary materials contain lots of additional experiments and ablation study, which are helpful to address a large portion of my concerns (for example, the sensitivity to other hyperparameters, comparing with more recent methods, etc). I suggest merging some of them into the main texts if possible.

Weakness and Question:
1) Regarding the disentanglement mapping, M is implicitly encouraged to map the inputs to a flatter space because we suppose unnatural samples can lead to a large loss. Some ablation study or statistics are required to validate the point.
2) Qimera is good at generating synthetic images for PTQ. But will a better decision boundary be learned when the network has better effective capability (when applied with advanced quantization methods such as better quantizer or mixed-precision quantization)? Can Qimera improve on top of SOTA quantization methods? (ZAQ + Qimera experiments in supplementary material is a start, but ZAQ is not specifically targeting quantizers/precisions, and the ZAQ + Qimera results are not super promising).
3) Why the distributions of the embeddings should be similar to the feature space in order to be flat?
4) In addition to PTQ, it would be interesting to see if Qimera can help quantization-aware training (QAT) with synthetic data?
5) From the ablation, solely applying SE can have a good improvement, but applying mixup on inputs doesn't work well. It would be good to see some experimental results on the source of contributions. Is embedding the only key factor? 

=========Post Rebuttal=========

Thanks the authors for the detailed rebuttal, which addressed many of my concerns. Based on the rebuttal and reviews from other reviewers, I will keep my rating and would like the paper to be accepted to NeurIPS 2021.
limitations_and_societal_impact: The potential negative societal impact of this work is regarding data privacy. 
Data-free quantization aims to quantize pretrained neural networks without access to the data they are trained on.
This generally requires an approximation of the distribution of training data derived/distilled from the neural networks.
The manuscript explicitly discuss this impact and show many results sampled in the approximated distribution of training data.
Since most samples are not interpretable by humans, the proposed method is good to use (suppose that the texture of training data is not private and interpretable).
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper addresses a popular data-free post-training quantization problem using a faux dataset of synthetically generated samples without depending on the BN layer statistics of the FP32 model. It proposes a superposed latent embeddings technique generating synthetic boundary samples to capture the distribution of the original data around the decision boundaries. The authors also introduce two auxiliary techniques to achieve additional performance gain.

main_review: This paper is well-written, and its goal is well-oriented by providing a good motivational experiment. The proposed Qimera approach provides a valuable insight that the synthetic samples of conventional data-free compression methods lack samples near the decision boundary of the FP32 model. The paper is well-organized and provides a unifying view of several generative data-free quantization techniques pioneered by ZeroQ.

The novelty is moderate but not strong enough because of the following reasons: Specifically, (1) a generative data-free quantization framework to generate a synthetic dataset has been studied in recent literature, say ZeroQ(CVPR'2020), GDFQ(ECCV'2020), ZAQ(CVPR'2021), and DSG(CVPR'2021). The proposed work seems to extend more like incremental modifications to the existing methods by combining already known concepts and ideas. Their philosophy and overall approaches are quite similar by following a generative data-free quantization framework originated from ZeroQ. Moreover, (2) the significance of its contribution appears limited in scope, as the boundary supporting samples have already been successful for knowledge distillation, especially in data-free network quantization with adversarial knowledge distillation(CVPR'2020) and knowledge distillation with adversarial samples supporting decision boundary(AAAI'2019). Therefore, the authors employed the main idea of generating boundary supporting samples to apply data-free quantization instead of knowledge distillation. 

Experimental results look good. Still, its application is quite restricted to the classification task because of the inherent drawback of superposed latent embeddings to create samples of multiple classes, which is inferior to the other approaches like ZAQ and DSG extending the application to image segmentation and object detection beyond image classification. 

Here are a few concerns and questions:
1. Please include the recently published DSG(Diversifying Sample Generation for Accurate Data-Free Quantization, CVPR'2021) paper as a reference. Comparison between the DSG and the proposed Qimera might be required. An additional evaluation comparing with DSG would be preferred. 
2. Both ZAQ and DSG have improved the previous GDFQ without using BN statistics. Adding their PCA plots would be better to understand Qimera's superiority over them. 
3. Is the amount of synthetic data generated by each benchmarking method all equal in the experiments conducted in this paper? It would be better to specify the amount of synthetic data used for fine-tuning in all methods.
4. For experiments with Cutmix and Mixup applied to GDFQ in Table 2, is it reasonable to apply these augmentation techniques to data focused on local textures as mentioned in the Discussion? Such augmentation techniques are thought to be efficient when global shapes are distinct. Therefore, it is inappropriate to conclude the performance degradation by applying Cutmix and Mixup to these synthetic data with a lack of boundary supporting samples. If appropriate, it would be better to visualize it or organize a separate experiment to prove it.
5. This paper experimentally demonstrates that synthetic samples generated near the decision boundary help the data-free quantized model's performance improve. The ablation study in Table 3 shows the performance changes of the model by Qimera's SE, DM, and EEI. The conceptual goal of SE seems to be on the opposite side of DM and EEI. Their combination would be quite strange because disentangling generally used in GAN is the opposite of superposing. The presented experimental results show the highest performance improvement is mainly due to the superposed latent embeddings. Does the ablation study setting still use the same amount of synthetic samples for SE, SE+DM, SE+EEI, and SE+DM+EEI? It would be good to provide valuable insights by carefully analyzing their performance difference because creating more synthetic samples could improve. Is there any other way to theoretically explain how the two auxiliary techniques improve using SE only? Or, if you visualize how DM and EEI can increase data near the decision boundary, you could see how boundary supporting samples improve the model's performance.


=================== post-rebuttal comment ===================

I thank the authors for their detailed response. The newly added experimental results address most of my concerns. Accordingly, I am happy to raise my rating. 

===========================================================
limitations_and_societal_impact: The authors adequately addressed the limitations and no sign of privacy invasion.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors proposed Qimera to improve data-free quantization by generating samples around the decision boundary. Qimera uses superposed latent embeddings for the generator to synthesize boundary supporting samples. To improve the superposition quality, the authors proposed a disentanglement mapping layer and extracted embedding initialization from the original fully connected layer to "flatten" the latent space. Experimental results show that Qimera outperforms existing work on data-free quantization.
main_review: Pros:
1. Firstly, the paper is generally well written and easy to follow. The motivation experiments clearly show the necessity for boundary supporting samples.
2. The proposed techniques to flatten the latent space further improve the performance of latent space superposition.
3. Experimental results are solid. The ablation study on applying mixup to the baseline methods is very informative. It shows that interpolating in the latent space is a more effective regularization compared to interpolating in the image space.

Comments:
1. I am not very clear why the disentanglement mapping alone can help make the latent space more "flatten". Is it similar to the style mapping in the StyleGAN family? If so, does it help to use an MLP for the mapping?
2. Regarding the data leaking issue, in the Dreaming to Distill paper [25], the synthesized images are actually quite visually interpretable. Could you please also show the images on the ImageNet dataset? What is the difference that makes the synthesized images less interpretable in this work?
3. Regarding "use noise with higher variance" (L148), is it possible to just use KD loss for the generated samples with large noises and get rid of the incorrect label issue?

limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper addresses post-training data-free quantization to compress neural networks, an important domain of NN research. The reviewers found the approach, generating samples around the NN's decision boundaries (boundary supporting samples), and associated methods well motivated (though there were some clarification requests in the reviews, which the authors fulfilled). There were some concerns about similarities to prior work, both in motivations and methods: the authors agreed to include these and I encourage the authors to be generous in giving motivational credit to these works. The authors also provide additional experimental results in the rebuttal, which further demonstrated the effectiveness in the method. Therefore I recommend acceptance as a poster.
Decision: Accept (Poster)