Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposed three different techniques to improve the quality of the post-training quantization (PT) results. The main claim is about Retro-Synthesis Data, which allows the calibration of the quantization parameters without the training data. There are two additional techniques, Hybrid Quantization and Non-Uniform Quantization, to increase the accuracy of PT. 

Although the overall paper is nicely written, the contents might need more novelty for being accepted in ICLR. Here are a few concerns related to it.

- The "Retro-Synthesis Data" scheme sounds interesting, but not sure how much practical impact it might bring in. The authors assumed the model deployment scenario where the full-precision model is given without any training data. But it does not necessarily mean that the unlabeled data is also not available; in most cases, test data (without labels) from the deployed application is available for tuning of parameters (E.g., Sec 4 of [Sun et al., NeurIPS19]). In such a case, it is not clear how much benefit we can expect from "synthesizing" the test data for tuning quantization parameters.

- Even if we believe that synthesizing the test data is necessary, it is not clear why "Retro-Synthesis Data" is well suited for the post-training quantization. Although the approach is interesting, how much representative the data created by the proposed method is for being used for the parameter tuning? It would be highly desirable to explain in the derivation of "Retro-Synthesis Data" how it can provide information particularly useful for better post-training quantization.

- The strong empirical evidence might say more than just a detailed explanation. But unfortunately, the performance gain in the experimental results seems to be marginal. The authors claim that the proposed method is good for the models without batchnorm, but it does not sound very convincing given the fact that batchnorm is extremely popular these days...

- The additional techniques, Hybrid Quantization and Non-Uniform Quantization, seem to be incremental from the previous work about quantization-sensitivity and outlier aware controls of neural networks (such as precision selection and channel splitting). Also, the authors claimed that there is 10~20% performance gain from Hybrid Quantization, but without much justification on how such hardware performance is measured/estimated.

To sum, this is a well-written paper but it might need more novelty for being accepted in ICLR. 


review_title: A nicely written paper, but it might need more novelty

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper considers the problem of data-free post-training quantization of classfication networks. It proposes three extensions of an existing framework ZeroQ (Cai et al., 2020): (1). in order to generate distilled data for network sensitivity analysis, the "Retro Synthesis" method is proposed to turn a random image into a one that represents a desired class label without relying on batch norm statistics like in ZeroQ; (2). a hybrid quantization strategy is proposed to optionally provide finer-grained per-channel quantization instead of the typical per-layer quantization; (3). a non-uniform quantization grid is proposed to better represent quantized weights, instead of uniform quantization as in ZeroQ.  Empirical evaluation demonstrate the effectiveness of the proposed approach.

==========================================================

Pros:
1. The proposed "Retro Synthesis" approach broadens the scope of ZeroQ to a wider range of neural network architectures by lifting the requirements of having batch normalization layers, and thus may significantly extend the practical applicability of data-free post-training quantization.
2. The proposed approach seems to run very fast (but on the same order of magnitude as ZeroQ).


==========================================================

Cons:
1. The significance of the contribution appears limited in scope, as the three proposed methods (especially the latter two, "hybrid quantization" and non-uniform quantization) read more like incremental modifications to components of the existing ZeroQ framework. As far as I understood, the overall approach of this paper follows the ZeroQ framework, e.g., uses the generated distilled data in the same way, also computes the effect of quantization based on KL divergence, and allocates per-layer/channel bit precision using the same Pareto frontier approach. The proposed "Retro Synthesis" is the main innovation, but is rather poorly explained (what's the motivation for having three components L_BN, L_G, L_C, to the total loss, and are they really weighted equally in the experiments?) and lacks insight into why it works.  And it is questionable whether the remaining two proposals count as contributions: "hybrid quantization" is a simple heuristic for deciding when to use per-channel vs per-layer quantization, and non-uniform quantization is a slightly more sophisticated quantization grid (simply divides weights into quartiles and does uniform quantization in each quartile). I have no issue with a simple method if it's well motivated and works well, but:
2. Moreover, the experiments are plagued with a lack of careful analysis of the proposed methods and details for reproducability. Since the three proposed improvements focus on different aspects of ZeroQ, ablations that analyze the effect of each in isolation  (i.e., baseline ZeroQ + proposed hybrid quantization, baseline ZeroQ + proposed non-uniform quantization) are crucial for evaluating their contributions. Details such as how the pre-trained models were obtained (were they trained from scratch, or pre-trained and obtained from the original ZeroQ authors) are not provided, which can significantly impact the empirical results and how they can be understood.  The issue is not specific to this paper -- the field of neural network compression can benefit from better reproducability and more robust evaluation, and particularly methods that operate post-training can evaluate on a shared repository of pre-trained models and report accuracy loss.

==========================================================

Comments & Questions:
1. Since the paper argues that "it is possible to generate image data with similar statistics" based on class scores (under the proposed "Retro Synthesis" approach), it makes sense to validate this is indeed the case qualitatively by comparing against the alternative "ground truth" approach based on matching batch norm statistics, e.g. by comparing the resulting model layer sensivity on distilled data produced by these two approaches, like in Figure 2 of the ZeroQ paper.
2. What's the Gaussian loss (LG) component of the main loss function (eq (1))? Similarly, step (e) of algorithm 1 is unclear -- what is \mu_0' and \sigma_0'?
3. Maybe I'm missing something, but wouldn't something obvious like k-means perform just as well (if not better) compared to the proposed non-uniform quantization approach based on uniformly quantizing within quartiles?

review_title: OK ideas that come across as incremental; good numbers but questionable science.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work uses post-training quantization without access to training data for privacy concerns. Instead, useful statistics are estimated using a retro-synthesis data obtained from the FP baseline. I have a few comments, some concerns and some suggestions I think can be used to improve this work.

Overview of prior work:
Most competing approaches are included and representative prior arts are mentioned which is good. In the descussion about post-training quantization methods, the authors conclude that prior arts observe accuracy degradations. There have been works on post-training quantization that theoretically predict such degradation and overcome it by increasing precision as needed using the concept of noise gains [1]. I think the authors should contrast such work with theirs. It is my impression that noise gains in [1] can be used to improve the presented method.

Comments about the method:
Retro-synthesis data generation:
The retro-synthesis data is used to determine activation ranges which are useful for quantization. This is a clever method. I wonder if that is really necessary though. Can't we follow an analysis similar to [2] in order to predict activation statistics from weight statistics (which are available)?

Hybrid quantization:
It seems the concept of per-tensor quantization has already been studied in [3] and also use the concept of noise gains as above to analytically determine the required number of bits for each tensor. I think this method can be useful to improve/validate the proposed hybrid quantization technique.

Non-uniform quantization:
I have two issues: (1) why not simply use the Lloyd-Max algorithm (which optimizes non-uniform quantization), (2) it is unclear how activations are quantized.

The experimental results look good.

[1] Sakr, C., Kim, Y., & Shanbhag, N. Analytical guarantees on numerical precision of deep neural networks. In International Conference on Machine Learning, ICML 2017.

[2] He, K., Zhang, X., Ren, S., & Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, CVPR 2015.

[3] Sakr, C., & Shanbhag, N. Per-tensor fixed-point quantization of the back-propagation algorithm. In 7th International Conference on Learning Representations, ICLR 2019.

Post Rebuttal Comments:

I thank the authors for their feedback. I have no modification to make to my original review.
review_title: Post-training quantization without dataset access to preserve privacy

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a data-independent post-training quantization scheme by generating a faux dataset without depending on the BN layer statistics of the FP32 model. The authors also introduce two variants of post-training quantization methods, hybrid quantization and non-uniform quantization methods. 

Strengths of the paper: 
- The proposed quantization method is practically applicable to privacy-constraint applications because no training data are required and the method works for any model. 
- The proposed Hybrid Quantization scheme addresses a couple of benefits, one is for improving accuracy and the other is for faster inference time. Both of them are important metrics for practical applicability.
- The paper is well written and easy to follow.

Weaknesses of the paper: 
- Novelty: The novelty is moderate but not strong enough. Specifically, (1) a zero-shot quantization framework to generate a synthetic dataset has been studied in the previous literature ZeroQ (using the distilled data engineered to match the statistics of BN layers to perform post-training quantization). But this paper employs an alternative faux dataset, called by the Retro-Synthesis Data, that does not depend on the BN layer statistics of the original data, which should be appreciated. Moreover, most of DNN models employ the BN layer in their construction, what is the merit in that case? (2) The same layer sensitivity metric as ZeroQ, the KLD between the original model and the quantized model has been used in the proposed Hybrid Quantization scheme. The main idea behind Hybrid Quantization using the KLD to determine whether a layer is suitable to be quantized using the per-tensor or per-channel scheme, seems to be same. Overall, the contribution is a bit incremental and seems not novel to me compared with ZeroQ and DFQ. 
- Evaluation: Further experiments and more ablation studies should be done. The provided experimental results are too weak to support the strength of using the retro-synthesis data.

Detailed comments: 

(1) Algorithm 1 to generate retro-synthesis data seems to have a dependency on the target class(C). If there are many classes to be classified, is the time complexity also increasing? The paper states that generating the retro-synthesis data takes 10~12 sec. Is this for CIFAR-10 or ImageNet dataset? The proposed scheme seems to be more time-consuming than ZeroQ's 3 sec for generating the distilled data. In the ZeroQ paper, the end-to-end quantization of ResNet50 on ImageNet data takes almost 30 sec, which takes 3 sec to generate the synthetic data, 12 sec to estimate the sensitivity for all layers, and 14 sec to perform Pareto Frontier optimization etc. Could you provide the detailed timing breakdown on which experimental setting?

(2) Computing the KLD in Algorithm 2 has more timing O/H than ZeroQ since it is executed twice for both per-tensor and per-channel quantization schemes per layer. Then does it take time twice more compared with ZeroQ? Could you provide the detailed timing breakdown or time O/H evaluation on your Hybrid Quantization configuration? Is the time O/H ignorable compared to the benefit of faster inference time by exploiting the hybrid quantization scheme? Approaches to achieve the goal of optimizing the inference time cause a conflict of interest.

(3) A threshold value Th is sub-optimal and determined heuristically to decide either per-channel or per-tensor quantization. Is there any way to find an optimal value? The paper uses Th=0.001, but there is little improvement over Th=0. Could you explain why this happens? Further ablation study on different Th values should be provided. In the 8-bits setting, there seems no significant difference b/w per-channel and per-layer schemes due to its perturbation effect. More explanation on the result is required.

(4) Results (in Table 3) of comparing the proposed method to ZeroQ with W8A8, W6A8, and W4A8 on ResNet model with CIFAR-10 are too weak to support the strength of using the retro-synthesis data. There seems to be a bit improvement over ZeroQ while the proposed scheme takes more time(10~12 sec for the retro-synthesis data) to generate a synthetic dataset compared with ZeroQ(3 sec for the distilled data).  

(5) Results on ImageNet data (in Table 4 and 5) have a weak contribution if the bitwidth used in the experiment is 8 bit(You should describe a precision setting in the result tables). More aggressive setting of bitwidths, say 6 or 4 bits, should be provided comparing with the SOTA post-training quantization schemes. 

(6) The paper also presents another variant of using the retro-synthesis data, a non-uniform quantization scheme. Further evaluation (in Table 5) should be done to support its superiority by comparing it with the SOTA non-uniform quantization schemes, e.g. PoT, APoT, DDQ, etc. Especially, the non-uniform quantization gets more important in the weaker representation range, say low-precision quantization schemes below 8 bits. Future more experiments should be provided. 

review_title: Official Blind Review #4

Meta-review: Four knowledgeable referees reviewed this paper; one reviewer (weakly) supports accept and other three indicate reject. Even with the rebuttal, all negative reviewers have concerns on the limited novelty and marginal performance improvement, and agree that the paper is not well qualified for the high standard of ICLR.
Decision: Reject