Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed a new analytical metric to quantify the impact of quantization for deep neural networks. The proposed metric provides an estimation of quantization impact as accurately as hessian methods yet it is more computationally efficient and its variance is more favorable. The authors compared the proposed method with the other metrics and demonstrate its superior performance.
strength_and_weaknesses: (Strength)
- A nice overview and derivation of the impact of quantization on the loss. 

- The nice properties of the proposed metric are that it is more robust (since it shows a smaller variance) and more efficient (in terms of computation)

- The comparison of correlations that show superior predictability of the proposed methods for the impact of quantization loss.

(Weaknesses)
- The evaluation in Table 2 seems to be very limited. By exploiting the computational efficiency of the proposed algorithm, the authors should also try challenging tasks, such as a large CNN for ImageNet classification or a large language model like BERT.



clarity,_quality,_novelty_and_reproducibility: - Some of the important analysis seems to be missed: 1) derivation of FIT for activation quantization. 2) detail comparison with the Fisher information matrix  (Li et al., 2021)
summary_of_the_review: This paper proposed a practical metric, FIT, for quantitatively analyzing the impact of quantization errors in deep neural networks. FIT's robustness (due to small variance) and computational efficiency would be very useful for the practitioners in this study. However, currently, the evaluation results are very weak; it would be highly appreciated if the authors could include more experimental results, such as ResNet50 on ImageNet.


====== Post rebuttal comments ======
The reviewer thanks the authors for their careful rebuttal. Demonstrating a solid accuracy boost on BERT models seems to make the paper much stronger, and I believe that such a demonstration confirms the applicability of the proposed method for researchers in this field to broader applications. Thus, I increase my score to Accept.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use Fisher Information Trace (FIT) to perform mixed quantization of deep neural networks. The author of the paper use FIT to measure sensitivity of parameters and activations regarding quantization, and shows improvement of the performance.
strength_and_weaknesses: Strengths

1. This paper demonstrate clear advantage of FIT to Hessian-based method.

2. The writing of the paper is good.


Weaknesses

1. There are no ImageNet results of the FIT quantization, which is the major weakness of this paper. The author mentioned ImageNet at the beginning of section 4.1, which uses ImageNet to analyze the proposed method, however, no final accuracy results are give,. Only CIFAR-10 and MNIST.

2. The experimental setting and demonstration is unclear. What is the bit width setting for each of the accuracy result? 

3. Many baselines regarding mixed precision quantization are not compared. For example: “Mix and match: A novel fpga-centric deep neural network quantization framework” (HPCA 2021), “RMSMP: A Novel Deep Neural Network Quantization Framework with Row-wise Mixed Schemes and Multiple Precisions” (CVPR 2021), and more.
  


clarity,_quality,_novelty_and_reproducibility: Please refer to strength and weaknesses.
summary_of_the_review: To sum up, the clarity and quality of this paper need to be improved. The author of the paper did some interesting works on model quantization but fails to demonstrate them with thorough experiments. Please refer to strengths and weaknesses for more information.

I think this paper needs to be revised, both on the technical contribution and experiments.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper finds a more efficient alternative than the Hessian Trace metric used in the mixed-precision HAWQ paper, named FIT, which is a further simplification of said metric based on the empirical fisher. They show the metric to be equivalent in many scenarios compared to the Hessian trace metric. 
strength_and_weaknesses: Strengths:
* It is very clear the authors know what they are talking about, as they mentioned all the relevant literature, and understand the underlying methods/ideas very well. 
* An excellent overview of the literature, and the derivations leading to the final FIM objective. This is a great overview for anyone to read that wants to dive into this topic. 

Weaknesses
* The metric itself has been introduced before, for both compression and quantization. I have seen it several times in the past. The first mention of it I know in the setting of quantization and compression is from this paper: https://arxiv.org/pdf/1810.06401.pdf - where it is derived from an information theoretical context. There is no novelty in the paper, since it is a small tweak to HAWQ, and the tweak itself already existed.
* I think the paper is lacking in terms of experimental results. It is good the authors included the accuracy comparison w.r.t. the target loss for this method and 'competitive' methods. However, these numbers don't mean much for the performance in the mixed-precision setting. For the correlations in Table 2, I wonder if those are high enough to do a proper mixed-precision setting that is non-trivial. For these networks, how well does the mixed precision method rank the networks, and how well can it find a non-trivial solution? Why is this analysis that was done for semantic segmentation also not done for these other models? I don't like hammering on more results, but since the metric itself is not novel, the onus is on the authors to really show that in this setting this metric works well and is the best we can do. The paper turns into a survey paper, rather than a novel method paper. 
* I also wonder about the sense of these metrics in general, and if there is no easier solution. I think this is related to a key underlying assumption at the start of the paper: "The exact (per parameter) perturbations δθ associated with quantization are often unknown."
This is a weird statement. If you 8-bit quantize a layer’s weights, with a given min-max parameter setting, you know exactly what δθ is. There is nothing inherently stochastic about it. Instead of making this stochastic assumption, you could just quantize each layer to the intended potential bit-widths, and measure this on your validation set. Here is your loss function approximation, no surrogate measure needed. This also holds for the activations, where there is arguable a distribution over your data (but not over noise on your model, as the formulation now has it). Please compare what your metric would improve over a simple eval step on a batch of data for per-layer/per-quantizer bit-width settings. In practice, one can generally not set intermediate bit-widths anyway, and generally you'd pick 4, 8, 16 or something, which will likely be almost as efficient as this method.
clarity,_quality,_novelty_and_reproducibility: * Some things are quite unclear on the implementation details. How exactly is the quantization done? Do we fold batch-norm? Do we do asymmetric/symmetric quantization? Per-channel or not? What bit-widths are included for the mixed precision results? I would like to see more details on this in the paper
* The quality of the paper is high, very well polished
* The novelty is not there, as I mentioned in the strenghts/weaknesses above, since literally this metric has been used before in the same context of quantization
* The authors noted they included sample code, so these specific experiments can be reproduced. 
summary_of_the_review: The paper is extremely well-written, and great in terms of clarity and exhaustiveness on the literature regarding Hessian-like estimates for deep learning. 
The method of the paper has been published before, in the context of quantization. The results do convince me that this metric works just as well as the HAWQ metric; They also convince me that they are better than the other metrics the authors found in the literature. However, the authors have not convinced me that these metrics are useful in mixed-precision, that the resulting algorithm actually does something that is non-trivial, and that this is a great way to do mixed precision in general.

I would like to see the following things:
* A proper set of mixed precision results on more than 1 network, not only the semantic segmentation one. The authors included a few more networks for their comparative analysis, but then do not show actual mixed precision results on these nets. This should include showing that the loss approximation is good enough to find non-trivial mixed-precision results
* A comparison to a very simple baseline - Just quantize each layer individually with your intended bit-width, and measure the output loss on a bit of data (few batches e.g.); Your metric is gauranteed to be a less accurate metric for mixed-precision by construction. How well does it perform in practice, and why would your metric be expected to perform better? 
* A more thorough experimental setup description. There's a lot of choices to be made in quantization, and these choices matter. 

If these are addressed in a satisfactory way, I am very willing to increase my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The presented work proposes a method for estimating model performance and mixed precision configuration without network retraining. The introduced method is based on the Fisher information, what allows for faster computations compared to other existing methods based on e.g., Hessians. 
strength_and_weaknesses: The work is clear and well structured. It also flows logically, with a detailed background, good comparison with the previous work, clearly explained FIM metric and experiments that include in-depth comparison with Hessian and analysis of model sensitivity based on the proposed criterion. The method is evaluated on a broad set of models and datasets to evaluate its performance and usability for different tasks. 

In general, I don't see any major omissions. Some sections that could be improved to further improve the quality of submissions are:
1) Could you elaborate more on the ideas of determining MPQ configurations from initialization instead of the trained models?
2) Could you provide more details. about the applied quantization scheme? Symmetric, asymmetric, per tensor, per channel? Have you performed experiments with other quantization schemes?
3) The abstract mentioned that hundreds of quantization configurations were used, but there was no clear summary of used configurations in the description, could you please explain it in more detail?
clarity,_quality,_novelty_and_reproducibility: The work is interesting. It proposes to use Fisher information for guiding mixed precision quantization, what to the best of my knowledge hasn't been used for such purpose before. Quality of the submission is good, the work flows logically. Additional details provided in appendix clearly explain all claims and can be useful for reproducing results. 
summary_of_the_review: This is a good quality interesting paper with detailed experiments and clear conclusions.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Model compression via quantization of weights and activations helps improve the efficiency and reduce memory requirements of DNNs.
However, quantifying the impact of quantization across the different layers in a DNN is challenging. For this purpose, the proposed work establishes FIT as a measure of DNN performance, when using multi-precision quantization (MPQ), without the need to retrain the network.
FIT is a metric that combines the impact of quantization on weights as well as activations in a unified manner such that the computation of the metric itself is quick and readily usable. 
Thus, FIT can be used to understand and readily gauge the impact of MPQ and avoid the pitfalls of bad performance.
strength_and_weaknesses: Strengths
- The proposed work provides a clear and concise explanation of the problem domain being tackled. This guides the reader from the general impact of the proposed work towards the specifics of the problem being handled.
- The evaluation schemes provided for the estimator as well as the application on various networks provide a good foundation.

Weaknesses
- Could the authors provide citations for the statement: "different layers, within different architectures, respond differently to quantization". (Pg. 1, Introduction, Paragraph 4, Line 1)
- The proposal of rank-correlation as a novel evaluation criterion is mentioned as a core contribution towards the end of the introduction. Could the authors describe possible evaluation schemes to compare against prior works, as baselines to judge the proposed scheme. Currently, rank-correlation exists as a readily available tool that has been repurposed for evaluating the task at hand. Multiple baselines could help highlight the contribution.
- Could the authors provide citations for the statement: " which is insufficient for prediction, and more challenging for practitioners to implement. In addition, trace computation can become very expensive for large networks". (Pg. 3, Paragraph 1, Lines 9-11).
- While the evaluation of the EF estimator in Table 1 shows a relatively model-agnostic behavior, it would be instructive to include evaluations with alternative DNN architectures that branch away from bottleneck-like structures (more densely packed connections) and alternative datasets, to help firmly establish the agnostic behavior observed.
- Could the authors normalize the X and Y scales across both plots in Figure 2? The visual comparison would provide a more compelling evidence in that regard.
- Could the authors discuss the type of information provided by FIT_A and FIT_W and try to provide more justification as to how they provide complementary information?
clarity,_quality,_novelty_and_reproducibility: Quality
The work provides strong intuition of the importance of evaluating the quality of a network post quantization without having to retrain the network. In terms of general concepts and building new ideas from foundational ones, the quality of the work is very good.

Clarity
The well-laid out context as well as thorough explanation of the method itself lend a lot of clarity to the reader.

Originality
While the general ideas are firmly established in existing work, the novelty of the proposed work comes from combining them and simplifying them for the task at hand.
summary_of_the_review: The proposed work establishes the need for a metric to analyze the sensitivity of networks to quantization without having to retrain the network. For this purpose Fisher Information Trace is selected as the measure of choice and its estimation process is defined in the proposed work. 
Overall, the evaluation of FIT against the hessian, proving its equivalence for the task at hand, and subsequent evaluation across various application domains helps validate the claims made about FIT.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces the Fisher Information Trace (FIT) metric as a method of predicting the effects of quantization on a neural network.  This approach is an alternative to prior work on Hessian AWare Quantization (HAWQ) [Dong et al., 2019].  Three reviewers favor accept, one gives a marginally below rating, and one favors reject.  A common request of reviewers (including Reviewer QyLp, who gave an initial reject) was for additional experiments on more challenging settings.  The authors responded with results using FIT to evaluating mixed-precision quantization configurations for BERT models.  Reviewers in favor of accept point to clear development of the idea and demonstrated advantages of FIT in comparison to Hessian-based methods.  Overall, the Area Chair agrees with this plurality of reviewers and also believes the additional results presented in the author response address some of the reviewer concerns on experiments.  The paper might benefit from further expanding experimental settings, such as including CNNs on ImageNet (suggested by Reviewer qJTT).
justification_for_why_not_higher_score: As an improved method for the existing setting of mixed-precision network quantization, the scope is probably not broad enough to justify a spotlight.
justification_for_why_not_lower_score: A novel idea with sufficient demonstrated advantages over prior work justifies an accept.
Decision: Accept: poster