FIT: A Metric for Model Sensitivity
Authors: Ben Zandonati
Adrian Alan Pol
Maurizio Pierini
Olya Sirkin
Tal Kopetz
Venue: ICLR 2023 poster (2023)
Keywords: Fisher Information
Quantization

Abstract: Model compression is vital to the deployment of deep learning on edge devices. Low precision representations, achieved via quantization of weights and activations, can reduce inference time and memory requirements. However, quantifying and predicting the response of a model to the changes associated with this procedure remains challenging. This response is non-linear and heterogeneous throughout the network. Understanding which groups of parameters and activations are more sensitive to quantization than others is a critical stage in maximizing efficiency. For this purpose, we propose FIT. Motivated by an information geometric perspective, FIT combines the Fisher information with a model of quantization. We find that FIT can estimate the final performance of a network without retraining. FIT effectively fuses contributions from both parameter and activation quantization into a single metric. Additionally, FIT is fast to compute when compared to existing methods, demonstrating favourable convergence properties. These properties are validated experimentally across hundreds of quantization configurations, with a focus on layer-wise mixed-precision quantization.

Published as a conference paper at ICLR 2023
FIT: A METRIC FOR MODEL SENSITIVITY
Ben Zandonati University of Cambridge baz2 30cam. ac. uk
Adrian Alan Pol
Maurizio Pierini CERN
Princeton University edu
maurizio, cturriioo.pcrinneecn.oo
Olya Sirkin
Tal Kopetz CEVA Inc.
CEVA Inc.
s rk rrkinollyee ya@gma 1 1.1.1.... comm
tal kopetz@ccea-dspppcooo.
ABSTRACT
Model compression is vital to the deployment of deep learning on edge devices. Low precision representations, achieved via quantization of weights and activations, can reduce inference time and memory requirements. However, quantifying and predicting the response model to the changes associated with this procedure remains challenging. This response is non-linear and heterogeneous throughout the network. Understanding which groups parameters and activations are more sensitive to quantization than others is critical stage in maximizing efficiency. For this purpose, we propose FIT. Motivated by an information geometric perspective, FIT combines the Fisher information with model of quantization. We find that FIT can estimate the final performance of a network without retraining. FIT effectively fuses contributions from both parameter and activation quantization into single metric. Additionally, FIT is fast to compute when compared to existing methods, demonstrating favourable convergence properties. These properties are validated experimentally across hundreds of quantization configurations, with a focus on layer-wiss wise maxed-precision quantization.
INTRODUCTION
The computational costs and memory footprints associated with deep neural networks (DNN) hamper their deployment to noourcc-conssrainee environments like mobile devices (Ignatov et al., 2018), self-driving cars (Liu et al., 2019), or high-energy physics experiments (Coelho et al., 2021). Latency, storage and even environmental limitations directly conflict with the current machine learning regime of performance improvement through scale. For deep learning practitioners, adhering to these strict requirements, whilst implementing state-of-the-art solutions, compression methods, such as quantization (Gray & Neuhoff, 1998) and pruning (Janowsky, 1989), have become essential stages in deployment the edge.
constant challenge. As a result,
In this paper, we focus on quantization. Quantization refers to the use of sentations for values within the network, like weights, activations and even gradients. This could, for example, involve reducing the values stored in 32-bit floating point (FP) single precision, to INT-8/4/2 integer precision (IP). This reduces the memory requirements whilst allowing models to meet strict latency and energy consumption criteria on high-performanc hardware such as FPGAs.
repre-
Despite these benefits, there trade-off associated with quantization. As full precision computation is approximated with less precise representations, the model often incurs drop in performance. In practice, this trade-off's is worthwhile for resource-conssrained applications. However, the DNN performance degradation associated with quantization schemes, where post-training quantization (PTQ) to 8 bits and below is applied to the whole network (Jacob et al., 2018). Quantization Aware Training (QAT) (Jacob et al., 2018) is often used to recover lost performance. However, even after QAT, aggressive quantization may still result in a large performance drop. The model performance is limited by sub-optimal quantization schemes.
become unacceptable under aggressive
is known that different layers, within different architectures, respond differently to quantization (Wu et al., 2018). Akin to how more detailed regions of images more challenging compress, as

---

Published as a conference paper at ICLR 2023
certain groups of parameters. As is shown clearly by Wu et al. (2018), uniform bit-width schemes fail to capture this heterogeneity. Mixed-precision quantization (MPQ), where each layer within the network is assigned different precision, allows to push the performance-comee comppessio rofe to the limit. However, determining which bit widths to assign to each layer is non-trivial. Furthermore, the search space of quantization configurations is exponential in the number of layers and activations.
Existing methods employ techniques such as neural architecture search and deep reinforcement learning, which are computationally expensive and less general. Methods which aim to explicitly capture the sensitivity (or importance) of layers within the network, present improved performance and reduced complexity. In particular, previous works employ the Hessian, taking the loss landscape curvature as sensitivity and achieving state-of-the -ottc--f-the-arttt compression. Even so, many explicit methods are slow to compute, grounded in intuition, and fail include activation quantization. Furthermore, previous works determine performance based on only a handful of configurations. Further elaboration presented in Section
The Fisher Information and the Hessian are closely related. In particular, many previous works in optimisation present the Fisher Information as an alternative to the Hessian. In this paper, we use the Fisher Information Trace as means of capturing the network dynamics. We obtain our final FIT metric which includes a quantization model, through a general proof in Section 3 grounded within the field of information geometry. The layer-wiss wise form of FIT closely resembles that of Hessian Aware Quantization (HAWQ), presented by Dong et al. (2020). Our contributions in this work are as follows:
1. We introduce the Fisher Information Trace (FTT) metric, to determine the effects of quanti- zation. To the best of our knowledge, this is the first application of the Fisher Information to generate MPQ configurations and predict final model performance. We show that FIT demonstrates improved convergence properties, is faster to compute than alternative metrics, and can be used to predict final model performance after quantization.
2. The sensitivity of parameters and activations to quantization is combined within FIT as single metric. We show that this consistently improves performance.
3. We introduce rank correlation evaluation procedure for mixed-precision quantization, which yields more significant results with which to inform practitioners.
2 PREVIOUS WORK
In this section, we primarily focus mixed-precision quantization (MPQ), and also give context to the information geometric perspective and the Hessian.
Mixed Precision Quantization As noted in Section I, the search space of possible quantization configurations, i.e. bit setting for each layer and/or activation, is exponential in the number of layers: ((.) where B3 is the set of bit precisions and L the layers. Tackling this large search space has proved challenging, however recent works have made headway in improving the state-of-the-are
CW-HAWQ (Qian et al., 2020), AutoQ (Lou al., 2019) and HAQ (Wang al., 2019) deploy Deep Reinforcement Learning (DRL) to automatically determine the required quantization configuration, given a set of constraints (c.g. accuracy, latency or size). AutoQ improves upon HAQ by employing a hierarchical agent with a hardware-aware objective function. CW-HAWQ seeks further improvements by reducing the search space with explicit second-order information, as outlined by Dong et al. (2020). The search space is also often explored using Neural Architecture Search (NAS). For instance, Wu al. (2018) obtain 10-20x model compression with little no accuracy degradation. Unfortunately, both the DRL and NAS approaches suffer from large computational resource requirements. As result, evaluation only possible on a small number of configurations. These methods explore the search space of possible model configurations, without explicitly capturing the dynamics of the network. Instead, this is learned implicitly, which restricts generalisation.
More recent works have successfully reduced the search space of model configurations through explicit methods, which capture the relative sensitiviry of layers to quantization. The bit-width assignment is based on this sensitivity. The eigenvalues of the Hessian matrix yield analogous heuristic to the local curvature. Higher local curvature indicates higher sensitivities to parameter perturbation, would result from quantization lower bit precision. Choi et al. (2016) exploit
2

---

Published as a conference paper at ICLR 2023
this to inform bit- bit-precision configurations. Popularised by Dong et al. (2020). HAWQ presents the following metric, where a trace-based method is combined with a measure quantization error:
L
Tr(Hh) (1Q(0.) -rll.
1=1
Here, 81 and Q(1,) represent the full precision and model parameters respectively, for each block of the model. (HH denotes the parameter normalised Hessian trace. HAWQ-VI (Dong et al., 2019) used the top eigenvalue as an estimate of block sensitivity. However, this proved less effective than using the trace as in HAWQ-VA V2 Dong et al. (2020). The ordering of quantization depth established over the set of network blocks reduces the search space of possible model configurations. The Pareto front associated with the trade-off between sensitivity and size is used to quickly determine the best MPQ configuration for given set of constraints. HAWQ-V3 (Yao et al., 2021) involves integer linear programming determine the quantization configuration. Although Dong et al. (2020) discusses activation quantization, it is considered separately from weight quantization. In addition, trace computation can become very expensive for large networks. This is especially the case for activation traces, which require large amounts of GPU memory. Furthermore, only a handful of configurations are analysed.
Other effective heuristics have been proposed, such as batch normalisation (Chen et al., 2021) and quantization (?) scaling parameters. For these intuition-grounded heuristies, it is more challenging to assess their generality. More complex methods of obtaining MPQ configurations exist. Kundu al. (2021) employ straight-througe estimators of the gradient (Hubara et al., 2016) with respect to the bit-setting parameter. Adjacent work closes the gap between final accuracy and quantization configuration. Liu et al. (2021), a classifier after each layer used to estimate the contribution to accuracy.
Previous works within the field of quantization have used the Fisher Information for adjacent tasks. Kadambi (2020), use it as regularisation method to reduce quantization error, whilst it is employed by Tu et al. (2016) as method for computing importance rankings for boock//prrmeetes. addition, Li et al. (2021) use the Fisher Information part of block/layer wise reconstruction loss during post-traine post-training quantization.
Connections to the loss landscape perspective Gradient preconditioning using the Fisher Infor- mation Metric (FIM) - the Natural Gradient acts to normalise the distortion the loss landscape. This information is extracted via the eigenvalues which are characterised by the trace. This is closely related to the Hessian matrix (and Netwon's 's method). The two coincide, provided that the model has converged the global minimum 0", and specific regularity conditions are upheld (Amari, 2016) (see Appendix G.3). The Hessian matrix H is derived via the second-ords -odder expansion of the loss function minimum. Using it as a method for determining the effects of perturbations common. The FIM is more general, as it is yielded (infinitesimally from the invariant f-divergences. As a result, FIT applies to greater subset of models, even those which have not converged to critical points.
Many previous works have analysed and provided examples for, the relation between second-order information, and the behaviour of the Fisher Information. (Kunstner al., 2019; Becker & Lecun, 1989; Martens, 2014; Li et al., 2020). This has primarily been with reference to natural gradient descent (Amari, 1998), preconditioning matrices, and Newton's method during optimization. Recent works serve to highlight the success of the layer-wise scaling factors associated with the Adam opti- miser (Kingma & Ba, 2014) whilst moving in stochastic gradient descent (SGD) directions (Agarwal al., 2020; 2022). This is consistent with previous work (Kunstner al., 2019), which highlights the issues associated with sub-optimal scaling for SGD, and erroneous directions for Empirical Fisher base- preconditioning. The EF and its properties are also explored by Karakida al. (2019).
It is important to make the distinction that FIT provides different use case. Rather than focusing on optimisation, FIT is used to quantify the effects of small parameter movements away from the full precision model, as would arise during quantization. Additionally, the Fisher-Rao metric has been previously suggested as a measure of network capacity (Liang et al., 2019). In this work, we consider taking the expectation over this quantity. From this perspective, FIT denotes expected changes in network capacity as result of quantization.
3

---

Published as a conference paper at ICLR 2023
3 METHOD
First, we outline preliminary notation. We then introduce an information geometric perspective quantify the effects of model perturbation. FIT is reached via weak assumptions regarding quantization. We then discuss computational details and finally connect FIT to the loss landscape perspective.
3.1 PRELIMINARY NOTATION.
Consider training a parameterised model an estimate of the underlying conditional distribution p(y|æ, 8), where the empirical loss on the training set is given by: L()) 1. this case, f E IR" are the model parameters, and f f) is the loss with respect to single member 2; = (I,, 3i) Rd y of the training dataset D = {(r,, 3i) }i)] drawn i.i.d. from the true distribution. Consider, for example, the cross-entropy criterion for training given by: [ Ve. 0) = log p(ys ( 0). Note that in parts we follow signal processing convention and refer to the parameter movement associated with change in precision quantization noise.
N: E fx, Yr. 8). In
3.2 FISHER INFORMATION TRACE (FIT) METRIC
Consider a general perturbation to the model parameters: ply|2, 0 80). For brevity, we denote p(y|æ, Φ) as Pg. To measure the effect of this perturbation on the model, we use an f-divergence (e.g. KL, total variation or x²) between them: (PellPe+5o). It well-known concept in information geometry (Amari, 2016; Nielsen, 2018) that the FIM arises from such a divergence: DK (pollpotoo) $80 (0)680. The FIM, (0). takes the following form:
T(0) (x. Vo logr(u/æ, VoVo logp(v|>;0)) 0)²1
In this case, po(x, 3) denotes the fact that expectation taken over the joint distribution of x and The Rao distance (Atkinson & Mitchell, 1981) between two distributions then follows.
The exact (per parameter) perturbations d0 associated with quantization are often unknown. As such, we assume they drawn from underlying quantization noise distribution, and obtain expectation this quadratic differential:
E [80 I(0)80] = E[00]" ((0) + Tr((()CCox888)
We assume that the random noise associated with quantization is symmetrically distributed around mean of zero: E[80] and uncorrelated: Coul86 Diag (B[)0))). This yields the FIT heuristic
in general form:
Ω Tr (I (E(()))))))
To develop this further, we assume that parameters within single layer block will have the same noise power, as this will be highly dependent on block-based configuration factors. As a concrete example, take the quantization noise power associated with heterogencous quantization across different layers, which is primarily dependent on layer-wise bit-precision. As such, we can rephrase FIT in layer-wise form:
L Σ Tr(I(O))- - EE
Where denotes a single layer/block in a set model layers/blocks.
3.2. EXTENDING THE NEURAL MANIFOLD
The previous analysis applies well to network parameters, however, quantization also applied to the activations themselves. As such, to determine the effects of the activation noise associated with quantization, we must extend the notion of neural manifolds to also include activation statistics, à, over the dataset well. The general perturbed model is denoted follows: p(y|æ,, 0 88, à 8ā). The primary practical change required for this extension involves taking derivatives w.r.t activations rather than parameters feature which is well supported in deep learning frameworks (see Appendix
4

---

Published as a conference paper at ICLR 2023
C for activation trace examples). After which, the expectation is taken over the data. Once again, this change simple, as the empirical form of the Fisher Information requires a similar approximation (Section 3.3, and the two can be computed at the same time. Having considered the quantization of activations and weights within the same space, these can now be meaningfully combined in the FIT heuristic. This is illustrated in Section 4.
3.3 COMPUTATIONAL DETAILS
The empirical form of FIT can be obtained via approximations of B[06?); and Tr([((:)). The former requires either Monte-Carlo estimate, or an approximate noise model (see Appendix F), whilst the latter requires computing the Empirical Fisher (EF): f(0). This yields the following form:
ΣnO 1 -.r((((()))) [|001]
Kunstner et al. (2019) illustrates the limitations, as well as the notational inconsistency, of the EF, highlighting several failure cases when using the EF during optimization. Recall that the FIM involves computing expectation over the joint distribution x and We do not have access to this distribution, fact, we only model the conditional distribution. This leads naturally to the notion of the EF:
i(0)
N Σ Vofo(=)Vofo(x)"
The EF trace far less computationally intensive than the Hessian matrix. Previous methods (Dong et al., 2020; Yao et al., 2021; ;; Yu et al., 2022) suggested the use of the Hutchinson algorithm (Hutchinson, 1989). The trace of the Hessian extracted matrix-free manner via zero-mean random variables with variance of one: Tr(H) & Ei= rf Hr,, where m is the number of estimator iterations. is common to use Rademacher random variables r [-1,1]. First, the above method requires second backwards pass through the network for every iteration. This is costly, especially for DNNs with many layers, where the increased memory requirements associated with storing the computation graph become prohibitive. Second, the variance of each estimator be large. This is given by: Vir Hr. = 2 (1 - E, H3) (see Appendix G.4). Even for the Hessian, which has a high diagonal norm, this variance still be large. This is validated empirically in Section 4. In contrast, the EF admits a simpler form (see Appendix G.4);
Tr( = N Σ [[(=,
The convergence of this trace estimator improves upon that of the Hutchinson estimator in having lower variance. Additionally, the computation is faster and better supported by deep learning frameworks: its computation can be performed with single network pass, second derivative required. similar scheme used in the Adam optimizer (Kingma Ba, 2014), where an exponential moving average is used. Importantly, the EF trace estimation has a more model-agnostic variance, also validated Section 4.
EXPERIMENTS
We perform several experiments determine the performance FIT. First, we examine the properties of FIT, and compare it with the commonly used Hessian. We show that FIT preserves the relative block sensitivities of the Hessian, whilst having significantly favourable convergence properties. Second, we illustrate the predictive performance of FIT, in comparison to other sensitivity metrics. We then show the generality of FIT, by analysing the performance on a semantic segmentation task. Finally, conclude by discussing practical implications.
4.1 COMPARISON WITH THE HESSIAN
To evaluate trace performance and convergence in comparison to Hessian- methods, we consider several computer vision architectures, trained the ImageNet (Deng al., 2009) dataset. We first
5

---

Published as a conference paper at ICLR 2023
illustrate the similarity between the EF trace and the Hessian trace and then illustrate the favourable convergence properties of the EF. Further analysis for BERT on SST-2 is given in Appendix A.
Trace Similarity.
Figure I shows that the EF preserves the relative block sensitivity of the Hessian. Substituting the EF trace for the Hessian will not affect the performance of heuristic-basedt search algorithm. Additionally, even for the Inception- V3 trace in Figure (d), the scaling discrepancy would present no change in the final generated model configurations because heuristic methods (which search for minimum) are scale agnostic.
30
Nuker
Hoss
Heotla
Hessin
Hecian
10"1
haw
2071
10-1
10
10
20
10
20
30
40
50
20
20
10
Mlocka
(a) ResNet-18 18
(b) ResNet-50
(c) Mobileeet---te V2
(d) Inception-V
Figure 1: Hessian and EF Parameter traces for four classification models. The Hessian and EF traces for the parameters are very similar. For Inception- V3, this holds up to a constant scaling factor.
Convergence Rate.
Across all models, the variance associated with the EF trace is orders of magnitude lower than that of the Hessian trace, as shown in Table 1. This is made clear in Figure 2, where the EF trace stabilises in far fewer iterations than the Hessian. The analysis in Section 3, where suggest that the EF estimation process has lower variance and converges faster, holds well in practice.
Importantly, whilst the Hessian variance shown in Table is very model dependent, the EF trace estimator variance more consistent across all models. These results also hold across differing batch sizes (see Appendix Dhe results of this favourable convergence illustrated in Table 1. For fixed tolerances, the model agnostic behaviour, faster computation, and lower variance, all contribute to large speedup.
Estimator
Variance Hessian
Iteration Time (ms) Hessian
Relative Speedup
EF
EF
ResNet-18
0.15 = 0.03
1.09 1.09 0.02
47.78 0.03
186.54 0.56
27.67 5.40
ResNet-50
0.31 = 0.04
6,91 + 1.52
152.02 + 0.38
639.13 1.02
94.24 = 34.06
MobileNet-N V2
0.24 = 0.01
4.81 = 0.38
58.84 + 0.55
2573.50 = 3.06
894.24 = 121.25
Inception-V V3
0.43 0.03
13.62 0.46
235.43 + 0.21
905.04 4.69
122.06 14,90
Table 1: Representative examples of the typical speedup associated with using the EF over the Hessian. Iteration times and variances are computed as sample statistics over multiple runs of many iterations, with batch size of 32. The resulting speedup is denoted for fixed tolerance, which can be practically computed via a moving variation of the mean trace. Early stopping is possible when we first reach the desired tolerance. The measurements were performed NVidia 2080Ti GPU.
4.2 FROM FIT TO ACCURACY
In this section, we use correlation as a novel evaluation criterion for sensitivity metries, used to inform quantization. Strong correlation implies that the metric is indicative of final performance, whilst low correlation implies that the metric less informative for MPQ configuration generation. In this MPQ setting, FIT is calculated as follows (Appendix F);
L Σ
IT(((O)))
Omaz Omin
2
Each bit configuration {b, yields unique FIT value from which we can estimate final perfor- mance. The space of possible models is large, outlined in Section thus randomly sample

---

Published as a conference paper at ICLR 2023
10
10
10
16
10
10
10
10
10
10
10
10
50
300
150
200
50
100
150
50
100
50
100
150
200
Eerotion
(a) ResNet-18
(b) ResNet-50
MobileNet-V2
(d) Inception-V
Figure 2: Comparison between Hessian and EF trace convergence for four classification models.
configuration space. 100 CNN models, with and without batch-normalisation,, are trained on the C0 and Mnist datasets, giving total of studies.
Comparison Metries.
As noted in Section 2 previous works (Kundu et al., 2021; Liu al., 2021; ?) provide layer-wise heuristics with which to generate quantization configurations. This differs from FIT, which yields a single value. These previous methods are not directly comparable. However, we take inspiration from Chen et al. (2021); ?. using the quantization ranges (QR: 10maz Omin)) l well the batch normalisation scaling parameter (BN: ?) (loffe & Szegedy, 2015), to evaluate the efficacy of FIT in characterising sensitivity.
In addition to these, we also perform ablation studies by removing components of FIT, resulting in FITw. w. FIT, and the isolated quantization noise model, i.c. E[60%]. Note that do not include HAWQ here as it generates results equivalent to FITw. Equations for these comparison heuristics shown in Appendix E. This decomposition helps determine how much the components which comprise FIT contribute to its overall performance.
Experiment
Dataset
BN
FIT
QR
Noise
FITw
QRw
FIT
QRA
BN
Cifar-10
0.89
0.76
0.85
0.87
0.86
0.38
0.36
0.33
B
Cifar- 10
0.77
0.67
0.60
0.65
0.61
0.61
0.60
Mnist
0.86
0.89
0.83
0.72
0.80
0.44
0.39
0.39
D
Mnist
0.90
0.58
0.70
0.72
0.72
0.55
0.44
Table 2: Rank correlation coefficient for each combination sensitivity and quantization experiment. W/A subscript indicates using only either weights or activations. BN indicates the presence of batch-normalisation within the architecture.
Figure 3 and Table 2 show the plots and rank-correlation results across all datasets and metrics, From these results, the key benefits of FIT are demonstrated.
FIT correlates well with final model performance. From Table 2, we can see that FIT has consistently high rank correlation coefficient, demonstrating its application in informing final model performance. Whilst other methods vary in correlation, FIT remains consistent across experiments.
FTT combines contributions from both parameters and activations effectively. We note from Table 2, that combining FITw and FIT consistently improves performance. The same is not the case for QR. Concretely, the average increase correlation with the inclusion of FIT is 0.12, whilst for ARA, the correlation decreases on average by 0.02. FIT scales the combination of activation and parameter contributions correctly, leading to a consistent increase in performance. Furthermore, from Table 2 observe that the inclusion of aatch-normalisation alters the relative contribution from parameters and activations. As result, whilst FIT combines each contribution effectively and remains consistently high, QR does not.
7

---

Published as a conference paper at ICLR 2023
Experiment
10
20
00
05
1.0
155
80
61
62
0.5
90.2
99.3
93.9
93.8
99.7
Experinent
98.6
99.3
99.0
93.9
93.8
99.7
99.6
98.5
Experiment
FIT
QR
Noise
FIT FITw/HAWQ
Figure 3: Plots of the chosen predictive heuristic against final model performance.
4.3 SEMANTIC SEGMENTATION
In this section, illustrate the ability of FIT to generalise to larger datasets and architectures, as well as more diverse tasks. We choose to quantify the effects of MPQ on the U-Net architecture (Ronneberger al., 2015), for the Cityscapes semantic segmentation dataset (Cordts et al., 2016). Additional analysis of BERT on SST-2 is shown in Appendix A. In this experiment, we train 50 models with randomly generated bit configurations for both weights and activations. For evaluation, we use Jaccard similarity. or more commonly, the mean Intersection over Union (mloU). The final tolerance
correlation coefficient between FIT and mloU, EF trace computation is stopped of 0.01, occurring at 82 individual iterations. The weight and activation traces for the trained, full precision, U-Net architecture shown in Figure Figure 4(c) shows the correlation between FIT and final model performance for the U-Net architecture on the Cityscapes dataset. In particular, we obtain a high final rank correlation coefficient 0.86.
T1a00
WMA
0.600
0.590
18
0.585
0,580
13.
0.00
0.01
0.02
0.03
(a) Weight Trace
(b) Activation Trace
(c) FIT against mloU
Figure 4: EF weight (a) and activation (b) traces for U-Net architecture on the Cityscapes dataset. (c) FIT against mloU for 50 random MPQ configurations of U-Net on Cityscapes semantic segmentation. Example configurations are highlighted, showing the average bit precision for weights and activations.
8

---

Published as a conference paper at ICLR 2023
4.4 FURTHER PRACTICAL DETAILS
Small perturbations In Section 3, we assume quantization noise, 80, is small with respect to the parameters themselves. This allows us to reliably consider a second-order approximation of the divergence. In Figure 5, plot every parameter within the model for every single quantization con- figuration from experiment A. Almost all parameters adhere to this approximation. We observe that setting covers practical situations, and we leave characterising more aggressive quantization (1/2 bit) to future work.
Distributional shift Modern DNNs often over-fit their training dataset. More precisely, models are able to capture the small distributional shifts between training and testing subsets. FIT is computed from the trained model, using samples from the training dataset. As such, the extent which FIT captures the final quantized model performance on the test dataset is, therefore, dependent on model over-fitting Consider for example dataset D. We observe high correlation of 0.98 between FIT and final training accuracy, which decreases to 0.90 during testing. This is further demonstrated in Figure For practitioners, FIT more effective where over-fitting less prevalent.
10.11
00.6
99.4
0.0
0.2
(a) Noise Parameter
(b) FIT against training
magnitude
accuracy on experiment D
Figure 5: (a) Noise parameter magnitude. The line indicates equal magnitude, and is shown for reference. (b) FIT against final training accuracy for experiment D. The correlation coefficient, in this case, is 0.98.
CONCLUSION
In this paper, we introduced FIT, metric for characterising how quantization affects final model performance. Such method is vital in determining high-perforning mixed-precision quantization configurations which maximise performance given constraints on compression.
We presented FIT from an information geometric perspective, justifying its general application and connection to the Hessian. By applying quantization-specif noise model, as well as using the empirical fisher, we obtained
groundedddd and practical form of FIT.
Empirically, we show that FIT correlates highly with final model performance, remaining consistent across varying datasets. architectures and tasks. Our ablation studies established the importance of including activations. Moreover, FIT fuses the sensitivity contribution from parameters and activations yielding single, simple use, and informative metric. In addition, we show that FIT has very favourable convergence properties, making it orders of magnitude faster to compute. We also explored assumptions which help guide practitioners.
Finally, by training hundreds of MPQ configurations, we obtained correlation metrics from which to demonstrate the benefits using FIT. Previous works train small number of configurations, We encourage future work in this field to replicate our approach, for meaningful evaluation.
Future Work.
Our contribution, FIT. worked quickly and effectively in estimating the final performance of quantized model. However, as noted in Section 4, FIT is susceptible to the distributional shifts associated with model over-fitting. In addition, FIT must be computed from the trained, full- precision model. We believe dataset agnostic methods provide promising direction for future research, where the MPQ configurations be determined from initialisation.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We have included sample code for generating the parameter and activation traces, well as generating and analysing quantized models. Further experimental details are presented in Appendix E. Relevant complete proofs are shown in Appendices F and G.
REFERENCES
Naman Agarwal, Rohan Anil, Elad Hazan, Tomer Koren, and Cyril Zhang. Disentangling adaptive gradient methods from learning rates, 2020. URL https: /arxivvvvvv../////i/ www.vorgg/aaa. www//bs//////ooooooo.//..../aaar aaaaaaas/00 2002.00022..0 11803.
Naman Agarwal, Rohan Anil, Elad Hazan, Tomer Koren, and Cyril Zhang. Learning rate grafting: Transferability of optimizer tuning, 2022. URL https: http///pppeewwweeeeeeeeee wwwprrvviiew.eeefouurr... nerumfffu/fmid FpKgG312 19.
Shun-ichi Amari. Natural gradient works efficiently learning, Neural Computation, 10(2):251-276. 1998. doi: 10. 00.116//899966693300000
Shun'ich Amari. Information geometry and its applications. Springer, 2016.
Colin Atkinson and Ann F. S. Mitchell. Rao's distance measure. Sankhyā; The Indian Journal of Statistics, Series A (1961-2002), 43(3):345-365, 1981. ISSN 0581572X. URL http: /www. jstto. stttor..rg/sttaale25555500
Suzanna Becker and Yann Lecun. Improving the convergence of back pack-propagation learning with second-orde -oooer methods. 01 1989.
Sung-En Chang, Yanyu Li, Mengshu Sun, Weiwen Jiang, Sijia Liu, Yanzhi Wang, and Xue Lin. Rmsmp: A novel deep neural network quantization framework with row-wise mixed schemes and multiple precisions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5251-5260 2021a.
Sung-En Chang. Yanyu Li, Mengshu Sun, Runbin Shi, Hayden K-H So, Xuehai Qian, Yanzhi Wang. and Xue Lin. Mix and match: A novel fpga-centric deep neural network quantization framework. In 2021 IEEE International Symposium on Hligh-Performace Computer Architecture (HPCA), pp. 208-220 IEEE, 2021b.
Boyu Chen, Peixia Li, Baopu Li, Chen Lin, Chuming Li, Ming Sun, Junjie Yan, and Wanli Ouyang. Bn-nas: Neural architecture search with batch normalization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 307-316, 2021.
Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Towards the limit of network quantization, 2016. URL https: /arxivv...... wwt.obog///...ooooo.o. http:/bbs/1111// 01543.
Claudionor N Coelho, Aki Kuusela, Shan Li, Hao Zhuang, Jennifer Ngadiuba, Thea Klaeboe Aarrestad, Vladimir Loncar, Maurizio Pierini, Adrian Alan Pol, and Sioni Summers. Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectors. Nature Machine Intelligence, 3(8):675-686, 2021.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /0/0. 04.005 2018.
10

---

Published as a conference paper at ICLR 2023
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precisioe Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 293-302, 2019.
Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trace-weighted quantization of neural networks. Advances in neural information processing systems, 13:18518-18822 2020.
Amir Gholami, Sehoon Kim, Zhen Dong. Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer.
A survey of quantization methods for efficient neural network inference, 2021. URL https: // Marxiv. 20000
R.M. Gray and D.L. Neuhoff. Quantization. IEEE Transactions on Information Theory, 44(6):
2325 2383, 1998. doi: 10.1109/18.72000
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- El-yaniv, and Yoshua Bengio. Binarized neural networks. Advances neural information processing systems, 29, 2016.
M.F. Hutchinson. A stochastic estimator the trace of the influence matrix for laplacian smoothing splines. Communication in Statistics- Simulation and Computation, 18:1059-1076 1076, 01 1989. doi: 10.1080003610919000 12866.
Andrey Ignatov, Radu Timofte, William Chou, Ke Wang, Max Wu, Tim Hartley, and Luc Van Gool. Ai benchmark: Running deep neural networks on android smartphones. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pp. 0-0, 2018.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-onll inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704-2713, 2018.
Steven Janowsky. Pruning versus clipping in neural networks. Phys. Rev. A, 1,3396600-66033 Jun 1989, doi: 10.1 Rew.// PhysRevA. 39. 6600.
URL https: /ttpps://iink aaps
Prad Kadambi. Comparing fisher information regularization with distillation for dnn quantization. 2020. URL https::/ hppp//vieeree.eeeeeemee https:://penneevie. wwt.:eet//e...e pdddddddddddd 1pws.
Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. Universal statistics of fisher information in deep neural networks: Mean field approach. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 1032-1041. PMLR, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv e-prints, art. arXiv:1412.69800 December 2014.
Souvik Kundu, Shikai Wang, Qirui Sun, Peter A. Beerel, and Massoud Pedram. Bmpq: Bit- gradient sensitivity driven mixed-precision quantization of dnns from scratch, 2021. URL https:
Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical fisher approx- imation for natural gradient descent. Advances in neural information processing systems, 32, 2019.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint N.ti::9100097000 2019.
Xinyan Li, Qilong Gu, Yingxue Zhou, Tiancong Chen, and Arindam Banerjee. Hessian based analysis of sgd for deep nets: Dynamics and generalization. In Proceedings of the 2020 SIAM International Conference Data Mining, pp. 190-198. SIAM, 2020.
11

---

Published as a conference paper at ICLR 2023
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2102.05426. 2021.
Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metric, geometry. and complexity of neural networks. In The 22nd international conference on artificial intelligence and statistics, pp. 888-896 896. PMLR, 2019.
Hongyang Liu, Sara Elkerdawy, Nilanjan Ray, and Mostafa Elhoushi. Layer importance estimation with imprinting for neural network quantization. In 2021 IEEE/CVE Conference Computer Vision and Puttern Recognition Workshops (CVPRW), pp. 2408-2417, 2021. doi: 10.1109/
CVPRW53098.202 202.0933.
Shaoshan Liu, Liangkai Liu, Jie Tang, Bo Yu, Yifan Wang, and Weisong Shi. Edge computing for autonomous driving: Opportunities and challenges. Proceedings of the IEEE, 167-8116 2019.
Qian Lou, Feng Guo, Lantao Liu, Minje Kim, and Lei Jiang. Autoq: Automated kernel-wise neural network quantization, 2019. URL httpt:: /aaxiv. org/ abs/ 1902. 0569C.
James Martens. New perspectives the natural gradient method. CoRR, abs/ 1e/1.112.11999 2014. URL http://arxiv..rog wwt.//iw...rraaooooo halggg/ab//4110 .1993.
Frank Nielsen. An elementary introduction to information geometry. arXiv:1808.08277 August 2018.
arXiv e-prints, art.
Xu Qian, Victor Li, and Crews Darren. Channel-wise hessian aware weighte quantization of neural networks, 2020. URL https: http//aaxvvv///////////./...av./ org. abs 2000 2008. 08284.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234 241. Springer. 2015.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https: /////aaaaaannttooooeoooaaa org//or 111//11131100
Ming Tu, Visar Berisha, Yu Cao, and Jae-sun Seo. Reducing the model order deep neural networks using information theory. In 2016 IEEE Computer Society Annual Symposium on VLSI (ISVLSI), pp. 98. IEEE, 2016
Kuan Wang. Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-awarr automated quantization with mixed precision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8612-8620, 2019.
Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Mixed precision quantization of convnets via differentiable neural architecture search, 2018. URL
https: // //tp://aaavv////aa/eaa ort/abs//11/... /ttt:s////ardvooorrr//////122 00090.
Zhewei Yao, Zhen Dong, Zhangcheng Zheng, Amir Gholami, Jiali Yu, Eric Tan, Leyuan Wang, Qijing Huang, Yida Wang, Michael Mahoney, et al. Hawq-v3: Dyadic neural network quantization. In International Conference on Machine Learning, pp. 11875-11111 1886. PMLR, 2021.
Shixing Yu, Zhewei Yao. Amir Gholami, Zhen Dong, Schoon Kim, Michael W Mahoney, and Kurt Keutzer. Hessian-awen aware pruning and optimal neural implant. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3880-3891, 2022.
12

---

Published as a conference paper at ICLR 2023
A ADDITIONAL EXPERIMENTS: BERT ON SST-2
FITERAA Acoiney
Unifor
2. A.ttentio outsut dense
b. dimeemedate dense
doutput
Lattertion seel
Lattentonn output dense
2. mimammdiate dense
Loutsut
Zatterion
Lationiinnsel
J.tthenton output
2. mammmdatae dense
2outru desse
Sustenttion nuety
Batteriin koy
B.ttiontinn outsot donse A Aimarmediate dense
3 Buubut donse
datteetion
outtertion se eeury
a.ttoniin ouisst
austercian koy o Batteriion value
Cimemedaae
Aoutsut cense
a anttemecace deree
Sattewiion seti
1 Lattertion sauurry
S.tterrion bey
1 add key
Latteriin ott value
Sattension output
B.memmdutt dense
5 Soutsct dasse Exttention sutr
2uttenion-ad key
zattemiin valiel 2 zintemelieee derce
3 aattertion eerr
S.uteniin..ad add key
mimemmdatt
Sutterrion ust value
Gounut
iittemetivee derce
Tattertion cutty
Luternio-aal kuy
Patteiion astu
aatteriin valie
pitemmtiine dence
Liniemedaate
batterbon..a
Toulpst denss
Sutterioo add key
Battemion sellouyy
Sattemiinn ant value
aattemtion sotttey
S Sintemerioee derce
attention outzut
cateemiann
douse
6.premedion derce
Sattention aerl
7. 7.dterrion sel key 7. 7attemioo sel value
aatiention outsut
9 sids
Ziitemetioee derne
Eustteriion nutr
dasse
Ripeemetion dense
Rattemtoon
10 ent
Sultter bo sor S sadtemiin selt key s. sutteriin set value Suntemmediae derce
atiection outsut
to
seff
I1 an
iuitemeeiine derce 13 alterttr ceeryery
11 si.ttianinnoiise dense
13. memmmmadaaa
33 sitttemiin..e aet key
31 altwerin velu 21 plintomedinee durse
(a) Weight Configuration
(b) Activation Configuration
Figure 6: Comparison between bit precision configurations for the uniform scheme: (W4, A8), and comparable (BOPs) configurations generated via FIT.
13

---

Published as a conference paper at ICLR 2023
RTDMM Accunry 91.68
Unitimm w2 42 Accu 50.52
Outtertion
Buttherion set
o ent
9. attention outsut desse
Lattoiionseffoory
toteeioooiisst dense
2.miemedate
2 aatentinaddivii
7. Z.tteetonn outst deste
2.meerddala dente
Bateeiinseef sole
Latrerton ount desse
2.mmmrdata
kylleniensdlky
4atooniinse sole
Battertisn eerryrr o auttemiion sul key o cattwernn set wilu
sateetion ouit dente
Gistomeciane dorce
5 ai
Lattwrin 3 1.ttemmetae derse
5. Aateenton outstt dente
Siniamedale dense
Southut
2 2estemion.aal koy
2.ttemmetaee derse
cattetton.see..mm
5. sattenton outct desse
iinermeditte dense
Kouttut
Aittemesaos deree
Lattertin suuuury
Laberiensdliky
w sel
Pattewiincee soue
<uttenion sdf volee
7.4tention dense
<irtemmstie derse
Xintemediltee dense
Zouttut
Sattemionse
Bulleniensdikey
5, Siitemesiine dence 6 autertoonaat...ay
5. Lateetion outsut denss
Gatterrien sel sattemion sef volue Kintemerivee dence
noutsut
Rutteriionn
T.atemmis
T.tterrian
Jipeemetion dence
3. R.ttenton outsut cense
9. simmemddate desse
Suttertonsata..a 8 a.tteonio adf key Rattemiann. aut velue Aittemetivee dense
Routtut
10 of
satemcio.aet key
12 Shauenion outsit dense
10 mereemeeele dense
Stinemecisee darse 10 diaatertiion.nn ueit.urr
10 a0.atteniion.. sef key
atttttiiinnaa velue
10
11 Manionouussttee dess
33, mamemmaaaaa dense
11ateerion.s
dens
31 altterin valu
11 iLinomeeisee Jurse
Stpiccioo
(a) Weight Configuration
(b) Activation Configuration
Figure 7: Comparison between bit precision configurations for the uniform scheme: (W2, A8), and comparable (BOPs) configurations generated via FIT.
This section extends our experimental evaluation of FIT to other challenging datasets and benchmarks. Notably, we focus on quantizing BERT-base Devlin et al. (2018) for the SST-2 dataset Socher et al. (2013). BERT-base is quantized via layer-wise mixed precision simulated asymmetric quantization of
14

---

Published as a conference paper at ICLR 2023
weights and activations. We chose 50 distinct MPQ configurations for BERT SST-2 across range of bit precisions [2,3,4,6,8]. We obtain a final correlation score of 0.752 between post-fine- fine--uiingg accuracy and FIT score for these MPQ configurations. Similar other experiments, this demonstrates that FIT has an excellent predictive power of the impact of quantization, even for large and deep case for FIT, Figures and 7 show highly non-trivial MPQ
models such BERT. To illustrate configurations compared to the uniform baseline methods constrained by operations (BOPs). In this case, we obtain higher MPQ accuracy comparable BOPs, showing the superior performance of FIT in selecting high-pent high-performing MPQ configurations. Notably, FIT can trade off with any arbitrary hardware constraint (e.g. latency, power), making it very flexible.
From a computational perspective, given batch size of 256, FIT converged with variance of 0.73 very few iterations. Conversely, the Hessian trace method could not properly converge even after 200 iterations, which is reflected in variance of 5256. This granularity of layer-wise quantization renders Hessian-bar based methods challenging to use in practice. The EF traces for the weights and activations of BERT are shown in Figure 8.
Trace
10-1
10-3
10-2
20
40
60
10
20
30
40
Layer
Layer
(a) Weight Trace magnitude
(b) Activation Trace
Figure 8: EF weight (a) and activation (b) traces for BERT-basc on the SST-2 dataset.
B QUANTIZATION AWARE TRAINING (QAT)
Quantization can lead to significant performance degradation Gholami et al. (2021). It is, therefore. necessary perform additional quantization aware training (QAT) in order recover this lost performance, QAT involves simulating the effects of quantization during training. The quantization function Q()) is applied to the floating point parameter values during the forward pass of the network. Q()) is piece-wise flat. As result, to propagate gradients, (STE) (Hubara et al., 2016). In effect, this bypasses the quantization function during gradient computation in the backwards pass. Figure 9 illustrates this process. QAT also involves learning the quantization ranges. For weights, a max-min approach is taken, whilst for activations, an exponential moving average is used. As result, the scaling and zero points for quantization mapped correctly. In addition, batch-normalisation parameters can be folded into weights for efficiency.
straight- straight-through estimator
Figure 9: Overview QAT Q(0) is applied during the forward pass, and an STE is used in the backwards pass.
15

---

Published as a conference paper at ICLR 2023
B.1 FURTHER QUANTIZATION DETAILS
During our experiments, we employ layer-wise synmetric/asymmettii simulated quantization. In addition, layer-wise quantization ranges accumulated for short period (30 iterations) the beginning of QAT to accurately tune the quantization function across the expected range. Note that this is particularly important for activation quantization. As a result of the simulated nature of the quantization, we do not perform batch-norm folding in our experiments. However, this may improve results further, given the additional scaling factor per channel (rather than just per-layer).
Notably, investigating FIT combined with other more complicated quantization schemes such as those proposed by Chang et al. (2021b) and Chang et al. (2021a) provide appealing directions for future work. In particular, we note that due to the formulation of FIT, differing quantization schemes are realised by changing the quantization noise model. We look forward to investigating this future.
C ACTIVATION TRACES
Achobon
10
10m
10a
10
10:4
Blddo
Biccoo
(a) ResNet-18
(b) ResNet-50
(c) Mobilenet-VV V2
(d) Inception-V V3
Figure 10: EF Activation traces for four classification models.
D ESTIMATOR COMPARISON
For the four classification models considered, Tables 3 and indicate the estimator variances and iteration times associated with the EF and Hessian for a variety batch sizes. Whilst the EF exhibits expected variance reduction behaviour, the Hessian does not, and requires minimum, depeeddeeend,nt, batch size to achieve stable behaviour. In all cases, the EF estimator variance is estimated over 3 runs
orders of magnitude lower than that of the Hessian. Means and variances of 200 samples. Deviations are normalised w.r.t. the trace magnitude, taking the average across blocks/layers. This ensures all the statistics are comparable, regardless of convergence bias. The relative speedup associated with fixed tolerance is computed as follows:
oy tH OFF tEF
Where o² indicates the estimator variance and f is iteration time. This follows the Monte-Carlo estimate variance properties.
E FURTHER EXPERIMENTAL DETAILS
For each of experiments A, B, C and D, we trained 100 convolutional classifiers on the Cifar10 and Mnist datasets. In both cases, we performed experiments with and without the inclusion of batch normalisation layers before each activation. The architecture used consisted of three convolutional layers, followed by a fully connected classification head, with ReLU activations between. The first two blocks also followed by MaxPool layers. This is shown in Figure 11. Between the Mnist and Cifar 10 datasets, the number of filters was scaled by two. To obtain the data, we first trained a full precision version of the network for 50 epochs using the Adam optimizer. A learning rate of
16

---

Published as a conference paper at ICLR 2023
Table Estimator Variances associated with the EF and Hessian for a variety of batch sizes, over runs of 200 iterations for each model and batch size.
Batch Size
EF
Hessian
2
3
Mean
Stdev
2
3
Mean
Stdev
ResNet-18 18
4
0.82
0.99
1.27
1.03
0.23
10.64
9.77
6.89
9.10
1.96
0.61
0.42
0.61
0.55
0.11
5.01
4.28
4.38
4.56
0.40
16
0.33
0.30
0.26
0.30
0.03
1.77
2.32
2.03
2.04
0.27
32
0.13
0.18
0.15
0.15
0.03
1.09
1.07
1.11
1.09
0.02
ResNet-50
1.81
2.44
2.03
2.10
0.32
8
1.15
1.06
1.19
1.13
0.07
22.57
45.45
76.14
48.05
26.88
16
0.55
0.71
0.51
0.59
0.11
17.68
22.41
14.56
18.21
3.95
32
0.26
0.34
0.32
0.31
0.04
8.60
5.65
6.49
6.91
1.52
MobileNet-V2
4
1.18
1.29
1.03
1.17
0.13
8
0.67
0.61
0.71
0.66
0.05
17.65
72.02
34.19
41.28
27.87
16
0.36
0.35
0.37
0.36
0.01
9.36
10.54
5,092.34
9.95
0.84
32
0.25
0.24
0.22
0.24
0.01
4.75
5.23
4.81
0.38
Inception V3
4
3.97
3,00
2.77
3.24
0.64
1.92
1.95
1.70
0.40
16
0.89
0.66
0.69
0.75
0.13
31.55
65.36
109.84
68.92
39.26
32
0.44
0.45
0.39
0.43
0.03
13.81
13.96
13.10
13.62
0.46
Table 4: Iteration times associated with the EF and Hessian for a variety of batch sizes, averaged over 3 runs 200 iterations for each model and batch size.
Batch Size
EF (ms)
Hessian (ms)
2
3
Mean
Stdev
3
Mean
Stdev
ResNet-18
11.06
11.29
11.29
11.21
0.13
41.91
40.99
40.58
41.16
0.68
16.98
62.80
61.16
61.83
0.86
16
24.07
24.46
24.12
24.22
0.21
100.01
95.40
98.38
97.93
2.34
32
47.81
47.77
47.75
47.78
0.03
186.92
186.80
185.90
186.54
0.56
ResNet-50
30.89
30.77
30.75
0.07
150.69
148.17
149.08
149.31
1.27
47.90
48.50
48.49
48.30
0.34
196.37
201.74
200.99
199.70
2,91
16
83.67
83.54
82.94
83.38
0.39
332.11
341.01
338.11
337.08
4.54
32
152.44
151.92
151.69
152.02
0.38
639.62
639.82
637.96
639.13
1.02
MobileNet-V2
4
19.32
19.38
19.57
19.42
0.13
709.06
705.77
704.53
706.45
2.34
8
21.49
21.87
21.96
21.77
0.25
913.47
917.50
914.63
915.20
2.07
16
33.85
33.83
33.73
33.80
0.06
1.460.03
1,464.95
1,460.89
1,461.96
2.62
32
58.65
58.80
$9.07
58.84
0.21
2,570.02
2,575.81
2,574.66
2,573.50
3.06
Inception-Ve V3
4
56.30
59.83
57.00
$7.71
1.87
265.04
276.99
279.17
273.73
7.61
8
83.77
83.21
82.41
83.13
0.68
325.06
335.16
332.37
330.86
5.21
16
135.46
132.97
132.58
133.67
1.56
530.60
537.80
535.57
534.66
3.69
32
236.07
235.14
235.09
235.43
0.55
901.72
908.35
955.94
905.04
4.69
0.01 was chosen, and increased to 0.1 with the inclusion of batch normalization. A cosine-annealing learning rate schedule was used. We then used this trained full precision model as a checkpoint to initialise randomly chosen mixed precision configurations, and training was continued for another 30 epochs with learning rate reduction of 0.1, using the same schedule. Quantization configurations chosen uniformly at random from the possible set of bit precisions: [8.6,4.3]. In these cases, initialisation and training were identical across all MPQ models, so as to compare the final performance.
17

---

Published as a conference paper at ICLR 2023
Figure 11: Small convolutional classifier architecture used in the experiments detailed in Section 4
E.1
FURTHER DETAILS FOR COMPARISON METRICS
QR: The QR baseline represents the use of the quantization ranges to replace the EF as the
sensitivity metric:
Oman 2
Omin
mmaz
mmin
BN: the BN baseline represents the use of the batch-norm scaling factor y replace the EF the sensitivity metric.
Σ
0 Omar Oriin
24
w/A: To obtain FTTw, we remove the component which takes into account activation quan- tization sensitivity. Similarly, to obtain FIT, we remove the component which takes into account weight quantization sensitivity. Recall that Section 3, we extend parameter space include the activation statistics. In these ablations, we remove (W) or retain (A) these contributions.
F QUANTIZATION AND NOISE MODEL
The following analysis serves to motivate the direct connection between model perturbation and bit configuration. During quantization, it is common historically to assume the quantization error is uncorrelated with the original signal, yielding an approximately uniform distribution. This leads to an important assumption: The quantization error of each parameter is independent and uniformly distributed with a mean zero. Figure 12 serves to motivate the validity of this assumption.
15000
.000
13000
3000
30000
300
8003
400
2003
180
-22
0.
63
-0015 6800
-2.03 -0005
od8s
od03
0d3s
(a) Weight Distribution
(b) Quantized Weight Distribution (c) Quantization Error Distribution
Figure 12: ResNet-18 block 12, quantization distribution analysis showing the validity of the uniform noise assumption.
Under this assumption, it is possible evaluate this noise power:
18

---

Published as a conference paper at ICLR 2023
A²
B[609]
12
where A denotes the step width of quantization, and can be modelled directly by the quantization scheme being used. In this case, uniform (min-max) quantization. Letting Q(#) - 0g denote the quantization function, uniform quantization can be expressed as follows:
0min
Omin
Oman
Ominn
2b
where b is the quantization bit precision. This yields the following quantization noise power:
1 Imar 0min
E[600]
12
2
Consider now our revised model perturbation (w.l.o.g. remove the constant factor),
Σ Tr(((()))
Omax
Omin
2
24
Each bit configuration {bi}i yields unique FIT value from which we can estimate final perfor- mance.
G FIT DETAILS
G.1 STANDARD RELATIONS IN INFORMATION GEOMETRY
The EF defines an estimate for the natural metric tensor of the statistical manifold induced by the parameterised model, obtained infinitesimalyy (in this case) from an expansion of the KL divergence.
Proposition
Proof. For brevity, denote 0) pa. Here describe the discrete case, however, the continuous follows similarly.
DxL (pol (po))p)458))
pa log
4-0
V YoPa"
1
Vipa
Vopa VoP="
Palog Pe
Pe
log Pa
50
60
80
pa
2
Pe
p3
2
V OPa
50
60
V 8D0
80 +
50
Σ
Vo log po Vo log T
80
2
2
T
2
V
80
T0
V?
80 + 60
Σ V log Pe V log
ps 80
T
2
2
D
D
2
50
Pe Vo log po Vo log
80
2
Vo log Vo log pa
50
N
D
= 50
f(0)50
19

---

Published as a conference paper at ICLR 2023
G.2 OBTAINING FIT
Proposition 2. FIT is obtained (under mild noise assumptions) by laking expectation over the Fisher-Rao metric.
Proof.
Ω [80 [(0)80]
= [[50]">(0))0) ((0)B[(0 + Tr(1(0)Coo(00)
We assume that the random noise is symmetrically distributed around mean of zero: E[60) = 0, and uncorrelated: Cov(60] Diag( [ 502)). This yields the FIT heuristic:
Tr (iag(())
And in block-wise empirical form, with an approximate noise model:
N Tr(i(O))) (8
In cases where we compute a numerical approximation of our noise model: m() ||50 rather than averaging over many trained models, we leverage the high dimensionality of the parameters in each layer to obtain good approximation.
G.3 CONNECTIONS TO THE HESSIAN
Proposition 3. The Fisher Information Matrix is equal to the expectation of the Hessian under the correctly specified model.
Proof.
Vap(ylz, 0)
Veplylæ 0) Vep(y|z, 0)
Epa. (x) [2]
Ey (z.2)
+ Ep (sra))
(1)
p(y|ar 0)
p(y|x,0) 0)2
Taking the first term of the right-hand side (RHS). We assume this case the function sufficiently smooth such that the order of integration and (second) differentiation be exchanged. In this case, the first term on the RHS evaluates to O:
Vap(yla 0)
Vap(ylz, 0)
Epp- (z.v)
plyl,, 0) p(y|2, 0)dz
(2)
p(yl,r,0
Vip(viz,00 0)
(3)
v3 plyl,, 0)
(4)
[ -
(5)
(6)
The second term on the RHS then simplified as follows, yielding our result:
Epo- (z) [2ff(,,0)
Ep (2.z)) /o lug.lu/ Logglgy 0)"] ((0)
(
20

---

Published as a conference paper at ICLR 2023
Proposition 4. Provided 0n is consistent estimator for the true model parameters 8", the Empirical Fisher Information will converge in probability to the Fisher Information as n -+ dc.
Proof.
i(,m)) - I(00) = - -[,) -
(8)
Considering the first term on the RHS, we are able to apply the uniform law of large numbers,
1 N
(((0,) 1(0,))
v Vf(z,,
(9)
sup 938
Vz (V)(2,))))))
-Epo 1ep
(10)
Py" 0,
(11)
provided the following regularity conditions are upheld:
1. @ must be compact
2. [(z 0) continuous at each almost all E
3. The aforementioned dominating function must be finite
Now considering the second term on the RHS. Our estimator in this case is defined consistent, and therefore 0, Pon 0"
Via the continuous mapping theorem, (T(0,) T(0*)] Pe" 0. We therefore arrive at the desired result.
G.4 COMPUTATION
Proposition 5. The trace of the EF can be computed via the sum of the square of the gradient.
Proof. The trace is linear operator, which allows each individual estimator to be extracted from the summation. Additionally, the trace of the second moment matrix can then be computed by the norm of the vector of parameters from which is composed:
N
Tr[[(0) =Ir
v V2,,, (),
N
1=1
1
Tr Vf 8) V f(2,
N
1
Vf(2,
N
N
Proposition 6. The variance of the Hutchinson estimator is given by: W/F Hr.
2 (1 Σ. Hz)
21

---

Published as a conference paper at ICLR 2023
Proof. Assuming follows Rademacher distribution, the first four moments follows: (0, 0,). As such, we can compute the variance of the quadratic form:
V(r Hr:] = (m4 - -3m²) Σ H + m² (Tr(H)² +2- Tr(H²)) - -Err Hr.
=2 (117712 Σ
- #
H ENVIRONMENTAL ANALYSIS
During the course of this research, we estimate our total emissions to be 10.8 kgCOzeq. ((100) hours of computation was performed which has a carbon efficiency of 0.432 kgCOpeq/kWh. Estimations were conducted using the MachineLearning Impact calculator presented by Lacoste al. (2019).
RTX 2080 Ti (TDP of 250W), using private infrastructure
22


Reviews:
[{"review_id": "xfTRXdcIMT", "reviewer": "Reviewer_orGP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use Fisher Information Trace (FIT) to perform mixed quantization of deep neural networks. The author of the paper use FIT to measure sensitivity of parameters and activations regarding quantization, and shows improvement of the performance.", "summary_of_the_review": "To sum up, the clarity and quality of this paper need to be improved. The author of the paper did some interesting works on model quantization but fails to demonstrate them with thorough experiments. Please refer to strengths and weaknesses for more information.\n\nI think this paper needs to be revised, both on the technical contribution and experiments. ", "strength_and_weaknesses": "Strengths\n\n1. This paper demonstrate clear advantage of FIT to Hessian-based method.\n\n2. The writing of the paper is good.\n\n\nWeaknesses\n\n1. There are no ImageNet results of the FIT quantization, which is the major weakness of this paper. The author mentioned ImageNet at the beginning of section 4.1, which uses ImageNet to analyze the proposed method, however, no final accuracy results are give,. Only CIFAR-10 and MNIST.\n\n2. The experimental setting and demonstration is unclear. What is the bit width setting for each of the accuracy result? \n\n3. Many baselines regarding mixed precision quantization are not compared. For example: “Mix and match: A novel fpga-centric deep neural network quantization framework” (HPCA 2021), “RMSMP: A Novel Deep Neural Network Quantization Framework with Row-wise Mixed Schemes and Multiple Precisions” (CVPR 2021), and more.\n  \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mXPpbDO0jmW", "reviewer": "Reviewer_qJTT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new analytical metric to quantify the impact of quantization for deep neural networks. The proposed metric provides an estimation of quantization impact as accurately as hessian methods yet it is more computationally efficient and its variance is more favorable. The authors compared the proposed method with the other metrics and demonstrate its superior performance.", "summary_of_the_review": "This paper proposed a practical metric, FIT, for quantitatively analyzing the impact of quantization errors in deep neural networks. FIT's robustness (due to small variance) and computational efficiency would be very useful for the practitioners in this study. However, currently, the evaluation results are very weak; it would be highly appreciated if the authors could include more experimental results, such as ResNet50 on ImageNet.\n\n\n====== Post rebuttal comments ======\nThe reviewer thanks the authors for their careful rebuttal. Demonstrating a solid accuracy boost on BERT models seems to make the paper much stronger, and I believe that such a demonstration confirms the applicability of the proposed method for researchers in this field to broader applications. Thus, I increase my score to Accept. ", "strength_and_weaknesses": "(Strength)\n- A nice overview and derivation of the impact of quantization on the loss. \n\n- The nice properties of the proposed metric are that it is more robust (since it shows a smaller variance) and more efficient (in terms of computation)\n\n- The comparison of correlations that show superior predictability of the proposed methods for the impact of quantization loss.\n\n(Weaknesses)\n- The evaluation in Table 2 seems to be very limited. By exploiting the computational efficiency of the proposed algorithm, the authors should also try challenging tasks, such as a large CNN for ImageNet classification or a large language model like BERT.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XIdNHzhpMB", "reviewer": "Reviewer_QyLp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper finds a more efficient alternative than the Hessian Trace metric used in the mixed-precision HAWQ paper, named FIT, which is a further simplification of said metric based on the empirical fisher. They show the metric to be equivalent in many scenarios compared to the Hessian trace metric. ", "summary_of_the_review": "The paper is extremely well-written, and great in terms of clarity and exhaustiveness on the literature regarding Hessian-like estimates for deep learning. \nThe method of the paper has been published before, in the context of quantization. The results do convince me that this metric works just as well as the HAWQ metric; They also convince me that they are better than the other metrics the authors found in the literature. However, the authors have not convinced me that these metrics are useful in mixed-precision, that the resulting algorithm actually does something that is non-trivial, and that this is a great way to do mixed precision in general.\n\nI would like to see the following things:\n* A proper set of mixed precision results on more than 1 network, not only the semantic segmentation one. The authors included a few more networks for their comparative analysis, but then do not show actual mixed precision results on these nets. This should include showing that the loss approximation is good enough to find non-trivial mixed-precision results\n* A comparison to a very simple baseline - Just quantize each layer individually with your intended bit-width, and measure the output loss on a bit of data (few batches e.g.); Your metric is gauranteed to be a less accurate metric for mixed-precision by construction. How well does it perform in practice, and why would your metric be expected to perform better? \n* A more thorough experimental setup description. There's a lot of choices to be made in quantization, and these choices matter. \n\nIf these are addressed in a satisfactory way, I am very willing to increase my score. ", "strength_and_weaknesses": "Strengths:\n* It is very clear the authors know what they are talking about, as they mentioned all the relevant literature, and understand the underlying methods/ideas very well. \n* An excellent overview of the literature, and the derivations leading to the final FIM objective. This is a great overview for anyone to read that wants to dive into this topic. \n\nWeaknesses\n* The metric itself has been introduced before, for both compression and quantization. I have seen it several times in the past. The first mention of it I know in the setting of quantization and compression is from this paper: https://arxiv.org/pdf/1810.06401.pdf - where it is derived from an information theoretical context. There is no novelty in the paper, since it is a small tweak to HAWQ, and the tweak itself already existed.\n* I think the paper is lacking in terms of experimental results. It is good the authors included the accuracy comparison w.r.t. the target loss for this method and 'competitive' methods. However, these numbers don't mean much for the performance in the mixed-precision setting. For the correlations in Table 2, I wonder if those are high enough to do a proper mixed-precision setting that is non-trivial. For these networks, how well does the mixed precision method rank the networks, and how well can it find a non-trivial solution? Why is this analysis that was done for semantic segmentation also not done for these other models? I don't like hammering on more results, but since the metric itself is not novel, the onus is on the authors to really show that in this setting this metric works well and is the best we can do. The paper turns into a survey paper, rather than a novel method paper. \n* I also wonder about the sense of these metrics in general, and if there is no easier solution. I think this is related to a key underlying assumption at the start of the paper: \"The exact (per parameter) perturbations δθ associated with quantization are often unknown.\"\nThis is a weird statement. If you 8-bit quantize a layer’s weights, with a given min-max parameter setting, you know exactly what δθ is. There is nothing inherently stochastic about it. Instead of making this stochastic assumption, you could just quantize each layer to the intended potential bit-widths, and measure this on your validation set. Here is your loss function approximation, no surrogate measure needed. This also holds for the activations, where there is arguable a distribution over your data (but not over noise on your model, as the formulation now has it). Please compare what your metric would improve over a simple eval step on a batch of data for per-layer/per-quantizer bit-width settings. In practice, one can generally not set intermediate bit-widths anyway, and generally you'd pick 4, 8, 16 or something, which will likely be almost as efficient as this method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6bAo4xnF1B", "reviewer": "Reviewer_cnF4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Model compression via quantization of weights and activations helps improve the efficiency and reduce memory requirements of DNNs.\nHowever, quantifying the impact of quantization across the different layers in a DNN is challenging. For this purpose, the proposed work establishes FIT as a measure of DNN performance, when using multi-precision quantization (MPQ), without the need to retrain the network.\nFIT is a metric that combines the impact of quantization on weights as well as activations in a unified manner such that the computation of the metric itself is quick and readily usable. \nThus, FIT can be used to understand and readily gauge the impact of MPQ and avoid the pitfalls of bad performance.", "summary_of_the_review": "The proposed work establishes the need for a metric to analyze the sensitivity of networks to quantization without having to retrain the network. For this purpose Fisher Information Trace is selected as the measure of choice and its estimation process is defined in the proposed work. \nOverall, the evaluation of FIT against the hessian, proving its equivalence for the task at hand, and subsequent evaluation across various application domains helps validate the claims made about FIT.", "strength_and_weaknesses": "Strengths\n- The proposed work provides a clear and concise explanation of the problem domain being tackled. This guides the reader from the general impact of the proposed work towards the specifics of the problem being handled.\n- The evaluation schemes provided for the estimator as well as the application on various networks provide a good foundation.\n\nWeaknesses\n- Could the authors provide citations for the statement: \"different layers, within different architectures, respond differently to quantization\". (Pg. 1, Introduction, Paragraph 4, Line 1)\n- The proposal of rank-correlation as a novel evaluation criterion is mentioned as a core contribution towards the end of the introduction. Could the authors describe possible evaluation schemes to compare against prior works, as baselines to judge the proposed scheme. Currently, rank-correlation exists as a readily available tool that has been repurposed for evaluating the task at hand. Multiple baselines could help highlight the contribution.\n- Could the authors provide citations for the statement: \" which is insufficient for prediction, and more challenging for practitioners to implement. In addition, trace computation can become very expensive for large networks\". (Pg. 3, Paragraph 1, Lines 9-11).\n- While the evaluation of the EF estimator in Table 1 shows a relatively model-agnostic behavior, it would be instructive to include evaluations with alternative DNN architectures that branch away from bottleneck-like structures (more densely packed connections) and alternative datasets, to help firmly establish the agnostic behavior observed.\n- Could the authors normalize the X and Y scales across both plots in Figure 2? The visual comparison would provide a more compelling evidence in that regard.\n- Could the authors discuss the type of information provided by FIT_A and FIT_W and try to provide more justification as to how they provide complementary information?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "4Mh4x9zfSf", "reviewer": "Reviewer_uWSp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The presented work proposes a method for estimating model performance and mixed precision configuration without network retraining. The introduced method is based on the Fisher information, what allows for faster computations compared to other existing methods based on e.g., Hessians. ", "summary_of_the_review": "This is a good quality interesting paper with detailed experiments and clear conclusions. ", "strength_and_weaknesses": "The work is clear and well structured. It also flows logically, with a detailed background, good comparison with the previous work, clearly explained FIM metric and experiments that include in-depth comparison with Hessian and analysis of model sensitivity based on the proposed criterion. The method is evaluated on a broad set of models and datasets to evaluate its performance and usability for different tasks. \n\nIn general, I don't see any major omissions. Some sections that could be improved to further improve the quality of submissions are:\n1) Could you elaborate more on the ideas of determining MPQ configurations from initialization instead of the trained models?\n2) Could you provide more details. about the applied quantization scheme? Symmetric, asymmetric, per tensor, per channel? Have you performed experiments with other quantization schemes?\n3) The abstract mentioned that hundreds of quantization configurations were used, but there was no clear summary of used configurations in the description, could you please explain it in more detail?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster