Title: MONTE CARLO DEEP NEURAL NETWORK ARITH-METIC

Abstract: Quantization is a crucial technique for achieving low-power, low latency and high throughput hardware implementations of Deep Neural Networks. Quantized floating point representations have received recent interest due to their hardware efficiency benefits and ability to represent a higher dynamic range than fixed point representations, leading to improvements in accuracy. We present a novel technique, Monte Carlo Deep Neural Network Arithmetic (MCDA), for determining the sensitivity of Deep Neural Networks to quantization in floating point arithmetic. We do this by applying Monte Carlo Arithmetic to the inference computation and analyzing the relative standard deviation of the neural network loss. The method makes no assumptions regarding the underlying parameter distributions. We evaluate our method on pre-trained image classification models on the CIFAR-10 and ImageNet datasets. For the same network topology and dataset, we demonstrate the ability to gain the equivalent of bits of precision by simply choosing weight parameter sets which demonstrate a lower loss of significance from the Monte Carlo trials. Additionally, we can apply MCDA to compare the sensitivity of different network topologies to quantization effects. 1 

1. INTRODUCTION:

Deep Neural Networks have achieved state-of-the-art performances in many machine learning tasks such as such as speech recognition (Collobert et al., 2011) , machine translation (Bahdanau et al., 2014) , object detection (Ren et al., 2015) and image classification (Krizhevsky et al., 2012) . However, excellent performance comes at the cost of significantly high computational and memory complexity, typically requiring teraops of computation during inference and Gigabytes of storage. To overcome these complexities, compression methods have been utilized, aiming to exploit the inherent resilience of DNNs to noise. These engender representations which maintain algorithm performance but significantly improve latency, throughput and power consumption of hardware implementations. In particular, exploiting reduced numerical precision for data representations through quantization has been emphatically promising, whereby on customizable hardware, efficiency scales quadratically with each bit of precision.

Quantization of fixed-point arithmetic (Q-FX) for DNN inference has been extensively studied, and more recently there has been increasing interest in quantized floating point (Q-FP) arithmetic for both DNN inference and training (Wang et al., 2018) . Q-FP has the advantage of higher dynamic range compared to equivalent Q-FX representations and reduced hardware cost over single-precision floating point (FP). This has influenced application specific integrated circuits (ASICs) such as Google's tensor processing unit (TPU), which supports 16-bit floating point (16-FP) and soft processors such as Microsoft's Project Brainwave which utilizes 8-FP.

To illustrate these hardware benefits, we synthesized arithmetic logic units (ALUs) in different formats and different precision on an FPGA and present performance estimates in operations per second (OPs) and area estimates in Look-up Tables (LUTs) per operation (LUTs/Op) in Figure 1 . As shown, 8-bit fixed point (8-FX) achieves improved performance and area over 8-FP. However, 7-FP is a significant improvement over 8 or 12-FX and 8 or 9-FP. These examples demonstrate substantial performance and area benefits from reducing FP precision by only 1 to 2-bits. Thus, if we Figure 1 : The estimated multiply-accumulate performance on a Xilinx VU9P FPGA, operating at 500MHz, in teraops (left) and area (right) for fixed (FX) and floating-point (FP) in various precisions (both weights and activations). can design networks which not only achieve high accuracy but are robust to quantization, higher performing hardware solutions are possible.

Since in Q-FP, we are trying to represent the infinite set of real numbers using a finite number of bits, quantization and rounding artefacts will be introduced, with inaccuracies being cascaded along the computation graph (IEEE, 1985; Goldberg, 1991; Higham, 2002) . This paper proposes Monte Carlo Deep Neural Network Arithmetic (MCDA), a novel way to apply Monte Carlo Arithmetic (MCA) (Parker et al., 2000) for determining the sensitivity of Deep Neural Networks to Q-FP representations. It allows hardware-software designers to quantify the impact of quantization, enabling more efficient systems to be discovered. We do this by exploiting Monte Carlo simulations under which rounding effects are randomized. This, in turn, allows one to infer the sensitivity of executing a computation graph to quantization effects.

Our MCDA technique is highly sensitive, allowing very small differences in quantization behaviour to be detected. The technique makes no assumptions regarding data distributions and directly measures the effects of quantization on the problem under study. This allows us to provide insights into the precision requirements of any inference network for any given dataset. Additionally we can use the technique to select weight parameters which are more robust to floating point rounding. The theoretical and practical contributions of this work can be summarized as follows:

• We introduce a novel and rigorous analysis technique, Monte Carlo Deep Neural Network Arithmetic (MCDA), which measures the sensitivity of Deep Neural Network inference computation to floating point rounding error. • When applied to neural network inference, we show MCDA can determine the precision requirements of different networks, rank them, and detect small differences between different neural network topologies and weight sets. • We demonstrate that while a network with the same topology but different weights may have the same loss and validation accuracy, their sensitivity to quantization can be vastly different. Using the CIFAR-10 and ImageNet datasets, we introduce a method to choose weights which are more robust to rounding error, resulting in a greatly improved accuracyarea tradeoff over state-of-the-art methods.

It is worth noting that although we consider convolutional neural networks for image classification, this method could be applied for any neural network model architectures and applications. Moreover, while the experiments in this paper are limited to inference, it may be possible to apply the same idea to analyze training algorithms.

2. RELATED WORK:

Low-precision representations of deep learning have been extensively studied. Many training methods have been developed to design representations for fixed point inference (Jacob et al., 2018; Faraone et al., 2018; Zhou et al., 2016) and training (Wu et al., 2018b; Yang et al., 2019; Gupta et al., 2015; Sakr & Shanbhag, 2019) . Other methods have also utilized Q-FP arithmetic for inference and training whilst maintaining single-precision accuracy. (Micikevicius et al., 2018) implemented 16-FP arithmetic training whilst storing a 32-FP master copy for the weight updates. Additionally, (Wang et al., 2018; Mellempudi et al., 2019) with 8-FP arithmetic whilst using a 16-FP copy of the weights and 16/32-bits for the accumulator. Techniques for determining per-layer sensitivity to quantization have also been studied (Choi et al., 2016; Sakr & Shanbhag, 2018) . Further, other studies have successfully determined the minimum fixed point precision requirements for a given DNN accuracy threshold (Sakr et al., 2017) . The accuracy and stability of various numerical algorithms in finite precision arithmetic has been studied in (Higham, 2002; Wilkinson, 1994) . This has led to techniques for tracking information lost from finite precision arithmetic using random perturbation such as Monte Carlo Arithmetic (Parker et al., 2000; Frechtling & Leong, 2015) . Monte Carlo methods have also been used in Bayesian Neural Networks (Buchholz et al., 2018; Blier & Ollivier, 2018) . In particular, (Achterhold et al., 2018) introduced a quantizing prior to learn weights which are either close to a quantized representation or have high variance. (Louizos et al., 2017) used hierarchical priors to prune nodes and posterior uncertainties to determine the optimal fixed point precision. (Blundell et al., 2015) use a Monte Carlo approach to learn a probability distribution on the weights of a neural network. To the best of our knowledge, our work is the first to present a technique for directly determining the sensitivity of DNNs to floating point rounding and to explicitly compute precision bounds of a trained network. These ideas can be very usefully applied to extending the limits of low-precision representations in deep learning applications.

3. BACKGROUND:

In this section, we describe background theory upon which our technique for determining the sensitivity of DNNs to floating point rounding is based.

3.1. FLOATING POINT ARITHMETIC:

The IEEE-754 binary floating point format (IEEE, 1985) represents most real numbers x by a subset in normal form as:

x = (-1) sx (1 + m x )2 ex (1)

where s x ∈ {0, 1} is the sign bit, e x is an integer representing the exponent of x and m x is the mantissa of x. Such number formats can be described as a (s x , e x , m x ) tuple. In binary form the representation is

(b s , b e 1 , b e 2 , ..., b e Be x , b m 1 , b m 2 ..., b m Bm x ) ∈ {0, 1} B

, with B ex and B mx being the number of exponent and mantissa bits, respectively. The infinite set of real numbers R is represented in a computer with B = 1+B ex +B mx bits, and we define the finite set of real numbers representable in floating point format as exact values, F ⊂ R. Real numbers which aren't representable are rounded to their nearest exact value. We call this set of numbers inexact values, I, where I ∪ F = R.

The approximation x = F(x) = x(1 + δ), given x ∈ I, introduces rounding error into the computation. The value of δ = x-x x , represents the relative error which is a function of the machine hardware precision, p, as δ ≤ , where = 2 -p (IEEE, 1985; Goldberg, 1991; Higham, 2002) .

In general, inexactness can be caused by finite representations or errors propagating from earlier parts of the computation.

Often the primary cause of error in floating point arithmetic is catastrophic cancellation which causes numerical inaccuracy.

Catastrophic cancellation occurs when for example, two near equal FP numbers, sharing k significant digits, are subtracted from one another as shown in (2) (Higham, 2002) .

0. f 1 f 2 ... f k f (k+1) ... f t -0. f 1 f 2 ... f k g (k+1) ... g t = 0. 0 0 .... 0 h (k+1) ... h t (2) 0. f 1 f 2 ... f k f (k+1) ... f t r (t+1) ... r p -0. f 1 f 2 ... f k g (k+1) ... g t r(t+1) ... rp = 0. 0 0 .... 0 h (k+1) ... h t i (t+1) ... i p (3)

In normalized form, the leading zeros are removed by shifting the result to the left and adjusting the exponent accordingly. The result is 0.h k+1 ...h t i 1 ...i k which has only (t -k) accurate digits and digits i which are unknown. Additionally, the remaining accurate digits h are most likely affected by rounding error in previous computations. This can significantly magnify errors, especially in computing large computational graphs such as that of state-of-the-art DNNs.

If either operand in (2) is inexact, then the digits h are no more significant than any other sequence of digits. Yet, FP arithmetic has no mechanism of recording this loss of significance. By padding both our operands with random digits r and r in (3), the resulting digits i are randomized. If k digits are lost in the result, then k random digits will be in the normalized result and when computed over many random trials, the results will disagree on the trailing k digits. In this case, we are able to detect catastrophic cancellation because the randomization over many trials provides a statistical simulation of round-off errors. We can use techniques from numerical analysis such as Monte Carlo methods to appropriately insert precision-dependant randomization in this way.

3.2. MONTE CARLO ARITHMETIC:

Monte Carlo methods can be used to analyze rounding by representing inexact values as random variables (Parker et al., 2000; Frechtling & Leong, 2015) . The real value x, as represented in (1), can be modelled to t digits, using:

inexact(x, t, δ) = x + 2 ex-t δ = (-1) sx (1 + m x + 2 -t δ)2 ex (4)

where δ ∈ U (-1 2 , 1 2 ) is a uniformly distributed random variable and t is a positive integer representing the virtual precision of concern. For the same input x in (4), we can run many Monte Carlo trials which will yield different values on each trial, where 0 < t < p so that the MCA can be run accurately on a computer with machine precision p. The ability to vary t is useful because it allows us to then evaluate the hardware precision requirements of a given system or computational graph for a given DNN.

MCA is a method to model the effect of rounding on a computational graph by randomizing all arithmetic operations. The randomization is applied for both generating inexact operands and also in rounding. In each operation using MCA, ideally both catastrophic cancellation and rounding error can be detected. An operation using MCA is defined as:

EQUATION

where • ∈ +, -, ×, ÷). By applying the inexact function to both operators we make it possible to detect catastrophic cancellation. Furthermore, applying the inexact function to the operation output and then rounding this value implements random rounding and hence is used to detect rounding error (Parker & Langley, 1997) . Hence, for the same input into the system, each trial will yield different operands and output.

After modifying the inexact and rounding operations as described, we use random sampling to simulate Monte Carlo trials. For each trial, we collect data on the resulting output of the system and compute summary statistics to quantify its behaviour (Parker et al., 2000) . With sufficiently large number of Monte Carlo trials and virtual precision t, the expected value of the output from these trials will equal the value from using real arithmetic. As explained in the next section, we can determine the total number of digits lost to rounding error and the minimum precision required to avoid a total loss of significance.

3.3. ANALYSIS:

The relative error is bounded by δ ≤ 2 -p from the design of IEEE FP arithmetic (Wilkinson, 1994; Goldberg, 1991) . With this inequality, we can determine the expected number of significant binary digits available from a p-digit FP system as p ≤ -log 2 (δ). These definitions can be adapted for MCA by replacing the precision of the FP system, p, by the virtual precision, t, of an MCA operation. Thus, the relative error of an MCA operation, for virtual precision t, is δ ≤ 2 -t and the expected number of significant binary digits in a t-digit MCA operation is at most t. Using this definition and the proof provided in (Parker & Langley, 1997), the total significant binary digits in a set of M trials is s = log 2 µ σ where µ is the mean and σ the standard deviation. The output of the system should be some scalar value so that we can perform such analysis. For experimentation, M trials are run for all of t ∈ {1, 2, 3, ..., t max }. The total number of base-2 significant digits lost in a set of M trials is K t , in (6):

EQUATION

where Θ = σ µ , for (µ = 0), is the relative standard deviation (RSD) of the MCA results. The virtual precision t controls the perturbation strength applied by the inexact function. For a given K t , as we reduce t, the RSD should increase according to equation 6. At some point, an unexpected loss of significance (Frechtling & Leong, 2015) is encountered due to the nonlinear effects of quantization. The value at which this occurs is defined as t min . The number of significant digits lost for the system being analyzed is then computed by averaging all K t whereby t > t min as shown in (7):

K = 1 tmax-tmin tmax t=tmin K t where t max > 1 K t

where t max = 1 (7)

For DNN inference, we propose to use K as a sensitivity measure for the network to FP rounding. The method for implementing this is discussed in the next section.

4. MONTE CARLO DEEP NEURAL NETWORK ARITHMETIC:

We now describe MCDA, a methodology for applying MCA techniques to DNN computation, allowing us to understand the sensitivity of a given network and its weight representation to FP rounding.

4.1. NETWORK MODEL:

We consider a generalized non-linear L-layer neural network with an output vector y L , input data vector x and learnable weight parameter tensor w = w l (l = 1, . . . , L), whereby y L = f (x; w).

To compute y, several layers consisting of general matrix multiplication (GEMM) operations (such as convolutional and fully-connected layers) between the layer input x l and weight parameters w l , followed by a non-linear activation function, h, producing intermediate layer outputs x l , i.e. y l = h(x l ⊗ w l ). The output of a given layer becomes the input to the subsequent layer, i.e. x l+1 = y l , with x 1 = x. A loss function is the objective function to minimize updating w via an optimizer such as stochastic gradient descent. For a given set of input data X, the total network loss during inference is calculated by applying a loss function loss(f (x; w), ŷ(x)) where ŷ(x) is the target ground truth output for x. The total loss for X is then a scalar output, such that:

EQUATION

Updates of w are usually done in small batches over subsets of X.

Naively applying MCA to each operation (fine-grained MCA) as described in equation ( 5) poses significant computational difficulties for DNN models. We observe two primary issues with employing fine-grained MCA to a DNN computational graph:

• Firstly, the number of required trials for Monte Carlo experiments to generate robust results can typically be in the hundreds or thousands. As DNN inference of state-of-the-art networks typically consist of billions of operations, the computational requirements of applying MCA after each operation will be very large, making the technique impractical. • Using the accuracy as the system output for MCA experiments is problematic because it is a discrete value. For high values of t, Monte Carlo results across different t then become indistinguishable and the standard deviation for a given t is potentially zero.

4.2. MONTE CARLO NETWORK INFERENCE:

To reduce the computational cost of Monte Carlo experiments, we employ MCDA, which is a coarsegrained approach to MCA for GEMM operations. Conveniently, these can be naturally implemented in modern machine learning frameworks such as PyTorch. Furthermore, to ensure the system output is a continuous value, the loss function output is used, rather than the accuracy. In this case, small perturbations in layer operands are more likely to produce observable changes in the output.

MCDA applies a vector version of (5) to the DNN inference computational problem in (8). Our operands in this case are vectors and • represents a neural network layer operation. For example, the output from performing a GEMM operation can thus be represented by:

y l = round(inexact(inexact(x l ) ⊗ inexact(w l ))) (9)

Since the inexact function is applied to the inputs and outputs of a GEMM operation, an optimized implementation can be used. This is in contrast to full MCA which requires the application of ( 5)

Figure 2 : Applying MCDA to residual blocks found in ResNet. The inexact function is applied to each input operand and both the inexact function and base-2 rounding is applied to the output.

to every individual scalar operation. The vector form in ( 9) is applied to each edge of neural network computational graph where multiply (division) and/or add (subtract) operations are performed. Hence, it is not applied to operations such as MaxPool and ReLu. As an example, in Figure 2 we show where the inexact function is applied for a residual block with folded batch normalization, which is a repeating sequence of layers found in ResNet models (He et al., 2015) . At the final output of the network, the loss is computed with ( 8) and the inexact function is applied to the outputs y and also the loss output scalar value L. From analyzing the behavior of the loss, we infer the sensitivity of the accuracy of the system to FP rounding. By using MCDA for the GEMM operations, we will not be able to detect all instances of catastrophic cancellation. However, we significantly reduce execution time over fine-grained MCA and show in the next section that we can still retrieve valuable information about our system. In fact, for one trial with one batch of 32 images on ImageNet running on a Nvidia Titan Xp GPU, the speed up of regular inference without MCDA is only 1.05× (for a single Monte Carlo trial). We also note that fine-grained MCA could be applied with a simple modification and would be possible given appropriate customized hardware support for parallel Monte Carlo computations (Yeung et al., 2011) .

5. EXPERIMENTAL RESULTS:

In this section, we present experimental results for applying MCDA to exemplary convolutional neural networks. We use the CIFAR-10 and ImageNet image classification datasets to compare MobileNet-v2 (Sandler et al., 2018) , EfficientNet (Tan & Le, 2019) , AlexNet, ResNet (He et al., 2015) , SqueezeNet (Iandola et al., 2016) and MnasNet (Tan et al., 2018) . For CIFAR-10, we use a batch size of 128, whilst we use a batch size of 32 for ImageNet experiments. Cross-entropy is used as the loss function for both datasets. For a given network, dataset, weight representation and t, we apply MCDA with M = 1000 trials. The resulting loss from each trial is computed with the same single batch of images and hence additional data is not required. We compute Θ t from our results for all t ∈ {1, 2, 3, ..., 16}. Following this, we run linear regression analysis using the MCALIB2 (Frechtling & Leong, 2015) R library, on our Θ t values, to determine t min and K. t min is defined as the point of lowest t where the difference between the regression line and the equivalent Θ t is less than half a binary digit, i.e. log 10 (2 0.5 ). Further detail on the calculation of K and t min from MCALIB can be found in Appendix A.1. We report Q-FP validation accuracy using the quantization function from (Wang et al., 2018) with stochastic rounding3 (See Appendix A.2).

5.1. DISTINGUISHING WEIGHT PARAMETER REPRESENTATIONS:

As discussed in Section 3.1, the inexactness in FP arithmetic largely depends on the numerical value of operands. Two instances of the same network and dataset, with the same validation accuracy, but vastly different weight representations, will likely produce differing sensitivities to FP rounding. We first train 8 instances of EfficientNet-b0 and MobileNet-V2 on CIFAR-10 from scratch with random initialization from (Glorot & Bengio, 2010) , all achieving within 1% validation accuracy of one another. Using MCDA, we calculate the K values for each model (See Appendix A.3). We then test their percentage validation accuracy decrease from using post-training quantization (i.e. no finetuning) with varying Q-FP precisions. In Figure 3 , we see that the models with higher K values typically experience a larger drop in Q-FP accuracy, indicating they are more sensitive to floating point rounding error. Notably, the model with lowest K for 7-bit MobileNet-v2 experiences a lower percentage validation accuracy drop than three of the 8-bit models. In this case, MCDA model selection enables the saving of a bit of precision while achieving smaller accuracy decrease than some of the trained 8-FP models.

5.2. COMPARISON TO PREVIOUS WORK:

One practical use case from the insights gained by MCDA is model selection for quantization. Typically when quantizing a given model trained on a given dataset, the model with highest validation accuracy is chosen and the sensitivity to quantization is assumed to be the same across models. As discussed, for Q-FP representations this is not necessarily the case. We can use K from MCDA to predict which models will be more robust to quantization. To demonstrate this, in Table 1 we compare post-training quantization results for model selection based on K from MCDA, against a baseline model chosen based on the highest single-precision validation accuracy. Evidently, even though the single-precision accuracy is initially as much as 0.9% higher, after quantizing the network to 8-5 bits, the accuracy of the network chosen by smallest K is always significantly higher.

5.3. NETWORK COMPARISON:

Modern DNNs consist of convolutional blocks with highly varying computational graphs (Wu et al., 2018a; Howard et al., 2017) . Using MCDA we can also compute and compare their sensitivites to floating point rounding error to determine which networks will be robust to Q-FP representations. In Figure 4 we show the Θ t of pre-trained models, trained on the ImageNet dataset from PyTorch4 5 , for differing values of t and run linear regression analysis over our data points. From here we can then assign a K value to each network and compare their loss of significance. At each t, the distance from the regression lines to the ideal line represents the values of K t , as described in (6). From the MCDA results, AlexNet is the least sensitive to rounding and ResNet-50 is the most, with various models in between these two. Additionally we compare EfficientNet at two different model scales and evidently the larger model has much larger sensitivity. We then also compare the validation accuracy percentage decrease of all models at 10, 9 and 8-FP post-training in Figure 5 . At 8-FP, besides MnasNet which experiences a large accuracy drop, K is able to predict validation accuracy degradation. Thus, MCDA provides very valuable information about Q-FP model design.

6. CONCLUSION:

We present a novel, highly sensitive, technique to quantify rounding error in DNNs. This is the first method to successfully compare the sensitivity of networks to floating point rounding error. Ultimately, this technique provides a tool for enabling the design of networks which perform higher when quantized. We do this by applying concepts from Monte Carlo Arithmetic theory to DNN computation. Furthermore, we show that by calculating the loss of significance metric K from MCDA, on the CIFAR-10 and ImageNet datasets, we can compare network sensitivities to floating point rounding error and gain valuable insights to potentially design better neural networks. This is an important contribution due to the increasing interest in low-precision floating point arithmetic for efficient DNN hardware systems. The theoretical and practical contributions of this paper will likely translate well to analyzing floating point rounding in backpropagation in future work.

A APPENDIX:

A.1 EXPERIMENTAL SETUP In all our experiments, for a given network, dataset, weight representation and t, we run 1000 Monte Carlo trials. The network loss output from each trial is computed with the same single batch of images from the training dataset. We then independently compute the Θ t of the network loss for t ∈ {1, 2, 3, ..., t max } where t max = 16.

Following this, we then run linear regression and calculate K and t min using MCALIB6 . To compute the linear regression, MCALIB uses a log transformed variable, with log(Θ) as the dependant variable and t as the exploratory variable (Frechtling & Leong, 2015) .

log 10 (Θ) = log 10 (2 K-t ) (10) = -log 10 (2)t + log 10 (2)K (11) = mt + c (12) Assume that m x is in fixed precision with z bits which needs to be rounded to z bits, then stochastic rounding is as follows:

Round(x) = (-1) sx (1 + m x + )2 ex with probability mx-mx (-1) sx (1 + m x )2 ex with probability 1 -mx-mx (15)

where m x is the truncation of z -z LSBs of m, = 2 -z . For each Q-FP accuracy reported in our experiments, we tested all possible combinations of the number of bits allowed for the exponent and mantissa which satisfied the desired precision. We then chose the combination which produced the highest accuracy.

A.3 CIFAR-10 REGRESSION ANALYSIS

In Figure 6 , we display the linear regression analysis using MCALIB for the 8 MobileNet-v2 and EfficientNet-b0 models trained on CIFAR-10. As shown, Monte Carlo trials were run for each of these models for t ∈ {1, 2, 3, ..., 16}. The corresponding t min and K values were then calculated using methods discussed in Appendix A.1. Thus, in Figure 3 , it is these K values which are plotted against 8 and 7-bit Q-FP validation accuracy. Also, the models with lowest K values are chosen for MCDA model selection in Table 1 .

:

Source code will be available if the paper is accepted

https://github.com/mfrechtling/mcalib

https://github.com/Tiiiger/QPyTorch

https://github.com/pytorch/vision/tree/master/torchvision

https://github.com/rwightman/pytorch-image-models

https://github.com/mfrechtling/mcalib

https://github.com/Tiiiger/QPyTorch

Figures and Tables:

Figure 3: CIFAR-10 percentage validation accuracy decrease from single-precision for 8 different trained models when using post-training Q-FP representations at different precisions.

Figure 4: Comparison of K and t min for various networks (Right) and EfficientNet variants (Left) at different virtual precisions on the ImageNet dataset.

Figure 5: Q-FP percentage validation accuracy decrease for different ImageNet models

Figure 6: Comparison of K and t min for different instances of EfficientNet-b0 (Left) and MobileNet-v2 (Right) at different virtual precisions on the ImageNet dataset.

Post-training Q-FP validation accuracy on CIFAR-10 when using K for model selection rather than single-precision validation accuracy

Table content: <table><tr><td/><td colspan="2">MobileNet-v2</td><td colspan="2">EfficientNet-b0</td></tr><tr><td>Precision</td><td>MCDA</td><td>(Wang et al., 2018)</td><td>MCDA</td><td>(Wang et al., 2018)</td></tr><tr><td colspan="2">(Weight, Act.) (K =1.42)</td><td>(K =1.56)</td><td>(K =1.46)</td><td>(K =1.89)</td></tr><tr><td>(32,32)</td><td>89.6</td><td>90.3</td><td>93.7</td><td>94.6</td></tr><tr><td>(8,8)</td><td>88.5</td><td>86.7</td><td>90.0</td><td>89.0</td></tr><tr><td>(6,6)</td><td>79.0</td><td>76.6</td><td>64.6</td><td>56.0</td></tr><tr><td>(5,5)</td><td>48.4</td><td>40.8</td><td>22.0</td><td>18.6</td></tr></table>

Peer review (iclr2020):
The premise of this paper is that quantization plays an important role in the deployment of deep neural networks; ie in the inference stage. However, errors due to quantization affect different neural architectures differently. It would be useful if we could predict ahead of time which models are more amenable to quantization. I think this is a very interesting premise and the paper is very well motivated. The paper is also very clear and well written, making the claims precise and backing these up with experiments. At the heart of the paper is the replacement of floating point numbers with inexact values, which are treated as random variables and defined precisely in equation 4. This definition enables the authors to apply Monte Carlo methods to obtain network predictions as shown in equation (10) and figure 2, and subsequently carry out sensitivity analysis. The experiments show that a measure of sensitivity (K) is indeed a good augmentation to cross-validation for model selection for the purpose of trading-off accuracy and resource consumption when launching deep neural networks with floating point rounding errors. One question I have for the authors is the following: There has been a large body of literature on Monte Carlo methods for Bayesian neural networks. Could those works have something to say in addressing some of the challenges posed in Section 4.1?