Monte Carlo Deep Neural Network Arithmetic
Authors: Julian Faraone
Philip Leong
Venue:  (2020)
Keywords: deep learning
quantization
floating point
monte carlo methods

Abstract: Quantization is a crucial technique for achieving low-power, low latency and high throughput hardware implementations of Deep Neural Networks. Quantized floating point representations have received recent interest due to their hardware efficiency benefits and ability to represent a higher dynamic range than fixed point representations, leading to improvements in accuracy.  We present a novel technique, Monte Carlo Deep Neural Network Arithmetic (MCA), for determining the sensitivity of Deep Neural Networks to quantization in floating point arithmetic.We do this by applying Monte Carlo Arithmetic to the inference computation and analyzing the relative standard deviation of the neural network loss.  The method makes no assumptions regarding the underlying parameter distributions. We evaluate our method on pre-trained image classification models on the CIFAR10 andImageNet datasets.  For the same network topology and dataset, we demonstrate the ability to gain the equivalent of bits of precision by simply choosing weight parameter sets which demonstrate a lower loss of significance from the Monte Carlo trials.   Additionally,  we can apply MCA to compare the sensitivity of different network topologies to quantization effects.

Under review as a conference paper at ICLR 2020
MONTE CARLO DEEP NEURAL NETWORK ARITH- METIC
Anonymous authors Paper under double-blind review
ABSTRACT
Quantization is a crucial technique for achieving low-power, low latency and high throughput hardware implementations of Deep Neural Networks. Quantized float- ing point representations have received recent interest due to their hardware effi- ciency benefits and ability to represent a higher dynamic range than fixed point representations, leading to improvements in accuracy. We present novel tech- nique, Monte Carlo Deep Neural Network Arithmetic (MCDA), for determining the sensitivity of Deep Neural Networks to quantization in floating point arith- metic. We do this by applying Monte Carlo Arithmetic to the inference compu- tation and analyzing the relative standard deviation of the neural network loss. The method makes no assumptions regarding the underlying parameter distribu- tions. We evaluate our method on pre-trained image classification models on the CIFAR-10 and ImageNet datasets. For the same network topology and dataset, we demonstrate the ability to gain the equivalent of bits of precision by simply choosing weight parameter sets which demonstrate lower loss of significance from the Monte Carlo trials. Additionally, we can apply MCDA to compare the sensitivity of different network topologies to quantization effects!
INTRODUCTION
Deep Neural Networks have achieved state-of-the-art performances in many machine learning tasks such as such as speech recognition (Collobert et al., 2011), machine translation (Bahdanau et al.) 2014), object detection (Ren al. 2015) and image classification (Krizhevsky al.] 2012). How- ever, excellent performance comes at the cost of significantly high computational and memory com- plexity, typically requiring teraops of computation during inference and Gigabytes of storage. To overcome these complexities, compression methods have been utilized, aiming to exploit the inher- ent resilience of DNNs to noise. These engender representations which maintain algorithm perfor- mance but significantly improve latency, throughput and power consumption of hardware imple- mentations. In particular, exploiting reduced numerical precision for data representations through quantization has been emphatically promising, whereby on eustomizable hardware, efficiency scales quadratically with each bit of precision.
Quantization of fixed-point arithmetic (Q-FX) for DNN inference has been extensively studied, and more recently there has been increasing interest in quantized floating point (Q-FP) arithmetic for both DNN inference and training (Wang al. 2018). Q-FP has the advantage of higher dynamic range compared to equivalent Q-FX representations and reduced hardware cost over single-precision floating point (FP). This has influenced application specific integrated circuits (ASICs) such as Google's 's tensor processing unit (TPU), which supports 16-bit floating point (16-FP) and soft pro- cessors such as Microsoft's s Project Brainwave which utilizes 8-FP.
To illustrate these hardware benefits, we synthesized arithmetic logic units (ALUs) in different for- mats and different precision on an FPGA and present performance estimates in operations per sec- ond (OPs) and area estimates in Look-up Tables (LUTss) per operation (LUTss/Op) in Figure [ As shown, 8-bit fixed point (8-FX) achieves improved performance and area over 8-FP. However, 7-FP is significant improvement over 8 or 12-FX and or 9-FP. These examples demonstrate substan- tial performance and area benefits from reducing FP precision by only to 2-bits. Thus, we
Source code will be available if the paper is accepted

---

Under review as a conference paper at ICLR 2020
03.07
4130
40.00
2 35.00
13.00
24.00
25.00
30.00
13:14
9.00
5,00
0.00
4-X
B-X
124%
7-IP
8-1P
9.77
4-XX
B-X
12-XX
7-PP
8-F
9-P
Quantiortion Type
Quartication Type
The estimated multiply- accumulate performance on a Xilinx VU9P FPGA, operating at 500MHz, in teraops (left) and area (right) for fixed (FX) and floating point (FP) in various precisions (both weights and activations).
Figure 1:
design networks which not only achieve high accuracy but robust to quantization, higher performing hardware solutions are possible.
Since in Q-FP, bits, quantization and rounding artefacts will be introduced, with inaccuracies being cascaded along the computation graph (TEE 1983 Goldbergi [991; Higham 2002). This paper proposes Monte Carlo Deep Neural Network Arithmetic (MCDA), a novel way to apply Monte Carlo Arithmetic (MCA) (Parker al. 2000) for determining the sensitivity of Deep Neural Networks to Q-FP rep- resentations. It allows sardware-softwan designers to quantify the impact of quantization, enabling more efficient systems to be discovered. We do this by exploiting Monte Carlo simulations under which rounding effects are randomized. This, in turn, allows one to infer the sensitivity of executing a computation graph to quantization effects.
trying represent the infinite of real numbers using finite number of
Our MCDA technique is highly sensitive, allowing very small differences in quantization behaviour to be detected, The technique makes no assumptions regarding data distributions and directly mea- sures the effects of quantization on the problem under study. This allows to provide insights into the precision requirements of any inference network for any given dataset. Additionally we can use the technique select weight parameters which are more robust to floating point rounding. The theoretical and practical contributions of this work can be summarized as follows:
We introduce novel and rigorous analysis technique, Monte Carlo Deep Neural Network Arithmetic (MCDA), which measures the sensitivity of Deep Neural Network inference computation to floating point rounding error.
When applied to neural network inference, we show MCDA determine the precision quirements of different networks, rank them, and detect small differences between different neural network topologies and weight sets.
We demonstrate that while network with the same topology but different weights may have the same loss and validation accuracy, their sensitivity to quantization can be vastly different. Using the CIFAR- 10 and ImageNet datasets, introduce method to choose weights which are more robust to rounding error, resulting in a greatly improved accuracy- tradeoff over state-of-the-art methods.
It is worth noting that although we consider convolutional neural networks for image classification, this method could be applied for any neural network model architectures and applications. Moreover, while the experiments in this paper are limited to inference, it may be possible to apply the same idea to analyze training algorithms.
RELATED WORK
Low-precision representations deep learning have been extensively studied. Many training meth- ods have been developed to design representations for fixed point inference (Jacob al. 2018 Faraone et al. 2018 Zhou et al. 2016) and training (Wu et al. 2018b] Yang et al. 2019 Gupta et al. 2015 Sakr & Shanbhag 2019). Other methods have also utilized Q-FP arithmetic for inference and training whilst maintaining single preecsiir accuracy. (Micikevicius et al.) (0018) implemented 16- FP arithmetic training whilst storing 32-FP master copy for the weight updates. Additionally,
2

---

Under review as a conference paper at ICLR 2020
Wang al. 2018: Mellempudi et al. 2019) with 8-FP arithmetic whilst using 16-FP copy of the weights and 16/32-bits for the accumulator. Techniques for determining per-layer sensitivity to quantization have also been studied (Choi et al. 2016 [akkr & Shanbhag 2018). Further, other stud- ies have successfully determined the minimum fixed point precision requirements for a given DNN accuracy threshold (Sakr et al., 2017). The accuracy and stability of various numerical algorithms in finite precision arithmetic has been studied in (Higham 2002 Wilkinson 1994). This has led to techniques for tracking information lost from finite precision arithmetic using random perturba- tion such as Monte Carlo Arithmetic (Parker et al. 2000, Frechtling & Leong, 2013). Monte Carlo methods have also been used Bayesian Neural Networks (Buchholz al. 2018] Blier & Ollivier, 2018). In particular, (Achterhold et al.) 2018) introduced quantizing prior to learn weights which are either close to a quantized representation or have high variance. (Louizos et al., (217) used hierarchical priors to prune nodes and posterior uncertainties to determine the optimal fixed point precision. (Blundell et al. 2015) use a Monte Carlo approach to Iearn a probability distribution on the weights of a neural network. To the best of our knowledge, our work is the first to present technique for directly determining the sensitivity of DNNs to floating point rounding and to explic- itly compute precision bounds of trained network. These ideas can be very usefully applied to extending the limits of low-precision representations in deep learning applications.
3 BACKGROUND
In this section, we describe background theory upon which our technique for determining the sensi- tivity of DNNs to floating point rounding is based.
3.1 FLOATING POINT ARITHMETIC
The IEEE-754 binary floating point format (EEE 1985) represents most real numbers by a subset in normal form as:
i (-1)* *= (1 m
(1)
where S, E {0, 13 the sign bit, er is an integer representing the exponent of á and my is the mantissa of 2. Such number formats can be described as a (S Ca, my)) tuple. In binary form the representation (b", by, bgg .... Db_,, byr b .... bBnm. E {0. 1] with Be, and Bm, being the number of exponent and mantissa bits, respectively. The infinite set of real numbers R is represented in computer with B = 1 Be, + Bm, bits, and we define the finite set of real numbers representable in floating point format as exact values, IF C R. Real numbers which aren't 't representable are rounded to their nearest exact value. We call this set of numbers inexact values, II, where I IUF- R.
The approximation à = =F))) = x(1+ given I E 1, introduces rounding error into the compu- tation. The value of = s- represents the relative error which a function of the machine hardware precision, p. as 8 < E, where = 2-P IEEB 1985 Goldberg 1991 Higham 2002).
In general, inexactness can be caused by finite representations or errors propagating from earlier parts of the computation. Often the primary cause error in floating point arithmetic is catastrophic cancellation which causes numerical inaccuracy. Catas- trophic cancellation occurs when for example, two near equal FP numbers, sharing significant digits,
subtracted from one another as shown in (2) (Higham 2002
0. fi f2 ... fú f(k+1) ... ft
fi f2 ... fN f(++1) ... f..fe f(r()))) ... Tp
0. f1 f2 ... fa g(++1) ... gt.
(2) 0. f1 f2 ... fo 9(++1) ... 91 f(+1)) ... Pp (3)
= 0.0 0 0 0 1(++1) he
= 0. 0 0... 0 h(++1) h4 ((++)) ... Lp
In normalized form, the leading zeros are removed by shifting the result to the left and adjusting the exponent accordingly. The result is 0.h441...fui.... which has only (t (t-28 k) accurate digits and digits i which are unknown. Additionally, the remaining accurate digits h are most likely affected by rounding error in previous computations. This can significantly magnify errors, especially in computing large computational graphs such as that of of art DNNs.
If either operand in (2) is inexact, then the digits h are no more significant than any other sequence of digits. Yet, FP arithmetic has no mechanism of recording this loss of significance. By padding both operands with random digits and f in 3.. the resulting digits are randomized. If
3

---

Under review as a conference paper at ICLR 2020
digits are lost in the result, then k random digits will be in the normalized result and when computed over many random trials, the results will disagree on the trailing & digits. In this case, we are able to detect catastrophic cancellation because the randomization over many trials provides statistical simulation of round-off errors. We can use techniques from numerical analysis such as Monte Carlo methods appropriately insert precision-dependant randomization in this way.
3.2 MONTE CARLO ARITHMETIC
Monte Carlo methods can be used to analyze rounding by representing inexact values as random variables (Parker al., 2000, Frechtling Leong. 2015). The real value x, as represented (.. can be modelled to t digits, using:
ineract(2, 1, = 2 20= (-1)*= (1 m2 + +-""""
(4)
where 5 E (1- is uniformly distributed random variable and is a positive integer repre- senting the virtual precision of concern. For the same input è in 4 we can run many Monte Carlo trials which will yield different values each trial, where 0
p that the MCA can be run accurately on a computer with machine precision p. The ability to vary is useful because it allows us then evaluate the hardware precision requirements of given system or computational graph for a given DNN.
MCA is a method to model the effect of rounding on computational graph by randomizing all arithmetic operations. The randomization is applied for both generating inxxact operands and also in rounding. In each operation using MCA, ideally both catastrophic cancellation and rounding error can be detected. An operation using MCA is defined as:
I o y = roundd (oundd(ieezaee(neeodot inezact(y)))
(5)
where E (+,-,x,+)). By applying the inexact function to both operators we make it possible to detect catastrophic cancellation. Furthermore, applying the inexact function to the operation output and then rounding this value implements random rounding and hence is used to detect rounding error (Parker& & Langley] 1997). Hence, for the same input into the system, each trial will yield different operands and output.
After modifying the inexact and rounding operations as described, we use random sampling to simulate Monte Carlo trials, For each trial, we collect data on the resulting output of the system and compute summary statistics to quantify its behaviour (Parker et al. (000). With sufficiently large number of Monte Carlo trials and virtual precision t, the expected value the output from these trials will equal the value from using real arithmetic. As explained in the next section, can determine the total number of digits lost to rounding error and the minimum precision required to avoid a total loss of significance.
3.3 ANALYSIS
The relative error is bounded by 5 < 2- from the design of IEEE FP arithmetic (Wilkinson 1994] Goldberg 1991). With this inequality, we can determine the expected number of significant binary digits available from p-digit FP system as p < -log2(0). These definitions can be adapted for MCA by replacing the precision the FP system, p. by the virtual precision, I, of an MCA operation. Thus, the relative error of an MCA operation, for virtual precision is 8 < 2- and the expected number of significant binary digits in a t-digit MCA operation is at most t. Using this definition and the proof provided in (Parker & Langley 1997), the total significant binary digits in a set of M trials is s' = logz " where j is the mean and a the standard deviation. The output of the system should be some scalar value so that can perform such analysis. For experimentation, M trials are run for all of E {1, ... 2... }. The total number of base-2 significant digits lost in set of M trials is Kr, 66
Ki = 1-
-t - log, θ +
= 5''
(6)
where E, for (u A 0), is the relative standard deviation (RSD) of the MCA results. The virtual precision controls the perturbation strength applied by the inexact function. For a given K,, as we reduce i, the RSD should increase according to equation 6 At some point, unexpected loss of
4

---

Under review as a conference paper at ICLR 2020
significance (Frechtling & Leong 2015) is encountered due to the nonlinear effects of quantization. The value at which this occurs is defined as fmin. The number of significant digits lost for the system being analyzed is then computed by averaging all K, whereby t min as shown in
Et=lmon
Kg where where
tmaz > 1
K
Fyear Kg
Emin
imax =
For DNN inference, we propose to use K as a sensitivity measure for the network to FP rounding. The method for implementing this is discussed in the next section.
4 MONTE CARLO DEEP NEURAL NETWORK ARITHMETIC
We now describe MCDA, methodology for applying MCA techniques to DNN computation, allow- ing us to understand the sensitivity of a given network and its weight representation to FP rounding.
4.1 NETWORK MODEL
We consider generalized non-linear L-layer neural network with output vector YL, input data vector x and learnable weight parameter tensor w = Uw, U. L.), whereby yı. = =/]=; w). To compute y, several layers consisting of general matrix multiplication (GEMM) operations (such as convolutional and fully-connected layers) between the layer input Xj and weight parameters Wis followed by non-linear activation function, h, producing intermediate layer outputs X,, L.c. Yi h(x1 & W)). The output of given layer becomes the input to the subsequent layer, i.c. XI with X1 x. loss function the objective function minimize updating via an optimizer such as stochastic gradient descent. For a given set of input data X, the total network loss during
inference calculated by applying loss function [oss[[(x; w). (xx) where (x) is the target ground truth output for x. The total loss for X is then a scalar output, such that:
L(X: [
Loss(](xs 9())
(8)
X
X
Updates of are usually done small batches over subsets of X.
Naively applying MCA to each operation (fine-grained MCA) as described in equation (5) poses sig- nificant computational difficulties for DNN models. We observe two primary issues with employing fine-grained MCA to a DNN computational graph:
Firstly, the number required trials for Monte Carlo experiments to generate robust re- sults can typically be in the hundreds or thousands. As DNN inference of state-of-the-art networks typically consist of billions of operations. the computational requirements of ap- plying MCA after each operation will be very large, making the technique impractical.
Using the accuracy as the system output for MCA experiments is problematic because it is discrete value. For high values of t, Monte Carlo results across different then become indistinguishable and the standard deviation for given / is potentially zero.
4.2 MONTE CARLO NETWORK INFERENCE
To reduce the computational cost of Monte Carlo experiments, we employ MCDA, which a coarse- grained approach to MCA for GEMM operations. Conveniently, these can be naturally implemented in modern machine learning frameworks such as PoTch. Furthermore, to ensure the system output continuous value, the loss function output is used, rather than the accuracy. In this case, small perturbations in layer operands are more likely to produce observable changes in the output.
MCDA applies vector version of (5) the DNN inference computational problem in (8). Our operands in this case are vectors and represents a neural network layer operation. For example, the output from performing GEMM operation can thus be represented by:
y1 round (inezacl (inercll (x1) 8 ineeaall (w1)))
Since the inexact function is applied to the inputs and outputs GEMM operation, an optimized implementation be used. This is in contrast to full MCA which requires the application of
5

---

Under review as a conference paper at ICLR 2020
Convolation
Botch, Norwaltation
Batck Normusization
Weighe
Scale
inssaci
Pustact
Relu
inctact
rrenact
rowers
Mnssace
rosnd
ducsart
Prenact
roend
Pretact
Axpur
Famare
Figure 2: Applying MCDA to residual blocks found in ResNet. The inexact function applied to each input operand and both the inexact function and base-2 rounding is applied to the output.
to every individual scalar operation. The vector form in is applied to each edge of neural net- work computational graph where multiply (division) and/or add (subtract) operations performed. Hence, is not applied to operations such as MaxPool and ReLu. As an example, in Figure [ we show where the inexact function applied for a residual block with folded batch normalization, which is repeating sequence of layers found in ResNet models (He et al. 2015). At the final output of the network, the loss is computed with (8) and the inexact function is applied to the outputs y and also the loss output scalar value L. From analyzing the behavior of the loss, we infer the sensitivity of the accuracy of the system FP rounding. By using MCDA for the GEMM operations, we will not be able to detect all instances of catastrophic cancellation. However, we significantly reduce execution time over fine-grained MCA and show in the next section that we can still retrieve valu- able information about our system. In fact, for one trial with one batch of 32 images on ImageNet running on a Nvidia Titan Xp GPU, the speed up of regular inference without MCDA is only 1.05 x (for a single Monte Carlo trial). We also note that fine-grained MCA could be applied with a sim- ple modification and would be possible given appropriate customized hardware support for parallel Monte Carlo computations (Yeung et al. 2011).
5 EXPERIMENTAL RESULTS
In this section, we present experimental results for applying MCDA to exemplary convolutional neural networks. We use the CIFAR-10 and ImageNet image classification datasets to compare MobileNet-v2 (Sandler et al.) 2018), EfficientNet Taan Lc 2019). AlexNet, ResNet (He et al. 2015), SqueezeNet (landola et al. 2016) and MnasNet (Tan et al. 2018). For CIFAR-10, we use batch size of 128, whilst we use a batch size of 32 for ImageNet experiments. Cross-entropy is used as the loss function for both datasets. For given network, dataset, weight representation and t, apply MCDA with M 1000 trials. The resulting loss from each trial is computed with the same single batch of images and hence additional data is not required. We compute O from our results for all / E (1, 2,
16). Following this, we run linear regression analysis using the MCALIE (Frechtling & Leong 2015) library, on our O values. determine tmmi and K. Emán is defined as the point of lowest where the difference between the regression line and the equivalent O is less than half binary digit, i.e. log (20.5). Further detail on the calculation of K and Emin from MCALIB can be found in Appendix A We report Q-FP validation accuracy using the quantization function from (Wang et al. 2018) with stochastic rounding (See Appendix A.2
5.1 DISTINGUISHING WEIGHT PARAMETER REPRESENTATIONS
As discussed in Section 3.1 the inexactness in FP arithmetic largely depends on the numerical value of operands. Two instances of the same network and dataset, with the same validation accuracy, but vastly different weight representations, will likely produce differing sensitivities to FP rounding. We first train 8 instances of EfficientNet--o bo and MobileNet-V2 V2 on CIFAR- from scratch with random initialization from (Gloot & Bengio 2010), all achieving within 1% validation accuracy of one another. Using MCDA, we calculate the K values for each model (See Appendix A.3
https://githuu.coom// https//gghhub.omm//mfccchhthnnna

---

Under review as a conference paper at ICLR 2020
36
1.4
12
10
8
Volidation
1.5
2,5
355
Loss Significance
Loss of Significaama 04)
(a) EfficientNet-bo (8-
(b) MobileNet-v2 (8-bit)
16
Validatton Acourady Decraage
1.4
12
10
1.5
2.5 Significance
3.5
1.5 Significncee
25
(c) EfficientNet-b0 (7-bit)
(d) MobileNet-v2 (7-bit)
Figure 3: CIFAR-10 10 percentage validation accuracy decrease from single-precision for different trained models when using post-training Q-FP representations at different precisions.
Table I:
Post-training Q-FP validation accuracy on CIFAR-I0 when using K for model selection
rather than single-precisiee validation accuracy
MobileNet-v2
EfficientNet-b0
Precision (Weight, Act.)
MCDA
Wang et al. 2018)
MCDA
Wang et al. (018))
(K =1.22)
(K =1.6
(K = -1.46
(A =1.99)
(32,32)
89.6
90.3
93.7
94.6
(8,8)
88.5
86.7
90.0
89.0
(6,6)
79.0
76.6
64.6
56,0
(5,5)
48.4
40.8
22.0
18.6
We then test their percentage validation accuracy decrease from using post-training quantization (i.e. finetuning) with varying Q-FP precisions. In Figure 3 K values typically experience a larger drop in Q-FP accuracy, indicating they are more sensitive to floating point rounding error. Notably, the model with lowest K for 7-bit MobileNet-v2 experiences a lower percentage validation accuracy drop than three of the 8-bit models. In this case, MCDA model selection enables the saving of bit of precision while achieving smaller accuracy decrease than some of the trained 8-FP models.
that the models with higher
5.2 COMPARISON TO PREVIOUS WORK
One practical use case from the insights gained by MCDA is model selection for quantization. Typ- ically when quantizing given model trained on a given dataset, the model with highest validation accuracy is chosen and the sensitivity to quantization is assumed to be the same across models. As discussed, for Q-FP representations this is not necessarily the case. We can use K from MCDA to predict which models will be more robust to quantization. To demonstrate this, in Table [ we compare post-training quantization results for model selection based on K from MCDA, against baseline model chosen based on the highest single-precision validation accuracy. Evidently, even though the single- preccis-en accuracy is initially as much as 0.9% higher, after quantizing the network to 8-5 bits, the accuracy of the network chosen by smallest K is always significantly higher.
5.3 NETWORK COMPARISON
Modern DNNs consist of convolutional blocks with highly varying computational graphs (Wu et al. 2018a) Howard et al. 2017). Using MCDA we can also compute and compare their sensitivites to floating point rounding error to determine which networks will be robust to Q-FP representations.
7

---

Under review as a conference paper at ICLR 2020
RAMI dn. Dey
Reeile Stl. Dev.
Rechbe Oue Kir
Aloobeet mn (oe1.77)
MtsaNek34 K=1.20)
Outtens
Effilinshles 50 (Taw K=1.60)
Minsobarrt de
Ideal (RSD 27)
kdoal (MBI 2
10
nual Phecasion 02
Vituual Proctsion (t)
Figure 4: Comparison K and Imi for various networks (Right) and EfficientNet variants (Left) at different virtual precisions on the ImageNet dataset.
18-be
9-brt
8-2it
Percentage Azuuncy Decmase
13.5
4.5
Atecheet (1.00)
Resheet 34 (122))
Resheet 18 (1.30)
Effiienthette 60 (1.60) Squeszeliet (1.87)
Mnaolett_.@..74 Eftcientien-11 (3.24) Restir 50 (3.50)
Figure 5: Q-FP percentage validation accuracy decrease for different ImageNet models
In Figure 4 we show the θ of pre-trained models, trained on the ImageNet dataset from Py Torch for differing values of and run linear regression analysis over our data points. From here we can then assign K value to each network and compare their loss of significance. At each f,, the distance from the regression lines to the ideal line represents the values of Ke, as described in (6). From the MCDA results, AlexNet is the least sensitive to rounding and ResNet-50 is the most, with various models in between these two. Additionally we compare EfficientNet at two different model scales and evidently the larger model has much larger sensitivity. We then also compare the validation accuracy percentage decrease of all models at 10, 9 and 8-FP post-training in Figure [ At 8-FP, besides MnasNet which experiences large accuracy drop, K is able predict validation accuracy degradation. Thus, MCDA provides very valuable information about Q-FP model design.
CONCLUSION
We present a novel, highly sensitive, technique to quantify rounding error in DNNs. This is the first method successfully compare the sensitivity of networks floating point rounding error Ultimately, this technique provides a tool for enabling the design of networks which perform higher when quantized. We do this by applying concepts from Monte Carlo Arithmetic theory to DNN computation. Furthermore, we show that by calculating the loss of significance metric K from MCDA. on the CIFAR-10 and ImageNet datasets, we can compare network sensitivities floating point rounding error and gain valuable insights to potentially design better neural networks. This an important contribution due to the increasing interest low-precision floating point arithmetic for efficient DNN hardware systems. The theoretical and practical contributions of this paper will likely translate well to analyzing floating point rounding in backpropagation future work.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jan Achterhold, Jan M. Köhler, Anke Schmeink, and Tim Genewein. Variational network quantiza- tion. In ICLR, 2018.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate, 2014. URL http://t//// httpp://aaa.vv.or/abb///////s wwtp:/b////....///.//// cite Nedi:1409.0473oommme
Accepted ICLR 2015 oral presentation.
Léonard Blier and Yann Ollivier. The description length of deep learning models. In Advances in Neural Information Processing Systems, pp. 2216 2226, 2018.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In Proceedings of the 32Nd International Conference on International Con- ference on Machine Learning Volume 37, ICML' 15, pp. 1613-1622. JMLR.org, 2015. URL http://dl. acm. org/ citation. c f ? d= 300451511111.35500 3045290
R.P. Brent, Algorithms for minimization without derivatives. Prentice-Hall, 1973. ISBN 0-13- 022335-2.
Alexander Buchholz, Florian Wenzel, and Stephan Mandt. Quasi-Monte Carlo variational in- ference. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 668-677, 677, Stockholmsmssan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http: //proceedings ml.. www.ml.....s......m v80 httpbuuchhlllc 8a html
Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Towards the limit of network quantization. ArXiv, abs/1612.01543. 2016.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. J. Mach. Learn. Rex., 12:2493- 2537, November 2011. ISSN 1532-4435. URL http:/ttt///d acm. org/eitation cfm?
id 1953493 2078186
Julian Faraone, Nicholas Fraser, Michaela Blott, and Philip HW Leong. Syq: Learning symmet- ric quantization for efficient deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4300-4309, 2018
Michael Frechtling and Philip H. W. Leong. Mcalib: Measuring sensitivity to rounding error with monte carlo programming. ACM Trans. Program. Lang. Syst. 37(2):5:1-5:25, April 2015. ISSN 0164-0925. doi: 10.1 145/26666650,, URL httl: //tp:://ccc.. wtt.cci.ccc.mmmmm acm. wttpoo////.........e...... 1et...1115//27777
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neu- ral networks. In In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATSIO). Society for Artificial Intelligence and Statistics, 2010.
David Goldberg. What every computer scientist should know about floating-point arithmetic. ACM Comput. Surv., 23(1):5 March 1991. ISSN 0360-0300. doi: 10.1145/103112.10316. URL
huttp://dot acm. or http://...111111111
103163
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In Proceedings of the 32Nd International Conference on Interna- fional Conference on Muchine Learning Volume 37, ICML'15, pp. 1737-1746. JMLR.org, 2015.
URL http:/// /estp://de /i. acm. /rg/ citation. cfm d= 3045118. 0111118..043300 3045303
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2016 IEÉE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770- 778, 2015.
Nicholas J. Higham. Accuracy and Stability of Numerical Algorithms. Society for Industrial and Applied Mathematies, Philadelphia, PA, USA, 2nd edition, 2002. ISBN 08987 15210.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. ArXiv, abs/1704.04861, 2017.

---

Under review as a conference paper at ICLR 2020
Peter J. Huber. Robust estimation of location parameter. Annals of Mathematical Statistics, 35 (1):73-101, March 1964. ISSN 0003-4851, doi: 201211//mm//111770033222 URL http: 7/dx. wwt..doo.org////0.... 1214/ /
Forrest N. landola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <Imb model size.
CoRR, abs/1602.07360, 2016. http: /ttpii:///axii.../ wttp://www.r//b/////2....00
JEEE. IEEE standard for binary floating-point arithmetic.. Institute of Electrical and Electronics Engineers, New York, 1985. Note: Standard 754-1985.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for effi cient integen-aritbmeeiconnly inference. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q- Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097- 1105. Curran Associates, Inc., 2012. URL http: http://paperss nips cc/paper/
4824 imagenet classification with deep wowvolutioonnnooee -ewoouutunonnnnuuuaaaaaaa neural networks.
pdf
Christos Louizos, Karen Ullrich, and Max Welling. Bayesian compression for deep learn- ing. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 3288-3298. Curran Associates, Inc., 2017. URL http::/papent nips. cc/paper/ 6921 1i21-bayesian -nn--ayyesaan-ccomressssooo ferrsiion-oo---ee poor-deep--o dew-deeeeereieee wew.eeenn.gg.oo......
Naveen Mellempudi, Sudarshan Srinivasan, Dipankar Das, and Bharat Kaul. Mixed precision train- ing with 8- bit floating point. CoRR, abs/1905.12334, 2019. URL htt.:tpp//axxv..or///ooo///
1905.12333
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In International Conference on Learning Representations, 2018. URL
https://cpeereviie net wett:/ooummiierlglseee dddnmunss=d11gg3g
Douglas Stott Parker and David Langley. Monte carlo arithmetic: exploiting randomness floating- point arithmetic. In Monte Carlo Arithmetic: exploiting randomness in floating-poii point arithmetic. Computer Science Department, University Of California, 1997.
Douglas Stott Parker, Brad Pierce, and Paul R. Eggert. Monte carlo arithmetic: how to gamble with floating point and win. Computing in Science and Engineering, 2:58-66 68, 2000.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-emn: Towards real-time object detection with region proposal networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Gareett (eds.), Advances in Neural Information Processing Systems 28, pp. 91-99. Curran Associates, Inc., 2015. URL /ttp::
cctpaper/ 5638 faster-r- -nneennn-tooon towards real- time object detection with region pdf
proposal-netwoxxe
Charbel Sakr and Naresh Shanbhag. An analytical method to determine minimum per-layer preci- sion of deep neural networks. 20/8 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1090-1094. IEEE, 2018.
Charbel Sakr and Naresh Shanbhag. Per-tensor fixed-point quantization of the prokapropn algorithm. In International Conference on Learning Representations, 2019. URL https:// openreview net/ forum? incerrmmiiiiie fideiiidiikaa (A
Charbel Sakr, Yongjune Kim, and Naresh Shanbhag. Analytical guarantees on numerical preci- sion of deep neural networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 3007-3016. JMLR. org, 2017.
10

---

Under review as a conference paper at ICLR 2020
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. 2018 IEEE//VF Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018.
Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning. volume 97 of Proceedings of Machine Learn- ing Research, pp. 6105-6114 Long Beach, Californa,, USA, 09-15 Jun 2019. PMLR. URL http:://proceeeinga. m.l...................m.... wwzess/v9/taaie
html
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V. Le. Mnasnet: Platform- aware neural architecture search for mobile. ArXiv, abs/1807. 11626, 2018.
Naigang Wang. Jungwook Choi, Daniel Brand, Chia- Yu Chen, and Kailash Gopalakrishnan. Train- ing deep neural networks with 8-bit floating point numbers. In Proceedings of the 32Nd In- ternational Conference on Neural Information Processing Systems, NIPS' 18, pp. 7686-7695, USA, 2018. Curran Associates Inc. URL http:://ll... acmp:///lllac...c wttp:/oggcctiiiioooooo cfm?idt 3327757.3327866
James H. Wilkinson. Rounding Errors in Algebraic Processes. Dover Publications, Inc., New York, NY, USA, 1994. ISBN 0486679993.
Bichen Wu, Alvin Wan, Xiangyu Yue, Peter Jin, Sicheng Zhao, Noah Golmant, Amir Gholaminejad, Joseph Gonzalez, and Kurt Keutzer. Shift: zero flop. zero parameter altemattve spatial convolutions. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
June 201 8a.
Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. In International Conference on Learning Representations, 2018b. URL ftttps://opereeviee
net Seeet/fooummii--GXXemmea.
Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, and Christo- pher De Sa. Swalp: Stochastic weight averaging in low precision training. In ICML, 2019.
Jackson H.C. Yeung, Evangeline F.Y. Young, and Philip H.W. Leong. A monte-cart floating- point unit for self-validating arithmetic. In Proceedings of the 19th ACM/SIGDA International Symposium on Field Programmable Gate Arrays, FPGA '11, pp. 199-208, New York, NY, USA, 2011. ACM. ISBN 978 4500-0054-9.. acm. www. 145/ 1555
doi: 10. 04114///150413311555555
URL http://doi.
Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. ArXiv, abs/ wwss///0........ (60600660,, 2016.
A
APPENDIX
A.1
EXPERIMENTAL SETUP
In all our experiments, for given network, dataset, weight representation and we run 1000 Monte Carlo trials. The network loss output from each trial is computed with the same single batch of images from the training dataset. We then independently compute the O, of the network loss for t E 2,,2,3 .... |maz) where Imaz 16.
Following this, we then run linear regression and calculate K and fmin using MCALIE To com- pute the linear regression, MCALIB uses log transformed variable, with log(O) as the dependant variable and as the exploratory variable (Frechtling & Leong 2015).
log ( logg1 (2
(10)
--log (2)t
(2)t + logio (2)
(11)
ml +
(12)
11

---

Under review as a conference paper at ICLR 2020
Algorithm Summary of Linear Regression Analysis for MCDA
Initialize: Pre-train single-precision DNN model. Set Eman- Set number of trials M.
Inputs: Batch of inputs & targets (X, Y), loss function (oss(](;; w), v())), current weights
Outputs: Emin and K
Monte Carlo Trials:
fort t 1 tmaz do
for trials=1 to M do
L(X; w) ForwardPath (X, y, un using (
end for
Compute μ and a of all trials
Compute O using 6
end for
Calculate K and Imind
for 1 mmaz do
Compute Pt using (10) - (14)
Compute Pt- O
P. θ < (20.5) then
bmin
break
else
continue
end if
end for
Compute K using (6) and 7
where m = 12) = 0.30103 is the slope and is the intercept such that logz 10%). Given these inputs, the intercept c is calculated by minimizing the following objective function using Brents method (Brent 1973) for single variable optimization:
PH (e:)
(13)
where e; = O (mt; + c) is the residual error, E [(O fmee mlman = 2m is the initial search space for the intercept, 0.75 and PH (c) is the Huber loss function (Huber 1964):
e²² for lel 22
le S
k
(e(e)
(14)
for
Jel > k
where k = 1.345a and is the standard deviation of the residual enror set, After determining the linear regression model, P: = mt + c, we determine whether each θ, is an outlier. If value for O difffers to the equivalent P, by more than half binary digit, then it is classed
outlier. tmin is
then defined as the lowest where P, O, < log1o (20.6). To compute K for a given network, we then average the values for K; whereby imin- This removes the outliers from the computation of K. We have summarized how the experiments were simulated in Algorithm
A.2 QUANTIZED FLOATING POINT WITH STOCHASTIC ROUNDING
To quantize our pre-trained networks to Q-FP representations, we used stochastic rounding as de- scribed in Wang al. 2018) and implemented in QPytoret Two common forms of rounding for FP arithmetic are round to neareet and stochastic rounding. The former discards information in the least significant bit (LSB) which is rounded off. This information loss can be significant, especially when quantizing to a small number of bits. Stochastic rounding provides method to capture this information loss from rounding off the LSB. Given = (-1)==(1 + m. ]2 as described
d1)
12

---

Under review as a conference paper at ICLR 2020
S6011
Reliiio SAt Cnn
56-33
Ron,
Run,o
Ron, (00.00))
So-55
Ran, jen Ad)
Ran (022311
2"))
10
16
min Precadion 02
V rual Precatio 05
Figure 6: Comparison of K and Eiá for different instances of EfficientNet-b0 (Left) and MobileNet-v2 (Right) at different virtual precisions on the ImageNet dataset.
Assume that m, is in fixed precision with bits which needs to be rounded to z bits, then stochastic rounding is follows:
(-1) (1+ [m,] + =)2*= with probability
TTL
Round(x)
(15)
1-1)*((1 )22= with probability
where [m,] the truncation of E' LSBB of m, 2- For each Q-FP accuracy reported in our experiments, we tested all possible combinations of the number of bits allowed for the exponent and mantissa which satisfied the desired precision. We then chose the combination which produced the highest accuracy.
A.3 CIFAR-10 REGRESSION ANALYSIS
In Figure we display the linear regression analysis using MCALIB for the MobileNet-v2 and EfficientNet-bO models trained on CIFAR-10. As shown, Monte Carlo trials were run for each of these models for E {1, 2, ..., 16). The corresponding min and K values were then calculated using methods discussed in Appendix A.1. Thus, in Figure 3 it is these K values which plotted against 8 and 7-bit Q-FP validation accuracy. Also, the models with lowest K values are chosen for MCDA model selection in T
13


Reviews:
[{"review_id": "rkgOn7jaYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The premise of this paper is that quantization plays an important role in the deployment of deep neural networks; ie in the inference stage. However, errors due to quantization affect different neural architectures differently. It would be useful if we could predict ahead of time which models are more amenable to quantization. I think this is a very interesting premise and the paper is very well motivated.\n\nThe paper is also very clear and well written, making the claims precise and backing these up with experiments.\n\nAt the heart of the paper is the replacement of floating point numbers with inexact values, which are treated as random variables and defined precisely in equation 4. This definition enables the authors to apply Monte Carlo methods to obtain network predictions as shown in equation (10) and figure 2, and subsequently carry out sensitivity analysis. The experiments show that a measure of sensitivity (K) is indeed a good augmentation to cross-validation for model selection for the purpose of trading-off accuracy and resource consumption when launching deep neural networks with floating point rounding errors.\n\nOne question I have for the authors is the following: There has been a large body of literature on Monte Carlo methods for Bayesian neural networks. Could those works have something to say in addressing some of the challenges posed in Section 4.1? \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJxUTXU9tS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper studies the sensitivity of a neural network with respect to quantizing its weights and activations. The idea is to use Monte Carlo Arithmetic (MCA) in order to calculate the number of significant bits in the training loss (e.g. cross entropy) that are lost due to floating-point arithmetic. The results show that the number of significant bits lost correlates with the reduction in classification accuracy when quantizing the weights and activations of the neural network.\n\nDecision:\n\nOverall, this is an interesting paper with interesting results. However, I think there is considerable room for improvement, and that more details are needed in order to assess the significance of the results, as I detail in the rest of my review. For these reasons, I recommend weak reject for now, but I encourage the authors to continue working on improving the paper and to provide more details in the updated version.\n\nContribution:\n\nThe paper considers an important problem, that of quantizing the weights and activations of a neural network in order to reduce computational and memory cost, while maintaining the machine-learning performance as high as possible. \n\nIn my opinion, the main contribution of the paper is the experimental findings, and in particular that the sensitivity of the training loss with respect to the precision of the weights and activations correlates with the accuracy of the network. It seems to me that these results may relate to work on Bayesian neural networks, sharp vs flat minima, and minimum-description length approaches to variational inference. Work on these areas has also shown that sensitivity of the training loss with respect to the precision of the weights (which intuitively happens when the network is at a \"sharp\" local minimum vs a \"flat\" one) is related to poor generalization performance, and vice versa. I would encourage the authors to explore the potential relationship of their work with these areas, and possibly discuss them in an updated version of the paper.\n\nOriginality:\n\nThe paper describes a method for assessing the sensitivity of a neural network with respect to the precision of the weights and activations. The method is a straightforward application of Monte Carlo Arithmetic (MCA) to neural networks. I believe that the application of MCA to neural networks for this particular purpose is novel, and that the results are original. However, the introduction of the paper gives the impression that the proposed method is brand new, and even uses the acronym MCA to refer to the proposed method, which can be confusing to readers. I would suggest to the authors to rewrite the introduction so as to reflect more accurately that the contribution is not a brand-new method, but rather the application of an existing method (MCA) in a novel way.\n\nWriting quality:\n\nThe paper is generally easy to read, but there is considerable room for improvement. There are mistakes, and often the writing is sloppy and imprecise. I give some more specific suggestions on what to improve later on.\n\nTechnical quality:\n\nThe method is well motivated and the experiments seem reasonable. However, there is very little detail on the experiments, which makes it hard to assess their correctness/significance. I would suggest to the authors to rewrite the experimental section with full detail, or put more details in an appendix. In particular:\n- Is each Monte Carlo trial done on the same batch of training images or a different one? If different, how are the trials averaged, and does that mean that the standard deviation over trials also includes a contribution due to different batches?\n- In sections 5.1 and 5.2, how were the results for different t combined/aggregated? Did you use linear-regression analysis as in section 5.3?\n- When you say \"accuracy\", do you mean accuracy on the training set, validation set, or test set? This is particularly important for assessing the significance of the results, and is something that is currently missing from the description of the experiments.\n- How was the quantization of the neural networks performed? It would be good to explain this at least on a high level, in addition to citing Wang et al., (2018).\n- In section 5.2, how was the model selection for each method performed exactly? In the baseline method, was the model to be quantized selected based on validation performance before quantization or after quantization?\n\nSpecific suggestions for improvement:\n\nThe citation format, i.e. (Smith et al., (2019)), is unusual and uses unnecessarily many parentheses. Use \\citep for (Smith et al., 2019), and \\citet for Smith et al. (2019).\n\nThe illustration of fig. 1 is not fully convincing as a motivation for floating-point arithmetic. Even though it makes the case that Float(7, 7) is more efficient than Float(8, 8) and Float (9, 9), the comparison between Float(7, 7) and Fixed(12, 12) is hard to interpret, as we can't conclude whether the efficiency gain is due to reducing the number of bits or to switching from fixed-point to floating-point arithmetic. A more convincing illustration would compare fixed-point with floating-point arithmetic using the same number of bits.\n\nIt would be better if fig. 1 were 2D, as 3D doesn't add anything but makes it harder to compare sizes visually.\n\nPlease avoid exaggerations, such as \"exquisitely sensitive\" or \"extremely sensitive\", when \"sensitive\" would suffice.\n\nSection 2 is grammatically sloppy:\n- arihtmetic --> arithmetic\n- Last line of page 2 seems to be missing a verb.\n- this has lead --> this has led\n\nThe related-work section is too short and in many cases it doesn't explain what previous work has actually done. For example, \"rounding of inexact values to their nearest FP approximation has been studied in several publications\" is vague: what exactly these publication have done? This lack of detail makes it hard to assess the originality of the current paper, and how it differs from existing work.\n\nSection 3 is often unclear with imprecise mathematical notation:\n- \"e is the base-2 exponent of x in binary floating point arithmetic\": surely, the exponent is represented as an integer?\n- (bs, be1, be2, ..., bex, bm1, bm2, ..., bmx) is sloppy, as it indicates that the indices run from 1 to x.\n- Bx = sx + ex + mx  is also sloppy; what is meant here is the number of bits to represent sx, ex, mx and not the values themselves.\n- F(x) = x(1 + θ), shouldn't θ be δ?\n- \"which is typically the cause of horrific numerical inaccuracy from numerical analysis literature\", the phrase \"from numerical analysis literature\" doesn't make much sense here.\n- In eq. (4), substituting the expression for x from eq. (1) doesn't yield the same result.\n- \"The number of trials is an important consideration because [...] it can produce adverse effects on results\". What is meant by \"adverse effects\"? Do you mean that with few trials Monte Carlo doesn't give accurate results? Please be more specific.\n- \"we can determine the expected number of significant binary digits available from a p-digit FP system as p ≥ −log2(δ)\". I'm unable to follow this statement, please explain further. Also, from applying logs to δ ≤ 2^{−p} one gets an inequality that doesn't match the one in p ≥ −log2(δ).\n- \"The relative error of an MCA operation is, for virtual precision t, is δ ≤ 2^-t\", \"is\" is used twice here.\n- \"the expected number of significant binary digits in a t-digit MCA operations is at least t\", operations --> operation. Also, shouldn't it be at most t, otherwise K becomes negative?\n-  is discussed in the section --> is discussed in the next section.\n\nSome mistakes in section 4.1:\n- y = (x; w) --> y = f(x; w)\n- Eq. (8) is sloppy, it uses X for both the set and its size. Use |X| or something similar for the size.\n- In the caption of fig. 2, baes --> base\n\nI'm not convinced by the second bullet point in section 4.1, that the averaging over many images used to obtain the accuracy is the reason why MCA doesn't work well. Surely, the training loss (cross entropy) is also an average over many images? To me it would seem more plausible that the main reason MCA works with training loss but not accuracy is because accuracy is discrete, whereas training loss is continuous.\n\nFig. 3 would be much easier to read if the axes were labelled, and if the axes had the same range (so that different plots can be compared visually).\n\nFig. 5 would be easier to read if the networks were sorted with respect to K.\n\nIn section 5, CIFAR-10 is sometimes written as CIFAR10.\n\nAppendix A is empty, so it should be removed.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "S1xgzttTFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a scalable method based on Monte Carlo arithmetic for quantifying the sensitivity of trained neural networks to floating point rounding errors. They demonstrate that the loss of significance metric K estimated from the process can be used for selecting networks that are more robust to quantization, and compare popular architectures (AlexNet, ResNet etc.) for their varying sensitivities.\n\nStrengths:\n- The paper tackles an important problem of analyzing sensitivity of networks to quantization and offers a well-correlated metric that can be computed without actually running models on quantized mode\n- Experiments cover a wide range of architectures in image recognition\n\nWeaknesses:\n- The proposed method in Section 4.2 appears to be a straightforward modification to MCA for NN\n- Experiments only demonstrate model selection and evaluating trained networks. Can this metric be used in optimization? For example, can you optimize for lowering K (say with fixed t) during training, so you can find a well-performing weight that also is robust to quantization? 1000 random samples interleaved in training may be slow, but perhaps you can use coarse approximation. This could significantly improve the impact of the paper. Some Bayesian NN literatures may be relevant (dropout, SGLD etc). \n\nOther Comments:\n- How is the second bullet point in Section 4.1 addressed in the proposed method?\n- Can you make this metric task-agnostic or input-distribution-agnostic (e.g. just based on variance in predictions over some input datasets)? (e.g. you may pick a difference loss function or different test distribution to evaluate afterwards) \n- Does different t give different K? If so, what’s the K reported? (are those points on Figure 3)?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the impact of rounding errors on deep neural networks. The                                                       
authors apply Monte Carlos arithmetics to standard DNN operations.                                                                 
Their results indeed show catastrophic cancellation in DNNs and that the resulting loss of                                         
significance in the number representation correlates with decrease in validation                                                   
performance, indicating that DNN performances are sensitive to rounding errors.                                                    
                                                                                                                                   
Although recognizing that the paper addresses an important problem (quantized /                                                    
finite precision neural networks), the reviewers point out the contribution of                                                     
the paper is somewhat incremental.                                                                                                 
During the rebuttal, the authors made an effort to improve the manuscript based                                                    
on reviewer suggestions, however review scores were not increased.                                                                 
                                                                                                                                   
The paper is slightly below acceptance threshold, based on reviews and my own                                                      
reading, as the method is mostly restricted to diagnostics and cannot yet be used                                                  
to help training low-precision neural networks.
Decision: Reject