Monte Carlo Deep Neural Network Arithmetic

1 INTRODUCTION . Deep Neural Networks have achieved state-of-the-art performances in many machine learning tasks such as such as speech recognition ( Collobert et al. , 2011 ) , machine translation ( Bahdanau et al. , 2014 ) , object detection ( Ren et al. , 2015 ) and image classification ( Krizhevsky et al. , 2012 ) . However , excellent performance comes at the cost of significantly high computational and memory complexity , typically requiring teraops of computation during inference and Gigabytes of storage . To overcome these complexities , compression methods have been utilized , aiming to exploit the inherent resilience of DNNs to noise . These engender representations which maintain algorithm performance but significantly improve latency , throughput and power consumption of hardware implementations . In particular , exploiting reduced numerical precision for data representations through quantization has been emphatically promising , whereby on customizable hardware , efficiency scales quadratically with each bit of precision . Quantization of fixed-point arithmetic ( Q-FX ) for DNN inference has been extensively studied , and more recently there has been increasing interest in quantized floating point ( Q-FP ) arithmetic for both DNN inference and training ( Wang et al. , 2018 ) . Q-FP has the advantage of higher dynamic range compared to equivalent Q-FX representations and reduced hardware cost over single-precision floating point ( FP ) . This has influenced application specific integrated circuits ( ASICs ) such as Google ’ s tensor processing unit ( TPU ) , which supports 16-bit floating point ( 16-FP ) and soft processors such as Microsoft ’ s Project Brainwave which utilizes 8-FP . To illustrate these hardware benefits , we synthesized arithmetic logic units ( ALUs ) in different formats and different precision on an FPGA and present performance estimates in operations per second ( OPs ) and area estimates in Look-up Tables ( LUTs ) per operation ( LUTs/Op ) in Figure 1 . As shown , 8-bit fixed point ( 8-FX ) achieves improved performance and area over 8-FP . However , 7-FP is a significant improvement over 8 or 12-FX and 8 or 9-FP . These examples demonstrate substantial performance and area benefits from reducing FP precision by only 1 to 2-bits . Thus , if we 1Source code will be available if the paper is accepted can design networks which not only achieve high accuracy but are robust to quantization , higher performing hardware solutions are possible . Since in Q-FP , we are trying to represent the infinite set of real numbers using a finite number of bits , quantization and rounding artefacts will be introduced , with inaccuracies being cascaded along the computation graph ( IEEE , 1985 ; Goldberg , 1991 ; Higham , 2002 ) . This paper proposes Monte Carlo Deep Neural Network Arithmetic ( MCDA ) , a novel way to apply Monte Carlo Arithmetic ( MCA ) ( Parker et al. , 2000 ) for determining the sensitivity of Deep Neural Networks to Q-FP representations . It allows hardware-software designers to quantify the impact of quantization , enabling more efficient systems to be discovered . We do this by exploiting Monte Carlo simulations under which rounding effects are randomized . This , in turn , allows one to infer the sensitivity of executing a computation graph to quantization effects . Our MCDA technique is highly sensitive , allowing very small differences in quantization behaviour to be detected . The technique makes no assumptions regarding data distributions and directly measures the effects of quantization on the problem under study . This allows us to provide insights into the precision requirements of any inference network for any given dataset . Additionally we can use the technique to select weight parameters which are more robust to floating point rounding . The theoretical and practical contributions of this work can be summarized as follows : • We introduce a novel and rigorous analysis technique , Monte Carlo Deep Neural Network Arithmetic ( MCDA ) , which measures the sensitivity of Deep Neural Network inference computation to floating point rounding error . • When applied to neural network inference , we show MCDA can determine the precision requirements of different networks , rank them , and detect small differences between different neural network topologies and weight sets . • We demonstrate that while a network with the same topology but different weights may have the same loss and validation accuracy , their sensitivity to quantization can be vastly different . Using the CIFAR-10 and ImageNet datasets , we introduce a method to choose weights which are more robust to rounding error , resulting in a greatly improved accuracyarea tradeoff over state-of-the-art methods . It is worth noting that although we consider convolutional neural networks for image classification , this method could be applied for any neural network model architectures and applications . Moreover , while the experiments in this paper are limited to inference , it may be possible to apply the same idea to analyze training algorithms . 2 RELATED WORK . Low-precision representations of deep learning have been extensively studied . Many training methods have been developed to design representations for fixed point inference ( Jacob et al. , 2018 ; Faraone et al. , 2018 ; Zhou et al. , 2016 ) and training ( Wu et al. , 2018b ; Yang et al. , 2019 ; Gupta et al. , 2015 ; Sakr & Shanbhag , 2019 ) . Other methods have also utilized Q-FP arithmetic for inference and training whilst maintaining single-precision accuracy . ( Micikevicius et al. , 2018 ) implemented 16- FP arithmetic training whilst storing a 32-FP master copy for the weight updates . Additionally , ( Wang et al. , 2018 ; Mellempudi et al. , 2019 ) with 8-FP arithmetic whilst using a 16-FP copy of the weights and 16/32-bits for the accumulator . Techniques for determining per-layer sensitivity to quantization have also been studied ( Choi et al. , 2016 ; Sakr & Shanbhag , 2018 ) . Further , other studies have successfully determined the minimum fixed point precision requirements for a given DNN accuracy threshold ( Sakr et al. , 2017 ) . The accuracy and stability of various numerical algorithms in finite precision arithmetic has been studied in ( Higham , 2002 ; Wilkinson , 1994 ) . This has led to techniques for tracking information lost from finite precision arithmetic using random perturbation such as Monte Carlo Arithmetic ( Parker et al. , 2000 ; Frechtling & Leong , 2015 ) . Monte Carlo methods have also been used in Bayesian Neural Networks ( Buchholz et al. , 2018 ; Blier & Ollivier , 2018 ) . In particular , ( Achterhold et al. , 2018 ) introduced a quantizing prior to learn weights which are either close to a quantized representation or have high variance . ( Louizos et al. , 2017 ) used hierarchical priors to prune nodes and posterior uncertainties to determine the optimal fixed point precision . ( Blundell et al. , 2015 ) use a Monte Carlo approach to learn a probability distribution on the weights of a neural network . To the best of our knowledge , our work is the first to present a technique for directly determining the sensitivity of DNNs to floating point rounding and to explicitly compute precision bounds of a trained network . These ideas can be very usefully applied to extending the limits of low-precision representations in deep learning applications . 3 BACKGROUND . In this section , we describe background theory upon which our technique for determining the sensitivity of DNNs to floating point rounding is based . 3.1 FLOATING POINT ARITHMETIC . The IEEE-754 binary floating point format ( IEEE , 1985 ) represents most real numbers x by a subset in normal form as : x̂ = ( −1 ) sx ( 1 +mx ) 2ex ( 1 ) where sx ∈ { 0 , 1 } is the sign bit , ex is an integer representing the exponent of x̂ and mx is the mantissa of x̂ . Such number formats can be described as a ( sx , ex , mx ) tuple . In binary form the representation is ( bs , be1 , b e 2 , ... , b e Bex , bm1 , b m 2 ... , b m Bmx ) ∈ { 0 , 1 } B , with Bex and Bmx being the number of exponent and mantissa bits , respectively . The infinite set of real numbers R is represented in a computer withB = 1+Bex+Bmx bits , and we define the finite set of real numbers representable in floating point format as exact values , F ⊂ R. Real numbers which aren ’ t representable are rounded to their nearest exact value . We call this set of numbers inexact values , I , where I∪F = R. The approximation x̂ = F ( x ) = x ( 1 + δ ) , given x ∈ I , introduces rounding error into the computation . The value of δ = ∥∥x−x̂ x ∥∥ , represents the relative error which is a function of the machine hardware precision , p , as δ ≤ , where = 2−p ( IEEE , 1985 ; Goldberg , 1991 ; Higham , 2002 ) . In general , inexactness can be caused by finite representations or errors propagating from earlier parts of the computation . Often the primary cause of error in floating point arithmetic is catastrophic cancellation which causes numerical inaccuracy . Catastrophic cancellation occurs when for example , two near equal FP numbers , sharing k significant digits , are subtracted from one another as shown in ( 2 ) ( Higham , 2002 ) . 0. f1 f2 ... fk f ( k+1 ) ... ft − 0. f1 f2 ... fk g ( k+1 ) ... gt = 0 . 0 0 .... 0 h ( k+1 ) ... ht ( 2 ) 0. f1 f2 ... fk f ( k+1 ) ... ft r ( t+1 ) ... rp − 0. f1 f2 ... fk g ( k+1 ) ... gt r̂ ( t+1 ) ... r̂p = 0 . 0 0 .... 0 h ( k+1 ) ... ht i ( t+1 ) ... ip ( 3 ) In normalized form , the leading zeros are removed by shifting the result to the left and adjusting the exponent accordingly . The result is 0.hk+1 ... hti1 ... ik which has only ( t − k ) accurate digits and digits i which are unknown . Additionally , the remaining accurate digits h are most likely affected by rounding error in previous computations . This can significantly magnify errors , especially in computing large computational graphs such as that of state-of-the-art DNNs . If either operand in ( 2 ) is inexact , then the digits h are no more significant than any other sequence of digits . Yet , FP arithmetic has no mechanism of recording this loss of significance . By padding both our operands with random digits r and r̂ in ( 3 ) , the resulting digits i are randomized . If k digits are lost in the result , then k random digits will be in the normalized result and when computed over many random trials , the results will disagree on the trailing k digits . In this case , we are able to detect catastrophic cancellation because the randomization over many trials provides a statistical simulation of round-off errors . We can use techniques from numerical analysis such as Monte Carlo methods to appropriately insert precision-dependant randomization in this way . 3.2 MONTE CARLO ARITHMETIC . Monte Carlo methods can be used to analyze rounding by representing inexact values as random variables ( Parker et al. , 2000 ; Frechtling & Leong , 2015 ) . The real value x , as represented in ( 1 ) , can be modelled to t digits , using : inexact ( x̂ , t , δ ) = x̂+ 2ex−tδ = ( −1 ) sx ( 1 +mx + 2−tδ ) 2ex ( 4 ) where δ ∈ U ( − 12 , 1 2 ) is a uniformly distributed random variable and t is a positive integer representing the virtual precision of concern . For the same input x̂ in ( 4 ) , we can run many Monte Carlo trials which will yield different values on each trial , where 0 < t < p so that the MCA can be run accurately on a computer with machine precision p. The ability to vary t is useful because it allows us to then evaluate the hardware precision requirements of a given system or computational graph for a given DNN . MCA is a method to model the effect of rounding on a computational graph by randomizing all arithmetic operations . The randomization is applied for both generating inexact operands and also in rounding . In each operation using MCA , ideally both catastrophic cancellation and rounding error can be detected . An operation using MCA is defined as : x ◦ y = round ( inexact ( inexact ( x ) ◦ inexact ( y ) ) ) ( 5 ) where ◦ ∈ ( + , − , × , ÷ ) . By applying the inexact function to both operators we make it possible to detect catastrophic cancellation . Furthermore , applying the inexact function to the operation output and then rounding this value implements random rounding and hence is used to detect rounding error ( Parker & Langley , 1997 ) . Hence , for the same input into the system , each trial will yield different operands and output . After modifying the inexact and rounding operations as described , we use random sampling to simulate Monte Carlo trials . For each trial , we collect data on the resulting output of the system and compute summary statistics to quantify its behaviour ( Parker et al. , 2000 ) . With sufficiently large number of Monte Carlo trials and virtual precision t , the expected value of the output from these trials will equal the value from using real arithmetic . As explained in the next section , we can determine the total number of digits lost to rounding error and the minimum precision required to avoid a total loss of significance .

Summary: The authors propose a scalable method based on Monte Carlo arithmetic for quantifying the sensitivity of trained neural networks to floating point rounding errors. They demonstrate that the loss of significance metric K estimated from the process can be used for selecting networks that are more robust to quantization, and compare popular architectures (AlexNet, ResNet etc.) for their varying sensitivities.