Monte Carlo Deep Neural Network Arithmetic

1 INTRODUCTION . Deep Neural Networks have achieved state-of-the-art performances in many machine learning tasks such as such as speech recognition ( Collobert et al. , 2011 ) , machine translation ( Bahdanau et al. , 2014 ) , object detection ( Ren et al. , 2015 ) and image classification ( Krizhevsky et al. , 2012 ) . However , excellent performance comes at the cost of significantly high computational and memory complexity , typically requiring teraops of computation during inference and Gigabytes of storage . To overcome these complexities , compression methods have been utilized , aiming to exploit the inherent resilience of DNNs to noise . These engender representations which maintain algorithm performance but significantly improve latency , throughput and power consumption of hardware implementations . In particular , exploiting reduced numerical precision for data representations through quantization has been emphatically promising , whereby on customizable hardware , efficiency scales quadratically with each bit of precision . Quantization of fixed-point arithmetic ( Q-FX ) for DNN inference has been extensively studied , and more recently there has been increasing interest in quantized floating point ( Q-FP ) arithmetic for both DNN inference and training ( Wang et al. , 2018 ) . Q-FP has the advantage of higher dynamic range compared to equivalent Q-FX representations and reduced hardware cost over single-precision floating point ( FP ) . This has influenced application specific integrated circuits ( ASICs ) such as Google ’ s tensor processing unit ( TPU ) , which supports 16-bit floating point ( 16-FP ) and soft processors such as Microsoft ’ s Project Brainwave which utilizes 8-FP . To illustrate these hardware benefits , we synthesized arithmetic logic units ( ALUs ) in different formats and different precision on an FPGA and present performance estimates in operations per second ( OPs ) and area estimates in Look-up Tables ( LUTs ) per operation ( LUTs/Op ) in Figure 1 . As shown , 8-bit fixed point ( 8-FX ) achieves improved performance and area over 8-FP . However , 7-FP is a significant improvement over 8 or 12-FX and 8 or 9-FP . These examples demonstrate substantial performance and area benefits from reducing FP precision by only 1 to 2-bits . Thus , if we 1Source code will be available if the paper is accepted can design networks which not only achieve high accuracy but are robust to quantization , higher performing hardware solutions are possible . Since in Q-FP , we are trying to represent the infinite set of real numbers using a finite number of bits , quantization and rounding artefacts will be introduced , with inaccuracies being cascaded along the computation graph ( IEEE , 1985 ; Goldberg , 1991 ; Higham , 2002 ) . This paper proposes Monte Carlo Deep Neural Network Arithmetic ( MCDA ) , a novel way to apply Monte Carlo Arithmetic ( MCA ) ( Parker et al. , 2000 ) for determining the sensitivity of Deep Neural Networks to Q-FP representations . It allows hardware-software designers to quantify the impact of quantization , enabling more efficient systems to be discovered . We do this by exploiting Monte Carlo simulations under which rounding effects are randomized . This , in turn , allows one to infer the sensitivity of executing a computation graph to quantization effects . Our MCDA technique is highly sensitive , allowing very small differences in quantization behaviour to be detected . The technique makes no assumptions regarding data distributions and directly measures the effects of quantization on the problem under study . This allows us to provide insights into the precision requirements of any inference network for any given dataset . Additionally we can use the technique to select weight parameters which are more robust to floating point rounding . The theoretical and practical contributions of this work can be summarized as follows : • We introduce a novel and rigorous analysis technique , Monte Carlo Deep Neural Network Arithmetic ( MCDA ) , which measures the sensitivity of Deep Neural Network inference computation to floating point rounding error . • When applied to neural network inference , we show MCDA can determine the precision requirements of different networks , rank them , and detect small differences between different neural network topologies and weight sets . • We demonstrate that while a network with the same topology but different weights may have the same loss and validation accuracy , their sensitivity to quantization can be vastly different . Using the CIFAR-10 and ImageNet datasets , we introduce a method to choose weights which are more robust to rounding error , resulting in a greatly improved accuracyarea tradeoff over state-of-the-art methods . It is worth noting that although we consider convolutional neural networks for image classification , this method could be applied for any neural network model architectures and applications . Moreover , while the experiments in this paper are limited to inference , it may be possible to apply the same idea to analyze training algorithms . 2 RELATED WORK . Low-precision representations of deep learning have been extensively studied . Many training methods have been developed to design representations for fixed point inference ( Jacob et al. , 2018 ; Faraone et al. , 2018 ; Zhou et al. , 2016 ) and training ( Wu et al. , 2018b ; Yang et al. , 2019 ; Gupta et al. , 2015 ; Sakr & Shanbhag , 2019 ) . Other methods have also utilized Q-FP arithmetic for inference and training whilst maintaining single-precision accuracy . ( Micikevicius et al. , 2018 ) implemented 16- FP arithmetic training whilst storing a 32-FP master copy for the weight updates . Additionally , ( Wang et al. , 2018 ; Mellempudi et al. , 2019 ) with 8-FP arithmetic whilst using a 16-FP copy of the weights and 16/32-bits for the accumulator . Techniques for determining per-layer sensitivity to quantization have also been studied ( Choi et al. , 2016 ; Sakr & Shanbhag , 2018 ) . Further , other studies have successfully determined the minimum fixed point precision requirements for a given DNN accuracy threshold ( Sakr et al. , 2017 ) . The accuracy and stability of various numerical algorithms in finite precision arithmetic has been studied in ( Higham , 2002 ; Wilkinson , 1994 ) . This has led to techniques for tracking information lost from finite precision arithmetic using random perturbation such as Monte Carlo Arithmetic ( Parker et al. , 2000 ; Frechtling & Leong , 2015 ) . Monte Carlo methods have also been used in Bayesian Neural Networks ( Buchholz et al. , 2018 ; Blier & Ollivier , 2018 ) . In particular , ( Achterhold et al. , 2018 ) introduced a quantizing prior to learn weights which are either close to a quantized representation or have high variance . ( Louizos et al. , 2017 ) used hierarchical priors to prune nodes and posterior uncertainties to determine the optimal fixed point precision . ( Blundell et al. , 2015 ) use a Monte Carlo approach to learn a probability distribution on the weights of a neural network . To the best of our knowledge , our work is the first to present a technique for directly determining the sensitivity of DNNs to floating point rounding and to explicitly compute precision bounds of a trained network . These ideas can be very usefully applied to extending the limits of low-precision representations in deep learning applications . 3 BACKGROUND . In this section , we describe background theory upon which our technique for determining the sensitivity of DNNs to floating point rounding is based . 3.1 FLOATING POINT ARITHMETIC . The IEEE-754 binary floating point format ( IEEE , 1985 ) represents most real numbers x by a subset in normal form as : x̂ = ( −1 ) sx ( 1 +mx ) 2ex ( 1 ) where sx ∈ { 0 , 1 } is the sign bit , ex is an integer representing the exponent of x̂ and mx is the mantissa of x̂ . Such number formats can be described as a ( sx , ex , mx ) tuple . In binary form the representation is ( bs , be1 , b e 2 , ... , b e Bex , bm1 , b m 2 ... , b m Bmx ) ∈ { 0 , 1 } B , with Bex and Bmx being the number of exponent and mantissa bits , respectively . The infinite set of real numbers R is represented in a computer withB = 1+Bex+Bmx bits , and we define the finite set of real numbers representable in floating point format as exact values , F ⊂ R. Real numbers which aren ’ t representable are rounded to their nearest exact value . We call this set of numbers inexact values , I , where I∪F = R. The approximation x̂ = F ( x ) = x ( 1 + δ ) , given x ∈ I , introduces rounding error into the computation . The value of δ = ∥∥x−x̂ x ∥∥ , represents the relative error which is a function of the machine hardware precision , p , as δ ≤ , where = 2−p ( IEEE , 1985 ; Goldberg , 1991 ; Higham , 2002 ) . In general , inexactness can be caused by finite representations or errors propagating from earlier parts of the computation . Often the primary cause of error in floating point arithmetic is catastrophic cancellation which causes numerical inaccuracy . Catastrophic cancellation occurs when for example , two near equal FP numbers , sharing k significant digits , are subtracted from one another as shown in ( 2 ) ( Higham , 2002 ) . 0. f1 f2 ... fk f ( k+1 ) ... ft − 0. f1 f2 ... fk g ( k+1 ) ... gt = 0 . 0 0 .... 0 h ( k+1 ) ... ht ( 2 ) 0. f1 f2 ... fk f ( k+1 ) ... ft r ( t+1 ) ... rp − 0. f1 f2 ... fk g ( k+1 ) ... gt r̂ ( t+1 ) ... r̂p = 0 . 0 0 .... 0 h ( k+1 ) ... ht i ( t+1 ) ... ip ( 3 ) In normalized form , the leading zeros are removed by shifting the result to the left and adjusting the exponent accordingly . The result is 0.hk+1 ... hti1 ... ik which has only ( t − k ) accurate digits and digits i which are unknown . Additionally , the remaining accurate digits h are most likely affected by rounding error in previous computations . This can significantly magnify errors , especially in computing large computational graphs such as that of state-of-the-art DNNs . If either operand in ( 2 ) is inexact , then the digits h are no more significant than any other sequence of digits . Yet , FP arithmetic has no mechanism of recording this loss of significance . By padding both our operands with random digits r and r̂ in ( 3 ) , the resulting digits i are randomized . If k digits are lost in the result , then k random digits will be in the normalized result and when computed over many random trials , the results will disagree on the trailing k digits . In this case , we are able to detect catastrophic cancellation because the randomization over many trials provides a statistical simulation of round-off errors . We can use techniques from numerical analysis such as Monte Carlo methods to appropriately insert precision-dependant randomization in this way . 3.2 MONTE CARLO ARITHMETIC . Monte Carlo methods can be used to analyze rounding by representing inexact values as random variables ( Parker et al. , 2000 ; Frechtling & Leong , 2015 ) . The real value x , as represented in ( 1 ) , can be modelled to t digits , using : inexact ( x̂ , t , δ ) = x̂+ 2ex−tδ = ( −1 ) sx ( 1 +mx + 2−tδ ) 2ex ( 4 ) where δ ∈ U ( − 12 , 1 2 ) is a uniformly distributed random variable and t is a positive integer representing the virtual precision of concern . For the same input x̂ in ( 4 ) , we can run many Monte Carlo trials which will yield different values on each trial , where 0 < t < p so that the MCA can be run accurately on a computer with machine precision p. The ability to vary t is useful because it allows us to then evaluate the hardware precision requirements of a given system or computational graph for a given DNN . MCA is a method to model the effect of rounding on a computational graph by randomizing all arithmetic operations . The randomization is applied for both generating inexact operands and also in rounding . In each operation using MCA , ideally both catastrophic cancellation and rounding error can be detected . An operation using MCA is defined as : x ◦ y = round ( inexact ( inexact ( x ) ◦ inexact ( y ) ) ) ( 5 ) where ◦ ∈ ( + , − , × , ÷ ) . By applying the inexact function to both operators we make it possible to detect catastrophic cancellation . Furthermore , applying the inexact function to the operation output and then rounding this value implements random rounding and hence is used to detect rounding error ( Parker & Langley , 1997 ) . Hence , for the same input into the system , each trial will yield different operands and output . After modifying the inexact and rounding operations as described , we use random sampling to simulate Monte Carlo trials . For each trial , we collect data on the resulting output of the system and compute summary statistics to quantify its behaviour ( Parker et al. , 2000 ) . With sufficiently large number of Monte Carlo trials and virtual precision t , the expected value of the output from these trials will equal the value from using real arithmetic . As explained in the next section , we can determine the total number of digits lost to rounding error and the minimum precision required to avoid a total loss of significance .

Summary: The paper studies the sensitivity of a neural network with respect to quantizing its weights and activations. The idea is to use Monte Carlo Arithmetic (MCA) in order to calculate the number of significant bits in the training loss (e.g. cross entropy) that are lost due to floating-point arithmetic. The results show that the number of significant bits lost correlates with the reduction in classification accuracy when quantizing the weights and activations of the neural network.