Accumulation Bit-Width Scaling For Ultra-Low Precision Training Of Deep Networks

1 INTRODUCTION . Over the past decade , deep learning techniques have been remarkably successful in a wide spectrum of applications through the use of very large and deep models trained using massive datasets . This training process necessitates up to 100 ’ s of ExaOps of computation and Gigabytes of storage . It is , however , well appreciated that a range of approximate computing techniques can be brought to bear to significantly reduce this computational complexity ( Chen et al. , 2018 ) - and amongst them , exploiting reduced numerical precision during the training process is extremely effective and has already been widely deployed ( Gupta et al. , 2015 ) . There are several reasons why reduced precision deep learning has attracted the attention of both hardware and algorithms researchers . First , it offers well defined and scalable hardware efficiency , as opposed to other complexity reduction techniques such as pruning ( Han et al. , 2015b ; a ) , where handling sparse data is needed . Indeed , parameter complexity scales linearly while multiplication hardware complexity scales quadratically with precision bit-width ( Zhou et al. , 2016 ) . Thus , any advance towards truly binarized networks ( Hubara et al. , 2016 ) corresponds to potentially 30x - 1000x complexity reduction in comparison to single precision floating-point hardware . Second , the mathematics of reduced precision has direct ties with the statistical theory of quantization ( Widrow and Kollár , 2008 ) . In the context of deep learning , this presents an opportunity for theoreticians to derive analytical trade-offs between model accuracy and numerical precision ( Lin et al. , 2016 ; Sakr et al. , 2017 ) . ∗Work done while at IBM Research . Most ongoing efforts on reduced precision deep learning solely focus on quantizing representations and always assume wide accumulators , i.e. , ideal summations . The reason being reduced precision accumulation can result in severe training instability and accuracy degradation , as illustrated in Figure 1 ( a ) for ResNet 18 ( ImageNet ) model training . This is especially unfortunate , since the hardware complexity in reduced precision floating-point numbers ( needed to represent small gradients during training ) ( Wang et al. , 2018 ; Micikevicius et al. , 2017 ) is dominated by the accumulator bit-width . To illustrate this dominance we developed a model underpinned by the hardware synthesis of low-precision floating-point units ( FPU ) , that translates precision into area complexity of the FPU . Comparisons obtained from this model are shown in Figure 1 ( b ) . We observe that accumulating in high precision severely limits the hardware benefits of reduced precision computations . This presents a new angle to the problem of reduced precision deep learning training which concerns determining suitable accumulation precision and forms the basis of our paper . Our findings are that the accumulation precision requirements in deep learning training are nowhere near 32-b , and in fact could enable further complexity reduction of FPUs by a factor of 1.5 ∼ 2.2× . 1.1 RELATED WORKS . Our work is concerned with establishing theoretical foundations for estimating the accumulation bit precision requirements in deep learning . While this topic has never been addressed in the past , there are prior works in both deep learning and high performance computing communities that align well with ours . Most early works on reduced precision deep learning consider fixed-point arithmetic or a variation of it ( Gupta et al. , 2015 ) . However , when considering quantization of signals involving the backpropagation algorithm , finding a suitable fixed-point configuration becomes challenging due to a weak handle on the scalar dynamic range of the back-propagated signals . Thus , hardware solutions have been sought , and , accordingly , other number formats were considered . Flexpoint ( Köster et al. , 2017 ) is a hybrid version between fixed-point and floating-point where scalars in a tensor are quantized to 16-b fixed-point but share 5-b of exponenent to adjust the dynamic range . Similarly , WAGE ( Wu et al. , 2018 ) augments Flexpoint with stochastic quantization and enables integer quantization . All of these schemes focused on representation precision , but mostly used 32-bit accumulation . Another option is to use reduced precision floating-point as was done in MPT ( Micikevicius et al. , 2017 ) , which reduces the precision of most signals to 16-b floating-point , but observes accuracy degradation when reducing the accumulation precision from 32-b . Recently , Wang et al . ( 2018 ) quantize all representations to 8-b floating-point and experimentally find that the accumulation could be in 16-b with algorithmic contrivance , such as chunk-based accumulation , to enable convergence . The issue of numerical errors in floating-point accumulation has been classically studied in the area of high performance computing . Robertazzi and Schwartz ( 1988 ) were among the first to statistically estimate the effects of floating-point accumulation . Assuming a stream of uniformly and exponentially distributed positive numbers , estimates for the mean square error of the floating- point accumulation were derived via quantization noise analysis . Because such analyses are often intractable ( due to the multiplicative nature of the noise ) , later works on numerical stability focus on worst case estimates of the accumulation error . Higham ( 1993 ) provide upper bounds on the error magnitude by counting and analyzing round-off errors . Following this style of worst case analysis , Castaldo et al . ( 2008 ) provide bounds on the accumulation error for different summing algorithms , notably using chunk-based summations . Different approaches to chunking are considered and their benefits are estimated . It is to be noted that these analyses are often loose as they are agnostic to the application space . To the best of our knowledge , a statistical analysis on the accumulation precision specifically tailored to deep learning training remains elusive . 1.2 CONTRIBUTIONS . Our contribution is both theoretical and practical . We introduce the variance retention ratio ( VRR ) of a reduced precision accumulation in the context of the three deep learning dot products . The VRR is used to assess the suitability , or lack thereof , of a precision configuration . Our main result is the derivation of an actual formula for the VRR that allows us to determine accumulation bit-width for precise tailoring of computation hardware . Experimentally , we verify the validity and tightness of our analysis across three benchmarking networks ( CIFAR-10 ResNet 32 , ImageNet ResNet 18 and ImageNet AlexNet ) . 2 BACKGROUND ON FLOATING-POINT ARITHMETIC . The following basic floating-point definitions and notations are used in our work : Floating-point representation : A b-bit floating-point number a has a signed bit , e exponent bits , and m mantissa bits so that b = 1 + e + m. Its binary representation is ( Bs , B ′ 1 , . . . , B ′ e , B ′′ 1 , . . . , B ′′ m ) ∈ { 0 , 1 } b and its value is equal to : a = ( −1 ) Bs × 2E × ( 1 + M ) where E = − ( 2e−1 − 1 ) + ∑e i=1B ′ i2 ( e−i ) and M = ∑m i=1B ′′ i 2 −i . Such number is called a ( 1 , e , m ) floating-point number . Floating-point operations : One of the most pervasive arithmetic functions used in deep learning is the dot product between two vectors which is the building block of the generalized matrix multiplication ( GEMM ) . A dot product is computed in a multiply-accumulate ( MAC ) fashion and thus requires two floating-point operations : multiplication and addition . The realization of an ideal floating-point operation requires a certain bit growth at the output to avoid loss of information . For instance , in a typical MAC operation , if c ← c + a × b where a is ( 1 , ea , ma ) and b is ( 1 , eb , mb ) , then c should be ( 1 , max ( ea , eb ) + 2 , ma + mb + 1 + ∆E ) , which depends on the bit-precision and the relative exponent difference of the operands ∆E . However , it is often more practical to pre-define the precision of c as ( 1 , ec , mc ) , which requires rounding immediately after computation . Such rounding might cause an operand to be completely or partially truncated out of the addition , a phenomenon called ” swamping ” ( Higham , 1993 ) , which is the primary source of accumulation errors . 3 ACCUMULATION VARIANCE . The second order statistics ( variance ) of signals are known to be of great importance in deep learning . For instance , in prior works on weight initialization ( Glorot and Bengio , 2010 ; He et al. , 2015 ) , it is customary to initialize random weights subject to a variance constraint designed so as to prevent vanishing or explosion of activations and gradients . Thus , such variance engineering induces fine convergence of DNNs . Importantly , in such analyses , the second order output statistics of a dot product are studied and expressed as a function of that of the accumulated terms , which are assumed to be independent and having similar variance . A fundamental assumption is : V ar ( s ) = nV ar ( p ) , where V ar ( s ) and V ar ( p ) are the variances of the sum and individual product terms , respectively , and n is the length of the dot product . One intuition concerning accumulation with reduced precision is that , due to swamping , some product terms may vanish from the summation , resulting in a lower variance than expected : V ar ( s ) = ñV ar ( p ) , where ñ < n. This constitutes a violation of a key assumption and effectively leads to the re-emergence of the difficulties in training neural networks with improper weight initialization which often harms the convergence behavior ( Glorot and Bengio , 2010 ; He et al. , 2015 ) . Input Feature Maps ( IFM ) Output Feature Maps ( OFM ) IFM Width ( IW ) OFM Width ( OW ) IF M H ei gh t ( IH ) O FM H ei gh t ( O H ) Input Channel Number ( ICN ) Output Channel Number ( OCN ) Kernel Width ( KW ) Ke rn el H ei gh t ( K H ) Accumulation length = ICN x KW x KH ( a ) Input Feature Maps ( IFM ) Derivatives Output Feature Maps ( OFM ) Derivatives IFM Width ( IW ) OFM Width ( OW ) IF M H ei gh t ( IH ) O FM H ei gh t ( O H ) Output Channel Number ( OCN ) Kernel Width ( KW ) Kernel Height ( KH ) Accumulation length = OCN x KW x KH Input Channel Number ( ICN ) ( b ) Elementwise Product of Input Activations and Output Derivatives Product Width ( PW ) Pr od uc t H ei gh t ( P H ) Accumulation length = BS x PW x PH Weight Gradients ( 2D Slice ) Batch Size ( BS ) ( c ) Figure 2 : The three GEMM calls , and hence accumulations , in one iteration of the back-propagation algorithm : ( a ) the forward propagation ( FWD ) , ( b ) the backward propagation ( BWD ) , and ( c ) gradient computation ( GRAD ) . The accumulation of these three GEMMs is across multiple dimensions ( mini-batch size , feature maps , output channels etc . ) and their lengths are usually very long . To explain the poor convergence of our ResNet 18 run ( Figure 1 ( a ) ) , we evaluate the behavior of accumulation variance across layers . Specifically , we check the three dot products of a backpropagation iteration : the forward propagation ( FWD ) , the backward propagation ( BWD ) , and the gradient computation ( GRAD ) , as illustrated in Figure 2 . Indeed , there is an abnormality in reduced precision GRAD as shown in Figure 3 . It is also observed that the abnormality of variance is directly related to accumulation length . From Figure 3 , the break point corresponds to the switch from the first to the second residual block . The GRAD accumulation length in the former is much longer ( 4× ) than the latter . Thus , evidence points to the direction that for a given precision , there is an accumulation length for which the expected variance can not be properly retained due to swamping . Motivated by these observations , we propose to study the trade-offs among accumulation variance , length , and mantissa precision . Before proceeding , it is important to note that our upcoming analysis differs , in style , from many works on reduced precision deep learning where it is common to model quantization effects as additive noise causing increased variance ( McKinstry et al. , 2018 ) . Our work does not contradict such findings , since prior arts have considered representation quantization whose effects are , by nature , different from intermediate roundings in partial sums .

Summary: There has been a lot of work on limited precision training and inference for deep learning hardware, but in most of this work, the accumulators for the multiply-and-add (FMA) operations that occur for inner products are chosen conservatively or treated as having unlimited precision. The authors address this with  an analytical method to predict the number of mantissa bits needed for partial summations during the forward, delta and gradient computation ops for convolutional and fully connected layers. They propose an information theoretic approach to argue that by using fewer bits of mantissa in the accumulator than necessary, the variance of the resulting sum is less than what it would have been if sufficient bits of mantissa were used. This is surprising to me, as quantization is usually modeled as _adding_ noise, leading to an _increase_ in variance (Mc Kinstry et al. 2018), so this is a nice counterexample to that intuition. Unfortunately the result is presented in a way that implies the variance reduction is what causes the degradation in performance, while obviously (?) it's just a symptom of a deeper problem. E.g., adding noise or multiplying by a constant to get the variance to where it should be, will not help the network converge. The variance is just a proxy for lost information. The authors should make this more clear.