A Block Minifloat Representation for Training Deep Neural Networks

1 INTRODUCTION . The energy consumption and execution time associated with training Deep Neural Networks ( DNNs ) is directly related to the precision of the underlying numerical representation . Most commercial accelerators , such as NVIDIA Graphics Processing Units ( GPUs ) , employ conventional floating-point representations due to their standard of use and wide dynamic range . However , double-precision ( FP64 ) and single-precision ( FP32 ) formats have relatively high memory bandwidth requirements and incur significant hardware overhead for general matrix multiplication ( GEMM ) . To reduce these costs and deliver training at increased speed and scale , representations have moved to 16-bit formats , with NVIDIA and Google providing FP16 ( IEEE-754 , 2019 ) and Bfloat16 ( Kalamkar et al. , 2019 ) respectively . With computational requirements for DNNs likely to increase , further performance gains are necessary in both datacenter and edge devices , where there are stricter physical constraints . New number representations must be easy to use and lead to high accuracy results . Recent 8-bit floating-point representations have shown particular promise , achieving equivalent FP32 accuracy over different tasks and datasets ( Wang et al. , 2018 ; Sun et al. , 2019 ) . We refer to such representations as minifloats in this paper . Minifloats are ideal candidates for optimization . By varying the number of exponent and mantissa bits , many formats can be explored for different trade-offs of dynamic range and precision . These include logarithmic and fixed point representations which provide substantial gains in speed and hardware density compared to their floating-point counterparts . For instance , 32-bit integer adders are approximately 10× smaller and 4× more energy efficient than comparative FP16 units ( Dally , 2015 ) . That said , fixed point representations still lack the dynamic range necessary to represent small gradients for backpropagation , and must be combined with other techniques for training convergence . Block floating point ( BFP ) in ( Yang et al. , 2019 ; Drumond et al. , 2018 ) share exponents across blocks of 8-bit integer numbers , and provide a type of coarse-grained dynamic range for training . This approach will typically incur some accuracy loss on more challenging datasets , however all dotproducts within the block can be computed with dense fixed point logic . In comparison , HFP8 ( Sun et al. , 2019 ) minifloats require larger floating-point units ( expensive FP16 adders in particular ) but have at least 5 exponent bits dedicated to each gradient and suffer zero degradation in training accuracy . It would seem that an ideal representation should bridge the gap between each of these approaches . Our work achieves this for 8-bit and sub 8-bit precision schemes , overcoming two key challenges in the process . These are listed below and discussed with related works . 1.1 CHALLENGES AND RELATED WORK . Minimising data loss with fewer bits : While several works have demonstrated training with fewer than 8 bits of precision , they typically lead to loss of accuracy on more complex problems and have performance bottlenecks because parts of the algorithm are left in high precision ( Hubara et al. , 2017 ; Zhou et al. , 2016 ; Miyashita et al. , 2016 ) . Therefore , training end-to-end with reduced precision representations that are persistent remains a key challenge . In this regard , 8 bit tensors with 16-bit updates can be trained effectively ( Banner et al. , 2018 ) . Data loss arises when formats do not have enough range to capture variations in tensor distributions during training . BFloat ( Kalamkar et al. , 2019 ) adds two extra exponent bits for a custom 16-bit representation , and the Apex library is used in ( Micikevicius et al. , 2017 ; Wang et al. , 2018 ; Sun et al. , 2019 ) for scaling the loss function into a numerically representable range . Block floating point and other variants apply similar functionality for fixed point numbers , but at a finer granularity . WAGE ( Wu et al. , 2018 ) uses layer-wise scaling factors , SWALP ( Yang et al. , 2019 ) shares exponents across feature maps and convolution channels , and HBFP ( Drumond et al. , 2018 ) does the same for dot-products , though their implementation requires caching of intermediate activations in FP32 and wide weight storage for better accuracy . S2FP8 ( Cambier et al. , 2020 ) replaces loss-scaling in FP8 ( Wang et al. , 2018 ) with squeeze and shift factors that center 8-bit minifloats over the mean exponent of the value distribution . Shift factors operate similarly to BFP shared exponents , whereas squeeze factors can divert precision away from high value regions leading to errors in dot-product calculations . We provide some empirical evidence of this effect in Section 4.5 . Finally , HFP8 ( Sun et al. , 2019 ) defines two minifloat formats that are optimized for range and precision requirements of forward and backward paths separately . In this work , we seek minifloat formats that are also optimized for arithmetic density . Increasing the performance density of floating-point : Most DNN training frameworks are developed with GEMM accumulation in FP32 . The authors in ( Wang et al. , 2018 ) reduced the accumulation width to FP16 with chunk-based computations and stochastic rounding . However , training minifloats with even denser dot-products has not been demonstrated . For DNN inference , ELMA ( Johnson , 2018 ) and posit number systems ( Gustafson & Yonemoto , 2017 ) describe arithmetic that accumulate minifloat-like numbers as integers . Such work is applicable when the number of exponent bits is small , however training under such regimes can lead to data loss due to limited dynamic range . 1.2 CONTRIBUTIONS . In this paper , we present the Block Minifloat ( BM ) representation which addresses both of the aforementioned challenges . BM is a modification of block floating point that replaces the fixed point values with minifloats , whilst maintaining shared exponents across blocks of numbers . BM formats generalise a far wider spectrum of reduced precision representations and produce better outcomes than previous 8-bit regimes . Specific contributions of our work include : • Block Minifloat ( BM ) , a more efficient alternative to INT8 and FP8 for end-to-end DNN training . Shared exponent biases provide dynamic range and accuracy , while small exponent encodings provide fine-grained dynamic range and reduce the hardware cost of GEMM accumulation . • A new 8-bit floating-point format that uses no more than 4 exponent bits , achieving equivalent accuracy to floating-point with denser hardware via efficient Kulisch accumulation . • An exploration of the BM design space showing high accuracy DNN training with sub 8-bit representations for all weights , activations and gradients . This includes two techniques for minimising data loss of a practical implementation , namely gradual underflow and cost-aware block designs . 50 |X| minifloat ( 3,2 ) log 2 |a| |a| value distribution Align with max . exponent ß 26 Figure 2 : Exponent bias shifts the minifloat distribution to align with the maximum exponent of the value distribution 2 BLOCK MINIFLOAT REPRESENTATION . 2.1 MINIFLOAT NUMBER FORMAT . Equation ( 1 ) computes the real value of a minifloat number , where ( e , m ) denote the number of exponent and mantissa bits in the representation . X 〈 e , m 〉 = { E = 0 , ( −1 ) s × 21−β × ( 0 + F × 2−m ) ( denormal ) otherwise , ( −1 ) s × 2E−β × ( 1 + F × 2−m ) ( normal ) ( 1 ) The decimal expansions of the exponent and mantissa are both unsigned integers , given by E and F respectively , s refers to the sign bit and β = 2e−1 − 1 is the exponent bias for the binaryoffset encoding scheme . This is consistent with IEEE-754 floating-point standards , except that our minifloats are considerably smaller ( 4-8 bits only ) , can generalise to multiple ( e , m ) configurations , and do not handle nan/infinity bit patterns . Instead , arithmetic is allowed to saturate at the limits of the representable range [ X+min , X + max ] . For example , a minifloat representation with X 〈 4 , 3 〉 have exponent and mantissas that range between [ 0 , 15 ] and [ 0 , 7 ] respectively . Therefore , the largest normal and smallest denormal positive numbers areX+max = 480 andX + min = 2 −9 . This corresponds to a dynamic range of 108 dB shown in Table 7 in Appendix A.1 . 2.2 SHARED EXPONENT BIAS . The main difference between minifloat and block minifloat ( BM ) representations are highlighted in Figure 1 . Minifloats have one exponent per element , but that exponent must be wide enough to tolerate changes in DNN tensor distributions during training ( i.e . 5 bits for gradients in FP8 ( Wang et al. , 2018 ) ) . In contrast , BM share exponent biases across blocks of N minifloat numbers . This provides equivalent dynamic range with respect to the block , but with fewer exponent bits than the original minifloat format . Block floating point ( BFP ) operates similarly , but all numbers within the block are integers ( Drumond et al. , 2018 ) . BM can generalise for this case , i.e . when e = 0 . The real value of the ith element from BM tensor a is given in Equation ( 2 ) , where X is an unbiased minifloat tensor , represented by ( e , m ) exponent and mantissa bits , and βa is the shared exponent bias . ai = Xi 〈 e , m 〉 × 2−βa ( 2 ) In this example , ai can only be represented accurately when the shared exponent bias βa ( calculated for the entire tensor ) and the distribution of X jointly captures the value distribution of a . For example , large and small values in a could saturate or be lost altogether if βa is too large or too small . However , some leeway exists when exponents are shared across dot products . This is because dot products are reduce operations , meaning their sum is dominated by the largest values in the inputs . For this reason , we calculate βa to specifically guard against overflow , and unlike ( Cambier et al. , 2020 ) we don ’ t apply any scaling which could divert precision away from larger value regions . Our method of updating β during training is illustrated in Figure 2 and formalized in Equation 3 below . βa = max ( blog2 |a|c ) − ( 2e − 1 ) ( 3 ) The first term denotes the maximum exponent for the tensor a , which changes and must be updated during training , while the second term is fixed and refers to the maximum exponent of X . In terms of hardware , shared biases ensure that all dot products can be computed with denser minifloat arithmetic . This is shown in Equation 4 for BM tensors a and b , each with N elements . a · b = N∑ i=1 ( ( Xai × 2−βa ) × ( Xbi × 2−βb ) ) = 2− ( βa+βb ) × ( Xa ·Xb ) ( 4 ) The dot product , Xa ·Xb , have minifloat formats with smaller exponents , while the cost of calculating , storing and aligning the exponent biases during training is amortized over the length of the dot-product . Next we show how minifloat formats with fewer exponent bits lead to faster and more compact hardware . 2.3 KULISCH ACCUMULATION In the above equations , kadd calculates the number of bits required for the largest product of two numbers , plus one extra bit for the addition , and kshift determines the maximum number of bits the mantissa product must be shifted to align with the addend . Crucially , by considering the size of kadd and kshift , BM formats can be designed to trade-off fine-grained dynamic range ( i.e . exponent bits ) for more precision and smaller hardware . In fact , formats with exponents up to 4 bits may yield kadd of approximately the same size as INT8/INT32 arithmetic units , while kadd becomes prohibitively wider and more expensive for larger exponents . This is clearly shown via example in Table 1 above , but more importantly , it is supported by hardware synthesis results given in Section 5 and Appendix A.4 . For example , an 8-bit minifloat format having 4 exponent bits achieves a 1.6× area reduction compared to HFP8 ( Sun et al. , 2019 ) with 5 exponent bits . Furthermore , through an extensive set of experiments we discover that such representations can also achieve high training accuracy which forms a key contribution of our work .

Summary: This paper introduced a new representation (Block Minifloat) for training DNNs with low precisions of 8-bit or less. This new representation combines FP8 formats and the shared exponent bias concept to cover the dynamic range of tensors needed for DNN training. Compared to other published FP8 format, this representation has smaller exponents, which allows to use a more efficient Kulisch accumulator. The representation has been verified on a spectrum of deep learning models and datasets.