A Block Minifloat Representation for Training Deep Neural Networks

1 INTRODUCTION . The energy consumption and execution time associated with training Deep Neural Networks ( DNNs ) is directly related to the precision of the underlying numerical representation . Most commercial accelerators , such as NVIDIA Graphics Processing Units ( GPUs ) , employ conventional floating-point representations due to their standard of use and wide dynamic range . However , double-precision ( FP64 ) and single-precision ( FP32 ) formats have relatively high memory bandwidth requirements and incur significant hardware overhead for general matrix multiplication ( GEMM ) . To reduce these costs and deliver training at increased speed and scale , representations have moved to 16-bit formats , with NVIDIA and Google providing FP16 ( IEEE-754 , 2019 ) and Bfloat16 ( Kalamkar et al. , 2019 ) respectively . With computational requirements for DNNs likely to increase , further performance gains are necessary in both datacenter and edge devices , where there are stricter physical constraints . New number representations must be easy to use and lead to high accuracy results . Recent 8-bit floating-point representations have shown particular promise , achieving equivalent FP32 accuracy over different tasks and datasets ( Wang et al. , 2018 ; Sun et al. , 2019 ) . We refer to such representations as minifloats in this paper . Minifloats are ideal candidates for optimization . By varying the number of exponent and mantissa bits , many formats can be explored for different trade-offs of dynamic range and precision . These include logarithmic and fixed point representations which provide substantial gains in speed and hardware density compared to their floating-point counterparts . For instance , 32-bit integer adders are approximately 10× smaller and 4× more energy efficient than comparative FP16 units ( Dally , 2015 ) . That said , fixed point representations still lack the dynamic range necessary to represent small gradients for backpropagation , and must be combined with other techniques for training convergence . Block floating point ( BFP ) in ( Yang et al. , 2019 ; Drumond et al. , 2018 ) share exponents across blocks of 8-bit integer numbers , and provide a type of coarse-grained dynamic range for training . This approach will typically incur some accuracy loss on more challenging datasets , however all dotproducts within the block can be computed with dense fixed point logic . In comparison , HFP8 ( Sun et al. , 2019 ) minifloats require larger floating-point units ( expensive FP16 adders in particular ) but have at least 5 exponent bits dedicated to each gradient and suffer zero degradation in training accuracy . It would seem that an ideal representation should bridge the gap between each of these approaches . Our work achieves this for 8-bit and sub 8-bit precision schemes , overcoming two key challenges in the process . These are listed below and discussed with related works . 1.1 CHALLENGES AND RELATED WORK . Minimising data loss with fewer bits : While several works have demonstrated training with fewer than 8 bits of precision , they typically lead to loss of accuracy on more complex problems and have performance bottlenecks because parts of the algorithm are left in high precision ( Hubara et al. , 2017 ; Zhou et al. , 2016 ; Miyashita et al. , 2016 ) . Therefore , training end-to-end with reduced precision representations that are persistent remains a key challenge . In this regard , 8 bit tensors with 16-bit updates can be trained effectively ( Banner et al. , 2018 ) . Data loss arises when formats do not have enough range to capture variations in tensor distributions during training . BFloat ( Kalamkar et al. , 2019 ) adds two extra exponent bits for a custom 16-bit representation , and the Apex library is used in ( Micikevicius et al. , 2017 ; Wang et al. , 2018 ; Sun et al. , 2019 ) for scaling the loss function into a numerically representable range . Block floating point and other variants apply similar functionality for fixed point numbers , but at a finer granularity . WAGE ( Wu et al. , 2018 ) uses layer-wise scaling factors , SWALP ( Yang et al. , 2019 ) shares exponents across feature maps and convolution channels , and HBFP ( Drumond et al. , 2018 ) does the same for dot-products , though their implementation requires caching of intermediate activations in FP32 and wide weight storage for better accuracy . S2FP8 ( Cambier et al. , 2020 ) replaces loss-scaling in FP8 ( Wang et al. , 2018 ) with squeeze and shift factors that center 8-bit minifloats over the mean exponent of the value distribution . Shift factors operate similarly to BFP shared exponents , whereas squeeze factors can divert precision away from high value regions leading to errors in dot-product calculations . We provide some empirical evidence of this effect in Section 4.5 . Finally , HFP8 ( Sun et al. , 2019 ) defines two minifloat formats that are optimized for range and precision requirements of forward and backward paths separately . In this work , we seek minifloat formats that are also optimized for arithmetic density . Increasing the performance density of floating-point : Most DNN training frameworks are developed with GEMM accumulation in FP32 . The authors in ( Wang et al. , 2018 ) reduced the accumulation width to FP16 with chunk-based computations and stochastic rounding . However , training minifloats with even denser dot-products has not been demonstrated . For DNN inference , ELMA ( Johnson , 2018 ) and posit number systems ( Gustafson & Yonemoto , 2017 ) describe arithmetic that accumulate minifloat-like numbers as integers . Such work is applicable when the number of exponent bits is small , however training under such regimes can lead to data loss due to limited dynamic range . 1.2 CONTRIBUTIONS . In this paper , we present the Block Minifloat ( BM ) representation which addresses both of the aforementioned challenges . BM is a modification of block floating point that replaces the fixed point values with minifloats , whilst maintaining shared exponents across blocks of numbers . BM formats generalise a far wider spectrum of reduced precision representations and produce better outcomes than previous 8-bit regimes . Specific contributions of our work include : • Block Minifloat ( BM ) , a more efficient alternative to INT8 and FP8 for end-to-end DNN training . Shared exponent biases provide dynamic range and accuracy , while small exponent encodings provide fine-grained dynamic range and reduce the hardware cost of GEMM accumulation . • A new 8-bit floating-point format that uses no more than 4 exponent bits , achieving equivalent accuracy to floating-point with denser hardware via efficient Kulisch accumulation . • An exploration of the BM design space showing high accuracy DNN training with sub 8-bit representations for all weights , activations and gradients . This includes two techniques for minimising data loss of a practical implementation , namely gradual underflow and cost-aware block designs . 50 |X| minifloat ( 3,2 ) log 2 |a| |a| value distribution Align with max . exponent ß 26 Figure 2 : Exponent bias shifts the minifloat distribution to align with the maximum exponent of the value distribution 2 BLOCK MINIFLOAT REPRESENTATION . 2.1 MINIFLOAT NUMBER FORMAT . Equation ( 1 ) computes the real value of a minifloat number , where ( e , m ) denote the number of exponent and mantissa bits in the representation . X 〈 e , m 〉 = { E = 0 , ( −1 ) s × 21−β × ( 0 + F × 2−m ) ( denormal ) otherwise , ( −1 ) s × 2E−β × ( 1 + F × 2−m ) ( normal ) ( 1 ) The decimal expansions of the exponent and mantissa are both unsigned integers , given by E and F respectively , s refers to the sign bit and β = 2e−1 − 1 is the exponent bias for the binaryoffset encoding scheme . This is consistent with IEEE-754 floating-point standards , except that our minifloats are considerably smaller ( 4-8 bits only ) , can generalise to multiple ( e , m ) configurations , and do not handle nan/infinity bit patterns . Instead , arithmetic is allowed to saturate at the limits of the representable range [ X+min , X + max ] . For example , a minifloat representation with X 〈 4 , 3 〉 have exponent and mantissas that range between [ 0 , 15 ] and [ 0 , 7 ] respectively . Therefore , the largest normal and smallest denormal positive numbers areX+max = 480 andX + min = 2 −9 . This corresponds to a dynamic range of 108 dB shown in Table 7 in Appendix A.1 . 2.2 SHARED EXPONENT BIAS . The main difference between minifloat and block minifloat ( BM ) representations are highlighted in Figure 1 . Minifloats have one exponent per element , but that exponent must be wide enough to tolerate changes in DNN tensor distributions during training ( i.e . 5 bits for gradients in FP8 ( Wang et al. , 2018 ) ) . In contrast , BM share exponent biases across blocks of N minifloat numbers . This provides equivalent dynamic range with respect to the block , but with fewer exponent bits than the original minifloat format . Block floating point ( BFP ) operates similarly , but all numbers within the block are integers ( Drumond et al. , 2018 ) . BM can generalise for this case , i.e . when e = 0 . The real value of the ith element from BM tensor a is given in Equation ( 2 ) , where X is an unbiased minifloat tensor , represented by ( e , m ) exponent and mantissa bits , and βa is the shared exponent bias . ai = Xi 〈 e , m 〉 × 2−βa ( 2 ) In this example , ai can only be represented accurately when the shared exponent bias βa ( calculated for the entire tensor ) and the distribution of X jointly captures the value distribution of a . For example , large and small values in a could saturate or be lost altogether if βa is too large or too small . However , some leeway exists when exponents are shared across dot products . This is because dot products are reduce operations , meaning their sum is dominated by the largest values in the inputs . For this reason , we calculate βa to specifically guard against overflow , and unlike ( Cambier et al. , 2020 ) we don ’ t apply any scaling which could divert precision away from larger value regions . Our method of updating β during training is illustrated in Figure 2 and formalized in Equation 3 below . βa = max ( blog2 |a|c ) − ( 2e − 1 ) ( 3 ) The first term denotes the maximum exponent for the tensor a , which changes and must be updated during training , while the second term is fixed and refers to the maximum exponent of X . In terms of hardware , shared biases ensure that all dot products can be computed with denser minifloat arithmetic . This is shown in Equation 4 for BM tensors a and b , each with N elements . a · b = N∑ i=1 ( ( Xai × 2−βa ) × ( Xbi × 2−βb ) ) = 2− ( βa+βb ) × ( Xa ·Xb ) ( 4 ) The dot product , Xa ·Xb , have minifloat formats with smaller exponents , while the cost of calculating , storing and aligning the exponent biases during training is amortized over the length of the dot-product . Next we show how minifloat formats with fewer exponent bits lead to faster and more compact hardware . 2.3 KULISCH ACCUMULATION In the above equations , kadd calculates the number of bits required for the largest product of two numbers , plus one extra bit for the addition , and kshift determines the maximum number of bits the mantissa product must be shifted to align with the addend . Crucially , by considering the size of kadd and kshift , BM formats can be designed to trade-off fine-grained dynamic range ( i.e . exponent bits ) for more precision and smaller hardware . In fact , formats with exponents up to 4 bits may yield kadd of approximately the same size as INT8/INT32 arithmetic units , while kadd becomes prohibitively wider and more expensive for larger exponents . This is clearly shown via example in Table 1 above , but more importantly , it is supported by hardware synthesis results given in Section 5 and Appendix A.4 . For example , an 8-bit minifloat format having 4 exponent bits achieves a 1.6× area reduction compared to HFP8 ( Sun et al. , 2019 ) with 5 exponent bits . Furthermore , through an extensive set of experiments we discover that such representations can also achieve high training accuracy which forms a key contribution of our work .

Summary: The authors proposes block-minifloat (BM), a floating-point format for DNN training. BM is a fairly simple extension to block floating-point (BFP), which was proposed in (Drumond 2018 and Yang 2019). In BFP, a block of integer mantissas share a single exponent. In BM, a block of narrow floats share a single exponent bias. The shared exponent bias helps to shorten the exponent field on each individual float element. This is a good contribution, though a bit trivial.