Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks

1 INTRODUCTION . Deep neural networks have achieved state-of-the-art performance on a wide variety of computer vision , audio , and natural language processing ( NLP ) tasks . This has resulted in an explosion of interest around techniques to reduce the memory footprint and energy consumption of neural network training and inference ( Guo , 2018 ) . Although there are a number of methods to address some of these issues for inference , the most effective method for training is using reduced precision numerical formats . While 32-bit floating point ( FP32 ) is the most common data format for neural network training , recent hardware have leveraged techniques that allow for training with 16-bit data formats ( Köster et al. , 2017 ; Micikevicius et al. , 2018 ) . However , 8-bit precision training remains an open challenge ( Johnson , 2018 ; Kalamkar et al. , 2019 ) . Current FP8 training methodologies ( Wang et al. , 2018 ; Mellempudi et al. , 2019 ) require either specialized chunk-based accumulation , stochastic rounding techniques , loss scaling or maintaining some layers of the network in higher precision . Tuning these knobs is non-intuitive and requires significant experimentation for each individual network . Accelerating the adoption of 8-bit data in training DNNs requires a hardware-friendly and out-ofthe-box implementation of FP8 . Due to the reduced number of mantissa bits , 8-bit multipliers are smaller and consume less power compared to higher bit representations . In this work we describe a novel 8-bit floating point ( FP8 ) format - shifted and squeezed FP8 ( S2FP8 ) - which has the following advantages compared to previously proposed 8-bit training methodologies : ∗Work performed during an internship at Intel †Equal contribution • S2FP8 eliminates the need for loss scaling , which requires significant tuning of the loss scale values and schedule for individual topologies • Leveraged by the forward and backward passes of model training , S2FP8 is effective in adjusting the range of gradients and also of activations and weights • S2FP8 does not require keeping the first and last layer in FP32 precision , which is needed for other approaches ( Mellempudi et al. , 2019 ) , however maintains the master weights and accumulations inside the matrix multipliers in FP32 We demonstrate across image classification , translation , and recommendation models that S2FP8 outperforms previous 8-bit approaches , and reaches the accuracy of FP32 models without any additional hyperparameter tuning . 2 RELATED WORK . The success of 32-bit floating point data type in training deep neural networks has increased interest in the feasibility of even lower precision training . The exponential demand for compute involved in training these deep neural networks has lead to multiple advancements in lower precision data types . Several studies have developed techniques such as loss scaling , stochastic rounding , and others to train effectively in 16-bit ( Micikevicius et al. , 2018 ; Das et al. , 2018 ; Azim ) , along with associated hardware support ( Markidis et al. , 2018 ) . Using 16-bit fixed point , ( Gupta et al. , 2015 ) showed that stochastic rounding techniques were crucial for model convergence even for simple convolutional neural networks . As noted in ( Kalamkar et al. , 2019 ) , Google ’ s bfloat16 format has the same number of exponent bits as FP32 , leading the success of that format without commonly requiring hardware intensive requirements such as stochastic rounding or other framework level techniques such as loss scaling . Although 8-bit formats have significant performance and memory advantages , convergence is especially challenging due to loss of accuracy in the backpropogated gradient values . Wang et al . ( 2018 ) demonstrated training models with matrix multiplications and convolutions in FP8 but they use FP16 with chunk-based accumulations and stochastic rounding hardware . Mellempudi et al . ( 2019 ) also demonstrated success with FP8 , accumulating in FP32 and using loss scaling techniques on ResNets , Transformer and GNMT networks . However , they too require the first and last layers of the model to be in FP32 , and similar to ( Banner et al. , 2018 ) leverage Stochastic Rounding techniques to maintain model accuracy . Unlike S2FP8 proposed in this work , both of these FP8 training techniques emphasize the need for efficient loss scaling , rounding hardware and restriction on some layers being in higher precision . Zhou et al . ( 2016 ) quantized weights , activations and gradients of AlexNet ( Krizhevsky et al. , 2012 ) to 1 , 2 and 6 bits respectively . But they also need to maintain the first and last convolution layers in full precision and stochastically quantize the gradients . Wu et al . ( 2018 ) demonstrate using integers for training LeNet-5 ( LeCun et al. , 1998 ) and AlexNet with 8-bits for activations , error and gradients and 2-bits for weights . However , these approaches also required custom tuning such as novel initialization techniques and layer wise scaling instead of Batch Normalization and Softmax . These approaches lack generalizability to other models , requiring significant fine tuning . To the best of our knowledge , there does not exist an out-of-the-box solution using FP8 in training deep learning topologies without the need for tuned loss scaling techniques , requirements of certain layers being in full precision along with efficient hardware rounding schemes like Stochastic Rounding . 3 SHIFTED AND SQUEEZED 8-BIT FLOATING POINT FORMAT . 3.1 CHALLENGES OF 8-BIT FLOATING POINT FORMAT . The FP8 format , with 2 bits of mantissa and 5 bits of exponent ( Mellempudi et al. , 2019 ) is both narrow ( i.e. , its dynamic range is very limited , from 2−16 to 216 ) and has lower accuracy ( the machine epsilon is only 2−3 ) . Figure A1 illustrates the range and accuracy of FP8 . In contrast , FP32 ranges from 2−149 to 2128 with a machine-epsilon of 2−24 ( Table A1 ) . On the other hand , tensors involved in neural networks ( weights , activations and gradients ) are spread across varying scales . As illustrated in Figure 1 , the tensor distributions change over the course of training , spanning different orders of magnitude . As a result , 8-bit training usually requires a combination of multiple techniques to capture the full dynamic range of values for model training . Some of these techniques include : • Loss scaling ( Micikevicius et al. , 2018 ) scales the loss L ( w ) by a constant λ before backpropagation . This makes the gradients artificially larger , allowing them to fit within the FP8 range . Gradients are then scaled down before being accumulated into the trainable weights as shown in Equation 6 • Stochastic rounding ( Maxfield , 2006 ) alleviate quantization errors by capturing some of the information discarded when truncating to lower precision at the output of a GEMM operation Between these two techniques , loss scaling is more critical ; once the magnitude of the gradients can no longer be represented in the FP8 range , training convergence will not be possible . However , loss scaling only modifies the gradients . Weights and activations can also ( albeit admittedly less frequently ) exceed the FP8 ’ s representable range of [ 2−16 , 216 ] . In those scenarios , convergence can also be affected . The issue with loss scaling is that it requires user interaction . Models have to be modified , and , more importantly , tedious empirical tuning is required to find the correct loss scaling schedule . While some networks can be trained with constant loss scaling , some , notably Transformers ( Mellempudi et al. , 2019 ) , require dynamic “ back-off ” and improved loss scaling . This requires significant trial and error to tune the scaling schedule , slowing down wide adoption of low-precision numerical formats . 3.2 SHIFTED AND SQUEEZED FP8 . To alleviate these issues and make neural network training possible with no model modifications or hyperparameter tuning , we propose a new 8-bit floating point format . Consider a tensor X of size N , i.e. , X = { Xi } Ni=1 . Instead of directly encoding each Xi in FP8 , we store X using N FP8 numbers { Yi } Ni=1 accompanied by two ( squeeze and shift ) factors α and β ( the “ statistics ” — see Figure 2 ) . -16 0 16 log2 |Y | 0 32 log2 |X| For Xi 6= 0 , X and Y are then related through log2 ( |Yi| ) = α log2 ( |Xi| ) + β ⇔ Yi = ±2β |Xi|α ( 1 ) where the ± is chosen so that Xi and Yi have the same sign . This representation allows for α and β be chosen so that together with tensor Y they capture most of the dynamic range of the tensor X . As we will see in section 4 , this is all that is necessary to train networks using 8-bit floating point numbers . In order for Y to be a tensor suitable to be represented by FP8 numbers , we enforce that it has zero mean and a maximum value within the dynamic range of FP8 ( e.g . 15 ) : N ′∑ i=1 log2 ( |Yi| ) = 0 and max i=1 , ... , N ′ log2 ( |Yi| ) = 15 ( = log2 ( 215 ) ) ( 2 ) where the ′ notation indicates that the sum and the max , respectively , ignore any i such that Yi = 0 . Those equations ensure that log2 ( |Y | ) values are distributed with zero mean and each is less than 15 , which is ideal for an FP8 format . By inserting Equation 2 into Equation 1 , and by denoting µ = N ′∑ i=1 log2 ( |Xi| ) and m = max i log2 ( |Xi| ) ( 3 ) we find α = 15 m− µ , β = −αµ ( 4 ) This new tensor format results in the training procedure ( forward pass , backward pass , weight update ) described in Figure 4 . Forward and backward MatMul use this new S2FP8 format . Master weights are kept in FP32 and updated using S2FP8 gradients . Accumulations inside the GEMM kernel are kept in full FP32 precision . Figure 3 illustrates the impact of α and β . By having those two extra degrees of freedom for each tensor , majority of the dynamic range of each tensor can now be captured , whether very small ( β > 0 ) , very large ( β < 1 ) , very narrow ( α > 1 ) ) or very wide ( α < 1 ) . 3.3 LEARNING THE TENSOR DISTRIBUTION . One way to interpret α and β is to consider them as parameters of a distribution generating the tensor values log2 ( |Xi| ) . We can then say that , by continuously computing α and β , we are effectively learning the distribution of log2 ( |Xi| ) . Figure 5c shows the evolution of µ , m , α and β for a particular tensor of ResNet-20 . We see that α and β converge to , approximately , 5 and 21 , respectively . From Equation 1 , we conclude that : • since α > 1 , this means that X is expanded into Y , i.e. , X is more narrow than what FP8 allows • since β > 0 , this means that X is right-shifted into Y , i.e. , X is smaller than what FP8 allows At convergence , those α and β values represent the distribution of each converged tensor . Notice that all statistics stabilize in the last third of the training , where the learning rate is decreased , indicating the network is converging to its final state .

Summary: The paper suggest the method to train neural networks using 8 bit floating point precision values. The method requires a hardware modification to evaluate per tensor statistics, which intuitively capture dynamic range that a given tensor values may fall into. Those per tensor statistics are used to map 8-bit representation into a rational number. Statistics themselves are calculated using 32 bit floating point numbers.