Mixed Precision Training With 8-bit Floating Point

1 INTRODUCTION . The unprecedented success of Deep Learning models in a variety of tasks including computer vision ( He et al. , 2016 ) , machine translation ( Wu et al. , 2016 ) and speech recognition ( Graves et al. , 2013 ; Hannun et al. , 2014 ) has led to the proliferation of deeper and more complex models . Algorithmic innovations such as large batch training ( Keskar et al. , 2016 ) and neural architecture search ( Zoph & Le , 2016 ) have enabled models to scale on large compute cluster to accelerate training . This enhanced performance has enabled the adoption of larger neural networks . As a consequence , the computational requirements for training Deep Learning models have been growing at an exponential rate ( Amodei & Hernandez ) over the past few years , outperforming Moore ’ s Law and hardware capabilities by a wide margin . One of the promising areas of research to address this growing compute gap is to reduce the numeric precision requirements for deep learning . Reduced precision methods exploit the inherent noise resilient properties of deep neural networks to improve compute efficiency , while minimizing the loss of model accuracy . Recent studies ( Micikevicius et al. , 2017 ; Das et al. , 2018 ) have shown that , deep neural networks can be trained using 16-bits of precision without any noticeable impact on validation accuracy across a wide range of networks . Today , state-of-the-art training platforms support 16-bit precision in the form of high-performance systolic array or GEMM engine ( General Matrix Multiply ) implementations ( Markidis et al. , 2018 ; Köster et al. , 2017a ) . There have been numerous attempts ( Hubara et al. , 2017 ; Zhou et al. , 2016 ; De Sa et al. , 2018 ; Wu et al. , 2018 ; Cai et al. , 2017 ) to train deep neural networks at lower precision ( below 16-bits ) with varying degrees of success . With the abundance of 8-bit integer deep learning ‘ ops ’ deployed to accelerate inference tasks , much of the research into training methods have also focused on integer based fixed-point numeric formats ( Zhou et al. , 2016 ; De Sa et al. , 2018 ; Wu et al. , 2018 ) . Training with 8-bit integers has been significantly more challenging because the dynamic range of such formats is not sufficient to represent error gradients during back-propagation . More recently , Wang et al . ( 2018 ) have shown that 8-bit floating representation can be used to train convolutional neural networks , with the help of specialized chunk-based accumulation and stochastic rounding hardware . While this method has shown promising results , it requires expensive stochastic rounding hardware built into the critical compute path making it unattractive for systolic array and GEMM accelerator implementations . Our paper extends the state of the art in 8-bit floating point ( FP8 ) training with the following key contributions : • We propose a scalable training solution that eliminates the need for specialized hardware designs ( Wang et al. , 2018 ) , thereby enabling efficient MAC designs with higher compute density . • We demonstrated state-of-the-art training results using 8-bit floating point representation ( for weight , activation , error and gradient tensors ) , across multiple data sets ( Imagenet-1K , WMT16 ) and a broader set of workloads ( Resnet , GNMT , Transformer ) than previously reported ( Wang et al. , 2018 ) . • We propose enhanced loss scaling method to compensate for the reduced subnormal range of 8-bit floating point representation for improved error propagation leading to better model accuracy . • We present a detailed study of the impact of quantization noise on model generalization and propose a stochastic rounding technique to address the gradient noise in the early epochs leading to better generalization . 2 RELATED WORK . The study of reduced precision methods for deep learning training is an active area of research . In the pursuit of improving compute efficiency , researchers have experimented with various numeric formats and hardware implementations . Gupta et al . ( 2015 ) demonstrated that deep neural networks can be trained with minimal loss in accuracy , using 16-bit fixed point representation . This was followed by studies employing other numeric formats such as , half-precision floating point ( Micikevicius et al. , 2017 ) and dynamic fixed point ( Köster et al. , 2017b ; Das et al. , 2018 ) , demonstrating state of the art results across residual ( He et al. , 2016 ) , recurrent ( Wu et al. , 2016 ) and generative networks . Today most of the neural network training in a production deployment has migrated to 16-bit hardware , resulting in significant improvements in performance ( Markidis et al. , 2018 ) . There have been several attempts to further reduce the precision requirements of DNNs to boost training performance . Zhou et al . ( 2016 ) have trained DoReFa-Net , a derivative of AlexNet ( Krizhevsky et al. , 2012 ) using bit-convolutions with 1-bit and 2-bits to represent weights and activations respectively , while the gradients are quantized to 6-bits of precision . Wu et al . ( 2018 ) have trained AlexNet ( Krizhevsky et al. , 2012 ) using 8-bit precision for activations , errors and weight gradients , while the weights are quantized to 2-bits of precision . However , both these methods have reported significant loss in validation accuracy . More recently , Wang et al . ( 2018 ) have successfully trained Resnet-50 ( He et al. , 2016 ) using 8-bit floating point numeric format with the help of a specialized hardware to compute chunk-based dotproduct computation and stochastic rounding on a 16-bit accumulator . The authors of this study have focused on reducing the accumulator precision and based on studies on smaller networks ( AlexNet Resnet-18 ) , attributed training issues related to error propagation and generalization on the choice of accumulator size . However , our studies on larger networks ( Resnet-34/50 ) using 32-bit accumulator indicate that , these issues are not related to the choice of accumulator size and should be addressed independently . We discuss these issues and our proposed solutions in greater detail in Sections3.1and 3.2 . Guided by these results , we decided to focus on studying the impact of using FP8 numeric format on training , while maintaining a high precision accumulator ( FP32 ) . We further believe that modern GEMM engine designs implementing progressive multiplier reduction ( Ibrahim & Gebali , 2016 ) techniques can effectively amortize the cost of a larger final accumulator , and do not benefit from building 16-bit accumulator ( Wang et al. , 2018 ) with additional hardware overheads in the critical compute path . 3 TRAINING METHOD . The choice of bit-level representation of floating point ( sign , exponent , mantissa ) , has a significant impact on the effectiveness of the numerical format – the trade-off between the dynamic range and precision is especially tricky at low bit-width representations . While it is important to maintain higher dynamic range for effective propagation of error gradients ( Micikevicius et al. , 2017 ) , it leads to having values that are too few and scattered to maintain fidelity required for gradient computations . After careful consideration of these facts and several failed experiments with other formats ( ex : with more exponent bits ) , we decided to use s=1 , e=5 , m=2 numeric format for representing 8-bit floating point . We also use a 32-bit floating point accumulator to accommodate the dynamic range of the output resulting from the dot-product operation . Therefore , each GEMM/convolution operation takes two input tensors in 8-bit floating point format and produces a 32-bit single precision floating point output . The 32-bit output must be down-converted to a 8-bit floating point value before passing to the next operation . We believe rounding plays an extremely important role during down-conversion operation to help recover the numeric accuracy . We present the results from the study of different rounding modes applied to this numeric format and their impact on training performacne in Section.3.2 Figure.1a shows the precision settings of various compute operations used in our mixed precision training setup . The ’ GEMM ’ operator shown in Figure.1a represents the key compute kernel used by deep neural networks during forward , backward , and gradient computation passes . Quantization nodes identified with the letter ’ Q ’ perform down-conversion and rounding on 32-bit output from GEMM operators and convert them them to 8-bit floating point format before passing on to the next layer . In our training experiments , we quantized the weight , activation , error and gradient tensors of all convolution and MatMul kernels to 8-bit floating point format in forward , backward and weight update paths . More details about workload specific quantization flow is discussed in Section 4 . Figure.1b shows the data flow during optimization and weight update steps . In the optimization path the L2-regularization term is added to the cross entropy , and the resulting loss is multiplied with loss scale parameter before initiating back propagation . The weight gradients are computed during back propagation and converted to 8-bit floating point format . During weight update , the weight gradients are re-scaled using the loss scale parameter , this step is performed in full precision to prevent any potential underflow . The gradients are then passed to the momentum optimizer , and the final gradients are applied to the master copy of weights . In our experiments , the master copy of weights are stored in half-precision floating point format , these values are up-converted to 32- bit during the update step and the weight update is performed at full precision . After the update , the master weights are converted to half-precision format before they are stored back into memory . Since this is a bandwidth bound operation , compute precision will not have any noticeable impact on the performance . 3.1 ENHANCED LOSS SCALING . Previous studies ( Micikevicius et al. , 2017 ) on half-precision floating point have shown that loss scaling technique can be used to push smaller error gradients into representable range and train neural networks successfully . The full range of numeric values represented by a floating point format include the ’ subnormal ’ values , the range of which is determined by the number of mantissa bits . Because of this property of floating point numbers , the proposed 8-bit floating point format will have significantly smaller subnormal range compared to a half-precision floating point which has the same number of exponent bits . Table.1 shows the dynamic range comparison between fullprecision ( FP32 ) , half-precision ( FP16 ) and the proposed 8-bit floating point ( FP8 ) formats . Half-precision training for convolution networks has been shown to converge using a constant loss scaling value ( ex : Resnet-50 uses 1000 ) ( Micikevicius et al. , 2017 ) . Networks like GNMT ( Wu et al. , 2016 ) and Transformer ( Vaswani et al. , 2017 ) experience significant variations in gradient distributions through the training cycle and perform better with dynamic loss scaling methods . We adopt the same loss scaling methods used by FP16 training for FP8 . For Resnet-50 , we increase the scaling factor to compensate for the smaller subnormal range of FP8 . Figure.2a shows results from our convergence studies on Resnet-50 using different loss scaling values . The model failed to converge with a scaling factor of 1000 , and progressively performed better with increasing loss scale values , converging at 10 000 . Transformer ( big ) model , uses the standard ’ DynamicLossScale ’ method implemented in TensorFlow and it works out-of-the-box without any additional changes . We use the OpenSeq2Seq ( Kuchaiev et al. , 2018 ) implementation for GNMT model , and it uses ’ back-off ’ method for automatic loss scaling . Both these loss scaling methods try to maintain a higher loss scale value while checking for numeric overflows at regular intervals ( =2000 ) , that result in a ’ NaN ’ during gradient computation . Our experiments showed that loss scaling experiences frequent gradient overflows for GNMT because of a few outliers , while a significant chunk of the gradients underflow . This irregular gradient distribution is likely a result of not having normalization layers in GNMT . The loss scaling algorithm responds to these overflows by dropping the loss scale value significantly . We addressed this by ignoring a few overflows which are likely a result of the outliers and continue to maintain a higher loss scale value . We accomplish this by setting a ‘ minimum threshold ’ for the loss scale value to prevent it from going below a certain threshold value even if an overflow occurs . Figure.2b shows the loss scaling schedule that worked for GNMT . The goal of this exercises was to maintain a higher loss scale value , while ignoring a few spurious overflows . This can also achieved by enhancing the loss scaling algorithm with few additional checks to ignore overflows unless they occur in succession for a few times that is predefined threshold value ( nan threshold ) . Overflows occurring at successive intervals is a more reliable indicator of a actual shift in the gradient distribution . We can also reduce the interval between loss scale updates ( from 2000 iterations to 500 iterations ) , to improve chances of recovering from any inadvertent drop in loss scale value . We describe this algorithm in 1 .

Summary: In this paper, the authors propose a method to train deep neural networks using 8-bit floating point representation for weights, activations, errors, and gradients. They use enhanced loss scale, quantization and stochastic rounding techniques to balance the numerical accuracy and computational efficiency. Finally, they get a slightly better validation accuracy compared to full precision baseline. Overall, this paper focuses on engineering techniques about mixed precision training with 8-bit floating point, and state-of-the-art accuracy across multiple data sets shows the effectiveness of their work.