Mixed Precision Training With 8-bit Floating Point
Authors: Naveen Mellempudi
Sudarshan Srinivasan
Dipankar Das
Bharat Kaul
Venue:  (2020)
Keywords: 8-bit training
8-bit floating point
low precision training
deep learning

Abstract: Reduced precision computation is one of the key areas addressing the widening’compute gap’, driven by an exponential growth in deep learning applications. In recent years, deep neural network training has largely migrated to 16-bit precision,with significant gains in performance and energy efficiency. However, attempts to train DNNs at 8-bit precision have met with significant challenges, because of the higher precision and dynamic range requirements of back-propagation.   In this paper,  we  propose  a  method  to  train  deep  neural  networks  using  8-bit  floating point representation for weights, activations, errors, and gradients.  We demonstrate state-of-the-art accuracy across multiple data sets (imagenet-1K, WMT16)and a broader set of workloads (Resnet-18/34/50, GNMT, and Transformer) than previously reported.   We propose an enhanced loss scaling method to augment the reduced subnormal range of 8-bit floating point, to improve error propagation.We also examine the impact of quantization noise on generalization, and propose a stochastic rounding technique to address gradient noise. As a result of applying all these techniques,  we report slightly higher validation accuracy compared to full precision baseline.

Under review as a conference paper at ICLR 2020
MIXED PRECISION TRAINING WITH 8- BIT FLOATING POINT
A Anonymous authors Paper under double-bling review
ABSTRACT
Reduced precision computation is one of the key areas addressing the widening 'compute gap'. driven by an exponential growth in deep learning applications. recent years, deep neural network training has largely migrated to 16-bit precision, with significant gains performance and energy efficiency. However, attempts to train DNNs at 8-bit precision have met with significant challenges, because of the higher precision and dynamic range requirements of proppaaaaaat.on. In this paper, we propose method to train deep neural networks using 8-bit floating point representation for weights, activations, errors, and gradients. We demon- strate state-of-the-ee accuracy across multiple data sets (imagenet- 1K, WMT16) and a broader set workloads previously reported. We propose enhanced loss scaling method augment the reduced subnormal range of 8-bit floating point, to improve error propagation. We also examine the impact of quantization noise on generalization, and propose a stochastic rounding technique to address gradient noise. As result of applying all these techniques, our models generalize as well as the full precision models, and achieve baseline accuracy.
GNMT, and Transformer) than
INTRODUCTION
The unprecedented success of Deep Learning models in a variety of tasks including computer vision (He 2016), machine translation (Wu et al.) 20166) and speech recognition (Graves al. 2013] Hannun et al. 2014) has led to the proliferation of deeper and more complex models. Algorithmic innovations such as large batch training (Keskar et al. 2016) and neural architecture search (Zoph) & [e 2016) have enabled models to scale on large compute cluster to accelerate training. This enhanced performance has enabled the adoption larger neural networks. As a consequence, the computational requirements for training Deep Learning models have been growing at an exponential rate (Amodei & Hernandez) over the past few years, outperforming Moore's 's Law and hardware capabilities by wide margin.
One of the promising areas of research to address this growing compute gap is to reduce the numeric precision requirements for deep learning. Reduced precision methods exploit the inherent noise resilient properties of deep neural networks to improve compute efficiency, while minimizing the loss of model accuracy. Recent studies (Micikevicius et al. 2017 Das et al., (018) have shown that, deep neural networks can be trained using 16-bits of precision without any noticeable impact on validation accuracy across wide range of networks. Today, state-of-the-art training platforms support 16-bit precision in the form of high- -eehh-perfrmancee systolic array or GEMM engine (General Matrix Multiply) implementations (Markidis et 2018
Köster al. 2017a).
There have been numerous attempts (Hubara et al. 2017:
Zhou et al. 2016: De Sa et al. 2018 Wu
al. 2018 Cai al., 2017) to train deep neural networks at lower precision (below 16-bits) with varying degrees of success. With the abundance of 8-bit integer deep learning 'ops' deployed to accelerate inference tasks, much the research into training methods have also focused on integer based fixed-point numeric formats (Zhou et al., 2016 De Sa et al. 2018 Wu et al. 2018). Train- ing with 8-bit integers has been significantly more challenging because the dynamic range of such formats is not sufficient to represent error gradients during propaaropag More recently, Wang jet al. (2018) have shown that 8-bit floating representation can be used to train convolutional neural networks, with the help of specialized chunk- accumulation and stochastic rounding hardware.

---

Under review as a conference paper at ICLR 2020
While this method has shown promising results, it requires expensive stochastic rounding hardware built into the critical compute path making unattractive for systolic array and GEMM accelerator implementations.
Our paper extends the state of the art 8-bit floating point (FP8) training with the following key contributions:
We propose scalable training solution that eliminates the need for specialized hardware designs (Wang et al. 2018), thereby enabling efficient MAC designs with higher compute density.
We demonstrated state-of-the-art training results using 8-bit floating point representation (for weight, activation, eπror and gradient tensors), across multiple data sets (Imagenet- IK, WMT16) and a broader set of workloads (Resnet, GNMT, Transformer) than previously reported (Wang et al. (2018).
We propose enhanced loss scaling method compensate for the reduced subnormal range of 8-bit floating point representation for improved error propagation leading to better model accuracy.
We present detailed study of the impact of quantization noise on model generalization and propose a stochastic rounding technique to address the gradient noise in the early epochs leading better generalization.
2 RELATED WORK
The study of reduced precision methods for deep learning training an active area of research. In the pursuit of improving compute efficiency, researchers have experimented with various numeric formats and hardware implementations. Gupta et al. (2015) demonstrated that deep neural networks can be trained with minimal loss in accuracy, using 16-bit fixed point representation. This was followed by studies employing other numeric formats such as, half- saff-precision floating point (Micike- vicius al. 2017) and dynamic fixed point (Köster et al.) 2017b] Das et al.) 2018). demonstrating state of the art results across residual (He et al.) (016)), recurrent (Wu et al. 2016) and generative networks. Today most of the neural network training 16-bit hardware, resulting in significant improvements in performance (Markidis et al.) 2018).
production deployment has migrated to
There have been several attempts to further reduce the precision requirements of DNNs to boost training performance. Zhou et al.] (2016) have trained DoReFa-Net, a derivative of AlexNet (Krizhevsky et al. 2012 using bit-convolutions with 1-bit and 2-bits to represent weights and acti- vations respectively. while the gradients are quantized 6-bits of precision. Wu et al. (2018) have trained AlexNet (Krizhevsky et al.] (012) using 8-bit precision for activations, errors and weight gradients, while the weights quantized 2-bits of precision. However, both these methods have reported significant loss in validation accuracy.
More recently. |Wang al. 2018) have successfully trained Resnet-50 (He al. 2016) using 8-bit floating point numeric format with the help of a specialized hardware to compute chunk-based dot- product computation and stochastic rounding on 16-bit accumulator. The authors of this study have focused on reducing the accumulator precision and based on studies on smaller networks (AlexNet Resnet-18) attributed training issues related to error propagation and generalization on the choice of accumulator size. However, our studies on larger networks (Resnet-34/500 using 32-bit accu- mulator indicate that, these issues are not related to the choice of accumulator size and should be addressed independently. We discuss these issues and our proposed solutions in greater detail in Sections3. I.n 3.2 Guided by these results, we decided to focus on studying the impact of using FP8 numeric format training, while maintaining high precision accumulator(TP322. We further believe that modern GEMM engine designs implementing progressive multiplier reduction (Ibrahim & Gebali 2016) techniques effectively amortize the cost of a larger final accumulator, and do not beneht from building 16-bit accumulator (Wang et al. 2018) with additional hardware overheads in the critical compute path.
2

---

Under review as a conference paper at ICLR 2020
3 TRAINING METHOD
The choice of bit-level representation of floating point (sign, exponent, mantissa), has signifi- cant impact on the effectiveness of the numerical format - the trade-off between the dynamic range and precision is especially tricky low bit-width representations. While it is important to maintain higher dynamic range for effective propagation of eπror (Miiiintt(Miiiiiiiiius et al., 2017), it lcads having values that are too few and scattered maintain fidelity required for gradient computa- tions. After careful consideration of these facts and several failed experiments with other formats (ex: with more exponent bits), we decided to use s=1,e=5,m=2 numeric format for representing 8-bit floating point. We also use 32-bit floating point accumulator to accommodate the dynamic range of the output resulting from the dot-product operation. Therefore, each GEMM/convolution operation takes two input tensors in 8-bit floating point format and produces a 32-bit single preci- sion floating point output. The 32-bit output must be down-convertee to a 8-bit floating point value before passing to the next operation. We believe rounding plays an extremely important role dur- ing down-conversion operation to help recover the numeric accuracy. We present the results from the study of different rounding modes applied to this numeric format and their impact on training performacne in Section 3.2
Figure Ia| shows the precision settings of various compute operations used in our mixed precision training setup. The 'GEMM' operator shown in Figure(Ta) represents the key compute kernel used by deep neural networks during forward, backward, and gradient computation passes. Quantization nodes identified with the letter 'Q" perform down- nown-conversion and rounding on 32-bit output from GEMM operators and convert them them 8-bit floating point format before passing on to the next layer. In our training experiments, we quantized the weight, activation, error and gradient tensors of all convolution and MatMul kernels to 8-bit floating point format in forward, backward and weight update paths. More details about workload specific quantization flow is discussed in Section
Figure lb shows the data flow during optimization and weight update steps. the optimization path the I.2rregllarization term is added to the cross entropy, and the resulting loss is multiplied with loss_scale parameter before initiating back propagation. The weight gradients are computed during back propagation and converted to 8-bit floating point format. During weight update. the weight gradients are re-scaled using the loss scale parameter, this step is performed in full precision to prevent any potential underflow. The gradients the final gradients are applied to the master copy of weights. In our experiments, the master copy of weights are stored in half-precision floating point format, these values are up-converted to 32- bit during the update step and the weight update is performed at full precision. After the update, the master weights are converted to malf-precision format before they are stored back into memory. Since this is a bandwidth bound operation, compute precision will not have any noticeable impact on the performance.
then passed to the momentum optimizer, and
PWD GEMM
cross entropy
weight decay
A1
FP32
scaled
weights (1Eb)
weights
weightss
regularisation
WG GEMM
loss scale
learning rate
weight gradienns"
weight update
weight gradients
FP32
gradienss
FEB
error gradienns
gradients w/ momontum (32b)
momentum
Fr32
(a)
(b)
Figure Mixed precision data flow for FP8 training. (left) precision settings for key compute kernels in Forward, Backward, and Weight Update passes, (right) flow diagram of the weight update rule.
3

---

Under review as a conference paper at ICLR 2020
3.1
ENHANCED LOss SCALING
Previous studies (Micikevicius et al., 2017) on half-precision floating point have shown that loss scaling technique can be used to push smaller error gradients into representable range and train neural networks successfully. The full range of numeric values represented by floating point format include the "subnormal' values, the range of which is determined by the number of mantissa bits. Because of this property of floating point numbers, the proposed 8-bit floating point format will have significantly smaller subnormal range compared to half- prefi-preiiion floating point which has the same number of exponent bits. precision(FP32), half-precision(FPP1) and the proposed 8-bit floating point (FP8) formats.
shows the dynamic range comparison between full-
Table 1: Dynamic range comparison between proposed FP8 and other floating point formats.
Data Type
Bit Format (s, e, m)
Max Normal
Min Normal
Min Subnormal
IEEE-754 float
8. 23
3.40e38 65 535
1.17e-38 6.10e-5
1.40e-45 -.
IEEE-754 half-float
5. 10
FP8 (proposed) 5, 2
57 344
5.10e-5
1
Half-precision training for convolution networks has been shown to converge using a constant loss scaling value (ex: Resnet-50 uses 1000) (Mciikeiiss et al. 2017). Networks like GNMT (Wu al. 2016) and Transformer (Vaswani et al. (017) experience significant variations gradient distributions through the training cycle and perform better with dynamic loss scaling methods. We adopt the same loss scaling methods used by FP16 training for FP8. For Resnet-50, we increase the scaling factor to compensate for the smaller subnormal range of FP8. Figure 2a shows results from our convergence studies on Resnet-50 using different loss scaling values. The model failed to converge with a scaling factor of 1000, and progressively performed better with increasing loss scale values, converging at 000. Transformer (big) model, uses the standard DynamicLossSSall method implemented in TensorFlow and it works out-of-the-box without any additional changes. We use the OpenSeq2Seq (Kuchaiev et al. 2018) implementation for GNMT model, and it uses 'back-off' method for automatic loss scaling. Both these loss scaling methods try to maintain a higher loss scale value while checking for numeric overflows at regular intervals(=2(000, that result in a 'NaN' during gradient computation. Our experiments showed that loss scaling experiences frequent gradient overllows for GNMT because of a few outliers, while a significant chunk of the gradients underflow. This irregular gradient distribution is likely a result of not having normalization layers in GNMT. The loss scaling algorithm responds to these overflows by dropping the loss scale value significantly. We addressed this by ignoring a few overflows which are likely a result of the outliers and continue to maintain higher loss scale value. We accomplish this by setting 'minimum threshold' for the loss scale value to prevent it from going below a certain threshold value even if an overflow occurs. Figure shows the loss scaling schedule that worked for GNMT.
The goal of this exercises was to maintain higher loss scale value, while ignoring a few spuri- ous overflows This can also achieved by enhancing the loss scaling algorithm with few additional checks to ignore overflows unless they occur in succession for a few times that is predefined thresh- old value (nan threshhrrehl.) Overflows occurring at successive intervals is more reliable indicator of a actual shift in the gradient distribution. We can also reduce the interval between loss scale updates (from 2000 iterations to 500 iterations), to improve chances of recovering from any inadvertent drop in loss scale value. We describe this algorithm in
3.2 QUANTIZATION NOISE AND GENERALIZATION
Reduced precision methods introduce significant amount of noise that can adversely effect conver- gence and accuracy of deep neural networks. Rounding techniques applied to quantization methods can be effective in regulating some of this noise. For extremely low precision representations with large rounding errors such as the one proposed here(c = 0.125). the choice of rounding method can have significant influence on the numeric accuracy and overall applicability of the numeric format. Previous studies (Gupta et al. (2155) have shown that stochastic rounding can be effective for train- ing neural networks using low-precision fixed point formats. The most widely supported rounding
4

---

Under review as a conference paper at ICLR 2020
baseline
1.2
b-bit, loss scale-1K 1K
200k
8-bit, Scale=4K bit,
GNMT, 8-bri, training loss
1.0
3.5
loss scaling factor
Validition enrer
2004
Waining
160c
2.0
100%
0.2
mine32700
0.0
epochs
Iterations
(a)
(b)
Figure 2: Convergence behaviour of FP8 training using enhanced loss scaling. (left) Resnet-50 (He al., 2016) failed converge with loss scale=1000, performed better with 2.3% accuracy loss loss scale=4000 and showed full convergence at loss scale=10 000, (right) Dynamic loss scaling with gradually increasing minimum threshold for the scaling factor.
Algorithm Enhanced loss scaling algorithm
Ensure: minval marval 2 interval 500, scale 2, nan_threshold iter = 0, last nat__an_iie iter =
1: nan count = 0, last_ter
2: procedure UPDATE LOSS_SCALE
has nan, iter)
3:
if has pus__nan-e True then
4:
if nan_count >= nanthresholdd then loss scale + loss scale +
5:
6:
nan count <-
7:
else if iter laaat_nnan nan_iter + then nan_count + nan_count + interval then loss_scale + loss_scale x
8:
9:
else if iter last.
10:
11:
return loss scale
method in hardware today is RNE (round to nearest even), because is easier implement and requires smaller silicon area. In this section, we explore the impact of both RNE and stochastic rounding methods model convergence and generalization.
Our early experiments showed that, for smaller networks such as Resnet-18 18 (He et al. 2016). RNE proved quite effective when trained on Imagenet-IK (Deng et al. 2209) data set. However, when we trained ResNet-50 (He et al. 20166) we observed some interesting results. Figure shows the convergence plots for Resnet-50 HHe al. 2016) using RNE rounding method applied quantized weights, activations and gradients. The model displayed significant over- wver-ftting behaviour as indi- cated by the increased validation error, while the training enror mostly follows the baseline as shown in as shown in Figure 3b, and 3a Multiple experiments indicated that this behaviour is caused by the noisy error gradients during early epochs which lead unconstrained growth model parameters. This is indicated by steep increase in L2 regularization parameter shown in Figure 3c Regular- ization loss is computed using the formula shown in Equation [ Increased regularization loss leads to more noisy gradients, which further exacerbates this behaviour.
An interesting observation about the L2 regularization loss is that for ResNet- 18, the L2-loss is low at the beginning and increases with gradually with iterations. On the other hand for ResNet-50, the L2-loss is high at the beginning due to the initialization of low fan-in Ixl (Glorot & Bengio 2010) convolutions, and needs to dip little before gradually rising again. We suspect that this property of
5

---

Under review as a conference paper at ICLR 2020
the initialization leads to noisy behavior of ResNet-50 in the earlier iterations compared to ResNet-18 18. Therefore for the ResNet-50 model stochastic rounding is essential.
W
L2_loss = X
W
(1)
k=0
Where, À is the weight decay parameter and W is the total number of weights.
In order to understand the issue of regularization independent the choice of rounding method, we conducted additional experiments using RNE with other forms of regularization. Figure com- pares the 'Dropout' method with no regularization' method which uses quantization noise as im- plicit regularizer with explicit regularization term. In both these cases, the models performed much better than using L2 regularization with RNE, leading us to the conclusion that RNE was ineffective in regulating quantization noise in gradients causing unconstrained growth in model pa- rameters.
120%
FP52. wulidiion
bevelin emm
FP37,1
1946, iraiiingrm
100%
19S selt, La
12g Autberknn tose
000
4001
2011
200081
200051
259123
400111
Neretions
(a)
(c)
Figure 3: Impact of quantization (with RNE rounding) noise model convergence with Resnet- 50 (a) comparison of training error, (b) validation error, and (c) L2 regularization loss with FP32 baseline.
Unlike deterministic rounding techniques, stochastic rounding computes the probability of rounding using information from several discarded bits of the input making less prone to introducing large rounding errors. We studied the error behaviour of Resnet-50 (He et al. (0116) by applying stochastic rounding on activations and gradients to regulate quantization noise the gradients, which in-turn can improve the effectiveness of explicit regularization methods.
Our stochastic rounding method is defined as follows:
round(x, k) = [ + with probability P (ar (x-1=)),+r
[ [0]a with probability 1 P
Where, k is the target precision, is machine epsilon, and r is random value generated by a pseudo random number generator.
Figure 4b shows the results from Resnet-50 (He et al. 2016) training experiment using combi- nation stochastic rounding and explicit 1.2 regularization. The convergence plots show the good generalization behavior that tracks with the full precision training. The accuracy numbers are sum- marized in Section
4 EXPERIMENTS AND RESULTS
We built a TensorFlow based training platform that can accurately emulate the numeric properties of 8-bit floating point on existing floating point hardware. Training experiments were conducted using open source model implementations from TensorFlow[] and and OpenSeq2Seq (Kuchaiev et al. 2018). Our training framework internally updates the training graph by inserting quantization nodes
Models: http::/ggiiaaaa..oo//eeesroooonh

---

Under review as a conference paper at ICLR 2020
100%
90%
FP32, baseline
FP32 baseline
FP8, mggllarization
80%
FP8, Stochastic rounding. 12 regularization
FP8, RNE, dropout
70%
validiation error
70%
60%
60%
validation
50%
40%
40%
3,9% loss
30%
30%
20%
10%
10%
0%
0%
21
41
61
81
epochs
epochs
(b)
Figure 4: (a) Comparing validation performance with 'dropout" and noise-based implicit regulariza- tion techniques using RNE(round to nearest even) (b) model performance with stochastic rounding with L2 regularization.
in the forward, backward, weight update paths for all convolution and GEMM kernels as described in Section
Using the proposed training method, we have successfully trained two residual networks (Resnet- 34/50) (He et al.] 2016) on Imagenet-1K (Deng et al. (209) data set. For 8-bit training experiments, we have quantized the Convolution, BatchNorm (stats are accumulated at full precision), ReLU and EltWise layers, while the data layer and SoftMax layers use full precision. We have maintained the first Convolution layer, the following BatchNorm layer, and final Dense layer at half- to minimize the accuracy loss. The first convolution layer is expected have minimum of eight man- tissa bits to accurately represent the normalized image data passed from the data layer. We also store the master copy of weights in half-precision; this can potentially speed up weight update operation by up to 2x, because the operation is memory bound. Since the weight update computation is per- formed at full precision (as descried in Section 3. to realize these bandwidth benefits the up/down conversion of the master weights between full and half precision formats must be strictly performed in the cache or near memory. We have used identical batch size (256), and hyper parameters for both full precision and 8-bit training experiments. Both sets of experiments also employ 'warm up" for the first 5 epochs, with an initial learning rate of 128, and train for a total of 90 epochs achieve the final accuracy.Tablee
summarizes the validation accuracy achieved by convolution networks on imagenet- KK (Deng et al. 2009) dataset.
Table 2: Top-1 validation accuracy for convolution networks Imagenet- 1 1 Deng et al. 2009
Model
Dataset
Batch-size
Epochs
FP32 (top-1 %)
FP8 (top-1 1 %)
Resnet-50
imagenet-11 -K
256
90
76.38
76.18
Resnet-34
imagenet- 1K
256
90
73.47
73. 18
Resnet-18
imagenet-IK
256
90
69.23
69.71
Jlsows the convergence plots for Resnet-34 and Resnet-50 comparing top-1 accuracy of FP8 training with the baseline FP32 training. It can be seen that the validation accuracy of FP8 training closely follow the baseline numbers indicating the robustness of the training method.
In addition to convolution networks, we have also trained two state of the art machine translation workloads (GNMT (wu et al., 2016) and Transformer (Vaswani et al. 2017)) and demonstrated BLEU scores matching single precision baselines. We trained an 8-layer GNMT (Wu et al.) 2016) encoder/decoder LSTM model with 1024 recurrent units and 1024 attention units. We trained this network using 8-bit floating point format for all MatMul operations, including the LSTM. attention modules, embedding lookup is performed at 8-bit while the updates are computed at full precision. The activation functions such as tanh and sigmoid use half-precision data type. Our 8- training

---

Under review as a conference paper at ICLR 2020
Table Comparison our method with the only other FP8 training method on Imagenet-IK (Deng et al. 2009) data set. W, A, E, G, MasterWts represent the precision setting for weights, activations, error, weight gradients and mater copy of weights respectively.
Method, Format
W.A.E,G
MasterWts
Resnet-18
Resnet-50 (top-1 error %)
(top-1 1 error %)
Wang et al.
2018
FP8
8,8,8,8 8,8,8,8
16
33.05 30.29
28.28
Ours, FP8
16
23.82
Remreh36, 790, Mibedpretitee
606
40%
30%
205
Opocts
Cooche
Figure 5: Convergence plots showing Top-1 validation accuracy for. (a) Resnet-34 (He et al. 2016) (b) Resnet-50 (He et al. 2016) on imagenet-IK (Deng et al. 2009) dataset.
hanffre lng
Tandurmnnig
23000
200000
300008
(a)
Figure 6: Convergence plots showing training loss for (a) 8-layer GNMT (Wu et al. 2016) and, (b) 6-layer Transformer (Vaswani et al. 2017) trained on WMT16 English -> German dataset.
experiments identical batch size and hyper parameters to that of the full precision baseline and the network is trained for 10 epochs. We used the loss scaling schedule described in Section 3.1
We also trained 6-layer Transformer-big (Vaswani al. 2017 translation network with roughly 200M parameters. For the 8-bit experiments, we quantized all the MatMul operations in the encoder, decoder, including attention layers. Embedding lookup operation is performed at 8-bit precision while the updates are computed at full precision. We use the identical configuration for both baseline and 8-bit training experiments. Both GNMT (Wu et al.) (0066) and Transformer (Vaswani et al.) 2017) models were trained on large scale, WMT2016 English- dataset consisting of 4.5 million sentence pairs. The results are summarized in Table
CONCLUSION
We demonstrate state-of-the-art accuracy across multiple data sets (imagenet-IK, WMT16) and broader set of workloads (Resnet-18/34/50, GNMT, Transformer) than previously reported. We
8

---

Under review as a conference paper at ICLR 2020
Table 4: sacreBLEU
Post 2018 score measured on WMT 2014 English dataset
Model
Dataset/ Task
Epochs
FP32
baseline
FP8 Mixed Precision
GNMT
WMT 2016 English-+German
10
24.3
24.6
Transformer-big
WMT 2016 English++
10
27.59
27.33
propose easy implement and scalable solution for building FP8 compute primitives, eliminating the need for stochastic rounding hardware in the critical compute path, as proposed by Wang et al. 2018), thereby reducing the cost and complexity of the MAC unit. We explore issues around gra- dient underflow and quantization noise that arise as a result of using the proposed 8-bit numeric format for large scale neural network training. We propose solutions deal with these problems in the form of enhanced loss scaling and stochastic rounding.
REFERENCES
Dario Amodei and Danny Hermandez. Al and Compute.
bttps://openai.ccom/b1a
Bi-and- compute/
Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half-wave gaussian quantization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5918-5926, 2017.
Dipankar Das, Naveen Mellempudi, Dheevatsa Mudigere, Dhiraj Kalamkar, Sasikanth Avancha, Kunal Banerjee, Srinivas Sridharan, Karthik Vaidyanathan, Bharat Kaul, Evangelos Georganas, et al. Mixed precision training of convolutional neural networks using integer operations. arXiv preprint I 809000000000.....0000000000000 2018
Christopher De Sa, Megan Leszezynski, Jian Zhang, Alana Marzoev, Christopher Aberger, Kunle Olukotun, and Christopher Ré. High-accuracy low-precision training. arXiv preprint 1rXiv:1803.040 03383, 2018.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Feii ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neu- ral networks. In In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS' 10). Society for Artificial Intelligence and Statistics, 2010.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6645-6649, IEEE, 2013.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Conference on Machine Learning, pp. 1737-1746, 2015.
Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint urXiv: 112.5567, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- El-ia,,, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Jour- nal of Machine Learning Research, 18(1):6869-6888 2017.
Atef Ibrahim and Fayez Gebali. Low power semi-systolic architectures for polynomial mul- tiplication over gf (2 m) using progressive multiplier reduction. Journal of Signal Processing Systems, 82(3)3331-343, 2016.

---

Under review as a conference paper at ICLR 2020
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batcl training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.044 04836, 2016.
Urs Köster, Tristan Webb, Xin Wang, Marcel Nassar, Arjun K Bansal, William Constable, Oguz Elibol, Scott Gray, Stewart Hall, Luke Hornff, et al. Flexpoint: An adaptive numerical format for efficient training of deep neural networks. In Advances in neural information processing systems, pp. 1742-1752, 2017a.
Urs Köster, Tristan Webb, Xin Wang, Marcel Nassar, Arjun K Bansal, William Constable, Oguz Elibol, Scott Gray, Stewart Hall, Luke Hornof, et al. Flexpoint: An adaptive numerical format for efficient training of deep neural networks. In Advances in neural information processing systems, pp. 1742-1752, 2017b.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Oleksii Kuchaiev, Boris Ginsburg, Igor Gitman, Vitaly Lavrukhin, Jason Li, Huyen Nguyen, Carl Case, and Paulius Micikevicius. sixed-precision training for nlp and speech recognition with openseq2seq. Computing Research Repository (CoRR), abs/1805. 10387 v2, 2018.
Stefano Markidis, Steven Wei Der Chien, Erwin Laure, Ivy Bo Peng, and Jeffrey S Vetter. Nvidia tensor core programmability, performance & precision. 2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp. 522-531. IEEE, 2018.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg. Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint
2017.
Matt Post. A call for clarity in reporting blcu scores. arXiv preprint
2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia- Yu Chen, and Kailash Gopalakr- ishnan. Training deep neural networks with 8-bit floating point numbers. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems 31, pp. 7675- 7684. Curran Associates, Inc., 2018. URL http:/ http:/aappaaaae/aa/eee nips cc ccppaerr//ppppap/p/po/ppp
1994-trainin deep- neural networks with s-bit floating- -on--nun pers. pdf
Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. In International Conference on Learning Representations, 2018. https::::
//openreview.coo wwtteeooomiidduuss......
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 08144, 2016.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 2016.
Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. CoRR, 1..0156 2016. URL http: owweeiii..rr./oooooe abs//blsl. 011e//111..1111
10


Reviews:
[{"review_id": "SyxwlXOjcH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Originality: The paper proposed a new scaling loss strategy for mixed-precision (8-bit mainly) training and verified the importance of rounding (quantization) error issue for low-precision training. \n\nQuality: The authors clearly illustrated the benefit of their proposed loss strategy and the importance of quantization error for two different tasks (image classification and NMT). The experiments are very clear and easy to follow.\n\nClarity: The paper is clearly written with some visualizations for readers to understand the 8-bit training. \n\nSignificance:\n1. The enhanced loss scaling strategy is interesting but the method seems hand-tuning. Is there any automatical way or heuristic deciding way?\n2. The stochastic rounding method is very intuitive. How do you choose the value of \"r\" in the equation? Is it a sensitive hyper-parameter or not?\n\nTypos:\nPage 7: with with roughly 200M ->  with roughly 200M \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Skxg0CjUtH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is about training deep models with 8-bit floating point numbers. The authors use an enhanced loss scaling method and stochastic rounding method to stabilize training. They do experiments on image classification and NLP tasks.\n\nThe paper is clearly written. However, I don’t think this paper passes the bar of ICLR. This paper lacks innovation and insightful analysis.\n\n1.Sec. 3.1 proposes enhanced loss scaling. Loss scaling is a heuristic to train low-precision neural networks. The authors train 8-bit GNMT with a changing scaling factor. However, this looks like some manually tuned result for GNMT only. I doubt if this generalizes to other models. Besides, there is no equation or algorithm flowchart to demonstrate their method. It’s not very readable.\n\n2.The logic of Sec. 3.2 is quite confusing. The authors first empirically show that the performance of ResNet-50 significantly drops with 8-bit training. Then they show the sum of the square of the weights in ResNet-50 is high at the beginning. With this observation, they claim it demonstrates the drawback of ‘rounding-to-nearest-even’. I cannot see the connection between the norm of weights and the rounding technique. Moreover, the stochastic rounding has already been used in 8-bit training.[1]\n\n3.The setting in the experiment section is not stated clearly. For example, what’s the hyper-parameter for loss scaling? Another question is the gradient. In Sec. 3, just above Fig. 1, the authors claim the weight update is performed in full-precision. In contrast, they claim the gradient is 8-bit in table 3. If the update is full-precision, [2] is an important baseline. \n\nSmall suggestions:\n1.For Fig. 6, I suggest the authors to smooth the loss curves to avoid overlap of two curves.   \n2.There are two ‘with’s in the last paragraph of page 7.\n\nReference:\n[1]Wang N, Choi J, Brand D, et al. Training deep neural networks with 8-bit floating point numbers[C]//Advances in neural information processing systems. 2018: 7675-7684.\n[2]Banner R, Hubara I, Hoffer E, et al. Scalable methods for 8-bit training of neural networks[C]//Advances in Neural Information Processing Systems. 2018: 5145-5153.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SkxP4oloYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a method to train deep neural networks using 8-bit floating point representation for weights, activations, errors, and gradients. They use enhanced loss scale, quantization and stochastic rounding techniques to balance the numerical accuracy and computational efficiency. Finally, they get a slightly better validation accuracy compared to full precision baseline. Overall, this paper focuses on engineering techniques about mixed precision training with 8-bit floating point, and state-of-the-art accuracy across multiple data sets shows the effectiveness of their work. \n\nHowever, there are some problems to be clarified.\n1. The authors apply several techniques to improve the precision for training with 8-bit floating point, but they do not show the gain for each individual. For example, how much improvement can this work achieve when just using enhanced loss scaling method or a stochastic rounding technique? This should be clearly presented and more experimental comparison is expected.\n\n2. The paper should present a bit more background knowledge and discussion on the adopted techniques. For instance, why the stochastic rounding method proposed in this article by adding a random value in probability can regulate quantization noise in the gradients? And why Resnet-50 demands a large scaling factor?\n\n3. On Table 3, in comparison with Wang et al. (2018), the authors use layers with FP32 (not FP16 in Wang). Thus, it is hard to say the improvement comes from the proposed 8-bit training. This should be clarified.\n\n4. How to set the hyper-parameters, such as scale, thresholds and so on, is not clear in the paper. There are no guidelines for readers to use these techniques.\n\n5. The authors did not give a clear description of the implement for the enhanced loss scaling. They apply different loss scaling methods for different networks. This should be explained in detail.\n\n6. In the experiment, for a single model, some layers are 8-bit, some layers are 32-bit and some layers are 16-bit.  Is the 8-bit training only applicable for a part of the model?  How do we know which layer is suitable for 8-bit training?  ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper propose a method to train DNNs using 8-bit floating point numbers, by using an enhanced loss scaling method and stochastic rounding method. However, the proposed method lacks novel and both the paper presentation and experiments need to be improved throughout. 
Decision: Reject