Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks
Authors: Leopold Cambier
Anahita Bhiwandiwalla
Ting Gong
Oguz H. Elibol
Mehran Nekuii
Hanlin Tang
Venue:  (2020)
Keywords: Low-precision training
numerics
deep learning

Abstract: Training with larger number of parameters while keeping fast iterations is an increasingly
adopted strategy and trend for developing better performing Deep Neural
Network (DNN) models. This necessitates increased memory footprint and
computational requirements for training. Here we introduce a novel methodology
for training deep neural networks using 8-bit floating point (FP8) numbers.
Reduced bit precision allows for a larger effective memory and increased computational
speed. We name this method Shifted and Squeezed FP8 (S2FP8). We
show that, unlike previous 8-bit precision training methods, the proposed method
works out of the box for representative models: ResNet50, Transformer and NCF.
The method can maintain model accuracy without requiring fine-tuning loss scaling
parameters or keeping certain layers in single precision. We introduce two
learnable statistics of the DNN tensors - shifted and squeezed factors that are used
to optimally adjust the range of the tensors in 8-bits, thus minimizing the loss in
information due to quantization.

Published as a conference paper at ICLR 2020
SHIFTED AND SQUEEZED 8-BIT FLOATING POINT FOR- MAT FOR LOW-PRECISION TRAINING OF DEEP NEU- RAL NETWORKS
Léopold Cambier"*1, Anahita Bhiwandiwalla"", Ting Gong",
Mehran Nekuii?, Oguz Elibol and Hanlin Tang"
ICME, Stanford University
Intel Al Lab
@cambler@stanfooe
[anahita www.wwhiwwi.......m............ wa laa ting. gong) @intel com
(mehran.. i nekuii, ogu.. www..nkeiii.u...eeeee.... ibol, han tang
ABSTRACT
Training with larger number of parameters while keeping fast iterations is an in- creasingly adopted strategy and trend for developing better performing Deep Neu- ral Network (DNN) models. This necessitates increased memory footprint and computational requirements for training. Here we introduce novel methodology for training deep neural networks using 8-bit floating point (FP8) numbers. Re- duced bit precision allows for a larger effective memory and increased computa- tional speed. We name this method Shifted and Squeezed FP8 (S2FP8), We show that, unlike previous 8-bit precision training methods, the proposed method works out-of-the-boo for representative models: ResNet-50, Transformer and NCF. The method can maintain model accuracy without requiring fine-tuning loss scaling parameters keeping certain layers in single precision. We introduce two learn- able statistics of the DNN tensors shifted and squeezed factors that are used to optimally adjust the range of the tensors 8-bits, thus minimizing the loss information due to quantization.
INTRODUCTION
Deep neural networks have achieved state-of-the-art performance wide variety of computer vision, audio, and natural language processing (NLP) tasks. This has resulted in an explosion of in- terest around techniques reduce the memory footprint and energy consumption of neural network training and inference (Guo) 2018). Although there are a number of methods to address some of these issues for inference, the most effective method for training is using reduced precision numeri- cal formats.
While 32-bit floating point (FP32) is the most common data format for neural network training, recent hardware have leveraged techniques that allow for training with 16-bit data formats (Köster al. [2077] Micikevicius et al., 2018). However, 8-bit precision training remains an open challenge (Johnson 2018 Kalamkar al. 2019). Current FP8 training methodologies (Wang al. 2018 Mellempudi et al. (2019) require either specialized chunk- baauk accumulation, stochastic rounding techniques, loss scaling maintaining some layers of the network in higher precision. Tuning these knobs is non-intuitive and requires significant experimentation for each individual network.
Accelerating the adoption of 8-bit data in training DNNs requires hardware-friendl and out-of- the-box implementation of FP8. Due to the reduced number of mantissa bits, 8-bit multipliers are smaller and consume less power compared to higher bit representations. In this work we describe novel 8-bit floating point (FP8) format shifted and squeezed FP8 (S2FP8)- which has the following advantages compared to previously proposed 8-bit training methodologies:
Work performed during an internship at Intel
I Equal contribution

---

Published as a conference paper at ICLR 2020
S2FP8 eliminates the need for loss scaling, which requires significant tuning of the loss scale values and schedule for individual topologies
Leveraged by the forward and backward passes of model training, S2FP8 effective in adjusting the range of gradients and also of activations and weights
S2FP8 does not require keeping the first and last layer in FP32 precision, which is needed for other approaches (Mellempudi et al. 2019), however maintains the master weights and accumulations inside the matrix multipliers in FP32
We demonstrate across image classification, translation, and recommendation models that S2FP8 outperforms previous 8-bit approaches, and reaches the accuracy of FP32 models without any addi- tional hyperparameter tuning.
2 RELATED WORK
The success of 32-bit floating point data type in training deep neural networks has increased interest in the feasibility of even lower precision training. The exponential demand for compute involved in training these deep neural networks has lead to multiple advancements in lower precision data types.
Several studies have developed techniques such as loss scaling, stochastic rounding, and others to train effectively in 16-bit (Micikevicius et al.) 2018 Das et al. 2018: (Azim), along with associated hardware support (Markidis al. (018). Using 16-bit fixed point, (Gupta et al. 2015) showed that stochastic rounding techniques were crucial for model convergence even for simple convolutional neural networks. As noted in (Kalamkar al. 2019), Google's 6 format has the same number of exponent bits as FP32, leading the success of that format without commonly requiring hardware intensive requirements such as stochastic rounding or other framework level techniques such as loss scaling.
Although 8-bit formats have significant performance and memory advantages, convergence es- pecially challenging due to loss of accuracy in the backpropogated gradient values. WWag et al. (2088) demonstrated training models with matrix multiplications and convolutions FP8 but they use FP16 with chunk-based accumulations and stochastic rounding hardware. Mellempudi et al.) 2019) also demonstrated success with FP8, accumulating FP32 and using loss scaling techniques on ResNets, Transformer and GNMT networks. However, they too require the first and last layers of the model to be in FP32, and similar to (Banner et al. 2018) leverage Stochastic Rounding tech- niques to maintain model accuracy. Unlike S2FP8 proposed in this work, both of these FP8 training techniques emphasize the need for efficient loss scaling, rounding hardware and restriction on some layers being in higher precision.
Zhou et al. 2016) quantized weights, activations and gradients of AlexNet (Krizhevsky et al. 2012) to 2,2 and 6 bits respectively. But they also need to maintain the first and last convolution layers in full precision and stochastically quantize the gradients. Wu et al. (2018) demonstrate using integers for training LeNet-5 (LeCun al. 1998) and AlexNet with 8-bits for activations, error and gradi- ents and 2-bits for weights. However, these approaches also required custom tuning such as novel initialization techniques and layer wise scaling instead of Batch Normalization and Softmax. These approaches lack generalizabilitg to other models, requiring significant fine tuning.
To the best of our knowledge, there does not exist an out-of-the-boo solution using FP8 in training deep learning topologies without the need for tuned loss scaling techniques, requirements of cer- tain layers being full precision along with efficient hardware rounding schemes like Stochastic Rounding.
3 SHIFTED AND SQUEEZED 8-BIT FLOATING POINT FORMAT
3.1 CHALLENGES OF 8-BIT FLOATING POINT FORMAT
The FP8 format, with bits of mantissa and 5 bits of exponent (ellempudi et al. 2019 is both nar- row (i.e., its dynamic range is very limited, from 2 16 to 2)) and has lower accuracy (the machine epsilon is only 2 3). Figure Al iillustrates the range and accuracy of FP8. In contrast, FP32 ranges from 249 to 122 with machine-epsilon of 2 Tabll A1).
2

---

Published as a conference paper at ICLR 2020
-00
50
40
-30
20
-10
Figure I: The distribution of tensor elements over the course of training for three tensors from the Transformer tiny model on the English- Vietnamese translation dataset. Blue bar indicates representable range of FP8. Left: Many of the tensor elements fall outside of FP8's representable range. Center: Few tensor elements fall outside of FP8's representable range. Right: Initially, most elements within FP8's representable range, but after training, many fall outside of the representable range
On the other hand, tensors involved in neural networks (weights, activations and gradients) are spread across varying scales. As illustrated in Figure the tensor distributions change over the course of training, spanning different orders of magnitude.
As result, 8-bit training usually requires combination of multiple techniques to capture the full dynamic range of values for model training. Some these techniques include:
Loss scaling (Micikevicius et al. 2018) scales the loss L(w) by a constant À before back- propagation This makes the gradients artificially larger, allowing them to fit within the FP8 range. Gradients are then scaled down before being accumulated into the trainable weights as shown Equation 6
Stochastic rounding (Maxfield, 2006 alleviate quantization errors by capturing some of the information discarded when truncating to lower precision at the output of GEMM operation
Between these two techniques, loss scaling is more critical; once the magnitude of the gradients longer be represented in the FP8 range, training convergence will not be possible. However, loss scaling only modifies the gradients. Weights and activations can also (albeit admittedly less frequently) exceed the FP8's representable range of [2 16 216]. In those scenarios, convergence can also be affected.
The issue with loss scaling is that requires user interaction. Models have to be modified, and, more importantly, tedious empirical tuning is required to find the correct loss scaling schedule. While some networks can be trained with constant loss scaling, some, notably Transformers (Mellempudi et al. 2019), require dynamic "back-off" and improved loss scaling. This requires significant trial and error to tune the scaling schedule, slowing down wide adoption of low-precision numerical formats.
3.2 SHIFTED AND SQUEEZED FP8
To alleviate these issues and make neural network training possible with no model modifications or hyperparameter tuning. propose new 8-bit floating point format. Consider a tensor X of size N, i.e., X {X, Instead of directly encoding each X, in FP8, we store X using N FP8 numbers [Y:).M accompanied by (squeeze and shift) factors and B (the "statistics"
Figure 2
Y,
Y,
y,
Y..
N FP8 numbers
8
2 FP32 numbers, "statistics"
Figure 2: The S2FP8 format. A tensor X N numbers is represented by a, 3 and N FP8 numbers Y, related to X through Equation

---

Published as a conference paper at ICLR 2020
-16
0
16
loga Y
0
32 logz [X]
(a) y, the usual FP8 distrihution.
(b) X, for a = 1 and 8 < 0
-32
0
32 logg [X]
(c) X. for < and / =
Figure 3: Impact of the Shifted and Squeezed transformation log, | a log, |X] + B. a let the distribution be as wide necessary (though, with associated loss of precision), and 3 let shift the distribution around any value.
For X; + O, X and Y are then related through
logz (IxG) = logz + Y, |2,
(1)
where the + chosen so that X; and Y, have the same sign. This representation allows for and 3 be chosen so that together with tensor Y they capture most of the dynamic range of the tensor X. As we will see in section 4| this is all that necessary to train networks using 8-bit floating point numbers.
In order for to tensor suitable to represented by FP8 numbers, we enforce that it has zero mean and maximum value within the dynamic range of FP8 (e.g. 15):
N loga (IVi)) = 0 and max i
(og)((Y)) = 15(= 12g2(215))
(2)
where the notation indicates that the sum and the max, respectively, ignore any i such that Y, = 0. Those equations ensure that ( values are distributed with zero mean and each is less than 15, which is ideal for an FP8 format.
By inserting Equation
into
Equation I and by denoting
log, (lx+l)
and m = max logz (Ix+)
(3)
find
15 m
0
B -aj
(4)
This tensor format results in the training procedure (forward pass, backward pass, weight up- date) described Figure 4 Forward and backward MatMul use this new S2FP8 format. Master weights kept in FP32 and updated using S2FP8 gradients. Accumulations inside the GEMM kernel are kept in full FP32 precision. Figure 3 illustrates the impact of a and B. By having those two extra degrees of freedom for each tensor, majority of the dynamic range of each tensor can now be captured, whether very small (B > 0), very large (3 < 1), very narrow (a > 1)) or very wide (a 1).
3.3 LEARNING THE TENSOR DISTRIBUTION
One way to interpret and 3 is to consider them parameters of a distribution generating the ten- sor values logæ (I (). We can then say that, by continuously computing a and 3, we are effectively learning the distribution of log, (l) Figure 5) shows the evolution of j., and 3 for partic- ular tensor ResNet-20. We see that a and 3 converge to, approximately, 5 and 21, respectively. From Equation 1, conclude that:
[X,)).

---

Published as a conference paper at ICLR 2020
FWD GEMM
Σ
(52FP6)
laser (52F98)
FP32
FF3P-9S0FF8
Mester weights
Tayer & (
FPS2-
Update
WS GEMM
FP32-952FP6
FP32
Lass gradients Iayer 6-1 (52FP8)
BWD GEMM
gradients
2 (52FP5)
FP32-922FF9
FP32
Figure 4: Low precision training with S2FP8. T represent the truncation described in Equation 5 from FP32 to S2FP8. When using S2FP8 for training, forward and backward GEMM's only use S2FP8. The master weights are kept in FP32 and updated during the update step.
since 1, this means that X is expanded into Y, i.., X is more narrow than what FP8 allows
since 3 > 0, this means that is right-shifted into Y, i.e., X smaller than what FP8 allows
At convergence, those and values represent the distribution of each converged tensor. Notice that all statistics stabilize in the last third of the training, where the learning rate is decreased, indicating the network converging to its final state.
EXPERIMENTAL RESULTS
In this section, we compare S2FP8 training with baseline FP32 and FP8 training with and with- out loss scaling for: Residual Networks (He al., 2016) varying depths on the CIFAR-10 and ImageNet (Deng al. 2009) datasets, Transformer (Vaswani et al. 2017) on IWSLT' English- Vietnamese dataset (Luong & Manning, 2015), and Neural Collaborative Filtering (NCF) (He al. 2017)
Moviel.ens i Million dataset (Harper & Konstan 2016).
For our experiments, we use the open source Tensorflow Models repository for ResNet and NCF, Tensor2Tensor (Vaswani al., (2018) for Transformer with added S2FP8 data type simulation sup- port using the methodology described in [subsection 4. For a given model, we keep the hyperpa- rameters consistent across FP32, FPS and S2FP8 evaluations.
4.1 SIMULATION METHODOLOGY
We simulated S2FP8 by inserting appropriate truncation function throughout the network, before and after every convolution and matrix-matrix product operations, during both the forward and backward passes. The rest of the network is kept in FP32, and those truncation simulate the low-precision training described in inlsubsection 3.2]
The truncation function takes as input tensor X, computes its magnitude mean and maximum, computes the appropriate a and 3 and finally truncates X by computing
Xtruncated
12- truncaterrs [[]
(5)
where truncaterps is usual FP8 truncation function with RNE (round-to-nearest with ties broken by rounding to even) rounding which easier to implement and most widely supported hardware.

---

Published as a conference paper at ICLR 2020
700
2.00+
1.40+43
300
600
200
(b) Distribution of magnitude log2((X() shifted and squeezed tensor Y with ||| 2 [X
(a) Distribution of the magnitude logg (lX) ) of
original tensor before scaling using and |
m
Q
3.8
40
-4
4.2
30
-2
4.4
20
4.6
-3
50k
100k
0
50k
100k
0
50k
100k
50k
100k
Step
Step
Step
Step
(c) The computed statistics during training for the scale (3), shift (a), as well as the mean of the log values (μ) and the maximum log value (m).
Figure 5: Evolution of the average and maximum magnitude, as well as a and / for CIFAR-10 with ResNet-20. This illustrates how the network is actually implicitly learning the tensors distribution, by repeatedly computing magnitudes a and 3 through and m.
4.2 RESIDUAL NETWORKS
We first present results with Residual Networks of varying depths on the CIFAR-10 image recogni- tion dataset. We trained the model on 1 GPU using standard parameters: 250 epochs, batchsize of 128, SGD with momentum 0.9, initial learning rate of 0. decreased by a factor of 10 after epochs 100, 150 and 200.
Table and Figure A2 presents the results. observe that S2FPS reaches almost exactly the FP32 baseline, sometimes poor accuracy. Finally, FP8 with constant loss scaling of 100 (FP8+LS(100) can reach the baseline. Both S2FP8 and FP8+LS(100) have similar performances, but S2FP8 can do without any extra hyperparameters or tuning from the user's perspective.
improving over it. Out-of-the-Be FP8 does not converge and has very
CIFAR-10
FP32
S2FP8
A
FP8
FP8+LS(100)
ResNet-20
91.5
91.1
0.4
17.9
91.1
ResNet-34
92.5
92.0
0.5
13.5
92.0
ResNet-50
93.0
93.2
-0.2
11.5
92.9
Table 1: Validation accuracy (in %) for image recognition on CIFAR-10 with 20/344.00/34/50.
We also evaluate S2FP8 on the 1000 class ImageNet dataset. Here, we trained the network on GPUs using standard parameters: 90 epochs, batchsize of 256, SGD with momentum of 0.9, initial learning rate of decreased by a factor of 10 after epochs 30, 60, 80 and 90. Table and Figure present the results.
Again, we observe that S2FP8 gets very close to the FP32 baseline. Out-of-the-box FP8 quickly diverges and does not converge all. For FP8 with loss scaling to converge, one has to not truncate the first and last layer, as consistent with (Mellempudi et al. 2019), which we denote as Ex in Table 2 below. A loss scaling of 10,000 can then be used reach the baseline (FP8+LS(10K)+En) Finally, stochastic rounding can be added and it slightly improves the precision (FP++S((100)+Exe> However, both those
out-o--the-box, they require loss scaling tuning and some layers

---

Published as a conference paper at ICLR 2020
to be kept in full precision. S2FP8 does not suffer from that, thanks to its improved quantization: all layers can be truncated and no loss scaling is required.
ImagenetIk
FP32
S2FP8
A
FP8
FP8+LS(10K)4EX
ResNet-18
70.3
69.6
-0.7
NaN
68.7
68.9
ResNet-50
76.2
75.2
-1.0
NaN
75.3
75.5
Table 2: Validation accuracy (in %) for image recognition on ImagenetIk with ResNet-
Top-1 1 accuracy (%)
Loss
L2 Loss
80
FP32
1.2
FP32
60
S2FP8
S2FP8
0.8
FP32
S2FP8
0.4
250k
500k
0
250k
500k
0
250k
500k
Step
Step
Step
Figure 6: Comparing Top-1 accuracy and Loss of S2FP8 with FP32 for ResNet-50 on Imagenetlk
4.3 TRANSFORMER
We also tested S2FP8 on a small Transformer (Transformer Tiny) on the English-Vietnamess Vietnamese dataset. The model has 2 hidden layers of size 128, and filter of size 512, and is trained using Adam optimizer (Kingma & Ba 2014).
Table Jnd Figure scaling. We tried many loss scaling schedules (constant and exponential, with various initializations) and report the best result. As one can see, S2FP8 reaches the bascline with no hyperparameter tuning. FP8, on the other hand, does not, even after extensive loss scaling tuning. This shows the value an out-of-the-boo method for the user.
Finurrrrre 7 show the result, where compare FP32, $2FP8 and FP8 with exponential loss
En-Vi Transformer tiny
FP32
S2FP8
A
FP8
FP8+LS(exp)
25.3
25.3
0.0
NaN
21.3
Table 3: BLEU Score (Papineni et al. 2002) Vietnamese dataset with Transformer tiny.
(from 0 to 100) for translation task on the English-
4.4 NEURAL COLLABORATIVE FILTERING
The Neural Collaborative Filtering (NCF) network comprises of embeddings for users and items from the Moviel ens dataset, that are then passed to Multi-Layer Perceptron (MLP) network to learn the user-item interaction. Matrix-multiplicatio operations are the building blocks of such models. We compare S2FP8 with FP32 and FP8 without loss scaling. We simulate Matrix-Multnn Matiix-Muliplliattions and look-ups from the embeddings in S2FPS and compare it to FP8 with RNE. We trained the model on the MovieLens Million dataset with the following standard paramaters: 20 iterations, batchsize of 1024 on 4 GPUs, predictive factors, learning rate of 0.0005 using the Adam optimizer. Figure and Table 4 show the result, where compare FP32, S2FP8 and FP8 without loss scaling.
This again shows that S2FPS easily reaches the baseline oft-of-the-Bee without tuning of any sort. FP8 gets relatively close, but cannot reach the baseline.

---

Published as a conference paper at ICLR 2020
BLEU Score
Loss
25
FP32
6
S2FP8
10
FP32
S2FP8
2
125k
250k
0
125k
250k
Step
Step
Figure 7: Comparing BLEU score and Loss of S2FP8 and FP32 for Transformer tiny on En-V
dataset
Hit Ratio
NDCG
Loss
0.4
0.65
0.3
FP32
S2FP8
0.6
0.35
0.25
0.55
FP32
FP32
0.3
S2FP8
S2FP8
0.5
0.2
20
10
20
10
20
Iteration
Iteration
Iteration
Figure 8: Comparing Hit Ratio, NDCG and Loss of S2FP8 and FP32 for NCF on MovieLens- IM
HARDWARE ASPECTS
S2FP8 is new data type and requires its own circuitry to be implemented in a tensor processing en- gine. However, the added overhead is very minimal and affects neither data throughput compute speed. order to convert FP32 tensors into S2FP8, two hardware (HW) components are needed. One is to calculate each tensor's statistics (Equation 3), which bring minimal HW complexity. To make compute operations even easier these statistics could be stored lower precision such as FPS/INT8. The other component is to adjust the exponent and mantissa of all those tensor elements by applying the squeeze (a) and shift (8) factors in Equation 4 before truncating them into their 8-bit placeholders. The shift could be done using simple element-wise add/subtract operations on the exponents, and element-wiss squeeze could be applied to the mantissa portions. Another con- sideration is within the tensor processing engine(e.g.. GEMM engine) which requires the and factors while doing the calculations. The FP32 result will be converted back to S2FP8 when needed (e.g., to store back in memory) as shown Figure
6 CONCLUSION
We introduce novel 8-bit floating point data type (S2FP8), that gives competitive performance in comparison to of FP32 baselines over a range of representative networks. S2FP8 makes of shifted and squeezed factors to shift and rescale the range of tensors prior to truncation. S2FP8 allows training of neural networks with an 8-bit format while eliminating the need for loss scaling tuning, hardware-complex rounding techniques. In addition, compared to existing FP8 implemen- tations we also eliminate the restriction of maintaining the first and last layers in FP32. Decreasing
Movielens million
FP32
S2FP8
FP8
NCF
0.666
0.663
0.003
0.633
Table 4: HR Score for NCF the Movielens 1 million dataset.
8

---

Published as a conference paper at ICLR 2020
the number of bits enables larger models to fit single device and results in faster training. As part of future work, we plan to extend the use of S2FP8 to train additional DNN topologies and also simplify the squeeze and shift statistics from hardware implementation point of view. We also plan to explore the use of reduced precision to store the statistics and the extendability of this ap- proach efficiently represent broader suite of low precision formats like 8-bit POSIT (Gustafson & Yonemoto 2017). 4-bit floating and integer data types.
ACKNOWLEDGMENTS
We would like to thank Naveen Mellempudi, Pratap Prasad, Prasanna Singamsetty and Cory Stephenson for insightful discussions.
REFERENCES
Anwarul Azim. Low precision arithmetic operations in deep neural networks: An overview.
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. In Advances in Neural Information Processing Systems, pp. 5145-5153, 2018.
Dipankar Das, Naveen Mellempudi, Dheevatsa Mudigere, Dhiraj Kalamkar, Sasikanth Avancha, Kunal Banerjee, Srinivas Sridharan, Karthik Vaidyanathan, Bharat Kaul, Evangelos Georganas, et al. Mixed precision training of convolutional neural networks using integer operations. arXiv preprint arXiv: 009000000000.....00000000 2018
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Yunhui Guo. A survey on methods and theories of quantized neural networks. 055/1808.047522 2018. URL htt::://xxxi..org// abs/ absp//bs//88 04752
CoRR,
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Conference on Machine Learning, pp. 1737-1746, 2015.
John L. Gustafson and Isaac T Yonemoto. Beating floating point at its own game: Posit arithmetic. Supercomputing Frontiers and Innovations, 4(2):71-88 86, 2017.
Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm
transactions on interactive intelligent systems (fiis) 5(4):19, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference computer vision, pp. 630-645. Springer, 2016.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural col- laborative filtering. In Proceedings of the 26th international conference world wide web, pp. 173-182. International World Wide Web Conferences Steering Committee, 2017.
Jeff Johnson. Rethinking floating point for deep learning. CoRR, abs/1811.01121, 2018 URL http://arxiv.. http://wwwg/b////////ea/ 001721
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka. Jianyu Huang, Hector Yuen, et al. A study of 6 for deep learning training. arXiv preprint arXiv: 1905. 12322, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.688 69802,6680.00 2014.
Urs Köster, Tristan Webb, Xin Wang, Marcel Nassar, Arjun K Bansal, William Constable, Oguz Elibol, Scott Gray, Stewart Hall, Luke Homof, et al. Flexpoint: An adaptive numerical format for efficient training of deep neural networks. In Advances in neural information processing systems, pp. 1742-1752, 2017.

---

Published as a conference paper at ICLR 2020
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (0:11)2778-2324 1998.
Minh-Thang Luong and Christopher D. Manning. Stanford neural machine translation systems for spoken language domain. In International Workshop on Spoken Language Translation, Da Nang, Vietnam, 2015.
Stefano Markidis, Steven Wei Der Chien, Erwin Laure, Ivy Bo Peng, and Jeffrey S Vetter. Nvidia tensor core programmability, performance & precision. In 2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp. 522-531. IEEE, 2018.
Clive Maxfield. An introduction different rounding algorithms. Programmable Logic Design Line, pp. 1-15, 2006.
Naveen Mellempudi, Sudarshan Srinivasan, Dipankar Das, and Bharat Kaul. Mixed precision train- ing with 8-bit floating point. arXiv preprint arXiv: /905. 12334, 2019.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In International Conference on Learning Representations, 2018. URL
net/ htttps://openrevee htwprrrrwwwe.eee//ouuiiiiiiieeee Forum?id=r1
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Anmual Meeting on Association for Computational Linguistics, ACI. A01, pp. 311-31111 1-318, Stroudsburg, PA, USA, 2002. Association for Computational Linguistics. doi: 10.3115/10733033117331 URL httph:ttp//pmmm httppp::///....r///1....
3115/1073083 1073135
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, fukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. Tensor2tensor for neural machine translation. CoRR, 2018. (ttt://t::///rtt http:///xxxvv...o/abbeee wwwp://ggg/aa/aao.e..oeeess abs//a/3.33 07416
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yo Yu Chen, and Kailash Gopalakrishnan. Train- ing deep neural networks with 8-bit floating point numbers. In Advances in neural information processing systems, pp. 7675-7684, 2018.
Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. Training and inference with integers in deep neural networks. arXiv preprint arXiv:1802.04660 2018.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 1rXiv:1606.00166, 2016.
10

---

Published as a conference paper at ICLR 2020
A
APPENDIX
A.1
SUPPLEMENTARY TABLES AND FIGURES
Format
Bits
s/e/m
Min sub-
Min
(Approx.) Max normal
Machine epsilon
Range
normal
mal
IEEE-FP32
32
1/8/23
2- 149
2-126
2128
2-24 24
2
IEEE-FPI6
16
1/5/10
2-24
2 2-14
216
2 2-11
240
BF16
16
1/8/7
2 2-133
2 2-126
2
2
2261
FP8
8
1/5/2
2 2-16
2 2-14 14
2
2
232
Table A1: Comparing several floating point formats. s/e/m indicates the number of sign (s), exponent (e) and mantissa (m) bits.
Models
Datasets
FP32
BF16
FP8
FP8+ether recipes
S2FP8
ResNet-20
CIFAR-I0
91.5
91.7
17.9
1 Scale=100)
91.1
ResNet-50
CIFAR-10
93.0
93.2
11.5
92.9(L Scale=100)
93.2
ResNet-50
ImageNet
76.2
76.5
NaN
75.3(Loss Scale=10K,
75.2
FP32 for first and last layers)
NCF
MovieLensIM
0.666
0.653
0.633
0,663
Transformer- En-Vi
25.3
25.6
NaN
21 2loss Scale=Exp)
25.3
tiny
Table A2: Comparing FP32, BF16, vanilla FP8, FP8 with tuning and S2FPS on the model ResNet copoooToo--acccccy..l NCF(Hit ceanssfommertinynBL... score).
Numbers density
4
3
2
1
16
0
8
16
loga ([
Figure A1: The range and precision of FP8. Bar indicate the number density between each power of 2. Since FP8 has 2 mantissa bit, the density is 4 (except in the denormals), and the associated machine epsilon is 2 1/8. The normal representable range goes from 2 2-14 to (1 - 2 32
with denormals from 2-16 2 2-14
A.2 SUPPLEMENTARY EQUATIONS
a(AC) 8w (w) X Øc
1((L) a Ow
(w) + 1 (k+1) = (A) (k)..
(6)
Ow
11

---

Published as a conference paper at ICLR 2020
Top-1 accuracy (%)
Loss
L2 Loss
3
100
FP32
FP32
S2FP8
S2FP8
0.3
80
FP32
0.2
S2FP8
0
50k
100k
50k
100k
50k
100k
Step
Step
Step
Figure A2: Convergence of ResNet-50 with the CIFAR-10 dataset
12


Reviews:
[{"review_id": "SkeEF48CKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "There has been a great deal of interest and research into reduced numerical precision of weights, activations and gradients of neural networks. If, for example, 16 bit floating point can be used instead of 32 bit floating point, then the memory bandwidth is halved along with significant gains in computational performance.\n\nIn this work the authors propose an 8-bit floating point format (denoted S2FP8) for tensors. In general, computing activations and gradients with such low precision at training time, has generally proved challenging without a number of tricks such as scaling the loss of each minibatch to within a reasonable range. Such ``tricks'' can be difficult to tune for each problem.\n\nThe key idea here is that for each tensor of 8-bit numbers, two 32 bit floating point statistics are recorded as well. These determine (in log-space) a scale and an offset for the 8-bit numbers (eq 1). This means that in this format tensors of significantly different scales can be well-represented (although larger scales necessarily implies low precision).\n\nMatrix multiplications are done in FP32 precision and then converted in S2FP8 format. This requires an additional step to accumulate the summary statistics of each tensor in order to convert from FP32 to S2FP8 (the mean and the max of the tensor elements in log space).\n\nThe weights of the network are stored in FP32 and the gradients and activations are computed in S2FP8 and used to update the weights.\n\nThey test this approach in ResNet, a small transformer and a MLP for collaborative filtering. They find it reaches similar performance to FP32 where standard FP8 format has worse performance or results in Nan's.\n\nImprovements in computational efficiency, both at training and inference, are active areas of research and this work contributes a novel approach using summary statistics. However, there are a several ways this work could be improved.\n\n1. There is no comparisons with bfloat16, which is becoming a widely used approach to lower precision and is gaining significant hardware support [1].\n\n2. Discussion and analysis regarding the need to gather summary statistics after each matrix multiplication (or other tensor operation). It is claimed that this brings minimal HW complexity, but this doesn't seem well justified. For a large tensor, this additional reduction to compute statistics may be expensive (in memory bandwidth and computation), particularly since this is done with FP32.\n\n3. Even with the current implementation on a GPU, it should be possible with kernel fusions to gain some significant savings in memory bandwidth (and therefore computational speed), but there is no attempt anywhere to show any runtime benefit on current hardware.\n\nMinor issues:\n\nSome captions are very terse and the figures would benefit from a clearer explanation (e.g. figure 6).\n\n[1] https://en.wikipedia.org/wiki/Bfloat16_floating-point_format", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Byl9vmZJir", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focuses on training neural networks using 8-bit floating-point numbers (FP8). The goal is highly motivated: training neural networks faster, with smaller memory footprint and energy consumption.\nThe proposed approach S2FP8 eliminates the need for loss scaling and does not require keeping some of the layers in FP32 precision as in Mellempudi et al. (2019).\nThe paper is well written, easy to follow, and provides a detailed background for readers who are not knowledgeable in this field.\n\nOn the downside, the first sections give the impression that FP32 is not used at all: \"S2FP8 does not require keeping the first and last layer in FP32 precision, which is needed for other approaches (Mellempudi et al., 2019).\". However, Section 3.2 says that \"Master weights are kept in FP32\" and that \"Accumulations inside the GEMM kernel are kept in full FP32 precision\". I think this should be stated earlier, because otherwise, the introduction is overclaiming.\n\nThe evaluation is very convincing - the approach is demonstrated for image classification, Transformer-based translation, and neural collaborative filtering. S2FP8 outperforms previous FP8 approaches and reaches the accuracy of FP32 out-of-the-box. \nIt would be interesting to see the Transformer results with S2FP8 on additional datasets rather than the English-Vietnamese dataset only, which is considered a low-resource dataset, and on Transformer-base rather than \"tiny\". If you have such results on additional NMT datasets, I would be interested to see them, even if the performance of S2FP8 is worse than FP32 (I will not reduce my rating because of lower results there).\n\nOverall, this is a good paper that presents an important contribution. Thus, I recommend accepting this paper.\n\nMinor:\n* Section 2, paragraph 3: \"(Mellempudi et al. 2019) also demonstrated ..\" the name of the authors should be outside of the parentheses (\\citet). Same in 4th paragraph: \"(Zhou et al., 2016) quantized ..\"\n* Equation (2) is a little difficult to read because of the sequence \"and max log\", in which each word has a different role. It might worth to break it into two lines, add brackets, or use logical \"and\" instead of the word \"and\"?\n* Equation (3) uses \"i-prime\" in the argument for \"max\", but \"i-prime\" is not used.\n* Figure 4 is referenced before Figure 3, this is a little confusing (the reader needs to scroll down for Figure 4, and scroll up for Figure 3).\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Bygzt_NR9S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper suggest the method to train neural networks using 8 bit floating point precision values. The method requires a hardware modification to evaluate per tensor statistics, which intuitively capture dynamic range that a given tensor values may fall into. Those per tensor statistics are used to map 8-bit representation into a rational number. Statistics themselves are calculated using 32 bit floating point numbers.\n\nWhile the method is interesting, I do not think it is practical due to the required hardware modifications. I am by no means not a hardware design expert, but I am not convinced that the gain of using 8 vs 16 bit floating point numbers outweights any extra complexity of hardware implementation. Mapping between representation and actual numerical values is much more complex than when using standard floating point representation (as any given numerical value is defined by a single its 8 bit representation + alpha + beta tensor statistics), integrated circuitry used to execute arithmetic operations would likely be much more complex than when using standard floating point operations. It is plausible that the added complexity (in terms of transistor count) would negate any potential price or energy savings over simply using 16 bit floating point representation. This should definitely be discussed in depth in this paper where the main contribution is an algorithm to be implemented in hardware.\n\nAs different layers will have different (alpha,beta) tensor statistics, from the computational perspective they would look like different data types which would be cast into the same common type before doing any arithmetic operations on them. This would almost certainly require extra computational steps to be done potentially negating computational benefits over simply using 16 bit floating point numbers.\n\nThe algorithm evaluates per-tensor and not per-tensor-element statistics. This may work well in cases where all entries in a given tensor are of the same dynamic range, but may break down in other cases: for example, diagonal entries in a matrix multiplication layer inside LSTM may have very different statistics comparing to non diagonal entries.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1xG2Yfljr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Good ideas\nImportant area\nImpressive results. \nCould be very useful for many embedded applications. \n\nI’m not a hardware expert but can see why this would be Useful.                            \n\n                                                                                                                                                                                                                                                                                                                    ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Main description:  paper focuses on training neural networks using 8-bit floating-point numbers (FP8). The goal is highly motivated: training neural networks faster, with smaller memory footprint and energy consumption.

Discussions
reviewer 3:  gives a very short review and is not knowledagble in this area (rating is weak accept)
reviewer 4: well written and convincing paper, some minor technical flaws (not very knowledgable)
reviewer 1: interesting paper but argues not very practical (not very knowledgable)
reviewer 2: this is the most thorough and knowledable review, and here the authors like the scope of the paper and its interest to ICLR. 
Recommendation: going mainly by reviewer 2, i vote to accept this as a poster
Decision: Accept (Poster)