FP8 Quantization: The Power of the Exponent
Authors: Andrey Kuzmin
Mart Van Baalen
Yuwei Ren
Markus Nagel
Jorn Peters
Tijmen Blankevoort
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neural network quantization
8-bit floating point
post-training quantization
quantization-aware training

Abstract: When quantizing neural networks for efficient inference, low-bit integers are the go-to format for efficiency. However, low-bit floating point numbers have an extra degree of freedom, assigning some bits to work on an exponential scale instead. This paper in-depth investigates this benefit of the floating point format for neural network inference. We detail the choices that can be made for the FP8 format, including the important choice of the number of bits for the mantissa and exponent, and show analytically in which settings these choices give better performance. Then we show how these findings translate to real networks, provide an efficient implementation for FP8 simulation, and a new algorithm that enables the learning of both the scale parameters and number of exponent bits in the FP8 format. Our chief conclusion is that when doing post-training quantization for a wide range of networks, the FP8 format is better than INT8 in terms of accuracy, and the choice of the number of exponent bits is driven by the severity of outliers in the network. We also conduct experiments with quantization-aware training where the difference in formats disappears as the network is trained to reduce the effect of outliers.

FP8 Quantization: The Power of the Exponent
Andrey Kuzmin', Mart Van Baalen', Yuwei Ren, Markus Nagel, Jorn Peters, Tijmen Blankevoort
Qualcomm Al Research" {akuzmin, mart, ren, mar.uuu, tinnesstiijmennin tijmen]qqti. s.jjeaajtti.uaaacm.ocoo qualcomm.com
Abstract
When quantizing neural networks for efficient inference, low-bit integers are the go-to format for efficiency. However, low-bit floating point numbers have extra degree of freedom, assigning some bits to work on an exponential scale instead. This paper in-depth investigates this benefit of the floating point format for neural network inference. We detail the choices that can be made for the FP8 format, including the important choice of the number of bits for the mantissa and exponent, and show analytically in which settings these choices give better performance. Then show how these findings translate to real networks, provide an efficient implementation for FP8 simulation, and a new algorithm that enables the learning of both the scale parameters and number of exponent bits the FP8 format. Our chief conclusion is that when doing post- pest-training quantization for wide range of networks, the FP8 format better than INT8 in terms of accuracy, and the choice of the number of exponent bits is driven by the severity of outliers in the network. We also conduct experiments with quantization-awar training where the difference in formats disappears as the network is trained to reduce the effect of outlier
Introduction
Neural network quantization is one of the most effective ways to improve the efficiency of neural networks. Quantization allows weights and activations to be represented in low bit-width formats, e.g. bit integers (INT8). When executing networks any device, this leads to reduction in data movement and enables the use of low bit-width computations, resulting in significantly faster inference and lower energy consumption.
Roughly speaking, values in neural networks are represented in either integer (INT) or floating- point formats (FP). Most quantization research has gone into low-bit integer formats such as INT8 and INT4, as the corresponding hardware for this is widely available. However, some research suggests there might be benefits to using low bit-width floating-point formats for neural network computations
In this paper, we set out to thoroughly investigate the potential benefits of the floating point format for neural network inference. We show that floating point numbers add an extra dimension on top of the INT8 format that makes outliers in the quantized distribution less harmful to the overall performance. We investigate the effect of the quantization formats on neural network quantization on three levels: 1) Analytically for several common data and weight distributions, 2) practically in INT8 and FP8 post-training quantization (PTQ) settings, and 3) quantization-an -wware training (QAT) settings with both INT8 and different FP8 formats. We will show there is strong agreement between our theoretical results and our practical results on real networks.
"Equal contribution
"Qullcomm AI Research is an initiative of Qualcoom Technologies, Inc. Code will be made available http::///sstttahccc/////aalaaaae
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Unsigned INT4 grid. unsigned FP4 2M2E grids with blases
Histagrams of quantbration
Unstgned 1972
Uniform cuant zation
PPB 5M2E
PPB 6N
2Hat; b=2
798 SK-8
7P6 2M30
16
2902, bm33
2
22 2- 2-
2044
2" 27 2-2 25
Figure 1: Left: Unsigned INT4 quantization compared to unsigned FP4 2M2E quantization with two different biases. Subnormal ranges marked with crosses, normal range with circles. Compared to uniform quantization, FP quantization has larger dynamic range, but less precision for larger values. Right: Histograms of the positive values representable by 8 bit signed integers and signed 8 bit floating point formats. Histogram bins are log-spaced, to show the relation between dynamic range (i.e. number of bins) and precision (i.e. values per bin) between different formats. FP8 formats allow values with larger dynamic range, at the expense of precision. Intuitively, changing the (floating point) bias value for each distribution would move these histograms to the left or right without changing their shape.
In order to compare the two formats, we introduce a novel floating-poing quantization simulation implementation that enables us to quickly run PTQ and QAT experiments with many floating point formats. Furthermore, this quantizer enables us to learn the FP8 exponent bias value, as well as the best trade-off between the number of bits used for the exponent and the mantissa, making good use of the flexibility the floating-poin format provides and providing a way to learn this automatically without manual intervention. Our conclusion from the study presented in this paper is that floating- point quantization can be better than integer quantization for inference, but it needs a carefully tuned bias term and correct bit-settings between the exponent and mantissa.
2 Background
2.1 Integer quantization
Quantization is often applied to enable efficient integer matrix multiplications instead of expensive 32-bit floating point (FP32) multiplications. A matrix X Rmmx is quantized an integer matrix X (int) with associated scale follows: X (éme) = clip (1 :Emin::aaaa where [-]
indicates the round-to-neares operation, and clip(;, indicates the element-wise clipping operation, which ensures that (int) can be represented in the chosen bit-width [3]]_ The dequantization operation then produces matrix ( which is a quantized approximation of the input matrix X:
(() x (int) = X. This allows the use of efficient integer matrix multiplication
R4 For
matrix Y e Bonk
XY = (
S2 X (int) (int).
2.2 Floating point number system
floating point number set F 18 a whose elements are defined follows:
d de
d
f=
1
(2)
2m
where s E {0, 13 is the sign bit, de E {0, 1} is the m-bit mantissa, p E Z;0 < < 2" is the e-bit exponent, and b is an integer exponent bias, commonly defined to be 2--1
Floating point numbers can be seen as uniform m-bit grid between two consecutive (integer) powers of two 2a, 2a+1. The distance between grid points in the range [2 2a+1] is 2a-m. Increasing the number of mantissa bits thus increases the number of grid points in each range [2". 20+1]. In the definition provided earlier in this section, the number of ranges [2n. 2=11 that can be represented by a floating point number system, is determined by the number of exponent bits e. Increasing the number of exponent bits thus increases the dynamic range (i.e. ratio between largest and smallest non-zero value) of values that can be represented. Any fixed bit-width floating point number system must make trade-off between the dynamic range of representable values (e), and the precision

---

on enTor
Quantiration erter weightidd by p100
Ouartiaatoo
1001 20e3)
1.54
Ouanttatten MEz
0.00
1100
5,000
EPVEEEER DE ELAON IUIRE A.100
EVDCIONIE 030% 431% 2.300
1804
Figure 2: (left) Rounding error for quantization grids: FP8 with 4-bit exponent, FP8 with 3-bit exponent, INT8. (middle) The rounding error weighted by the probability density function from the first column. (right) Exponent bias ablation for zero mean Gaussian distribution for FP8 format. We consider three versions of the format: fixed format with the bias 2n- integer bias, FP32 scale.
(m.) For example, IEEE-754 32-bit 'full-precision" floating point numbers (FP32) use sign bit, 23 mantissa bits, and 8 exponent bits. The resulting effeet that, compared to integer formats, floating point formats have more precision close to zero, as the ranges [2 2 will be smaller for lower values of a, and less precision away from This is visualized in the left plot in Fig. [I Intuitively, they a better match for peaked distributions like Gaussians that have more density around O, and better fit for distributions with large tails and outliers like the Student's t-distribution. The right plot in Fig. shows the different distributions of values for various FP8 formats.
Note that this definition does not allow for a representation of To allow 0 to be represented, the exponent value p 0 is reserved to indicate subnormal numbers. In this case, the exponent value is implicitly set to 1, and f (-1)*21-b (0 + d1 + 22 +... 2 representation of 0, subnormal numbers also allow a graceful representation of values close See Fig. M for an intuition behind this. In Section we show (de)quantization operations analogous to those INT8.
dm Besides allowing the exact
2.3 Assumptions and extensions
In this work we make number of assumptions and consider several extensions of the standard floating point format. We assume that the exponent bias b not restricted to the value 2--1, Instead, we introduce (per-tensor or per-channel) quantization scale similar to the quantization scale used in integer quantization [3 26]. Since there is no standard allocation of mantissa and exponent bits for 8 bit floating point formats, consider various allocations of mantissa and exponent bits, assume that the choice of trade-off between precision and dynamic range will have more impact for lower floating point bit- bit-widths. We use the notation xMyE to denote format with mantissa bits and exponent bits. Specifically, we consider the formats 5M2E, 4M3E, 3M4E and 2M5E.
Expected quantization error
In this section we perform an analytical analysis of the FP8 and INT8 formats and show that formats with more exponent bits perform better when outliers are factor. This theoretical result lays foundation for the comparison between the formats that is predictive of when to use the FP8 format and how many exponent bits to use.
We investigate the error induced by quantizing values drawn from several insightful distributions, considering the expected MSE of these values it has been shown to be indicative of the final loss in a neural network
3.1 Expected quantization error
Given quantization grid a {ai, a.,..... ). we can define the quantization operation Qo (w) and the corresponding quantization error Ra (w))
(w) as, i = aigmin/w ail.
R.(w) Qa(w)
3

---

13TC
1.13
915
17044
S4MM:
Meggu
Socki
Figure 3: (left) Quantization expected SQNR for different distributions and different formats including Lloyd-Max non-uniform quantizer 30 52) The Student's- s-t distribution clipped at [-100, 100] to avoid an unbounded error. (middle) scalar product expected SQNR, (right) Expected SQNR for Resnet18 layer.
Examples of the rounding error function for the 3M4E, 4M3E, and INT8 formats are shown in fig. left). We model neural network weights as a random variable W Pw.(w). The expected value of the quantization MSE can expressed as follows:
Chmmn
(w amin) ) pw + / (amaz w) dw,
B ((R)(W))
(w) +
where Omin min; OE and Qyma max; O; are the limits of the quantization range. The first term corresponds the rounding error, while the remaining two terms correspond the clipping error. We will use this formulation to analytically compute the quantization errors for different distributions. This is done by splitting the integration into sub-intervals corresponding to each point of the quantization grid. We present further details of this procedure the appendix A. An example of the rounding error function weighted by the probability density is given in fig. middle).
3.2 Scalar product quantization error
We can now model the expected output error for a quantized scalar product, the essential building- block for both fully-connected and convolutional layers. In order model the activation values, we im- troduce random variable X ~ (z) and the corresponding quantization grid | = {B,, 6,,33 32..... ... B 3, R. While computing the scalar product of X and W, we only consider the error incurred by quantization of the inputs, as the error incurred by quantization of the output Yis identical to the input quantization error as described the previous section.
We denote the output tensor Y = X, and its quantized version as Yq. The quantization error AY then can be expressed
AY YQ= Y Qa (W))Q) WX WR2(X) (R), (22
((())
(5)
The sample mean value AY corresponds to the mean error of the scalar product between two vectors W E R and X E RD, where W, (w)),, X; ~ (x) weights and activations samples:
AY - (Q(W), Q(x)) (W, x
(6)
Computing E(AY²) for different distributions and quantization grids allows us to analyze how they would work in practical scenarios. many cases, the expected output MSE can be approximated by the following weighted sum of the rounded errors on W and X:
E(AY?) (W)) [ d3po(z)ddr +(G)))++E(RR(X)))
w² p'pwww)ddw.
We see that the output error mainly depends on the quantization error of the inputs and the spread of the distributions of the inputs. We describe the full procedure for computing E(AY2) and further analysis in appendix A.2

---

Discussion We use these formulations to analytically compute the quantization errors for several distributions. We consider the uniform distribution, Gaussian distribution and the student-t distribution. The latter is essentially a more heavy-tailed Gaussian, allowing us to model what happens when outliers are added into the picture.
It is important to choose the bias term in the FP8 format properly. We see in fig. rright) that the standard fixed format of setting the bias 2 can fail. This happens when values are either clipped too much, or not enough grid-points are used as the representable range is too big. We also see that having an integer bias performs worse than having a floating-Pol point bias. This is similar to what happens in INT8 quantization, where setting the scale parameter correctly can have a large impact on the network's performance [4 As the bias shifts the dynamic range, this effect less strong for the FP8 formats with a high amount of exponent bits that have a high dynamic range. For lower exponent-bit one would preferably use a per-channel bias/scale parameter, similar to what is done in per-channel quantization [26]. Further justification for this can be found in section 5.2
We also analyze the effect of different settings of the number of mantissa and exponent bits. The results of this are presented in fig. 3 We observe a clear pattern. For a uniform distribution, the INT8 format performs the best. For the Gaussian distribution, the format with 2 exponent bits performs best. Then, when increasing the relative outliers by decreasing the degrees of freedom in the Student's t-distribution, the best format tends towards having more exponent bits.
In neural networks, most weights and activations can be modeled relatively well by a Gaussian distribution. So based on this analysis, we would expect the 5M2E format work the best for most well-behaved layers with little outliers. Specifically in the weights outliers are infrequent, as the weights often explicitly, and otherwise implicitly regularized [5 For example, consider a layer of Resnet18 8 model pre-trained on ImageNet. We fit Gaussian distributions in the weights and activations sample, and compute the expected MSE analytically fig. Bl (right). We see that the 5M2E format works the best in this case. A more detailed per-layer study is given in appendix B.]
However, the stronger the outliers number of bits will help with its quantized representation. We would thus predict that for networks with severe outliers, formats like M4E3 would perform better. We will show in section 5 that these predictions hold for real networks, and that networks that are known to have larger activation outliers such transformers benefit the most from more exponent bits.
in the distribution, the more the FP8 format with higher
4 FP8 quantization simulation
In the previous section our analysis showed that FP8 quantization can theoretically yield better tensor reconstruction MSE than INT8 quantization, but careful selection of the division between exponent and mantissa bits, well as the value of the exponent bias are crucial. We investigate whether these findings can be extended to FP8 quantization of neural networks. In order to do so, we need an efficient simulation of FP8 guantization. Methods commonly used for FP8 quantization are either too slow, e.g. nearest grid point search or bit-masking [22]] or too complicated, e.g. custom CUDA kernels 400. for quick experimentation.
In this section we introduce novel method of FP8 quantization simulation. This method can easily be implemented in common deep learning frameworks. An added benefit of this method is that it exposes the parameters of the FP8 quantizer (i.c., the number of mantissa/cxponent bits and the exponent bias), thus allowing the parameters to be learned by back-propagation.
4.1 FP8 simulated quantization formulation
In our FP8 quantizer we exploit the fact that FP8 quantization can be seen as the union of m-bit uniform guantization grids between consecutive integer powers of two [2n-b, 2p+1- This means we can simulate FP8 quantization of an input vecto x using the same method as for simulation of uniform quantization as described in Section 2.1. with the distinction that each element 2; in æ has its own associated scale s;:
(g)
S
20
(8)
5;
This method extends readily matrices or higher- ciaheer-imeniionall tensors.
5

---

where x;;") denotes xa quantized to floating point. The scale 8; depends on the number of mantissa bits m and the range [2p-6, 20)1- in which 2; falls. This is given by log2 8; P, [ m.
To ensure that i; can be represented given m, e and b, both x; (9) and need to be clipped. Values of (4) greater than maximum value or smaller than -c are clipped and - respectively, where c = (2 2- " - is the largest representable value for a given floating point format. Since 21-b-1 is the smallest representable value, values of smaller than 1 - b m are clipped to 1-b- m.
Note that this approach identical to the quantization operation defined in Eq 3 provided that rounding mode in Eq. 3 matches the tie- fie-breaking procedure in Eq for numbers equidistant two numbers F. See Fig. for an intuition.
2 lo x
2 2llogs xd-M
0
2-1
20
2
Figure 4: Graphic depiction of the FP8 quantizer.
In case the scaling factor 7 # this needs be reflected in ps- In order to accommodate the scaling factor, we first fold it into a reparameterized bias value b = loga 7 We then compute p; as follows:
logal m +b]
[m]] if log2l0//+ >
Pi
(9)
1 [m],
otherwise.
4.2 Quantization-awant training with FP8
To enable QAT using this quantizer, we need to make a few changes. First, allow gradients to flow through each step of the quantizer, we use the of non-diffcrentiable rounding operations. Second, we find that learning the maximum clipping value e instead of improves training stability. T can be found from as follows: b 2 - llgg
estimator (STE) for gradients
(2 - 2 2- - 1. Lastly, we treat logg|=;|++) as a constant that receives no gradient. This
prevents the (sometimes extremely large) gradients of this operation
I; propagate backwards.
The result of this is that æ receives the "straight-through" gradient for the full quantization procedure, i.. 0 (I;, m, c) 1, where F(.:))) denotes the FP8 quantizer.
8z.
4.3 Toy experiment: Learning minimal MSE on common distributions
To investigate whether our quantizer can indeed learn the maximum value and number of mantissa bits we run toy experiment. We sample 10 values from (O, 1), and initialize an FP8 quantizer with 3M4E and bias of 8, which corresponds
maximum value of 240. We then use SGD to learn the values of and m that minimize MSE the reconstruction loss: L(M, c)
E, (x; F(r,, m, 2 After 500 iterations, has converged to 4.35 and oscillates around 5.5. The oscillation behavior can be explained by the fact that MSE can be further minimized by increasing precision through higher m. however increasing to 6 yields uniform quantizer, which results in higher MSE. Similar behavior was observed in uniform quantization by [35]. A line search shows that indeed 5 and 4.37 minimize the MSE the target data, meaning our algorithm found the optimal values in this example. See Fig. for details.
Experiments
In this section, we aim to empirically validate the analytical findings from section 3 on full neural networks. We use our analytical results to investigate various FP8 formats for weight and activation tensors neural networks, and show that Finally, we perform quantization-awan training (QAT) on FP8 quantized models.

---

Sorpet cae Learaad cortig oeaan
Weatr um velies
30%
423
Figure 5: Left: Histograms of target data, and representable values of the initial format (3M4E) and the learned format (5M2E). Right: evolution of maximum value and number of mantissa bits. After roughly 350 iterations the number of mantissa bits starts oscillating around a value of 5.5.
5.1 Baselines and experimental setup
We run our experiments on variety of different tasks, datasets, and models: We experiment on
ResNet18 [19] MobileNerV2 (38] and ViT [14] for ImageNet classification [37]: BERT-base [[2) for
language understanding on the GLUE benchmark
[+ HRNet 39 for semantic segmentation on the
Cityscapes dataset [0] DeepLabV3 7 for semantic segmentation on the Pascal VOC dataset [[;; and SalsaNext [TI] for LIDAR point cloud segmentation the SemanticKITTI dataset [2]. For each
model, we report the 32-bit floating point (FP32) results on the same model instance (trained weights and code base) used in other experiments.
As baselines we compare against models quantized using INT8 quantization. Following we do not apply batch normalization folding, and re-estimate the batch normalization statistics (running mean and variance) before final validation, as this improved results for every model we considered. For each model, we consider several methods for range estimation separately for weights and activations, per-channel and per-tensor range estimation for weights. Results reported are those for the best combination of range estimation setting and per-channel or per-tensor range estimation. Our code written in Porrrc and all our experiments are performed using NVIDIA Tesla V100 and A100 GPUs.
35
5.2 Post-training quantization results
We compare post-training quantization using INT8 and FP8 quantization. To allow an apples-to- apples comparison, we do not any methods to improve post-training quantization that have been developed in the past years (e.g. those mentioned in 44! other than the methods described in Section 5
For our FP8 results, we report the best fully fixed format, i.c. the same combination of m, e and used for the full network, the best flexible bias format, i.e. m and fixed for the full network, but applied per channel tensor, whichever yields best results, and fully flexible format where and e are set for each tensor, and b set for each channel, to minimize MSE. The full procedure for this approach is detailed in Section B For example setting, see Figures I and 12 for example of FP8 formats that minimize MSB for each tensor ResNet18 and BERT, respectively.
Discussion PTQ results can be found in Table this table we can see that networks with large activation outliers (ViT, Bert, SalsaNext and HRNet) show better results for formats with more exponent bits, and thus a larger dynamic range. Furthermore, we see that, as predicted in Section A. formats with more mantissa bits are better able to represent weight and activation tensors in convolutional neural networks. However, since increasing the number of mantissa bits implies reducing dynamic range, finding the right value for the bias for each channel is important. The full set of results for our PTQ experiments detailed in Table Appendix Section !|
Lastly, we see that fully flexible formats only sometimes outperform fixed m/e formats, with slim improvements when they do. This is surprising, a network's 's tensors. We attribute this discrepancy to our local greedy method for assigning m, and b, which may not find settings that optimal globally.
more flexible format should be able better fit

---

Model
FP32
INT8
Best fixed
Best flex bias
Best flexible
Format
Score
Format
Score
Score
ResNet18
69.72
69.55
3M4E9B
68.52
5M2E
69.66
69.64
MobileNetV2
71.70
70.94
4M3E3B
68.49
5M2E
71.06
71.28
ViT
77.75
76.39
3M4E8B
77.37
4M3E
77.71
77.62
BERT-base
83.06
71.03
3M4E9B
83.08
3M4E
82.80
83.02
SalsaNext
55.80
54.22
4M3E3B
55.79
4M3E
55.80
55.02
DeepLabbV3
72.91
71.24
3M4E8B
9.42
5M2E
72.58
72.72
HRNet
81.05
80.93
3M4E8B
80.71
4M3E
81.04
81.06
Table 1: PTQ results with for fixed formats, flexible bias formats, and fully flexible formats, compared to original model (FP32) and INT8 results.
Model
FP32
INT8
Best fixed
Best flex bias
Best flexible
accuracy
accuracy
accuracy
ResNet18 PTQ
69.55
68.44
69.66
69.64
ResNet18 QAT
70.43
69.16
69.82
70.28
69.72
c learning
69.30
69.97
70.15
m learning
70.10
70.25
70.24
MobileNetV2 PTQ
70.94
68.49
71.06
71.28
MobileNetV2 QAT
71.82
71.03
71.54
71.60
71.70
learning
70.93
71.67
71.66
+ m learning
71.34
71.69
71.74
BERT PTQ
71.03
83.08
82.80
83.02
BERT QAT
83.26
83.56
83.70
83.17
83.06
c learning
83.89
83.86
83.72
+ m learning
83.85
83.84
83.70
Table 2: QAT results with for fixed formats, flexible bias formats, and fully flexible formats, compared to original model (FP32) and INT8 QAT results.
5.3 Quantization-aware training
We investigate whether quantization in mind (quantization-awe awwae training, QAT). We compare against INT8 QAT, which has previously been shown to result in INT8 models with accuracy near the original FP32 models.
improve the FP8 PTQ results by performing training with FP8
We consider three different initializations based on our PTQ results from Table D best fixed format, best flexible bias format and best fully flexible format. For each of these initializations we perform QAT with the format fixed at initialization, with learnable maximum value (cf. range learning in INT8 QAT), and with learning both and the number of mantissa bits m. We run experiments with various learning rates for model and quantization parameters, as well as per-tensor and per- channel quantization, and report results for the best learning setup. We train models for 20 epochs and use Adam for the model parameters and SGD for the quantization parameters. Results of these experiments can be found in Table 2 Full experimental details and results be found in Appendix
Discussion From these results, see that QAT can improve performance of FP8 quantized models. We also see that, for ResNet18 and MobileNetV2 learning both maximum value c and number of exponent bits improves results compared just learning weights, for fully fixed and flexible bias formats. See also Figure 6 for an example of how initialized e and m differ from learned and m after 20 epochs of training. However, this difference disappears for fully flexible PTQ initialization, and learning e and m slightly harms performance for BERT, although the difference is too small to be significant.
Generally, we see that QAT reduces the accuracy gap between different formats, as the weights the network can adapt to the distributions the quantizers represent.

---

Maxval before and QAT
Mantiissa QAT
4.5
Lesrred
09.5
52.4
Intialized maxval
Intialized mantissa bits
Figure 6: Values of maximum value e and number of mantissa bits m before and after training. In this experiment, maximum value was initialized to 60, and number of mantissa bits was initialized to m
6 Related work
Integer quantization Integer quantization, sometimes also called fixed-poing quantization, has been an active field of research since neural networks became more popular and the need for efficient inference became important. On a high level there are two lines of work, post- post-training quantization
(PTQ) 29 36
2X and quantization-awaan -wware training (QAT) [T8 23.
5 22
35
For more details we refer the interested read to the surveys of [27
34 17 Our approach
to learning the FPS configuration is inspired by [15] which assume the straight through estimator
(STE) A on the rounding operation in order to derive gradient for the scaling factor of the fixed point format. 42 extends this to jointly learn the bit width and scaling factor, which we explore for learning the trade-off between the number mantissa and exponent bits.
Floating point formats The accuracy and stability of floating point formats and algorithms are well studied in computer science and electrical engineering, refer the interested read (20) However, these studies usually focus on high bit with floating point formats (e.g. FP32 or FP64) and do not consider the impact on neural networks which significantly more robust noise.
Early work by 444 showed that convolutional networks can be trained using FP16 and FP8 using the 2M5E format. The work of 40 introduces hybrid FP8 format and first discovered that 3M4E has better inference performance than 2M5E, While they use a different FP8 format for the backwards path, they keep the bias and mantissa and exponent division fixed for all layers in the forward path. [22] introduces a fully flexible FP8 format similar to the one we consider. They perform a layer-wise optimization to find the best configuration (mantissa, exponent, sign bit, bias). Their work does not go in-depth in the analysis of why some formats are better, and it has a fairly limited results section with easy to quantize models like ResNet1 and VGG (27]. [4]] uses a custom FP4 format for the backwards path and INT4 for the forward path to enable full 4 bit training with a small drop in performance.
To the best of our knowledge we are the first work with an extensive study of the different FP8 formats based on both analytical insights and empirical results across several tasks and data modalities. As opposed to our new FP8 simulation method, other works rely on dedicated FP8 simulations that do not allow for efficient gradient based learning of the bias mantissa/exponent bit configuration.
7 Impact and Limitations
Impact FP8 is becoming widespread as an alternative to INT8, with multiple chips from vendors like Nvidia, Intel, Graphcore, AMD and IBM moving to support 3M4E and/or 2M5E FP8 formats, often with fixed bias values. In this paper we show that these formats are not optimal and that many networks benefit from FP8 formats with more mantissa bits and bias values that can be flexibly set per channel. We hope that the results in this paper will help guide FP8 hardware design decisions in the broader community.
Limitations In this paper, restrict ourselves to studying the impact of various FP8 formats model accuracy and ignore hardware implementation-speee sppccifcc impact on power consumption and latency. Assessing the difference in hardware impact between INT8 and FP8 for all networks is not trivial. From a data transfer bandwidth perspective, the two 8-bit formats incur similar overhead, while
http::/bit. hyttttt.yy3S/et

---

for compute limited models, the relative overheads depend on the exact implementation. Generally, FP8 units use more power for additions and multiplications [2T] however, surrounding logic (e.g. accumulator design, NaN/overflow checks etc) might amortize this difference, and in multi-purpose designs which support not only FP8, but e.g. also FP16/32, INT8/16 the difference in the overheads might disappear. Therefore we limited this work to comparing the different formats on a bit width level. Hardware design teams use our accuracy analysis to make trade-offs on the hardware side for their specific use-case and design.
8 Conclusion
In our analysis of the FP8 format and its benefits for quantized neural network inference we have touched on many points. We have shown that analytically the FP8 format can improve on the INT8 format for Gaussian distributions that are common in neural networks, and that higher exponent bits work well when outliers occur. This paper introduced a new way to simulate FP8 quantization in FP32 hardware that speeds up FP8 quantization simulation and makes it possible to learn the bias and mantissa-exponent bit-width trade-off. We validated the FP8 format for many networks in post-training quantization setting, showing that generally for neural networks the 5M2E and 4M3E FP8 format works the best, and that for networks with more outliers like transformers increasing the number exponent bits works best. We have also shown that when doing quantization-awe -aware training, many of these benefits of the format disappear, the network learns to perform well for the INT8 quantization grid as well.
Acknowledgments
We would like to thank Christos Louizos and Yin Huang for helpful discussions and valuable feedback.
References
[1] Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment. In Advances in Neural Information Processing Systems, 2019.
[2] J. Behley, M. Garbade, A. Milioto. J. Quenzel, S. Behnke, C. Stachniss, and J. Gall. Se- manticKITTI: Dataset for Semantic Scene Understanding of LiDAR Sequences. In Proc. of the IEEE/CVF International Conf. on Computer Vision (ICCV), 2019.
[3] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 1308:1300.3444 2013.
[4] Yoshua Bengio, Nicholas Léonard. and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 1rXivv1308..34. 3432, 2013.
[5] Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020.
[6] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. arXiv preprint arXiv:2001.0028/ 2020.
[7] Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation, 2017.
[8] Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. PACT: parameterized clipping activation for quantized neural networks. arXiv preprint arxiv:805.06s 06085, 2018
[9] Yoni Choukroun, Eli Kravchik, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. International Conference on Computer Vision (ICCV), 2019.
[10] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
10

---

[11]
Tiago Cortinhal, George Tzelepis, and Eren Erdal Aksoy. Salsanext: Fast, uncertainty- aware semantic segmentation of lidar point clouds for autonomous driving, 2020.
[12] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018.
[13] Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed mixed-precision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 293-302, 2019.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2020.
[15] Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S. Modha. Learned step size quantization. In International Conference on Learning Representations (ICLR), 2020.
[16] Mark Everingham. S.
M. Ali Eslami, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 11 111(1))98-136, 2015.
[17] Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer.
A survey of quantization methods for efficient neural network inference, 2021.
[18] Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Conference on Machine Learning. (ICML), 2015.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference Computer Vision and Pattern Recognition, 2016.
[20] Nicholas J Higham. Accuracy and stability of numerical algorithms. SIAM, 2002.
[21] M. Horowitz. 1.1 computing's s energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pages 10-14, 2014.
[22] Cheng-Wa Wein Huang, Tim-Wei Chen, and Juinn-Dar Huang. All-you-can- fit 8-bit flexible floating- point format for accurate and memory-efficient inference of deep neural networks, 2021.
[23] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient sinnger-arithmeticoonll inference. Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[24] Sambhav R. Jain, Albert Gural, Michael Wu, and Chris Dick. Trained uniform quantization for accurate and efficient neural network inference fixed-point hardware. arxiv preprint 1V03v:190.00000 2019.
[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXivv1412.6980, 2014.
[26] Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference:
A whitepaper. arXiv preprint arXiv: 1806::1806.8833 08342, 2018.
[27] Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference:
A whitepaper. arXiv preprint arXiv: 1806 08342, 2018.
[28] Yuhang Li, Ruihao Gong. Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations (ICLR), 2021.
[29] Darryl D. Lin, Sachin S. Talathi, and Sreekanth Annapureddy. Fixed point quantization of deep convolutional networks. In International Conference on Machine Learning, 2016.
[30] Stuart Lloyd. Least squares quantization in pcm. IEEE transactions on information theory, 2892):129-137. 1982.
[31] Christos Louizos, Mathias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling. Relaxed quantization for discretized neural networks. In International Conference on Learning Representations (ICLR), 2019.
11

---

[32] Joel Max. Quantizing for minimum distortion. IRE Transactions on Information Theory, 6(1):7-12, 1960.
[33] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? Adaptive rounding for post-training quantization. In International Conference on Machine Learning (ICML), 2020.
[34] Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yclysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. ArXiv, abs/2106.08295,
2021.
[35] Markus Nagel, Marios Fournarakis, Yelysei Bondarenko, and Tijmen Blankevoort. Overcoming oscillations in quantization-awar training. arXiv preprint arXiv:2203. 1086, 2022.
[36] Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In International Conference on Computer Vision (ICCV), 2019.
[37] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015.
[38] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chickl Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Conference on Computer Vision and Puttern Recognition (CVPR), 2018.
[39] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[40] Xiao Sun, Jungwook Choi, Chia-Yu Chen, Naigang Wang, Swagath Venkataramani, Vijay- alakshmi (Viji) Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. In H. Wallach, H. Larochelle. A. Beygelzimer, F. d' d'Alché-Buu E. Fox, and R. Gamett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[41] Xiao Sun, Naigang Wang. Chia- Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swa- gath Venkataramani, Kaoutar El Maghraoui, Vijayalakshmi (Viji) Srinivasan, and Kailash Gopalakrishnan. Ultra-low precision 4-bit training of deep neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1796-1807. Curran Associates, Inc., 2020.
[42] Stefan Uhlich, Lukas Mauch, Fabien Cardinaux, Kazuki Yoshiyama, Javier Alonso Garcia, Stephen Tiedemann, Thomas Kemp, and Akira Nakamura. Mixed precision duns: All you need is a good parametrization. In International Conference Learning Representations (ICLR), 2020.
[43] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics.
[44] Naigang Wang. Jungwook Choi, Daniel Brand, Chia- Yu Chen, and Kailash Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Besa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[45] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-
115. 2021.
[46] Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. Dorefa-net net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv: 1600.66666660. 2016.
12


Reviews:
[{"review_id": "nDSwYlXQ8yH", "reviewer": "Reviewer_tuZq", "summary": "This paper investigates how the choices that can be made to leverage the FP8 format (i.e. including the important choice of the number of bits for the mantissa and exponent), and these investigations deliver analytical results on which settings for better performance. Following their analytic contributions, this work also shows how these findings translate to real networks, provide an efficient implementation for FP8 simulation, and a new algorithm that enables the learning of both the scale parameters and number of exponent bits in the FP8 format. The main conclusion from this work is that FP8 format is more suitable than INT8 in terms of the accuracy, when doing post-training quantization for a wide range of networks, the FP8 format is better than INT8 in terms of accuracy. Also, this work delivers a recommendation that the choice of the number of exponent bits is driven by the severity of outliers in the network (under FP8 format). A sensitivity study on quantization-aware training is also carried out.", "questions": "- Can your method benefit the quantization-aware training in terms of the efficiency? If yes, how?\n- Can you comment on the potential impacts on the inference performance, given the solution and your recommendations (i.e. 5M2E & 4M3E)?", "limitations": "- The comparison is performed between FP8 and INT8 fake quantization, and I suggest the authors to make it clear upfront.\n\n- The accuracy is expected to be good by using FP8 format, since it can have a wider coverage than INT8. However, this can bring huge overheads in terms of the inference performance (see next), and it shall not be overlooked.\n\n- The inference performance is hard to benefit from the concluded 5M2E and 4M3E: Minifloats are commonly used in embedded devices, which need to be emulated in software. These overheads are extra burdens to the inference over INT8 fake quantization. \n", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- A sensible insight and the solution is convincing (for the accuracy)\n- A method to make simulated quantization fast\n- Evaluation results are sufficient to back up the current claim\n\nWeaknesses:\n- Limited insights (i.e., this work essentially describes a usage of minifloats for DNNs)\n- The proposed method is only tailored to an efficient quantization procedure\n- No discussions on the potential impacts on the inference performance\n- Low reproducibility (No source codes available)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GrfRXUN8wtz", "reviewer": "Reviewer_iumQ", "summary": "FP8 is discussed both in the context of quantization-aware training and in the context of post-training quantization.  A very nice analysis of the expected quantization error is provided by the authors when the input to the quantizer is normal/ uniform/student's distribution. One of the most important and striking conclusions is that FP8 is a better format for inference than INT8, even though the latter has been the standard format up to now. In my opinion, this is a critical conclusion since switching between floating-point formats (used for training) and integer formats (used for inference) is a pain. \n\n", "questions": "1. Bert-base opts for 3M4E in table 1. This may be due to heavy-tailed distributions of weights and activations. Could you please estimate the type of distribution Bert has in the FWD pass? Could a plot be included?\n2. What task does Bert base use here (word masking)? I'm not familiar with the baseline accuracy of  \"83.06\" in table 1 (baseline accuracy should be ~86). \n\nXXXXX\nThank you for your detailed response. Could you please share the code? \n\n\n ", "limitations": "The current work has no negative societal impact. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. The paper provides a rigourous approach to proving that FP8 outperforms INT8 in inference. 5M2E is found to minimize MSE of normal distributions, and thus is of particular interest. This is a well-written, novel paper with clear practical implications. \n2. The results of the analysis and simulation are in close agreement. \n\nWeakness: \n1. I do not find the FP simulator to be novel or interesting. \n2. It was a great work by the authors to provide MSE based on clipping value (set via exponent bias). Next, they should find the derivative, set it to zero, and find the roots or, if that is not possible, numerically solve it. Otherwise, analytical expressions have less practical value. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CQRxXEL9-T", "reviewer": "Reviewer_5G84", "summary": "This paper first analyzes the effect of the number of bits for the mantissa and exponent in FP8 data format, and then proposes a method to learn them with gradient descent. Empirical results on various tasks with various models show that the proposed FP8 quantization has a clear performance improvement for post-training quantization but performs similarly or even slightly worse than previously commonly used uniform INT8 quantization.", "questions": "Please address the questions above.", "limitations": "I did not find extra sections discussing about the limitations or broader impact, but some are talked about in the related work and conclusion. \nIt would be better if the authors can discussion about them in more detail.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is overall well written and easy to follow. The discussions on the effect of using different #bits for the mantissa and exponent to the quantization error are also intuitive. Extensive experiments on various tasks and various domains are performed to validate the efficacy \nof the proposed method.\n\nStill some parts of the method and experiment sections need some more clarifications.\n- Lines 119-120: Why the MSE can be approximated by (7). Are there any intuitions, explanations, or references?\n- Table1: most of the time, \"Fixed\" performs similarly to \"flex bias\" and \"flexible\" on all models except DeepLabV3. Are there any explanations for this observation?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "60VDeKEkrLF", "reviewer": "Reviewer_5HMP", "summary": "In this paper, the FP8 format quantization algorithm is realized based on a common FP32 deep learning framework for post-training quantization and quantization-aware training. The effects of choosing different bits for the mantissa and exponent on the quantization error and the final accuracy of the model are studied. Experiment results show that FP8 format is better than INT8 format.", "questions": "1. What is the major contribution and novelty of this paper? I think the authors should make it more clear. Is it the \"chief conclusion\" saying that FP8 format has a larger representation range and so has a smaller quantization error on the tensor with outlier? This is a somewhat trivial knowledge. Is it the innovation of this paper to conduct “fast” FP8 quantization simulation in the FP32 framework? If so, the implementation of this paper should be compared with other quantitative simulation frameworks.\n\n2. Line 93, \"it has been shown to be indicative of the final loss in a neural network\". Can it be confirmed by experiments in this paper?\n\n3. This paper proposes to select the appropriate bits for mantissa and exponent for each layer (best flexible in the experiment? ) Is this off-line selection obtained by traversing the selection of mantissa and exponent bits for the weight and activation of each layer respectively? When selecting the quantization format for each layer, do you aim to minimize the MSE of the tensor, or minimize the scalar product quantization error, or maximize the final model score? If MSE is minimized, will the result with the largest model score be obtained?\n\n4. \"best flexible\" does not seem to be significantly better than “best flex bias”. The tensor is divided into blocks, and each block adopts different exponent bias, which is equivalent to the integer format of block quantization, also called share exponent. This idea has been put forward in Flexpoint and other work [1][2][3]. Is the block +FP8 scheme used in this paper better than block + INT8? Further consider that the actual deployment requires hardware design. The use of best flex bias or best flexible requires support for various m/e combinations on hardware, while block + INT8 only needs to support one.\n\n5. There should not be a \"(PTQ)\" under the \"INT8\" in Table 2.\n\n6. From the incomplete results in Table 2, QAT is not necessary for best flexible? In other words, applying QAT to the best fixed scheme or even INT8 scheme is enough to achieve the optimal accuracy. Is the complex FP8 format such as best flexible necessary?\n\n7. Because the colors in Figure 5 overlap, new colors are generated, which is difficult to distinguish. Maybe change to a line chart?\n\n8. Line 146, \"we fit Gaussian distributions in the weights and activations sample, and compute the expected MSE analytically in Fig. 3 (right)\" If we are experimenting with the data in the actual model, why should we fit the Gaussian distribution? We should directly use the original data to calculate the quantization error.\n\n9. The MAC results of FP8 tensors are in high-precision (such as FP32). The result after an operator needs to be intercepted as FP8, so, how are the operators divided in various network models? For example, before adding the shortcut path and the main path in ResNet, will they be both quantized as FP8?\n", "limitations": "The scheme in this paper might make the hardware design more complex, because it needs to support different m/e/b, which is mentioned in [3] (see the ref section in \"Strengths and Weakness\"). I recommend the authors to discuss on the implication for actual hardware implementation.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\n1. This paper did a good work on clearly describing the floating-point format and quantization process.\n2. This paper realized the quantization simulation based on the FP32 framework.\n3. Experimented with models on various tasks.\n\n## Weakness\n\nI think the major weakness of this work is the lack of novelty and insufficient comparison with baselines. Please see the question section for more detailed discussion.\n\n### Novelty\n1. Using floating-point low-bit format in neural network quantization is very common. For example, group-wise quantization [1][2][3], different quantization parameters for different layers [4][5], special schemes for tensor with outliers [6], and QAT scheme for reducing quantization error [7][8] are well discussed in the literature.\n2. It seems the experiments do not bring new findings or knowledge beyond existing studies. \n\n### Insufficient comparison\n3. There are many other low-bit quantization schemes in the literature (as shown above), this paper should compare with more baselines other than the classical INT8 quantization scheme.\n\n## References\n[1] Köster, Urs et al. \"Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks.\" NIPS (2017).\n\n[2] Rouhani, Bita Darvish et al. \"Pushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point.\" NeurIPS (2020).\n\n[3] Zhong, Kai et al. \"Exploring the Potential of Low-bit Training of Convolutional Neural Networks.\" IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (2022).\n\n[4] Banner, Ron et al. \"Post training 4-bit quantization of convolutional networks for rapid-deployment.\" NeurIPS (2019).\n\n[5] Dong, Zhen et al. \"HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision.\" 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (2019): 293-302.\n\n[6] Zhao, Ritchie et al. \"Overwrite Quantization: Opportunistic Outlier Handling for Neural Network Accelerators.\" ArXiv abs/1910.06909 (2019).\n\n[7] Choi, Jungwook et al. \"PACT: Parameterized Clipping Activation for Quantized Neural Networks.\" ArXiv abs/1805.06085 (2018).\n\n[8] Yamamoto, Kohei. \"Learnable Companding Quantization for Accurate Low-bit Neural Networks.\" 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021): 5027-5036.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept