All-You-Can-Fit 8-Bit Flexible Floating-Point Format for Accurate and Memory-Efficient Inference of Deep Neural Networks
Authors: Juinn-Dar Huang
Cheng-Wei Huang
Tim-Wei Chen
Venue:  (2021)
Keywords: 8-bit floating-point format
accuracy loss minimization
numerics
memory-efficient inference
deep learning

Abstract: Modern deep neural network (DNN) models generally require a huge amount of weight and activation values to achieve good inference outcomes. Those data inevitably demand a massive off-chip memory capacity/bandwidth, and the situation gets even worse if they are represented in high-precision floating-point formats. Effort has been made for representing those data in different 8-bit floating-point formats, nevertheless, a notable accuracy loss is still unavoidable. In this paper we introduce an extremely flexible 8-bit floating-point (FFP8) format whose defining factors – the bit width of exponent/fraction field, the exponent bias, and even the presence of the sign bit – are all configurable. We also present a methodology to properly determine those factors so that the accuracy of model inference can be maximized. The foundation of this methodology is based on a key observation – both the maximum magnitude and the value distribution are quite dissimilar between weights and activations in most DNN models. Experimental results demonstrate that the proposed FFP8 format achieves an extremely low accuracy loss of $0.1\%\sim 0.3\%$ for several representative image classification models even without the need of model retraining. Besides, it is easy to turn a classical floating-point processing unit into an FFP8-compliant one, and the extra hardware cost is minor.

Under review as a conference paper at ICLR 2021
ALL-YOU-CAN-FIT 8-BIT FLEXIBLE FLOATING- POINT FORMAT FOR ACCURATE AND MEMORY- EFFICIENT INFERENCE OF DEEP NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Modern deep neural network (DNN) models generally require huge amount weight and activation values to achieve good inference outcomes. Those data in- evitably demand massive off-chip memory capacity/bnddwidth, and the situation gets even worse if they represented in high-precision floating-pon -ointt formats. Effort has been made for representing those data in different 8-bit floating- point formats, nevertheless, notable accuracy loss is still unavoidable. In this paper we introduce an extremely flexible 8-bit floating-pont ppont (FFP8) format whose defining factors the bit width of exponent/fraction field, the exponent bias, and even the presence of the sign bit - are all configurable. We also present a methodology to properly determine those factors so that the accuracy of model inference be maximized. The foundation of this methodology is based on key observation both the maximum magnitude and the value distribution are quite dissimilar be- tween weights and activations in most DNN models. Experimental results demon- strate that the proposed FFP8 format achieves an extremely low accuracy loss of 0.1% ~ 0.3% for several representative image classification models even without the need of model retraining. Besides, it is easy to turn a classical floating-por point processing unit into FFP8-compliant one, and the extra hardware cost is minor.
INTRODUCTION
With the rapid progress of deep neural network (DNN) techniques, innovative applications of deep learning in various domains, such as computer vision and natural language processing (NLP), are getting more mature and powerful (Huang et al. 2017 Vaswani et al. 2017 Szegedy et al. 2015 2017: Krizhevsky et al. 2012).
Howard et al. 2017: He et al.
To improve the model accuracy, one of the most commonly used strategies to add more layers into network, which inevitably increases the number of weight parameters and activation values of a model. Today, it is typical to store weights and activations in the 32-bit IEEE single-precision floating-point format (FP32). Those 32-bit data accesses thus become extremely heavy burden on the memory subsystem in a typical edge or AloT device, which often has very limited memory capacity and bandwidth. Even for high-end GPU or dedicated network processing unit (NPU) based computing platforms, off-chip DRAM bandwidth is still a major performance bottleneck.
To relieve the issue of memory bandwidth bottleneck, several attempts of various aspects have been made including (but not limited to) weight pruning (Li et al.] 2017] Han et al.) 2016). weight/activation quantization (Courbariaux most straightforward way: storing weights and activations in shorter format (Köster et al. 2017)
al. 2015 Hubara al. 20117 and probably the
One trivial way to do is to adopt the 16-bit IEEE half- An FP16 number consists of sign bit, 5 exponent bits, and 10 fraction bits. In addition, Google proposed another 16-bit format, named Brain Floating Pooting-PPint Format (BFP16), simply by truncating the lower half of the FP32 format (Kalamkar et al.) 2019). Compared with FP16, BFP16 allows significantly wider dynamic value range at the cost of 3-bit precision loss. Note that the exponent bias in all of the above formats is not free design parameter. Conventionally, the value is solely determined by the exponent size. For example, for FP16 with 5-bit exponent, the exponent bias is automatically fixed to 15 (25-1_. 1).
flotting-poin format (FP16).

---

Under review as a conference paper at ICLR 2021
To make the data even shorter, 8-bit fixed-point signed/unsigned integer formats (INT8 and UINT8) are also broadly adopted. However, the 8-bit fixed-pont point format inherently has a narrower dynamic value range that the model accuracy loss is usually not negligible even after extra symmetric or asymmetric quantization. As a consequence, there are a number of attempts concentrating on utilizing mixed-precision or pure 8-bit floating- point numbers in deep learning applications.
Various techniques have been developed for mixed-precisiin training (Banner et al. 2018 Micike- Vicius et al. 2018 Das et al., 2018 Zhou et al. 2016). Moreover, recent studies proposed several training frameworks that produces weights only in 8-bit floating-poin formats (Wang & Choi 2018) Cambier al., 2020 Sun et al., 2019). In these studies, the underlying 8-bit floating-poin -oont numbers in training and inference are represented in the format of FP8(1, 5, 2) or FP8(1, 4. 3), where the enclosed three parameters indicate the bit length of sign, exponent, and fraction, respectively. Note that 4 or 5 bits are essential for the exponent in their frameworks, or the corresponding dynamic range may not cover both weight and activation values well. Consequently, only 2 or bits are available for fraction, which inevitably leads to lower accuracy.
In this paper, we present an extremely flexible 8-bit floating- point (FFP8) number format. In FFP8, all -arameters-ther bit width of exponent/fraction the exponent bias, and the presence of the sign bit are configurable. Three major features of our inference methodology associated with the proposed FFP8 format are listed as follows. First, it is observed that both the maximum magnitude and the value distribution are quite dissimilar between weights and activations in most DNNs. It suggests the best exponent size and exponent bias for weights should be different from those for activations to achieve higher accuracy. Second, a large set of commonly-used activation functions always produce nonnegative outputs (e.g., ReLU). It implies that activations are actually unsigned if one of those activation functions is in use. Hence, it implies the sign bit is not required for those activations, which makes either exponent or fraction 1-bit longer. Note that even one bit can make a big impact since only 8 bits are available. Third, all aforementioned studies require their own sophisticated training frameworks to produce 8-bit floating-poin models. Our flow does not. Our flow simply takes model generated by any conventional FP32 training framework as the input. Then, it simply converts the given pre-trained FP32 model into an FFP8 model.
The rest of this paper is organized as follows. Section 2 briefly introduces related work. In Section 3, we elaborate more on the proposed FFP8 format and how to properly convert a pre-trained FP32 model into an FFP8 one. Section 4 demonstrates the experimental results on various DNN models. The system and hardware design issues for the support of FFP8 numbers are discussed in Section 5. Finally, the concluding remarks are given in Section 6.
2 RELATED WORK
DNNs are becoming larger and more complicated, which means they require bigger memory space and consume more energy during inference. As a result, is getting harder to deploy them on systems with limited memory capacity and power budget (c.g.. edge devices). (Han et al. 2016 Zhao et al. 2020 Horowitz 2014) also demonstrated that off-chip DRAM access is responsible for significantly big share of system power consumption. Hence, it remains an active research topic about how to reduce the memory usage for weights and activations. As mentioned in the previous section, one way to do is to use short 8-bit floating- point number formats.
Wang & Choi (2018) introduced a DNN training methodology using 8-bit floating-poit numbers in FPS(1, 5, 2) format. The methodology features chunk-based accumulation and stochastic rounding methods for accuracy loss minimization. Besides, to achieve a better trade-off between precision and dynamic range during model training, (Sun et al. (2019) proposed an improved methodology that utilizes two different 8-bit floating-point formats- - FPS(L, 4. 3) for forward propagation and FPS(1, 5, 2) for backward propagation. Nevertheless, both methodologies fail to make DNN model entirely in 8-bit numbers: the first and the last layers of the given model are still in 16-bit floating-point numbers; otherwise, the model suffers about 2% accuracy degradation. Cambier et al. (2000) then proposed the S2FP8 format, which allows DNN model represented in 8-bit floating point numbers completely. By adding a scaling factor and a shifting factor, data can thus be well represented in FP8(1, 5, 2) after proper shifting and squeezing operations, which eliminates the need of 16-bit floating point numbers. However, the S2FP8 format still results in about 1 1% accuracy drop in ResNet-50 (He et al. 2018).
2

---

Under review as a conference paper at ICLR 2021
FP16
10
FFP8
X
(1,5,10)
(1.y.z.b)
5
(1.5.2)
FP8
e
5
number (-1)xx1.fx2
number = (-1) 1.f
b= -= -1
FFP8 (0,y.z.b)
FP8
(1.4.3)
number = 1.fx22=b
(a)
(b)
Figure 1: Various floating-pots formats: (a) conventional ones, and (b) the proposed FFP8 format
3 FLEXIBLE 8-BIT FLOATING-POINT (FFP8) FORMAT
3.1 DEFINITION OF THE FFP8 FORMAT
In this subsection, a flexible 8-bit floating-point (FFP8) format, which leads to more accurate in- ference outcomes of deep neural networks, is presented. A typical floating-point format consists of sign (s), exponent (e), and fraction (f) fields, and the bit length of cach field is specified æ, y, 2, respectively. Besides, one more parameter, exponent bias (b), is required to completely specify a floating-poin point number. Conventionally, b is always implicitly set to 2
1. In this paper, n-bit floating-poin format is denoted as (x. y, 2, b) or (I, y, 2), where n = x + y + 2. If is missing, the default value is implicitly used.
Figure ((a) illustrates one 16-bit FP16 format and two commonly used 8-bit formats: (1, 5, 2, 15) and (1, 4, 3, 7). Note that there is actually only one parameter, the bit width of the exponent (y), that can be freely chosen when defining a new n-bit conventional floating-point format since x is always
always n - y - and 6 is always 2 1. The above fact motivated us think out-of-the-box and thus develop a more flexible format, named flexible 8-bit floating-pon -oinn (FFP8) format and shown in Figure I(b). addition to the size of exponent (y)) the FFP8 format offers two more parameters. First, the exponent bias (b) is not necessarily equal to 2 - and can be set to any integer, which helps cover the value distributions shorter exponent size (v). Second, the sign bit is present or not (i.e., I can be 0 or 1). Without the sign bit (r = 0), unsigned activations can be better represented in higher precision. To be more precise, there only two restrictions or 1. Next, we are about to show how the proposed FFP8 format improves the inference accuracy.
both weights and activations well in
n-bit FFP8 format: 1) x + y + n: 2) x must be 0
3.2 WEIGHT DISTRIBUTION AND THE WAYS VARIOUS 8-BIT FORMATS COPE WITH IT
Overall speaking. the magnitudes of weights in most DNN models are usually small. Take a popular image classification model V6G-16 (Simonyan & Zisserman 2015) magnitude of weights in the whole model is less than 2. Figure gives the overall weight distribution (in log scale) of VGG-16 trained via conventional FP32 framework. Figure 2(a) illustrates how the conventional FP8 4, 3) copes with those weights. The rectangle in red is called the range window, which specifies the value range that FP8(1, 4, 3) can represent. The purple vertical dash line further partitions the window into the norm region (right side) and the denorm region (left side). The star marks the position where the weight of the maximum magnitude locates. Note that virtually the entire right half of the range window in Figure 2(a) covers no weights, while 9.6% of leftmost weights cannot be included in the range window. In order contain almost every weight in a range window, FP8(1, 5, 2) can be selected alternatively, as shown in Figure 2(b). A bigger exponent size (4 to 5) results in a larger range window. However, there are only 4 (22) instead of 8 (23) representative values available for each æ in the norm region since the fraction size decreases from 3 to 2, which potentially results in a lower accuracy. Note that FP8 4, 3) and FP8(1, 5, 2) are two most commonly used 8-bit floating point formats in the previous studies.
example, the maximum
With the help of the proposed FFP8 format, things can change a lot. Figure 2(c) shows what happens if FFPS(1, 4, 3, 15) is in use. The range window is of the same size as that in Figure 2(a) but is left- shifted by 8 positions due to the exponent bias is set to 15 instead of the default value 7. It is obvious that FFP8(I, 4, 3, 15) can better cope with weights in VGG-16 than FP8(1, 4. 3) and FP(1, 5, 2). Furthermore, Figure 2(d) shows what if FFP8(1, 3, 4, 7) is in use. Comparing FFP8(1, 3,
4, 7) against FFP8(1, 4, 3, 15), weights in the norm region (over 35%) represented in 1-bit
3

---

Under review as a conference paper at ICLR 2021
Modmum
F,8(1,4,3)
F,8(1,5,2) values for each x norm
12
-0
8 values for each @ norm
Clennty (Na
02
30
122
(21) [a)
32-33
Magrinde 121 (b)
Madiumm
FFP8(1,43,15)
FFP811,3.4.7)
values
Demily (h)
16 values for each x @ norm
each @ norm
Maceitud
(c)
Magrinuee
(c)
Figure 2: The range windows of various 8-bit floating-point formats versus the weight distribution of VGG-16; (a) in FP8(1, 4, 3), (b) in FPS(1, 2), (c) in FFP8(I, 3, 15), (d) in FFP8(1, 7)
higher precision whereas 4.6% of leftmost weights those out-of-the-window weights can be regarded as some sort of pruned weights. That is, more investigation should be further conducted determine whether FFPS(I, 4, 7) FFPS(11 4, 3, 15) is better for the weights in VGG-16. One way or another, it is clear that the proposed FFP8 formats can achieve certain improvement that the conventional 8-bit formats cannot.
included the range window. However,
3.3 FFP8-BASE BASED INFERENCE FRAMEWORK AND BEST-FIT FORMAT SELECTION
In the proposed memory-efficient inference framework, weights and activations stored in off-chip memory are in FFP8 formats while operations inside the computing engine are still in FP32, which is common strategy widely adopted in current state-of-the -fte-of-thheart art computing platforms (Intel 2018 Nvidia 2017). Hence, a systematic process, shown Figure 3, is required to transform FP32 numbers into FFP8 ones in the following two scenarios: 1) quantizing pre-trained FP32 weights to FFP8 ones in advance, and 2) converting FP32 output activations into FFP8 ones before writing them back to off-chip memory. The process first prepares list of 256 values that can be precisely represented by the specified FFP8 format. Then, the given FP32 inputs (weights or activations) are converted into FFP8 outputs using the Dound-to-nearest-evar method.
In order to explore best-fit 8-bit formats for weights and activations in VGG-16, we have made a of attempts, and the results are summarized in Table 1. First, it is unwise represent weights in FP(1, 5, 2). Though Figure 2(b) shows FP(1, 5, 2) can cover virtually all weights, the accuracy loss is serious due to 2-bit extremely low precision. Next, FP(I, 3) performs much better than FP(1, 5, 2), which suggests one extra precision bit does make notable improvement here though 9.6% smallest weights are clear the proposed FFP8 format can surely do better. FFPS(1, 4, 3, 15) further outperforms FP(1, 4. 3) because weights are better covered by left-shifted range window, as shown Figure 2(c).
(i.e.. pruned away), as depicted in Figure 2(a). Moreover,
Inspired by the fact that FP(1, 4, 3) outperforms FP(1, 5, 2), we further examine whether an even smaller exponent size can help or not. First, it is unwise to represent weights FPS(1, 3, 4) without modifying the default exponent bias (=3). Though the fraction size increases from 3 to 4, 64.1% lefumost weights are out of the range window, which is way too much. However, by setting the exponent bias to 7, which is equivalent to sliding the range window to the left by 4, the resultant FFP8(1, 3, 4, 7) successfully achieves a higher accuracy since it has bigger 4-bit fraction size and only prunes away 4.6% smallest weights, illustrated in Figure 2(d).
4

---

Under review as a conference paper at ICLR 2021
Weights Activations in FP32
Conversion: Round to Nearest Even
Weights / vations In FFP8
FP32
FFP8 Parameters (x, b)
256 Values
Generation
Figure 3: The process converting FP32 inputs into FFP8 outputs with the given format parameters
Table I:
Top-1 and Top-5 accuracy of VGG-16 on ImageNet dataset (Deng al. 2009 in various number formats; delta (A) indicates the accuracy drop compared to FP32
Weight
Activation
Top-1 1 (A)
Top-5 (A)
FP32
FP32
71.59%
90.38%
(1,5,2,15)
(1,4,3,7)
69.89% (-1.70%)
89.29% (-1.09%)
(1,4,3,7)
(1,4,3,7)
70.86% (-0.73%)
90.02% (-0.36%)
(1,4,3,15)
(1,4,3,7)
70.96% (-0.63%)
90.10% (-0.28%)
(1,3,4.3)
(1,4.3.7)
70.18% (-1.41%)
89.56% (-0.82%)
(1,3,4.7)
(1,4,3.7)
71.19%
(-0.40%)
90.12% (-0.26%)
(1,3,477)
(1.43.7))((4444))
71.19% (-0.40%)
90.14% (-0.24%)
In addition to weights, it is certainly worth finding out best-fit formats for activations as well. For those attempts made above, activations are always FFPS(1, 4. 3, 7) for two reasons: 1) the overall distribution of activations is wider than that of weights, and 2) the maximum magnitude of activa- tions is much bigger than that of weights. The detailed distribution of activations will be given in Section 3.4 later. Meanwhile, it is also worth noting that large set of commonly used activation functions always produce nonnegative outputs, e.g., ReLU, ReLU6, and sigmoid. That is, if those outputs are represented in any signed format, a half of the code space is actually wasted. It may not be a problem for 32-bit and 16-bit formats with long enough fraction bits; however, is indeed a serious issue for any 8-bit format, which merely has 256 available codes in total. Since VGG-16 utilizes ReLU as its activation function, it is feasible to select signed FFP8(1, 4, 3, 7) for the first layer and unsigned FFPS(O, 4, 4, 7) for all succeeding layers. It implies that 256 instead of 128 codes are available to represent those unsigned activations in all layers except for the first one. With no surprise, the accuracy further improved since the range window remains untouched while the fraction size gains one extra bit. In the last configuration, the Top-1 accuracy loss is only 0.4% when compared against the FP32 baseline, as indicated in Table 1.
3.4 LAYER-WISS WISE OPTIMIZATION
Figure and Figure illustrate several distributions of weights and activations in VGG- respec- tively. Each figure includes the distributions of one whole model and three selected individual layers. In Section 3.3, the best-fit FFP8 format is determined by the overall distribution of the whole model. Here we have three key observations from those distributions: 1) the distributions of weights quite dissimilar to those of activations, 2) even the distributions across different layers are dissimilar for both weights and activations, and 3) the distribution of an individual layer is narrower than that of the whole model. The above observations clearly suggest that applying layer-wise optimization (LWO) properly on number format selection very likely improve the accuracy further.
For instance, after comparing Figure (b)), it is found that the distribution of the whole model is wider than that of the first layer, and the maximum log magnitudes of the whole model and the first layer are 8 and 1, respectively. As consequence, selecting FFP8(1. 3, 4, 6) instead of FFP8(1, 4, 3, 7) for activations the first layer can further increase the Top-1 accuracy by 0.1%% (from 71.19% to 71.38%), as indicated in Table 2. Next, we examine new configuration that makes weights of all layers are in FIP8 2. 5, 3). is obvious an unwise attempt since 37.6% leftmost weights are out of the range window according to Figure 4(a). However, it may not be bad idea to use FFPS(1, 2, 5, 5) in Layer 6 and FFP8(L, (1, 2, 5, 6) in the last layer because only 6.9% and 5.3% smallest weights out of the range window respectively according Figure (cd)) Therefore,
examine another new configuration that makes weights of all layers in FFPS(1, 2, 5, *). Here the asterisk (*) represents the largest possible exponent bias, which ensures that the maximum weight is still inside
5

---

Under review as a conference paper at ICLR 2021
the range window. This LWO weights successfully increases the Top-1 accuracy by 0.1% (from 71.38% to 7..48%). Similarly, we can apply the LWO on activations, which again raises the Top-5 accuracy by 0.05%. The selected FFP8 format of each layer after LWO is reported in Table 3.
Previous studies usually choose FP8(1, 4, 3) or FP8(1, 5, 2) formats because the exponent size has to be large enough to cover most weights and activations at the cost of even smaller fraction size. However, the exponent size for weights can be as small as after LWO in our flow. Consequently, the larger 5-bit fraction size does help in accuracy improvement, indicated Table 2.
Note that the Top-1 accuracy achieved by the final configuration, which applies layer-wise optimiza- tion both weights and activations, is merely 0. % lower compared to that of the FP32 baseline (71.4%% vs. 71.59%). More importantl, model retraining is not applied yet. In other words, all the accuracy improvements made so far are simply from properly re-expressing weights and activations of pre-trained FP32 model in their best- fit FFP8 formats.
15
(7)
Naggenue
(8)
(b)
30
<-35 -15 -80 -12 (12-11
46
-35
+10
Magritue
Magnlade
(6)
Figure 4: Weight distributions of (a) whole model, (b) first layer, (c) Layer 6, (d) last layer
Dergy 66
Doodh
Macsttce (8)
Magntudk (21)
(b)
70
60
07
50
Magiiaale
Maga made
[c)
(d)
Figure 5: Activation distributions of (a) whole model, (b) first layer, (c) Layer 6, (d) last layer

---

Under review as a conference paper at ICLR 2021
Table 2: Top-1 and Top-5 accuracy of VGG-16 after layer-wise optimization
Weight
Activation
Top-1 (A)
Top-5 (A)
FP32
FP32
71.59%
90.38%
(1,3,4,7)
(1.43..7)((044..)
71.19% (-0.40%)
90.14% (-0.24%)
(1,3447)
(1.3,46))+(04444))))
71.38% (-0.21%)
90.33% (-0.05%)
(1,2,53)
(1.3.46))((0444.))))
71.24% (-0.35%)
90.22% (-0.16%)
(1,2,5,)
(13344))))(04447)))
71.48% (-0.11%)
90.27% (-0.11%)
(1,2,5,*)
(1344.)))44444)))
71.8%% (-0.11%)
90.32% (-0.06%)
Table 3: The FFP8 format of each layer in VGG-16 16 after layer-wise optimization
Layer
Weight
Activation
Layer
Weight
Activation
(1,2,5.3)
(1,3,4,6)
(1.2,5,5)
(0,4,4,8)
2
(1,2.5,4)
(0,4,4,11)
9
(1,2,5,5)
(0,4,4,8)
(1,2,5.4)
(0,4,4,10)
10
(1.2,5,6)
(0,4,4,8)
4
(1,2,5.5)
(0,4,4,10)
11
(1,2,5.5)
(0,4,4.7)
(1,2,5,4)
(0,4,4.9)
12
(1,2,5,5)
(0,4,4,7)
6
(1,2,5,5)
(0,4,4.9)
13
(1,2,5,6)
(0,4,4,8)
(1,2,5,4)
(0,4,4.9)
3.5 ACCURACY IMPROVEMENT VIA MODEL RETRAINING
All FFP8 weights in our previous experiments simply converted from pre-trained weights gen- erated from a typical FP32 training framework. However, reported that quantization-avee model retraining whether can successfully apply to our work. We first train the ResNet- model an FP32 frame- work and the corresponding Top-1 accuracy is 69.76%. Next, an FFP8-basee bared inference is performed under the following settings: 1) all weights are in FFPS(1, 3, 4, 7), 2) activations of the first layer are in FFPS(1, 3, 4, 6), and 3) activations of the other layers in FFP8(0, 4, 4, 7). The Top-1 accuracy for the above configuration is down to 69.44%. Then, a quantization-awwe aware retraining process of 15 epochs is applied and the resultant Top-1 accuracy goes back 69.76%. Note that the retraining is done simply in a typical FP32 framework without the need of special training skills.
usually improve the accuracy (Jacob al. 2017), which motivates to check
4 EXPERIMENTAL RESULTS ON VARIOUS DNN MODELS
In this section, we intend to demonstrate that the proposed FFP8 format constantly performs well in various DNN models in addition VGG-16. Table reports the accuracy results of various mod- els under different format configurations. Configuration A gives the results of the FP32 baseline. Configuration B utilizes conventional FP8(1, 4, 3) format with default exponent bias (7), which incurs a penalty of roughly 1% drop on Top-1 accuracy. With the help of the FFP8 format, Configu- ration C adopts the best-fit format for weights, which effectively reduces the Top-1 accuracy loss to 0.4-0.75%. Configuration D further selects two best-fit formats for activations (the signed one for the first layer and the unsigned one for the rest), which minimizes the Top- accuracy loss to 0.3%. Note that neither layer-wise optimization nor model retraining is even applied for this achievement.
Table 4: Accuracy results of various models under different format configurations
VGG-16
ResNet-50
ResNet-34
ResNet-18
Cfg.
Weight
Activation
Top-1 / Top-5
Top-1 Top-5
Top-1 Top-5
Top-1 Top-5
FP32
FP32
71.59 / 90.38
76.13792.85
73.31791.42
69.76/ / 89,08
(1,4,3,7)
(1,4,377)
70.86 / 90.02
75.24 92.52
72.39 90.97
68.70/ 88.50
(1,3,4.7)
(1,4.3.7)
71.19 90.12
75.38 12.668
72.81 91.16
69.25 88.80
D
(1,3,477)
(1.3.4.))((.444.7)))
71.38 / 90.33
75.85 92.81
73.12 / 91.33
69.44 88.93

---

Under review as a conference paper at ICLR 2021
Wgt. /Act.
BFP16 to FP32
FP32 FMA
Wgt. / Act.
FFP8 to
FP32 FMA
BFP16
FFP8
FP32
off-chip
on-chip
off-chip
on-chip
Act.
Quantization
Act.
Quantization
in BFP16
FFP8
(a)
(b)
00000 001 LOUTITIO
off-chip
on-chip
X
y
format
127
8-bit exponent
1-bit
8-bit
register
127 b
32
0000
To FP32 FMA
0101
8
select ctrl
8: bit
mux-based data-bit
00 1100
001
selector
15-bit
padding O's
(C)
Figure 6: Memory-efficiene system architectures: off-chip external data (a) in BFP16, (b) in FFP8; and (c) an FFP8 to FP32 hardware converter
In addition to image classification, we also want to know whether the proposed FFP8 format per- forms equally well in other application domains. Here, examine two more applications: semantic segmentation and ECG check. First, FCN32s is a popular CNN model for semantic segmentation (Long al. 2014). If FCN32s is in FP32, the mIOU is 63.63% using the VOC2011 dataset (Ever- ingham et al.) Afternatively, all the weights are in FFPS(1. (1, 4, 3, 14) and all the activations are in FFPS(I, 4, 2), the resultant mIOU would be 63. 63.45% 9,, slight drop of 0.18%. Second, LSTM model for ECG check (Physionet 2017). has also been tested. If the LSTM model is in FP32, the check accuracy is 81.12% If all the weights are in FFP8(1, 3, 4, 6), the first-layer activations are in FFP8(1, 3, 4. 5), and activations in the other layer are in FFP8(1, 4, 3, 16), the resultant accuracy would be 81.53%, an accuracy gain of 0.41%. The experimental results once again demonstrate that FFP8 performs very well in these two categories as well.
ASPECTS OF SYSTEM AND HARDWARE
The target of this work is to develop a memory-efficiem inference system, especially for those with limited memory capacity and bandwidth. A current state-of-the-an of-the-art system, proposed by Nvidia and Intel, has successfully cut the required memory size and traffic through representing external data (weights activations) in BFP16/INTS instead of FP32, illustrated in Figure 6(a) (Intel 2018 2019 Nvidia 2017 2020). It not only reduces the memory size, alleviates the performance bottleneck due memory bandwidth limitation but also saves significant amount of energy due to fewer power-consuming external memory access operations. To preserve the computation accuracy the same time, external BFP16 data are converted to FP32 data right before entering the FP32 mused-multiply-ade (FMA) unit. That is, internal computations can all be in FP32. Those FP32 data are converted back to BFP16 only if they are about to written back to external memory.
In this work, we propose system architecture that is very similar to the previous Figure 6(b). The key difference that external data are FFP8 instead BFP16, which implies the proposed system merely demands quarter of the memory size and throughput required by today's 's FP32-based counterparts.
shown in
It is easy convert BFP16 number into its FP32 equivalent by concatenating 16-bit pattern of all Os at its least significant end. In fact, it is also easy to transform an FFP8 number to its FP32 equivalent via converter, depicted in Figure 6(c). Few register bits allocated to store the current format settings of x, y. and b within the converter. Then, x is used to recover the sign bit; the biased exponent and fraction can be extracted via & and y; finally the exponent be further
8

---

Under review as a conference paper at ICLR 2021
corrected by the exponent bias b. Hence, it is apparent that the required hardware logic for the converter is indeed minimal and the extra hardware cost is truly minor as well. Furthermore, updates of those format registers are extremely infrequent. Even the layer-wise optimization is applied, those registers are only modified at the start of each layer. In other words, virtually no runtime overhead imposed due to those register updates.
CONCLUSION
In this work, we propose the flexible 8-bit floating-point (FFP8) format for accurate and memory- efficient inference of deep neural networks. Our FFP8 format offers three adjustable options: 1) the size of exponent/fraction field, 2) the value of exponent bias, and 3) the presence of the sign bit, whereas those rigid conventional formats simply leave nothing- In this paper, we explain how the exponent size and bias jointly define the representable value range of given FFP8 format. We also demonstrate how explore the best-fit signed/unsigned FFP8 formats for weights and activations to achieve more accurate inference outcomes. Besides, layer-wise optimization flow, which discovers the best-fit formats for each individual layer, is presented to further improve the accuracy. model retraining methodology that can be carried out in typical FP32 frameworks without the need of special training skills is also introduced. The experimental results on various DNN models show that the proposed FFP8- based inference flow achieves an extremely low accuracy loss of 0.1% 0.3% as compared to the FP32 baseline even without model retraining. Moreover, we also show that the extra hardware for supporting the FFP8 format is minimal. Therefore, it is conclusive that the proposed FFP8-based inference framework should be a better solution for those computing systems with limited memory capacity and bandwidth, e.g., edge and AloT devices.
REFERENCES
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. In Advances in Neural Information Processing Systems, 2018.
Leopold Cambier, Anahita Bhiwandiwalla, Ting Gong, Mehran Nekuii, Oguz HElibol, and Hanlin Tang. Shifted and squeezed 8-bit floating point format for low-precision training of deep neural networks. In International Conference on Learning Representutions, 2020.
Matthieu Courbariaux, Yoshua Bengio, and Jean Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems, 2015.
Dipankar Das, Naveen Mellempudi, Dheevatsa Mudigere, Dhiraj Kalamkar, Sasikanth Avancha, Kunal Banerjee, Srinivas Sridharan, and et al. Mixed precision training of convolutional neural networks using integer operations. In International Conference on Learning Representationss 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition, 2009,
M. Everingham, L.
Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Vi- sual Object Classes Challenge 2011 (VOC2011) Results, http:/ www. www.paaaaaaleeee....... http:/hhaaleeeee//////oo/e/eeee/ hatp://///ooooo22001//oaaaao//// http//owwwwhw/o/eeeoeeoe hitp://idxxx... html
Song Han, Mao Huizi, and Dally William Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In International Conference on Learning Representations, 2016.
Song Han, Xingyu Liu, Huizi Mao. Jing Pu, Ardavan Pedram, Mark A. Horowitz, and William J. Dally. EIE: efficient inference engine on compressed deep neural network. arXiv e-prints, art. arXiv:1602.01528. February 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Conference Computer Vision and Pattern Recognition, 2018.

---

Under review as a conference paper at ICLR 2021
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Vat-Seng Chua. Neural collab- orative filterin. arXiv e-prints, art. arXiv: 1ee.1178.000000 August 2017.
M. Horowitz. Computing's ss energy problem (and what we can do about it). 2014 IEEE Interna- tional Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pp. 10-14 2014. doi:
D0,1109//SCCC.21146677777
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv e-prints, art. arXiv:17044 04861,, April 2017.
Gao Huang, Liu Zhuang, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks, In Conference Computer Vision and Pattern Recognition, 2017.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv,, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. In The Journal of Machine Learning Research, 2017.
Intel. Bfloat16 hardware numerics definition white paper. 2018.
Intel. Leadership performance with 2nd rederreerttion intel xeon scalable processors, 2019.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for effi- cient watneger-rritmmtiiooon inference. arXiv e-prints, art. arXiv:1712.05811 December 2017.
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. study of bfloat16 for deep learning training. arXiv e-prints, arXiv: 905. 12322, May 2019.
Urs Köster, Tristan Webb, Xin Wang, Marcel Nassar, Arjun K Bansal, William Constable, Oguz Elibol, Scott Gray, Stewart Hall, and Luke et al Hornof. Flexpoint: An adaptive numerical format for efficient training of deep neural networks. In Advances in neural information processing systems, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, 2012.
Hao Li, Asim Kadav, Igor Durdanovic, Graf. Samet, and Hans Pete. Pruning filters for effecient convnets. In International Conference on Learning Representations, 2017.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. arXiv e-prints, arXiv:1411.4033 November 2014.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, and Boris et al Ginsburg. Mixed precision training. In International Conference on Learning Representations, 2018.
Nvidia. Nvidia tesla v100 gpu architecture, 2017.
Nvidia. Nvidia a100 tensor core gpu architecture, 2020.
Physionet. Af classification from short single lead eog recording the physionet computing in cardiology challenge. bttts::/phhsiinne 0/ 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015.
Xiao Sun, Jungwook Choi, Chia- Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalakshmi Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. In Neural Information Processing Systems, 2019.
10

---

Under review as a conference paper at ICLR 2021
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Computer Vision and Pattern Recognition, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Erhan, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, 2017.
Naigang Wang and Jungw Choi. Training deep neural networks with 8-bit floating point numbers. In Advances in Neural Information Processing Systems, 2018.
Yang Zhao, Xiaohan Chen, Yue Wang. Chaojian Li, Haoran You, Yonggan Fu, Yuan Xie, Zhangyang Wang, and Yingyan Lin. Smartexchange: Trading higher-cost memory storage/access for lower- cost computation. In International Symposium on Computer Architecture (ISCA), 2020.
Shuchang Zhou, Zekun Ni, He Wen, Yuxin Wu, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. In CoRR, abs/1606.06160, 2016.


Reviews:
[{"review_id": "wNn5UIC_-29", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a flexible floating-point format that can save storage space by being highly configurable in terms of the bit width of the exponent/fraction field, the exponent bias, and the presence of the sign bit. The experiments in the paper demonstrate that the proposed format achieves a very low accuracy loss of < 0.3% compared to the regular float32 format for several popular image classification models. \n\nStrengths\n---\n- The concept of flexible floating-point format makes sense and can potentially result in significant space savings for large models. \n- The loss drop in the models compared to the standard float32 format seems to be minimal\n- The authors evaluated the proposed format over multiple models: VGG 16, ResNet-50, ResNet-34, and ResNet-18 \n\nWeaknesses\n---\n- No full system implementation to demonstrate the performance/memory gains. I that the updates to the form registers will be infrequent and cheap \n- The hardware details are very sketchy. Figure 6 seems to be only showing the high-level components.\n\n\nOverall, the paper’s ideas are promising but my score reflects the fact that the authors presented end-to-end experiments demonstrating the performance/storage gains.", "extra_scores": {}, "extra_text": {}}, {"review_id": "lNbu8u5AsKO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new flexible floating point format (FFP8) on 8 bits, to help alleviate the high memory demand of deep networks inference, while preserving high accuracy. There is a large body of literature on reducing the data format, typically from 32 bits to 16, 8 and even below. There is previous work on using an 8-bit floating point FP(8), usually (1,4,3) or (1,5,2) where 1 bit is used for sign, 5 or 4 bits are used for the exponent and 3 or 2 bits are used for the fraction. \n\nThe new FFP8 proposed in the paper offers more configurable parameters: (1) the bit width of exponent/fraction, (2) the exponent bias (usually this is implicit in FP8 but it can be changed in FFP8), (3) the presence of the sign bit (this can be removed and the bit can be used for exponent or fraction). By allowing flexible control of the dynamic range of the 8-bit FFP8, the accuracy loss is minimized. The authors observe that both the maximum magnitude and the value distribution are quite dissimilar between weights and activations in most DNNs. Therefore, the variable exponent size and exponent bias are better suited. Some activation functions always produce non-negative results (e.g., ReLU), therefore the sign bit can be reclaimed. \n\nThe paper includes experimental evaluation where a best fit format selection shows a minimal accuracy loss for the VGG-16 network. Further optimization can be achieved over each layer, because distributions across different layers are dissimilar for both weights and activations, and the distribution of an individual layer is typically narrower than the one of the entire model.\n\nI advocate for the acceptance of the paper. Although there is a long line of research examining the data format for training and inference in deep networks, the paper provides evidence of the usefulness of some flexibility in the 8-bit format. The proposed would not require hardware changes except for simple translations between FP32 and FFP8.\n\nMinor comments / typos:\n\nFig. 4 and 5 - the vertical axis might look better with the same max values (range).\n\npg.2 : Moreover, recent studies proposed several training frameworks that __producing__ weights only in 8-bit floating-point formats\n\npg.3 : Note that there is actually only one parameter, the bit width of the exponent (y), __that__ can be freely chosen when defining", "extra_scores": {}, "extra_text": {}}, {"review_id": "aYkeJ-w9ILV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary:\n\nThe paper introduces a new floating point format FFP8 that can adaptively choose the exponent bias as well as the existence of the sign bit. FFP8 is more flexible than other 8-bit floating point formats with fixed exponent biases. The authors show how FF8 can be used to cover the dynamic range of NN weights and activations while still achieving higher precision than commonly used FP 8 formats.\n\n## Strengths:\n\n* Experiments on how various settings of FFP8 can be used to represent weights and activations of different layers and comparison with existing FP 8 formats.\n* Varying exponent bias is a good insight to shift the dynamic range of FP formats.\n\n## Weaknesses and Questions for the Authors\n\n* Limited details on how hardware support for FFP8 look like.\nThe authors claim in section 5 that the extra hardware support required for FFP8 would be minimal. However, no data or simulations are there to back this claim. In a possible hardware implementation of FFP8, each floating point instruction should be preceded by an instruction which sets (x,y,b) registers for FFP8 to F32 conversion. The frequency of setting these registers may vary based on the application, but nevertheless is an overhead compared to the current FP 8 implementations. The authors should quantify this overhead with a hardware simulation. Alternatively, the authors can describe a different and an efficient implementation of FP 8. This is an important detail that is missing from the paper.\n* Compiler support for FFP8\n The authors have not described how FFP 8 would be emitted, specially when different settings are used for different layers. Do you expect the programmer to hand code this? Or will there be compiler support? If so, what would the compiler support look like? What would the compiler optimization algorithm look like to produce FFP 8 code? These details are not answered by the paper.\n* More evaluation on other commonly used NNs\nThe authors present results for VGG-16. I would like to see more results on different networks. The authors can cut down on some of the graphs or make them smaller to get more space (specially Figure 5). Also, in general I feel like rather than giving out all the results that bring us to the same conclusion, the authors can summarize the results or graphs to make the message / conclusion of the paper more clearer.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BuPDcYXImy", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores 8-bit floating point formats for the inference of deep neural networks. The quantization is applied on the weight and activation tensors, but computation engine remains in FP32.  To cover the different ranges of weight and activation tensors, the authors propose to use exponent bias. The authors did ablation studies on the impact of numerical formats, e.g. bit-width and exponent biases, on model accuracies. The experiments are performed on vision models on ImageNet, including VGG and ResNet 50/34/18. \n\nThe paper is relatively clear written, and experiments seem sound, however, I have some major concerns on the objective and novelty of this paper. \n\n1). It is not clear to me the objective of this paper. This paper introduces an 8-bit quantized inference framework. However, it is well-known that, 8-bit precision can be applied to popular DNN models to accelerate inference while maintaining model accuracies and many such systems have already been put into products (e.g. TPU). The state-of-the-art inference quantization work are primarily in the precision of 4 bit or less. But this paper did not compare their work to any of the latest inference works. Instead, the three references this paper compared to, i.e. (Wang & Choi, 2018; Cambier et al., 2020; Sun et al., 2019), are all for the acceleration of DNN TRAINING, where the challenging can be very different, e.g. quantization of gradient tensors.\n\n2). On the same note, this paper claimed advantages of using FP32 computation engine, however, the three training papers, by definition, can also be used in FP32 computations through the same high precision converting described in this paper. In addition, it is not clear to me how much one can benefit from quantization while keeping computation in 32bit? For example, one may save some memories space, but the whole inference will be limited by the FP32 computations in terms of throughput, latency, power and cost. Can the authors provide an application case where only quantizing data is beneficial?\n\n3). The main contribution claimed in the paper are to use exponent bias to cover tensors with different range of distribution. However, this technique has be introduced and discussed in (Sun et at., 2019) paper and has also been investigated in detail in (Cambier et al., 2020) paper. The authors of this paper did not provide any new insight although using in a much simpler case, i.e. only forward paths.\n\nI think this paper lacks clear objective, novelty and insightful analysis, therefore not good enough for ICLR.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After reading the paper, reviews and authors’ feedback. The meta-reviewer agrees with the reviewers that the paper has limited novelty as there are already previous studies on setting floating point configurations. Additionally, the particular hardware setting that the authors provide seems to rely on a fp32 FMA, which defeats the purpose of a low bit floating point(where smaller FMA could have been used).Therefore this paper is rejected.

Thank you for submitting the paper to ICLR. 

Decision: Reject