All-You-Can-Fit 8-Bit Flexible Floating-Point Format for Accurate and Memory-Efficient Inference of Deep Neural Networks

1 INTRODUCTION . With the rapid progress of deep neural network ( DNN ) techniques , innovative applications of deep learning in various domains , such as computer vision and natural language processing ( NLP ) , are getting more mature and powerful ( Huang et al. , 2017 ; Vaswani et al. , 2017 ; Szegedy et al. , 2015 ; Howard et al. , 2017 ; He et al. , 2017 ; Krizhevsky et al. , 2012 ) . To improve the model accuracy , one of the most commonly used strategies is to add more layers into a network , which inevitably increases the number of weight parameters and activation values of a model . Today , it is typical to store weights and activations in the 32-bit IEEE single-precision floating-point format ( FP32 ) . Those 32-bit data accesses thus become an extremely heavy burden on the memory subsystem in a typical edge or AIoT device , which often has very limited memory capacity and bandwidth . Even for high-end GPU or dedicated network processing unit ( NPU ) based computing platforms , off-chip DRAM bandwidth is still a major performance bottleneck . To relieve the issue of memory bandwidth bottleneck , several attempts of various aspects have been made including ( but not limited to ) weight pruning ( Li et al. , 2017 ; Han et al. , 2016 ) , weight/activation quantization ( Courbariaux et al. , 2015 ; Hubara et al. , 2017 ) , and probably the most straightforward way : storing weights and activations in a shorter format ( Köster et al. , 2017 ) . One trivial way to do so is to adopt the 16-bit IEEE half-precision floating-point format ( FP16 ) . An FP16 number consists of 1 sign bit , 5 exponent bits , and 10 fraction bits . In addition , Google proposed another 16-bit format , named Brain Floating-Point Format ( BFP16 ) , simply by truncating the lower half of the FP32 format ( Kalamkar et al. , 2019 ) . Compared with FP16 , BFP16 allows a significantly wider dynamic value range at the cost of 3-bit precision loss . Note that the exponent bias in all of the above formats is not a free design parameter . Conventionally , the value is solely determined by the exponent size . For example , for FP16 with 5-bit exponent , the exponent bias is automatically fixed to 15 ( 25−1 − 1 ) . To make the data even shorter , 8-bit fixed-point signed/unsigned integer formats ( INT8 and UINT8 ) are also broadly adopted . However , the 8-bit fixed-point format inherently has a narrower dynamic value range so that the model accuracy loss is usually not negligible even after extra symmetric or asymmetric quantization . As a consequence , there are a number of attempts concentrating on utilizing mixed-precision or pure 8-bit floating-point numbers in deep learning applications . Various techniques have been developed for mixed-precision training ( Banner et al. , 2018 ; Micikevicius et al. , 2018 ; Das et al. , 2018 ; Zhou et al. , 2016 ) . Moreover , recent studies proposed several training frameworks that produces weights only in 8-bit floating-point formats ( Wang & Choi , 2018 ; Cambier et al. , 2020 ; Sun et al. , 2019 ) . In these studies , the underlying 8-bit floating-point numbers in training and inference are represented in the format of FP8 ( 1 , 5 , 2 ) or FP8 ( 1 , 4 , 3 ) , where the enclosed three parameters indicate the bit length of sign , exponent , and fraction , respectively . Note that 4 or 5 bits are essential for the exponent in their frameworks , or the corresponding dynamic range may not cover both weight and activation values well . Consequently , only 2 or 3 bits are available for fraction , which inevitably leads to lower accuracy . In this paper , we present an extremely flexible 8-bit floating-point ( FFP8 ) number format . In FFP8 , all parameters – the bit width of exponent/fraction , the exponent bias , and the presence of the sign bit – are configurable . Three major features of our inference methodology associated with the proposed FFP8 format are listed as follows . First , it is observed that both the maximum magnitude and the value distribution are quite dissimilar between weights and activations in most DNNs . It suggests the best exponent size and exponent bias for weights should be different from those for activations to achieve higher accuracy . Second , a large set of commonly-used activation functions always produce nonnegative outputs ( e.g. , ReLU ) . It implies that activations are actually unsigned if one of those activation functions is in use . Hence , it implies the sign bit is not required for those activations , which makes either exponent or fraction 1-bit longer . Note that even one bit can make a big impact since only 8 bits are available . Third , all aforementioned studies require their own sophisticated training frameworks to produce 8-bit floating-point models . Our flow does not . Our flow simply takes a model generated by any conventional FP32 training framework as the input . Then , it simply converts the given pre-trained FP32 model into an FFP8 model . The rest of this paper is organized as follows . Section 2 briefly introduces related work . In Section 3 , we elaborate more on the proposed FFP8 format and how to properly convert a pre-trained FP32 model into an FFP8 one . Section 4 demonstrates the experimental results on various DNN models . The system and hardware design issues for the support of FFP8 numbers are discussed in Section 5 . Finally , the concluding remarks are given in Section 6 . 2 RELATED WORK . DNNs are becoming larger and more complicated , which means they require a bigger memory space and consume more energy during inference . As a result , it is getting harder to deploy them on systems with limited memory capacity and power budget ( e.g. , edge devices ) . ( Han et al. , 2016 ; Zhao et al. , 2020 ; Horowitz , 2014 ) also demonstrated that off-chip DRAM access is responsible for a significantly big share of system power consumption . Hence , it remains an active research topic about how to reduce the memory usage for weights and activations . As mentioned in the previous section , one way to do so is to use short 8-bit floating-point number formats . Wang & Choi ( 2018 ) introduced a DNN training methodology using 8-bit floating-point numbers in FP8 ( 1 , 5 , 2 ) format . The methodology features chunk-based accumulation and stochastic rounding methods for accuracy loss minimization . Besides , to achieve a better trade-off between precision and dynamic range during model training , Sun et al . ( 2019 ) proposed an improved methodology that utilizes two different 8-bit floating-point formats – FP8 ( 1 , 4 , 3 ) for forward propagation and FP8 ( 1 , 5 , 2 ) for backward propagation . Nevertheless , both methodologies fail to make a DNN model entirely in 8-bit numbers : the first and the last layers of the given model are still in 16-bit floating-point numbers ; otherwise , the model suffers about 2 % accuracy degradation . Cambier et al . ( 2020 ) then proposed the S2FP8 format , which allows a DNN model represented in 8-bit floating point numbers completely . By adding a scaling factor and a shifting factor , data can thus be well represented in FP8 ( 1 , 5 , 2 ) after proper shifting and squeezing operations , which eliminates the need of 16-bit floating point numbers . However , the S2FP8 format still results in about 1 % accuracy drop in ResNet-50 ( He et al. , 2018 ) . 3 FLEXIBLE 8-BIT FLOATING-POINT ( FFP8 ) FORMAT . 3.1 DEFINITION OF THE FFP8 FORMAT . In this subsection , a flexible 8-bit floating-point ( FFP8 ) format , which leads to more accurate inference outcomes of deep neural networks , is presented . A typical floating-point format consists of sign ( s ) , exponent ( e ) , and fraction ( f ) fields , and the bit length of each field is specified as x , y , z , respectively . Besides , one more parameter , exponent bias ( b ) , is required to completely specify a floating-point number . Conventionally , b is always implicitly set to 2y−1 − 1 . In this paper , an n-bit floating-point format is denoted as ( x , y , z , b ) or ( x , y , z ) , where n = x + y + z . If b is missing , the default value is implicitly used . Figure 1 ( a ) illustrates one 16-bit FP16 format and two commonly used 8-bit formats : ( 1 , 5 , 2 , 15 ) and ( 1 , 4 , 3 , 7 ) . Note that there is actually only one parameter , the bit width of the exponent ( y ) , that can be freely chosen when defining a new n-bit conventional floating-point format since x is always 1 , z is always n − y − 1 , and b is always 2y−1 − 1 . The above fact motivated us to think out-of-the-box and thus develop a more flexible format , named flexible 8-bit floating-point ( FFP8 ) format and shown in Figure 1 ( b ) . In addition to the size of exponent ( y ) , the FFP8 format offers two more parameters . First , the exponent bias ( b ) is not necessarily equal to 2y−1 − 1 and can be set to any integer , which helps cover the value distributions of both weights and activations well in a shorter exponent size ( y ) . Second , the sign bit is present or not ( i.e. , x can be 0 or 1 ) . Without the sign bit ( x = 0 ) , unsigned activations can be better represented in higher precision . To be more precise , there are only two restrictions on an n-bit FFP8 format : 1 ) x + y + z = n ; 2 ) x must be 0 or 1 . Next , we are about to show how the proposed FFP8 format improves the inference accuracy . 3.2 WEIGHT DISTRIBUTION AND THE WAYS VARIOUS 8-BIT FORMATS COPE WITH IT . Overall speaking , the magnitudes of weights in most DNN models are usually small . Take a popular image classification model VGG-16 ( Simonyan & Zisserman , 2015 ) as an example , the maximum magnitude of weights in the whole model is less than 2 . Figure 2 gives the overall weight distribution ( in log scale ) of VGG-16 trained via a conventional FP32 framework . Figure 2 ( a ) illustrates how the conventional FP8 ( 1 , 4 , 3 ) copes with those weights . The rectangle in red is called the range window , which specifies the value range that FP8 ( 1 , 4 , 3 ) can represent . The purple vertical dash line further partitions the window into the norm region ( right side ) and the denorm region ( left side ) . The star marks the position where the weight of the maximum magnitude locates . Note that virtually the entire right half of the range window in Figure 2 ( a ) covers no weights , while 9.6 % of leftmost weights can not be included in the range window . In order to contain almost every weight in a range window , FP8 ( 1 , 5 , 2 ) can be selected alternatively , as shown in Figure 2 ( b ) . A bigger exponent size ( 4 to 5 ) results in a larger range window . However , there are only 4 ( 22 ) instead of 8 ( 23 ) representative values available for each x in the norm region since the fraction size decreases from 3 to 2 , which potentially results in a lower accuracy . Note that FP8 ( 1 , 4 , 3 ) and FP8 ( 1 , 5 , 2 ) are two most commonly used 8-bit floating point formats in the previous studies . With the help of the proposed FFP8 format , things can change a lot . Figure 2 ( c ) shows what happens if FFP8 ( 1 , 4 , 3 , 15 ) is in use . The range window is of the same size as that in Figure 2 ( a ) but is leftshifted by 8 positions due to the exponent bias is set to 15 instead of the default value 7 . It is obvious that FFP8 ( 1 , 4 , 3 , 15 ) can better cope with weights in VGG-16 than FP8 ( 1 , 4 , 3 ) and FP ( 1 , 5 , 2 ) . Furthermore , Figure 2 ( d ) shows what if FFP8 ( 1 , 3 , 4 , 7 ) is in use . Comparing FFP8 ( 1 , 3 , 4 , 7 ) against FFP8 ( 1 , 4 , 3 , 15 ) , weights in the norm region ( over 35 % ) are represented in 1-bit higher precision whereas 4.6 % of leftmost weights are not included in the range window . However , those out-of-the-window weights can be regarded as some sort of pruned weights . That is , more investigation should be further conducted to determine whether FFP8 ( 1 , 3 , 4 , 7 ) or FFP8 ( 1 , 4 , 3 , 15 ) is better for the weights in VGG-16 . One way or another , it is clear that the proposed FFP8 formats can achieve certain improvement that the conventional 8-bit formats can not .

Summary: This paper explores 8-bit floating point formats for the inference of deep neural networks. The quantization is applied on the weight and activation tensors, but computation engine remains in FP32.  To cover the different ranges of weight and activation tensors, the authors propose to use exponent bias. The authors did ablation studies on the impact of numerical formats, e.g. bit-width and exponent biases, on model accuracies. The experiments are performed on vision models on ImageNet, including VGG and ResNet 50/34/18.