All-You-Can-Fit 8-Bit Flexible Floating-Point Format for Accurate and Memory-Efficient Inference of Deep Neural Networks

1 INTRODUCTION . With the rapid progress of deep neural network ( DNN ) techniques , innovative applications of deep learning in various domains , such as computer vision and natural language processing ( NLP ) , are getting more mature and powerful ( Huang et al. , 2017 ; Vaswani et al. , 2017 ; Szegedy et al. , 2015 ; Howard et al. , 2017 ; He et al. , 2017 ; Krizhevsky et al. , 2012 ) . To improve the model accuracy , one of the most commonly used strategies is to add more layers into a network , which inevitably increases the number of weight parameters and activation values of a model . Today , it is typical to store weights and activations in the 32-bit IEEE single-precision floating-point format ( FP32 ) . Those 32-bit data accesses thus become an extremely heavy burden on the memory subsystem in a typical edge or AIoT device , which often has very limited memory capacity and bandwidth . Even for high-end GPU or dedicated network processing unit ( NPU ) based computing platforms , off-chip DRAM bandwidth is still a major performance bottleneck . To relieve the issue of memory bandwidth bottleneck , several attempts of various aspects have been made including ( but not limited to ) weight pruning ( Li et al. , 2017 ; Han et al. , 2016 ) , weight/activation quantization ( Courbariaux et al. , 2015 ; Hubara et al. , 2017 ) , and probably the most straightforward way : storing weights and activations in a shorter format ( Köster et al. , 2017 ) . One trivial way to do so is to adopt the 16-bit IEEE half-precision floating-point format ( FP16 ) . An FP16 number consists of 1 sign bit , 5 exponent bits , and 10 fraction bits . In addition , Google proposed another 16-bit format , named Brain Floating-Point Format ( BFP16 ) , simply by truncating the lower half of the FP32 format ( Kalamkar et al. , 2019 ) . Compared with FP16 , BFP16 allows a significantly wider dynamic value range at the cost of 3-bit precision loss . Note that the exponent bias in all of the above formats is not a free design parameter . Conventionally , the value is solely determined by the exponent size . For example , for FP16 with 5-bit exponent , the exponent bias is automatically fixed to 15 ( 25−1 − 1 ) . To make the data even shorter , 8-bit fixed-point signed/unsigned integer formats ( INT8 and UINT8 ) are also broadly adopted . However , the 8-bit fixed-point format inherently has a narrower dynamic value range so that the model accuracy loss is usually not negligible even after extra symmetric or asymmetric quantization . As a consequence , there are a number of attempts concentrating on utilizing mixed-precision or pure 8-bit floating-point numbers in deep learning applications . Various techniques have been developed for mixed-precision training ( Banner et al. , 2018 ; Micikevicius et al. , 2018 ; Das et al. , 2018 ; Zhou et al. , 2016 ) . Moreover , recent studies proposed several training frameworks that produces weights only in 8-bit floating-point formats ( Wang & Choi , 2018 ; Cambier et al. , 2020 ; Sun et al. , 2019 ) . In these studies , the underlying 8-bit floating-point numbers in training and inference are represented in the format of FP8 ( 1 , 5 , 2 ) or FP8 ( 1 , 4 , 3 ) , where the enclosed three parameters indicate the bit length of sign , exponent , and fraction , respectively . Note that 4 or 5 bits are essential for the exponent in their frameworks , or the corresponding dynamic range may not cover both weight and activation values well . Consequently , only 2 or 3 bits are available for fraction , which inevitably leads to lower accuracy . In this paper , we present an extremely flexible 8-bit floating-point ( FFP8 ) number format . In FFP8 , all parameters – the bit width of exponent/fraction , the exponent bias , and the presence of the sign bit – are configurable . Three major features of our inference methodology associated with the proposed FFP8 format are listed as follows . First , it is observed that both the maximum magnitude and the value distribution are quite dissimilar between weights and activations in most DNNs . It suggests the best exponent size and exponent bias for weights should be different from those for activations to achieve higher accuracy . Second , a large set of commonly-used activation functions always produce nonnegative outputs ( e.g. , ReLU ) . It implies that activations are actually unsigned if one of those activation functions is in use . Hence , it implies the sign bit is not required for those activations , which makes either exponent or fraction 1-bit longer . Note that even one bit can make a big impact since only 8 bits are available . Third , all aforementioned studies require their own sophisticated training frameworks to produce 8-bit floating-point models . Our flow does not . Our flow simply takes a model generated by any conventional FP32 training framework as the input . Then , it simply converts the given pre-trained FP32 model into an FFP8 model . The rest of this paper is organized as follows . Section 2 briefly introduces related work . In Section 3 , we elaborate more on the proposed FFP8 format and how to properly convert a pre-trained FP32 model into an FFP8 one . Section 4 demonstrates the experimental results on various DNN models . The system and hardware design issues for the support of FFP8 numbers are discussed in Section 5 . Finally , the concluding remarks are given in Section 6 . 2 RELATED WORK . DNNs are becoming larger and more complicated , which means they require a bigger memory space and consume more energy during inference . As a result , it is getting harder to deploy them on systems with limited memory capacity and power budget ( e.g. , edge devices ) . ( Han et al. , 2016 ; Zhao et al. , 2020 ; Horowitz , 2014 ) also demonstrated that off-chip DRAM access is responsible for a significantly big share of system power consumption . Hence , it remains an active research topic about how to reduce the memory usage for weights and activations . As mentioned in the previous section , one way to do so is to use short 8-bit floating-point number formats . Wang & Choi ( 2018 ) introduced a DNN training methodology using 8-bit floating-point numbers in FP8 ( 1 , 5 , 2 ) format . The methodology features chunk-based accumulation and stochastic rounding methods for accuracy loss minimization . Besides , to achieve a better trade-off between precision and dynamic range during model training , Sun et al . ( 2019 ) proposed an improved methodology that utilizes two different 8-bit floating-point formats – FP8 ( 1 , 4 , 3 ) for forward propagation and FP8 ( 1 , 5 , 2 ) for backward propagation . Nevertheless , both methodologies fail to make a DNN model entirely in 8-bit numbers : the first and the last layers of the given model are still in 16-bit floating-point numbers ; otherwise , the model suffers about 2 % accuracy degradation . Cambier et al . ( 2020 ) then proposed the S2FP8 format , which allows a DNN model represented in 8-bit floating point numbers completely . By adding a scaling factor and a shifting factor , data can thus be well represented in FP8 ( 1 , 5 , 2 ) after proper shifting and squeezing operations , which eliminates the need of 16-bit floating point numbers . However , the S2FP8 format still results in about 1 % accuracy drop in ResNet-50 ( He et al. , 2018 ) . 3 FLEXIBLE 8-BIT FLOATING-POINT ( FFP8 ) FORMAT . 3.1 DEFINITION OF THE FFP8 FORMAT . In this subsection , a flexible 8-bit floating-point ( FFP8 ) format , which leads to more accurate inference outcomes of deep neural networks , is presented . A typical floating-point format consists of sign ( s ) , exponent ( e ) , and fraction ( f ) fields , and the bit length of each field is specified as x , y , z , respectively . Besides , one more parameter , exponent bias ( b ) , is required to completely specify a floating-point number . Conventionally , b is always implicitly set to 2y−1 − 1 . In this paper , an n-bit floating-point format is denoted as ( x , y , z , b ) or ( x , y , z ) , where n = x + y + z . If b is missing , the default value is implicitly used . Figure 1 ( a ) illustrates one 16-bit FP16 format and two commonly used 8-bit formats : ( 1 , 5 , 2 , 15 ) and ( 1 , 4 , 3 , 7 ) . Note that there is actually only one parameter , the bit width of the exponent ( y ) , that can be freely chosen when defining a new n-bit conventional floating-point format since x is always 1 , z is always n − y − 1 , and b is always 2y−1 − 1 . The above fact motivated us to think out-of-the-box and thus develop a more flexible format , named flexible 8-bit floating-point ( FFP8 ) format and shown in Figure 1 ( b ) . In addition to the size of exponent ( y ) , the FFP8 format offers two more parameters . First , the exponent bias ( b ) is not necessarily equal to 2y−1 − 1 and can be set to any integer , which helps cover the value distributions of both weights and activations well in a shorter exponent size ( y ) . Second , the sign bit is present or not ( i.e. , x can be 0 or 1 ) . Without the sign bit ( x = 0 ) , unsigned activations can be better represented in higher precision . To be more precise , there are only two restrictions on an n-bit FFP8 format : 1 ) x + y + z = n ; 2 ) x must be 0 or 1 . Next , we are about to show how the proposed FFP8 format improves the inference accuracy . 3.2 WEIGHT DISTRIBUTION AND THE WAYS VARIOUS 8-BIT FORMATS COPE WITH IT . Overall speaking , the magnitudes of weights in most DNN models are usually small . Take a popular image classification model VGG-16 ( Simonyan & Zisserman , 2015 ) as an example , the maximum magnitude of weights in the whole model is less than 2 . Figure 2 gives the overall weight distribution ( in log scale ) of VGG-16 trained via a conventional FP32 framework . Figure 2 ( a ) illustrates how the conventional FP8 ( 1 , 4 , 3 ) copes with those weights . The rectangle in red is called the range window , which specifies the value range that FP8 ( 1 , 4 , 3 ) can represent . The purple vertical dash line further partitions the window into the norm region ( right side ) and the denorm region ( left side ) . The star marks the position where the weight of the maximum magnitude locates . Note that virtually the entire right half of the range window in Figure 2 ( a ) covers no weights , while 9.6 % of leftmost weights can not be included in the range window . In order to contain almost every weight in a range window , FP8 ( 1 , 5 , 2 ) can be selected alternatively , as shown in Figure 2 ( b ) . A bigger exponent size ( 4 to 5 ) results in a larger range window . However , there are only 4 ( 22 ) instead of 8 ( 23 ) representative values available for each x in the norm region since the fraction size decreases from 3 to 2 , which potentially results in a lower accuracy . Note that FP8 ( 1 , 4 , 3 ) and FP8 ( 1 , 5 , 2 ) are two most commonly used 8-bit floating point formats in the previous studies . With the help of the proposed FFP8 format , things can change a lot . Figure 2 ( c ) shows what happens if FFP8 ( 1 , 4 , 3 , 15 ) is in use . The range window is of the same size as that in Figure 2 ( a ) but is leftshifted by 8 positions due to the exponent bias is set to 15 instead of the default value 7 . It is obvious that FFP8 ( 1 , 4 , 3 , 15 ) can better cope with weights in VGG-16 than FP8 ( 1 , 4 , 3 ) and FP ( 1 , 5 , 2 ) . Furthermore , Figure 2 ( d ) shows what if FFP8 ( 1 , 3 , 4 , 7 ) is in use . Comparing FFP8 ( 1 , 3 , 4 , 7 ) against FFP8 ( 1 , 4 , 3 , 15 ) , weights in the norm region ( over 35 % ) are represented in 1-bit higher precision whereas 4.6 % of leftmost weights are not included in the range window . However , those out-of-the-window weights can be regarded as some sort of pruned weights . That is , more investigation should be further conducted to determine whether FFP8 ( 1 , 3 , 4 , 7 ) or FFP8 ( 1 , 4 , 3 , 15 ) is better for the weights in VGG-16 . One way or another , it is clear that the proposed FFP8 formats can achieve certain improvement that the conventional 8-bit formats can not .

Summary: The paper proposes a new flexible floating point format (FFP8) on 8 bits, to help alleviate the high memory demand of deep networks inference, while preserving high accuracy. There is a large body of literature on reducing the data format, typically from 32 bits to 16, 8 and even below. There is previous work on using an 8-bit floating point FP(8), usually (1,4,3) or (1,5,2) where 1 bit is used for sign, 5 or 4 bits are used for the exponent and 3 or 2 bits are used for the fraction.