Exploring  the Potential of Low-Bit Training of Convolutional Neural Networks

1 INTRODUCTION . Convolutional neural networks ( CNNs ) have achieved state-of-the-art performance in many computer vision tasks , such as image classification ( Krizhevsky et al. , 2012 ) and object detection ( Redmon et al. , 2016 ; Liu et al. , 2016 ) . However , deep CNNs are both computation and storage-intensive . The training process could consume up to hundreds of ExaFLOPs of computations and tens of GBytes of storage ( Simonyan & Zisserman , 2014 ) , thus posing a tremendous challenge for training in resource-constrained environments . At present , the most common training method is to use GPUs , but it consumes much energy . The power of a running GPU is about 250W , and it usually takes more than 10 GPU-days to train one CNN model on ImageNet ( Deng et al. , 2009 ) . It makes AI applications expensive and not environment-friendly . Reducing the precision of NNs has drawn great attention since it can reduce both the storage and computational complexity . It is pointed out that the power consumption and circuit area of fixed-point multiplication and addition units are greatly reduced compared with floating-point ones ( Horowitz , 2014 ) . Many studies ( Jacob et al. , 2017a ; Dong et al. , 2019 ; Banner et al. , 2018b ) focus on amending the training process to acquire a reduced-precision model with higher inference efficiency . Besides the studies on improving inference efficiency , there exist studies that accelerate the training process . Wang et al . ( 2018 ) and Sun et al . ( 2019 ) reduce the floating-point bit-width to 8 during the training process . Wu et al . ( 2018 ) implements a full-integer training procedure to reduce the cost but fails to get acceptable performance . As shown in Tab . 1 , Conv in the training process accounts for the majority of the operations . Therefore , this work aims at simplifying convolution to low-bit operations , while retaining a similar performance with the full-precision baseline . The contributions of this paper are : 1 . This paper proposes a low-bit training framework to improve the energy efficiency of CNN training . We design a low-bit tensor format with multi-level scaling ( MLS format ) , which can strike a better trade-off between the accuracy and bit-width , while taking the hardware efficiency into consideration . The multi-level scaling technique extracts the common exponent of tensor elements as much as possible to reduce the element-wise bitwidth , thus improving the energy efficiency . To leverage the MLS format efficiently , we develop the corresponding dynamic quantization and the MLS tensor convolution arithmetic . 2 . Extensive experiments demonstrate the effectiveness of our low-bit training framework . One only needs 1-bit mantissa and 2-bit exponent to train ResNet-20 on CIFAR-10 while retaining the same accuracy as the full-precision training . On ImageNet , using 4-bit mantissa and 2-bit exponent is enough for training ResNet-18 , with a precision loss within 1 % . Our method achieves higher energy efficiency using fewer bits than previous floating-point training methods and better accuracy than previous fixed-point training methods . 3 . We estimate the hardware energy that implements the MLS convolution arithmetic . Using our MLS tensor format , the energy efficiency of convolution can be improved by over 6.8× , than the full-precision training , and over 1.2× than previous low-bit training methods . 2 RELATED WORK . 2.1 POST-TRAINING QUANTIZATION . Earlier quantization methods like ( Han et al. , 2015 ) focused on the post-training quantization , and quantized the pre-trained full-precision model using the codebook generated by clustering or other criteria ( e.g. , SQNR Lin et al . ( 2015 ) , entropy Park et al . ( 2017 ) ) . Banner et al . ( 2018b ) selected the quantization bit-width and clipping value for each channel through the analytical investigation . Jacob et al . ( 2017b ) developed an integer arithmetic convolution for efficient inference , but it ’ s hard to be used in training because the scale of the output tensor should be known before calculation . These quantization methods need pretrained models , and can not accelerate the training process . 2.2 QUANTIZE-AWARE TRAINING . Quantize-aware training considered quantization effects in the training process . Some methods trained an ultra low-bit network like binary ( Rastegari et al. , 2016 ) or ternary ( Li et al. , 2016 ) networks , with a layer-wise scaling factor . Despite that the follow-up studies ( Liu et al. , 2020 ; Qin et al. , 2019 ) have been proposing training techniques to improve the performance of binary networks , the extremely low bit-width still causes notable performance degradation . Other methods sought to retain the accuracy with relatively higher precision , such as 8-bit ( Jacob et al. , 2017a ) . Gysel et al . ( 2018 ) developed a GPU-based training framework to get dynamic fixed-point models . These methods focus on accelerating the inference process and the training process is still using floating-point operations . 2.3 LOW-BIT TRAINING . To accelerate the training process , studies have been focusing on design a better floating-point data format . Dillon et al . ( 2017 ) proposed a novel 16-bit floating-point format that is more suitable for CNN training , while Köster et al . ( 2017 ) proposed the Flexpoint that contains 16-bit mantissa and 5-bit tensor-shared exponent ( scale ) , which is similar to the dynamic fixed-point format proposed by Gysel et al . ( 2018 ) . Recently , 8-bit floating-point ( Wang et al. , 2018 ; Sun et al. , 2019 ) was used with chunk-based accumulation and hybrid format to solve swamping . ...... Tensor Scale Group 0 Scale s Exp # Man # … … … s Exp # Man # Exp $ Man $ Exp % M Group 0 Group 1 Scale Exp % M Group 1 Group N Scale Exp % M Group N s Exp # Man # … … … s Exp # Man # s Exp # Man # … … … s Exp # Man # Figure 1 : Illustration of the multi-level scaling ( MLS ) low-bit tensor data format . Some studies used fixed-point in both the forward and backward processes ( Zhou et al. , 2016 ) .Wu et al . ( 2018 ) ; Yang et al . ( 2020 ) implemented a full-integer training framework for integer-arithmetic machines . However , their methods caused notable accuracy degradation . Banner et al . ( 2018a ) used 8-bit and 16-bit quantization based on integer arithmetic ( Jacob et al. , 2017b ) to achieve a comparable accuracy with the full-precision baseline . But it ’ s not very suitable for training as we discussed earlier . These methods reduced both the training and inference costs . In this paper , we seek to strike a better trade-off between accuracy and bit-width . 3 MULIT-LEVEL SCALING LOW-BIT TENSOR FORMAT . In this paper , we denote the filters and feature map of the convolution operation as weight ( W ) and activation ( A ) , respectively . In the back-propagation , the gradients of feature map and weights are denoted as error ( E ) and gradient ( G ) , respectively . 3.1 MAPPING FORMULA OF THE QUANTIZATION SCHEME . In quantized CNNs , floating-point values are quantized to use the fixed-point representation . In a commonly used scheme ( Jacob et al. , 2017b ) , the mapping function is float = scale× ( Fix+Bias ) , in which scale and Bias are shared in one tensor . However , since data distribution changes over time during training , one can not simplify the Bias calculation as Jacob et al . ( 2017b ) did . Thus , we adopt an unbiased quantization scheme , and extend the scaling factor to three levels for better representation ability . The mapping formula of our quantization scheme is X [ i , j , k , l ] = Ss [ i , j , k , l ] × St × Sg [ i , j ] × X̄ [ i , j , k , l ] ( 1 ) where [ · ] denotes the indexing operation , Ss is a sign tensor , St is a tensor-wise scaling factor shared in one tensor , and Sg is a group-wise scaling factor shared in one group , which is a structured subset of the tensor . Our paper considers three grouping dimensions : 1 ) grouping by the 1-st dimension of tensor , 2 ) the 2-nd dimension of tensor , or 3 ) the 1-st and the 2-nd dimensions simultaneously . St , Sg , and X̄ use the same format , which we refer to as 〈E , M〉 , a customized floating-point format with E-bit exponent and M-bit mantissa ( no sign bit ) . A value in the format 〈E , M〉 is float = I2F ( Man , Exp ) = Frac× 2−Exp = ( 1 + Man 2M ) × 2−Exp ( 2 ) where Man and Exp are the M-bit mantissa and E-bit exponent , and Frac ∈ [ 1 , 2 ) is a fraction . 3.2 DETAILS ON THE SCALING FACTORS . The first level tensor-wise scaling factor St is set to be an ordinary floating-point number ( 〈Et , Mt〉 = 〈8 , 23〉 ) , which is the same as unquantized data in training to retain the precision as much as possible . Considering the actual hardware implementation cost , there are some restrictions on the second level group-wise scaling factor Sg . Since calculation results using different tensor groups need to be aggregated , using Sg in an ordinary floating-point format will make more expensive conversions and operations necessary in the hardware implementation . We proposed two hardware-friendly group-wise scaling scheme , whose formats can be denoted as 〈Eg , 0〉 , and 〈Eg , 1〉 . The scaling factor in the 〈Eg , 0〉 format is simply a power of two , which can be implemented easily as shifting on the hardware . From Eq . 2 , a Sg = I2F ( Mang , Expg ) value in the 〈Eg , 1〉 format can be written as Sg = ( 1 + Mang 2 ) × 2−Expg = { 2−Expg + 2−Expg−1 Mang = 1 2−Expg Mang = 0 ( 3 ) which is a sum of two shifting , and can be implemented with small hardware overhead . The third level scaling factor Sx = I2F ( 0 , Expx ) = 2−Expx is the element-wise exponent in X̄ = Sx ( 1 + Manx 2 ) , and we can see that the elements of X̄ in Eq . 1 are in a 〈Ex , Mx〉 format . The specific values of Ex and Mx determine the type and the cost of the basic multiplication and accumulation ( MAC ) operation , which will be discussed later in Sec . 5.2 . 4 LOW-BIT TRAINING FRAMEWORK OF CNN . As shown in Fig . 2 , the convolution operation ( Conv ) is followed by batch normalization ( BN ) , nonlinear operations ( e.g . ReLU , pooling ) . Since Convs account for the majority computational cost , we apply quantization right before Convs in the training process , including three types : Conv ( W , A ) , Conv ( W , E ) , and Conv ( A , E ) . Note that the output data of Convs is in floating-point format , and other operations operate on floating-point numbers . An iteration of this low-bit training process is summarized in Appendix Alg . 2 , in which the major differences from the vanilla training process are the dynamic quantizaiton procedure DynamicQuantization and the low-bit tensor convolution arithmetic LowbitConv .

Summary: This paper investigated the low-bit training problem and proposed a novel method, which can reduce the element-wise bit-width to simplify floating-point computations to nearly fixed-point. The major contribution can be summarized as follows: a multi-level scaling (MLS) tensor format, a dynamic quantization and the low-bit tensor convolution arithmetic. Experiments show the proposed method could strike a good trade-off between the accuracy and bit-width, while taking the hardware efficiency into consideration.