F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization

1 INTRODUCTION . Real-time inference on resource-constrained and efficiency-demanding platforms has long been desired and extensively studied in the last decades , resulting in significant improvement on the trade-off between efficiency and accuracy ( Han et al. , 2015 ; Liu et al. , 2018 ; Mei et al. , 2019 ; Tanaka et al. , 2020 ; Ma et al. , 2020 ; Mishra et al. , 2020 ; Liang et al. , 2021 ; Jin et al. , 2021 ; Liu et al. , 2021 ) . As a model compression technique , quantization is promising compared to other methods , such as network pruning ( Tanaka et al. , 2020 ; Li et al. , 2021 ; Ma et al. , 2020 ; 2021a ; Yuan et al. , 2021 ) and slimming ( Liu et al. , 2017 ; 2018 ) , as it achieves a large compression ratio ( Krishnamoorthi , 2018 ; Nagel et al. , 2021 ) and is computationally beneficial for integer-only hardware . The latter one is especially important because many hardwares ( e.g. , most brands of DSPs ( Ho , 2015 ; QCOM , 2019 ) ) only support integer or fixed-point arithmetic for accelerated implementation and can not deploy models with floating-point operations . However , the drop in performance , such as classification accuracy , caused by quantization errors , restricts wide applications of such methods ( Zhu et al. , 2016 ) . To address this challenge , many approaches have been proposed , which can be categorized into simulated quantization , integer-only quantization , and fixed-point quantization ( Gholami et al. , 2021 ) . Fig . 1 shows a comparison between these implementations . For simulated quantization , previous works propose to use trainable clipping-levels ( Choi et al. , 2018 ) , together with scaling techniques on activations ( Jin et al. , 2020b ) and/or gradients ( Esser et al. , 2019 ) , to facilitate training for the quantized models . However , some operations in these works , such as batch normalization ( BN ) , ∗Work done during an internship at Snap Inc. Code is available at https : //github.com/snap-research/F8Net . Published as a conference paper at ICLR 2022 Int32 Multiplication ( Int32 ) Int32 Multiplication ( Int8 ) Int32 Accumulation ( Int32 ) Bit Shifting ( Int32 ) Output ( Int8 ) Weight ( Int8 ) Input ( Int8 ) Int32 Multiplication ( Int8 ) Accumulation ( Int32 ) Int32 Bit Shifting ( Int32 ) Output ( Int8 ) Weight ( Int8 ) Input ( Int8 ) Int32 Multiplication ( Int32 ) Int32 Multiplication ( Int8 ) Int32 Accumulation ( Int32 ) Bit Shift ng ( Int32 ) Output ( Int8 ) Weight ( Int8 ) Input ( Int8 ) Int32 Multiplication ( Int8 ) Accumulation ( Int32 ) Int32 Bit Shift ng ( Int32 ) Output ( Int8 ) Weight ( Int8 ) Input ( Int8 ) Int32 Accumulation ( Int32 ) Requantization Output ( Int8 ) Int32 Multiplication ( Int8 ) Dequantization FP32 Weight ( Int8 ) Input ( Int8 ) FP32 Multiplication ( FP32 ) Output ( FP32 ) Accumulation ( FP32 ) FP32 Weight ( FP32 ) Input ( FP32 ) Int32 Accumulation ( Int32 ) Requantization Output ( Int8 ) Int32 Multiplication ( Int8 ) Dequantization FP32 Weight ( Int8 ) Input ( Int8 ) FP32 Multiplication ( FP32 ) Output ( FP32 ) Accumulation ( FP32 ) FP32 Weight ( FP32 ) Input ( FP32 ) are conducted with full-precision to stabilize training ( Jin et al. , 2020b ; Esser et al. , 2019 ) , limiting the practical application of integer-only hardware . Meanwhile , integer-only quantization , where the model infere ce can be implemented with integer multiplication , addition , and bit shifting , has shown significant progress in recent studies ( Jacob et al. , 2018 ; Yao et al. , 2021 ; Kim et al. , 2021 ) . Albeit floating-point operations are removed to enable models running on devices with limited support of operation types , INT32 multiplication is still required for these methods . On the other hand , fixed-point quantization , which also applies low-precision logic for arithmetic , does not require INT32 multiplication or integer division . For example , to replace multiplication by bit shifting , Jain et al . ( 2019 ) utilize trainable power-of-2 scale factors to quantize the model . In this work , we adopt fixed-point quantization . Our work differs from previous efforts ( Jain et al. , 2019 ) in three major aspects . First , to determine the minimum error quantization threshold , we conduct statistical analysis on fixed-point numbers . Second , we unify parameterized clipping activation ( PACT ) and fixed-point arithmetic to achieve high performance and high efficiency . Third , we discuss and propose quantization fine-tuning methods for different models . We dub our method as F8Net , as it consists in only Fixed-point 8-bit multiplication employed for Network quantization . We thoroughly study the problem with fixed-point numbers , where only INT8 multiplication is involved , without any INT32 multiplication , neither floating-point nor fixed-point types . Throughout this paper we focus on 8-bit quantization , the most widely supported case for different devices and is typically sufficient for efficiency and performance requirements . Our contribution can be elaborated as follows . • We show 8-bit fixed-point number is able to represent a wide range of values with negligible relative error , once the format is properly chosen ( see Fig . 3 and Fig . 4 ) . This critical characteristic enables fixed-point numbers a much stronger representative capability than integer values . • We propose a method to determine the fixed-point format , also known as fractional length , for weights and activations using their variance . This is achieved by analyzing the statistical behaviors of fixed-point values of different formats , especially those quantized from random variables with normal distribution of different variances . The analysis reveals the relationship between relative quantization error and variance , which further helps us build an approximated formula to determine the fractional length from the variance . • We develop a novel training algorithm for fixed-point models by unifying fixed-point quantization and PACT ( Choi et al. , 2018 ) . Besides , we show the impact of fractional length sharing for residual blocks , which is also important to obtain good performance for quantized models . • We validate our approach for various models , including MobileNet V1/V2 and ResNet18/50 on ImageNet for image classification , and demonstrate better performance than existing methods that resort to 32-bit multiplication . We also integrate the recent proposed fine-tuning method to train quantized models from pre-trained full-precision models with ours for further verification . 2 RELATED WORK . Quantization is one of the most widely-used techniques for neural network compression ( Courbariaux et al. , 2015 ; Han et al. , 2015 ; Zhu et al. , 2016 ; Zhou et al. , 2016 ; 2017 ; Mishra et al. , 2017 ; Park et al. , 2017 ; Banner et al. , 2018 ) , with two types of training strategies : Post-Training Quantization directly quantizes a pre-trained full-precision model ( He & Cheng , 2018 ; Nagel et al. , 2019 ; Fang et al. , 2020a ; b ; Garg et al. , 2021 ) ; Quantization-Aware Training uses training data to optimize quantized models for better performance ( Gysel et al. , 2018 ; Esser et al. , 2019 ; Hubara et al. , 2020 ; Tailor et al. , 2020 ) . In this work , we focus on the latter one , which is explored in several directions . One area uses uniform-precision quantization where the model shares the same precision ( Zhou et al. , 2018 ; Wang et al. , 2018 ; Choukroun et al. , 2019 ; Gong et al. , 2019 ; Langroudi et al. , 2019 ; Jin et al. , 2020a ; Bhalgat et al. , 2020 ; Chen et al. , 2020 ; Yang et al. , 2020 ; Darvish Rouhani et al. , 2020 ; Oh et al. , 2021 ) . Another direction studies mixed-precision that determines bit-width for each layer through search algorithms , aiming at better accuracy-efficiency trade-off ( Dong et al. , 2019 ; Wang et al. , 2019 ; Habi et al. , 2020 ; Fu et al. , 2020 ; 2021 ; Yang & Jin , 2020 ; Zhao et al. , 2021a ; b ; Ma et al. , 2021b ) . There is also binarization network , which only applies 1-bit ( Rastegari et al. , 2016 ; Hubara et al. , 2016 ; Cai et al. , 2017 ; Bulat et al. , 2020 ; Guo et al. , 2021 ) . Despite the fact that quantization helps reduce energy consumption and inference latency , it is usually accompanied by performance degradation . To alleviate this problem , several methods are proposed . One type of effort focuses on simulated quantization . The strategy is to leave some operations , e.g. , BN , in full-precision for the stabilized training of quantized models ( Choi et al. , 2018 ; Esser et al. , 2019 ; Jin et al. , 2020b ) . Nevertheless , these methods limit the application of the quantized models on resource-demanding hardware , such as DSP , where full-precision arithmetic is not supported for accelerated computing ( QCOM , 2019 ; Ho , 2015 ) . To completely eliminate floating-point operations from the quantized model , integer-only quantization techniques emulate the full-precision multiplication by 32-bit integer multiplication followed by bit shifting ( Jacob et al. , 2018 ; Zhu et al. , 2020 ; Wu et al. , 2020 ; Yao et al. , 2021 ; Kim et al. , 2021 ) . However , the calculation of INT32 multiplication in these works requires one more operation , which results in extra energy and higher latency ( Gholami et al. , 2021 ) . In parallel , recent work ( Jain et al. , 2019 ) proposes to restrict all scaling factors as power-of-2 values for all weights and activations , which belongs to fixed-point quantization methods ( Lin et al. , 2016 ; Jain et al. , 2019 ; Kim & Kim , 2021 ; Mitschke et al. , 2019 ; Enderich et al. , 2019b ; Chen et al. , 2017 ; Enderich et al. , 2019a ; Zhang et al. , 2020 ; Goyal et al. , 2021 ) . This enables the model to only incorporate INT8 or even INT4 multiplications , followed by INT32 bit shifting . However , there still a lack of a thorough study of the benefits of using fixed-point arithmetic . Also , the power-of-2 scaling factors are directly determined from the training data without theoretical analysis and guidance . In this work , we give an extensive analysis , especially on the potential and theoretical principle of using fixed-point values for quantized models , and demonstrate that with proper analysis and design , a model quantized with only INT8 multiplication involved is able to achieve comparable and even better performance to the integer-only methods implemented with INT32 multiplication . 3 ANALYSIS OF FIXED-POINT REPRESENTATION . In this section , we first introduce the fixed-point multiplication ( Smith et al. , 1997 ; Tan & Jiang , 2018 ) and analyze the distribution of weight from different layers in a well-trained full-precision model ( Sec . 3.1 ) . We then investigate the statistical property of fixed-point numbers , and demonstrate the potential of approximating full-precision values by 8-bit fixed-point numbers with different formats ( Sec . 3.2 ) . After that , we study the relationship between standard deviation of random variables and the optimal fixed-point format with the smallest quantization error . Finally , we derive an approximated formula relating the standard deviation and fixed-point format , which is verified empirically and employed in our final algorithms ( Sec . 3.3 ) .

Summary: The paper proposes a low-precision DNN inference models with 8-bit fixed point. To realize the number of fraction bits, the author uses the variance of DNN parameters and combines it with PACT approach in QAT. The new approach is evaluated in various neural networks such as MobileNet V1/V2 and ResNet18/50 on ImageNet for image classification and the result are mostly par with the state of the art approaches.