F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization

1 INTRODUCTION . Real-time inference on resource-constrained and efficiency-demanding platforms has long been desired and extensively studied in the last decades , resulting in significant improvement on the trade-off between efficiency and accuracy ( Han et al. , 2015 ; Liu et al. , 2018 ; Mei et al. , 2019 ; Tanaka et al. , 2020 ; Ma et al. , 2020 ; Mishra et al. , 2020 ; Liang et al. , 2021 ; Jin et al. , 2021 ; Liu et al. , 2021 ) . As a model compression technique , quantization is promising compared to other methods , such as network pruning ( Tanaka et al. , 2020 ; Li et al. , 2021 ; Ma et al. , 2020 ; 2021a ; Yuan et al. , 2021 ) and slimming ( Liu et al. , 2017 ; 2018 ) , as it achieves a large compression ratio ( Krishnamoorthi , 2018 ; Nagel et al. , 2021 ) and is computationally beneficial for integer-only hardware . The latter one is especially important because many hardwares ( e.g. , most brands of DSPs ( Ho , 2015 ; QCOM , 2019 ) ) only support integer or fixed-point arithmetic for accelerated implementation and can not deploy models with floating-point operations . However , the drop in performance , such as classification accuracy , caused by quantization errors , restricts wide applications of such methods ( Zhu et al. , 2016 ) . To address this challenge , many approaches have been proposed , which can be categorized into simulated quantization , integer-only quantization , and fixed-point quantization ( Gholami et al. , 2021 ) . Fig . 1 shows a comparison between these implementations . For simulated quantization , previous works propose to use trainable clipping-levels ( Choi et al. , 2018 ) , together with scaling techniques on activations ( Jin et al. , 2020b ) and/or gradients ( Esser et al. , 2019 ) , to facilitate training for the quantized models . However , some operations in these works , such as batch normalization ( BN ) , ∗Work done during an internship at Snap Inc. Code is available at https : //github.com/snap-research/F8Net . Published as a conference paper at ICLR 2022 Int32 Multiplication ( Int32 ) Int32 Multiplication ( Int8 ) Int32 Accumulation ( Int32 ) Bit Shifting ( Int32 ) Output ( Int8 ) Weight ( Int8 ) Input ( Int8 ) Int32 Multiplication ( Int8 ) Accumulation ( Int32 ) Int32 Bit Shifting ( Int32 ) Output ( Int8 ) Weight ( Int8 ) Input ( Int8 ) Int32 Multiplication ( Int32 ) Int32 Multiplication ( Int8 ) Int32 Accumulation ( Int32 ) Bit Shift ng ( Int32 ) Output ( Int8 ) Weight ( Int8 ) Input ( Int8 ) Int32 Multiplication ( Int8 ) Accumulation ( Int32 ) Int32 Bit Shift ng ( Int32 ) Output ( Int8 ) Weight ( Int8 ) Input ( Int8 ) Int32 Accumulation ( Int32 ) Requantization Output ( Int8 ) Int32 Multiplication ( Int8 ) Dequantization FP32 Weight ( Int8 ) Input ( Int8 ) FP32 Multiplication ( FP32 ) Output ( FP32 ) Accumulation ( FP32 ) FP32 Weight ( FP32 ) Input ( FP32 ) Int32 Accumulation ( Int32 ) Requantization Output ( Int8 ) Int32 Multiplication ( Int8 ) Dequantization FP32 Weight ( Int8 ) Input ( Int8 ) FP32 Multiplication ( FP32 ) Output ( FP32 ) Accumulation ( FP32 ) FP32 Weight ( FP32 ) Input ( FP32 ) are conducted with full-precision to stabilize training ( Jin et al. , 2020b ; Esser et al. , 2019 ) , limiting the practical application of integer-only hardware . Meanwhile , integer-only quantization , where the model infere ce can be implemented with integer multiplication , addition , and bit shifting , has shown significant progress in recent studies ( Jacob et al. , 2018 ; Yao et al. , 2021 ; Kim et al. , 2021 ) . Albeit floating-point operations are removed to enable models running on devices with limited support of operation types , INT32 multiplication is still required for these methods . On the other hand , fixed-point quantization , which also applies low-precision logic for arithmetic , does not require INT32 multiplication or integer division . For example , to replace multiplication by bit shifting , Jain et al . ( 2019 ) utilize trainable power-of-2 scale factors to quantize the model . In this work , we adopt fixed-point quantization . Our work differs from previous efforts ( Jain et al. , 2019 ) in three major aspects . First , to determine the minimum error quantization threshold , we conduct statistical analysis on fixed-point numbers . Second , we unify parameterized clipping activation ( PACT ) and fixed-point arithmetic to achieve high performance and high efficiency . Third , we discuss and propose quantization fine-tuning methods for different models . We dub our method as F8Net , as it consists in only Fixed-point 8-bit multiplication employed for Network quantization . We thoroughly study the problem with fixed-point numbers , where only INT8 multiplication is involved , without any INT32 multiplication , neither floating-point nor fixed-point types . Throughout this paper we focus on 8-bit quantization , the most widely supported case for different devices and is typically sufficient for efficiency and performance requirements . Our contribution can be elaborated as follows . • We show 8-bit fixed-point number is able to represent a wide range of values with negligible relative error , once the format is properly chosen ( see Fig . 3 and Fig . 4 ) . This critical characteristic enables fixed-point numbers a much stronger representative capability than integer values . • We propose a method to determine the fixed-point format , also known as fractional length , for weights and activations using their variance . This is achieved by analyzing the statistical behaviors of fixed-point values of different formats , especially those quantized from random variables with normal distribution of different variances . The analysis reveals the relationship between relative quantization error and variance , which further helps us build an approximated formula to determine the fractional length from the variance . • We develop a novel training algorithm for fixed-point models by unifying fixed-point quantization and PACT ( Choi et al. , 2018 ) . Besides , we show the impact of fractional length sharing for residual blocks , which is also important to obtain good performance for quantized models . • We validate our approach for various models , including MobileNet V1/V2 and ResNet18/50 on ImageNet for image classification , and demonstrate better performance than existing methods that resort to 32-bit multiplication . We also integrate the recent proposed fine-tuning method to train quantized models from pre-trained full-precision models with ours for further verification . 2 RELATED WORK . Quantization is one of the most widely-used techniques for neural network compression ( Courbariaux et al. , 2015 ; Han et al. , 2015 ; Zhu et al. , 2016 ; Zhou et al. , 2016 ; 2017 ; Mishra et al. , 2017 ; Park et al. , 2017 ; Banner et al. , 2018 ) , with two types of training strategies : Post-Training Quantization directly quantizes a pre-trained full-precision model ( He & Cheng , 2018 ; Nagel et al. , 2019 ; Fang et al. , 2020a ; b ; Garg et al. , 2021 ) ; Quantization-Aware Training uses training data to optimize quantized models for better performance ( Gysel et al. , 2018 ; Esser et al. , 2019 ; Hubara et al. , 2020 ; Tailor et al. , 2020 ) . In this work , we focus on the latter one , which is explored in several directions . One area uses uniform-precision quantization where the model shares the same precision ( Zhou et al. , 2018 ; Wang et al. , 2018 ; Choukroun et al. , 2019 ; Gong et al. , 2019 ; Langroudi et al. , 2019 ; Jin et al. , 2020a ; Bhalgat et al. , 2020 ; Chen et al. , 2020 ; Yang et al. , 2020 ; Darvish Rouhani et al. , 2020 ; Oh et al. , 2021 ) . Another direction studies mixed-precision that determines bit-width for each layer through search algorithms , aiming at better accuracy-efficiency trade-off ( Dong et al. , 2019 ; Wang et al. , 2019 ; Habi et al. , 2020 ; Fu et al. , 2020 ; 2021 ; Yang & Jin , 2020 ; Zhao et al. , 2021a ; b ; Ma et al. , 2021b ) . There is also binarization network , which only applies 1-bit ( Rastegari et al. , 2016 ; Hubara et al. , 2016 ; Cai et al. , 2017 ; Bulat et al. , 2020 ; Guo et al. , 2021 ) . Despite the fact that quantization helps reduce energy consumption and inference latency , it is usually accompanied by performance degradation . To alleviate this problem , several methods are proposed . One type of effort focuses on simulated quantization . The strategy is to leave some operations , e.g. , BN , in full-precision for the stabilized training of quantized models ( Choi et al. , 2018 ; Esser et al. , 2019 ; Jin et al. , 2020b ) . Nevertheless , these methods limit the application of the quantized models on resource-demanding hardware , such as DSP , where full-precision arithmetic is not supported for accelerated computing ( QCOM , 2019 ; Ho , 2015 ) . To completely eliminate floating-point operations from the quantized model , integer-only quantization techniques emulate the full-precision multiplication by 32-bit integer multiplication followed by bit shifting ( Jacob et al. , 2018 ; Zhu et al. , 2020 ; Wu et al. , 2020 ; Yao et al. , 2021 ; Kim et al. , 2021 ) . However , the calculation of INT32 multiplication in these works requires one more operation , which results in extra energy and higher latency ( Gholami et al. , 2021 ) . In parallel , recent work ( Jain et al. , 2019 ) proposes to restrict all scaling factors as power-of-2 values for all weights and activations , which belongs to fixed-point quantization methods ( Lin et al. , 2016 ; Jain et al. , 2019 ; Kim & Kim , 2021 ; Mitschke et al. , 2019 ; Enderich et al. , 2019b ; Chen et al. , 2017 ; Enderich et al. , 2019a ; Zhang et al. , 2020 ; Goyal et al. , 2021 ) . This enables the model to only incorporate INT8 or even INT4 multiplications , followed by INT32 bit shifting . However , there still a lack of a thorough study of the benefits of using fixed-point arithmetic . Also , the power-of-2 scaling factors are directly determined from the training data without theoretical analysis and guidance . In this work , we give an extensive analysis , especially on the potential and theoretical principle of using fixed-point values for quantized models , and demonstrate that with proper analysis and design , a model quantized with only INT8 multiplication involved is able to achieve comparable and even better performance to the integer-only methods implemented with INT32 multiplication . 3 ANALYSIS OF FIXED-POINT REPRESENTATION . In this section , we first introduce the fixed-point multiplication ( Smith et al. , 1997 ; Tan & Jiang , 2018 ) and analyze the distribution of weight from different layers in a well-trained full-precision model ( Sec . 3.1 ) . We then investigate the statistical property of fixed-point numbers , and demonstrate the potential of approximating full-precision values by 8-bit fixed-point numbers with different formats ( Sec . 3.2 ) . After that , we study the relationship between standard deviation of random variables and the optimal fixed-point format with the smallest quantization error . Finally , we derive an approximated formula relating the standard deviation and fixed-point format , which is verified empirically and employed in our final algorithms ( Sec . 3.3 ) .

Summary: The paper analyzes the relationship between relative quantization errors and fixed-point formats for zero-centered normal distributions and finds a linear model which fits the best exponent length of the fixed-point data type given a standard deviation. These insights are then unified with parameterized clipping activation (PACT) to normalize incoming floating point data into the desired fixed point range. To handle the network with sole 8-bit multiplications a forward pass in floating precision is used to compute batch norm statistics for the main 8-bit forward/backward pass. Additional adjustments are made between successive layers and residual layers which rely on reusing some statistics of the previous layer.