Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks

We propose Additive Powers-of-Two ( APoT ) quantization , an efficient nonuniform quantization scheme for the bell-shaped and long-tailed distribution of weights and activations in neural networks . By constraining all quantization levels as the sum of Powers-of-Two terms , APoT quantization enjoys high computational efficiency and a good match with the distribution of weights . A simple reparameterization of the clipping function is applied to generate a better-defined gradient for learning the clipping threshold . Moreover , weight normalization is presented to refine the distribution of weights to make the training more stable and consistent . Experimental results show that our proposed method outperforms state-of-the-art methods , and is even competitive with the full-precision models , demonstrating the effectiveness of our proposed APoT quantization . For example , our 4-bit quantized ResNet-50 on ImageNet achieves 76.6 % top-1 accuracy without bells and whistles ; meanwhile , our model is capable to decrease 22 % computational cost compared with the uniformly quantized counterpart . 1 1 INTRODUCTION . Deep Neural Networks ( DNNs ) have made a significant improvement for various real-world applications . However , the huge memory and computational cost impede the mass deployment of DNNs , e.g. , on resource-constrained devices . To reduce memory footprint and computational burden , several model compression methods such as quantization ( Zhou et al. , 2016 ) , pruning ( Han et al. , 2015 ) and low-rank decomposition ( Denil et al. , 2013 ) have been widely explored . In this paper , we focus on the neural network quantization for efficient inference . Two operations are involved in the quantization process , namely clipping and projection . The clipping operation sets a full precision number to the range boundary if it is outside of the range ; the projection operation maps each number ( after clipping ) into a predefined quantization level ( a fixed number ) . We can see that both operations incur information loss . A good quantization method should resolve the two following questions/challenges , which correspond to two contradictions respectively . How to determine the optimal clipping threshold to balance clipping range and projection resolution ? The resolution indicates the interval between two quantization levels ; the smaller the interval , the higher the resolution . The first contradiction is that given a fixed number of bits to represent weights , the range and resolution are inversely proportional . For example , a larger range can clip fewer weights ; however , the resolution becomes lower and thus damage the projection . Note that slipshod clipping of outliers can jeopardize the network a lot ( Zhao et al. , 2019 ) although they may only take 1-2 % of all weights in one layer . Previous works have tried either pre-defined ( Cai et al. , 2017 ; Zhou et al. , 2016 ) or trainable ( Choi et al. , 2018b ) clipping thresholds , but how to find the optimal threshold during training automatically is still not resolved . ∗Equal Contribution . Y. L. completed this work during his internship at NUS . 1Code is available at https : //github.com/yhhhli/APoT_Quantization . How to design quantization levels with consideration for both the computational efficiency and the distribution of weights ? Most of the existing quantization approaches ( Cai et al. , 2017 ; Gong et al. , 2019 ) use uniform quantization although non-uniform quantization can usually achieve better accuracy ( Zhu et al. , 2016 ) . The reason is that projection against uniform quantization levels are much more hardware-friendly ( Zhou et al. , 2016 ) . However , empirical study ( Han et al. , 2015 ) has shown that weights in a layer of DNN follow a bell-shaped and long-tailed distribution instead of a uniform distribution ( as shown in the right figure ) . In other words , a fair percentage of weights concentrate around the mean ( peak area ) ; and a few weights are of relatively high magnitude and out of the quantization range ( called outliers ) . Such distribution also exists in activations ( Miyashita et al. , 2016 ) . The second contradiction is : considering the bell-shaped distribution of weight , it is well-motivated to assign higher resolution ( i.e . smaller quantization interval ) around the mean ; however , such non-uniform quantization levels will introduce high computational overhead . Powers-of-Two quantization levels ( Miyashita et al. , 2016 ; Zhou et al. , 2017 ) are then proposed because of its cheap multiplication implemented by shift operations on hardware , and super high resolution around the mean . However , the vanilla powers-of-two quantization method only increases the resolution near the mean and ignores other regions at all when the bit-width is increased . Consequently , it assigns inordinate quantization levels for a tiny range around the mean . To this end , we propose additive Powers-of-Two ( APoT ) quantization to resolve these two contradictions , our contribution can be listed as follows : 1 . We introduce the APoT quantization scheme for the weights and activations of DNNs . APoT is a non-uniform quantization scheme , in which the quantization levels is a sum of several PoT terms and can adapt well to the bell-shaped distribution of weights . APoT quantization enjoys an approximate 2× multiplication speed-up compared with uniform quantization on both generic and specific hardware . 2 . We propose a Reparameterized Clipping Function ( RCF ) that can compute a more accurate gradient for the clipping threshold and thus facilitate the optimization of the clipping threshold . We also introduce weight normalization for neural network quantization . Normalized weights in the forward pass are more stable and consistent for clipping and projection . 3 . Experimental results show that our proposed method outperforms state-of-the-art methods , and is even competitive with the full-precision implementation with higher computational efficiency . Specifically , our 4-bit quantized ResNet-50 on ImageNet achieve 76.6 % Top-1 and 93.1 % Top-5 accuracy . Compared with uniform quantization , our method can decrease 22 % computational cost , demonstrating the proposed algorithm is hardware-friendly . 2 METHODOLOGY . 2.1 PRELIMINARIES . Suppose kernels in a convolutional layer are represented by a 4D tensor W ∈ RCout×Cin×K×K , where Cout and Cin are the number of output and input channels respectively , and K is the kernel size . We denote the quantization of the weights as Ŵ = ΠQ ( α , b ) bW , αe , ( 1 ) where α is the clipping threshold and the clipping function b· , αe clips weights into [ −α , α ] . After clipping , each element ofW is projected by Π ( · ) onto the quantization levels . We denote Q ( α , b ) for a set of quantization levels , where b is the bit-width . For uniform quantization , the quantization levels are defined as Qu ( α , b ) = α× { 0 , ±1 2b−1 − 1 , ±2 2b−1 − 1 , ±3 2b−1 − 1 , . . . , ±1 } . ( 2 ) For every floating-point number , uniform quantization maps it to a b-bit fixed-point representation ( quantization levels ) in Qu ( α , b ) . Note that α is stored separately as a full-precision floating-point number for each whole W . Convolution is done against the quantization levels first and the results are then multiplied by α. Arithmetical computation , e.g. , convolution , can be implemented using low-precision fixed point operations on hardware , which are substantially cheaper than their floating-point contradictory ( Goldberg , 1991 ) . Nevertheless , uniform quantization does not match the distribution of weights ( and activations ) , which is typically bell-shaped ( Han et al. , 2015 ) . A straightforward solution is to assign more quantization levels ( higher resolution ) for the peak of the distribution and fewer levels ( lower resolution ) for the tails . However , it is difficult to implement the arithmetical operations for the non-uniform quantization levels efficiently . 2.2 ADDITIVE POWERS-OF-TWO QUANTIZATION . To solve the contradiction between non-uniform resolution and hardware efficiency , Powers-ofTwo ( PoT ) quantization ( Miyashita et al. , 2016 ; Zhou et al. , 2017 ) is proposed by constraining quantization levels to be powers-of-two values or zero , i.e. , Qp ( α , b ) = α× { 0 , ±2−2 b−1+1 , ±2−2 b−1+2 , ... , ±2−1 , ±1 } . ( 3 ) Apparently , as a non-uniform quantizer , PoT has a higher resolution for the value range with denser weights because of its exponential property . Furthermore , multiplication between a Powers-of-two number 2x and the other operand r can be implemented by bit-wise shift instead of bulky digital multipliers , i.e. , 2xr = r if x = 0 r < < x if x > 0 r > > x if x < 0 , ( 4 ) where > > denotes the right shift operation and is computationally cheap , which only takes 1 clock cycle in modern CPU architectures . However , we find that PoT quantization does not benefit from more bits . Assume α is 1 , as shown in Equation ( 3 ) , when we increase the bit-width from b to b+ 1 , the interval [ −2−2b−1+1 , 2−2b−1+1 ] will be split into 2b−1 − 1 sub-intervals , whereas all other intervals remain unchanged . In other words , by increasing the bit-width , the resolution will increase only for [ −2−2b−1+1 , 2−2b−1+1 ] . We refer this phenomenon as the rigid resolution of PoT quantization . TakeQp ( 1 , 5 ) as an example , the two smallest positive levels are 2−15 and 2−14 , which is excessively fine-grained . In contrast , the two largest levels are 2−1 and 20 , whose interval is large enough to incur high projection error for weights between [ 2−1 , 20 ] , e.g. , 0.75 . The rigid resolution is demonstrated in Figure 2 ( b ) . When we change from from 3-bit to 4-bit , all new quantization levels concentrate around 0 and thus can not increase the model ’ s expressiveness effectively . To tackle the rigid resolution problem , we propose Additive Powers-of-Two ( APoT ) quantization . Without loss of generality , in this section , we only consider unsigned numbers for simplicity2 . In APoT quantization , each level is the sum of n PoT terms as shown below , Qa ( α , kn ) = γ × { n−1∑ i=0 pi } where pi ∈ { 0 , 1 2i , 1 2i+n , ... , 1 2i+ ( 2k−2 ) n } , ( 5 ) where γ is a scaling coefficient to make sure the maximum level in Qa is α. k is called the base bit-width , which is the bit-width for each additive term , and n is the number of additive terms . When the bit-width b and the base bit-width k is set , n can be calculated by n = bk . There are 2 kn = 2b levels in total . The number of additive terms in APoT quantization can increase with bit-width b , which provides a higher resolution for the non-uniform levels . We use b = 4 and k = 2 as an example to illustrate how APoT resolves the rigid resolution problem . For this example , we have p0 ∈ { 0 , 20 , 2−2 , 2−4 } , p1 ∈ { 0 , 2−1 , 2−3 , 2−5 } , γ = 2α/3 , and Qa ( α , kn ) = { γ × ( p0 + p1 ) } for all ( 2b = 16 ) combinations of p0 and p1 . First , we can see the smallest positive quantization level in Qa ( 1 , 4 ) is 2−4/3 . Compared with the original PoT levels , APoT allocates quantization levels prudently for the central area . Second , APoT generates 3 new quantization levels between 20 and 2−1 , to properly increase the resolution . In Figure 2 , the second row compares the 3 quantization methods using 4 bits for range [ 0 , 1 ] . APoT quantization has a reasonable distribution of quantization levels , with more levels in the peak area ( near 0 ) and relatively higher resolution than the vanilla PoT quantization at the tail ( near 1 ) . Relation to other quantization schemes . On the one hand , the fixed-point number representations used in the uniform quantization is a special case of APoT . When k = 1 in Equation ( 5 ) , the quantization levels is a sum of b PoT terms or 0 . In the fixed-point representations , each bit indicates one specific choice of the additive terms . On the other hand , when k = b , there is only one PoT term and Qa ( α , b ) becomes Qp ( α , b ) , i.e. , PoT quantization . We can conclude that when k decreases , APoT levels are decomposed into more PoT terms , and the distribution of levels becomes more uniform . Our experiments use k = 2 , which is an intermediate choice between the uniform case ( k = 1 ) and the vanilla PoT case ( k = b ) . Computation . Multiplication for fixed-point numbers can be implemented by shifting the multiplicand ( i.e. , the activations ) and adding the partial product . The n in Equation ( 5 ) denotes the number of additive PoT terms in the multiplier ( weights ) , and control the speed of computation . Since n = bk , either decreasing b or increasing k can accelerate the multiplication . Compared with uniform quantization ( k = 1 ) , our method ( k = 2 ) is approximately 2× faster in multiplication . As for the full precision α , it is a coefficient for all weights in a layer and can be multiplied only once after the multiply-accumulate operation is finished . Figure 3 shows the hardware accelerator , the weights buffer takes k-bit as a PoT term and shift-adds the activations . Generalizing to 2n+ 1 bits . When k = 2 , APoT quantization can only leverages 2n-bit width for quantization . To deal with 2n+ 1-bit quantization , we choose to add n+ 1 PoT terms , one of which 2To extend the solution for the signed number , we only need to add 1 more bit for the sign . only contains 2 levels . The formulation is given by Qa ( α , 2n+ 1 ) = γ × { n−1∑ i=0 pi + p̃ } where pi ∈ { 0 , 1 2i , 1 2i+n , 1 2i+2n+1 } , p̃ ∈ { 0 , 1 2i+2n } . ( 6 ) Take 3-bit APoT quantization as an example , every level is a sum of one p0 and one p̃ , where p0 ∈ { 0 , 2−1 , 2−2 , 2−4 } and p̃ ∈ { 0 , 2−3 } . The forward function is plotted in Figure 2 ( c ) .

Summary: The paper presents an approach based on power-two quantization to compress the weights of neural networks. The authors elaborate on a coding scheme that adds several quantized values with different log scales (typically 2^(-2k) and 2^(-2k+1) for varying k, for the sum of two log scales). More importantly, they emphasize on the importance of weight scaling and learning the clipping threshold. experiments are carried out with several versions of ResNet on CIFAR-10 and ImageNet, and the authors show better performances than competing baseline for a fixed quantization budget per float.