CSQ: Centered Symmetric Quantization for Extremely Low Bit Neural Networks

1 INTRODUCTION . Quantized neural networks ( QNNs ) ( Krishnamoorthi , 2018 ; Esser et al. , 2019 ; Lee et al. , 2021 ; Li et al. , 2021 ; Nagel et al. , 2020 ) can reduce both computational complexity and memory requirement quite effectively over the full-precision ( i.e. , floating-point ) version , and hence are commonly used for deployment . Recent state-of-the-art QNNs ( Esser et al. , 2019 ; Lee et al. , 2021 ; Li et al. , 2019 ) can achieve near-full-precision accuracy even at 3-bit for ImageNet classification . However , most of the existing quantization method target 3-bit or higher . At 2-bit there is not much room for optimization , even with non-uniform step-size quantization ( Li et al. , 2019 ) . While some previous work , e.g . ( Li et al. , 2019 ) , treats 2-bit as ternary ( { −1 , 0 , 1 } ) , it wastes one quantization level . But to make the most of the 2-bit precision , the conventional 2-bit quantizer allocates two vs. one quantization levels to the negative side , causing a severe imbalance . As an alternative to the conventional quantizer , we propose ( zero- ) centered symmetric quantizer ( CSQ ) , which is most effective at low precision ( 2∼3-bit ) though the method itself is applicable to any precision . Through a set of analyses and experiments we show that CSQ improves QNN performance over the conventional linear quantizer ( CLQ ) in quantization-aware training ( QAT ) .1 We note that the particular quantizer function we propose has been used in some previous work ( Choi et al. , 2018a ; Boo et al. , 2021 ; Gong et al. , 2019 ; Lee et al. , 2021 ; Chen et al. , 2021 ) . However , no previous work has proposed it explicitly nor provided any analysis on the effectiveness of such a scheme ( see Section 2.3 ) . Moreover , while low-precision CLQ can be efficiently implemented in hardware via multiple invocations of XNOR-popcount such as on BNN ( Binarized Neural Network ) hardware , doing the same for CSQ poses new challenges that have never been addressed . In this paper we make the following contributions . • We propose CSQ that uses perfectly symmetrical quantization levels with uniform step size . We provide analytical and empirical evidence showing that using CSQ for weight instead of CLQ improves performance of low-bit QNNs ( 2∼3-bit ) . 1We use CSQ ( and CLQ ) to refer to both a quantizer and a quantization method . • We propose a binary coding scheme and a mapping method for CSQ , which allows for an efficient hardware implementation of QNNs using bitwise operations on BNN hardware . 2 BACKGROUND AND RELATED WORK . 2.1 QUANTIZATION PRIMER . A quantizer is a function from a real number x to a discrete value or an integer xQ . To train a network while simulating the effect of quantizer , one often maps the discrete value xQ back to a real value x̃ . Instead of quantizing the entire range of inputs , clipping off the extreme values and only mapping the mid-range values uniformly is often more efficient . A generalized version of a quantizer used in Neural Networks can be defined as : x̃ = clip ( ⌊x s ⌉ , L , U ) · s ( 1 ) where L and U ( the lower and upper bound ) are the minimum and maximum integer values that xQ can take , and clip ( x , a , b ) = min ( max ( x , a ) , b ) . Quantization using the above quantizer in ( 1 ) is called uniform step-size quantization , also called linear quantization . We do not consider nonuniform step-size quantization in this paper . Based on the input range ( i.e. , whether input is signed vs. un-signed ) , quantizers are classified as symmetric vs. asymmetric quantizers . For the linear quantizer in ( 1 ) , symmetric and asymmetric quantizers can be defined as follows , where b is the number of bits used for representation . Symmetric quantizer ( for signed input ) : L = −2b−1 , U = 2b−1 − 1 ( 2 ) Asymmetric quantizer ( for un-signed input ) : L = 0 , U = 2b − 1 ( 3 ) From here on we will refer to quantization method defined by ( 1 ) and ( 2 ) as the Conventional Linear Quantizer ( CLQ ) . 2.2 PREVIOUS WORK ON DNN QUANTIZATION . Most of the previous works on DNN quantization ( Jung et al. , 2019 ; Choi et al. , 2018b ; Esser et al. , 2019 ) use essentially the same linear quantizer , but each work proposes a slightly different way to optimize quantizer functions . Often the differences come from different formulations of the quantizer function . For instance , QIL ( Jung et al. , 2019 ) optimizes quantization boundary , PACT ( Choi et al. , 2018b ) optimizes a clipping parameter to train the quantized activations , and LSQ ( Esser et al. , 2019 ) optimizes the step size.We are not aware of any previous work explicitly proposing a new set of quantization levels for uniform quantization . Though the formulation of a quantizer function varies , in the end they all assume the same integer hardware that expects 2 ’ s complement number representation . 2.3 SIMILAR APPROACHES . The idea of using a zero centered quantizer is also seen in ( Choi et al. , 2018a ; Boo et al. , 2021 ; Gong et al. , 2019 ; Lee et al. , 2021 ; Chen et al. , 2021 ) . All of these works present a novel training method for their quantizer . However , none of these works analyze zero-centered quantization or present it as a source of performance gain . Furthermore , to the best of our knowledge , no previous work has attempted to address the realization of zero-centered quantization on hardware . Realizing zerocentered quantization on hardware can be a challenging task , because it introduces new quantization points which can not be represented using 2 ’ s complement number representation . 3 CENTERED SYMMETRIC QUANTIZATION ( CSQ ) . 3.1 EXACT ZERO REPRESENTATION VS . PERFECT SYMMETRY . Any linear quantization to b-bit precision results in 2b quantization levels . Therefore , inclusion of zero among the quantization points , naturally results in an asymmetry in the number of positive and negative quantization levels . While exact representation of zero may be important due to common operations like zero padding ( Krishnamoorthi , 2018 ) , it is also true that the asymmetry among positive and negative quantization levels grows large as precision becomes low . Thus one of our aims in this paper is to explore the trade-off between exact zero representation and perfect symmetry in the context of weight quantization of neural networks . 3.2 ALTERNATIVES . From the perspective of quantization levels , one can consider the following quantization schemes ( see Table 1 ) . Conventional Linear Quantization ( CLQ ) uses the quantization levels represented by 2 ’ s complement , which is also used by most of the previous uniform step-size quantizers , e.g . ( 1 ) . Alternatively , the asymmetry between the positive and negative sides can be reversed ; i.e. , by setting L = −2b−1 + 1 and U = 2b−1 . Reduced Symmetric Quantization ( RSQ ) uses one less quantization level , thereby achieving both exact zero representation and perfect symmetry ; i.e. , L = −2b−1 +1 and U = 2b−1−1 . This scheme wastes one quantization level and is expected to result in inferior performance . Extended Symmetric Quantization ( ESQ ) , on the other hand , uses one more quantization level to achieve zero representation and symmetry ; i.e. , L = −2b−1 and U = 2b−1 . However , since ESQ requires more than b-bit precision , it is not feasible for practical deployment , but included here for comparison . Non-uniform Symmetric Quantization ( NSQ ) achieves both exact zero representation and perfect symmetry with exactly 2b quantization levels , but the step size is not uniform , leading to a completely different quantizer function and QAT methods . Since NSQ is non-uniform quantization , we do not consider it for the scope of this paper . Centered Symmetric Quantization ( CSQ ) stipulates uniform step size and perfect symmetry between the positive and negative sides , while compromising on the exact representation of zero . It can also be represented using integers only by scaling all quantization levels by 2 as shown in the last row of the table . For the remainder of the paper we focus on CSQ and CLQ , as they are the most practical . 3.3 QUANTIZER FUNCTION FOR CSQ . We define the quantizer for CSQ as follows : v̇ = ⌊v s + 0.5 ⌉ − 0.5 ( 4 ) v̄ = clip ( v̇ , −Q , Q ) ( 5 ) v̂ = v̄ × s ( 6 ) where v is any input value , s is the step size , and Q = 2b−1 − 0.5 with b being the quantization precision ( i.e. , the number of bits ) . As usual , b·e is the round operation , and clip ( x , a , b ) = min ( max ( x , a ) , b ) . Here v̄ is not an integer . Nevertheless , it is an exact value that can be represented by the b-bit CSQ format . Moreover , our proposed CSQ format permits efficient hardware and software realizations ( see Section 5 ) . Therefore v̄ represents the value that is computed by b-bit hardware . Finally , v̂ is the scaled-back version of v̄ , defined and used for the purpose of training . The proposed quantizer provides equal representation for the positive and negative sides of the input distribution . Figure 1a shows the 2-bit quantizer functions for conventional linear quantization and our CSQ . Figure 1b shares the gradient for step size which is a quantization parameter . Detailed training method is described in Appendix A . 4 ANALYSIS OF CSQ . 4.1 IMPROVED REPRESENTATIONAL CAPACITY OF CSQ . Even though both CSQ and CLQ have the same number of quantization levels , and therefore the same representational capacity on the operand level , multiplication result , or the product of weight and activation , may have different representational capacity depending on the choice of quantizer . We compare the representational capacity of the product of weight and activation , when using CSQ vs. CLQ for weight . For activation quantization we consider unsigned CLQ ( denoted by CLQu ) for unsigned activation , and either CSQ or signed CLQ ( denoted by CLQs ) for signed activation . We start by defining the range of quantization levels of each scheme : CLQu ∼ { 0 , 1 , · · · , 2b − 1 } ( 7 ) CLQs ∼ { −2b−1 , −2b−1 + 1 , · · · , 2b−1 − 1 } ( 8 ) CSQ ∼ { −2b−1 + 0.5 , −2b−1 + 1.5 , · · · , 2b−1 − 0.5 } ( 9 ) where b is the precision . It should be noted that CSQ can only be signed . Table 2 shows the range of weights , activations and their product for 2-bit precision using different quantization methods . Table 3 shows that using CSQ for weight quantization almost always increases the representational capacity of quantized multiplication . The only exception is 2-bit precision for signed activation where it remains the same . Given that improving representational capacity is crucial for increasing the overall performance of QNNs ( Liu et al. , 2018 ) , our analysis suggests that CSQ has a definite advantage over CLQ for quantizing weights . Furthermore , the distribution of product resembles non-uniform quantization which provides higher accuracy . For more details refer to Appendix C. In the case of signed activation , our result in Table 3 suggests that using CLQs for activation and CSQ for weight provides the highest representational capacity .

Summary: This paper presents a centered symmetric quantizer (CSQ) that can map floating point tensors into zero-centered quantized integer values. The CSQ method works well for ultra-low bit quantization such as 2-bit. The authors also propose a binary coding method to run efficiently on hardware.