CSQ: Centered Symmetric Quantization for Extremely Low Bit Neural Networks

1 INTRODUCTION . Quantized neural networks ( QNNs ) ( Krishnamoorthi , 2018 ; Esser et al. , 2019 ; Lee et al. , 2021 ; Li et al. , 2021 ; Nagel et al. , 2020 ) can reduce both computational complexity and memory requirement quite effectively over the full-precision ( i.e. , floating-point ) version , and hence are commonly used for deployment . Recent state-of-the-art QNNs ( Esser et al. , 2019 ; Lee et al. , 2021 ; Li et al. , 2019 ) can achieve near-full-precision accuracy even at 3-bit for ImageNet classification . However , most of the existing quantization method target 3-bit or higher . At 2-bit there is not much room for optimization , even with non-uniform step-size quantization ( Li et al. , 2019 ) . While some previous work , e.g . ( Li et al. , 2019 ) , treats 2-bit as ternary ( { −1 , 0 , 1 } ) , it wastes one quantization level . But to make the most of the 2-bit precision , the conventional 2-bit quantizer allocates two vs. one quantization levels to the negative side , causing a severe imbalance . As an alternative to the conventional quantizer , we propose ( zero- ) centered symmetric quantizer ( CSQ ) , which is most effective at low precision ( 2∼3-bit ) though the method itself is applicable to any precision . Through a set of analyses and experiments we show that CSQ improves QNN performance over the conventional linear quantizer ( CLQ ) in quantization-aware training ( QAT ) .1 We note that the particular quantizer function we propose has been used in some previous work ( Choi et al. , 2018a ; Boo et al. , 2021 ; Gong et al. , 2019 ; Lee et al. , 2021 ; Chen et al. , 2021 ) . However , no previous work has proposed it explicitly nor provided any analysis on the effectiveness of such a scheme ( see Section 2.3 ) . Moreover , while low-precision CLQ can be efficiently implemented in hardware via multiple invocations of XNOR-popcount such as on BNN ( Binarized Neural Network ) hardware , doing the same for CSQ poses new challenges that have never been addressed . In this paper we make the following contributions . • We propose CSQ that uses perfectly symmetrical quantization levels with uniform step size . We provide analytical and empirical evidence showing that using CSQ for weight instead of CLQ improves performance of low-bit QNNs ( 2∼3-bit ) . 1We use CSQ ( and CLQ ) to refer to both a quantizer and a quantization method . • We propose a binary coding scheme and a mapping method for CSQ , which allows for an efficient hardware implementation of QNNs using bitwise operations on BNN hardware . 2 BACKGROUND AND RELATED WORK . 2.1 QUANTIZATION PRIMER . A quantizer is a function from a real number x to a discrete value or an integer xQ . To train a network while simulating the effect of quantizer , one often maps the discrete value xQ back to a real value x̃ . Instead of quantizing the entire range of inputs , clipping off the extreme values and only mapping the mid-range values uniformly is often more efficient . A generalized version of a quantizer used in Neural Networks can be defined as : x̃ = clip ( ⌊x s ⌉ , L , U ) · s ( 1 ) where L and U ( the lower and upper bound ) are the minimum and maximum integer values that xQ can take , and clip ( x , a , b ) = min ( max ( x , a ) , b ) . Quantization using the above quantizer in ( 1 ) is called uniform step-size quantization , also called linear quantization . We do not consider nonuniform step-size quantization in this paper . Based on the input range ( i.e. , whether input is signed vs. un-signed ) , quantizers are classified as symmetric vs. asymmetric quantizers . For the linear quantizer in ( 1 ) , symmetric and asymmetric quantizers can be defined as follows , where b is the number of bits used for representation . Symmetric quantizer ( for signed input ) : L = −2b−1 , U = 2b−1 − 1 ( 2 ) Asymmetric quantizer ( for un-signed input ) : L = 0 , U = 2b − 1 ( 3 ) From here on we will refer to quantization method defined by ( 1 ) and ( 2 ) as the Conventional Linear Quantizer ( CLQ ) . 2.2 PREVIOUS WORK ON DNN QUANTIZATION . Most of the previous works on DNN quantization ( Jung et al. , 2019 ; Choi et al. , 2018b ; Esser et al. , 2019 ) use essentially the same linear quantizer , but each work proposes a slightly different way to optimize quantizer functions . Often the differences come from different formulations of the quantizer function . For instance , QIL ( Jung et al. , 2019 ) optimizes quantization boundary , PACT ( Choi et al. , 2018b ) optimizes a clipping parameter to train the quantized activations , and LSQ ( Esser et al. , 2019 ) optimizes the step size.We are not aware of any previous work explicitly proposing a new set of quantization levels for uniform quantization . Though the formulation of a quantizer function varies , in the end they all assume the same integer hardware that expects 2 ’ s complement number representation . 2.3 SIMILAR APPROACHES . The idea of using a zero centered quantizer is also seen in ( Choi et al. , 2018a ; Boo et al. , 2021 ; Gong et al. , 2019 ; Lee et al. , 2021 ; Chen et al. , 2021 ) . All of these works present a novel training method for their quantizer . However , none of these works analyze zero-centered quantization or present it as a source of performance gain . Furthermore , to the best of our knowledge , no previous work has attempted to address the realization of zero-centered quantization on hardware . Realizing zerocentered quantization on hardware can be a challenging task , because it introduces new quantization points which can not be represented using 2 ’ s complement number representation . 3 CENTERED SYMMETRIC QUANTIZATION ( CSQ ) . 3.1 EXACT ZERO REPRESENTATION VS . PERFECT SYMMETRY . Any linear quantization to b-bit precision results in 2b quantization levels . Therefore , inclusion of zero among the quantization points , naturally results in an asymmetry in the number of positive and negative quantization levels . While exact representation of zero may be important due to common operations like zero padding ( Krishnamoorthi , 2018 ) , it is also true that the asymmetry among positive and negative quantization levels grows large as precision becomes low . Thus one of our aims in this paper is to explore the trade-off between exact zero representation and perfect symmetry in the context of weight quantization of neural networks . 3.2 ALTERNATIVES . From the perspective of quantization levels , one can consider the following quantization schemes ( see Table 1 ) . Conventional Linear Quantization ( CLQ ) uses the quantization levels represented by 2 ’ s complement , which is also used by most of the previous uniform step-size quantizers , e.g . ( 1 ) . Alternatively , the asymmetry between the positive and negative sides can be reversed ; i.e. , by setting L = −2b−1 + 1 and U = 2b−1 . Reduced Symmetric Quantization ( RSQ ) uses one less quantization level , thereby achieving both exact zero representation and perfect symmetry ; i.e. , L = −2b−1 +1 and U = 2b−1−1 . This scheme wastes one quantization level and is expected to result in inferior performance . Extended Symmetric Quantization ( ESQ ) , on the other hand , uses one more quantization level to achieve zero representation and symmetry ; i.e. , L = −2b−1 and U = 2b−1 . However , since ESQ requires more than b-bit precision , it is not feasible for practical deployment , but included here for comparison . Non-uniform Symmetric Quantization ( NSQ ) achieves both exact zero representation and perfect symmetry with exactly 2b quantization levels , but the step size is not uniform , leading to a completely different quantizer function and QAT methods . Since NSQ is non-uniform quantization , we do not consider it for the scope of this paper . Centered Symmetric Quantization ( CSQ ) stipulates uniform step size and perfect symmetry between the positive and negative sides , while compromising on the exact representation of zero . It can also be represented using integers only by scaling all quantization levels by 2 as shown in the last row of the table . For the remainder of the paper we focus on CSQ and CLQ , as they are the most practical . 3.3 QUANTIZER FUNCTION FOR CSQ . We define the quantizer for CSQ as follows : v̇ = ⌊v s + 0.5 ⌉ − 0.5 ( 4 ) v̄ = clip ( v̇ , −Q , Q ) ( 5 ) v̂ = v̄ × s ( 6 ) where v is any input value , s is the step size , and Q = 2b−1 − 0.5 with b being the quantization precision ( i.e. , the number of bits ) . As usual , b·e is the round operation , and clip ( x , a , b ) = min ( max ( x , a ) , b ) . Here v̄ is not an integer . Nevertheless , it is an exact value that can be represented by the b-bit CSQ format . Moreover , our proposed CSQ format permits efficient hardware and software realizations ( see Section 5 ) . Therefore v̄ represents the value that is computed by b-bit hardware . Finally , v̂ is the scaled-back version of v̄ , defined and used for the purpose of training . The proposed quantizer provides equal representation for the positive and negative sides of the input distribution . Figure 1a shows the 2-bit quantizer functions for conventional linear quantization and our CSQ . Figure 1b shares the gradient for step size which is a quantization parameter . Detailed training method is described in Appendix A . 4 ANALYSIS OF CSQ . 4.1 IMPROVED REPRESENTATIONAL CAPACITY OF CSQ . Even though both CSQ and CLQ have the same number of quantization levels , and therefore the same representational capacity on the operand level , multiplication result , or the product of weight and activation , may have different representational capacity depending on the choice of quantizer . We compare the representational capacity of the product of weight and activation , when using CSQ vs. CLQ for weight . For activation quantization we consider unsigned CLQ ( denoted by CLQu ) for unsigned activation , and either CSQ or signed CLQ ( denoted by CLQs ) for signed activation . We start by defining the range of quantization levels of each scheme : CLQu ∼ { 0 , 1 , · · · , 2b − 1 } ( 7 ) CLQs ∼ { −2b−1 , −2b−1 + 1 , · · · , 2b−1 − 1 } ( 8 ) CSQ ∼ { −2b−1 + 0.5 , −2b−1 + 1.5 , · · · , 2b−1 − 0.5 } ( 9 ) where b is the precision . It should be noted that CSQ can only be signed . Table 2 shows the range of weights , activations and their product for 2-bit precision using different quantization methods . Table 3 shows that using CSQ for weight quantization almost always increases the representational capacity of quantized multiplication . The only exception is 2-bit precision for signed activation where it remains the same . Given that improving representational capacity is crucial for increasing the overall performance of QNNs ( Liu et al. , 2018 ) , our analysis suggests that CSQ has a definite advantage over CLQ for quantizing weights . Furthermore , the distribution of product resembles non-uniform quantization which provides higher accuracy . For more details refer to Appendix C. In the case of signed activation , our result in Table 3 suggests that using CLQs for activation and CSQ for weight provides the highest representational capacity .

Summary: This paper analyzed the difference in quantization methods between the conventional linear quantizer (CLQ) and centered symmetric quantization (CSQ). The authors explained that CSQ might exploit the limited quantization states better to represent the data than CLQ, with supporting analysis on representation capacity. The authors further proposed a bitwise implementation of CLQ and CSQ. The evaluation in terms of the quantization accuracy and GPU implementation speed is given for their analysis.