CSQ: CENTERED SYMMETRIC QUANTIZATION FOR EXTREMELY LOW BIT NEURAL NETWORKS
Year: 2021

Abstract:
Recent advances in quantized neural networks (QNNs) are closing the performance gap with the full precision neural networks. However at very low precision (i.e., ≤ 3-bits), QNNs often still suffer significant performance degradation. The conventional uniform symmetric quantization scheme allocates unequal numbers of positive and negative quantization levels. We show that this asymmetry in the number of positive and negative quantization levels can result in significant quantization error and performance degradation at low precision. We propose and analyze a quantizer called centered symmetric quantizer (CSQ), which preserves the symmetry of latent distribution by providing equal representations to the negative and positive sides of the distribution. We also propose a novel method to efficiently map CSQ to binarized neural network hardware using bitwise operations. Our analyses and experimental results using state-of-the-art quantization methods on ImageNet and CIFAR-10 show the importance of using CSQ for weight in place of the conventional quantization scheme at extremely low-bit precision (2∼3 bits).

heading: 1 INTRODUCTION
text: Quantized neural networks (QNNs) (Krishnamoorthi, 2018; Esser et al., 2019; Lee et al., 2021; Li et al., 2021; Nagel et al., 2020) can reduce both computational complexity and memory requirement quite effectively over the full-precision (i.e., floating-point) version, and hence are commonly used for deployment. Recent state-of-the-art QNNs (Esser et al., 2019; Lee et al., 2021; Li et al., 2019) can achieve near-full-precision accuracy even at 3-bit for ImageNet classification. However, most of the existing quantization method target 3-bit or higher. At 2-bit there is not much room for optimization, even with non-uniform step-size quantization (Li et al., 2019).
While some previous work, e.g. (Li et al., 2019), treats 2-bit as ternary ({−1, 0, 1}), it wastes one quantization level. But to make the most of the 2-bit precision, the conventional 2-bit quantizer allocates two vs. one quantization levels to the negative side, causing a severe imbalance. As an alternative to the conventional quantizer, we propose (zero-)centered symmetric quantizer (CSQ), which is most effective at low precision (2∼3-bit) though the method itself is applicable to any precision. Through a set of analyses and experiments we show that CSQ improves QNN performance over the conventional linear quantizer (CLQ) in quantization-aware training (QAT).1
We note that the particular quantizer function we propose has been used in some previous work (Choi et al., 2018a; Boo et al., 2021; Gong et al., 2019; Lee et al., 2021; Chen et al., 2021). However, no previous work has proposed it explicitly nor provided any analysis on the effectiveness of such a scheme (see Section 2.3). Moreover, while low-precision CLQ can be efficiently implemented in hardware via multiple invocations of XNOR-popcount such as on BNN (Binarized Neural Network) hardware, doing the same for CSQ poses new challenges that have never been addressed.
In this paper we make the following contributions.
• We propose CSQ that uses perfectly symmetrical quantization levels with uniform step size. We provide analytical and empirical evidence showing that using CSQ for weight instead of CLQ improves performance of low-bit QNNs (2∼3-bit).
1We use CSQ (and CLQ) to refer to both a quantizer and a quantization method.
• We propose a binary coding scheme and a mapping method for CSQ, which allows for an efficient hardware implementation of QNNs using bitwise operations on BNN hardware.

heading: 2 BACKGROUND AND RELATED WORK

heading: 2.1 QUANTIZATION PRIMER
text: A quantizer is a function from a real number x to a discrete value or an integer xQ. To train a network while simulating the effect of quantizer, one often maps the discrete value xQ back to a real value x̃. Instead of quantizing the entire range of inputs, clipping off the extreme values and only mapping the mid-range values uniformly is often more efficient. A generalized version of a quantizer used in Neural Networks can be defined as:
x̃ = clip( ⌊x s ⌉ , L, U) · s (1)
where L and U (the lower and upper bound) are the minimum and maximum integer values that xQ can take, and clip(x, a, b) = min(max(x, a), b). Quantization using the above quantizer in (1) is called uniform step-size quantization, also called linear quantization. We do not consider nonuniform step-size quantization in this paper.
Based on the input range (i.e., whether input is signed vs. un-signed), quantizers are classified as symmetric vs. asymmetric quantizers. For the linear quantizer in (1), symmetric and asymmetric quantizers can be defined as follows, where b is the number of bits used for representation.
Symmetric quantizer (for signed input): L = −2b−1, U = 2b−1 − 1 (2) Asymmetric quantizer (for un-signed input): L = 0, U = 2b − 1 (3)
From here on we will refer to quantization method defined by (1) and (2) as the Conventional Linear Quantizer (CLQ).

heading: 2.2 PREVIOUS WORK ON DNN QUANTIZATION
text: Most of the previous works on DNN quantization (Jung et al., 2019; Choi et al., 2018b; Esser et al., 2019) use essentially the same linear quantizer, but each work proposes a slightly different way to optimize quantizer functions. Often the differences come from different formulations of the quantizer function. For instance, QIL (Jung et al., 2019) optimizes quantization boundary, PACT (Choi et al., 2018b) optimizes a clipping parameter to train the quantized activations, and LSQ(Esser et al., 2019) optimizes the step size.We are not aware of any previous work explicitly proposing a new set of quantization levels for uniform quantization. Though the formulation of a quantizer function varies, in the end they all assume the same integer hardware that expects 2’s complement number representation.

heading: 2.3 SIMILAR APPROACHES
text: The idea of using a zero centered quantizer is also seen in (Choi et al., 2018a; Boo et al., 2021; Gong et al., 2019; Lee et al., 2021; Chen et al., 2021). All of these works present a novel training method for their quantizer. However, none of these works analyze zero-centered quantization or present it as a source of performance gain. Furthermore, to the best of our knowledge, no previous work has attempted to address the realization of zero-centered quantization on hardware. Realizing zerocentered quantization on hardware can be a challenging task, because it introduces new quantization points which can not be represented using 2’s complement number representation.

heading: 3 CENTERED SYMMETRIC QUANTIZATION (CSQ)

heading: 3.1 EXACT ZERO REPRESENTATION VS. PERFECT SYMMETRY
text: Any linear quantization to b-bit precision results in 2b quantization levels. Therefore, inclusion of zero among the quantization points, naturally results in an asymmetry in the number of positive and
negative quantization levels. While exact representation of zero may be important due to common operations like zero padding (Krishnamoorthi, 2018), it is also true that the asymmetry among positive and negative quantization levels grows large as precision becomes low. Thus one of our aims in this paper is to explore the trade-off between exact zero representation and perfect symmetry in the context of weight quantization of neural networks.

heading: 3.2 ALTERNATIVES
text: From the perspective of quantization levels, one can consider the following quantization schemes (see Table 1). Conventional Linear Quantization (CLQ) uses the quantization levels represented by 2’s complement, which is also used by most of the previous uniform step-size quantizers, e.g. (1). Alternatively, the asymmetry between the positive and negative sides can be reversed; i.e., by setting L = −2b−1 + 1 and U = 2b−1. Reduced Symmetric Quantization (RSQ) uses one less quantization level, thereby achieving both exact zero representation and perfect symmetry; i.e., L = −2b−1 +1 and U = 2b−1−1. This scheme wastes one quantization level and is expected to result in inferior performance. Extended Symmetric Quantization (ESQ), on the other hand, uses one more quantization level to achieve zero representation and symmetry; i.e., L = −2b−1 and U = 2b−1. However, since ESQ requires more than b-bit precision, it is not feasible for practical deployment, but included here for comparison. Non-uniform Symmetric Quantization (NSQ) achieves both exact zero representation and perfect symmetry with exactly 2b quantization levels, but the step size is not uniform, leading to a completely different quantizer function and QAT methods. Since NSQ is non-uniform quantization, we do not consider it for the scope of this paper. Centered Symmetric Quantization (CSQ) stipulates uniform step size and perfect symmetry between the positive and negative sides, while compromising on the exact representation of zero. It can also be represented using integers only by scaling all quantization levels by 2 as shown in the last row of the table. For the remainder of the paper we focus on CSQ and CLQ, as they are the most practical.

heading: 3.3 QUANTIZER FUNCTION FOR CSQ
text: We define the quantizer for CSQ as follows: v̇ = ⌊v s + 0.5 ⌉ − 0.5 (4)
v̄ = clip (v̇,−Q,Q) (5) v̂ = v̄ × s (6)
where v is any input value, s is the step size, and Q = 2b−1 − 0.5 with b being the quantization precision (i.e., the number of bits). As usual, b·e is the round operation, and clip(x, a, b) = min(max(x, a), b). Here v̄ is not an integer. Nevertheless, it is an exact value that can be represented by the b-bit CSQ format. Moreover, our proposed CSQ format permits efficient hardware and software realizations (see Section 5). Therefore v̄ represents the value that is computed by b-bit hardware. Finally, v̂ is the scaled-back version of v̄, defined and used for the purpose of training. The proposed quantizer provides equal representation for the positive and negative sides of the input distribution. Figure 1a shows the 2-bit quantizer functions for conventional linear quantization and our CSQ. Figure 1b shares the gradient for step size which is a quantization parameter. Detailed training method is described in Appendix A.

heading: 4 ANALYSIS OF CSQ

heading: 4.1 IMPROVED REPRESENTATIONAL CAPACITY OF CSQ
text: Even though both CSQ and CLQ have the same number of quantization levels, and therefore the same representational capacity on the operand level, multiplication result, or the product of weight and activation, may have different representational capacity depending on the choice of quantizer. We compare the representational capacity of the product of weight and activation, when using CSQ vs. CLQ for weight. For activation quantization we consider unsigned CLQ (denoted by CLQu) for unsigned activation, and either CSQ or signed CLQ (denoted by CLQs) for signed activation. We start by defining the range of quantization levels of each scheme:
CLQu ∼ {0, 1, · · · , 2b − 1} (7) CLQs ∼ {−2b−1,−2b−1 + 1, · · · , 2b−1 − 1} (8) CSQ ∼ {−2b−1 + 0.5,−2b−1 + 1.5, · · · , 2b−1 − 0.5} (9)
where b is the precision. It should be noted that CSQ can only be signed. Table 2 shows the range of weights, activations and their product for 2-bit precision using different quantization methods.
Table 3 shows that using CSQ for weight quantization almost always increases the representational capacity of quantized multiplication. The only exception is 2-bit precision for signed activation where it remains the same. Given that improving representational capacity is crucial for increasing the overall performance of QNNs (Liu et al., 2018), our analysis suggests that CSQ has a definite advantage over CLQ for quantizing weights. Furthermore, the distribution of product resembles non-uniform quantization which provides higher accuracy. For more details refer to Appendix C.
In the case of signed activation, our result in Table 3 suggests that using CLQs for activation and CSQ for weight provides the highest representational capacity.

heading: 4.2 CSQ AND CLQ AS DIFFERENT ZERO-POINTS OF AFFINE QUANTIZATION
text: In this section we analyze and compare CSQ and CLQ using a more general framework of affine quantization, which is defined as follows (Krishnamoorthi, 2018):
xint = ⌊x s ⌉ + z, x̄ = clip(xint, 0, 2 b − 1), x̂ = s(x̄− z) (10)
where z is called zero-point, and must be an integer since the other terms on both sides are all integers. However if we relax the zero-point to be any real value, CLQ and CSQ can both be considered as a special case of (relaxed) affine quantizer. Then the zero-point for CLQ (zCLQ) and CSQ (zCSQ) can be defined as follows:
zCLQ = 2 b−1, zCSQ = 2 b−1 − 0.5 (11) For instance, at b = 2, the term ⌊ x s ⌉ has the range of {−2,−1, 0, 1} in (signed) CLQ. Thus to satisfy (10), we have zCLQ = 2.
We can consider four cases depending on the restriction on zero-point (ZP): (i) ZP can take any real value (i.e., relaxed affine quantizer), (ii) ZP can take any integer value (i.e., affine quantizer), (iii) ZP is fixed to zCLQ, and (iv) ZP is fixed to zCSQ. Among these, CLQ is a special case of integer zero-point (affine quantizer) while CSQ is not (see Figure 2). At the same time, there is a hardware overhead for (i) and (ii), though integer zero-point can be implemented more efficiently than realvalue zero-point. We show in Section 5 that CSQ can be implemented as efficiently as CLQ.
The above view provides us with a methodology to somewhat objectively compare CSQ and CLQ by the distance of the optimal real-value ZP (z∗) to zCSQ vs. zCLQ; of which we present our results in Section 6.5. Also, it follows from (11) that the difference between zCLQ and zCSQ, relative to the entire quantization range, diminishes exponentially as b increases (see Appendix H), suggesting that the performance difference between CLQ and CSQ will be negligible for large b.

heading: 5 EFFICIENT REALIZATION OF CSQ
text: The high efficiency of BNN hardware comes from the bit-parallel processing (i.e., bitwise XNOR) of large arrays, which can be extended to efficient processing of very low-precision QNNs by interchanging the order of summation between precision and vector dimensions as observed by Zhou et al. (2016).
The challenge is how to extend the bitwise-operation based inner-product computation method to CSQ. Since there are multiple combinations, here we show the result for the case of a CLQu vector x and a CSQ vector v, each of which is N -dimensional and n-bit.
An n-bit binary number an−1an−2...a0 (n ≥ 2) represents the following value: Ab = ∑n−1 i=0 ai2 i. We define the CSQ number format as exemplified in Table 4, which leads to the following equalities (the second equality is not obvious, but is correct).
Acsq = n−1∑ i=0 ai2 i − (2n − 1)/2 = n−1∑ i=0 (−1)ai+12i−1 (12)
To avoid dealing with fractional numbers, let us use the 2× scaled version of CSQ (see Table 1). Then,
Acsq2 = n−1∑ i=0 (−1)ai+12i, (13)
which has a very similar mathematical structure as a CLQ number, allowing us to use the same trick of changing the order of precision (n) and vector dimension (N ) as in the inner-product computation of two CLQ numbers. Finally, we arrive at the following inner-product computation method (for more detail, see Appendix I):
v · x = (vH · xH << 2) + (vH · xL << 1) + (vL · xH << 1) + vL · xL (14)
which is shown for the 2-bit case (n = 2). xH and xL (similarly for vH and vL) are the N - dimensional bit-vectors of x containing only the higher and lower bits only, respectively, and << is the bitwise shift-left operation. Each product on the right-hand side of (14) can be computed on BNN hardware in a single cycle. Thus v · x can be computed in four cycles, using an additional adder/accumulator. It is worth mentioning that the same method as illustrated in (14) is also used when computing the inner-product of two CLQ vectors.
Now for the inner-product of two bit-vectors (e.g. vH · xH ) we can use the same structure of a bitwise operation followed by popcount, with a slight variation.
vcsq2 · xclqu = 2 · popcount(AND(vcsq2,xclqu))− popcount(xclqu) (15)
We subtract popcount(xclqu), since the bits of the CSQ bit-vector corresponding to the zero elements of the CLQ bit-vector must be ignored. Note that BNN hardware, e.g. (Umuroglu et al., 2017), relies on the same method for the inner-product computation of two bit-vectors, except that we use AND instead of XNOR and to subtract a popcount value, we need an additional popcount operation. However, accelerating QNNs with the CLQ format also requires AND operations in the exact same manner, and the additional popcount operation does not increase the hardware cost. Thus the bit-vector-level complexity of using CSQ is nearly the same as that of using CLQ. Since there is no distinction at the vector level, the inner-product operation with CSQ can be implemented as efficiently as with CLQ.

heading: 6 EXPERIMENTS

heading: 6.1 EXPERIMENTAL SETUP
text: We conduct experiments on CIFAR-10 (Krizhevsky et al., 2009) and ImageNet (Russakovsky et al., 2015) using ResNet-18, ResNet-20, ResNet-34 (He et al., 2016) and MobileNet-v2 (Sandler et al., 2018). All quantized models on ImageNet are initialized with the weights of pretrained full-precision model of the same network. The first and last layers are kept at 8-bit precision. Other than convolution and fully connected layers, all the other layers, e.g. batch norm, are kept in full precision.
For ImageNet experiments we use the training recipe of Esser et al. (2019). We use stochastic gradient descent (SGD) optimizer, with 0.9 momentum, cosine learning rate decay (Loshchilov & Hutter, 2016) without restarts, and the initial learning rate of 0.01. Weight decay is 0.25 × 10−4 for 2- bit, 0.5 × 10−4 for 3-bit, and 10−4 for 4-bit quantization. The quantized models are fine-tuned for 90 epochs. For ResNet-18 ImageNet experiments, we train the full-precision model ourselves from scratch, using the same experimental setup except 0.1 initial learning rate and 10−4 weight decay.
For CIFAR-10 experiments the weights for quantized models are trained from scratch. We use the same setting as with ImageNet except the following: the initial learning rate is 0.1, weight decay is 10−4, and each model is trained for 300 epochs. All experiments on CIFAR-10 are conducted using ResNet-20 network. All ImageNet and CIFAR-10 models were implemented in PyTorch.

heading: 6.2 CIFAR-10 RESULT
text: We compare our method with CLQ using a state-of-the-art QAT method by Esser et al. (2019). Each case is repeated five times; mean± std. dev. is reported for each case. The results are summarized in Table 5, which confirms that our proposed method performs significantly better than CLQ at 2-bit, while at 3- and 4-bit, there is little difference in performance between CLQ and CSQ.

heading: 6.3 IMAGENET RESULT
text: We compare CLQ vs. CSQ on ImageNet classification using the standard models such as ResNet and MobileNet. We also compare our method with PACT (Choi et al., 2018b), LQ-Nets (Zhang et al., 2018), QIL (Jung et al., 2019) and LSQ+ (Bhalgat et al., 2020). Since we use the training method by LSQ (Esser et al., 2019), the CLQ case also represents the previous work (LSQ). However, LSQ use pre-activation ResNet (He et al., 2016) which has higher performance than the standard ResNet architecture, and their trained models are not available. Therefore for a fair comparison, we have implemented LSQ ourselves, and use it as the baseline. For CSQ we use the same model and replace only the weight quantizer with CSQ, except in the first and last layers which are quantized to 8-bit CLQ. LSQ+ (Bhalgat et al., 2020) uses affine quantizer for activation that allows floatingpoint zero-point. However, ResNet and MobileNet use ReLU activation which results in unsigned activation which can not fully utilize the capability of affine quantizer. Therefore, we do not use affine quantizer for activations in our experiments. For comparison with LSQ+ we use their reported results. Lee et al. (2021) is another recent work that presents a training method for QNNs, that outperforms LSQ. However, they already use a quantizer than results in zero-centered quantization levels, similar to the one proposed in this paper. Therefore, we do not compare our results with them.
The results are summarized in Table 6 which shows that CSQ outperforms CLQ for all cases. We can observe that the performance gain by our method diminishes as precision increases. At 2- and 3-bit precision CSQ outperforms LSQ+ as well. We attribute this gain in performance to the improved representational capacity of CSQ, discussed in Section 4.1, and reduced quantization error shown in Section 6.4.

heading: 6.4 QUANTIZATION ERROR USING LSQ
text: We study the quantization error using CLQ and CSQ. We use full precision weights trained on ImageNet, which are also used to initialize weights for quantization-aware training. We compare
mean square error 〈(w−wq)2〉 between CLQ and CSQ, where w is the full precision weight and wq is the quantized weight. For each experiment we find a step size (s) that minimizes each error metric, by an exhaustive search. We report percentage error reduction by CSQ for each layer. which we define as: error reduction(%) = (ECLQ−ECSQ )ECLQ × 100, where ECLQ and ECSQ are the quantization errors using CLQ and CSQ, respectively. The results are presented in Figure 3.
It can be seen that CSQ reduces the overall quantization error compared to CLQ. At 2-bit precision CSQ provides a clear advantage over CLQ. At 3-bit precision CSQ is still significantly better than CLQ but the improvement in quantization error using CSQ is less comapred to 2-bit. Finally at 4-bit precision the improvement using CSQ seems much smaller than 2 and 3-bit precision. Additionally CLQ provides significantly better results in some layers compared to CSQ. Based on these results we can conclude that CSQ provides significant advanatge at 2-bit precision, which diminishes as we increase the precision.

heading: 6.5 ANALYZING CSQ USING AFFINE QUANTIZER
text: To analyze the effectiveness of CSQ, we train ResNet-20 for 2, 3 and 4-bit precision on CIFAR10, using affine quantization defined in (10). The latent weights for quantized model, are initialized using pre-trained full precision weights. We initialize and train the step size using LSQ (Esser et al., 2019), similar to experiments in Section 6.2 and Section 6.3. Zero-point is relaxed to be a real value and initialized to zCLQ from (11). We choose the initialization biased towards CLQ to show that zero-point converges closer to CSQ irrespective of initialization.
The results are shown in Figure 4. It can be seen that despite our initialization being biased towards CLQ, the zero-point distribution is always closer to CSQ than CLQ. Considering that the zero point distribution moves away from CSQ as the precision increases, the results suggest that there is an inverse relation between the effectiveness of CSQ and precision of the network, which is also consistent with our analytical result in Section 4.2, that CSQ is most effective at low precision (i.e., 2-bit) and loses effectiveness as we increase precision. The result in Figure 4 also indicates that CSQ provides significantly better approximation of affine quantization than CLQ.

heading: 6.6 GPU IMPLEMENTATION RESULTS
text: To see the speedup on GPU, we have implemented three custom kernels that use concatenation and bitwise operations to compute matrix multiplication (see Appendix I.2). We compare these custom kernels with cuBLAS. The first kernel is 1-bit CSQ, which is the same as 1-bit BNN. The second and third kernels are 2-bit, each using CSQ and CLQ.
The result is summarized in Table 7, which is the average of five runs. We observe exactly the same speed between CSQ and CLQ, which demonstrates that our CSQ is as efficient as the conventional binary representation on GPU. Note that apart from the main computation kernel, our CSQ is exactly the same as CLQ (such as memory requirement) and therefore has the same overall runtime as CLQ. Also our 2-bit CLQ is 2.9 times faster than cuBLAS while being only 2.3 times slower than 1-bit.

heading: 6.7 DISCUSSION
text: We can see from experimental results and analyses that CSQ provides significant advantage at 2-bit precision. It is also evident that CSQ loses significance as we increase precision. The question is, “what is the exact precision where CSQ loses its superiority to CLQ ?”. Experimental results and quantization error experiments in Section 6.4 demonstrate that superiority of CSQ is significantly reduced at 4-bit precision. However representational capacity analysis in Section 4.1 and affine quantization experiments in Section 6.5 suggest that CSQ provides significant advantage at 4-bit precision as well.
We know that 4-bit precision does not suffer from performance degradation using QuantizationAware Training (QAT). In fact with current state-of-the-art 4-bit precision achieves similar or even higher accuracy than full-precision networks for ResNet. This can also be seen in our experimental results in Table 6. In other words, for ResNet-18 4-bit precision can provide more than sufficient representation irrespective of quantization method and lack of representation capacity does not remain a bottleneck in performance. That is why at 4-bit precision we observe very similar results whether we use CSQ or CLQ. Based on this we conclude that CSQ achieves its limit at 4-bit precision for ResNet and beyond 4-bit precision CLQ and CSQ should perform similarly. While the exact precision threshold cannot be generalized to other networks or datasets (e.g. CSQ outperforms CLQ on ResNet-18 and ResNet-34 on ImageNet but does not provide the same superiority on ResNet-20 on CIFAR-10), the criteria to find the threshold are generalizable; that is, CSQ is superior while representation capacity is the bottleneck.
Finally, though we have used LSQ only (Esser et al., 2019) as the quantization method in our experiments, our quantizer can be applied with any quantization method. In Appendix D we present our PTQ (post-training quantization) result using another state-of-the-art quantization method (Li et al., 2021).

heading: 7 CONCLUSION
text: In this paper we provided an in-depth analysis of CSQ for extreme low-bit quantization, which is both completely symmetric around zero and trainable using existing linear quantization methods. Our analyses and experimental results using state-of-the-art quantization methods with CIFAR-10 and ImageNet datasets show that a simple change of quantization levels can result in significant performance improvement for extremely low bit quantized neural networks (≤ 4 bits). We also show that CSQ can be realized efficiently on BNN hardware and GPUs. Considering there are very few previous works targeting 2-bit network performance, CSQ can be a very useful tool for optimizing extreme low-precision neural networks for deployment.

heading: A OUR TRAINING METHOD
text: We have employed the method presented in Esser et al. (2019) to train the quantization parameters. To optimize the step size s using gradient descent, we use the following derivative formula.
∂v̂ ∂s =  −v/s + v̇ if −Q < v̇ < Q −Q if v̇ ≤ −Q Q if v̇ ≥ Q
(16)
Then computing the loss gradient w.r.t. step size is straightforward. Similar to gradient scaling in (Esser et al., 2019), the gradient of step size is scaled by factor g = 1/ √ NW 2p, where NW is the
number of weight parameters. The weights are initialized as 2〈|v|〉/ √ Q, where 〈.〉 represents the notation for mean of a distribution. Figure 1b shares the gradient of step size parameter at 2-bit precision. Our training method is based on LSQ (Esser et al., 2019) but it should be noted that CSQ can be used with any training method.

heading: B LIMITAIONS OF AFFINE QUANTIZER
text: We have defined affine quantizer by Krishnamoorthi (2018) in (10). Another variant of affine quantizer has also been proposed by Bhalgat et al. (2020):
x̄ = ⌊ clip ( x− z s , n, p )⌉ , x̂ = s · x̄ + z (17)
where n and p are clipping intervals. Bhalgat et al. (2020) does not restrict their zero-point to integer values. This approach can lead to significant hardware overhead. To tackle this problem they propose the affine quantizer for activations only, in which case zero-point can be implemented simply as a bias term, since
ŵ · x̂ = (w̄ · sw) (x̄ · sx + z) = w̄x̄swsx + zsww̄︸ ︷︷ ︸ bias
(18)
and w̄ is known in advance. This resolves the hardware overhead problem of affine quantization for activation. However, the same method cannot be applied to weight because it would involve pre-computing x̄ in advance, which is impossible.
Affine quantization for weights still results in some hardware overhead as the the zero-point of weights quantizer can not be implemented as the bias. This can be shown as:
ŵx̂ = (w̄ × sw + zw) (x̄× sx + zx) = w̄x̄swsx + zxsww̄ + zwzx︸ ︷︷ ︸ bias + zwsxx̄︸ ︷︷ ︸ overhead
(19)
This is because x̄ can not be pre-computed as it will change with each input. Similarly weight quantization overhead is shown by (Krishnamoorthi, 2018) as:
y(k, l, n) = swsx conv (w̄(k, l,m;n)− zw, x̄(k, l,m)− zx) (20)
y(k, l, n) = conv (w̄(k, l,m;n), x̄(k, l,m))− zw K−1∑ k=0 K−1∑ l=0 N−1∑ m=0 x̄(k, l,m) (21)
− zx K−1∑ k=0 K−1∑ l=0 N−1∑ m=0 w̄(k, l,m;n) + zxzw (22)
Considering these limitations of affine quantizer we have proposed CSQ as an approximation of affine quantizer which can be efficiently realized on hardware.

heading: C MORE ON REPRESENTATIONAL CAPACITY
text: In Section 4.1 we have shown that CSQ improves the representational capacity compared to CLQ. We have considered using signed, as well as unsigned activations. The most common activation
(a) 2-bit precision with signed activations (b) 3-bit precision with signed activations (c) 4-bit precision with signed activations
Figure 6: Representational capacity of quantized product with signed activations, using different quantization methods for 2, 3 and 4-bit precision.
function is ReLU, which produces unsigned activations. In case of signed activations such as PReLU and h-swish, affine quantization (Krishnamoorthi, 2018; Bhalgat et al., 2020) provides the best representation. However, affine quantizer can also be represented by unsigned CLQ before applying zero-points. Therefore unsigned CLQ for activations encapsulates affine quantizer as well. We also compare the representational capacity of quantized operation using symmetric activation range. Such an activation is unusual as the activation function introduces some non-linearity. However, it is possible in some cases e.g output of Tanh activation. In this case the activation is quantized using signed CLQ or CSQ. We show the normalized distribution of product with unsigned activations in Figure 5. It can be observed that the distribution of product values when using CSQ for weight is perfectly symmetric, and densely populated in the middle of the output range while being sparse near the boundary. This bell-shaped distribution better resembles layer activations, which is often exploited by non-uniform quantization schemes (Li et al., 2019; Lee et al., 2017) to achieve higher quantization performance. On the other hand, the product values when using CLQ for weight, are more uniformly distributed for the entire output range, with an exceptional outlier in the negative range and no matching positive outlier, introducing asymmetry to the output as well.
Figure 6 shows the normalized distribution of product with signed activation. It can be seen that using CSQ for weights and CLQ for activations provides significantly higher representational capacity compared to using CLQ for weights or CSQ for activations.

heading: D POST TRAINING QUANTIZATION RESULTS (COMPARISON WITH BRECQ)
text: BRECQ (Li et al., 2021) uses affine quantizer with integer zero-point, which extends the linear quantizer in a different direction than our CSQ. Thus it is very interesting to see how our method compares with BRECQ. Note that integer zero-point requires some additional hardware overhead whereas CSQ does not. At the same time, the two schemes are orthogonal in the sense that one can combine both schemes, viz. integer zero-point and CSQ, so that zero-point take any integer or half-integer value, at the cost of some hardware overhead. Here we compare only CSQ vs. BRECQ,
but not the combination. Since we have proposed CSQ for weight quantization, we only quantize the weights in BRECQ experiments.
To implement CSQ we simply fix the zero point value to CSQ which is shown as ZCSQ in (11). The shared results may be different from reported results. The results have been reproduced using the officially shared code. We share our results on ResNet-18 and ResNet-34. We used channelwise quantization and all the hyper-parameters are same as shown in Li et al. (2021). Table 8 shows the experimental results using BRECQ on ImageNet data. It can be seen that at 2-bit precision CSQ outperforms affine quantizer. This is especially interesting because unlike affine quantization, CSQ does not incur any hardware overhead. At 3 and 4-bit precision, affine quantizer gives better performance. However, CSQ also provides competitive results. This shows that CSQ is a very strong approximation of (relaxed) affine quantizer.

heading: E ADDITIONAL EXPERIMENTS USING KNOWLEDGE DISTILLATION
text: To further demonstrate the generalization-ability of CSQ, we conduct some experiments with knowledge distillation (Hinton et al., 2015) loss. We use LSQ (Esser et al., 2019) for training the quantization parameters and show that CSQ performs superior to CLQ at low precision even when we use advanced training methods such as knowledge distillation over state-of-the-art quantization-aware training method.
The experimental methodology and training setups are the same as described in Section 6.1, except that we use a weight decay of 0.25e-4 and knowledge distillation loss for all experiments. For knowledge distillation loss we set temperature as 1 and give equal weight to the standard loss and the distillation loss following (Esser et al., 2019). Our experimental results in Table 9 show that at extremely low precision, i.e., at 2- and 3-bit, CSQ outperforms CLQ. This is consistent with experimental results in Table 6. We have already discussed in Section 6.7 why CSQ may lose its advantage at 4-bit precision.

heading: F WHY CSQ IS ALWAYS BETTER THAN CLQ AT 2-BIT
text: In the weight quantization error experiment of Section 6.4, optimizing for minimum quantization error faces the challenge of maximizing the utility of limited quantization levels. The utility is max-
imized (i) if each quantization level has equal number of real values mapped to it, in the same way as the Lloyd-Max quantization (Max, 1960) is optimal. Also it can be helped a lot (ii) if the quantization levels have the same distribution as the underlying data. In other words, for the first objective the underlying data (when mapped to quantization levels) should be distributed as uniformly as possible, while for the second objective the distribution of quantization levels should resemble that of the underlying data (e.g., Gaussian). This usually creates conflicting requirements, but not in the case of 2-bit CSQ. At 2-bit, CSQ has only three quantization thresholds, {−1, 0, 1}, and therefore can satisfy both requirements simultaneously: (i) real-valued weight data are uniformly distributed across quantization levels, and at the same time, (ii) quantization levels follow Gaussian (or any symmetric) distribution, as illustrated in Figure 7. Note that this is not possible for 2-bit CLQ, as in other precision for either CLQ or CSQ, which explains why CSQ always shows better performance than CLQ at 2-bit. From the figure, the scale parameter can be determined as s = Φ−1(0.75), where Φ is the CDF of standard normal distribution, since P (0 ≤ X < s) = 0.25 when X ∼ N(0, 1)). Also our experimental results confirm that indeed real-valued weight data are mapped uniformly to 2-bit CSQ quantization levels as shown in Figure 8.

heading: G EXHAUSTIVE SEARCH METHOD FOR QUANTIZATION ERROR EXPERIMENTS
text: We present quantization error experiments in Section 6.4. To find the step size that minimizes the Quantization Error we use an exhaustive search method. Our exhaustive search method is very similar to the method used by Esser et al. (2019) for their quantization error experiments. The exhaustive search goes as follows. First, we initialize step sizes as:
s0 = 〈|w|〉
2p − 1 (23)
where w represents full precision weights,〈.〉 represents the mean operation, and p is the bit-width. Then for the search space S = {0.01s0, 0.02s0, 0.03s0, · · · , 5s0}, we exhaustively find the value of s ∈ S that minimizes the target quantization error metric. This helps us find the minimum quan-
tization error using CLQ and CSQ, for any given bit-width. Experimental results and analysis has been presented in Section 6.4. https://www.overleaf.com/project/608a0e19c95387366f0de2f6

heading: H MORE ABOUT ZERO-POINT
text: The difference between zCLQ and zCSQ is constant at 0.5 (see (11)) while the possible range of values that can be taken by the quantizer grows exponentially with b. Thus we can give the percentage difference between zero-points of CLQ vs. CSQ relative to the entire quantization range by:
D = zCLQ − zCSQ 2b × 100 = 1 2b+1 × 100 (24)
This equation implies that D decreases as we increase the precision. For example, at 2-bit precision, D is 12.5% but at 4-bit precision, D reduces to mere 3.125%. This shows that at higher precision the difference between CSQ and CLQ becomes negligible compared to the entire distribution range and they should provide similar performance. This is consistent with our experimental results in Section 6.
H.1 ZERO REPRESENTATION
CSQ does not provide an exact representation for zero. Instead, zero or values slightly less than zero are rounded to −0.5 while values slightly greater than zero are rounded to 0.5 (before scale factor). Despite this it has been shown in Section 6.4 that CSQ can reduce quantization error compared with CLQ. Furthermore, our experimental results have also shown superior performance with CSQ. This serves as an evidence that exact zero representation is not critical for weight quantization, especially at low precision (< 4 bits).
H.2 ZERO PADDING
Zero padding is needed for activation only, not for weight. In case, if zero padding were used for weight as well, we can simulate the exact effect of rounding zeros during QAT, and adjust weight accordingly. So it would not contribute to any performance degradation.
Now, vector and tensor processors (e.g. TPU) must process an array of values together, and may “fill” some elements with zeros as needed. This zero filling is needed for both weight and activation. For CSQ weight, inexact zero representation may introduce an error or discrepancy between algorithm and realization. This error can be eliminated or minimized, depending on hardware dataflow, by resetting corresponding activation values to zeroes or filling with both +0.5 and −0.5.
H.3 WHY PROPOSE CSQ FOR WEIGHT ONLY
CSQ can be used for signed activation. However, activation frequently involves zero padding. Therefore, activation quantization strictly demands exact zero representation to support zero padding. Furthermore Section 4.1 shows that when using CSQ for weight, using CLQs for activation is much better in terms of representational capacity than using CSQ for both weight and activation. To summarize, using CLQ for activation is not only important to support zero padding, but also for best performance. Thus we recommend that using CSQ for weight quantization and CLQ for activation quantization, which provides the best performance with no hardware overhead.

heading: I EFFICIENT HARDWARE REALIZATION OF CSQ
text: I.1 DIGITAL HARDWARE IMPLEMENTATION
CSQ can be efficiently implemented in hardware and software. To avoid using fractional numbers, in this section we assume the 2× scaled version of CSQ (see Table 1). We can think of CSQ as an n-bit extension of BNN encoding. We will show in the section below how we use the XNOR-popcount based inner-product method of BNNs to realize the product of two CSQ vectors. Conventional BNN hardware takes two 1-bit CSQ bit-vectors as an input. Assuming two N -element bit-vectors v and
x, the inner-product is calculated as follows:
vcsq · xcsq = 2 · popcount(XNOR(vcsq,xcsq))−N (25)
In the case of the product between CSQ and unsigned CLQ, we should ignore the values of CSQ when CLQu is 0. Therefore, we subtract the number of CSQ values after the popcount operation, which is identical to popcount(xclqu).
vcsq · xclqu = 2 · popcount(AND(vcsq,xclqu))− popcount(xclqu) (26)
We now extend (26) to the multi-bit case so that the inner-product between CSQ and unsigned CLQ vectors can be performed efficiently. Consider a 2-bit CSQ vector and a 2-bit unsigned CLQ vector of N elements.
vcsq = [ aN−11 a N−1 0 a N−2 1 a N−2 0 ... a 0 1a 0 0 ] , xclqu = [ bN−11 b N−1 0 b N−2 1 b N−2 0 ... b 0 1b 0 0 ] (27)
Each vector can be split into two bit-vectors by dividing the lower bits and the higher bits of each element.
vHcsq = [ aN−11 a N−2 1 ... a 0 1 ] , vLcsq = [ aN−10 a N−2 0 ... a 0 0 ] (28)
xHclqu = [ bN−11 b N−2 1 ... b 0 1 ] , xLclqu = [ bN−10 b N−2 0 ... b 0 0 ] (29)
The product of v and x can be computed as follows (<< is the bitwise shift-left operation):
v · x = (vH · xH << 2) + (vH · xL << 1) + (vL · xH << 1) + vL · xL (30)
Each product on the right-hand side of (30) can be computed on the proposed hardware in a single cycle. Thus v · x can be computed in four cycles, using an additional adder/accumulator.
I.2 FAST GPU IMPLEMENTATION
It is possible to speed up BNNs on GPU by using bitwise operations (Hubara et al., 2016), through a custom GPU kernel to boost matrix multiplication on GPU. We have implemented a custom kernel to compute the inner product of two (N -dimensional) vectors of n-bit CSQ numbers.
a = [ aN−1n−1 a N−1 n−2 · · · a N−1 0 a N−2 n−1 a N−2 n−2 · · · a N−2 0 · · · a0n−1a0n−2 · · · a00 ] (31)
b = [ bN−1n−1 b N−1 n−2 · · · b N−1 0 b N−2 n−1 b N−2 n−2 · · · b N−2 0 · · · b0n−1b0n−2 · · · b00 ] (32)
The kernel first concatenates the most significant bits into a single N -bit number (if N > 32, we break it to a multiple of 32). Repeating this for all n bits, each vector is converted to an n-element array.
ac = [ aN−1n−1 a N−2 n−1 ...a 0 n−1 a N−1 n−2 a N−2 n−2 · · · a0n−2 · · · a N−1 0 a N−2 0 · · · a00 ] (33)
bc = [ bN−1n−1 b N−2 n−1 · · · b0n−1 b N−1 n−2 b N−2 n−2 · · · b0n−2 · · · b N−1 0 b N−2 0 · · · b00 ] (34)
Then we perform the XNOR-popcount operation n2 times, accumulating its result.
s← s + {popcount(XNOR(ac[i],bc[j])) << (i + j)} (35)

Review summaries:
This paper presents a centered symmetric quantizer (CSQ) that can map floating point tensors into zero-centered quantized integer values. The CSQ method works well for ultra-low bit quantization such as 2-bit. The authors also propose a binary coding method to run efficiently on hardware.

This paper analyzed the difference in quantization methods between the conventional linear quantizer (CLQ) and centered symmetric quantization (CSQ). The authors explained that CSQ might exploit the limited quantization states better to represent the data than CLQ, with supporting analysis on representation capacity. The authors further proposed a bitwise implementation of CLQ and CSQ. The evaluation in terms of the quantization accuracy and GPU implementation speed is given for their analysis.

This paper points out that conventional uniform quantization methods do not allocate the same quantization level in the positive and negative areas and proposes Centered Symmetric Quantizer (CSQ), a zero-centered symmetric quantizer, to address the resulting degradation at very low precision. It also presents a method of mapping the proposed CSQ to BNN hardware through bit-wise operations. This paper claims that no specific analysis of CSQ has been made in previous studies and that low-precision quantization results in poor performance due to the allocation of asymmetric quantization levels in the positive and negative sides of the distribution.