CSQ: Centered Symmetric Quantization for Extremely Low Bit Neural Networks
Authors: Faaiz Asim
Jaewoo Park
Azat Azamat
Jongeun Lee
Venue: ICLR 2022 Submitted (2022)
Keywords: deep learning
classification
low precision
uniform symmetric quantization
binary neural network hardware

Abstract: Recent advances in quantized neural networks (QNNs) are closing the performance gap with the full precision neural networks. However at very low precision (i.e., $\le 3$-bits), QNNs often still suffer significant performance degradation. The conventional uniform symmetric quantization scheme allocates unequal numbers of positive and negative quantization levels. We show that this asymmetry in the number of positive and negative quantization levels can result in significant quantization error and performance degradation at low precision. We propose and analyze a quantizer called centered symmetric quantizer (CSQ), which preserves the symmetry of latent distribution by providing equal representations to the negative and positive sides of the distribution. We also propose a novel method to efficiently map CSQ to binarized neural network hardware using bitwise operations. Our analyses and experimental results using state-of-the-art quantization methods on ImageNet and CIFAR-10 show the importance of using CSQ for weight in place of the conventional quantization scheme at extremely low-bit precision (2$\sim$3 bits).

Under review as a conference paper at ICLR 2022
CSQ: CENTERED SYMMETRIC QUANTIZATION FOR EXTREMELY Low BIT NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Recent advances in quantized neural networks (QNNs) are closing the perfor- mance gap with the full precision neural networks. However at very low precision (i.e., S 3-bits), QNNs often still suffer significant performance degradation. The conventional uniform symmetric quantization scheme allocates unequal numbers of positive and negative quantization levels. We show that this asymmetry in the number of positive and negative quantization levels can result in significant quan- tization error and performance degradation at low precision. We propose and ana- lyze a quantizer called centered symmetric quantizer (CSQ), which preserves the symmetry of latent distribution by providing equal representations to the negative and positive sides of the distribution. We also propose a novel method to effi- ciently map CSQ binarized neural network hardware using bitwise operations. Our analyses and experimental results using state-of-the-art quantization meth- ods ImageNet and CIFAR-10 show the importance of using CSQ for weight in place of the conventional quantization scheme at extremely low-bit precision (2~3 bits).
INTRODUCTION
Quantized neural networks (QNNs) (Krishnamoorthi] 2018 Esser et al. 2019: Lee et al. 2021 1
Li
al. 202 Nagel et al. 2020) can reduce both computational complexity and memory requirement quite effectively over the full-precision (i.e.. floating-poing version, and hence are commonly used for deployment. Recent ffate-of-the-ass QNNs (Esser al. 2019 Lee al. 2021 Li 2019) can achieve near-full-pe full-precision accuracy even at 3-bit for ImageNet classification. However, most of the existing quantization method target 3-bit or higher. At 2-bit there is not much room for optimization, even with non-uniform siep-size quantization (Li et al. 2019).
While some previous work, e.g. (Li et al. 2019). treats 2-bit as ternary ((-1, 0, 1)), wastes one quantization level. But to make the most of the 2-bit precision, the conventional 2-bit quantizer allocates two vs. one quantization levels to the negative side, causing a severe imbalance. As an alternative to the conventional quantizer, we propose (zero-)centered) symmetric quantizer (CSQ), which most effective at low precision (2~3-bbit) though the method itself is applicable to any precision. Through set of analyses and experiments we show that CSQ improves QNN performance over the conventional linear quantizer (CIQQ) in quantization-aware training (QAT)
We note that the particular quantizer function propose has been used in some previous work (Choi jet al. 2018a] Boo et al. 2021 Gong et al., 2019 Lee et al., [2021] Chen et al. 2021). However, no previous work has proposed explicitly nor provided any analysis on the effectiveness of such scheme (see Section 2.3). Moreover, while low low-precision CLQ can be efficiently implemented in hardware via multiple invocations of XNOR-popcour such as on BNN (Binarized Neural Network) hardware, doing the same for CSQ poses new challenges that have never been addressed.
In this paper we make the following contributions.
We propose CSQ that uses perfectly symmetrical guantization levels with uniform step size. We provide analytical and empirical evidence showing that using CSQ for weight instead of CLQ improves performance of low-bit QNNs (2~~3bitt).
We use CSQ (and CL-Q) refer both quantizer and quantization method.

---

Under review as a conference paper at ICLR 2022
We propose binary coding scheme and a mapping method for CSQ, which allows for an efficient hardware implementation QNNs using bitwise operations on BNN hardware.
BACKGROUND AND RELATED WORK
2.1 QUANTIZATION PRIMER
A quantizer is function from real number to discrete value an integer zQ- To train network while simulating the effect of quantizer, one often maps the discrete value zQ back to a real value ž. Instead of quantizing the entire range of imputs, clipping off the extreme values and only mapping the mid-range values uniformly is often more efficient. A generalized version of a quantizer used in Neural Networks be defined
z z= clipp
L.U)
(1)
where and U (the lower and upper bound) are the minimum and maximum integer values that Te take, and clip(x, a, 6) min min(max(z, a), b). Quantization using the above quantizer in (I) called uniform step-size quantization, also called linear quantization. We do not consider non-
uniform step-size quantization in this paper.
Based on the input range (i.e., whether input is signed vs. un-signed), quantizers are classified as symmetric vs. asymmetric quantizers. For the linear quantizer in ( symmetric and asymmetric quantizers can be defined as follows, where b the number of bits used for representation.
Symmetric quantizer (for signed input):
L --
U = 2-2
(2)
Asymmetric quantizer (for un-signed input):
L= 0,
U U=2 -1
From here on we will refer to quantization method defined by and 2 as the Conventional Linear Quantizer (CLQ).
2.2
PREVIOUS WORK ON DNN QUANTIZATION
Most of the previous works on DNN quantization (Jung et al., 2019 Choi et al., 2018b Esser al. 2019) use essentially the same linear quantizer, but each work proposes a slightly different way to optimize quantizer functions. Often the differences come from different formulations of the quantizer function. For instance, QIL (Jung et al. 2019) optimizes quantization boundary, PACT (Choi et al. 2018b) optimizes a clipping parameter to train the quantized activations, and LSQ(Esser jet al., 2019) optimizes the step size. Wi are not aware of any previous work explicitly proposing set of quantization levels for uniform quantization. Though the formulation of quantizer function varies, in the end they all assume the same integer hardware that expects 2's complement number representation.
2.3 SIMILAR APPROACHES
The idea of using a zero centered quantizer is also seen in (Choi et al. 2018a Boo et al. 2021 Gong al. 20199] Lee et al. 2021 Chen et 202T). All of these works present novel training method for their quantizer. However, none of these works analyze zero-centered quantization a source of performance gain. Furthermore, to the best of our knowledge, no previous work has attempted to address the realization of zero-centered quantization centered quantization on hardware can be a challenging task, because introduces new quantization points which can not be represented using 2's complement number representation.
present
hardware. Realizing zero-
CENTERED SYMMETRIC QUANTIZATION (CSQ)
3.1 EXACT ZERO REPRESENTATION VS. PERFECT SYMMETRY
Any linear quantization b-bit precision results in 2a quantization levels. Therefore, inclusion of zero among the quantization points, naturally results in asymmetry in the number of positive and
2

---

Under review as a conference paper at ICLR 2022
Table 1: Quantization levels by representation (2-bit example)
Case
Quantization levels
Representation
1
1-2-110, 1}
CLQ
2
{-1.0,1,2}
CLQ-altemative
3
{-1,0.1}
RSQ (Ternary)
-2,-11,,1,,]
ESQ
5
[-2.-1,1,2)
NSQ
6
[-1,5,
1.5}
CSQ
7
(-3. 1,333
CSQ (scaled by 2)
"Note: ESQ requires more than 2 bits.
negative quantization levels. While exact representation of zero may be important due to common operations like zero padding (Krishnamoorthi (018)), it is also true that the asymmetry among posi- tive and negative quantization levels grows large as precision becomes low. Thus one of our aims in this paper is to explore the trade-off between exact zero representation and perfect symmetry in the context of weight quantization of neural networks.
3.2 ALTERNATIVES
From the perspective of quantization levels, one can consider the following quantization schemes (see Table Conventional Linear Quantization (CLQ) uses the quantization levels represented by 2's complement, which is also used by most of the previous uniform step-size quantizers, e.g. [T. Alternatively, the asymmetry between the positive and negative sides can be reversed; i.e., by setting L = -2h + 1 and U 2 Reduced Symmetric Quantization (RSQ) uses one less quantization level, thereby achieving both exact zero representation and perfect symmetry; i.e., I. = - and U = 26-1 - 1. This scheme wastes one quantization level and is expected to result in inferior performance. Extended Symmetric Quantization (ESQ), on the other hand, uses one more quantization level to achieve zero representation and symmetry: i... L -2 and U 2 However, since ESQ requires more than b-bit precision, it is not feasible for practical deployment, but included here for comparison. Non-uniform Symmetric Quantization (NSQ) achieves both exact zero representation and perfect symmetry with exactly 2 quantization levels, but the step size not uniform, leading completely different quantizer function and QAT methods. Since NSQ is non-uniform quantization, we do not consider it for the scope of this paper. Centered Symmetric Quantization (CSQ) stipulates uniform step size and perfect symmetry between the positive and negative sides, while compromising on the exact representation of zero. It can also be represented using integers only by scaling all quantization levels by 2 as shown the last row of the table. For the remainder of the paper we focus on CSQ and CLQ, they are the most practical.
3.3 QUANTIZER FUNCTION FOR CSQ
We define the quantizer for CSQ follows:
0.5 0.5
(4)
clip (b, -0,000 Q)
(5)
0 = 7 x
(6)
where any input value, s is the step size, and Q 2b-1 0.5 with b being the quantiza- tion precision (i.e., the number of bits). As usual. is the round operation, and clip(æ, a, b) = min (ma b). Here is not an integer. Nevertheless, sented by the b-bit CSQ format. Moreover, our proposed CSQ format permits efficient hardware and software realizations (see Section 5}. Therefore represents the value that is computed by b-bit hardware. Finally, 5 is the scaled-back version of defined and used for the purpose of training. The proposed quantizer provides equal representation for the positive and negative sides of the input distribution. Figure Ia shows the 2-bit quantizer functions for conventional linear quantization and our CSQ. Figure [b] shares the gradient for step size which is a quantization parameter. Detailed training method is described in Appendix A
an exact value that can be repre-
3

---

Under review as a conference paper at ICLR 2022
CLQ
gradient
0.2
CSQ
quantized
00
06
-1
0.2
float
(a) Quantization of 2-bit signed data using conventional linear quantization (CI.Q)
(b) The gradient of the CSQ quantization output w.r.. step size.
CSQ (ours).
Figure I: CSQ quantizer and gradient.
Table 2: List of values in the product term (2-bit example, unsigned activation)
W-A: CLQ,-CLO.
W-A: CSQ-CLQ.
W (signed)
(-2. 1,00,1
1.5, 0.5, 0.5. 1.5)
A (unsigned)
10,1,
(0, 1,,3))
W A
6, 44 -3, -2, 12,1,0,1,233
(-4.5 3, 1.5, -1, 0.5,0 15,3,455
4 ANALYSIS OF CSQ
4.1 IMPROVED REPRESENIATIONAL CAPACITY OF CSQ
Even though both CSQ and CLQ have the same number of quantization levels, and therefore the same representational capacity the operand level, multiplication result, the product of weight and activation, may have different representational capacity depending on the choice quantizer. We compare the representational capacity of the product of weight and activation, when using CSQ vs. CLQ for weight. For activation quantization we consider unsigned CLQ (denoted by CLQu) for unsigned activation, and either CSQ or signed CLQ (denoted by CLQ²) for signed activation. We start by defining the range of quantization levels of each scheme:
CLQ,, {0, ,2
CT.Q (-20-1 22--22011 1, 1,... 2 1}
(8)
CSQ (-2h-1 + 0.5, 2 + 1.5, 20-1 0.5)
where b is the precision. It should be noted that CSQ can only be signed. Table shows the range of weights, activations and their product for 2-bit precision using different quantization methods.
Table 3 shows that using CSQ for weight quantization almost always increases the representational capacity of quantized multiplication. The only exception is 2-bit precision for signed activation where it remains the same. Given that improving representational capacity is crucial for increasing the overall performance of QNNs (Liu et al.) 2018), our analysis suggests that CSQ has a definite advantage over CLQ for quantizing weights. Furthermore, the distribution of product resembles non-uniform quantization which provides higher accuracy. For more details refer to Appendix C
In the case of signed activation, our result in Table 3 suggests that using CLQ₂ for activation and CSQ for weight provides the highest representational capacity.
4.2 CSQ AND CLQ AS DIFFERENT ZERO-POINTS OF AFFINE QUANTIZATION
In this section we analyze and compare CSQ and CLQ using more general framework of affine quantization, which is defined as follows (Krishnamoorthi 2018):
Fint
+ F = =ciip(iint 0,2º - 1), 4 = s( - =2)
(10)
4

---

Under review as a conference paper at ICLR 2022
Table 3: Number of unique values in the product term
Unsigned Activation (W-A)
Signed Activation (W-A)
#bits
CLQ,-CLQu
CSQ-CLQ,
CLQ.-CLO,
CSQ-CLQ.
CSQ-CSQ
2-bit
9
(+22.2%%
(450.0%%
6 (0.0%)
3-bit
35
43 (+22.9%%
18
31 (+72.2%)
20 (+11.0%)
4-bit
120
155 (+29.2%)
60
105 (+75.0%)
(+10.0%)
b+ Z
1
1/2
2
3
(= Zcsp)
(= Zano
Figure 2: Zero-point for CLQ and CSQ 2-bit precision. Affine quantizer allows integer zero-point
only (shown in green circles). In relaxed affine quantizer, zero-point can take any real value.
where 2 is called zero-point, and must be an integer since the other terms on both sides are all inte- gers. However if we relax the zero-point to be any real value, CLQ and CSQ can both be considered as a special case of (relaxed) affine quantizer. Then the zero-point for CLQ (ZCT.Q) and CSQ (ECSQ) can be defined as follows:
ZCLQ = 26-1 cCs = 26 - 0.5
(11)
For instance, at b = 2, the term has the range of 1, 1} in (signed) CLQ. Thus to satisfy (10), we have ZCLQ = 2.
We can consider four cases depending on the restriction zero-point (ZP): (i) ZP can take any real value (i.e., relaxed affine quantizer), (ii) ZP can take any integer value (i.c., affine quantizer), (ii)) ZP is fixed to SCLQ, and (iv) ZP is fixed to 2csQ. Among these, CLQ is special case of integer zero-point (affine quantizer) while CSQ is not (see figure2). At the same time, there is a hardware be implemented more efficiently than real- value zero-point. We show in Section 5 that CSQ can be implemented as efficiently as CLQ.
overhead for (i) and (ii), though integer zero-point
The above view provides with methodology to somewhat objectively compare CSQ and CLQ by the distance of the optimal real-value ZP (=") to #CsQ vs. ECLQ; of which we present our results in Section 6.5 Also, follows from (IT) that the difference between SCLQ and EcsQ. relative the entire quantization range, diminishes exponentially as b increases (see Appendix H.. suggesting that the performance difference between CLQ and CSQ will be negligible for large
EFFICIENT REALIZATION OF CSQ
The high efficiency of BNN hardware comes from the bit-parallel processing (i.e., bitwise XNOR) of large arrays, which can be extended efficient processing of very low-precision QNNs by terchanging the order of summation between precision and vector dimensions as observed by Zhou ct al. (2016).
The challenge is how to extend the bitwise-operation based inner-product computation method to CSQ. Since there are multiple combinations, here we show the result for the case of a CLQu, vector and CSQ vector v, each of which is N --dimensional and n-bit.
An ritit binary number fn=110--...4pp (n > 2) represents the following value: As = =i= 0,24. We define the CSQ number format as exemplified in Table which leads to the following equalities (the second equality is not obvious, but is correct).
Aesg
0, (2" 11/2
(110:
(12)
To avoid dealing with fractional numbers, let us use the 2x scaled version of CSQ (see Table Then,
1
Aeeg2
+1
(13)
5

---

Under review as a conference paper at ICLR 2022
Table 4: Comparison of number representations: CLQ vs. CSQ (2-bit example)
2-bit binary
CLQ₂
CLQu
CSQ
00
0
0
- -1.5
01
0.5
10
-2
0.5
11
1.5
which has a very similar mathematical structure as a CLQ number, allowing us to use the same trick of changing the order of precision (n) and vector dimension (N) in the inner-product computation of two CLQ numbers. Finally, we arrive at the following inner-product computation method (for more detail, see Appendix
v.x (VH XH 2< + (v Xt. << + (Vr. XH << 1) + Xr.
(14)
which is shown for the 2-bit case (n 2). X and xy (similarly for VH and vL) are the N- dimensional bit-vectors of x containing only the higher and lower bits only, respectively, and << the bitwise shift-left operation. Each product on the right-hand side of ([4) can be computed on BNN hardware in single cycle. Thus v x can be computed in four cycles, using an additional adder/accumulator. It is worth mentioning that the same method as illustrated in (14) is also used when computing the inner-product of two CLQ vectors.
Now for the inner-product of two bit-vectors (e.g. Vи X1)) we can use the same structure of bitwise operation followed by popcount, with slight variation.
Vceg2 Xolg. = 2 popcount (AND(V Xelqu)) )) popcount(X.tg.)
(15)
We subtract popcount(Xxto.)) ), since the bits of the CSQ bit-vector corresponding to the zero el- ements of the CLQ bit-vector must be ignored. Note that BNN hardware, e.g. (Umuroglu et al, 2017). relies on the same method for the inner-product computation of two bit-vectors, except that we use AND instead of XNOR and to subtract a popcount value, we need an additional popcount operation. However, accelerating QNNs with the CLQ format also requires AND operations in the exact same manner, and the additional popcount operation does not increase the hardware cost. Thus the bit-vector-eevel complexity of using CSQ is nearly the same as that using CLQ. Since there is no distinction at the vector level, the inner-product operation with CSQ can be implemented as efficiently with CL.Q.
6 EXPERIMENTS
6.1 EXPERIMENTAL SETUP
We conduct experiments CIFAR- 10 Krizhevsky et al. 2009 and ImageNet (Russakovsky et al, 2015 using ResNet-18. ResNet-20. ResNet-34 (He al. 20166) and MobileNet-v2 (Sandler al., 2018 All quantized models on ImageNet are initialized with the weights of pretrained pree-porii model of the same network. The first and last layers are kept at 8-bit precision. Other than convolu- tion and fully connected layers, all the other layers, e.g. batch norm, are kept in full precision.
For ImageNet experiments we use the training recipe Esser al. (2019). We use stochastic gradi- ent descent (SGD) optimizer, with 0.9 momentum, cosine learning rate decay (Loshchilov & Hutter 2016) without restarts, and the initial learning rate of 0.01. Weight decay 0.25 x 10-1 for 2- bit, 0.5 10-4 for 3-bit, and 10-4 for 4-bit quantization. The quantized models are fine-tuned for 90 epochs. For ResNet-18 18 ImageNet experiments, we train the full-precision model ourselves from scratch, using the same experimental setup except 0.1 initial learning rate and 10 weight decay.
For CIFAR-10 experiments the weights for quantized models are trained from scratch. We use the same setting with ImageNet except the following: the initial learning rate is 0. weight decay is 10 10-4, and each model is trained for 300 epochs. All experiments on CIFAR-10 are conducted using ResNet-20 network. All ImageNet and CIFAR-10 models implemented in PyTorc..

---

Under review as a conference paper at ICLR 2022
Table 5: Comparison of CLQ and CSQ (ours) on ResNet-20 for CIFAR-10.
Top-1 Accuracy @ Precision
Network
ResNet-20 Full Precision: 91.22
Precision (W/A)
2/2
3/3
4/4
CLQ (LSQ)
89.09
90.7010.18
91.07=0.13
CSQ (ours)
89.46.0.2255
90.8010.12
90.96=1.225
Table 6: Comparison of CLQ. CSQ and other quantization-avaal training methods on ImageNet.
The CLQ case also represents LSQ (Esser et al. 2019). MobileNet-v2 2-bit case did not converge.
Top-1 Accuracy Precision
Network
Re
ResNet-34
MobileNet-v2
Full Precision:
70.58
Full Precision: 73.31
Full Precision: 71.88
Precision (W/A)
2/2
3/3
4/4
2/2
3/3
4/4
2/2
3/3
4/4
PACT
64.40
68.10
69.20
LQ-Nets
64.90
68.20
69.30
69.80
71.90
QIL
65.70
69.20
70.10
70.60
73.10
73.70
CLQ (LSQ)
66.59
69.38
70.52
70.56
73.21
73.82
60.41
66.82
LSQ+
66.80
69.40
70.80
CSQ (ours)
66.92
69.48
70.63
70.82
73.29
74.01
60.89
66.98
6.2 CIFAR-10 10 RESULT
We compare our method with CLQ using a state-of-the-e QAT method by Esser et al. (2019). Each case is repeated five times; mean + std. dev. is reported for each case. The results summarized in 5 which confirms that our proposed method performs significantly better than CLQ at 2-bit, while 3- and 4-bit, there little difference in performance between CLQ and CSQ.
6.3 IMAGENET RESULT
We compare CLQ vs. CSQ on ImageNet classification using the standard models such ResNet and MobileNet. We also compare our method with PACT (Choi et al. 20188). LQ-Nets (Zhang et al. 2018). QIL (Jung et al. 2019) and LSQ+ (Bhalgat al. (020). Since we use the training method by LSQ (Esser et al. 2019). the CLQ case also represents the previous work (LSQ). However, LSQ use pre-activation ResNet (He et al. 2016) which has higher performance than standard ResNet architecture, and their trained models
not available. Therefore for fair comparison, have implemented LSQ ourselves, and use as the baseline. For CSQ we use the same model and replace only the weight quantizer with CSQ, except in the first and last layers which are quantized to 8-bit CLQ. LSQ+ (Bhalgat et al. 2020) uses affine quantizer for activation that allows floating- point zero-point. However, ResNet and MobileNet activation which can not fully utilize the capability of affine quantizer. Therefore, we do not use affine quantizer for activations in experiments. For comparison with LSQ+ use their reported results. Lee et al. (202I) is another recent work that presents training method for QNNs, that outperforms LSQ. However, they already use quantizer than results in centered quantization levels, similar to the one proposed in this paper. Therefore, we do not compare our results with them.
ReLU activation which results in unsigned
The results are summarized T which shows that CSQ outperforms CLQ for all cases. We can observe that the performance gain by our method diminishes as precision increases. At 2- and 3-bit precision CSQ outperforms LSQ+ as well. We attribute this gain in performance to the improved representational capacity of CSQ, discussed in Section 4.1 and reduced quantization error shown in Section 6.4
6.4 QUANTIZATION ERROR USING LSQ
We study the quantization error using CLQ and CSQ. We use full precision weights trained on ImageNet, which are also used initialize weights for quantization-aware training. We compare

---

Under review as a conference paper at ICLR 2022
15
5R SESUCTIO 15
21 100000000000
BR-OX REDUCTIONNN
10-
20-
LANERS
LATE1O
LAWERS
(a) 2-bit precision
(b) 3- bit precision
(c) 4-bit precision
Figure 3: Percentage error reduction using CSQ. The results are shown for ResNet-18 trained on ImageNet.
6-6
20
6.00 airr Dare
artinaa dara PLRT
2.5
Zero Points
Zero Pointe
(a) 2-hit precision
(b) 3-bit precision
(c) 4-hit precision
Figure 4: Distribution optimal zero-point values of relaxed affine quantizer, across layers. The
graphs also show the zero-points for CSQ and CLQ, represented vertical lines.
mean square error ((w (ww-wy) )²)²) between CLQ and CSQ, where w is the full precision weight and ty the quantized weight. For each experiment we find a step size (s) that minimizes each error metric, by exhaustive search. We report percentage error reduction by CSQ for each layer. which we define as: error reduction(%)
(Fezg- Fesq) x 100, where ECLQ and Easq are the quantization
Ecco
errors using CLQ and CSQ. respectively. The results are presented in Figure 3
It can be seen that CSQ reduces the overall quantization error compared CLQ. At 2-bit precision CSQ provides a clear advantage over CLQ. At 3-bit precision CSQ is still significantly better than CLQ but the improvement in quantization error using CSQ is less comapred to 2-bit. Finally 4-bit precision the improvement using CSQ seems much smaller than and 3-bit precision. Additionally CLQ provides significantly better results in some layers compared to CSQ. Based on these results we can conclude that CSQ provides significant advanatge at 2-bit precision, which diminishes as we increase the precision.
6.5 ANALYZING CSQ USING AFFINE QUANTIZER
To analyze the effectiveness of CSQ, we train ResNet-20 for 2, 3 and 4-bit precision on CIFAR- 10, using affine quantization defined (TO). The latent weights for quantized model, are initialized using pre-trained full precision weights. We initialize and train the step size using LSQ (Esser et al., 2019), similar experiments in Se. and Section 6.3 Zero-point relaxed to be real value and initialized to ÆCLQ from (1T). We choose the initialization biased towards CLQ to show that zero-point converges closer to CSQ irrespective of initialization.
The results are shown in Figure It can be seen that despite our initialization being biased towards CLQ, the zero-point distribution is always closer to CSQ than CLQ. Considering that the zero point distribution moves away from CSQ the precision increases, the results suggest that there is an inverse relation between the effectiveness sistent with our analytical result in Section 2 that CSQ is most effective low precision (i.e., 2-b and loses effectiveness as we increase precision. The result in Figure also indicates that CSQ provides significantly better approximation of affine quantization than CL.Q.
CSQ and precision the network, which also con-
8

---

Under review as a conference paper at ICLR 2022
Table 7: Matrix multiplication runtime (size: 16384x 16384) on Nvidia RTX 2080 TI
Kernel
Runtime (ms)
Relative Speed
cuBLAS
579.81 + 9.16
1-bit
85.44 + 11.99
6.8
2-bit CSQ
197.85 + 6.82
2.9
2-bit CLQ
197.85 10.20
2.9
6.6 GPU IMPLEMENTATION RESULTS
To see the speedup on GPU, we have implemented three custom kernels that use concatenation and bitwise operations to compute matrix multiplication (see Appendix 1.2]. We compare these custom kernels with cuBLAS. The first kernel is 1-bit CSQ, which is the same as 1-bit BNN. The second and third kernels are 2-bit, each using CSQ and CLQ.
The result is summarized in Table 7 which is the average of five runs. We observe exactly the same speed between CSQ and CL.Q, which demonstrates that our CSQ is as efficient as the conventional binary representation on GPU. Note that apart from the main computation kernel, our CSQ is exactly the same as CLQ (such as memory requirement) and therefore has the same overall runtime as CLQ. Also our 2-bit CLQ is 2.9 times faster than cuBLAS while being only 2.3 times slower than 1-bit.
6.7 DISCUSSION
We can see from experimental results and analyses that CSQ provides significant advantage at 2-bit precision. It is also evident that CSQ loses significance as we increase precision. The question is, "what is the exact precision where CSQ loses its superiority to CLQ ?". Experimental results and quantization error experiments in Section 6.4 demonstrate that superiority of CSQ is significantly reduced at 4-bit precision. However representational capacity analysis in Section 4. and affine quantization experiments Section 6.5 suggest that CSQ provides significant advantage at 4-bit precision as well.
We know that 4-bit precision does not suffer from performance degradation using Quantization- Aware Training (QAT). In fact with current state-of-the-art 4-bit precision achieves similar higher accuracy than full-precision networks for ResNet. This can also be seen in our experimental results in Table In other words, for ResNet-18 4-bit precision can provide more than sufficient representation irrespective of quantization method and lack of representation capacity does not re- main bottleneck in performance. That is why at 4-bit precision we observe very similar results whether we use CSQ or CLQQ Based on this we conclude that CSQ achieves its limit at 4-bit preci- sion for ResNet and beyond 4-bit precision CLQQ and CSQ should perform similarly. While the exact precision threshold cannot be generalized to other networks or datasets (e.g. CSQ outperforms CLQ on ResNet-18 and ResNet-34 on ImageNet but does not provide the same superiority on ResNet-20 on CIFAR-10), the criteria to find the threshold are generalizable; that is, CSQ is superior while representation capacity is the bottleneck.
Finally, though we have used LSQ only (Esser et al. 2019) as the quantization method in our exper- iments, our quantizer can be applied with any quantization method. In Appendix D we present our PTQ (post-training quantization) result using another state-of-the-art quantization method (Li et al., 2021).
CONCLUSION
In this paper we provided an in-depth analysis of CSQ for extreme low-bit quantization, which both completely symmetric around zero and trainable using existing linear quantization methods. Our analyses and experimental results using state-of-the-art quantization methods with CIFAR- and ImageNet datasets show that a simple change of quantization levels can result in significant performance improvement for extremely low bit quantized neural networks (S 4 bits). We also show that CSQ can be realized efficiently on BNN hardware and GPUs. Considering there are very few previous works targeting 2- network performance, CSQ can be a very useful tool for optimizing extreme low-precision neural networks for deployment.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 696-697, 2020.
Yoonho Boo, Sungho Shin, Jungwook Choi, and Wonyong Sung. Stochastic precision ensemble: self-knowledge distillation for quantized deep neural networks. In Proceedings of the AAAI Con- ference Artificial Intelligence, volume 35, pp. 6794 6802, 2021.
Peng Chen, Jing Liu, Bohan Zhuang, Mingkui Tan, and Chunhua Shen. Aqd: Towards accurate quantized object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 104-113, 2021.
Jungwook Choi, Pierce I-Jen Chuang, Zhuo Wang. Swagath Venkataramani, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Bridging the accuracy gap for 2-bit quantized neural net- works (qnn). arXiv preprint 18Xiv:1807.000 2018a.
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprini arXiv: 1805.06085, 2018b.
Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra S Modha. Learned step size quantization. In International Conference on Learning Repre- sentations, 2019.
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4852-4861, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint 1XXx::1503.02531 2015.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 4114 4122, 2016.
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4350-4359, 2019.
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.0m (00.:188600832,, 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
E. H. Lee, D. Miyashita, E. Chai, B. Murmann, and S. S. Wong. Lognet: Energy-efficient neu- ral networks using logarithmic computation. In 2017 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pp. 5900-590000 2017. doi: 10.1109/ICASPP22017
7953288.
Junghyup Lee, Dohyung Kim, and Bumsub Ham. Network quantization with element- wise gradient scaling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 6448 6457, 2021.
10

---

Under review as a conference paper at ICLR 2022
Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-twe quantization: An efficient non- uniform discretization for neural networks. In International Conference on Learning Representa- tions, 2019.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2102.05426. 2021.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1 1-bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European conference on computer vision (ECCV), pp- 722- 737, 2018.
lya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.08 03983, 2016.
Joel Max. Quantizing for minimum distortion. IRE Transactions on Information Theory, 6(1):7-12, 1960.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort, Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learming. pp. 7197-7206. PMLR, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet. International journal of computer vision, 1 115(3):211-2522 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Liang-Chieh
Yaman Umuroglu. Nicholas J Fraser, Giulio Gambardella, Michaela Blott, Philipp Leong, Magnus Jahre, and Kees Vissers. Finn: A framework for fast, scalable binarized neural network inference. In Proceedingy of the 2017 ACM/SIGDA International Symposium Field- Programmable Gate Arrays, pp. 65-74, 2017.
Dongging Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. Proceedings of the European conference on computer vision (ECCV), pp. 365-382, 2018.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 1rXxv:16066.6160,,
2016. 2016.

---

Under review as a conference paper at ICLR 2022
A OUR TRAINING METHOD
We have employed the method presented in Esser et al. (2019) to train the quantization parameters. To optimize the step size using gradient descent, the following derivative formula.
00 w/s
+
if <i<Q if S -Q i QQ
(16)
Ös
Then computing the loss gradient w.r.t. step size is straightforward. Similar to gradient scaling in [Esser et al. 2019), the gradient of step size is scaled by factor ///////ww/2 2P, where Nw is the number of weight parameters. The weights are initialized as 2(lll)/VZ where (.) represents notation for mean of distribution. Figure Th shares the gradient of step size parameter at 2-bit
precision. Our training method is based on LSQ (Esser et 2019) but should be noted that CSQ can be used with any training method.
B LIMITAIONS OF AFFINE QUANTIZER
We have defined affine quantizer by Krishnamoorth 2018 in 10 Another variant of affine quan- tizer has also been proposed by Bhalgat al. (2020):
2 p.
I clip
s- 8- 2
(17)
where n and p are clipping intervals. Bhalgar gat et al. (2000) does not restrict their zero-point to integer values. This approach can lead to significant hardware overhead. To tackle this problem they propose the affine quantizer for activations only, in which case zero-point can be implemented simply as a bias term, since
i 5 = (w 8w) (x 82 + 2) 8. + 28w
(18)
and is known in advance. This resolves the hardware overhead problem of affine quantization for activation. However, the same method cannot be applied to weight because it would involve pre-computing F in advance, which is impossible.
Affine quantization for weights still results in some hardware overhead as the the zero-point of weights quantizer can not be implemented as the bias. This can be shown
. (w x Sw + Zwn) (I x $z + 2x) = w.Sw + 82 + z Eg + Sy
(19)
bios
overhead
This is because E not be pre-computed it will change with each input. Similarly weight quantization overhead is shown by (Krishnamoorthi (018) as:
ukk, 1, n.) Sw conv (w(k, m: n) Zw: _(k, 1, m)
-2z)
(20)
K N-1
L,) conv (w(k, m; n), F F(k (k,,,)) Zw Σ Σ 1kk4,m)
(21)
0m0
K K-IK-IN-1
2. ΣΣΣ M==
(h, I. m,; + Ze
(22)
Considering these limitations of affine quantizer have proposed CSQ as an approximation of affine quantizer which can be efficiently realized on hardware.
C MORE ON REPRESENTATIONAL CAPACITY
In Section 4.1 we have shown that CSQ improves the representational capacity compared to CLQ. We have considered using signed, as well unsigned activations. The most common activation
12

---

Under review as a conference paper at ICLR 2022
66 mo.
6.7
0.11
0.1
0.23
025
025
6.0
4.50
1.00
4.50
Rurber wleaaaa
13
09
150
19
36
(a) 2-bit precision
(b) 3-bit precision
(c) 4-bit precision
Figure 5: Distribution of quantized product values when using CSQ vs. CLQ₂ for weight, and CLQu. for unsigned activation.
190
mn 1n6
630
035
639
639
A08
(a) 2-bit precision with signed ac- tivations
(b) 3-bit precision with signed ac- tivations
(c) 4-bit precision with signed ac- tivations
Figure 6: Representational capacity of quantized product with signed activations, using different quantization methods for 2, and 4-bit precision.
function is ReLU, which produces unsigned activations. case of signed activations such as PReLU and h-swish, affine quantization (Krishnamoorthi 2018: Bhalgat al. 2020 provides the best rep- resentation. However, affine quantizer can also be represented by unsigned CLQ before applying zero-points. Therefore unsigned CLQ for activations encapsulates affine quantizer well. We also compare the representational capacity of quantized operation using symmetric activation range. Such activation is unusual the activation function introduces some non-linearity. However, it is pos- sible some cases e.g output of Tanh activation. In this case the activation is quantized using signed CLQ or CSQ. We show the normalized distribution of product with unsigned activations in Figure It can be observed that the distribution of product values when using CSQ for weight is perfectly symmetric, and densely populated in the middle of the output range while being sparse near the boundary. This bell-shaped distribution better resembles layer activations, which is often exploited by non-umin non-uniform quantization schemes (Li et al., 20199) Lee et al. (017) to achieve higher quanti- zation performance. On the other hand, the product values when using CLQ for weight, are more uniformly distributed for the entire output range, with an exceptional outlier the negative range and matching positive outlier, introducing asymmetry the output well.
FigurefBshows the normalized distribution of product with signed activation. can seen that using CSQ for weights and CLQ for activations provides significantly higher representational capacity compared to using CLQ for weights or CSQ for activations.
D POST TRAINING QUANTIZATION RESULTS (COMPARISON WITH BRECQ
BRECQ (Li et al. I021) uses affine quantizer with integer zero-point, which extends the linear quantizer in different direction than CSQ. Thus it is very interesting see how our method compares with BRECQ. Note that integer zero-point requires some additional hardware overhead whereas CSQ does not. At the same time, the two schemes can combine both schemes, viz. integer zero-point and CSQ, so that zero-point take any integer or half-integer value, at the cost of some hardware overhead. Here compare only CSQ vs. BRECQ,
orthogonal in the sense that one
13

---

Under review as a conference paper at ICLR 2022
Table 8: Comparison of CSQ with affine quantizer using BRECQ on ImageNet.
Accuracy
Network ResNet-18
Method
2/32
3/32
4/32
Full Precision:
71.08
BRECQ
66.60
69.82
70.64
BRECQ-CSQ (ours)
66.93
69.81
70.62
ResNet-50
Full Precision:
77.00
BRECQ
72.16
75.68
76.41
BRECQ-CSQ (ours)
72.24
75.52
76.35
but not the combination. Since we have proposed CSQ for weight quantization, we only quantize the weights in BRECQ experiments.
To implement CSQ we simply fix the zero point value to CSQ which is shown as Zaso in The shared results may be different from reported results. The results have been reproduced using the officially shared code. We share our results on ResNet-18 18 and ResNet- 34. We used channel- wise quantization and all the hyper-parameters
same shown in Li et al. (021). Table shows the experimental results using BRECQ on ImageNet data. can be seen that at 2-bit precision CSQ outperforms affine quantizer. This especially interesting because unlike affine quantization, CSQ does not incur any hardware overhead. At 3 and 4-bit precision, affine quantizer gives better performance. However, CSQ also provides competitive results. This shows that CSQ is very strong approximation of (relaxed) affine quantizer.
E ADDITIONAL EXPERIMENTS USING KNOWLEDGE DISTILLATION
To further demonstrate the generalization-abiltty CSQ, we conduct some experiments with knowl- edge distillation (Hinton et al.) 2015) loss. We use LSQ (Esser et al. 2019) for training the quanti- zation parameters and show that CSQ performs superior to CLQ at low precision even when we use advanced training methods such as knowledge distillation over state-of-the-art quantization-wware training method.
The experimental methodology and training setups the cept that we use a weight decay of 0.25c-4 and knowledge distillation loss for all experiments. For knowledge distillation loss the distillation loss following (Esser et al. 2019). Our experimental results in Table show that at extremely low precision, i.e., 2- and 3-bit, CSQ outperforms CL.Q This is consistent with experi- mental results in 6 We have already discussed in Section 6.7 why CSQ may lose its advantage 4-bit precision.
described in Section 6.
temperature as 1 and give equal weight to the standard loss and
F WHY CSQ IS ALWAYS BETTER THAN CLQ AT 2-BIT
In the weight quantization enror experiment of Section 6.4 emror faces the challenge of maximizing the utility of limited quantization levels. The utility is max-
optimizing for minimum quantization
x =025
X
-11/2 -1/2 1/2 11/2
Figure 7: With 2-bit CSQ, quantization levels (shown the æ-axis) resemble Gaussian distri- bution while, at the same time, real-valued weight data are also uniformly mapped to them (scale parameter is omitted for brevity).
14

---

Under review as a conference paper at ICLR 2022
Table 9: Comparison of CSQ vs. CLQ using knowledge distillation training on ImageNet.
Top-1
Accuracy @ Precision
Network
ResNet-18 18
Full Precision:
70.52
Precision (W/A)
2/2
3/3
4/4
CLQ (LSQ) + Knowledge Distillation
66.99
69.77
70.63
CSQ + Knowledge Distillation (ours)
67.24
69.90
70.56
(a) CL.Q 2-bit
(b) CSQ 2-hit
Figure 8: How real-valued weight data are mapped to CLQ vs. CSQ after training ResNet-18 on ImageNet.
imized (i) if each quantization level has equal number of real values mapped to it, in the same way as the Lloyd-Max quantization (Max 1960) is optimal. Also it can be helped a lot (ii) if the quanti- zation levels have the same distribution the underlying data. In other words, for the first objective the underlying data (when mapped to quantization levels) should be distributed as uniformly as pos- sible, while for the second objective the distribution quantization levels should resemble that of the underlying data (e.g.. Gaussian). This usually creates conflicting requirements, but not in the case of 2-bit CSQ. At b-t CSQ has only three quantization thresholds, f-1, O. 1}, and therefore can satisfy both requirements simultaneously: (i) real-valued weight data are uniformly distributed across quantization levels, and at the same time, (ii) quantization levels follow Gaussian (or any symmetric) distribution, illustrated in Figure 7 Note that this is not possible for 2-bit CLQ, as in other precision for either CLQQ or CSQ, which explains why CSQ always shows better performance than CLQ at 2-bit. From the figure, the scale parameter can be determined as Φ - (0.75). where is the CDF of standard normal distribution, since P(0 S X < s) 0.25 when X ~ (0, 1)). Also experimental results confirm that indeed real-valued weight data mapped uniformly to 2-bit CSQ quantization levels as shown in Figure g
G EXHAUSTIVE SEARCH METHOD FOR QUANTIZATION ERROR EXPERIMENTS
We present quantization error experiments in Section 6.4 To find the step size that minimizes the Quantization Error we use an exhaustive search method. Our exhaustive search method very simi- lar the method used by Esser (2019) for their quantization error experiments. The exhaustive search goes as follows. First, we initialize step sizes as:
(|w|)
8p = 2
(23)
where represents full precision weights.(.) represents the mean operation, and is the bit-width. Then for the search space = (0.01sp. 0.02s,, 0.03sg. 5. we exhaustively find the value of S that minimizes the target quantization error metric. This helps find the minimum quan-
15

---

input. Assuming N-element bit-vectors and Under review as a conference paper at ICLR 2022
tization error using CLQ and CSQ, for any given bit-width. Experimental results and analysis has been presented in Section 6.4 http://wwwwocltaccoooooooooooooa
H MORE ABOUT ZERO-POINT
The difference between ECLQ and ECSQ is constant at 0.5 (see (11)) while the possible range of val- ues that can be taken by the quantizer grows exponentially with Thus we can give the percentage difference between zero-points of CLQ vs. CSQ relative to the entire quantization range by:
D ZCLQ - 2CSQ X 100 = 2b+1 X 100
(24)
2
This equation implies that D decreases
increase the precision. For example, at 2-bit precision, D is 12.5% but at 4-bit precision, D reduces to mere 3. 3.5%. This shows that at higher precision the difference between CSQ and CLQ becomes negligible compared to the entire distribution range and they should provide similar performance. This is consistent with our experimental results in Section 6
H. ZERO REPRESENTATION
CSQ does not provide an representation for zero. Instead, zero values slightly less than are rounded to -0.5 while values slightly greater than zero are rounded to 0.5 (before scale factor). Despite this has been shown in Section 6..4 that CSQ can reduce quantization error compared with CLQ. Furthermore, our experimental results have also shown superior performance with CSQ. This serves an evidence that exact zero representation is not critical for weight quantization, especially at low precision (< 4 bits).
H.2 ZERO PADDING
Zero padding is needed for activation only, not for weight. In case, if zero padding were used for weight as well, we can simulate the exact effect rounding zeros during QAT, and adjust weight accordingly. So it would not contribute to any performance degradation.
Now, vector and tensor processors (e.g. TPU) must process an array of values together, and may "fill" some elements with zeros as needed. This zero filling is needed for both weight and activation. For CSQ weight, inexact zero representation may introduce an eигог or discrepancy between algorithm and realization. This error be eliminated minimized, depending hardware dataflow, by resetting corresponding activation values to zeroes or filling with both +0.5 and -0.5.
H.3 WIIY PROPOSE CSQ FOR WEIGHT ONLY
CSQ can be used for signed activation. However, activation frequently involves zero padding. There- fore, activation quantization strictly demands exact zero representation to support zero padding. Furthermore Section 4.1 shows that when using CSQ for weight, using CLQ, for activation is much better in terms of representational capacity than using CSQ for both weight and activation. To sum- marize, using CLQ for activation is not only important to support zero padding, but also for best performance. Thus we recommend that using CSQ for weight quantization and CI.Q for activation quantization, which provides the best performance with no hardware overhead.
I EFFICIENT HARDWARE REALIZATION CSQ
I.1 DIGITAL HARDWARE IMPLEMENTATION
CSQ can be efficiently implemented in hardware and software. To avoid using fractional numbers, in this section we assume the 2x scaled version of CSQ (see Table We can think of CSQ as an n-bit extension of BNN encoding. We will show in the section below how based anerrprroduct method BNNs to realize the product of two CSQ vectors. Conventional BNN hardware takes two 1-bit CSQ bit-vectors
the XNOR- popcouut
16

---

Under review as a conference paper at ICLR 2022
the inner-product is calculated as follows:
Vesq
Xas popcoun((XNORI (Vesgs Xesq)) N
(25)
In the case of the product between CSQ and unsigned CLQ, we should ignore the values of CSQ when CLQu is O. Therefore. subtract the number of CSQ values after the popcount operation, which is identical to )xpcount(xdt...
Verg Xelg. = 2 popcount (ANDunntAA Xelg.)) )) popcount(xalv)
(26)
We now extend (26) to the multi-bit case so that the inner-produce between CSQ and unsigned CLQ vectors can be performed efficiently. Consider 2-bit CSQ vector and 2-bit unsigned CLQ vector of elements.
Veag ap
-1
af a
afag]
[b] -6 -
Xeig.
(27)
Each vector can be split into two bit-vectors by dividing the lower bits and the higher bits of each element.
Ve = [a] N-1
N-
01 VLegg = [00 ag
N-1
N
(28)
[bN-1
by N
bi]
XLetau
160
bo N-
(29)
The product of v and x can be computed as follows (<< is the bitwise shift-left operation):
v (VH жи << << + (vH X1, << 1) + (vy, XH << 1) + VL. Xt.
(30)
Each product on the right-hand side of 330 can computed on the proposed hardware in a single cycle. Thus v can be computed in four cycles, using an additional adder/accuumllatrr
1.2 FAST GPU IMPLEMENTATION
is possible to speed up BNNs on GPU by using bitwise operations (Hubara et al. 2016), through a custom GPU kernel to boost matrix multiplication on GPU. We have implemented custom kernel to compute the inner product of two (N-dimensional) vectors of n-bit CSQ numbers.
a | n-2 0
N-1
N-1
N-
... 0
N-2
1 0
af]
(31)
b 6N
N-1
2
6 66]
(32)
The kernel first concatenates the most significant bits into single N-bit number (if N > 32, break it to multiple of 32). Repeating this for all n bits, each vector is converted to an n-clement array. a [ a ...0--1
N-2
N-1 N-2 nn
dg 4g
N N-2
(33)
b. NN2
bQ
b
b
2
(34)
Then we perform the XNOR- operation n² times, accumulating its result.
+- s + (popcoun(XNNOR(a.ll) b.(l)) << (1 (i+1))
(35)
17


Reviews:
[{"review_id": "brXauWYZ0bh", "reviewer": "Reviewer_hDhb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: A new idea of applying centered symmetric quantizer for model quantization.\n\nWeakness:\n\n1. Speedup setting: The authors only compare the single matrix multiplication runtime, which does not necessarily represent the speedup and the accuracy of the whole network [1]. \n\n2. Due to the new quantization level, the authors are encouraged to conduct experiments on other tasks (e.g., detection and NLP tasks.), which can present the generalization and robustness of CSQ.\n\n3. The accuracy improvement is marginal than CLQ on the ImageNet dataset.\n\n4. If people optimize sparse mand quantized models jointly [2, 3], is the CLQ with sparsity equal to CSQ? \n\n\n[1]. MQBench: Towards Reproducible and Deployable Model Quantization Benchmark\n\n[2]. Compressing Deep Neural Networks with Pruning, Trained\n\n[3]. A unified framework of dnn weight pruning and weight clustering/quantization using admm", "summary_of_the_paper": "This paper presents a centred symmetric quantizer(CSQ), a new symmetrical quantization scheme.", "summary_of_the_review": "The analysis and experiments have some issues (see weakness). I will give a borderline due to those concerns and change it accordingly based on rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Q4O8GRCiLru", "reviewer": "Reviewer_PPJy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work's main idea is well-written and pretty straightforward by comparing the proposed CSQ with CLQ (Conventional Linear Quantizer) and explaining that some quantization representations are wasted in the conventional quantization methods. The idea of using a zero-centered quantizer is ok, but it might need more novelty because it has already been used in several previous works. That's why the authors claim that this paper has other contributions in that they propose an explicit quantizer function for CSQ and provide some analysis on the effectiveness of such a scheme at very low precision. However, it is not clear to the reviewer if their claim is technically sound because theoretical analysis or empirical evidence does not support their main contributions strongly. Moreover, there are some cons to demonstrating its effectiveness in the experimental results. Here are a few details regarding this concern. \n\n1. Apart from explaining the problem of CLQ and the expressive power of existing quantization methods, the reviewer is unsure about the necessity of allocating symmetric quantization levels. The learned weight distribution as described in Figure 2 of DFQ [1] may not be symmetrical. Because the weight distribution being learned may be similar to the normal distribution but not a perfectly symmetric structure, allocating symmetric quantization levels need not be assigned to the positive and negative regions. Previous QAT studies such as clipping methods [2], QIL [3], or LSQ [4] have been attempted to reduce the dynamic range of non-uniform quantization or to learn the quantization intervals for this reason. It seems that applying CSQ is not always better than the previous approaches in the above situation. The author should explain first the necessity of allocating symmetric quantization levels by comparing these methods to claim the motivation of this study. Is there any particular reason why the authors excluded non-uniform quantization methods or other QAT approaches to solve the same problem from the discussion?\n\n2. In the experiment, it is explained that CSQ is applied only to weight quantization. However, it doesn't seem easy to conclude that the results presented in Table 5, Table 6, and Figure 3 are entirely caused by symmetrical quantization of CSQ. Nevertheless, it should be possible to clearly explain that CSQ majorly contributes to these results, excluding the effects of activation quantization and learning weights. \n\n3. Overall, the improvement due to CSQ seems to be minor and insignificant for both CIFAR-10 and ImageNet because there is little performance gap when looking at the results presented in Table 5 and Table 6 and results of applying PTQ covered in the appendix. Therefore, it seems not convincing to validate the effectiveness of the proposed CSQ scheme.\n\n4. Some recent works (Boo et al.(2021), Lee et al.(2021), Chen et al.(2021)) have used a quantizer that results in zero-centered quantization levels similar to the one proposed in this paper. So why not compare the experimental results with them?\n\nMinor Comments: \n1. The second equality in equation (12) is not intuitive, so it needs to be proved. \n2. Equation (30) on page 6 should be replaced with equation (14). \n3. There are a few typos.\n\nReferences: \n[1] M. Nagel, M. V. Baalen, T. Blankevoort, and M. Welling, \"Data-free quantization through weight equalization and bias correction,\" ICCV 2019. \n[2] R. Banner, Y. Nahshan, E. Hoffer, and D. Soudry, \"ACIQ: Analytical Clipping for Integer Quantization of Neural Networks,\" arXiv e-print, arXiv:1810.05723, Oct 2018.\n[3] S. Jung, C. Son, S. Lee, J. Son, J. Han, Y. Kwak, S. Hwang, and C. Choi, \"Learning to quantize deep networks by optimizing quantization intervals with task loss,\" CVPR 2019.\n[4] S. Esser, J. McKinstry, D. Bablani, R. Appuswamy, and D. Modha, \"Learned step size quantization,\" ICLR 2019.\n", "summary_of_the_paper": "This paper points out that conventional uniform quantization methods do not allocate the same quantization level in the positive and negative areas and proposes Centered Symmetric Quantizer (CSQ), a zero-centered symmetric quantizer, to address the resulting degradation at very low precision. It also presents a method of mapping the proposed CSQ to BNN hardware through bit-wise operations. This paper claims that no specific analysis of CSQ has been made in previous studies and that low-precision quantization results in poor performance due to the allocation of asymmetric quantization levels in the positive and negative sides of the distribution.", "summary_of_the_review": "To sum up, this paper seems to have several technical flaws for the claims to be well supported. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Kr7FewT9Af3", "reviewer": "Reviewer_FpP9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper is generally well written.\n\nThe authors show that CSQ can be efficiently expressed by binary representations, and then be calculated using XNOR operations. I appreciate the details and analysis provided in the main text and supplementary materials.\n\nCSQ has quantization results for mobilenets, which are compact models that generally are harder for quantization. Experiments on large datasets (such as ImageNet) are also helpful to validate the effectiveness of CSQ.\n\nIn addition, some ablation study are meaningful, such as the BRECQ-CSQ experiments in the supplementary material.\n\nDespite the merits, I think the paper still has the following weaknesses:\n1. The scope of application is quite limited. From the paper we can see that CSQ can consistently outperform CLQ on 2-bit, but is not guaranteed to be better on 3-bit or 4-bit. Plus, there are other orthogonal methods that can alleviate the quantization degradation on 2-bit. Although the authors claim the difference between CSQ and previous methods (Section 2.3, Stochastic precision ensemble etc.), the novelty of CSQ is not particularly strong.\n2. In Table 5, the ESQ results mentioned on the caption are missing.\n3. The experimental results of CSQ didn't show great improvement over CLQ. For example, on ImageNet with 2-bit ResNet34, the gain is 0.26%. I wonder if the accuracy gain will remain if advanced quantization-aware training methods (with or without distillation) or mixed-precision methods are applied together with CLQ/CSQ.\n4. Some state-of-the-art results need to be compared with, for example:\n[1] Distribution-Aware Adaptive Multi-Bit Quantization.\n5. The ResNet34 baseline (73.3%) used in the experiment is not state-of-the-art (>74%). I suggest using ResNet50 which is a more common and standard choice.", "summary_of_the_paper": "This paper presents a centered symmetric quantizer (CSQ) that can map floating point tensors into zero-centered quantized integer values. The CSQ method works well for ultra-low bit quantization such as 2-bit.\nThe authors also propose a binary coding method to run efficiently on hardware.", "summary_of_the_review": "I think CSQ is a simple but effective method that can be helpful for ultra-low bit quantization. However, the novelty and application scope are limited and the experiments have room for improvement. Consequently, I think the paper is marginally below the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "K99GLQJiOV", "reviewer": "Reviewer_Z3wR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper seems to be the first to deep dive into the difference between CLQ and CSQ, although these quantization techniques are not new. However, the resulting analysis seems to be straightforward for the following reasons:\n\n- The conclusion that CSQ could be (marginally) better seems to be as expected, since CLQ might not be efficient for representing the signed (or centered) distribution. Although this paper explicitly revealed this fact, it does not seem to provide new insights.\n\n- The analysis based on the representation capacity seems to be weak. Table 3 reveals that the number of distinct output states varies for different combinations of quantizers. Therefore, it does not necessarily prioritize CSQ to CLQ. Furthermore, the importance of the representation capacity is not clear; although CSQ-CLQs has a 50% larger representation capacity for the signed activation, its accuracy gain seems to be marginal (~0.4% from Table 5,6). \n\n- The accuracy results reported in Table 6 seem to be misleading. 1) The authors did not put the accuracy results of LSQ (= CLQ case) for ResNet18 and ResNet34 (67.6% and 71.6%, respectively), which are significantly higher than CSQ. 2) In the case of MobileNetV2, the reported accuracy results CSQ seem to be significantly lower than the state-of-the-art (e.g., PROFIT(ECCV2020) 4-bit MobileNet-V2 = 69.06% compared to CSQ=66.98%). \n\n", "summary_of_the_paper": "This paper analyzed the difference in quantization methods between the conventional linear quantizer (CLQ) and centered symmetric quantization (CSQ). The authors explained that CSQ might exploit the limited quantization states better to represent the data than CLQ, with supporting analysis on representation capacity. The authors further proposed a bitwise implementation of CLQ and CSQ. The evaluation in terms of the quantization accuracy and GPU implementation speed is given for their analysis.", "summary_of_the_review": "This paper seems to provide a better understanding of the low-precision (= 2-bit) quantizers, but the discovery itself seems to be marginally significant. Therefore, I am inclined toward rejection. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: ### Description
The paper investigates the choice of a fixed quantization grid for weights. Namly, the paper observes that symmetric uniform quantization levels such as {-1.5,-0.5,0.5,1.5} lead to better results than non-symmetric ones, e.g. {-2,-1,0,1}. While it is a small thing, it can be appreciated that it is investigated systematically and pedantically, proposing an explanation and showing experimentally that the effect is constantly present in favour of symmetric quantization. While the improvement is small, it comes almost at no cost. A part of the contribution proposes an efficient implementation. 

### Decision
Reviewers and AC came to a consensus that the contribution of the paper is marginal. Symmetric quantization schemes themseleves were already employed by many models, albeit without analysis or even a discussion of such choice. The analysis presented in the paper was found unconvincing by the reviewers (see below). The efficient implementation follows from basic linear algebra (see below). The potential impact of the work was considered as limited due to a rather marginal observed improvement. The average rating of the paper was 4.5. Therefore must reject.

### Details
Regarding the proposed analysis of CSQ, it is not clear, why the number of quantization levels of an elementary product matters, given that these numbers are then summed over all corresponding input channels and spatial dimensions of a convolution kernel applied at a single location. It is questionable whether the number of these quantization levels indeed corresponds to the representation capacity. Finally, the paper misses to demonstrate the effect on binary (1 bit) networks. In this case the standard approach is to use {-1,1} weights and {-1,1} activatinos. The paper could investigate the case of {0,1} activations, where there would be 50% more unique possible outputs from the product, namely {-1,0,1} to validate their hypothesis. If the hypothesis holds, an improvement in the binary case would be observed. This is important since the binary case is know to be the hardest and since the respective recommendation of representations would be non-standard. It could be further questioned why the distribution of real-valued weights has any relevance (such as in the arguments in appendix E) if the model is trained from scratch? A training method need not keep any real-valued latent weights in the first place.

The technical part in section 5 "efficient realization" adds very little, if anything, to the paper's contribution. A simple linear algebra suffices to see that 

$(W-0.5) \ast x = W* x - 0.5 I \ast x,$

where $I$ is the kernel of ones of the same shape as $W$. It is clear that the convolution $I \ast x$ can be implemented efficiently (e.g. it is just a sum over channels followed by a separable spatial only convolution) and is not a bottleneck and. The final detail such as whether to slice by bits and use popcount for it or to use 8-bit addition, depend very much on the choice of the bit-packed representation and the hardware available. It would be known to engineers in the field how to implement it efficiently.
Decision: Reject