Redistribution of Weights and Activations for AdderNet Quantization
Authors: Ying Nie
Kai Han
Haikang Diao
Chuanjian Liu
Enhua Wu
Yunhe Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: AdderNet
Quantization

Abstract: Adder Neural Network (AdderNet) provides a new way for developing energy-efficient neural networks by replacing the expensive multiplications in convolution with cheaper additions (i.e., L1-norm). To achieve higher hardware efficiency, it is necessary to further study the low-bit quantization of AdderNet. Due to the limitation that the commutative law in multiplication does not hold in L1-norm, the well-established quantization methods on convolutional networks cannot be applied on AdderNets. Thus, the existing AdderNet quantization techniques propose to use only one shared scale to quantize both the weights and activations simultaneously. Admittedly, such an approach can keep the commutative law in the  L1-norm quantization process, while the accuracy drop after low-bit quantization cannot be ignored. To this end, we first thoroughly analyze the difference on distributions of weights and activations in AdderNet and then propose a new quantization algorithm by redistributing the weights and the activations. Specifically, the pre-trained full-precision weights in different kernels are clustered into different groups, then the intra-group sharing and inter-group independent scales can be adopted. To further compensate the accuracy drop caused by the distribution difference, we then develop a lossless range clamp scheme for weights and a simple yet effective outliers clamp strategy for activations. Thus, the functionality of full-precision weights and the representation ability of full-precision activations can be fully preserved. The effectiveness of the proposed quantization method for AdderNet is well verified on several benchmarks, e.g., our 4-bit post-training quantized adder ResNet-18 achieves an 66.5% top-1 accuracy on the ImageNet with comparable energy efficiency,  which is about 8.5% higher than that of the previous AdderNet quantization methods. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/AdderQuant.

Redistribution of Weights and Activations for AdderNet Quantization
Ying Nie!
Kai Han
Haikang Diao² Chuanjian Liu' Enhua Wu Yunhe Wang!"
HHuawi Noah's Ark Lab
State Key Lab of Computer Science, ISCAS & UCAS
$ School of Integrated Circuits, Peking University University of Macau fying nie, kai. han, yunhe. wang/ com
Abstract
Adder Neural Network (AdderNet) provides new way for developing energy- efficient neural networks by replacing the expensive multiplications in convolution with cheaper additions (i.e., (n-oo))). To achieve higher hardware efficiency, it is necessary to further study the low-bit quantization of AdderNet. Due to the limitation that the commutative law in multiplication does not hold in f--nrmm the well-established quantization methods on convolutional networks cannot be applied AdderNets. Thus, the existing AdderNet quantization techniques propose use only one shared scale to quantize both the weights and activations simultane- ously. Admittedly, such an approach can keep the commutative law the -norm quantization process, while the accuracy drop after low-bit quantization cannot be ignored. To this end, we first thoroughly analyze the difference on distributions of weights and activations in AdderNet and then propose new quantization algo- rithm by redistributing the weights and the activations. Specifically, the pre-trained full-precision weights in different kernels are clustered into different groups, then the intra-group sharing and inter-group independent scales can be adopted. To further compensate the accuracy drop caused by the distribution difference, we then develop a lossless range clamp scheme for weights and a simple yet effective outliers clamp strategy for activations. Thus, the functionality of full-precision weights and the representation ability of full- pel-iircciioon activations can be fully preserved. The effectiveness of the proposed quantization method for AdderNet is well verified on several benchmarks, e.g. our 4-bit post-training quantized adder ResNet- 18 achieves 66.5% top-1 accuracy the ImageNet with com- parable energy efficiency, which is about 8.5% higher than that of the previous AdderNet quantizaiin methods. Code will be available at at/https: htttt:://gitte.e com/ mundppen/nobaas/rrr////teeeeeeee htt.://saarrrcccoo//dduuuaaaaaa
Introduction
Convolutional neural networks (CNNs) have achieved extraordinary performance on many vision tasks. However, the huge energy consumption of massive multiplications makes it difficult to deploy CNNs on portable devices like mobile phones and embedded devices. As a result, substantial research efforts have been devoted reducing the energy consumption of CNNs 12
16 R
32
37 29
Beyond pioncering model compression approaches, Chen et al. 4 advocate the use of & -norm instead of cross-correlation for similarity measure between weights and activations in conventional convolution operation, thus produces a series of adder neural networks (AdderNets) without massive
"Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Activations (FP)
Waight (FP)
Activations (NT44)
Waights (NNT))
Scales (FF)
:Eq.
Adder Conv
sEq. 2
clamps
Wi
XSx
X5
W2
We
x55x
(a) The input full precision activations and weights
or
Activations (INTA)
Weights (INT4)
Scales (FF)
Vanilla Conv
Bits waste
X
Wi
Sx Swi
X
W>
Sx Sw2
W3
x
Sr Swa De-Quant.
Quant
X
Quant
De-Quant
(b) Quantization for vanilla conv
(c) Quantization for adder conv
Figure 1: Comparisons of quantization methods of vanilla convolution and adder convolution (symmetric 4-bit an example). The independent scales adopted by vanilla convolution quantization can properly handle the challenge of large differences in weights and activations. One shared scale adopted in existing adder convolution quantization methods cannot handle this challenge.
number of multiplications. Compared with multiplication, addition is more energy efficiency, e-g- full-precision (FP) multiplication takes about 4x energy as full-precision addition [15]
39
33 23),
AdderNets have shown impressive performance in large scale image classification 4 38 object
detection [5] elc. which shed light into the design of low-power hardware for Al applications.
The operation of low-bit addition has much lower energy consumption than the full-precision addition [T5] 39 33 23]. To further improve the efficiency of AdderNet, low-bit quantization is a
crucial option. Quantization algorithms have been studied in conventional convolution networks
for a long time and achieved remarkable performance [20] 1 IT] 28 24 18 10 36 30]. As the
law of commutation holds in the quantization process of multiplication, the scales of weights and
activations for quantization can be independent of each other (Eq. ( In practice, weights tend to be quantized with multiple scales based on the output channels, while activations scale based on the whole layer (Figure (b)). However, the law of commutation does not hold in the é fi-norm quantization, which restricts the weights and activations in AdderNet be quantized with shared scale (Eq. TO). Therefore, existing AdderNet quantization techniques [55] [5] adopt only one shared scale to quantize both the weights and activations simultancously. practice, the shared scale is calculated based on the activations of the whole layer (Top of Figure (c)) or the weights of the whole layer (Bottom of Figure (c)) However, given pre-trained full ail--rrcision AdderNet model, the ranges of weights usually vary widely between output channels, and the ranges of activations and weights also vary widely, both of which pose huge challenge for quantization. If the range of activations is adopted to quantize weights, a large proportion of weights will be clamped, i.e., over clamp. If the range of weights is adopted quantize activations, large proportion of precious bits will be never used, i.e., bits waste. In both cases, poor quantized accuracy will be incurred. In contrast, multiple independent scales adopted in the vanilla convolution can avoid this challenge.
quantized with one
To enhance the accuracy of quantized low-bit AdderNets, we first thoroughly analyze the problems of the one shared scale paradigm for quantization in existing AdderNet quantization methods. Then, we develop novel quantization algorithm for AdderNet. Specifically, we first propose a scheme of group-shared scales with negligible increase in computational workload address the challenge that the ranges of weights vary widely between output channels. The pre-trained full-precision countization-oriennte clustering process. Since the
weights are divided into multiple groups by weights within group are similar, the mntra-group sharing and inter-group independent scales are thus used our method. further improve the performance of quantized adder neural network, we introduce a lossless range clamp scheme for weights to further reduce the difference on distribution of activations and weights AdderNet. In practice, for groups where the range of the weights exceeds the range of the activations, clamp the weights to the range of activations and then incorporate

---

the clamped values into the following bias term. Next, for those activations that contain outliers, simple yet effective outliers clamp strategy for activations is presented eliminate the negative impact of outliers. The extensive experiments conducted on several benchmarks demonstrate the effectiveness of the proposed quantization method.
2
Preliminaries and Motivation
In this section, we briefly revisit the necessary fundamental of adder neural network and existing quantization process for conventional CNN and AdderNet, then we describe the motivation.
2.1
Preliminaries
Adder Neural Network (AdderNet). Conventional convolution uses massive multiplications for computation, while adder convolution utilize addition to replace multiplication for reducing computa- tional cost. Given the input activations X E IR Hx xein and the weights W E Bdxxxxii XCout the traditional convolutional operation is defined as:
nm),
n, X
ΣΣΣ X(m + in + X ((i,j,kee) =1j=1km1
where H and W are the height and width of activations, c is the index of output channels, d x d denotes the kernel size, Gin and Cout are the number of input and output channels, respectively.
To avoid massive floating number multiplications, Chen et al. 4 advocate the use of -norm instead of cross-correlation for similarity measure between activations and weights in CNN:
Y (m, c) X W
[m + ii + 1. (i,j,k,
(2)
where |- the absolute value function. Since Eq Eq.2 only contains addition operation, the energy costs of the adder neural network can be effectively reduced [15 33 39]. With modified vack-propagation approach and adaptive learning rate strategy, AdderNet achieves satisfactory performance on image
classification 4 38 35), object detection [5), etc.
Quantization for CNN. The common uniform quantization for CNN linearly maps full- null-precision real numbers into low-bit integer representations, which is preferred by hardware efficiency. We take the symmetric mode without point example describe the procedure of quantization. Given a full-precision input value e and bit-width b, the quantized value T can be defined as:
i == clamp cllmpp("//]l.lllll In. 9n:
3
where the scale s = * max(lo))/(22 1), On = - and Op 2h-1 L-1 denotes the round function and clampp(z , indicates that value is clamped between ri and r2. Correspondingly, de-quantized data can computed by:
x s.
(4)
And the quantization loss can be defined as:
T Iaquant = 10 vl.
(5)
Since the commutative law holds in multiplication, the multiplication operation of weights and activations in the conventional convolution can be converted as follows:
& ~ X @ W (s @ Ww = (s2Sw) x (X @ W),
where @ denotes the vanilla convolution operation in Eq. [ In common settings 24 19 20 6
the weights adopt independent s for each output channel, and all activations in layer share one 8g- From Eq. 6 the data format of vanilla convolution operation is changed from full-precision to low-bit integer, which can reduce the energy consumption effectively.
Quantization for AdderNet. Due to the fact that the commutative law does not hold in adder convolution, the independent scales sw and s, cannot be adopted when quantifying weights and activations in adder convolution as we do in conventional convolution, that is:
X@ W ~ X W (s + (sw * (Sw x (X + W),
3

---

where @ denotes the adder convolution operation in Eq.22 Therefore, the existing works on AdderNet quantization 335 all adopt one shared scale s for quantifying weights and activations. Thus, the quantized and de-quantized weights are defined as:
W clamp( L /=]. qwn: 9wp), W = sW,
(8)
similarly, the quantized and de-quantized activations are defined as:
X clamp( /x], qan, {xp), = sX,
where s = 2 * mar mar([x)) ( - 1) or = 2 * max W 1) - 1). Therefore, the quantized adder
convolution can be implemented:
Xe W X W = (sX) @ (sW) (X + W).
(10)
2.2 Motivation
Empirically, the values of weights and activations tend to vary widely in pre-trained full-precision AdderNet. We take the knyerI.1.come in pre-trained adder ResNet-20 [14] as an ex- ample to visualize the statisties the absolute ranges of activa- tions and weights in Figure 2 Obviously, the absolute ranges of weights vary widely between output channels, and the ab- solute ranges of activations and weights also vary widely, both of which pose a huge challenge for the quantization with one shared scale. After thorough analysis, we conclude that when weights and activations in AdderNet are quantized with one shared scale, no matter whether the scale is calculated based on weights or activations, a large quantization loss (Eq. 5 will be incurred, resulting in a poor accuracy, especially in the case of low bits. For example, the Top-1 I accuracy loss of 5-bit quantized adder ResNet- 18 on the ImageNet is 3.3%, while the accuracy loss of 4-bit quantized adder ResNet-18 18 increases to 9.9% (35].
12,
010 90)
4.8
48%
110, 20
19.9%
1)
75.3%
79.255
(a) Activations
(b) Weights
Figure 2:
The statistics of the ab- solute ranges of activations and weights in the pre-trained full- precision AdderNet.
Proposition 1. Denote and Tw as the ranges of activations and weights in AdderNet, respectively, r is adopted to quantize weights, a large proportion of weights will be clamped. If is adopted to quantize activations, large proportion of precious bits will be never used. In both cases, large quantization loss is incurred, resulting in a poor accuracy.
Proof. For the first case where r2 is adopted to quantize weights, according to Eq. 8 the detailed quantization of weights can be formulated as:
W clamp([we' - 1)/2ro -2 12--11 1).
(11)
Without losing generality. suppose W 50r,. then [W 11/2r=1 25 1), 25 1) is
clamped between -26-1 and 2--1 then the proportion of the clamped value is (25 (2) 1)
2h-1) 1/25 (2th - 1)) (49 2h-1 (5))(50) 2 25) 0.98. Besides, the quantization loss IW W| ~ Ir.- 50r,| 49r.. which will result in a significant accuracy degradation.
For the second case where Tw is adopted to quantize activations, according to Eq. 9 the detailed quantization of activations can be formulated as:
X clamp((xxe 11/2r. 1, -- 20 1).
(12)
Similarly, suppose X = 0.02r w. then [x(2* 112h-1177 1)/2rw 10.01(2' - -1)1, if we clamp (0.01 1)] between -2h and 2 the remaining 2 10.01(2 -1)1 = 0.98 * 2 bits will be never used, which huge waste for precious bits.
3 Approach
In this section, we will describe the proposed quantization algorithm for AddenNet in detail. Firstly, we introduce the method of group-shared scales based on clustering. Then, the clamp for weights and activations are proposed, respectively.

---

[a] Clustering- basen Weights Grouping
(b) Outliers Clamp for Acts
0.5
1.2
-0.7
-122
0.5
06
-1.2
0.5
00.
066
-14
12
168
12.3
-154
1.0
-14
12
1.1
16.8
12.3
-154
-15
0.0
06
113
21.0
-19.2
-1.0
W3
-1.
1.3
W2
04
0.8
WW3
-04
0.8
1.6
X
10.0
18.4
1622
W/
(d) Quantization
Bias
-8
7
(c) Range Clamp for Weights
Bias
0.5
1.2
-0.7
8
-7
-5
-1.2
0.5
-0.8
X
0.6
-14
1.2
11
1.6
1.6
-1.6
-7
6
7
-8
WA
066
-1.0
W2
1.6
16
-16
1.3
W>
XS22
-26.2
-0.4
0.8
1.6
-16
1.6
1.6
-126.2
2
4
7
-8
7
Quant De-Quant
Ws
Figure 3: An illustration of the proposed quantization method for AdderNet (symmetric 4-bit example). The pre-trained full-precision weights are clustered into different groups. Then the clamp scheme for weights and activations explored respectively make efficient use of the precious bits and eliminate the negative impact of outliers.
3.1 Clustering-baset Weights Grouping
To address the challenge that the ranges of weights in pre-trained full-precision AdderNet vary widely between output channels, we propose a novel quantization method with group-shared scales illustrated in Figure (a). Formally,
X @ W concat(on conca((o,,.....)). 0g),
(13)
where Oj 8j (X [[s:;;,,)).
where j = 1 is the index of groups, 9 is the number of pre-defined groups. I, indicates a set of indexes of weights belonging to the j-th group, and 8j indicates the j-th scale. The set Il is achieved by clustering the weights in the pre-trained full-precision model. Since the scale 8 = 2 max( max((v))/(2²
1), which indicates that and the maximum of [o] are positively correlated. Thus, cluster based the maximum of [v]. all the elements of v. In practice, the absolute maximum in each output channels of weights are taken as the clustering feature. Formally, the generated feature vector of weights denoted as fe....!
Se max([W(::;;;;: [
(14)
where e 1,, .,., Cout is the index of output channels. Next, the feature vector is divided into pre- defined clusters I I= (I1,, I2,,.., Ig... Any pair of points within each cluster should be as close to each other as possible:
min
Σ 1oc 13
(15)
j=1cE1,
where μú is the mean of points in cluster Ily- The method like k-means 13 can be adopted for clustering weights with the objective function in Eq. 15 Finally, the intra-group sharing and inter- group independent scales can be adopted for quantization. In practice, for groups where the range of the weights is within the range of the activations, the difference between the range the weights and the activations is usually small, the scale calculated based on the range of weights can be adopted for quantifying both the weights and activations. Conversely, for groups where the range of weights exceeds the range of activations, the range of weights and activations tends to differ significantly, making it difficult to quantize both weights and activations with one shared scale, which will be addressed in the next subsection.
Analysis on Complexities. Compared to quantifying weights and activations with only one scale, the proposed quantization scheme with group-shared scales can improve the performance of quantized AdderNet, but it will result small increase in Floating Point Operations (FLOPs). After thorough analysis, conclude that the increased FLOPs is negligible.

---

Before Camp
After Clamp
Adder activations
Adder weights
Adder activations
Adder weights
Adder bias
Figure 4: The distributions activations and weights pre-trained full-precision AdderNet.
Proposition 2.
Without losing generality, suppose Cout Can C, then the additional FLOPs ratio introduced by the group-shared scales approximately equal to 1/(6c + 1), which is negligible.
The detailed proof is provided in the supplementary material under the assumption that 9 considering that the magnitude is generally in the tens or hundreds for common networks, thus the additional FLOPs ratio very small.
3.2 Clamp for Weights and Activations
To address another challenge that the ranges of activations and weights in pre-trained full-precision AdderNet vary widely, more specifically, the range of weights far exceeds the range of activations (Left of Figure we present lossless range clamp scheme for weights (Figure (c)). Besides, a simple yet effective outliers clamp strategy for activations also introduced eliminate the negative impact of outliers (Figure (b)).
Range Clamp for Weights. A lossless range clamp scheme for weights is first proposed. Formally, denote We as any one output channel weights, i.e., W. = W|:,;, i, cl. E [1,.... Cout), Tn and Tw as the ranges of activations and weights, respectively. In practice, we clamp the full-precision values of We to the range of [-Ta, Tz]. then add the clamped values to the following bias
Theorem 1. The proposed range clamp scheme for weights has no any effect on the accuracy, i.e., lossless. Formally,
+ W. X @ clamp(W., -2,
where = ΣE
Cun
max([Wali.....mmm All - r2,0). 0).
(16)
j=1
Proof.
esn
d d cin
X
ΣΣ>
( (m W.(..A.)
>Σ>>> ==
S(, (x,,,((,j.))), (17)
1=1 1=1
where f(X, S.,,W.(,,. k)) can be divided into three cases:
if (.(,. Fa then f(X, Wc(,,,,k) (r. +
r2).
if -rx (e(i.j, k) S then -
if W.(L,j,k) - then f(X, Wc(i.,
The above three cases be unified into:
W. X @ Clamp(W., -w,
max max(WW.l..j.kk) 0).
(18)
km
After the lossless range clamp process for weights, Fw is equal to rz. Thus, directly adopt rz to quantize the weights and activations.
Outliers Clamp for Activations. As shown in the left of Figure 4 some activations clearly have outliers. avoid the negative impact of these outliers during quantization, we propose simple yet effective outliers clamp strategy for activations. Specifically, the absolute activations are denoted as

---

x= f=oool...., / } and the sorted X in ascending order is denoted as X, where n is the number of the values. We select the value r. a (n 1)]] as the range of activations for the calculation of scale, where a E (0, 1] is a typer-parameet controlling the ratio discarded activations. Similarly, there is another option, i.e,, rr = X/n 1] a. However, this method can only reduce the negative impact of outliers, whereas ours can eliminate the negative impact of outliers.
4 Experiments
In this section, we conduct extensive experiments. Thorough ablation studies are also provided.
4.1 Experiments on CIFAR
Re-train FP Networks. Following the pioneering work of AdderNet 4 we first conduct exper- iments on CIFAR-10 10 and CIFAR-100 [21]. The CIFAR-IO1 10/100000000-0 dataset consists of 60,000 color images 10/100 classes with 32 x 32 size, including 50,000 training and 10,000 validation images. The training images are padded by 4 pixels and then randomly cropped and flipped. We re-train the three relevant full-precision networks, including VGG-Small B], ResNet-20 [I4] and ResNet-32 [[4], according to the experimental settings in AdderNet [4]. Specifically, the first and last layers of the network are kept as conventional convolution, and the other layers are replaced with adder convolu- tion. We employ learning rate starting 0.1 and decay the learning rate with cosine learning rate schedule. SGD with momentum of 0.9 is adopted as our optimization algorithm. The weight decay is set to 5 x 10-4, We train the models for 400 epochs with a batch size of 256. The final results provided in in the supplementary material, and are basically consistent with the results in [4. The baseline results of CNN and binary neural network (BNN) [1] are cited from
Low-bit Quantization. Following the general setting quantization quantize the first and the last layers. The proposed quantization method can be employed in either post-training quantization (PTQ) or quantization-awe aware training (QAT). In practice, when the bits is 4, we begin to perform QAT procedure. Unless otherwise specified, the number of groups we use is 4. The hyper-parameter a controlling the ratio of discarded outliers in activations is empirically set to 0.999 in CIFAR experiments. In the QAT procedure, we keep the same settings the FP training, except that the learning rate is changed to 1 x 10-4, and only 50 epochs are used for saving the training time. The trriigh-throouge estimator 2 is adopted for avoiding the zero- vero-gradiens problem.
27 25
we do not
Table 1: Quantization results of AdderNets CIFAR- 10 and CIFAR- 100 datasets.
CIFAR (0)) (%)
CIFAR- 100 (%)
Model
Bits
PTQ
QAT
PTQ 73.56 (-0.04)
QAT
93.42 (-0.02)
6
93.48 (+0.04)
73.62 (+0.02)
VGG-Small
5
93.41 (-0.03)
73.50 (-0.10)
4
93.20 (-0.24)
93.46 (+0.02)
73.15 (-0.45)
73.58 (-0.02)
91.40 (-0.02)
67.58 (-0.01)
6
91.32 (-0.10)
67.63 (+0.04)
ResNet-20
5
91.36 (-0.06)
67.46 (-0.13)
90.72 (-0.70)
91.21 (-0.21)
65.76 (-1.83)
67.35 (-0.24)
92.76 (+0.04)
70.46 (40.29)
92.62 (-0.10)
70.08 (-0.09)
ResNet-32
5
92.59 (-0.13)
70.09 (-0.08)
91.73 (-0.99)
92.35 (-0.37)
68.24 (-1.93)
69.38 (-0.79)
The quantization results of AdderNets are reported in Table For all networks, there is almost no accuracy loss in 8-bit quantization compared to the full-precision counterparts. For example, the accuracy of 8-bit quantized ResNet-32 is even 0.29% higher. For the 6-bit and 5-bit quantization, the accuracy loss exists, but is still within an acceptable range. For the more challenging b-bit quantization, the accuracy loss of PTQ increases significantly. Therefore, QAT procedure is necessary this case. After 50 epochs training, the accuracy loss was greatly reduced. For example, in the case of 4-bit ResNet-20 on CIFAR-100, the accuracy loss is reduced from 1.83% to 0.24%. If equipped with more refined QAT techniques, such as FWGS 22 KD 27 or simply a longer training epoch, the accuracy loss can be further reduced, but is not the focus of this paper.
7

---

4.2 Experiments on ImageNet
Re-train FP Networks. We further conduct evaluation on ImageNet dataset [9. which contains over 1.2M training images and 50K validation images belonging to 1,000 classes. The proccssscs and data augmentation follow the same protocols as in [14]. We re-train two full-precision networks including ResNet-18 18 and ResNet-50 [4] according to the experimental settings 44. The entire training takes 150 epochs with weight decay of 1 10-4 and the other same as that in CIFAR experiments. The final results are provided the supplementary material and the baseline results of CNN and BNN are also cited from [4]
is the
Low-bit Quantization. We adopt the same quantization settings discussed in CIFAR experiments except that the hyper-parameter a is set to 0.9992. Besides, in the QAT procedure, we also keep the
same settings as the full-training training, except that the learning rate is changed to 10-3 and only 20 epochs are used for saving training time.
Table Quantization results of AdderNets on ImageNet.
Top-1
Acc (%)
Top-5 (%)
Model
Bits
PTQ
QAT
PTQ
QAT
8
67.7 (-0.2)
87.7 (-0.1)
6
67.7 (-0.2)
87.6 (-0.2)
ResNet-18
67.4 (-0.5)
87.3 (-0.5)
66.5 (-I.44
(-0.5)
86.6 (-1.2)
(-04
8
75.0 (-0.0)
91.9 (-0.0)
6
75.0 (-0.0)
91.8 (-0.1)
ResNet-50
74.7 (-0.3)
91.5 (-0.4)
(-1.8)
73.7 (-1.3)
90.7 (-1.2)
90.9 (-1.0)
The quantization results are reported in Table For both networks, there almost no loss in 8-bit and 6-bit quantization compared the full-precision counterparts. For the 5-bit quantization, the accuracy loss is still within an acceptable range, i.e., a maximum loss of 0.5%. Similar to CIFAR experiments, QAT is adopted in 4-bit quantization. After 20 epochs training, the accuracy loss was greatly reduced. For example, the accuracy loss of 4-bit ResNet-18 is reduced from 1.4% to 0.5%.
4.3 Comparisons with Sfate-of-the-arts
Image Classification. We compare the proposed method with the existing AdderNet quantization technique QSSF [35] on image classification task. We first compare the PTQ results on ImageNet with it. The detailed Top-1 accuracy loss comparison with full-precision Adder ResNet-18 18 are reported in Figure (a). QSSF (35) simply adopt only one shared scale quantize both the weights and activations simultaneously, which ignores the properties the distribution of the weights and activations of AdderNet, leading the problems of "Over clamp" and "Bits waste", further leading to a poor accuracy. In contrast, we propose a quantization method consisting of three parts: clustering- based weights grouping, lossless range clamp for weights and outliers clamp for activations. The problems of "Over clamp" and "Bits waste" can be effectively resolved with our quantization method, further leading high quantized accuracy. As the number of bits decreases, the advantages of our method become more pronounced. For example, in the case of 4-bit, our method achieves 8.5% higher accuracy than QSSF (35]
05
44
1.15
Oure-Train Loss
75
0.7
1.10
geg-Train Loss
70-
-3.5
2 00
0.95
0.90
60
0ura 053F
0.85
Bure-train Icc
3.9
10
20
30
50
55
Bits
Epoch
Epoch
(a) Comparison of Adder 18 on ImageNet
(b) Comparisan of Adder ResNet-20 on CIFAR-100
Figure 5: Comparisons with QSSF 35 on image classification task.

---

In addition, we also compare the QAT results on CIFAR-100 100 with it. In Figure (b), we visualize the training loss, training accuracy and testing accuracy of the 4-bit Adder ResNet-20. For the accuracy curve, the solid lines and dash lines represent the testing accuracy and training accuracy, respectively. Our quantization method achieves higher accuracy on both training and testing. Specifically, under the same epoch, our method achieves 67.35% test accuracy, which is higher than 64.61% % achieved by QSSF [35].
Object Detection. AdderDet [ presents an empirical study of AdderNet on object detection task. Besides, the PTQ mAP with one shared scale of 8-bit Adder FCOS +4 is also reported in this work, i.e,, 36.2 on COCO val2017 [26], which is 0.8 lower than the full- full-precision counterpart. To verify the generality of our quantization method on detection task, we conduct the PTQ experiment of the 8-bit Adder FCOS following the settings in AdderDet [5]. Specifically, the adder convolutions in a set to
the backbone and neck are quantized with 4-group shared scales, and the 0.9992. The final 8-bit quantized mAP with our method is 36.5, which is 0.3 higher.
Energy Cost. In Figure 6 we compare the accu- racy and energy cost of different network architectures after quantization, including AdderNet, CNN and Shif- IAddNet [39]. The experiments are based on ResNet-20 network and CIFAR- 100 dataset. We follow the related
Shit
6Dit
Bbit
Sbit
4bit
4bit
works [39 35 23 to measure the practical energy cost
(1)
of ResNet-20 with various bits and basic operations
3bit
ONN
on FPGA platform. Besides, the quantized results of CNN achieved by the BRECQ (24) PTQ method.
62
StAdtAN
The quantized accuracy of ShiftAddNet is cited from the original paper [39] and the energy ShiftAddNet is calculated from the relative ratio of the energy of ShiftAddNet and AdderNet the original paper [39], ie., 22.2%%. From Figure 6 under similar energy cost,
abit
1.5
2.0
2.5
3.5
Energy
Figure 6: Comparisons of accuracy and en-
ergy cost of ResNet-20 on CIFAR-100. the quantized Adder ResNet-20 with our method achieves higher accuracy.
4.4 Ablation Studies
In this subsection, we conduct thorough ablation studies based on the 4-bit quantized Adder ResNet-20 network CIFAR-100 dataset.
Analysis on Sub-Methods. We first conduct analysis on three sub-methods as reported in Table 3 including group-shared scales, lossless range clamp for weights and outliers clamp for activations. The complete method achieves the highest accuracy of 67.35%, which is higher than the initial accuracy of 57.88%. Besides, the introduction of any sub-method contributes to the improvement of accuracy.
Table 3: Analysis on sub-methods.
Method
Croup-Shared Scales
Weights Clamp
Activations Clamp
Acc (%)
57.88
60.42
62.30
61.55
65.17
67.35
Analysis on Group Number. The results with various group number are reported in Table 4 and the relative FLOPs are calculated under the assumption that the number of channels of input and output is 32. From Table 4 when the group number is 4, the quantized model achieves the highest accuracy with negligible increase in FLOPs. More group does not lead to higher accuracy.
Analysis on Group Methods. We also conduct analysis on the group methods. The group number we used is 4, and the results are reported in Table 5 Uniform indicates that the weights are evenly grouped based on the output channel without clustering. Compared with other group methods, the clustering method based on maz([W) achieves the highest accuracy.

---

Table 4: Analysis on the group number.
Table 5: Analysis on group methods.
Group
Relative FLOPs
Acc (%)
Group Method
Ace (%)
63.91
Uniform
65.72
1.002
66.38
All
66.61
4
1.005
67.35
Mean 66.86
1.0T2
67.11
Max 67.35
Conclusion
To further improve the efficiency of AdderNet, this paper studies the quantization algorithm for AdderNet. Considering that the weights of different channels may differ greatly, and the ranges of weights and activations are different, propose scheme of group-shared scales and clamp strategy for weights and activations to improve the performance of quantized AdderNet. The effectiveness of the proposed quantization method for AdderNet is well-verified state-of-the-an results on various quantized network architectures and datasets. We believe that the advantages of low-bit quantized AdderNet can shed light into the design of low-power hardware for AI applications.
several benchmarks. We achieve
Acknowledgement
We gratefully acknowledge the support of MindSpore (17), CANN(Compute Architecture for Neural Networks) and Ascend AI Processor used for this research. This research is supported by NSFC (62072449, 61972271); National Key R&D Program of China (2020YFC2004000; University of Macau Grant (MYRG2019-0006-FFST)
References
[1] Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment. Advances in Neural Information Processing Systems, 32, 2019.
[2] Yoshua Bengio, Nicholas Léonard, Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1rXiv:1308.3432. 2013.
Zhaowei Cai. Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by hale-wave gaussian quantization. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5918-5926, 2017.
[4] Hanting Chen, Yunhe Wang, Chunjing Xu, Boxin Shi, Chao Xu, Qi Tian, and Chang Xu. Addernet Do really need multiplications in deep learning? In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1468-1477, 2020.
[5] Xinghao Chen, Chang Xu, Minjing Dong, Chunjing Xu, and Yunhe Wang. An empirical study of adder neural networks for object detection. Advances Neural Information Processing Systems, 34, 2021.
[6] Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv: 805,06085, 2018.
[7] Kanghyun Choi, Deokki Hong, Noscong Park, Youngsok Kim, and Jinho Lec. Qimera: Data-free quantization with synthetic boundary supporting samples. Advances in Neural Information Processing Systems, 34, 2021.
[8] Matthieu Courbariaux, Yoshua Bengio, and lean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems. pages 3123-3131 2015.
[9] Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and puttern recognition, pages 248-255. leee, 2009.
[10] Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer Hawq v2: Hessian aware mace-weighted quantization of neural networks. Advances in neural information processing systems, 33:18518 13118518-18529 2020.
10

---

[11] Steven K Esser, Jeffrey L McKinstry. Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. Learned step size quantization. arXiv preprint urXiv:1902.08/53, 2019.
[12] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICI.R, 2016.
[13] John A Hartigan and Manchek A Wong. Algorithm as 136: A k-means clustering algorithm. Journal of the royal statistical society. series (applied statistics), 28(1):100- 108, 1979.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[15] Mark Horowitz. computing's energy problem (and what we can about it). 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pages 10-14. IEEE, 2014.
[16] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 17X0:1170.00861,,1
[17] Huawei. Mindspore,
cn/| 2020.
[18] Iay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran EI- Yanya, and Yoshua Bengio. Binarized neural networks. Advances in neural information processing systems, 29, 2016.
[19] Benoit Jacoh, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer- arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704-2713, 2018.
[20] Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv: 083402 2018
[21] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[22] Junghyup Lee, Dohyung Kim, and Bumsub Ham. Network quantization with element-wise gradient scaling. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6448-6457, 2021.
[23] Wenshuo Li, Hanting Chen, Mingqiang Huang. Xinghao Chen, Chunjing Xu, and Yunhe Wang. Winograd algorithm for addernet. In Proceedings of the International Conference on Machine Learning, pages 6307-631 1007-6315,220 2021.
[24] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang. Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2102.05426, 2021.
[25] Mingbao Lin, Rongrong Ji, Zihan Xu, Baochang Zhang, Yan Wang, Yongjian Wu, Feiyue Huang, and Chia-Wen Lin. Rotated binary neural network. Advances neural information processing systems, 33:7474-7488 2020.
[26] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[27] Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In European Conference on Computer Vision, pages 143-159. Springer. 2020.
[28] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pages 7197-7206. PMLR, 2020.
[29] Ying Nie, Kai Han, Zhenhua Liu, An Xiao, Yiping Deng, Chunjing Xu, and Yunhe Wang. Ghostsr: Learning ghost features for efficient image super-resolution. arXiv preprint preivirepriit arXiv:2101.0852 2021.
[30] Ying Nie, Kai Han, and Yunhe Wang. Multi-bit adaptive distillation for binary neural networks.
[31] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision, pages 525-542. Springer. 2016.
11

---

[32] Gil Shomron, Freddy Gabbay, Samer Kurzum, and Uri Weiser. Post-training sparsity-aware quantization. Advances in Neural Information Processing Systems, 34, 2021.
[33] Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S Emer. Efficient processing of deep neural networks: A tutorial and survey. Proceedings of the IEEE, 105(12)2295--229, 2017.
[34] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9627-9636, 2019.
[35] Yunhe Wang. Mingqiang Huang, Kai Han, Hanting Chen, Wei Zhang, Chunjing Xu, and Dacheng Tao. Addernet and its minimalist hardware design for energy-efficient artificial intelligence. arXiv preprint arXiv:2101. 10015, 2021.
[36] Yixing Xu, Kai Han, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Learning frequency domain approximation for binary neural networks. Advances in Neural Information Processing Systems, 34:25553-25565, 2021.
[37] Yixing Xu, Yunhe Wang, Hanting Chen, Kai Han, Chunjing Xu, Dacheng Tao, and Chang Xu. Positive- unlabeled compression on the cloud. Advances in Neural Information Processing Systems, 32, 2019.
[38] Yixing Xu, Chang Xu, Xinghao Chen, Wei Zhang, Chunjing Xu, and Yunhe Wang. Kernl based progressive distillation for adder neural networks. In Proceedings of the Neural Information Processing Systems, 2020.
[39] Haoran You, Xiaohan Chen, Yongan Zhang, Chaojian Li, Sicheng Li, Zihao Liu, Zhangyang Wang. and Yingyan Lin. Shiftaddnet: hardware-inspired deep network. Advances in Neural Information Processing Systems, 33:2771- 2783, 2020.
[40] Dongging Zhang, Jiaolong Yang. Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pages 365-382, 2018.
[41] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Derefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv: 1606.06 60. 2016.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Section D.
(b) Did you describe the limitations of your work? [Yes] We have discussed the limitations in the section of limitations and societal impacts in the supplemental material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We have discussed the potential negative societal impacts in the section of limitations and societal impacts in the supplemental material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 2.2 Section and Section 3.2..
(b) Did you include complete proofs of all theoretical results? [Yes] See Section 2.2 Section 3.riionn.nnam and Section
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1 and Section 4.2)
(c) Did you report error bars (e-g., with respect to the random seed after running experi-
ments multiple times)? [No] The common settings on ImageNet.
12

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'le 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "mhvJdh4d-_e", "reviewer": "Reviewer_2x7U", "summary": "This paper first thoroughly analyzes the difference in distributions of weights and activations in AdderNet and then proposes a new quantization algorithm by redistributing the weights and the activations.", "questions": "1. The authors show the quantization results from 8bit to 4bit, and I am curious about the accuracy on lower-bit quantization, e.g., 3bit?\nFig. 6 shows energy and accuracy comparisons with CNN quantized via BRECQ PTQ method. Could you show comparisons with more CNN quantization methods?\n\n2. Is the quantization bit of output (before adding the bias) consistent with that of input and weight? Do you also quantize gradients and errors during training?\n\n3. The authors cluster weights in AdderNet into several groups, and then leverage different inter-group scale factors to quantize both weights and input. By doing this, I am concerned about whether it will consume more energy to transfer the aforementioned different quantized inputs when deployed onto mobile devices.\n\n4. Do you have any plan to open source the kernel implementation of AdderNet and corresponding quantization schemes?\n\n---\nAfter reading the author's post, most of my questions are resolved and I am tending to accept it. The new experiments or tables can be added to the revision.", "limitations": "The comparison with CNN quantization method seems to be not very adequate.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. This paper conducts a thorough study of the dilemma in AdderNet quantization, and proposes an effective method to solve this problem. \n2. The paper is clearly presented.\n3. I am glad to see that the accuracy drop is within 2% for ImageNet even at 4 bits.\n\nWeaknesses (suggestions): \n\n1. The accuracy and energy comparisons with quantized CNN seem to be not very adequate. Better to compare the accuracy drops in quantized CNNs as well as the currently presented accuracy drop in AdderNet. So the reader can get the full information whether AdderNet is more advanced as compared to CNNs in terms of quantization.\n\n2. AdderNet is a specific neural network, it is not clear whether the proposed methods can be generalized to other neural networks will similar distribution properties.\n\n3. Only classification results are shown, how about other downstrem tasks, e.g., detection, segemetation. Or other network architectures, e.g., ViTs with adder kernels.\n\n4. Does the AdderNet compatiable with SSL pretraining? e.g., MAE pretraining, and how the quantization scheme different for pretraining stage and fine-tuning or normal training stages?\n\n5. How is the latency or throughput in real devices? I am curious of this since there is no available efficient CUDA implementation of AdderNet open-sourced as of now. Also, quantization is also not efficient for general CPU/GPU devices. Have you deployed the trained models to real devices yet?\n\nI am open to further boost the score or champion this paper if the rebuttal is also sound and can somewhat solve my quesions/concerns.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Fw55hZ1lwor", "reviewer": "Reviewer_QZcy", "summary": "Quantization is an effective method to further reduce the energy consumption of AdderNets. However, previous AdderNets quantization methods cannot properly handle the challenge of large differences in weights and activations and often lead to a large accuracy degradation, especially in the case of low-bit (4-bit). This paper first reveals the key reasons for the poor accuracy of previous AdderNets quantization methods, namely “over clamp” and “bits waste”. Then a novel quantization method for AdderNets is proposed. Experiments on several datasets and models demonstrate the effectiveness of the proposed method.", "questions": "1. Is it a common phenomenon that the distribution of weights and activations in pre-trained adder neural networks differs greatly?\n2. Can the authors further explain the meaning of the values ​​marked in red in Fig.3 (b) and Fig.3 (c)? How are these values ​​calculated?\n3. The results in Fig.5 (a) are impressive, yet the authors are advised to make more discussions on the results to make the paper stronger.\n4. Writing suggestion: Line 166: “both weights and features” should be “both weights and activations”, to be consistent with the full text.", "limitations": "The authors have discussed the limitations and potential negative societal impact of their work in Appendix.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The paper is extremely well structured and easy to follow, with motivation well-explained.\n2. To my knowledge, this paper is by far the most comprehensive and systematic study of the quantization of AdderNets. Through thorough analysis, this paper concludes two main reasons for the poor accuracy of previous AdderNets quantization methods, namely “over clamp” and “bits waste”, which are insightful. The proposed scheme of the clustering-based weights grouping and the lossless range clamp for weights are interesting and novel.\n3. Extensive experiments on different models and datasets. Superior performance compared to other AdderNets quantization methods. The thorough ablation studies verifies the effectiveness of each components. The distributions of weights and activations (Fig.1 in Appendix) demonstrate that the proposed method can effectively solve the problem of “over clamp” and “bits waste”, leading to a higher quantized performance.\nWeaknesses\n1. The values in Fig. 4 are too small to read. The authors are required to refine them.\n2. The histogram for INT4 weights ​​adjacent to “over clamp” is significantly higher (Fig.1 in Appendix), however, this phenomenon is not expressed in the top of Fig.1 (c). The authors are advised to revise this detail for better presentation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "56aLZhvEFfB", "reviewer": "Reviewer_R6QH", "summary": "This manuscript focuses on the problem of the quantization of AdderNet. The author has investigated the difference between AdderNet and traditional networks. Based on the differences, the dedicated quantization is achieved by redistributing the weights and activation of AdderNet. In the quantization method, three techniques are proposed to overcome the bit waste and over clamp problems, including clustering-based grouping quantization, range clamp of weights, and outlier clamp of activations. Experimental results show the effectiveness of the proposed method for quantizing AdderNet with different bit widths.", "questions": "Please see the weakness part.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n- The manuscript is easy to follow. The analysis of the difference between conventional quantization methods for CNN and that for AdderNet is interesting. The statistics of the activations and weights of a pre-trained AdderNet are good.\n\n- As a new kind of efficient neural network (AdderNet), how to effectively quantize it is a challenging problem. Quantization of such NN would put forward a faster and more energy-efficient model.\n\n- The proposed method, includes clustering-based grouping quantization, range clamp of weights, and outlier clamp of activations, is promising for addressing the bit waste or over-clamp issues within AdderNet, which is also verified by the extension experiments.\n\nCons:\n\n- Besides the FLOPs and energy, it would be great to report the inference time of the proposed method.\n\n- It is highly recommended to add a more detailed recap about AdderNet, which will make the whole manuscript smoother, especially for those who do not familiar with it.\n\n- In Fig. 3, how is the `-126.2` calculated? There is no detailed explanation about it.\n\n- In line 45, “L1-norm quantization” is unclear. Does it mean an L1-norm-based quantization method or quantization for L1-norm operation?\n\nMinor issues\n\n-There is a strange “rec” symbol in Line 193\n\n- There are several minor grammar issues. For example,  in line 21: “well-verified” should be “well verified”.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "16elRJ5NSKq", "reviewer": "Reviewer_zyyr", "summary": "The paper proposes a new quantization scheme for Addernets. Specifically, the authors propose to cluster model weights in the channel dimension, where each cluster is assigned its own scale value for quantization. This ensures that the scales can better represent the range of values, which may be different for different weight channels. The authors further propose to absorb the error caused by clamping the weights inside the layer bias, which helps restore accuracy. Finally, the proposed method removes outliers when quantizing the activations, therefore tailoring the scale to better represent the valid range of data.", "questions": "Questions:\n- It appears that the models are not trained with a weight regularization term which leads to a wide range of weight values in Figure 2. A standing question is whether adding strict regularization to training can change the effectiveness of the proposed method? In that scenario, the weights are forced to be within certain bounds, which may reduce the benefits of clustering as the same scale may likely work for all weights. \n\nLimitations:\n- The I term in equation 13 of the paper merely depends on the weight values and not the activations. Therefore, while the derived scale is perhaps optimized for the weights, there is no guarantee that it will work for the range of activations. The accuracy gains of the proposed method may therefore be due to the more fine-grained weight scales, rather than the fact that the scales are actually tailored to both the activation and the weights, which is what is assumed after reading the introduction. ", "limitations": "The authors have not discussed the limitations or potential negative social impact of their work.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written and the ideas are clearly explained.\n- The proposed method significantly improves the accuracy after quantization, compared to prior methods.\n\nWeaknesses:\n- Some of the claims are not backed up by the method. Specifically, the authors mention that a shortcoming of prior work is using the same scale for weights and activations which is decided based on either of the two and therefore may not best fit the other. The proposed method also adopts the same scheme, where the scales are still determined by either the weights or the activations with the only difference being the increased granularity of the scale choices due to the channel clustering. Please find more details on this in the next section.\n- Some questions remain regarding applying the method to new models, e.g., how to determine the number of clusters for new benchmarks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept