Towards Stable and Robust AdderNets
Authors: Minjing Dong
Yunhe Wang
Xinghao Chen
Chang Xu
Venue: NeurIPS 2021 Poster (2021)
Keywords: AdderNet
Adversarial Robustness
Network Stability

Abstract: Adder neural network (AdderNet) replaces the original convolutions with massive multiplications by cheap additions while achieving comparable performance thus yields a series of energy-efficient neural networks. Compared with convolutional neural networks (CNNs), the training of AdderNets is much more sophisticated including several techniques for adjusting gradient and batch normalization. In addition, variances of both weights and activations in resulting adder networks are very enormous which limits its performance and the potential for applying to other tasks. To enhance the stability and robustness of AdderNets, we first thoroughly analyze the variance estimation of weight parameters and output features of an arbitrary adder layer. Then, we develop a weight normalization scheme for adaptively optimizing the weight distribution of AdderNets during the training procedure, which can reduce the perturbation on running mean and variance in batch normalization layers. Meanwhile, the proposed weight normalization can also be utilized to enhance the adversarial robustness of resulting networks. Experiments conducted on several benchmarks demonstrate the superiority of the proposed approach for generating AdderNets with higher performance.

Towards Stable and Robust AdderNets
Minjing Dong Yunhe Wang?; Xinghao Chen', Chang Xu' School of Computer Science, University of Sydney HHaawii Noah's Ark Lab mdon0736@uni. wywp3deennisyyyyy..oo.e..o edu. edu.au, yunhe wang@huawei. com, xinghao. wwwn@huwwei.ccmmmm com, c. c.xulsydney, edu au
Abstract
Adder neural network (AdderNet) replaces the original convolutions with massive multiplications by cheap additions while achieving comparable performance thus yields series of energy-efficient neural networks. Compared with convolutional neural networks (CNNs), the training of AdderNets is much more sophisticated including several techniques for adjusting gradient and batch normalization. addition, variances of both weights and activations in resulting adder networks are very enormous which limits performance and the potential for applying other tasks. To enhance the stability and robustness of AdderNets, we first thor- oughly analyze the variance estimation of weight parameters and output features of an arbitrary adder layer. Then, we develop weight normalization scheme for adaptively optimizing the weight distribution AdderNets during the training procedure, which can reduce the perturbation running mean and variance in batch normalization layers. Meanwhile, the proposed weight normalization can also be utilized to enhance the adversarial robustness of resulting networks. Ex- periments conducted on several benchmarks demonstrate the superiority the proposed approach for generating AdderNets with higher performance.
1
Introduction
By hint of the success of Deep Convolutional Neural Networks (DCNNs), a wide range of computer vision tasks can be tackled with satisfactory performance, including image classification [13 9 [1]
25), super-resolution [32 12 22]. object detection [16
7 g etc.. DCNNs are mainly trained and inferred on graphics processing unit (GPU) devices which suffice fulfill the high computational complexity and enormous energy consumption. However, is intractable for DCNNs to be deployed on low-power devices due the limited computational speed and power capacity, such as mobile and embedded devices. As result, substantial research efforts have been devoted to energy reduction and speed acceleration of DCNNs
19
29 26]..
Recently, Chen et al. [ advocate the use of fi-distance for similarity measure instead of cross- correlation in CNNs to replace multiplications with additions. Compared with multiplications, addition operations are much cheap, which benefits the power-efficiency 28 23 30 Adder neural network (ANN) has demonstrated extraordinary performance several computer vision tasks with huge energy reduction, which can be seen as a good complement to the classical CNNs.
The performance achieved by ANNs on classification is impressive, but some observations raise our concerns. e.g., the optimization of ANNs not stable as CNNs where the test accuracy curve has dramatic fluctuations during the optimization, ANNs has a large variation until the end of training while that of convolution networks is much more stable. Furthermore, weights of ANNs have demonstrated significantly different statistical
shown in Figure I (a). The test accuracy of
*Correponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

3
(a). Loss Curve
(b). Adder feature after BN
(c). Adder weight
(d). Conv weight
Figure I: Observations of AdderNet. Training and testing loss curves of ANN and CNN in (a). Histograms over the AdderNet features after batch normalization layer follows Gaussian distribution in (b). Histograms over the AdderNet weight follow Laplace distribution with large variance while Conv weight follow Gaussian distribution with small variance in (c) and (d) respectively.
property from those of CNN weights. The histograms over ANN and CNN weights shown in Figure (c) and (d) respectively. The red curve in (c) denotes Laplace distribution with a mean of 0.32 and variance of 88.47 and the one in (d) denotes a normal distribution with mean of -0.002 and a variance of 0.001. Although the means of these two distributions are similar, there exists serious discrepancy between their variances. Given this statistical property difference, those sophisticated training strategies previously developed for CNN might not fulfill their potential when straightforwarly applied to ANNs.
In this paper, we provide an exhaustive analysis of the variance ANNs. Taking a one-layer adder forward as example, we demonstrate the large variance of ANN weights can be the major cause of the instability of running mean and variance in batch normalization layer which vibrates the test accuracy. Weight normalization is therefore natural idea to constrain the variance of weights. To preserve the representation capability of the normalized weights, trainable scaling and shift coefficients are further introduced to achieve doing so, the variance of ANN weights can be normalized to acceptable levels which significantly stabilizes the batch normalization layers and makes pretrained ANN easily applied to other tasks. Notably, we identify a natural advantage of ANNs to be robust to adversarial perturbations, stemming from disittance and the running mean of batch normalization layer in ANNs. The reduction of weight variation across channels further boosts the defense ability against attacks. Without adversarial training, AWN with ANN on ResNet-32 improves adversarial accuracy by 59.73% compared with CNN under PGD attacks, AWN presents superior stability under a series of evaluations and outperforms vanilla ANN in detection task on VOC benehmark with 2.7 mAP improvement.
Adaptive Weight Normalization (AWN). By
2 Preliminary
Adder Neural Networks. To significantly reduce energy costs, Chen el al. 11] proposed an Adder Neural Network release the burden of multiplications in traditional convolution networks by replacing them with additions. Consider an intermediate layer in deep convolution neural network with weight W dx Xx,,,,Xe XCout where d denotes the kernel size and Cin and Cout are the number of input and output channel respectively. Given the input feature map X E R xein where H and W are the height and width of input feature, the adder operation is defined as
(m,
X(mm + iin j.k) ((.
d=1
Although the adder operation enables ANNs to achieve similar performance in classification tasks with energy efficiency, the observations in Figure Souure[saae shadow ANNs. The unstable test accuracy curve and large variance almost everywhere in ANNs arouse our interest in the analysis of ANN variance.
Adversarial Attack. Given the input I E R and the annotated label y RCC where D is the dimension of input and C' is the number of classes, the network A maps perturbed input ă to 8. The objective of adversarial attacks is to find the perturbed input
(ā; W) where which maximizes the classification loss as
argmax ((N W),v), a -1ll-
(2)

---

where the perturbation is constrained by its (p-oom. Through variance study of ANNS and & fi-iittance analysis, we demonstrate the potential adversarial robustness of ANNs can be activated through proposed inference strategy with AWN.
3 Variance Study of AdderNet
To analyze the variance of ANNs, we first consider a one-layer adder forward with batch normalization [11] and ReLU as activation function. Given output feature y1-1 from previous layer, the forwarding of layer is formulated as
3/ 01
= maz 0, 11-1), sí
Σ 2 Wil. y1
+ B
(3)
where Yu and 0; denote the mean and standard deviation, respectively, Wi is the weight, represents the size of weight of one output channel n = d x Cins and yy and / stand for the rescale and shift parameters.
Before we proceed to the weight variance analysis, we first make several assumptions that are empirically plausible. After the batch normalization, 1/-1 is supposed to follow normal distribution ((0, d²). From empirically observation of a ResNet- 18 with ANN on ImageNet, as shown in Figure (b) and (c), adder feature after BN layer follows distribution with mean of 0.02 and variance of 0.38 while the corresponding weight with mean of 0.32 and variance of 88.47, the adder weights follow Laplace distributions with large variance but their mean is close to 0 after training. Thus, assume W, follows C(v,) b) with the mean as and the variance as 2h where 21 > o².
Now compute the mean and variance of each variable. The activation follows the Rectified Gaussian distribution. With the law total expectation, the mean æ forms the one-side truncation of lower tail E[ Jaz > 0] which can be computed based on the property of Truncated normal distribution as
Elæy] = | |2) > 0] P(x > 0) + E| 5 P(n; 5)
o(0)
(4)
= E|maa > 0] P(n > 0) + 0 = 00 +
1 (0) 2 = V2TT
where (b)) denotes the probability of standard normal distribution and ((.) denotes the cumulative
distribution function. Similarly, with the law of total expectation, the variance of xą is broken down into variants of expectation and the variance of one-side truncation of lower tail Var(ay > 0), which can be computed through Truncated normal distribution as
Var[z1]
E (E(=i))² (E(m))) E[ai In > > P(n > 0)
P( 0)[V ar( > 0) (E[1) >0)) - (E(m))²
(5)
= 1o 6(0) 12) ( 4
4
o² = 0² =2
1
2
2n
2 2m
Since we assume 21/ >> o² and the variance is further reduced after activation, the distribution of
It W, is overwhelmed by W, to form a Laplace distribution as L( / 1 + 45 ). For
simplicity, let Tμ .2= ja and b + Ta V - (m-1)=² Note that the standard deviation of output last layer and the mean of weight μ are both close zero, which makes Tμ and To small values. Although it is difficult to directly derive the closed-form distribution expression of y/, it can be approximated based on variable x WT- W Ty. According the properties of Laplace distribution, the absolute function of C(0, b) follows an Exponential distribution, with which of can be approximated as
vi =
Σ
bt Wi
Σ
||
W₁ Tul + ITull.
where [2]
Wi Tyl ~ Exp( + Ta] 1=))
where Exp denotes Exponential distribution. Based on the property of Exponential distribution, Ella: Will b + and Var[iz] Will (b + )2 With Eq. 6 the lower boundary of E[vl]
3

---

can be derived as
B|a - Wil v B| Wi Tul + lmdb
Elyil 2 (b + To + |T
With inequality in Eq. 7 the lower boundary of V V (vrlv can derived by the law of the unconscious statistician as
Varlyl = > [B|| - wil - (El Will)a
[Var(z) Wi) + (E(m) Wi)) (E( Wi 2 m[Var(æi - Wi) + (B(m)-Wi)) - - (Biz) Wi Tul =
(8)
+ - + ),)²² + (b To [T,))²]
m|(b + +T,² - (b
T.)]]
Taking the lower boundary of mean and variance of in Eq. the batch normalization layer can be computed as
1/ 3/ + n + (Tu)
+ 3.
Vn((b+ 2 To)
Note that the batch normalization layers use running mean and variance of current batch in training phase while moving averages in testing phase. Although precise Ely'l and Varlyí] can be computed training phase, the moving averages vary dramatically since b in Eq. grows from small value to a large one during optimization, as shown in Figure (c) where the distribution of W, with initial variance of 1.0 is optimized to the one with variance of 88.47. Thus, the large variation of batch statistics results in the instability of testing loss curve while the training loss curve is similar to the one of CNNs, as shown in Figure
3.1 Adaptive Weight Normalization for AdderNet
Based on these observations and analysis, we propose to make some efficient modifications to current ANN optimization. From Eq. 9 the running mean and variance mainly depend on the standard deviation of adder weights so that large magnitude of weights could destabilize the statistics in batch normalization, which indicates that adder weights need normalization to prevent them from being updated to distribution with large variance. A naive approach is Weight Standardization proposed by Qiao et al. [8 as
W Hw, where pw; ow,
W
Wid+
1=1
(10)
OW:
( juw;) +
where W RReet denotes the permuted adder weight, || denotes concatenation operation and is added for numerical stability. In Eq. [0] each output channel of adder weight is normalized to a distribution with mean of 0 and variance of 1, which stabilizes the running mean and variance of batch normalization layer according to Eq, 9 However, with weight standardization directly applied to ANNs, there exists dramatic accuracy drop. For example, ANN with weight standardization achieves 90.62% in ResNet-20 on CIFAR- while original ANN achieves 91.84%, which causes 1.22% accuracy drop. Although adder weights are normalized to guarantee a stable test phase, rigorous mean and variance are strictly assigned to adder weights, which constrains the representation power of ANNs. Since the similarity between filter and input feature is measured by & didiittncce in ANNs, the magnitude of weight values can be rather sensitive for network expression capability. Thus, directly applying weight standardization to ANNs can be easily stunk in the local optimum without exploring wider space of adder weights. Instead, we propose to normalize adder weights with trainable variables for each output channel, which preserves the representation power. Eq. 10 can be rewritten as

---

W = Wi JAW, + Ui
(11)
ow,
where v; and vú are trainable variables similar to 8 and y in batch normalization layer. Thus, the magnitude of weight values can be automatically adjusted to fit the potential levels of freedom of adder weights. Furthermore, previous analysis demonstrates that the magnitude of gradient w.r. the input X and the filter W in ANNs is much smaller than that in CNNs ([] With the incorporation of these two parameters, the gradient w.r.t the filter W can be automatically adjusted. The gradient of loss & w.r.t the weight Wa,j is computed as
de
V
Oe
OW 1,c [1+ (W13
de
W WW -
(12)
aW.j
n nww,
OW
ow:
In Eq. 12 the gradient of W is amplified by vs which relieve the gradient reduction in ANNs.
4 Activate Potential Adversarial Robustness
Consider the intermediate layer forwarding of ANNs and CNNs with perturbation òt, the disturbance of an element on the output feature map before BN layer can be computed as
MONN VECNN
[ 51,4) Wi,1 21, Widl
(-,)) x Wi,
(13)
MANN MIANN
Σ |,,, Wt,1 - 100,5 δ1,5 Wi,1 ~
|
where + denotes the choices of two possible signs including addition and subtraction. Note that lay, Wi,l follows distribution with large mean and variance in ANNs. We assume that dj follows Gaussian distribution with zero mean and smaller variance N(0, az) where o3 < [Wi] with which the subtraction |æt_ Wi,i |2i - 61,5 Wi,il has a high probability to be either 6j or -ō.. We show that it is difficult for +|0 to have large variance in ANNs under adversarial attacks. In Eq. 13 if V or [=])) grows in ANNs, all the elements in | tend to select different signs, which automatically eliminate each other make 3/ ANN VIANN R O. Thus, if attacks succeed, /or needs reduction to guarantee larger perturbation on feature map. To ensure maximum disturbance, all the elements + | have the same signs, which forms a sum of n half-normal distributions. On the contrary, each ōt,i in CNNs is transformed by different Wi, in Eq. 13 The variance of disturbance before BN layer can computed as
Varl - VlCNN = Var Σ
((,) x Wi,1 (V [W]] + (E(W))
2=1
(14)
|6, = no 2
Varl - VIANN RO Varl
Eq. [4]iniccsss the output disturbance variation in CNNs depends on both the statistics of and and could vary for different output channels while the one in ANNs only depends on the statistics of 8 for all the output channels. The major difference lies in BN layer where both ANN and CNN disturbances rescaled and the variance of perturbation next layer can be computed as Varlō4+1CNNN no ar [Wi)
2 Varlō++1ANN = n1 ( //(Std(ilaN))
(15)
Note that the variance of 8/ ANN is much larger than B Thus, ANNs have a much smaller
disturbance variance on + 1 layer, which suggests that the perturbations of all the elements on i/ are similar and can be eliminated through subtraction of scalar value while CNNs cannot copy that success. To activate the adversarial robustness through utilizing this property of ANNs, simple yet effective method is utilizing the running mean in batch normalization layer for automatic disturbance
5

---

Table 1: Adversarial robustness CIFAR-10 under white-box attacks without adversarial training. -R denotes robust inference strategy which uses the running mean in batch normalization layer instead
of tracked ones. BIM denotes iterative attack with 7 steps. The best results in bold and the second
best with underline.
Model
Method
#Mul.
#Add.
Clean
FGSM
BIM
PGD
MIM
RFGSM
CNN
41.17M
41.17M
92.68
16.33
0.00
0.00
0.01
0.00
ANN
0.45M
81.89M
91.72
18.42
0.00
0.00
0.04
0.00
ResNet-20
CNN-R
41.17M
41.17M
90.62
17.23
3,46
3.67
4.23
0.06
ANN-R
0.45M
81,8MM
90.95
29.93
29.30
29.72
32.25
3.38
ANN-R-AWN
0.45M
81,89M
90.55
43.93
42.62
43.39
46.52
18.36
CNN
69.12M
69.12M
92.78
23.55
0.00
0.01
0.10
0.00
ANN
0.45M
137.79M
92.48
35.85
0.03
0.11
1.04
0.02
ResNet-32
CNN-R
69.12M
69.12M
91.32
20.41
5.15
5.27
6.09
0.07
ANN-R
0.45M
137.79M
91.68
19.74
15.96
16.08
17.48
0.07
ANN-R-AWN
0.45M
137.79M
91.25
61.30
59.41
59.74
61.54
39.79
elimination. In Eq. 9 T₁ becomes Ty. E[o] while b + To nearly remains the same under attack settings since the disturbance has small variance. The forwarding of BN layer can be computed as
V' + n(b T To + T4 F(5]))
1 Y
+3
(1 TT 2 F. Ta)
3/ Σ. = + n To [nl) |[8]
+
(16)
To)² T/T - E E(ō]((b + To)
y' + n(b + To + (T,))
+ 3.
+ To)² 2 FT
[b+][(b+Ta)
The difference between Eq. and Eq. [6 in the rumning variance, which demonstrates that the disturbance can be significantly relieved through computing the running mean in BN layer while the variance is expected to be constant to eliminate the difference brought by Elō). Thus, propose ANN robust inference strategy as
y 3 running mean
tracked variance + 3.
(17)
However, considering the actual case where the signs of +18,,al not the same everywhere, the proposed ANN robust inference strategy cannot work appropriately and results in residual perturbations which will be rescaled by tracked variance after BN layer. In Eq. [3] the perturbations of different output channels are similar in ANNs, which indicates that the noise of feature map can hardly cross channels. However, this property will be broken if tracked variances have enormous differences among channels. Thus, our proposed ANN-AWN can successfully relieve the variation across channels to further improve adversarial robustness, which verified in later empirical evaluations.
Experiments
In this section, we empirically evaluate the superiority of the proposed approach different tasks and datasets, including the adversarial robustness, object detection and optimization stability.
5.1 Adversarial Robustness Evaluation
White-box Attacks Setup. To demonstrate how the proposed adaptive weight normalization activates the potential adversarial robustness of AdderNet, we conduct series of experiments. Following [I], we first train both CNNs and ANNs with ResNer20 and ResNet32 on CIFAR-10 under the same settings. CIFAR-10 dataset contains 50K training images and 10K validation images with size of 32 x 32 over 10 classes. We use SGD optimizer with an initial learning rate of 0.1, momentum of 0.9 and a weight decay which takes 400 epochs with batch size of 256 and cosine learning rate schedule. The learning rate of trainable parameter vs and u in AWN rescaled by a hyper-parameter which we set to be 1 x 10-5, For adversarial robustness evaluation, we conduct white-box attacks these models
10-4 The model trained on single V100,

---

(b)
(c)
Figure 2: The evaluation of adversarial robustness under different PGD attack size is shown in (a) and different PGD attack steps in (b). The performance of intermediate weights sampled from ANN
and ANN-AWN through linear interpolation in (c). Test loss curves CNN, ANN, ANN-WS and ANN-AWN are shown in (d).
including Fast Gradient Sign Method (FGSM) [2], Basic Iterative Method (BIM)
14
Projected
Gradient Descent (PGD) [17. Momentum Iterative Method (MIM) 6 and RFGSM
27 to generate
adversarial examples. Following previous adversarial literature [7 31 the adversarial perturbation
considered under 2o norm with the total perturbation size of 8/ 8/255. In iterative attack settings, the step size is set to 2/255. The number of iterations is set to 7 for PGD and BIM attacks, and 5 for MIM and RFGSM attacks. Note that different from traditional defense algorithms which generate adversarial samples for adversarial training 21] or search robust architectures 15, our proposed algorithm utilizes the properties of ANNs to achieve adversarial robustness without any training tricks or modifications of architecture. The experimental results shown in Table IT the models are trained with clean images, which dismisses the expensive adversarial training.
Note that all
Against White-box Attacks. As shown in the first two rows in Table 1 these adversarial attacks successfully mislead both CNN and ANN to provide wrong predictions. Although ANN has slightly better adversarial accuracy compared to CNN, such as 0.01% - 0.04% under MIM attack, the potential adversarial robustness shown in Eq. 13 cannot be activated with normal settings. Thus, we replace the tracked mean in batch normalization layer with the running mean of current batch based on Eq. a7]an compare ANN with CNN the third and fourth rows, which are denoted as -R. Comparing CNN-R and ANN-R, there exists an enormous adversarial robustness improvement. For example, ANN-R achieves 26.05% accuracy increment from 3.67% + 29.72% under PGD compared with CNN-R. and improves the accuracy by 32. 21% from 0.04% - 32. 22.25% under MIM compared with ANN. The comparison between ANN and ANN-R empirically demonstrates that appropriate utilization of the running mean in batch normalization layer of ANNs can significantly activate the adversarial robustness of ANNs. Furthermore, the comparison between CNN-R and ANN-R shows strong evidence of the natural robustness difference between CNNs and ANNs and indicates that the & distance and the independence between perturbation and adder weight provides much better defense than CNNs, which is consistent with the aforementioned variance analysis. The evaluation of proposed AWN is shown in the fifth row. For all the attacks, ANN-R-AWN achieves the best results, which demonstrates the effectiveness of proposed AWN. Comparing ANN-R and ANN-R-AWS A-WNN, AWN shows obvious superior adversarial robustness. For example, AWN improves adversarial accuracy by 13.67 13.67% from 79% - 43.39% under PGD? and 14. 98% from 3.38% + 18.36% under RFGSM which illustrates that adversarial robustness can be further boosted through narrowing the difference among channels to relieve the perturbation transformation. On ResNet-32, ANN-R achieves worse performance than the one with ResNet- 18, which demonstrates that proposed robust inference strategy is not sufficient for superior adversarial robustness. However, ANN-R-AWN consistently achieves better performance, which outperforms other baselines in all the adversarial scenarios, which indicates that the robustness of proposed AWN can generalize to deeper models.
Against Gradually Enhanced Attacks. We highlight the superiority of proposed ANN-R-AWN through enhancing the attacks from different aspects to evaluate the adversarial robustness under more powerful attacks. We use the naturally trained models with ResNet-20 on CIFAR- 10 for evaluation. In the first scenario, the total perturbation size of PGD attack increases from 0.01 to 0.1 with step size of c/7. In the second scenario, the iterations of PGD attacks are enhanced from 1 to 50. The evaluation results are shown in Figure (a) and (b). ANN-R-AWN obtains much better defense ability than ANN-R, CNN-R and ANN-R-W- WS baselines when the attack size grows. For example, the adversarial accuracy of CNN-R quickly dive to 2. 2.96, ANN-R to 27.67% and ANN-R-WS to 28.66 when the PGD attack size reaches 0.06 while ANN-R-AWN maintains 32.67% when the PGD attack size reaches O. Similarly, ANN-R-AWN also achieves superior robustness over other baselines with increasing steps of PGD attack. For example, the adversarial accuracy of ANN-R
7

---

+150
Teaturas
-140
1100
-9000
3000
-100
(a). ANN Layer-2
[b). ANN-AWN Layer-2
[c). ANN Layer-19
(d). ANN-AWN Layer-19
Figure 3: Distributions of features from different layers of ANN and ANN-AWN on different epochs.
under PG becomes 26.48% while ANN-R-AWN maintains 36.52% with 10.04% increment, which demonstrates the effectiveness of AWN. The enormous accuracy drop of ANN-R-W; along with attack epoch and coherent robustness of our proposed AWN illustrate the necessity of introduced adaptive rescale and shift parameters in weight normalization. Thus, the advantage of AWN becomes more obvious under more powerful attacks, which highlights the superiority of proposed AWN.
5.2 Stability of AdderNets
Stability Evaluation through Linear Interpolation. To conduct quantitative evaluation of AdderNet stability, we propose to evaluate the smoothness of optimization landscape through tracking the performance of AdderNet parameters which are sampled between different training epochs. To highlight the stability of our proposed AWN, we select adjacent epochs for ANN and ANN-AWN trained with ResNet-20 on CIFAR- 10 and sample 9 intermediate weights between two epochs through linear interpolation. The comparison is shown in Figure 2 (c). To evaluate the stability of early training stage, we select epoch 160, 164 for ANN as the blue curve and select epoch 125, 135 for ANN-AWN as the green curve. Although we set earlier epoch with larger intervals for ANN-AWN, our proposed algorithm forms much more smooth curve with smaller variance 336.00 compared to the one of ANN 587.28. In the later training stage, we select epoch 290, 294 for ANN as the orange curve and select epoch 283 294 for ANN-AWN the red curve. Comparing with ANN, our proposed algorithm achieves variance reduction of 585.97 from 627.54 to 41.67. The performance of parameters sampled from linear interpolation reflects the stability of models and the smoothness of the optimization landscape. With the proposed AWN, the optimization of AdderNet can be significantly stabilized, which could benefit the situations where adder layers are difficult optimize.
Trade-offs Between Stability and Accuracy. To further illustrate how AWN takes effect, we keep track of the test loss during AdderNet optimization and the results of both CNN and ANN with ResNet-20 on CIFAR-10 are shown in Figure 2
(d). The test loss curve of AdderNet is quite unstable before 300 epochs while CNNs, ANN with weight standardization and ANN with adaptive weight normalization all achieve relatively smoothed loss curve, which indicates that the instability of AdderNet mainly comes from the large variance of adder weights and the proposed AWN eliminate successfully. The dotted area covers from 300 to 400 epochs. Although AdderNet cannot achieve stability due the large variance of weights, ANNs can still achieve similar performance as CNNs since the variation of adder weights is reduced during the end of training. However, WS fails to achieve a similar classification performance and is stuck at local optimum, which demonstrates that the normalization on adder weights could hurt the expressive power. On the contrary, our proposed AWN achieves relatively better performance through incorporating the adaptive trainable parameters for adder weights, which enables them to shift and rescale back to restore the original performance.
Feature Distribution Analysis. We visualize and track the feature distributions of AdderNet with or without AWN during the training, as shown in Figure 3 We randomly sample features before batch normalization layer at different epochs and compute histograms over them. Both ANN and ANN-AWN feature distributions in the 2nd and 19th layers are shown in Figure (a). and (d) respectively. ANN feature distributions vary dramatically during the optimization, which enormously disturbs the tracking of mean and variance in batch normalization layer. For example, the sampled feature on 21-th epoch in (c) has mean of -2738. 58 and standard deviation of 1042.44 while the one on 386-th epoch has mean of - -1476.52 and standard deviation of 266.97 with 1262.06 increment on mean and 775.47 reduction on standard deviation. However, those in ANN-AWN become much milder, which stabilizes ANNs. For example, the sampled feature on 21-th epoch in (d) has mean of -940.89 and standard deviation of 269.39 and the one on 385-th epoch has mean of -564.59 and standard deviation of 60.65 with 376.3 increment mean and 208.7 74 reduction standard
8

---

deviation. Furthermore, the difference among channels are effectively reduced by AWN to constrain the perturbation space of adversarial examples, e.g., the standard deviation of ANN features on the second layer after training becomes 87.41 while that of ANN-AWN becomes 31.13 with a massive Table 2: ANN stability evaluation on object de- tection task. Comparison of proposed approach on ANNs with other settings on PASCAL VOC 2012 benchmark. The [:] in backbone denotes the classification accuracy of pretrained network on ImageNet.
Table 3: Adversarial robustness comparison of WS and AWN under different inference strategy with ResNet-20 ing. -R denotes using running mean of current batch in BN layer. denotes using both running mean and variance.
CIFAR-10 with natural train-
Model
Backbone
Neck
mAP
Model Cleun FGSM
BIM
PGD
MIM RFGSM"
CNN-FPN Res18-CNN [69.8]
CNN
69.3
A
88. 10 49.24 21.46
22.96
32.26
7.31
ANN-FPN
Res18-ANT [67.0]
ANN
68.6
ANN-F-WW
89.81
49.85
22.54
24.27
32.54
8.26
ANN-WS-FPN
ResI8.ANN.WS. [64.1]
ANN-WS
67.0
ANN-R-WS
89,38
40.65
31.35
36,08
42.3
7,78
ANN-AWN FPN ResIS-ANN-AAN [67.1]
ANN AWN 69.4
ANN-R-AWN
90.55
45.93
42.62
43.39
46.52
18.36
reduction, which potentially provides better defense ability against more powerful adversarial attacks.
5.3 Experiments on Object Detection
To further illustrate the advantage of imposing stability on AdderNets, we conduct experiments on object detection with ANNs on PASCAL VOC (VOC) dataset. VOC contains 20 object classes, the training set includes 10K images which are the union of VOC 2007 and VOC 2012, and the VOC 2007 test set with 4.9K images is used for evaluation. The mAP scores using lou at 0.5 are reported. All the models are trained with the same setting. Based on the variance study of ANNs, we unfreeze BatchNorm during the training. Following (2), we insert more shortcuts in the neck part. We use an initial learning rate of 0.008 with a linear warmup for 500 iterations, momentum of 0.9, weight decay of 1 x 10-4 and cosine learning rate strategy. All the models are trained on 4 V100 GPUs with SGD optimizer for 12 epochs with a batch size of 4. For the detector baseline, we include both CNN and vanilla ANN for comparison. ANN-FPN replaces the convolution layers with adder layers in the pretrained ResNet-18 backbone and neck of Faster R-CNN (20]. Through applying different types of ANNs to detection, we conduct comparison among the CNN, vanilla ANN (Н]. ANN-WS WS and ANN-AW The detailed evaluation is shown in Table 2 In Backbone column, the number in brackets denotes the classification accuracy pretrained on ImageNet. Comparing ANN with our proposed AWN, although they achieve similar classification performance, AWN improves the mAP score by 0.8 from 68.6 - 69.4. Even comparing with CNN-FPN which has superior classification performance, our proposed AWN still outperforms it, which demonstrates the necessity of stability. With a much more smooth loss landscape, the optimization of AdderNet on other tasks can be easier and more stable. However, ANN-WS is not competitive with other baselines, with 2.4 mAP reduction compared with AWN, which empirically verifies that directly normalizing adder weights could limit the ability of feature extraction and performance for other tasks. Note that there exists an enormous accuracy drop of pretrained ANN-WS, which significantly constrains its detection performance. Thus, besides stability, the representation power of ANNs can be rather important in terms of applying ANNs to other tasks. On the contrary, our proposed AWN achieves better trade-on offs between classification performance and stability through an adaptive scheme, which together achieves the superior mAp score in detection task.
5.4 Ablation Studies
We conduct ablation studies on proposed ANN-R-AWN to illustrate the effectiveness of adaptive
training parameters and proposed robust inference strategy. We have already shown that weight
standardization can casily be stuck at a local optimum in Figure (d). Although the analysis in Sec 4
works for both WS and AWN, we empirically verifies that the gaps of clean accuracy between WS
and AWN still exist in adversarial accuracy. The results are shown in Table 3 where WS and AWN
are further attacked and evaluated to demonstrate the influence of performance drop on clean and adversarial accuracy. Comparing WS and AWN, our proposed AWN consistently outperforms WS in both adversarial and clean accuracy, which indicates that directly normalizing adder weights could hurt the representation power of ANNs and restrict the adversarial accuracy of ANNs. With proposed parameters v and u in Eq. [T] AWN successfully relieves this problem through exploring the balance between expressive power and weight magnitude reduction, which achieves better classification
9

---

performance and adversarial robustness. We further evaluate the effectiveness of proposed ANN robust inference strategy as Eq. [7 We denote the strategy which replaces both tracked mean and variance with running ones as -r and our proposed one as -R. Comparing two inference methods, our proposed strategy consistently outperforms -r, which verifies that the activated robustness mainly comes from the running mean which automatically eliminates the perturbations.
6 Conclusion
In this paper, we investigate the major concerns of AdderNets through approximating the mean and variance of output features of an arbitrary adder layer. With derived lower boundary, show that the instability of AdderNets mainly comes from drastic fluctuations of running mean and variance in batch normalization layer whose computation is dominated by the variance of weights. Our proposed adaptive weight normalization (AWN) works with AdderNets to optimize adder weight distributions adaptively, which significantly improves the stability and leads smooth landscape. Our analysis of the adder layer forwarding reveals the potential superior defense ability of AdderNets against perturbations and proposed robust inference strategy together with AWN successfully activate the adversarial robustness. Experiments conducted on stability and robustness demonstrate the superior performance of the proposed ANN-AWN.
Acknowledgment
The authors would like to thank the Area Chair and the reviewers for their constructive comments. This work was supported in part by the Australian Research Council under Projects DE180101438 and DP210101859.
References
[1] H. Chen, Y. Wang, C. Xu, Shi, C. Xu, Q. Tian, and C. Xu. AdderNet: Do we really need multiplications in deep learning? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[2] X. Chen, C. Xu, M. Dong, C. Xu, and Y. Wang. An empirical study of adder neural networks for object detection. In NeurIPS, 2021.
[3] M. Courbariaux and Y. Bengio. Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1. CoRR, abs/1662.028300, 2016. URL http:://rxii wrg. www.aas/..6
02333
[4] C. Dong, C. C. Loy, K. Hc, and X. Tang. Leaming a deep convolutional network for image super-resolution. In European conference computer vision, pages 194-199. Springer, 2014.
[5] M. Dong, L. Y. Wang, and C. Xu. Adversarially robust neural architectures. CoRR, abs/2009.00902, 2020. BtLLhtts:://// :/atxii arxiv. org/ abs 2009. 00902
[6] Y. Dong, F. Liao, T. Pang. H. Su, J. Zhu, X. Hu, and J. Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9185 9193, 2018.
[7] G. Ghiasi, T.-Y. Lin, and Q. V. Le. Nas-fpu: Learning scalable feature pyramid architecture for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7036-7045, 2019.
[8] J. Guo, K. Han, Y. Wang, C. Zhang, Z. Yang. H. Wu, X. Chen, and C. Xu. Hit-detector: Hierarchical trinity architecture search for object detection. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 11 11405-11414 11414, 2020.
[9] K. He, X. Zhang. S. Ren, and Sun. Deep residual learning for image recognition. CoRR, ab2/1552.033855 2015. URL //tp///avvv///////////////.///wa www.orgggbbb...... wat.s/1112.... 03385
[10] G. Huang. Z. Liu, and K. Q. Weinberger. Densely connected convolutional networks. CoRR. abs/1608.06990 069933, 2016. URL hRtt:://// [ull:atee//aarav.oooaaaaaaLLLL 06993
[11] S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.03167. 2015. UBL.Attp.//sevvv.or///aas 03167
10

---

[12] J. Kim. J. Kwon Lee, and K. Mu Lee. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1646-1654, 2016.
[13] A. Krizhevsky, L. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, pages 1097-1105, 2012.
[14] A. Kurakin, L. Goodfellow, and S. Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.0255 2016.
[15] Y. Li. Z. Yang. Y. Wang, and C. Xu. Neural architecture dilation for adversarial robustness. In Advances in Neural Information Processing Systems, 2021.
[16] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg. Ssd: Single shot multibox detector. In European conference on computer vision, pages 21-37. Springer, 2016.
[17] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant adversarial attacks. arXiv preprint arXiv: (70.000330.08883 2017.
[18] S. Qiao, H. Wang, C. Liu, W. Shen, and A. L.. Yuille. Weight standardization. CoRR, ahs/1903. 2019. URL wttp:::::/axxv..oooto www.oo/ggg/baaeeeeeeeeee 10520
[19] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779-788, 2016.
[20] S. Ren, He, R. B. Girshick, and Sun. Faster R-CNN: towards real-time object detectionn with region proposal networks. CoRR, abs/1506.11497, 2015. /tLL:/ttpp:/aaxw...rro//// http://pri....r///a////5311..1..
[21] A. Shafahi, M. Najibi, A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, and Goldstein. Adversarial training for free! CoRR, atw//104..12200.0 12843, 2019. URL /tK.itttpp//rrr//.rr/////// 12843
[22] D. Song, Y. Wang. H. Chen, C. Xu, C. Xu, and D. Tao. Addersr: Towards energy efficient image super-resolution. arXiv preprint arXiv:2009.0880 108.0999.881..... 2020.
[23] V. Sze, Y.HH. Chen, T.-J. Yang, and J. S. Emer. Efficient processing of deep neural networks: A tutorial and survey. Proceedings of the IEEE, 105(12):2295-232) 2017.
[24] C. Szegedy, W. Zaremba, Sutskever, J. Bruna, D. Erhan, Goodfellow, and R. Fergus. Intriguing properties of neural networks. arXiv e-prints, art. arXiv: 1312.6199, Dec 2013.
[25] M. Tan and Q V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. CoRR, abs/1905.1 1196,, 2019.
905. 11946
[26] Y. Tang, Y. Wang, Xu, Y. Deng, C. Xu, D. Tao, and C. Xu. Manifold regularized dynamic network pruming. CoRR, ahs/21103.0552 2021. URL httpsstts::/// http:://axxvvvvo www.ivi..ggr..oo. wwt..//ggg/a///a//e abs/2103 103.003.00 0586T
[27] F. Tramèr, A. Kurakin, N. Papemot, I. Goodfellow, D. Bonch, and P. McDanicl. Ensemble adversarial training: Attacks and defenses. arXiv preprint a7Xi::1705.07204, 2017.
[28] M. Wang, W. Fu, X. Hc, S. Hao, and X. Wu. A survey on large-scale machine learning. IEEE Transactions on Knowledge and Data Engineering, 2020.
[29] B. Wu, X. Dai, P. Zhang, Y. Wang. F. Sun, Y. Wu, Y. Tian, P. Vajda, Y. Jia, and K. Keutzer. Fbnet: Hardware- aware efficient convnet design via differentiable neural architecture search. CoRR, abs/1812.033443, 2018. htllhtttp///axiv.coo
812. 03443
[30] H. You, X. Chen, Y. Zhang, C. Li, S. Li, Z. Liu, Z. Wang, and Y. Lin. Shiftaddnet: A hardware-inspired 2020.
deep network. arXiv preprint arXiv:2010.
[31] H. Zhang, Y. Yu, J. Jiao, E. P. Xing, I. E. Ghaoui, and M. I. Jordan. Theoretically principled trade-off between robustness and accuracy. arXiv preprint arXiv:/901.0000 1901,0853,
2019. 2019.
[32] Y. Zhang, K. Li, K. Li, L. Wang. B. Zhong, and Y. Fu. Image super-resolution using very deep residual channel attention networks. CoRR, abs/1807.00255 2018. URL htth:htttap/x/// wwtp:/iiiwrgggaaaao wrtp//////a/b///// 02758
11


Reviews:
[{"review_id": "vS7chG27FAi", "reviewer": "Reviewer_xDpY", "summary": "This is an interesting paper to discuss the variance in the adder neural network. The authors have an in-depth comparison between ANN and CNN. Also, the adder operation in the neural network can enjoy some nature advantage over CNN in terms of the adversarial robustness. The authors have conducted comprehensive experiments to support these analyses.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In Line 86, the authors stated that \"the adder weights follow Laplace distributions\". But I did not get how the authors achieve such a conclusion. Why not the adder weights follow a gaussian distribution? I guess by default, the network is initialized with a gaussian distribution.\n\nWhat do you mean by \"activation xl follows the Rectified Gaussian distribution\"?\n\nIn Eq. (15), how the authors get the conclusion that the second line has a smaller variance than that of the first line? I could agree that the denominator for ANN is larger than that of CNN, but the numerators are different.\n\nThe authors proposed using running mean to improve the adversarial robustness. Will this trick be applied over CNN and what the effect will be? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "oME806LDA3c", "reviewer": "Reviewer_8kiu", "summary": "This paper aims at enhancing the stability of AdderNets for performance improvement in downstream tasks. For an adder layer, approximations of feature variance and expectation are derived, with which the variance of weight is found to be the major cause of instability. Adaptive weight normalization (AWN) is thus introduced for addressing this problem. Together with the feature variance approximation, the potential adversarial robustness of AdderNets is also analyzed through comparing the perturbation variance in both AdderNets and CNNs. A simple robust inference strategy with BN layer is thus designed for disturbance elimination.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this paper provides a good analysis on the variance and exposes an important problem in AdderNets that the instability could constrain the performance of other downstream tasks, such as object detection which requires pre-trained AdderNets. Proposed AWN normalizes weights and involves affine transformation to explore the trade-offs between representation power and stability, which can be treated as a modified version of weight standardization. Extensive experiments on stability evaluation and object detection task demonstrates the necessity of proposed method. \n\nThe further discussion of adversarial robustness seems interesting. Due to the difference of addition and multiplication, the paper provides a contrastive analysis of disturbance in AdderNets and CNNs to highlight the potential robustness in AdderNets. The BN statistics in Eq. 9 is further extended to the one under attack in Eq. 16, where the disturbance can be automatically eliminated by the running mean in BN layer, which seems efficient and effective.\n\nOverall, the paper is a good contribution to Adder Neural Networks and adversarial robustness. My major concerns come from the comparison with WS. Proposed AWN is apparently a modified version of existing WS, however, the reason why such a modification is necessary is not fully discussed. Based on the analysis in Eq. 8 and Eq. 9, the instability is highly correlated with the variance of weight, which can be solved through WS. Similarly, WS also holds true for the potential adversarial robustness discussed in Eq. 16. Although I notice that all the empirical evaluations demonstrate that AWN performs better than WS in Table 1-3 and Fig 2, the key factor which brings this improvement is not well-explained in the paper.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of AdderNets have been discussed by the authors in the introduction. For AWN, there exist trade-offs between stability and classification performance, which is discussed in the Sec E of supplementary material.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours."}, "extra_text": {}}, {"review_id": "FSNdlDPF2Wc", "reviewer": "Reviewer_VLQB", "summary": "The paper reveals the existing instability of Adder Neural Networks (ANNs) through the variance estimation of output features and proposes adaptive weight normalization to tackle this issue. Additionally, the adversarial robustness of ANNs is highlighted through further analysis of perturbation variance in both ANNs and CNNs, and the paper proposes to automatically eliminate the perturbations based on the statistics in the batch normalization layer. Experiments on several tasks validate the effectiveness of the proposed method in improving the stability and robustness of ANNs.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The entire paper is easy to follow. The observations in ANNs seem interesting and the motivation is straightforward, which aims at eliminating instability of batch normalization layers according to the lower boundary of feature variance in Eq. 8. The variance study in Section 3 seems well-organized and reasonable. The proposed adaptive weight normalization then comes naturally. Through the discussion of perturbation influence in ANNs compared with CNNs, the paper explains that batch normalization layers can form automatic perturbation eliminations and introduced a simple yet effective inference strategy in Eq. 17.\n- Sufficient experimental results are provided to verify the effectiveness of proposed adaptive weight normalization for improving ANN stability on object detection tasks and stability studies in Sections 5.2 and 5.3. Adversarial robustness is also evaluated through white-box attacks in Table 1. All the experiments show obvious superiority in both network stability and robustness over other baselines.\n \n\nCons:\n\n- My main concern is about the normalization of ANNs. The analysis in Section 3 indicates that the instability mainly comes from the statistics of batch normalization layers in Eq. 9. However, there exist various normalization layers without the statistics involving batch, such as layer normalization. Although the analysis shows that a large variance of feature comes from the weight parameters, I wonder why not directly apply the aforementioned normalization variants to ANNs for improving stability? This discussion or comparison is missing in the paper.\n\nAdditionally, some texts in figures are too small to read, such as Figures 1 and 2.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The proposed ANN-R-AWN seems sensitive to the batch size since the inference strategy involves running mean of batch normalization layer, which has been discussed by the authors. \n\nThere is no immediate social impact.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concern", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "3FYVSLe222R", "reviewer": "Reviewer_oKa2", "summary": "This paper investigates the mean and variance of features for AdderNets and reveals that the instability of AdderNets mainly comes from fluctuations of running mean and variance in batch normalization layers. To this end, this paper proposes the adaptive weight normalization (AWN) to optimize adder weight distributions adaptively. Extensive experiments demonstrate that the proposed method improves AdderNets on both stability and robustness.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1.\tThis paper is technically sound and well written.\n2.\tThe motivation of the proposed method is clear and the proposed idea is quite interesting. The theatrical analysis well justifies the proposed adaptive weight normalization.\n3.\tThe analysis of adversarial robustness for AdderNets makes sense and the proposed method for improving robustness is simple yet effective.\n4.\tThe proposed method shows strong performance for adversarial robustness and presents good stability for downstream tasks like object detection.\n\nCons:\n\n1.\tIn Line 193, the learning rate of trainable parameters in AWN is rescaled by a hyper-parameter 1e-5. How is this parameter determined? Are there more ablation studies for the impact of this hyper-parameter?\n2.\tThe concatenation operations in Eq. (10) and Eq. (11) seem a bit weird and I think the concatenation is not necessary. Please carefully check the formulations.\n3.\tLine 10: “weigh normalization” should be “weight normalization”. Please carefully proofread the manuscript.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please discuss a bit how AdderNets affect the future design of neural networks and corresponding chips since they are tailored for energy-efficient networks.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "36"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on enhancing the stability of AdderNets for performance improvement in downstream tasks. The proposal is the adaptive weight normalization (AWN) to optimize adder weight distributions adaptively. The philosophy behind sounds quite interesting to me, namely, batch normalization layers can form automatic perturbation eliminations. This philosophy leads to a simple yet effective inference algorithm design I have never seen.

The clarity and novelty are clearly above the bar of NeurIPS. While the reviewers had some concerns on the significance, the authors did a particularly good job in their rebuttal. Thus, all of us have agreed to accept this paper for publication! Please include the additional experimental results and merge the reviewers' comments in the next version.
Decision: Accept (Poster)