Towards Stable and Robust AdderNets
Year: 2022

Abstract:
Adder neural network (AdderNet) replaces the original convolutions with massive multiplications by cheap additions while achieving comparable performance thus yields a series of energy-efficient neural networks. Compared with convolutional neural networks (CNNs), the training of AdderNets is much more sophisticated including several techniques for adjusting gradient and batch normalization. In addition, variances of both weights and activations in resulting adder networks are very enormous which limits its performance and the potential for applying to other tasks. To enhance the stability and robustness of AdderNets, we first thoroughly analyze the variance estimation of weight parameters and output features of an arbitrary adder layer. Then, we develop a weight normalization scheme for adaptively optimizing the weight distribution of AdderNets during the training procedure, which can reduce the perturbation on running mean and variance in batch normalization layers. Meanwhile, the proposed weight normalization can also be utilized to enhance the adversarial robustness of resulting networks. Experiments conducted on several benchmarks demonstrate the superiority of the proposed approach for generating AdderNets with higher performance.

heading: 1 Introduction
text: By hint of the success of Deep Convolutional Neural Networks (DCNNs), a wide range of computer vision tasks can be tackled with satisfactory performance, including image classification [13, 9, 10, 25], super-resolution [32, 4, 12, 22], object detection [16, 19, 7, 8], etc.. DCNNs are mainly trained and inferred on graphics processing unit (GPU) devices which suffice to fulfill the high computational complexity and enormous energy consumption. However, it is intractable for DCNNs to be deployed on low-power devices due to the limited computational speed and power capacity, such as mobile and embedded devices. As a result, substantial research efforts have been devoted to energy reduction and speed acceleration of DCNNs [3, 29, 1, 26].
Recently, Chen et al.[1] advocate the use of ℓ1-distance for similarity measure instead of crosscorrelation in CNNs to replace multiplications with additions. Compared with multiplications, addition operations are much cheap, which benefits the power-efficiency [28, 23, 30]. Adder neural network (ANN) has demonstrated extraordinary performance on several computer vision tasks with huge energy reduction, which can be seen as a good complement to the classical CNNs.
The performance achieved by ANNs on classification is impressive, but some observations raise our concerns. e.g., the optimization of ANNs is not stable as CNNs where the test accuracy curve has dramatic fluctuations during the optimization, as shown in Figure 1 (a). The test accuracy of ANNs has a large variation until the end of training while that of convolution networks is much more stable. Furthermore, weights of ANNs have demonstrated a significantly different statistical
∗Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
property from those of CNN weights. The histograms over ANN and CNN weights are shown in Figure 1 (c) and (d) respectively. The red curve in (c) denotes a Laplace distribution with a mean of 0.32 and a variance of 88.47 and the one in (d) denotes a normal distribution with a mean of −0.002 and a variance of 0.001. Although the means of these two distributions are similar, there exists a serious discrepancy between their variances. Given this statistical property difference, those sophisticated training strategies previously developed for CNN might not fulfill their potential when straightforwardly applied to ANNs.
In this paper, we provide an exhaustive analysis of the variance in ANNs. Taking a one-layer adder forward as an example, we demonstrate the large variance of ANN weights can be the major cause of the instability of running mean and variance in batch normalization layer which vibrates the test accuracy. Weight normalization is therefore a natural idea to constrain the variance of weights. To preserve the representation capability of the normalized weights, trainable scaling and shift coefficients are further introduced to achieve an Adaptive Weight Normalization (AWN). By doing so, the variance of ANN weights can be normalized to acceptable levels which significantly stabilizes the batch normalization layers and makes pretrained ANN easily applied to other tasks. Notably, we identify a natural advantage of ANNs to be robust to adversarial perturbations, stemming from ℓ1-distance and the running mean of batch normalization layer in ANNs. The reduction of weight variation across channels further boosts the defense ability against attacks. Without adversarial training, AWN with ANN on ResNet-32 improves adversarial accuracy by 59.73% compared with CNN under PGD attacks. AWN presents superior stability under a series of evaluations and outperforms vanilla ANN in detection task on VOC benchmark with 2.7 mAP improvement.

heading: 2 Preliminary
text: Adder Neural Networks. To significantly reduce energy costs, Chen et al.[1] proposed an Adder Neural Network to release the burden of multiplications in traditional convolution networks by replacing them with additions. Consider an intermediate layer in deep convolution neural network with weight W ∈ Rd×d×cin×cout where d denotes the kernel size and cin and cout are the number of input and output channel respectively. Given the input feature map X ∈ RH×W×cin where H and W are the height and width of input feature, the adder operation is defined as
Y (m,n, c) = − d∑
i=1 d∑ j=1 cin∑ k=1 |X(m+ i, n+ j, k)−W (i, j, k, c)|. (1)
Although the adder operation enables ANNs to achieve similar performance in classification tasks with energy efficiency, the observations in Figure 1 shadow ANNs. The unstable test accuracy curve and large variance almost everywhere in ANNs arouse our interest in the analysis of ANN variance.
Adversarial Attack. Given the input x ∈ RD and the annotated label y ∈ RC where D is the dimension of input and C is the number of classes, the network N maps perturbed input x̃ to ỹ = N (x̃;W ) where x̃ = x+ δ. The objective of adversarial attacks is to find the perturbed input which maximizes the classification loss as
x̃ = argmax x̃:∥x̃−x∥p⩽ϵ ℓ(N (x̃,W ), y), (2)
where the perturbation is constrained by its lp-norm. Through variance study of ANNs and ℓ1-distance analysis, we demonstrate the potential adversarial robustness of ANNs can be activated through proposed inference strategy with AWN.

heading: 3 Variance Study of AdderNet
text: To analyze the variance of ANNs, we first consider a one-layer adder forward with batch normalization [11] and ReLU as activation function. Given output feature yl−1 from previous layer, the forwarding of layer l is formulated as
xl = max(0, yl−1), y ′ l = − n∑ |xl −Wl|, yl = γ
y′l − µl σl + β, (3)
where µl and σl denote the mean and standard deviation, respectively, Wl is the weight, n represents the size of weight of one output channel n = d× d× cin, and γ and β stand for the rescale and shift parameters.
Before we proceed to the weight variance analysis, we first make several assumptions that are empirically plausible. After the batch normalization, yl−1 is supposed to follow a normal distribution N (0, σ2). From empirically observation of a ResNet-18 with ANN on ImageNet, as shown in Figure 1 (b) and (c), adder feature after BN layer follows a distribution with mean of 0.02 and variance of 0.38 while the corresponding weight with mean of 0.32 and variance of 88.47, the adder weights follow Laplace distributions with a large variance but their mean is close to 0 after training. Thus, we assume Wl follows L(µ, b) with the mean as µ and the variance as 2b2 where 2b2 ≫ σ2. Now we compute the mean and variance of each variable. The activation xl follows the Rectified Gaussian distribution. With the law of total expectation, the mean of xl forms the one-side truncation of lower tail E[xl|xl > 0] which can be computed based on the property of Truncated normal distribution as
E[xl] = E[xl|xl > 0] · P (xl > 0) + E[xl|xl ≤ 0] · P (xl ≤ 0)
= E[xl|xl > 0] · P (xl > 0) + 0 = [0 + σ ϕ(0) 1− Φ(0) ] · 1 2 = σ√ 2π , (4)
where ϕ(·) denotes the probability of standard normal distribution and Φ(·) denotes the cumulative distribution function. Similarly, with the law of total expectation, the variance of xl is broken down into variants of expectation and the variance of one-side truncation of lower tail V ar(xl|xl > 0), which can be computed through Truncated normal distribution as
V ar[xl] = E[x 2 l ]− (E[xl])2 = −(E[xl])2 + E[x2l |xl > 0] · P (xl > 0)
= P (xl > 0)[V ar(xl|xl > 0) + (E[xl|xl > 0])2]− (E[xl])2
= [σ2(1− ( ϕ(0) 1− Φ(0) )2) + 4σ2 2π ] · 1 2 − σ 2 2π = σ2( 1 2 − 1 2π ).
(5)
Since we assume 2b2 ≫ σ2 and the variance is further reduced after activation, the distribution of xl −Wl is overwhelmed by Wl to form a Laplace distribution as L( σ√2π − µ, √ b2 + (π−1)σ 2 4π ). For
simplicity, let τµ = σ√2π − µ and b + τσ = √ b2 + (π−1)σ 2
4π . Note that the standard deviation of output last layer σ and the mean of weight µ are both close to zero, which makes τµ and τσ small values. Although it is difficult to directly derive the closed-form distribution expression of y′l, it can be approximated based on variable xl−Wl−τµ. According to the properties of Laplace distribution, the absolute function of L(0, b) follows an Exponential distribution, with which y′l can be approximated as
y′l = − n∑ |xl −Wl| ≥ − n∑ [|xl −Wl − τµ|+ |τµ|], where |xl −Wl − τµ| ∼ Exp([b+ τσ]−1), (6)
where Exp denotes Exponential distribution. Based on the property of Exponential distribution, E[|xl −Wl|] = b+ τσ and V ar[|xl −Wl|] = (b+ τσ)2. With Eq. 6, the lower boundary of E[y′l]
can be derived as E|xl −Wl| ≤ E|xl −Wl − τµ|+ |τµ|, E[y′l] ≥ −n(b+ τσ + |τµ|).
(7)
With inequality in Eq. 7, the lower boundary of V ar[y′l] can be derived by the law of the unconscious statistician as
V ar[y′l] = n∑ [E[|xl −Wl|2]− (E[|xl −Wl|])2]
= n∑ [V ar(xl −Wl) + (E(xl −Wl))2 − (E(|xl −Wl|))2]
≥ n[V ar(xl −Wl) + (E(xl −Wl))2 − (E|xl −Wl − τµ| + |τµ|)2] = n[2(b+ τσ)2 + τ2µ − (b+ τσ + |τµ|)2]
= n[(b+ τσ) 2 − 2|τµ|(b+ τσ)].
(8)
Taking the lower boundary of mean and variance of y′l in Eq. 7, 8, the batch normalization layer can be computed as
yl = γ y′l + n(b+ τσ + |τµ|)√
n[(b+ τσ)2 − 2|τµ|(b+ τσ)] + β. (9)
Note that the batch normalization layers use running mean and variance of current batch in training phase while moving averages in testing phase. Although precise E[y′l] and V ar[y ′ l] can be computed in training phase, the moving averages vary dramatically since b in Eq. 9 grows from a small value to a large one during optimization, as shown in Figure 1 (c) where the distribution of Wl with initial variance of 1.0 is optimized to the one with variance of 88.47. Thus, the large variation of batch statistics results in the instability of testing loss curve while the training loss curve is similar to the one of CNNs, as shown in Figure 1 (a).

heading: 3.1 Adaptive Weight Normalization for AdderNet
text: Based on these observations and analysis, we propose to make some efficient modifications to current ANN optimization. From Eq. 9, the running mean and variance mainly depend on the standard deviation of adder weights so that large magnitude of weights could destabilize the statistics in batch normalization, which indicates that adder weights need normalization to prevent them from being updated to a distribution with large variance. A naive approach is Weight Standardization proposed by Qiao et al.[18] as
W ′i,j = Wi,j − µWi
σWi , where µWi =
1
n n∑ j=1 Wi,j ,
σWi = √√√√ 1 n n∑ j=1 (Wi,j − µWi)2 + ϵ,
(10)
where W ∈ Rcout×n denotes the permuted adder weight, f
denotes concatenation operation and ϵ is added for numerical stability. In Eq. 10, each output channel of adder weight is normalized to a distribution with a mean of 0 and variance of 1, which stabilizes the running mean and variance of batch normalization layer according to Eq. 9. However, with weight standardization directly applied to ANNs, there exists a dramatic accuracy drop. For example, ANN with weight standardization achieves 90.62% in ResNet-20 on CIFAR-10 while original ANN achieves 91.84%, which causes 1.22% accuracy drop. Although adder weights are normalized to guarantee a stable test phase, rigorous mean and variance are strictly assigned to adder weights, which constrains the representation power of ANNs. Since the similarity between filter and input feature is measured by ℓ1-distance in ANNs, the magnitude of weight values can be rather sensitive for network expression capability. Thus, directly applying weight standardization to ANNs can be easily stunk in the local optimum without exploring wider space of adder weights. Instead, we propose to normalize adder weights with trainable variables for each output channel, which preserves the representation power. Eq. 10 can be rewritten as
W ′i,j = νi Wi,j − µWi
σWi + υi, (11)
where νi and υi are trainable variables similar to β and γ in batch normalization layer. Thus, the magnitude of weight values can be automatically adjusted to fit the potential levels of freedom of adder weights. Furthermore, previous analysis demonstrates that the magnitude of gradient w.r.t the input X and the filter W in ANNs is much smaller than that in CNNs [1]. With the incorporation of these two parameters, the gradient w.r.t the filter W can be automatically adjusted. The gradient of loss ℓ w.r.t the weight Wi,j is computed as
∂ℓ
∂Wi,j = n∑ c=1 νi n2σWi { ∂ℓ ∂W ′i,j − ∂ℓ ∂W ′i,c [1 + (Wi,j −Wi,c)(Wi,c − µWi) σWi ] } . (12)
In Eq. 12, the gradient of W is amplified by νi, which relieve the gradient reduction in ANNs.

heading: 4 Activate Potential Adversarial Robustness
text: Consider the intermediate layer forwarding of ANNs and CNNs with perturbation δl, the disturbance of an element on the output feature map before BN layer can be computed as
ỹ′lCNN − y ′ lCNN = n∑ i=1 [(xl,i − δl,i)×Wl,i − xl,i ×Wl,i] = n∑ i=1 (−δl,i)×Wl,i,
ỹ′lANN − y ′ lANN = n∑ i=1 [|xl,i −Wl,i| − |xl,i − δl,i −Wl,i|] ≈ n∑ i=1 ±|δl,i|, (13)
where ± denotes the choices of two possible signs including addition and subtraction. Note that |xl,i −Wl,i| follows a distribution with large mean and variance in ANNs. We assume that δl follows a Gaussian distribution with zero mean and smaller variance N (0, σ2δ ) where σ2δ < V ar[Wl], with which the subtraction |xl,i − Wl,i| − |xl,i − δl,i − Wl,i| has a high probability to be either δl or −δl. We show that it is difficult for ±|δl,i| to have large variance in ANNs under adversarial attacks. In Eq. 13, if V ar[±|δl,i|] grows in ANNs, all the elements in ±|δl,i| tend to select different signs, which automatically eliminate each other to make ỹ′lANN − y ′ lANN ≈ 0. Thus, if attacks succeed, V ar[±|δl,i|] needs reduction to guarantee larger perturbation on feature map. To ensure maximum disturbance, all the elements in ±|δl,i| have the same signs, which forms a sum of n half-normal distributions. On the contrary, each δl,i in CNNs is transformed by different Wl,i in Eq. 13. The variance of disturbance before BN layer can be computed as
V ar[ỹ′l − y ′ l]CNN = V ar[ n∑ i=1 (−δl,i)×Wl,i] = nσ2δ (V ar[Wl] + (E[Wl])2),
V ar[ỹ′l − y ′ l]ANN ≈ V ar[ n∑ i=1 |δl,i|] = nσ2δ (1− 2 π ).
(14)
Eq. 14 indicates the output disturbance variation in CNNs depends on both the statistics of W and δ, and could vary for different output channels while the one in ANNs only depends on the statistics of δ for all the output channels. The major difference lies in BN layer where both ANN and CNN disturbances are rescaled and the variance of perturbation on next layer can be computed as
V ar[δl+1]CNN = nσ 2 δ (V ar[Wl] + (E[Wl]) 2)/[Std(ỹ′lCNN )] 2,
V ar[δl+1]ANN ≈ nσ2δ (1− 2
π )/[Std(ỹ′lANN )]
2. (15)
Note that the variance of ỹ′lANN is much larger than ỹ ′ lCNN . Thus, ANNs have a much smaller disturbance variance on l + 1 layer, which suggests that the perturbations of all the elements on ỹ′l are similar and can be eliminated through subtraction of a scalar value while CNNs cannot copy that success. To activate the adversarial robustness through utilizing this property of ANNs, a simple yet effective method is utilizing the running mean in batch normalization layer for automatic disturbance
elimination. In Eq. 9, τµ becomes τµ − E[δ] while b + τσ nearly remains the same under attack settings since the disturbance has small variance. The forwarding of BN layer can be computed as
ỹ = γ ỹ′ + n(b+ τσ + |τµ − E[δ]|)√
n[(b+ τσ)2 − 2|τµ − E[δ]|(b+ τσ)] + β
≈ γ y′ + ∑n i ±|δi|+ n(b+ τσ + |τµ|)− n|E[δ]|√
n[(b+ τσ)2 − 2|τµ − E[δ]|(b+ τσ)] + β
≈ γ y ′ + n(b+ τσ + |τµ|)√
n[(b+ τσ)2 − 2|τµ − E[δ]|(b+ τσ)] + β.
(16)
The difference between Eq. 9 and Eq. 16 lies in the running variance, which demonstrates that the disturbance can be significantly relieved through computing the running mean in BN layer while the variance is expected to be constant to eliminate the difference brought by E[δ]. Thus, we propose ANN robust inference strategy as
y∗ = γ ỹ − running mean tracked variance + β. (17)
However, considering the actual case where the signs of ±|δl,i| are not the same everywhere, the proposed ANN robust inference strategy cannot work appropriately and results in residual perturbations which will be rescaled by tracked variance after BN layer. In Eq. 13, the perturbations of different output channels are similar in ANNs, which indicates that the noise of feature map can hardly cross channels. However, this property will be broken if tracked variances have enormous differences among channels. Thus, our proposed ANN-AWN can successfully relieve the variation across channels to further improve adversarial robustness, which is verified in later empirical evaluations.

heading: 5 Experiments
text: In this section, we empirically evaluate the superiority of the proposed approach on different tasks and datasets, including the adversarial robustness, object detection and optimization stability.

heading: 5.1 Adversarial Robustness Evaluation
text: White-box Attacks Setup. To demonstrate how the proposed adaptive weight normalization activates the potential adversarial robustness of AdderNet, we conduct a series of experiments. Following [1], we first train both CNNs and ANNs with ResNet20 and ResNet32 on CIFAR-10 under the same settings. CIFAR-10 dataset contains 50K training images and 10K validation images with size of 32 × 32 over 10 classes. We use SGD optimizer with an initial learning rate of 0.1, momentum of 0.9 and a weight decay of 5 × 10−4. The model is trained on single V100, which takes 400 epochs with a batch size of 256 and a cosine learning rate schedule. The learning rate of trainable parameter ν and υ in AWN is rescaled by a hyper-parameter which we set to be 1 × 10−5. For adversarial robustness evaluation, we conduct white-box attacks on these models
including Fast Gradient Sign Method (FGSM) [24], Basic Iterative Method (BIM) [14], Projected Gradient Descent (PGD) [17], Momentum Iterative Method (MIM) [6], and RFGSM [27] to generate adversarial examples. Following previous adversarial literature [17, 31], the adversarial perturbation is considered under l∞ norm with the total perturbation size of 8/255. In iterative attack settings, the step size is set to 2/255. The number of iterations is set to 7 for PGD and BIM attacks, and 5 for MIM and RFGSM attacks. Note that different from traditional defense algorithms which generate adversarial samples for adversarial training [21] or search robust architectures [5, 15], our proposed algorithm utilizes the properties of ANNs to achieve adversarial robustness without any training tricks or modifications of architecture. The experimental results are shown in Table 1. Note that all the models are trained with clean images, which dismisses the expensive adversarial training.
Against White-box Attacks. As shown in the first two rows in Table 1, these adversarial attacks successfully mislead both CNN and ANN to provide wrong predictions. Although ANN has slightly better adversarial accuracy compared to CNN, such as 0.01% → 0.04% under MIM5 attack, the potential adversarial robustness shown in Eq. 13 cannot be activated with normal settings. Thus, we replace the tracked mean in batch normalization layer with the running mean of current batch based on Eq. 17 and compare ANN with CNN in the third and fourth rows, which are denoted as -R. Comparing CNN-R and ANN-R, there exists an enormous adversarial robustness improvement. For example, ANN-R achieves 26.05% accuracy increment from 3.67% → 29.72% under PGD7 compared with CNN-R, and improves the accuracy by 32.21% from 0.04% → 32.25% under MIM5 compared with ANN. The comparison between ANN and ANN-R empirically demonstrates that appropriate utilization of the running mean in batch normalization layer of ANNs can significantly activate the adversarial robustness of ANNs. Furthermore, the comparison between CNN-R and ANN-R shows strong evidence of the natural robustness difference between CNNs and ANNs and indicates that the ℓ1 distance and the independence between perturbation and adder weight provides much better defense than CNNs, which is consistent with the aforementioned variance analysis. The evaluation of proposed AWN is shown in the fifth row. For all the attacks, ANN-R-AWN achieves the best results, which demonstrates the effectiveness of proposed AWN. Comparing ANN-R and ANN-R-AWN, AWN shows obvious superior adversarial robustness. For example, AWN improves adversarial accuracy by 13.67% from 29.72% → 43.39% under PGD7 and 14.98% from 3.38% → 18.36% under RFGSM5, which illustrates that adversarial robustness can be further boosted through narrowing the difference among channels to relieve the perturbation transformation. On ResNet-32, ANN-R achieves worse performance than the one with ResNet-18, which demonstrates that proposed robust inference strategy is not sufficient for superior adversarial robustness. However, ANN-R-AWN consistently achieves better performance, which outperforms other baselines in all the adversarial scenarios, which indicates that the robustness of proposed AWN can generalize to deeper models.
Against Gradually Enhanced Attacks. We highlight the superiority of proposed ANN-R-AWN through enhancing the attacks from different aspects to evaluate the adversarial robustness under more powerful attacks. We use the naturally trained models with ResNet-20 on CIFAR-10 for evaluation. In the first scenario, the total perturbation size ϵ of PGD attack increases from 0.01 to 0.1 with a step size of ϵ/7. In the second scenario, the iterations of PGD attacks are enhanced from 1 to 50. The evaluation results are shown in Figure 2 (a) and (b). ANN-R-AWN obtains much better defense ability than ANN-R, CNN-R and ANN-R-WS baselines when the attack size grows. For example, the adversarial accuracy of CNN-R quickly dive to 2.96%, ANN-R to 27.67% and ANN-R-WS to 28.66 when the PGD attack size reaches 0.06 while ANN-R-AWN maintains 32.67% when the PGD attack size reaches 0.1. Similarly, ANN-R-AWN also achieves superior robustness over other baselines with increasing steps of PGD attack. For example, the adversarial accuracy of ANN-R
under PGD50 becomes 26.48% while ANN-R-AWN maintains 36.52% with 10.04% increment, which demonstrates the effectiveness of AWN. The enormous accuracy drop of ANN-R-WS along with attack epoch and coherent robustness of our proposed AWN illustrate the necessity of introduced adaptive rescale and shift parameters in weight normalization. Thus, the advantage of AWN becomes more obvious under more powerful attacks, which highlights the superiority of proposed AWN.

heading: 5.2 Stability of AdderNets
text: Stability Evaluation through Linear Interpolation. To conduct a quantitative evaluation of AdderNet stability, we propose to evaluate the smoothness of optimization landscape through tracking the performance of AdderNet parameters which are sampled between different training epochs. To highlight the stability of our proposed AWN, we select adjacent epochs for ANN and ANN-AWN trained with ResNet-20 on CIFAR-10 and sample 9 intermediate weights between two epochs through linear interpolation. The comparison is shown in Figure 2 (c). To evaluate the stability of early training stage, we select epoch 160, 164 for ANN as the blue curve and select epoch 125, 135 for ANN-AWN as the green curve. Although we set earlier epoch with larger intervals for ANN-AWN, our proposed algorithm forms a much more smooth curve with a smaller variance 336.00 compared to the one of ANN 587.28. In the later training stage, we select epoch 290, 294 for ANN as the orange curve and select epoch 283− 294 for ANN-AWN as the red curve. Comparing with ANN, our proposed algorithm achieves variance reduction of 585.97 from 627.54 to 41.67. The performance of parameters sampled from linear interpolation reflects the stability of models and the smoothness of the optimization landscape. With the proposed AWN, the optimization of AdderNet can be significantly stabilized, which could benefit the situations where adder layers are difficult to optimize.
Trade-offs Between Stability and Accuracy. To further illustrate how AWN takes effect, we keep track of the test loss during AdderNet optimization and the results of both CNN and ANN with ResNet-20 on CIFAR-10 are shown in Figure 2 (d). The test loss curve of AdderNet is quite unstable before 300 epochs while CNNs, ANN with weight standardization and ANN with adaptive weight normalization all achieve relatively smoothed loss curve, which indicates that the instability of AdderNet mainly comes from the large variance of adder weights and the proposed AWN eliminate it successfully. The dotted area covers from 300 to 400 epochs. Although AdderNet cannot achieve stability due to the large variance of weights, ANNs can still achieve similar performance as CNNs since the variation of adder weights is reduced during the end of training. However, WS fails to achieve a similar classification performance and is stuck at local optimum, which demonstrates that the normalization on adder weights could hurt the expressive power. On the contrary, our proposed AWN achieves relatively better performance through incorporating the adaptive trainable parameters for adder weights, which enables them to shift and rescale back to restore the original performance.
Feature Distribution Analysis. We visualize and track the feature distributions of AdderNet with or without AWN during the training, as shown in Figure 3. We randomly sample features before batch normalization layer at different epochs and compute histograms over them. Both ANN and ANN-AWN feature distributions in the 2nd and 19th layers are shown in Figure 3 (a),(b),(c) and (d) respectively. ANN feature distributions vary dramatically during the optimization, which enormously disturbs the tracking of mean and variance in batch normalization layer. For example, the sampled feature on 21-th epoch in (c) has mean of −2738.58 and standard deviation of 1042.44 while the one on 386-th epoch has mean of −1476.52 and standard deviation of 266.97 with 1262.06 increment on mean and 775.47 reduction on standard deviation. However, those in ANN-AWN become much milder, which stabilizes ANNs. For example, the sampled feature on 21-th epoch in (d) has mean of −940.89 and standard deviation of 269.39 and the one on 385-th epoch has mean of −564.59 and standard deviation of 60.65 with 376.3 increment on mean and 208.74 reduction on standard
deviation. Furthermore, the difference among channels are effectively reduced by AWN to constrain the perturbation space of adversarial examples, e.g., the standard deviation of ANN features on the second layer after training becomes 87.41 while that of ANN-AWN becomes 31.13 with a massive reduction, which potentially provides better defense ability against more powerful adversarial attacks.

heading: 5.3 Experiments on Object Detection
text: To further illustrate the advantage of imposing stability on AdderNets, we conduct experiments on object detection with ANNs on PASCAL VOC (VOC) dataset. VOC contains 20 object classes, the training set includes 10K images which are the union of VOC 2007 and VOC 2012, and the VOC 2007 test set with 4.9K images is used for evaluation. The mAP scores using Iou at 0.5 are reported. All the models are trained with the same setting. Based on the variance study of ANNs, we unfreeze BatchNorm during the training. Following [2], we insert more shortcuts in the neck part. We use an initial learning rate of 0.008 with a linear warmup for 500 iterations, momentum of 0.9, weight decay of 1× 10−4 and a cosine learning rate strategy. All the models are trained on 4 V100 GPUs with SGD optimizer for 12 epochs with a batch size of 4. For the detector baseline, we include both CNN and vanilla ANN for comparison. ANN-FPN replaces the convolution layers with adder layers in the pretrained ResNet-18 backbone and neck of Faster R-CNN [20]. Through applying different types of ANNs to detection, we conduct comparison among the CNN, vanilla ANN [1], ANN-WS and ANN-AWN. The detailed evaluation is shown in Table 2. In Backbone column, the number in brackets denotes the classification accuracy pretrained on ImageNet. Comparing ANN with our proposed AWN, although they achieve similar classification performance, AWN improves the mAP score by 0.8 from 68.6 → 69.4. Even comparing with CNN-FPN which has superior classification performance, our proposed AWN still outperforms it, which demonstrates the necessity of stability. With a much more smooth loss landscape, the optimization of AdderNet on other tasks can be easier and more stable. However, ANN-WS is not competitive with other baselines, with 2.4 mAP reduction compared with AWN, which empirically verifies that directly normalizing adder weights could limit the ability of feature extraction and performance for other tasks. Note that there exists an enormous accuracy drop of pretrained ANN-WS, which significantly constrains its detection performance. Thus, besides stability, the representation power of ANNs can be rather important in terms of applying ANNs to other tasks. On the contrary, our proposed AWN achieves better trade-offs between classification performance and stability through an adaptive scheme, which together achieves the superior mAP score in detection task.

heading: 5.4 Ablation Studies
text: We conduct ablation studies on proposed ANN-R-AWN to illustrate the effectiveness of adaptive training parameters and proposed robust inference strategy. We have already shown that weight standardization can easily be stuck at a local optimum in Figure 2 (d). Although the analysis in Sec 4 works for both WS and AWN, we empirically verifies that the gaps of clean accuracy between WS and AWN still exist in adversarial accuracy. The results are shown in Table 3 where WS and AWN are further attacked and evaluated to demonstrate the influence of performance drop on clean and adversarial accuracy. Comparing WS and AWN, our proposed AWN consistently outperforms WS in both adversarial and clean accuracy, which indicates that directly normalizing adder weights could hurt the representation power of ANNs and restrict the adversarial accuracy of ANNs. With proposed parameters ν and υ in Eq. 11, AWN successfully relieves this problem through exploring the balance between expressive power and weight magnitude reduction, which achieves better classification
performance and adversarial robustness. We further evaluate the effectiveness of proposed ANN robust inference strategy as in Eq. 17. We denote the strategy which replaces both tracked mean and variance with running ones as -r and our proposed one as -R. Comparing two inference methods, our proposed strategy consistently outperforms -r, which verifies that the activated robustness mainly comes from the running mean which automatically eliminates the perturbations.

heading: 6 Conclusion
text: In this paper, we investigate the major concerns of AdderNets through approximating the mean and variance of output features of an arbitrary adder layer. With a derived lower boundary, we show that the instability of AdderNets mainly comes from drastic fluctuations of running mean and variance in batch normalization layer whose computation is dominated by the variance of weights. Our proposed adaptive weight normalization (AWN) works with AdderNets to optimize adder weight distributions adaptively, which significantly improves the stability and leads to smooth landscape. Our analysis of the adder layer forwarding reveals the potential superior defense ability of AdderNets against perturbations and proposed robust inference strategy together with AWN successfully activate the adversarial robustness. Experiments conducted on stability and robustness demonstrate the superior performance of the proposed ANN-AWN.

heading: Acknowledgment
text: The authors would like to thank the Area Chair and the reviewers for their constructive comments. This work was supported in part by the Australian Research Council under Projects DE180101438 and DP210101859.

Review summaries:
This is an interesting paper to discuss the variance in the adder neural network. The authors have an in-depth comparison between ANN and CNN. Also, the adder operation in the neural network can enjoy some nature advantage over CNN in terms of the adversarial robustness. The authors have conducted comprehensive experiments to support these analyses.

The paper reveals the existing instability of Adder Neural Networks (ANNs) through the variance estimation of output features and proposes adaptive weight normalization to tackle this issue. Additionally, the adversarial robustness of ANNs is highlighted through further analysis of perturbation variance in both ANNs and CNNs, and the paper proposes to automatically eliminate the perturbations based on the statistics in the batch normalization layer. Experiments on several tasks validate the effectiveness of the proposed method in improving the stability and robustness of ANNs.

This paper aims at enhancing the stability of AdderNets for performance improvement in downstream tasks. For an adder layer, approximations of feature variance and expectation are derived, with which the variance of weight is found to be the major cause of instability. Adaptive weight normalization (AWN) is thus introduced for addressing this problem. Together with the feature variance approximation, the potential adversarial robustness of AdderNets is also analyzed through comparing the perturbation variance in both AdderNets and CNNs. A simple robust inference strategy with BN layer is thus designed for disturbance elimination.

This paper investigates the mean and variance of features for AdderNets and reveals that the instability of AdderNets mainly comes from fluctuations of running mean and variance in batch normalization layers. To this end, this paper proposes the adaptive weight normalization (AWN) to optimize adder weight distributions adaptively. Extensive experiments demonstrate that the proposed method improves AdderNets on both stability and robustness.