Towards Stable and Robust AdderNets

1 Introduction . By hint of the success of Deep Convolutional Neural Networks ( DCNNs ) , a wide range of computer vision tasks can be tackled with satisfactory performance , including image classification [ 13 , 9 , 10 , 25 ] , super-resolution [ 32 , 4 , 12 , 22 ] , object detection [ 16 , 19 , 7 , 8 ] , etc .. DCNNs are mainly trained and inferred on graphics processing unit ( GPU ) devices which suffice to fulfill the high computational complexity and enormous energy consumption . However , it is intractable for DCNNs to be deployed on low-power devices due to the limited computational speed and power capacity , such as mobile and embedded devices . As a result , substantial research efforts have been devoted to energy reduction and speed acceleration of DCNNs [ 3 , 29 , 1 , 26 ] . Recently , Chen et al . [ 1 ] advocate the use of ℓ1-distance for similarity measure instead of crosscorrelation in CNNs to replace multiplications with additions . Compared with multiplications , addition operations are much cheap , which benefits the power-efficiency [ 28 , 23 , 30 ] . Adder neural network ( ANN ) has demonstrated extraordinary performance on several computer vision tasks with huge energy reduction , which can be seen as a good complement to the classical CNNs . The performance achieved by ANNs on classification is impressive , but some observations raise our concerns . e.g. , the optimization of ANNs is not stable as CNNs where the test accuracy curve has dramatic fluctuations during the optimization , as shown in Figure 1 ( a ) . The test accuracy of ANNs has a large variation until the end of training while that of convolution networks is much more stable . Furthermore , weights of ANNs have demonstrated a significantly different statistical ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . property from those of CNN weights . The histograms over ANN and CNN weights are shown in Figure 1 ( c ) and ( d ) respectively . The red curve in ( c ) denotes a Laplace distribution with a mean of 0.32 and a variance of 88.47 and the one in ( d ) denotes a normal distribution with a mean of −0.002 and a variance of 0.001 . Although the means of these two distributions are similar , there exists a serious discrepancy between their variances . Given this statistical property difference , those sophisticated training strategies previously developed for CNN might not fulfill their potential when straightforwardly applied to ANNs . In this paper , we provide an exhaustive analysis of the variance in ANNs . Taking a one-layer adder forward as an example , we demonstrate the large variance of ANN weights can be the major cause of the instability of running mean and variance in batch normalization layer which vibrates the test accuracy . Weight normalization is therefore a natural idea to constrain the variance of weights . To preserve the representation capability of the normalized weights , trainable scaling and shift coefficients are further introduced to achieve an Adaptive Weight Normalization ( AWN ) . By doing so , the variance of ANN weights can be normalized to acceptable levels which significantly stabilizes the batch normalization layers and makes pretrained ANN easily applied to other tasks . Notably , we identify a natural advantage of ANNs to be robust to adversarial perturbations , stemming from ℓ1-distance and the running mean of batch normalization layer in ANNs . The reduction of weight variation across channels further boosts the defense ability against attacks . Without adversarial training , AWN with ANN on ResNet-32 improves adversarial accuracy by 59.73 % compared with CNN under PGD attacks . AWN presents superior stability under a series of evaluations and outperforms vanilla ANN in detection task on VOC benchmark with 2.7 mAP improvement . 2 Preliminary . Adder Neural Networks . To significantly reduce energy costs , Chen et al . [ 1 ] proposed an Adder Neural Network to release the burden of multiplications in traditional convolution networks by replacing them with additions . Consider an intermediate layer in deep convolution neural network with weight W ∈ Rd×d×cin×cout where d denotes the kernel size and cin and cout are the number of input and output channel respectively . Given the input feature map X ∈ RH×W×cin where H and W are the height and width of input feature , the adder operation is defined as Y ( m , n , c ) = − d∑ i=1 d∑ j=1 cin∑ k=1 |X ( m+ i , n+ j , k ) −W ( i , j , k , c ) | . ( 1 ) Although the adder operation enables ANNs to achieve similar performance in classification tasks with energy efficiency , the observations in Figure 1 shadow ANNs . The unstable test accuracy curve and large variance almost everywhere in ANNs arouse our interest in the analysis of ANN variance . Adversarial Attack . Given the input x ∈ RD and the annotated label y ∈ RC where D is the dimension of input and C is the number of classes , the network N maps perturbed input x̃ to ỹ = N ( x̃ ; W ) where x̃ = x+ δ . The objective of adversarial attacks is to find the perturbed input which maximizes the classification loss as x̃ = argmax x̃ : ∥x̃−x∥p⩽ϵ ℓ ( N ( x̃ , W ) , y ) , ( 2 ) where the perturbation is constrained by its lp-norm . Through variance study of ANNs and ℓ1-distance analysis , we demonstrate the potential adversarial robustness of ANNs can be activated through proposed inference strategy with AWN . 3 Variance Study of AdderNet . To analyze the variance of ANNs , we first consider a one-layer adder forward with batch normalization [ 11 ] and ReLU as activation function . Given output feature yl−1 from previous layer , the forwarding of layer l is formulated as xl = max ( 0 , yl−1 ) , y ′ l = − n∑ |xl −Wl| , yl = γ y′l − µl σl + β , ( 3 ) where µl and σl denote the mean and standard deviation , respectively , Wl is the weight , n represents the size of weight of one output channel n = d× d× cin , and γ and β stand for the rescale and shift parameters . Before we proceed to the weight variance analysis , we first make several assumptions that are empirically plausible . After the batch normalization , yl−1 is supposed to follow a normal distribution N ( 0 , σ2 ) . From empirically observation of a ResNet-18 with ANN on ImageNet , as shown in Figure 1 ( b ) and ( c ) , adder feature after BN layer follows a distribution with mean of 0.02 and variance of 0.38 while the corresponding weight with mean of 0.32 and variance of 88.47 , the adder weights follow Laplace distributions with a large variance but their mean is close to 0 after training . Thus , we assume Wl follows L ( µ , b ) with the mean as µ and the variance as 2b2 where 2b2 ≫ σ2 . Now we compute the mean and variance of each variable . The activation xl follows the Rectified Gaussian distribution . With the law of total expectation , the mean of xl forms the one-side truncation of lower tail E [ xl|xl > 0 ] which can be computed based on the property of Truncated normal distribution as E [ xl ] = E [ xl|xl > 0 ] · P ( xl > 0 ) + E [ xl|xl ≤ 0 ] · P ( xl ≤ 0 ) = E [ xl|xl > 0 ] · P ( xl > 0 ) + 0 = [ 0 + σ ϕ ( 0 ) 1− Φ ( 0 ) ] · 1 2 = σ√ 2π , ( 4 ) where ϕ ( · ) denotes the probability of standard normal distribution and Φ ( · ) denotes the cumulative distribution function . Similarly , with the law of total expectation , the variance of xl is broken down into variants of expectation and the variance of one-side truncation of lower tail V ar ( xl|xl > 0 ) , which can be computed through Truncated normal distribution as V ar [ xl ] = E [ x 2 l ] − ( E [ xl ] ) 2 = − ( E [ xl ] ) 2 + E [ x2l |xl > 0 ] · P ( xl > 0 ) = P ( xl > 0 ) [ V ar ( xl|xl > 0 ) + ( E [ xl|xl > 0 ] ) 2 ] − ( E [ xl ] ) 2 = [ σ2 ( 1− ( ϕ ( 0 ) 1− Φ ( 0 ) ) 2 ) + 4σ2 2π ] · 1 2 − σ 2 2π = σ2 ( 1 2 − 1 2π ) . ( 5 ) Since we assume 2b2 ≫ σ2 and the variance is further reduced after activation , the distribution of xl −Wl is overwhelmed by Wl to form a Laplace distribution as L ( σ√2π − µ , √ b2 + ( π−1 ) σ 2 4π ) . For simplicity , let τµ = σ√2π − µ and b + τσ = √ b2 + ( π−1 ) σ 2 4π . Note that the standard deviation of output last layer σ and the mean of weight µ are both close to zero , which makes τµ and τσ small values . Although it is difficult to directly derive the closed-form distribution expression of y′l , it can be approximated based on variable xl−Wl−τµ . According to the properties of Laplace distribution , the absolute function of L ( 0 , b ) follows an Exponential distribution , with which y′l can be approximated as y′l = − n∑ |xl −Wl| ≥ − n∑ [ |xl −Wl − τµ|+ |τµ| ] , where |xl −Wl − τµ| ∼ Exp ( [ b+ τσ ] −1 ) , ( 6 ) where Exp denotes Exponential distribution . Based on the property of Exponential distribution , E [ |xl −Wl| ] = b+ τσ and V ar [ |xl −Wl| ] = ( b+ τσ ) 2 . With Eq . 6 , the lower boundary of E [ y′l ] can be derived as E|xl −Wl| ≤ E|xl −Wl − τµ|+ |τµ| , E [ y′l ] ≥ −n ( b+ τσ + |τµ| ) . ( 7 ) With inequality in Eq . 7 , the lower boundary of V ar [ y′l ] can be derived by the law of the unconscious statistician as V ar [ y′l ] = n∑ [ E [ |xl −Wl|2 ] − ( E [ |xl −Wl| ] ) 2 ] = n∑ [ V ar ( xl −Wl ) + ( E ( xl −Wl ) ) 2 − ( E ( |xl −Wl| ) ) 2 ] ≥ n [ V ar ( xl −Wl ) + ( E ( xl −Wl ) ) 2 − ( E|xl −Wl − τµ| + |τµ| ) 2 ] = n [ 2 ( b+ τσ ) 2 + τ2µ − ( b+ τσ + |τµ| ) 2 ] = n [ ( b+ τσ ) 2 − 2|τµ| ( b+ τσ ) ] . ( 8 ) Taking the lower boundary of mean and variance of y′l in Eq . 7 , 8 , the batch normalization layer can be computed as yl = γ y′l + n ( b+ τσ + |τµ| ) √ n [ ( b+ τσ ) 2 − 2|τµ| ( b+ τσ ) ] + β . ( 9 ) Note that the batch normalization layers use running mean and variance of current batch in training phase while moving averages in testing phase . Although precise E [ y′l ] and V ar [ y ′ l ] can be computed in training phase , the moving averages vary dramatically since b in Eq . 9 grows from a small value to a large one during optimization , as shown in Figure 1 ( c ) where the distribution of Wl with initial variance of 1.0 is optimized to the one with variance of 88.47 . Thus , the large variation of batch statistics results in the instability of testing loss curve while the training loss curve is similar to the one of CNNs , as shown in Figure 1 ( a ) .

Summary: This is an interesting paper to discuss the variance in the adder neural network. The authors have an in-depth comparison between ANN and CNN. Also, the adder operation in the neural network can enjoy some nature advantage over CNN in terms of the adversarial robustness. The authors have conducted comprehensive experiments to support these analyses.