Handling Long-tailed Feature Distribution in AdderNets
Authors: Minjing Dong
Yunhe Wang
Xinghao Chen
Chang Xu
Venue: NeurIPS 2021 Poster (2021)
Keywords: AdderNet
Efficient Networks
Multivariate Skew Laplace

Abstract: Adder neural networks (ANNs) are designed for low energy cost which replace expensive multiplications in convolutional neural networks (CNNs) with cheaper additions to yield energy-efficient neural networks and hardware accelerations. Although ANNs achieve satisfactory efficiency, there exist gaps between ANNs and CNNs where the accuracy of ANNs can hardly be compared to CNNs without the assistance of other training tricks, such as knowledge distillation. The inherent discrepancy lies in the similarity measurement between filters and features, however how to alleviate this difference remains unexplored. To locate the potential problem of ANNs, we focus on the property difference due to similarity measurement. We demonstrate that unordered heavy tails in ANNs could be the key component which prevents ANNs from achieving superior classification performance since fatter tails tend to overlap in feature space. Through pre-defining Multivariate Skew Laplace distributions and embedding feature distributions into the loss function, ANN features can be fully controlled and designed for various properties. We further present a novel method for tackling existing heavy tails in ANNs with only a modification of classifier where ANN features are clustered with their tails well-formulated through proposed angle-based constraint on the distribution parameters to encourage high diversity of tails. Experiments conducted on several benchmarks and comparison with other distributions demonstrate the effectiveness of proposed approach for boosting the performance of ANNs.

Handling Long-tailed Feature Distribution in AdderNets
Minjing Dong' :2, Yunhe Wang"; Xinghao Chen', Chang Xu' Schooo of Computer Science, University of Sydney Huawei Noah's Ark Lab
mdon07360uni. sydney....yyyyy.y..... edu. adu,au, yunhe wang@huawei com, xinghao.. www.newwween....cccmmmmmmmm com, x.udsyyneey.cm edu.au au
Abstract
Adder neural networks (ANNs) are designed for low energy cost which replace expensive multiplications in convolutional neural networks (CNNs) with cheaper additions to yield energy-efficient neural networks and hardware accelerations. Although ANNs achieve satisfactory efficiency, there exist gaps between ANNs and CNNs where the accuracy of ANNs can hardly be compared to CNNs without the assistance of other training tricks, such as knowledge distillation. The inherent discrepancy lies in the similarity measurement between filters and features, however how alleviate this difference remains unexplored. To locate the potential problem of ANNs. we focus on the property difference due to similarity measurement. We demonstrate that unordered heavy tails ANNs could the key component which prevents ANNs from achieving superior classification performance since fatter tails tend overlap in feature space. Through pre-defining Multivariate Skew Laplace distributions and embedding feature distributions into the loss function, ANN features can be fully controlled and designed for various properties. We further present novel method for tackling existing heavy tails in ANNs with only a modification of classifier where ANN features are clustered with their tails well- formulated through proposed angle-based constraint the distribution parameters to encourage high diversity of tails. Experiments conducted on several benchmarks and comparison with other distributions demonstrate the effectiveness of proposed approach for boosting the performance of ANNs.
Introduction
Deep Convolutional Neural Networks have been widely adopted in various computer vision tasks due to their satisfactory performance, including image classification [12 10 20). object detection
13 17 6, 7), super-resolution [29 [5] [8), etc.. However, the success of CNNs can hardly be presented for practical usage without further modifications since the majority of computer vision tasks usually deployed on low-power platforms, such as portable and embedded devices, where the computational resources are significantly constrained while prompt inference is required. As result, the acceleration of deep neural networks and reduction of energy cost become urgent requirements and attract massive research efforts in recent years 8
24 15
30 21].
Recently, Chen et al. (3] introduced Adder Neural Network to replace the cross-correlation in CNNs with formrmm for the similarity measurement between input features and filters, which eliminates the massive multiplications in CNNs. Through replacing existing multiplications with additions in deep neural networks, ANNs achieve considerable energy reduction since addition is a cheaper operation than multiplications [19 23 Furthermore, ANN can be more friendly for hardware designs of deep
learning acceleration 25 28 Although ANNs achieve comparable results with CNNs, there exist
"Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Owwlagged tall
Diserantienst
(a) ANN weght
(00 ANN felure
Skewness construnt
(a) and (b) denote histograms of ANN weights and features respectively. (c) denotes a toy example of overlapped heavy tails of feature distributions and a solution through skewness constraint.
Figure 1:
gaps in classification task (e.g. 67.0% top-1 accuracy on ImageNet with ANN-18 while 69.8% with CNN-18). Progressive Kernel Based Knowledge Distillation (PKKD) has been proposed to establish ANN with better performance. However concomitant CNN with the same architecture and parallel optimization of both ANN and CNN are required, which largely raises the cost training [26]. The natural difference between CNNs and ANNs mainly lies in the similarity measurement, which is always ignored and how to relieve the gaps from inherent properties of ANNs remains unexplored. Thus, the difference caused by the replacement of cosine distance with & norm is shed light to explore potentially more powerful classification capability of ANNs.
The discrepancy between ANNs and CNNs due to different similarity measurements lies in various aspects. this work, we mainly focus on the ANN features for classification. A well-studied property of ANNs is that the features corresponding to different classes are clustered towards different centers since & norm is used for similarity measurement [3. Meanwhile, ANN weights tend to follow a Laplace distribution which denotes the prior of &1- Actually the heavy-tailed distributions widely exist not only in ANN weights but also in ANN features, as shown in Figure (a) and (b). Since the features clustered for different classes, the existing fatter tails can be easily overlapped with each other, which could hurt the classification performance.
In this paper, propose to pre-define the feature distributions in order to model the heavy-tailedhee in ANNs. We demonstrate the limitations of Gaussian for ANNs and instead make use of a mixture of Multivariate Skew Laplace Distributions which not only involves mean and variance for optimization but also includes higher-order moment skewness. With skewness, the overlapped areas of heavy tails can be reduced for preventing the entanglement of feature distributions. We propose to embed this mixture of skew Laplace into the loss function through substituting the distribution parameters for the classifier head. A likelihood regularization comes naturally for fitting ANN features pre-defined distributions. With distribution parameters, we introduce a well-formulated angle-based constraint the feature distributions based their locations, covariance and skewness, which drives the distribution tails to different angle regions for disentanglemennt Our proposed method improves the classification accuracy by 0.7% both CIFAR-100 with ResNet-20 and ImageNet with ResNet-18 -8 compared to vanilla ANN with only a modification to the ANN classifier head.
2 Preliminaries
Adder Neural Networks. Chen el al. proposed the Adder Neural Network to eliminate the multiplications in traditional convolution networks and replace them with additions significantly reduce computational and energy costs. Consider an intermediate feature map X E R ff "f' XXXWxx, x in deep neural network with weight W x X where H, W denote the height and width of input feature, denotes the kernel size, and Cin, Cout denote the number of input and output channel respectively. The convolution and adder operation are defined as
NO
Y Yoonu (m, n, c)
X(m + i, monij,k j,kk x W(i,i,k,e),
10
w
Cin
Yadder (m, c)
X(m + i, j.k) W(i.j,
Comparing the adder operation with the traditional convolution in Eq. 1 the dot product is replaced by the norm for measuring the similarity between the filter and input feature. Although ANN can achieve similar performance to CNN, there still exist gaps between ANNs and CNNs. For example,

---

ANN has 0.7% accuracy drop with ResNet- 32 on CIFAR- 100 compared to CNN. Since the major difference between ANN and CNN lies in the operation, we mainly focus on addressing the potential adverse properties of this operation substitution for improving ANN performance.
3 Skew Laplace Mixture Loss with Angle-based Constraint
In this section, we analyze potential problems of ANNs and propose a novel framework which replaces classifier head with appropriate trainable distribution parameters. Furthermore, we introduce an angle-based constraint for controlling feature distribution tails to avoid potential entanglement.
3.1 Embedding Skew Laplace Mixture into Loss Function
With 4--ormm as similarity measurement, the weights in ANNS close to Laplace distribution since the prior of fn-nrrm is Laplace distribution, as discussed in 3. Laplace density is expressed by the absolute difference from mean while Gaussian density is expressed by the squared difference, which results in a fatter tails of Laplace distributions compared to Gaussian distributions. We empirically verifies the heavy tails in both ANN weights and features through histograms of sampled layer from pre-trained ANN with ResNet32 on CIFAR-10, as shown in Figure (() important property lies in the features for classification. As discussed and verified in (3), ANN features are clustered and classified by Manhattan distance while CNN features by cosine distance. Combing aforementioned properties, a concern arises that the heavy tails could become potentially troublesome issue for & nn-ro based clustering classification since the feature tails corresponding to different classes are more likely to be overlapped, which significantly reduces the classification margin and constrains the generalization of ANNs, as illustrated in Figure (c). Directly tackling this issue can be rather difficult since the distributions of features are unknown, which motivates us to pre-define an appropriate feature distribution for ANNs to obtain high-level control of heavy tails.
(a) and (b). Another
Thus, make assumption that deep features of neural networks follow learnable distributions, with which the feature distribution can be better formed to achieve expected properties. An naive selection can be Multivariate Gaussian distribution (H,,) 2) where μ denotes the mean and Σ denotes the covariance matrix. However, there exist several concerns in ANNs. For example, the features and weights of ANN always form long-tailed distributions instead of the bell curves with Gaussian distributions, which indicates that pre-defined Gaussian distributions might not fit the real ANN features well. Moreover, Gaussian distribution contains two parameters μl and E for optimization. However, the heavy-tailedness can hardly be controlled by these two lower-order moments, which indicates higher-order moment is required to be involved in pre-defined distribution. Thus, Multivariate Gaussian distribution needs to be replaced by a more appropriate one in ANNs.
In this work, we make use of Multivariate Skew Laplace distribution (SL.) ① to fulfill aforementioned requirements. Considering classification task with K classes, the density of ANN last-layer features _2d with respect its corresponding class k is given by
1=/21/2 1/2
fsz uu, Tx) (d-1)/1)//
(B-ML)"EE (x-pe (e- m EE TH
(2)
where 14, E, y denote the location, covariance and skewness parameters respectively, ag /1 /1+ vf 2 7k and I' denotes the Gamma function. Skew Laplace distribution is selected for several reasons. First, Laplace distribution fits ANN well since ANN filters empirically follow Laplace distributions and the heavy tails of ANN features can be well-approximated by Laplace distributions. Moreover, the skewness parameters in Multivariate Skew Laplace naturally incorporate a high-level control of distribution tails in ANNs. According to the definition of skewness, it indicates the direction and relative magnitude of distribution deviation from its center location. Through optimizing skewness parameters to appropriate values, the overlapped area of heavy tails can be easily eliminated since skewness directly controls the shape of distributions, especially the direction of heavy tails, to achieve disentanglement in angle space, as shown in [
Note that each class k follows independent SC(H,, ΣR, Yk). which makes ANN feature x follows a mixture of Multivariate Skew Laplace distributions. The distribution of x can be computed as
p(x) =
SL pas Ex, (k)())
(3)
3

---

Stlan
Consiaant
PyIl B
1A,000
Sclpa S.
Sclpu Ei
SClps Eo
Feeiune
lcodaan
Affinity
Milchrolis disaroo
Input
Festure Exhaction
Clessifier
Figure 2: An illustration of proposed method. ANN feature extraction network remains the same while proposed classifier imposes constraints on features which involves both Mahalanobis and cosine distance to form clustering classification with distribution tails dispersed in different directions.
where pkk denotes the prior probability. Although the real distribution of ANN feature is hard to derive, how well the extracted features fit this mixture of Multivariate Skew Laplace distributions can be measured through sampling subsets from training set and computing the negative log likelihood as
Lill logg C(u,, E, 7/x)] -log[/sI (a) Es. m))p()))
N
= Okk + E. + v (o: px) (ar p4k)
Σ
(4)
==1
(x: un)
> - log[p(A))]
where L denotes the likelihood. Through incorporating the negative log likelihood into the objective loss, ANN features can be well-formulated learned SL(Mk, Σg, Tk). Since ANN features now follow mixture of Multivariate Skew Laplace distributions, the classification loss can be easily derived. With the predefined distribution with its parameters, we can derive the conditional probability of class label y; E [1, K] through Bayes' theorem. Similar to the traditional classification loss which computes the cross-entropy between affinity scores and one-hot encoding of class labels, the classification loss can be computed through treating p(us (24) as the affinity score as
N
log p( p(ailv))((s)) P p(x:)
L.
loglp(u
N
N
1=1
(5)
1
( My: Ee Vos )p(()) Σ S'S (245 Mua, Eg, (p)p(k))
logl
N
Through combing Eq. and 5 we embed the feature distributions into the loss function and make follow a mixture of SL distributions, where the observed heavy tails can be parameterized and direct connection between ANN features and trainable distribution parameters can be established to provide a focus for tackling existing overlapping issue.
3.2
Angle-based Constraint
Note that the prior probability p(k) = K and the distribution parameters of different classes are independent from each other, which indicates there no direct correlation among different feature distributions. If we directly optimize SL(M,. Σx, 7k:) for each class, it could be difficult to reduce the overlapping of distribution tails since each class is formulated independently in Sec without taking feature distributions of other classes into consideration. Thus, we take the feature distributions of all the classes as a whole and propose to explore the constraints among classes which diverge the heavy tails of their feature distributions enlarge classification margin. Since the type and corresponding parameters of feature distributions are available during training phase, constraints can be easily applied the distribution parameters Σ and y to achieve desired performance.

---

We now introduce simple yet effective constraint on the feature distributions. which leads to diversity of distribution tails. We first explore the existing similarity measurements in proposed classifier. The
location 1s and covariance form Mahalanobis distance V(r - -= - (1x)) in Eq. 2 for
measuring the distance from feature x to the center of class k. Similarly, the involvement of skewness
forms inner product (xr - pu,) ) Ex 1 in Eq. 2 for measuring the cosine distance from normalized feature z to the skewness of class k. Now we iocus on the similarity measurement among classes. Given random pair of classes m and n, the relative distance from class m to can be easily derived through replacing feature x with the distribution parameters of other classes in aforementioned distance. Since our objective is incorporating disentanglement of feature distributions, we propose to arrange their heavy tails to different angle regions. In order fulfill this potential, we measure the distance of feature distribution skewness among different classes and impose constraint to encourage
tail divergence in angle space, Given set of skewness parameters "1:K [?, Y2, .... .k]] E RR K xd we compute the skewness distance D, E R through computing the cosine similarity between each class pair. For example, Dy between class m and n is given by
7'm Th2
Dy n) = cos0(mmm (m))
(6)
7l
Although D., can be casily derived, the target D. remains undefined. Note that the skewness itself denotes the direction and relative magnitude of its corresponding distribution heavy-tailednen which controls the shape of distribution and provides an implicit similarity measurement among classes, as illustrated in Figure 2 Meanwhile, the first-order moments ju provides a more explicit measurement for describing the distance among classes, which suggests the distances among the centers of different clusters could become an indicator for obtaining target skewness distance among classes. Thus, we propose to make use of location parameters to obtain an adaptive target D.,. For class m, we first take the location parameter fmm as the center point and make use of square of Mahalanobis distance for measuring the location difference between (4u and the feature distributions of other classes as
(7) where SLn, denotes the Multivariate Skew Laplace distribution corresponding to class n and D denotes the Mahalanobis distance. With Eq. 6 and 7 we design the constraint based on two simple principles: (a). For class m, target (,,,,, n) where n E 2..,.... K] can always achieve the maximum and minimum values of cosine distance to fully utilize the angle space; (b). (,(m,,)) n) determine how D,, (m, n) are distributed in range [-1.1] 1] where the tail direction difference D. is inverse proportion to the location difference D,.. Thus, the propose angle-based constraint is given by
(m, n) = D'sc, (Hm) = (Im - =,))' (1m - -Ma),
K
K
Lac where D, n) max (D 1:K)) - -2 22D,,(m,
(D, (m,,,) m)) - (m,, n)),
K2
m-
n=1
(8)
max(PP, (m,K)
where function max takes the maximum value among all K classes. Eq. 8 computes the mean squared error between current skewness distance and target one, where D.,, purely depends on current location and covariance parameters of feature distributions to form an adaptive target distance. D, is designed by projecting the pre-computed Mahalanobis distance into range [-1, in reverse order. Note that D. (m, m) becomes 1 since (m,, m) 0 and (,,,,, n)) becomes -1 when class n has the maximum Mahalanobis distance (m,,,, n) = max maax(D,ll (m,, 1:K)). An illustration of how proposed constraint influence the pre-defined feature distributions is shown in Figure 2 With the combination of Mahalanobis and cosine distance for angle- based constraint, Skew Laplace distribution introduces a clustering classification without long tails entanglement through controlling the heavy tails. Finally, combing Eq. 4. Eq. and Eq. g the objective loss is defined as
Lakj L. + XLall + 3Lac,
(9)
where \ and 3 are the hyper- papre-parameters for balancing these terms.
3.3 Optimization
Our final objective loss named Skew Laplace Loss with angle-based constraint (SLACC can be simply optimized through stochastic gradient decent (SGD). Similar to vanilla ANN, we make use
5

---

Algorithm Skew Laplace Mixture Loss with Angle-based Constraint for AdderNet
The training set [AK))) y); Batch size Hyperparameter X, 93 Initialize ANN network / with W where W is initialized by uniform distribution; Initialize Multivariate Skew Laplace distributions SL( Σ, where ja is initialized by Xavier initialization, Σ by 1.0 and 7 by 0.0;
Input:
while not converge do Sample a batch of data "K from (X Forward ANN network to get feature -7= (1 W): Forward classifier to get affinity score y 2==n with Eq. 5
Calculate objective loss Loby with Eq.
Update the SL, 7) through SGD and pass gradients to A with Eq. 10:
Update ANN parameters W through full-pre full-precision gradient with SGD; end while
of full- call-precision gradient to update filters W, which is computed based on norm The major difference lies in the gradients passed by the classifier. In SLAC ANN, the partial derivative of Lobj with respect to last-layer feature is calculated as
aLobj Ox
-1-110 p(u((() + (a ((x (yy)) ( E (x (0-1)))) (gy))) -1/2 (x Myy) + 7o.)
(10)
Σ p(()))(E Th aax(1 DE 2 - paa) - 1/2 k
pk)).
Although proposed constraint Lae is not directly involved in Eq. 10 the gradients passed to ANN layers from classifier are heavily determined by the feature distribution parameters μ4, E and y, which are significantly influenced by Lac. Similar to Eq. T0 the partial derivative of Lotj with respect to
J., Σ and y can be computed through SGD accordingly. The entire algorithm named Skew Laplace Mixture Loss with Angle-based constraint for AdderNet (SLAC-ANN) is shown in Algorithm [ Since our proposed SLAC-ANN involves a constraint on the distribution parameters to encourage tail diversity in angle space which significantly improves the classification margin, the complexity of optimization is relatively enlarged The superiority of our method becomes more obvious when the number of epochs are enlarged, which will be verified in empirical evaluations.
4 Experiments
In this section, conduct empirical evaluation of the proposed SLAC ANN on several image classi- fication benchmarks, including 100,R-10, CIFAR- 100 and ImageNet. ANN variants comparison and ablation studies are provided to demonstrate the effectiveness of the proposed algorithm.
4.1 Experiments on CIFAR
CIFAR-10 10 and CIFAR- dataset contain 50K training images and 10K validation images with size of 32 x 32 from 10 categories. All the training and validation sets are pre-processed according to the same protocol as in [9] We make use of SGD optimizer with an initial learning rate of 0. 0.1, weight decay of 5 x 10-4, momentum of 0.9 and a cosine learning rate schedule. The entire training takes 800 epochs with batch size of 256. The learning rate of trainable parameter Σ is downscale set to 0.01 and 3 to 0.1. For comparison, we include CNN, ANN and binary
by x 10a, neural network (BNN) 300 as baselines. Note that the first and last layers are set to full-precision convolutional layer in vanilla ANN B For a fair comparison, SLAC-ANN adopts a similar setting, which uses convolutional operation for the first layer but replaces the last layer by the proposed SL distributions parameters. All the baseline results are cited from [
The classification results are reported in Table I We first evaluate the performance of proposed SLAC-ANN on VGG-small model 2 SLAC-ANN achieves better performance on both CIFAR- 10 and CIFAR- 100 compared to other baselines. For example. SLAC-ANN improves vanilla ANN by 0.99% 772.64% > 73. 73.63%] on CIFAR-100. We further compare on the widely used ResNet models 19. For ResNet-20. SLAC-ANN achieves 92. 29% and 68.31% accuracy on CIFAR-10 and CIFAR-100 respectively, which improves vanilla ANN by 0.45% on CIFAR-10 and 0.71% on CIFAR- 100. Although higher efficiency can be achieved, BNN cannot achieve competitive classification

---

Table 1: Classification results on CIFAR-10 and CIFAR-100 datasets
Model
Method
#Mul.
#Add.
#XNOR.
CIFAR-IO
CIFAR-
CNN
0.65G
0.65G
93.80%
72.73%
BNN
0.05G
0.65G
0.60G
89.80%
67.24%
VGG-small
ANN
0.05G
1.25G
93.72%
72.64%
SLAC-ANN
0.05G
1.25G
93.96%
73.63 %
CNN
41.17M
41.17M
92.25%
68.14%
BNN
0.45M
41.17M
40.72M
84.87%
54.14%
ResNet-20
ANN
0.45M
81.89M
91.84%
67.60 67.60%
SLAC-ANN
0.46M
81.90M
0
92.29%
68.31 %
CNN
69.12M
69.12M
O
93.29%
69.74%
BNN
0.45M
69.12M
68.67M
86.74%
56.21
ResNet-32
ANN
0.45M
137.79M
93.01% %
69.02%
SLAC-ANN
0.46M
137.80M
93.24%
69.83%
performance. Through formulating the feature distributions, our proposed algorithm successfully alleviates the unfavorable gap between ANNs and CNNs. Compared CNN, SLAC-ANN achieves competitive results on CIFAR- 100 with only 0.44% accuracy drop. The evaluation on ResNet-32 also demonstrates the superiority of our algorithm. Similarly, SLAC-ANN reduces the existing accuracy gap by 0.23% on CIFAR-10 [0.28% - 0.05%] compared to CNN and improves ANN by 0.81% on CIF,R-100, attribute this superiority to elimination of existing overlapped areas of distribution tails, which boosts the potential classification performance of ANNs.
4.2 Experiments on ImageNet
We further conduct evaluation ImageNet dataset 12] which contains 1.2. training images and 50k testing images with size of 224 x 224 from 1000 categories. The pre-processing and data augmentation follow the same protocols in (9) We make of SGD optimizer with initial learning rate of 0.1, weight decay of x 10 momentum of 0.9 and a cosine learning rate schedule. The entire training takes 300 epochs with batch size of 256. The learning rate of trainable parameter Σ is downscale by 1 x 103, À is set to 0.01 and B to 0.1. The models are trained on 4 NVIDIA Tesla V100 GPUs. Similar CIFAR experiments,
include CNN, ANN and BNN for comparison.
The results are reported in Table 2 We evaluate the performance of SLAC-ANN on ResNet-18 model. XNOR-Net replaces the multiplication by XNOR operations in ResNet trained ImageNet [6]. Although high efficiency can achieved, the performance gap is tremendous compared with CNN, with only 51.2% top-1 accuracy and 73.2% top-5 accuracy. On the contrary, ANNs can achieve competitive results. With proposed algorithm, we further narrow the existing gap between ANN and CNN. Comparing with vanilla ANN,, SLAC-ANN improves the top-1 accuracy by 0.7% [67.0% + 67.7%) and top-5 accuracy by 0.3% (87.6% - 87.9 97..). Similarly, we conduct experiments ResNet-50. Comparing with vanilla ANN, SLAC-ANN improves the top-1 accuracy by 0.4% [74.9% + 75.3%] and top-5 accuracy by 0.9% [91.7% - 92.6%]. which demonstrates that ANN classification performance can be improved through eliminating the overlapping tails.
4.3 Comparison with Loss Variants
To illustrate the necessity of the proposed mixture of multivariate skew Laplace distribution with angle-based constraint, we introduce several loss variant for comparison. We first include L-SoftMax
Table 2: Classification results on ImageNet datasets.
Model
Method
#Mul.
#Add.
#XNOR.
Top-I Acc
Top-5 Aec
CNN
1.8G
1.8G
0
69.8%
89.1%
BNN
0.1G
1.8G
1.7G
51.2%
73.2%
R8
ANN
0.1G
3.5G
0
67.0%
87.6%
SLAC-ANN
0.1G
3.5G
0
67.7 %
87.9%
CNN
3.9G
3.9G
0
76.2%
92.9%
BNN
0.1G
3.9G
3.8G
55.8%
78.4%
ResNet-50
ANN
0.1G
7.6G
0
74.9%
91.7%
SLAC-ANN
0.1G
7.6G
0
75.3%
92.6%

---

(a LGMAANN
(h) Mfitun GMANN
(c) SLANN
SLAC
Figure 3: Visualization of feature distributions of LeNet on MNIST with various ANN variants. (a)
denotes ANN with Large-margin Gaussian Mixture loss. (b) denotes ANN with mixture of Gaussian mixture. (c) denotes ANN with mixture of skew Laplace distributions. (d) denotes ANN with mixture of skew Laplace distributions equipped with angle-based constraint.
Table 3: Comparison of different ANN variants CIFAR-I0 10 and CIFAR- datasets.
Classifier
L-SoftMax
LGM
MoM
SL
SLAC
10
91.83%
92.00%
91.95%
91.99%
92.29%
CIFAR-100
67.53%
67.56%
67.73%
67.51%
68.31 %
14 as the baseline. We further compare SLAC-ANN with different types of distributions, such as the mixture of Gaussian distributions with large-margin (LGM) [22]. To further demonstrate the superiority of SLAC-ANN, we also include a modified version of LGM named a mixture of Gaussian mixture (MoM)
stronger baseline which replaces the Gaussian distribution corresponding each class in 22 with a mixture of two Gaussian distributions since the representation power of Gaussian mixture can be quite powerful explore potential optimal distribution for ANNs. Skew Laplace distribution (SL) without based constraint is also included for comparison.
First, visualize the feature distributions of aforementioned baselines in Figure 3 Consistent with the analysis in Sec 3.c the direction of distribution peavy-tailednees cannot be optimized with only lower-order moments μ and Σ, shown in Figure (a). We further visualize the feature distributions of MoM and SL ANNs, as shown in Figure (b) and (c). MoM, we replace the Gaussian distribution for each class with mixture of Gaussian explore larger space of distributions. The feature distribution tails of both MoM-ANN and SL.-ANN form minor and irregular directions, which demonstrates that pre-defining the feature distributions alone cannot achieve desired properties that the heavy tails of different feature distributions form angle discrepancy. A visualization of proposed SL loss with angle-based constraint is shown in Figure 3 (d) where every class successfully forms cluster with its distribution tail separated in various angle areas.
We further compare the performance of these ANN variants. All the variants trained under the same training setting on CIFAR- 1FEAR-10/100 10/100 with ResNet-20 mentioned in Sec [.1] Note that all the baselines are trained with 800 epochs for a fair comparison. For the hyper- parameters of LGM and MoM, we adopt the same setting in [22] The results are reported Table 3 L-SoftMax can be treated naive approach for addressing the distribution overlapping issue through enlarging classification margin. However, L-SSftMax achieves the similar performance as vanilla ANN, which demonstrates that directly including margin optimization SoftMax loss function could be ineffective in ANNs due to the heavy-tailedness. Comparing different distributions on CIFAR-I0 10 dataset, ANN with LGM, MoM, and SL distribution achieve similar performance while SLAC-ANN achieves the best accuracy with 0.29% improvement compared to LGM. On CIFAR-100, the gaps become more obvious. Both LGM and SL cannot achieve competitive results. For LGM, we attribute this gap the natural limitation that single Gaussian distribution can hardly fit the long-tailed tailed one in ANNs. For SL, although the heavy tails in ANNs can be easily fitted, the overlapped areas of them cannot be handled, which leads to misclassification MoM achieves similar performance compared to vanilla ANN, which indicates the enlarged distribution space is not the major contributor to the improvement. SLAAC achieves the best performance among all the baselines, which demonstrates its effectiveness.
4.4 Ablation Studies
Number of Epochs. As mentioned in Sec. 3.3 we enlarge the number of epochs due to the higher training complexity of SLAC-ANN. To demonstrate the effectiveness of proposed algorithm, we enlarge the training epochs to 800 for vanilla ANN and conduct experiments under different training epochs. We first compare the training and testing curves of SLAC-ANN and vanilla ANN, As

---

6.V0
5.00
Aay
CP10-10
D00
130
64 CIEAR 10 Cuave
03 CIPARLOO Cuav
Trannsiin Epodis
Figure 4: Training and testing accuracy curves with 800 training epochs on CIFAR-10 abd CIFAR- 100 in (a) and (b) respectively. (c) denotes the testing accuracy of vanilla ANN and SLAC-ANN on CIFAR-10 and CIFAR-100 under different training epochs.
Table 4: Ablation Studies SLAC-ANN on CIFAR-ION 100.
Location
Covariance
Skewness
3
Accuracy
0.01
0.1
64.82%
SL Parameters
x
0.01
0.1
67.73%
X
0.01
0.1
62.04%
0.0
0.0
67.39%
Loss Terms
0.01
0.0
67.51%
0.0
0.1
67.73%
SLAC-ANN
0.01
0.1
68.31%
shown in Figure 4 (a), we train both SLAC-ANN and ANN with ResNet-20 on CIFAR-10 where SLAC-ANN consistently has higher training accuracy than vanilla ANN, which enables SLAC-ANN to achieve better classification performance on CIFAR-10. The superiority becomes more obvious on CIFAR-100, as shown in Figure (b). There exist large gaps between SLAC-ANN and vanilla ANN in terms of both training and testing curves, which empirically verifies that SLAC-ANN successfully improves the classification performance besides the enlargement of epochs. To further demonstrate the influence of training epochs, we conduct experiments on CIFAR-10 and CIFAR-100 under different training epochs including 400, 800 and 1200 epochs, as shown in Figure (c). Through enlarging the training epochs CIFAR-IO, both SLAC-ANN and ANN achieve better performance, and SLAC-ANN consistently surpasses vanilla ANN with around 0. 0.1% to 0.2% improvement under different epochs. On CIFAR- 100, although SLAC-ANN has similar performance with ANN with 400 epochs, the superiority becomes more obvious with increasing epochs. We attribute this gap to the natural training complexity of SLAC which introduces clustering algorithm with angle-based based constraint on tails. After enlarge epochs to 800, SLAC-ANN achieves 0.52% accuracy improvements compared with ANN. When the epochs enlarged to 1200, both SLAC-ANN and vanilla ANN reach sub-optimal areas with accuracy reduction. Thus, we set the training epochs to 800.
Effectiveness of Different Components. We conduct ablation studies of the proposed SLAC-ANN to verify the effectiveness of the distribution parameters and loss terms. Since we assume ANN features follow mixture of Multivariate Skew Laplace distributions, all the distribution parameters including location, covariance, and skewness are studied. The influence of different loss terms is included in our studies. We make of ResNet-20 with ANN the baseline model and evaluate all the variants on CIFAR-100. The results are reported in Table For SL parameters, X denotes fixed after initialization while / denotes involved in optimization.
corresponding parameters As shown in rows of SL Parameters, all the distribution parameters contribute to the classification accuracy. Location and skewness play more importance roles since they the key factors in clustering and proposed constraint while covariance models the variance of each feature dimension for different classes. Without the involvement of regularization term angle-based constraint, all the variants have relatively large accuracy drop, as shown in the rows of Loss Terms. Through incorporating the optimization of all distribution parameters and including proposed loss terms, SLAC-ANN achieves the best performance.

---

5 Conclusion
Adder Neural Networks are more efficient for hardware designs and achieve a satisfactory energy reduction, which has wide application potential for computer vision tasks. To alleviate the existing performance gaps between ANNs and CNNs in classification task, we propose to investigate the potential natural weakness due to the replacement of operations. We argue that the heavy-tailed feature distributions in ANNs could lead to worse classification and propose to pre-define ANN features to follow a mixture of Multivariate Skew Laplace distributions, with which the heavy tails in ANNs can be better controlled with high order moment skewness. We introduce an angle-based constraint distribution parameters incorporate high diversity of distribution tails in angle space so that the overlapping can be eliminated. We conduct experiments on various models and datasets where the proposed SLAC-ANN consistently achieves superior performance.
Acknowledgment
The authors would like to thank the Area Chair and the reviewers for their constructive comments. This work was supported in part by the Australian Research Council under Projects DE180101438 and DP210101859.
References
[1] O. Arslan. An alternative multivariate skew laplace distribution: properties and estimation. Statistical Papers, 51(4):865-887, 2010.
2010:1:4: :::0::::p::66--8:: 887
[2] Z. Cai, X. He, J. Sun, and N. Vasconcelos. Deep learning with low precision by half-wave gaussian quantization. CoRR, abs/1702.00953. 2017. /ttp://:://aaxxx org/
[3] H. Chen, Y. Wang. C. Xu, B. Shi, C. Xu, Q. Tian, and C. Xu. AdderNet: Do really need multiplications in deep learning? In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition (CVPR), June 2020.
[4] Courbariaux and Y. Bengio. Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1. CoRR, abs/1602.02830, 2016. RtLL/http///axii orgoorggg/ao /aba/ /60//.60...0 1602.02830 02830
[5] C. Dong, C. C. Loy, He, and X. Tang. Leaming a deep convolutional network for image super-resolution. In Eurupean conference on computer vision, pages 184-199. Springer, 2014.
[6] G. Ghiasi, T.-Y. Lin, and Q. V. Le. Nas-fpn: Learning scalable feature pyramid architecture for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7036-7045 2019.
[7] J. Guo, K. Han, Y. Wang, C. Zhang, Z. Yang, H. Wu, X. Chen, and C. Xu. Hit-detector: Hicrarchical trinity architecture search for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11405-11444 11414, 2020.
[8] S. Han, J. Pool. J. Tran, and W. J. Dally. Learning both weights and connections for efficient neural networks. CoRR, 506./660.. 2015. URL /ttp:://ray arg/abss/106. 026266
[9] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual leamning for image recognition. CoRR, abs/ 151200333 2015. URL URL lttt:://axxxii www./ogg////.eeee.eeoo.o 1et./.11....00 03385
[10] G. Huang, Z. Liu, and K. Q. Weinberger. Densely connected convolutional networks. abs/1608.06993, 2016. URL htl./attp://axxvv.oom www..v..rg/bbbbaab wrwg/ba/////o0o httba///6..00999999
CoRR,
[11] Kim, J. Kwon Lee, and Mu Lee. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2016.
[12] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1 105, 2012.
[13] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg. Ssd: Single shot multibox detector. In European conference computer vision, pages 21-37. Springer, 2016.
10

---

[14] W. Liu, Y Wen, Z. Yu, and M. Yang. Large- softmax loss for convolutional neural networks, 2017.
[15] Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang. Learning efficient convolutional networks through network slimming. CoRR, abs/1708.06519, 2017.
[16] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks, 2016.
[17] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, You only look once: Unified, real-time object detection. In Proceedings of the IEFE conference computer vision and pottern recognition, pages 779-788, 2016.
[18] D. Song, Y. Wang, H. Chen, C. Xu, C. Xu, and D. Tao. Addersr: Towards energy efficient image super-resolution. arXiv preprint arXiv:2009.088)) 2020.
[19] V. Sze, Y.HH. Chen, T.-. Yang, and J. Emer. Efficient processing of deep neural networks: A tutorial and survey. Proceedings of the IEEE, 105(12):229--333) 2329, 2017.
[20] M. Tan and Q. V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. CoRR, abs/1905.1 1946, 2019. URL http:::::/aaiwi.org//aaaaaa htt///aaa/999/ 11946
[21] Y. Tang, Y. Wang, Y. Xu, Y. Deng, C. Xu, D. Tao, and C. Xu. Manifold regularized dynamic network pruning. CoRR, abs/2103.00 10..05661, 2021. https:s:: org/ att/a/bbb///... 05861
[22] W. Wan, Y. Zhong, T. Li, and J. Chen. Rethinking feature distribution for loss functions in image classification. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 9117-9126, 2018.
[23] M. Wang. W. Fu, X. He, S. Hao, and X. Wu. A survey large-scale machine learning. IEEE Transactions Knowledge and Data Engineering, 2020.
[24] Y. Wang. C. Xu, S. You, D. Tao, and C. Xu. Cnnpack: Packing convolutional neu- ral networks in the frequency domain. In D. D. Lee, M. Sugiyama, Guyon, and Garnett, editors, Advances in Neural Information Processing Systems 29, pages 253-261. Curran Associates, Inc.. 2016. URL http::/ppapers cctppprrccpppe .pdf
Luxburg,
6390 6390- ce000onnnacc-ppp
[25] Y. Wang, M. Huang, K. Han, H. Chen, W. Zhang, C. Xu, and D. Tao. Addernet and its minimalist hardware design for energy-efficient artificial intelligence, 2021.
[26] Y. Xu, C. Xu, X. Chen, W. Zhang, C. Xu, and Y. Wang. Kernel hased progressive distillation for adder neural networks. arXiv preprint arXiv:2009. 13044, 2020.
[27] Z. Yang, Y. Wang, X. Chen, B. Shi, C. Xu, C. Xu, Q. Tian, and C. Xu. Cars: Continuous evolution for efficient neural architecture search, 2020.
[28] H. You, X. Chen, Y. Zhang, C. Li, S. Li, Z. Liu, Z. Wang, and Y. Lin. Shiftaddnet: A hardware-inspired deep network. arXiv preprint arXiv:2010. 12785, 2020.
[29] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu. Image super-resolution using very deep residual channel attention networks. CoRR, abs/1807.000 00758, 2018. httpp/////aa org/ httgg/ab/1887... 1807 02758
[30] S. Zhou, Y. Wa, Z. Ni, X. Zhou, H. Wen, and Y. Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv: 2016.


Reviews:
[{"review_id": "rEfwMAPmoP7", "reviewer": "Reviewer_xfwL", "summary": "**Post Rebuttal**:\n\nI have read the author's response and other reviewers' comments. I appreciate the authors' effort in the variance study. However, as reviewer 5BV4 pointed out, there are some issues with this paper. For example, the comparison between BWN and the ANN. I found the authors run an experiment of BWN and reach 64.4% accuracy. However, the result of IR-net [Qing et al., 2020] is 66.5% and it requires much fewer epochs. Based on this, I'll keep my original rating. \n  \n------\n\nThis work presents two complement prior objectives for optimizing the adder neural networks (ANN). Start by analyzing the difference between the convolutional network (CNN) and ANN, the authors propose a new prior distribution for ANN, namely the Skew Laplace Mixture Loss. Then an Angle based constraint is provided. The authors verify the SLAC-ANN in imagenet and cifar classification tasks. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "To be honest, I am not an expert on AdderNet, so maybe I cannot give a precise review. \n\nPros:\n\n+ well-organized structure.\n\n+ clear motivation and technically sound solution.\n\nCons: \n\n- No error bars are reported. Some improvements of results are within the randomness of seed. \n\n- The authors said the kernel-based KD costs more training costs. However, the SLAC-ANN also double the training time. I wonder how to compare the efficiency? \n- Training 800 epochs for CIFAR10 seems too long. Will Binary NN get improved for longer training? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "My concerns are listed above. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "q1vdrDBSqM2", "reviewer": "Reviewer_9FVq", "summary": "This paper identifies a unique long-tailed problem in Adder neural network (ANN) which measures the $l$-1 term between weight and activation and thus incurs Laplacian distributions. Further, this paper solves it by proposing to leverage Multivariate Skew Laplace distribution (SL) to fit feature distributions and to integrate it to the objective loss function for controlling the long-tails. Also, this paper considers adding angle-based constraints to the objective for promoting the disentanglement among features of all classes.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Originality\n\nHigh originality towards more accurate ANNs.\n\n* Quality\n\nThis paper provides decent visualization to demonstrate and motivate readers to understand the long-tail distribution problem in ANNs. The discussion of potential distributions to fit the feature maps is also interesting. Results show the first time that ANNs perform better than CNNs.\n\n* Clarity\n\nOverall clarity is acceptable, but better to specify the new contributions of this paper in the introduction part and explain more clearly how the distribution-related loss is formulated into the objective functions.\n\n* Significance\n\nThis paper is of great significance to an efficient network community. I expect to see it open-source to facilitate the growth of this research direction.\n\n* Weakness\n\n(1) This assumption of Multivariate Skew Laplace distribution (SL) in features needs to be justified. Also, I was confused about the overlapped tails since it is not shown in your real trained networks as shown in Fig. 3 (not like your illustrations in Fig. 1 and 2).\n\n(2) How many epochs do you use for CNN baselines? The training recipes of this paper are different from the original ANN work (e.g., 800 epochs instead of 400 epochs), is the slightly improved accuracy mainly attributed to unfair comparison?\n\n(3) This paper fails to provide any results about the efficiency but highlights that in the introduction. How is the training/inference speed in modern GPUs and customized hardware platforms as compared to baselines?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "* Limitations\n\nAlthough the authors claim they describe the limitations, I only find the limitation of Gaussian distribution to fit the features instead of the proposed techniques.\n\n* Societal impact\n\nThe authors are apparently veterans in ANN designing. However, I do not find any promise of open-source. That may impede the development of ANNs and related research works since others may not be able to reproduce the reported accuracies.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "o12eHnVznIz", "reviewer": "Reviewer_5ZaN", "summary": "This paper proposes trainable distribution parameters for ANNs to tackle the overlapping between classes caused by long feature tails. The authors also add an angle-based constraint to further eliminate overlapping.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors first analyze the limitations of ANNs and focus on the long feature tail problem. To tackle this problem, they introduce Skew Laplace Mixture as the pre-defined prior and embed it into loss function. To consider the dependency among different classes, they also introduce an angle-based constraint. An ablation study is given to clarify the effectiveness of each component.\n\nHowever, the experimental results need crucial analysis to confirm the effectiveness of the proposed approach. For example, the classification results on ImageNet show that SLCA-ANN still underperforms CNN. What's the reason? Is that still caused by the long-tail overlapping issue?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "h7waCgUMl52", "reviewer": "Reviewer_5BV4", "summary": "The paper studies the feature distribution overlapping problem of AdderNets and proposes a Skew Laplace Mixture Loss to alleviate it.\n\nThe overall logic and organization are good. And the technique is somehow novel.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The long-tail problem also exists and has been investigated in ImageNet, and it is not a unique property for AdderNet. What about SLAC loss apply to full-precision CNNs. And the author should investigate the problem on other datasets [5].  \n\nThe analysis is not convincing. Why does Fig1 use the CIFAR-10 dataset, but does Figure 3 use MNIST?\n\nThe training epoch with 300 epochs seems a little strange. The popular training setting is 100 epochs[1] or extends with 120 epochs[2]. The BWN[4] can achieve 60.8% with 90 epochs, and the author should report the result using 90/100 epochs for a fair comparison.\n\nThe addernet is similar to low-precision neural networks. The addernet should report latency or power consumption on GPUs, CPUs or FPGA compared with the Integer-arithmetic quantization method (e.g. 8 bits[1], 4 bits, binary).\n\nThe experiment should include other neural architecture on ImageNets (e.g., MobileNets).\n\n[1]. Deep Residual Learning for Image Recognition\n\n[2]. Bag of Tricks for Image Classification with Convolutional Neural Networks\n\n[3]. Quantization and training of neural networks for efficient integer-arithmetic-only inference\n\n[4]. XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks\n\n[5]. Large-scale long-tailed recognition in an open world", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "about 5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: AdderNets (ANNs) are a non-trivial variant of convolutional networks (CNNs) where each conv layers performs L1 comparisons of the input image/feature patches to its filters instead of multiplying the patches by its filters. These networks use essentially no multiplication operations and are thus potentially more efficient than CNNs. Surprisingly, ANNs perform almost as well as CNNs on CIFAR and Imagenet, and this paper aims at narrowing the remaining performance gap by proposing a new loss function which is tailored to overcome a specific problem they identify in ANNs.

The reviewers thought the contribution of this paper was novel and well explained. However, they found the experimental evidence to be pretty weak, and the long-tail hypothesis wasn't given proper empirical support (beyond the slightly higher test accuracies achieved by their method, which could be attributed to other factors). They recommend that the authors resubmit their manuscript after improving these aspects.
Decision: Accept (Poster)