Handling Long-tailed Feature Distribution in AdderNets
Authors: Minjing Dong
Yunhe Wang
Xinghao Chen
Chang Xu
Venue: NeurIPS 2021 Poster (2021)
Keywords: AdderNet
Efficient Networks
Multivariate Skew Laplace

Abstract: Adder neural networks (ANNs) are designed for low energy cost which replace expensive multiplications in convolutional neural networks (CNNs) with cheaper additions to yield energy-efficient neural networks and hardware accelerations. Although ANNs achieve satisfactory efficiency, there exist gaps between ANNs and CNNs where the accuracy of ANNs can hardly be compared to CNNs without the assistance of other training tricks, such as knowledge distillation. The inherent discrepancy lies in the similarity measurement between filters and features, however how to alleviate this difference remains unexplored. To locate the potential problem of ANNs, we focus on the property difference due to similarity measurement. We demonstrate that unordered heavy tails in ANNs could be the key component which prevents ANNs from achieving superior classification performance since fatter tails tend to overlap in feature space. Through pre-defining Multivariate Skew Laplace distributions and embedding feature distributions into the loss function, ANN features can be fully controlled and designed for various properties. We further present a novel method for tackling existing heavy tails in ANNs with only a modification of classifier where ANN features are clustered with their tails well-formulated through proposed angle-based constraint on the distribution parameters to encourage high diversity of tails. Experiments conducted on several benchmarks and comparison with other distributions demonstrate the effectiveness of proposed approach for boosting the performance of ANNs.

Handling Long-tailed Feature Distribution in AdderNets
Minjing Dong' :2, Yunhe Wang"; Xinghao Chen', Chang Xu' Schooo of Computer Science, University of Sydney Huawei Noah's Ark Lab wang@huawei com,
mdon0736@uni. sydney....yyyyy.y... edu. adu,au, yunhe xinghao.. www.nnwwween....cccmmmmmmmm com, x.udsyyneey.cm edu.au au
Abstract
Adder neural networks (ANNs) are designed for low energy cost which replace expensive multiplications in convolutional neural networks (CNNs) with cheaper additions to yield energy-efficient neural networks and hardware accelerations. Although ANNs achieve satisfactory efficiency, there exist gaps between ANNs and CNNs where the accuracy of ANNs can hardly be compared to CNNs without the assistance of other training tricks, such as knowledge distillation. The inherent discrepancy lies in the similarity measurement between filters and features, however how alleviate this difference remains unexplored. To locate the potential problem of ANNs. we focus on the property difference due to similarity measurement. We demonstrate that unordered heavy tails ANNs could the key component which prevents ANNs from achieving superior classification performance since fatter tails tend overlap in feature space. Through pre-defining Multivariate Skew Laplace distributions and embedding feature distributions into the loss function, ANN features can be fully controlled and designed for various properties. We further present novel method for tackling existing heavy tails in ANNs with only a modification of classifier where ANN features are clustered with their tails well- formulated through proposed angle-based constraint the distribution parameters to encourage high diversity of tails. Experiments conducted on several benchmarks and comparison with other distributions demonstrate the effectiveness of proposed approach for boosting the performance of ANNs.
Introduction
Deep Convolutional Neural Networks have been widely adopted in various computer vision tasks due to their satisfactory performance, including image classification [12 10 20). object detection
13 17 6, 7), super-resolution [29 [5] [8), etc.. However, the success of CNNs can hardly be presented for practical usage without further modifications since the majority of computer vision tasks usually deployed on low-power platforms, such as portable and embedded devices, where the computational resources are significantly constrained while prompt inference is required. As result, the acceleration of deep neural networks and reduction of energy cost become urgent requirements and attract massive research efforts in recent years 8
24 15
30 21].
Recently, Chen et al. (3] introduced Adder Neural Network to replace the cross-correlation in CNNs with -orm for the similarity measurement between input features and filters, which eliminates the massive multiplications in CNNs. Through replacing existing multiplications with additions in deep neural networks, ANNs achieve considerable energy reduction since addition is a cheaper operation than multiplications [19 23 Furthermore, ANN can be more friendly for hardware designs of deep
learning acceleration 25 28 Although ANNs achieve comparable results with CNNs, there exist
"Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Owwlagged tall
Diserantienst
(a) ANN weght
(00 ANN fesure
Skewness construnt
(a) and (b) denote histograms of ANN weights and features respectively. (c) denotes a toy example of overlapped heavy tails of feature distributions and a solution through skewness constraint.
Figure 1:
gaps in classification task (e.g. 67.0% top-1 accuracy on ImageNet with ANN-18 while 69.8% with CNN-18). Progressive Kernel Based Knowledge Distillation (PKKD) has been proposed to establish ANN with better performance. However concomitant CNN with the same architecture and parallel optimization of both ANN and CNN are required, which largely raises the cost training [26]. The natural difference between CNNs and ANNs mainly lies in the similarity measurement, which is always ignored and how to relieve the gaps from inherent properties of ANNs remains unexplored. Thus, the difference caused by the replacement of cosine distance with & norm is shed light to explore potentially more powerful classification capability of ANNs.
The discrepancy between ANNs and CNNs due to different similarity measurements lies in various aspects. this work, we mainly focus on the ANN features for classification. A well-studied property of ANNs is that the features corresponding to different classes are clustered towards different centers since & norm is used for similarity measurement [3. Meanwhile, ANN weights tend to follow a Laplace distribution which denotes the prior of &1- Actually the heavy-tailed distributions widely exist not only in ANN weights but also in ANN features, as shown in Figure (a) and (b). Since the features clustered for different classes, the existing fatter tails can be easily overlapped with each other, which could hurt the classification performance.
In this paper, propose to pre-define the feature distributions in order to model the heavy-tailedhee in ANNs. We demonstrate the limitations of Gaussian for ANNs and instead make use of a mixture of Multivariate Skew Laplace Distributions which not only involves mean and variance for optimization but also includes higher-order moment skewness. With skewness, the overlapped areas of heavy tails can be reduced for preventing the entanglement of feature distributions. We propose to embed this mixture of skew Laplace into the loss function through substituting the distribution parameters for the classifier head. A likelihood regularization comes naturally for fitting ANN features pre-defined distributions. With distribution parameters, we introduce a well-formulated angle-based constraint the feature distributions based their locations, covariance and skewness, which drives the distribution tails to different angle regions for disentanglementt Our proposed method improves the classification accuracy by 0.7% both CIFAR-100 with ResNet-20 and ImageNet with ResNet-18 -8 compared to vanilla ANN with only a modification to the ANN classifier head.
2 Preliminaries
Adder Neural Networks. Chen el al. proposed the Adder Neural Network to eliminate the multiplications in traditional convolution networks and replace them with additions significantly reduce computational and energy costs. Consider an intermediate feature map X E R ff "f' XXXWxx, x in deep neural network with weight W x X where H, W denote the height and width of input feature, denotes the kernel size, and Cin, Cout denote the number of input and output channel respectively. The convolution and adder operation are defined as
NO
Y Yoonu (m, n, c)
X(m + i, monij,k j,kk x W(i,i,k,e),
10
w
Cin
Yadder (m, c)
X(m + i, j.k) W(i.j,
Comparing the adder operation with the traditional convolution in Eq. 1 the dot product is replaced by the norm for measuring the similarity between the filter and input feature. Although ANN can achieve similar performance to CNN, there still exist gaps between ANNs and CNNs. For example,

---

ANN has 0.7% accuracy drop with ResNet- 32 on CIFAR- 100 compared to CNN. Since the major difference between ANN and CNN lies in the operation, we mainly focus on addressing the potential adverse properties of this operation substitution for improving ANN performance.
3 Skew Laplace Mixture Loss with Angle-based Constraint
In this section, we analyze potential problems of ANNs and propose a novel framework which replaces classifier head with appropriate trainable distribution parameters. Furthermore, we introduce an angle-based constraint for controlling feature distribution tails to avoid potential entanglement.
3.1 Embedding Skew Laplace Mixture into Loss Function
With 4--ormm as similarity measurement, the weights in ANNS close to Laplace distribution since the prior of fn-nrrm is Laplace distribution, as discussed in 3. Laplace density is expressed by the absolute difference from mean while Gaussian density is expressed by the squared difference, which results in a fatter tails of Laplace distributions compared to Gaussian distributions. We empirically verifies the heavy tails in both ANN weights and features through histograms of sampled layer from pre-trained ANN with ResNet32 on CIFAR-10, as shown in Figure (() important property lies in the features for classification. As discussed and verified in (3), ANN features are clustered and classified by Manhattan distance while CNN features by cosine distance. Combing aforementioned properties, a concern arises that the heavy tails could become potentially troublesome issue for & nn-ro based clustering classification since the feature tails corresponding to different classes are more likely to be overlapped, which significantly reduces the classification margin and constrains the generalization of ANNs, as illustrated in Figure (c). Directly tackling this issue can be rather difficult since the distributions of features are unknown, which motivates us to pre-define an appropriate feature distribution for ANNs to obtain high-level control of heavy tails.
(a) and (b). Another
Thus, make assumption that deep features of neural networks follow learnable distributions, with which the feature distribution can be better formed to achieve expected properties. An naive selection can be Multivariate Gaussian distribution (H,,) 2) where μ denotes the mean and Σ denotes the covariance matrix. However, there exist several concerns in ANNs. For example, the features and weights of ANN always form long-tailed distributions instead of the bell curves with Gaussian distributions, which indicates that pre-defined Gaussian distributions might not fit the real ANN features well. Moreover, Gaussian distribution contains two parameters μl and E for optimization. However, the heavy-tailedness can hardly be controlled by these two lower-order moments, which indicates higher-order moment is required to be involved in pre-defined distribution. Thus, Multivariate Gaussian distribution needs to be replaced by a more appropriate one in ANNs.
In this work, we make use of Multivariate Skew Laplace distribution (SL.) ① to fulfill aforementioned requirements. Considering classification task with K classes, the density of ANN last-layer features _2d with respect its corresponding class k is given by
1=/21/2 1/2
fsz uu, Tx) (d-1)/1)//
(B-ML)"EE (x-pe (E- m EE TH
(2)
where 14, E, y denote the location, covariance and skewness parameters respectively, ag /1+ vf 2 7k and I' denotes the Gamma function. Skew Laplace distribution is selected for several reasons. First, Laplace distribution fits ANN well since ANN filters empirically follow Laplace distributions and the heavy tails of ANN features can be well-approximated by Laplace distributions. Moreover, the skewness parameters in Multivariate Skew Laplace naturally incorporate a high-level control of distribution tails in ANNs. According to the definition of skewness, it indicates the direction and relative magnitude of distribution deviation from its center location. Through optimizing skewness parameters to appropriate values, the overlapped area of heavy tails can be easily eliminated since skewness directly controls the shape of distributions, especially the direction of heavy tails, to achieve disentanglement in angle space, as shown in [
Note that each class k follows independent SC(H,, ΣR, Yk). which makes ANN feature x follows a mixture of Multivariate Skew Laplace distributions. The distribution of x can be computed as
p(x) =
SL pas Ex, (k)(k)
(3)
3

---

Stlan
Consiaant
PyIl B
1A,000
Sclpa S.
Sclpu Ei
SClps Eo
Feeiune
lcodaan
Affinity
Milchrolis disaroo
Input
Festure Exhaction
Clessifier
Figure 2: An illustration of proposed method. ANN feature extraction network remains the same while proposed classifier imposes constraints on features which involves both Mahalanobis and cosine distance to form clustering classification with distribution tails dispersed in different directions.
where pkk denotes the prior probability. Although the real distribution of ANN feature is hard to derive, how well the extracted features fit this mixture of Multivariate Skew Laplace distributions can be measured through sampling subsets from training set and computing the negative log likelihood as
Lill logg C(u,, E, 7/x)] -log[/sI (a) Es. m))p()))
N
= Okk + E. + v (o: px) (ar p4k)
Σ
(4)
==1
(x: un)
> - log[p(k))]
where L denotes the likelihood. Through incorporating the negative log likelihood into the objective loss, ANN features can be well-formulated learned SL(Mk, Σg, Tk). Since ANN features now follow mixture of Multivariate Skew Laplace distributions, the classification loss can be easily derived. With the predefined distribution with its parameters, we can derive the conditional probability of class label y; E [1, K] through Bayes' theorem. Similar to the traditional classification loss which computes the cross-entropy between affinity scores and one-hot encoding of class labels, the classification loss can be computed through treating p(us (24) as the affinity score as
N
log p( p(ailv))((s)) P p(x:)
L.
loglp(u
N
N
1=1
(5)
1
( My: Ee Vos )p(()) Σ S'S (245 Mua, Eg, (p)p(k))
logl
N
Through combing Eq. and 5 we embed the feature distributions into the loss function and make follow a mixture of SL distributions, where the observed heavy tails can be parameterized and direct connection between ANN features and trainable distribution parameters can be established to provide a focus for tackling existing overlapping issue.
3.2
Angle-based Constraint
Note that the prior probability p(k) = K and the distribution parameters of different classes are independent from each other, which indicates there no direct correlation among different feature distributions. If we directly optimize SL(M,. Σx, 7k:) for each class, it could be difficult to reduce the overlapping of distribution tails since each class is formulated independently in Sec without taking feature distributions of other classes into consideration. Thus, we take the feature distributions of all the classes as a whole and propose to explore the constraints among classes which diverge the heavy tails of their feature distributions enlarge classification margin. Since the type and corresponding parameters of feature distributions are available during training phase, constraints can be easily applied the distribution parameters Σ and y to achieve desired performance.

---

We now introduce simple yet effective constraint on the feature distributions. which leads to diversity of distribution tails. We first explore the existing similarity measurements in proposed classifier. The
location 1s and covariance form Mahalanobis distance V(r - -= - (1x)) in Eq. 2 for
measuring the distance from feature x to the center of class k. Similarly, the involvement of skewness
forms inner product (xr - pu,) ) Ex 1 in Eq. 2 for measuring the cosine distance from normalized feature z to the skewness of class k. Now we iocus on the similarity measurement among classes. Given random pair of classes m and n, the relative distance from class m to can be easily derived through replacing feature x with the distribution parameters of other classes in aforementioned distance. Since our objective is incorporating disentanglement of feature distributions, we propose to arrange their heavy tails to different angle regions. In order fulfill this potential, we measure the distance of feature distribution skewness among different classes and impose constraint to encourage
tail divergence in angle space, Given set of skewness parameters "1:K [?, Y2, .... .k]] E RR K xd we compute the skewness distance D, E R through computing the cosine similarity between each class pair. For example, Dy between class m and n is given by
7'm Th2
Dy n) = cos0 (m))
(6)
7l
Although D., can be casily derived, the target D. remains undefined. Note that the skewness itself denotes the direction and relative magnitude of its corresponding distribution heavy-tailednen which controls the shape of distribution and provides an implicit similarity measurement among classes, as illustrated in Figure 2 Meanwhile, the first-order moments j provides a more explicit measurement for describing the distance among classes, which suggests the distances among the centers of different clusters could become an indicator for obtaining target skewness distance among classes. Thus, we propose to make use of location parameters to obtain an adaptive target D.,. For class m, we first take the location parameter fmm as the center point and make use of square of Mahalanobis distance for measuring the location difference between (4u and the feature distributions of other classes as
(7) where SLn, denotes the Multivariate Skew Laplace distribution corresponding to class n and D denotes the Mahalanobis distance. With Eq. 6 and 7 we design the constraint based on two simple principles: (a). For class m, target (,,,,, n) where n E 2..,.... K] can always achieve the maximum and minimum values of cosine distance to fully utilize the angle space; (b). (,(m,,)) n) determine how D,, (m, n) are distributed in range [-1.1] 1] where the tail direction difference D. is inverse proportion to the location difference D,.. Thus, the propose angle-based constraint is given by
(m, n) = D'sc, (Hm) = (Im - =,))' (1m - -Ma),
K
K
Lac
(D, (m,,,) m)) - (m,, n)),
K2
m-
n=1
(8)
where D, n) max (D 1:K)) - -2 22D,,(m,
max(PP, (m,K)
where function max takes the maximum value among all K classes. Eq. 8 computes the mean squared error between current skewness distance and target one, where D.,, purely depends on current location and covariance parameters of feature distributions to form an adaptive target distance. D, is designed by projecting the pre-computed Mahalanobis distance into range [-1, in reverse order. Note that D. (m, m) becomes 1 since (m,, m) 0 and (,,,,, n)) becomes -1 when class n has the maximum Mahalanobis distance (m,,,, n) = max maax(D,ll (m,, 1:K)). An illustration of how proposed constraint influence the pre-defined feature distributions is shown in Figure 2 With the combination of Mahalanobis and cosine distance for angle- based constraint, Skew Laplace distribution introduces a clustering classification without long tails entanglement through controlling the heavy tails. Finally, combing Eq. 4. Eq. and Eq. g the objective loss is defined as
Lakj L. + XLall + 3Lac,
(9)
where \ and 3 are the hyper- papre-parameters for balancing these terms.
3.3 Optimization
Our final objective loss named Skew Laplace Loss with angle-based constraint (SLACC can be simply optimized through stochastic gradient decent (SGD). Similar to vanilla ANN, we make use
5

---

Algorithm Skew Laplace Mixture Loss with Angle-based Constraint for AdderNet
The training set [AK))) y); Batch size Hyperparameter X, 93 Initialize ANN network / with W where W is initialized by uniform distribution; Initialize Multivariate Skew Laplace distributions SL( Σ, where ja is initialized by Xavier initialization, Σ by 1.0 and 7 by 0.0;
Input:
while not converge do Sample a batch of data "K from (X Forward ANN network to get feature -7= (1 W): Forward classifier to get affinity score y 2==n with Eq. 5
Calculate objective loss Loby with Eq.
Update the SL, 7) through SGD and pass gradients to A with Eq. 10:
Update ANN parameters W through full- full-precision gradient with SGD; end while
of full- call-precision gradient to update filters W, which is computed based on norm The major difference lies in the gradients passed by the classifier. In SLAC ANN, the partial derivative of Lobj with respect to last-layer feature is calculated as
aLobj Ox
Myy) + 7o.) -1-110 p(u((() + (a ((x (yy)) ( E (x (0-1)))) (gy))) 5T. (x -1/2
(10)
Σ p(()))(E Th aax(1 DE 2 - paa) - 1/2 pk)).
k
Although proposed constraint Lae is not directly involved in Eq. 10 the gradients passed to ANN layers from classifier are heavily determined by the feature distribution parameters μ4, E and y, which are significantly influenced by Lac. Similar to Eq. T0 the partial derivative of Lotj with respect to
J., Σ and y can be computed through SGD accordingly. The entire algorithm named Skew Laplace Mixture Loss with Angle-based constraint for AdderNet (SLAC-ANN) is shown in Algorithm [ Since our proposed SLAC-ANN involves a constraint on the distribution parameters to encourage tail diversity in angle space which significantly improves the classification margin, the complexity of optimization is relatively enlarged The superiority of our method becomes more obvious when the number of epochs are enlarged, which will be verified in empirical evaluations.
4 Experiments
In this section, conduct empirical evaluation of the proposed SLAC ANN on several image classi- fication benchmarks, including 100,R-10, CIFAR- 100 and ImageNet. ANN variants comparison and ablation studies are provided to demonstrate the effectiveness of the proposed algorithm.
4.1 Experiments on CIFAR
CIFAR-10 10 and CIFAR- dataset contain 50K training images and 10K validation images with size of 32 x 32 from 10 categories. All the training and validation sets are pre-processed according to the same protocol as in [9] We make use of SGD optimizer with an initial learning rate of 0. 0.1, weight decay of 5 x 10-4, momentum of 0.9 and a cosine learning rate schedule. The entire training takes 800 epochs with batch size of 256. The learning rate of trainable parameter Σ is downscale set to 0.01 and 3 to 0.1. For comparison, we include CNN, ANN and binary
by x 10a, neural network (BNN) 300 as baselines. Note that the first and last layers are set to full-precision convolutional layer in vanilla ANN B For a fair comparison, SLAC-ANN adopts a similar setting, which uses convolutional operation for the first layer but replaces the last layer by the proposed SL distributions parameters. All the baseline results are cited from [
The classification results are reported in Table I We first evaluate the performance of proposed SLAC-ANN on VGG-small model 2 SLAC-ANN achieves better performance on both CIFAR- 10 and CIFAR- 100 compared to other baselines. For example. SLAC-ANN improves vanilla ANN by 0.99% 772.64% > 73. 73.63%] on CIFAR-100. We further compare on the widely used ResNet models 19. For ResNet-20. SLAC-ANN achieves 92. 29% and 68.31% accuracy on CIFAR-10 and CIFAR-100 respectively, which improves vanilla ANN by 0.45% on CIFAR-10 and 0.71% on CIFAR- 100. Although higher efficiency can be achieved, BNN cannot achieve competitive classification

---

Table 1: Classification results on CIFAR-10 and CIFAR-100 datasets
Model
Method
#Mul.
#Add.
#XNOR.
CIFAR-IO
CIFAR-
CNN
0.65G
0.65G
93.80%
72.73%
BNN
0.05G
0.65G
0.60G
89.80%
67.24%
VGG-small
ANN
0.05G
1.25G
93.72%
72.64%
SLAC-ANN
0.05G
1.25G
93.96%
73.63 %
CNN
41.17M
41.17M
92.25%
68.14%
BNN
0.45M
41.17M
40.72M
84.87%
54.14%
ResNet-20
ANN
0.45M
81.89M
91.84%
67.60 67.60%
SLAC-ANN
0.46M
81.90M
0
92.29%
68.31 %
CNN
69.12M
69.12M
O
93.29%
69.74%
BNN
0.45M
69.12M
68.67M
86.74%
56.21
ResNet-32
ANN
0.45M
137.79M
93.01% %
69.02%
SLAC-ANN
0.46M
137.80M
93.24%
69.83%
performance. Through formulating the feature distributions, our proposed algorithm successfully alleviates the unfavorable gap between ANNs and CNNs. Compared CNN, SLAC-ANN achieves competitive results on CIFAR- 100 with only 0.44% accuracy drop. The evaluation on ResNet-32 also demonstrates the superiority of our algorithm. Similarly, SLAC-ANN reduces the existing accuracy gap by 0.23% on CIFAR-10 [0.28% - 0.05%] compared to CNN and improves ANN by 0.81% on CIF,R-100, attribute this superiority to elimination of existing overlapped areas of distribution tails, which boosts the potential classification performance of ANNs.
4.2 Experiments on ImageNet
We further conduct evaluation ImageNet dataset 12] which contains 1.2. training images and 50k testing images with size of 224 x 224 from 1000 categories. The pre-processing and data augmentation follow the same protocols in (9) We make of SGD optimizer with initial learning rate of 0.1, weight decay of x 10 momentum of 0.9 and a cosine learning rate schedule. The entire training takes 300 epochs with batch size of 256. The learning rate of trainable parameter Σ is downscale by 1 x 103, À is set to 0.01 and B to 0.1. The models are trained on 4 NVIDIA Tesla V100 GPUs. Similar CIFAR experiments,
include CNN, ANN and BNN for comparison.
The results are reported in Table 2 We evaluate the performance of SLAC-ANN on ResNet-18 model. XNOR-Net replaces the multiplication by XNOR operations in ResNet trained ImageNet [6]. Although high efficiency can achieved, the performance gap is tremendous compared with CNN, with only 51.2% top-1 accuracy and 73.2% top-5 accuracy. On the contrary, ANNs can achieve competitive results. With proposed algorithm, we further narrow the existing gap between ANN and CNN. Comparing with vanilla ANN,, SLAC-ANN improves the top-1 accuracy by 0.7% [67.0% + 67.7%) and top-5 accuracy by 0.3% (87.6% - 87.9 97..). Similarly, we conduct experiments ResNet-50. Comparing with vanilla ANN, SLAC-ANN improves the top-1 accuracy by 0.4% [74.9% + 75.3%] and top-5 accuracy by 0.9% [91.7% - 92.6%]. which demonstrates that ANN classification performance can be improved through eliminating the overlapping tails.
4.3 Comparison with Loss Variants
To illustrate the necessity of the proposed mixture of multivariate skew Laplace distribution with angle-based constraint, we introduce several loss variant for comparison. We first include L-SoftMax
Table 2: Classification results on ImageNet datasets.
Model
Method
#Mul.
#Add.
#XNOR.
Top-I Acc
Top-5 Aec
CNN
1.8G
1.8G
0
69.8%
89.1%
BNN
0.1G
1.8G
1.7G
51.2%
73.2%
R8
ANN
0.1G
3.5G
0
67.0%
87.6%
SLAC-ANN
0.1G
3.5G
0
67.7 %
87.9%
CNN
3.9G
3.9G
0
76.2%
92.9%
BNN
0.1G
3.9G
3.8G
55.8%
78.4%
ResNet-50
ANN
0.1G
7.6G
0
74.9%
91.7%
SLAC-ANN
0.1G
7.6G
0
75.3%
92.6%

---

(a LGMAANN
(h) Mfitun GMANN
(c) SLANN
SLAC
Figure 3: Visualization of feature distributions of LeNet on MNIST with various ANN variants. (a)
denotes ANN with Large-margin Gaussian Mixture loss. (b) denotes ANN with mixture of Gaussian mixture. (c) denotes ANN with mixture of skew Laplace distributions. (d) denotes ANN with mixture of skew Laplace distributions equipped with angle-based constraint.
Table 3: Comparison of different ANN variants CIFAR-I0 10 and CIFAR- datasets.
Classifier
L-SoftMax
LGM
MoM
SL
SLAC
10
91.83%
92.00%
91.95%
91.99%
92.29%
CIFAR-100
67.53%
67.56%
67.73%
67.51%
68.31 %
14 as the baseline. We further compare SLAC-ANN with different types of distributions, such as the mixture of Gaussian distributions with large-margin (LGM) [22]. To further demonstrate the superiority of SLAC-ANN, we also include a modified version of LGM named a mixture of Gaussian mixture (MoM)
stronger baseline which replaces the Gaussian distribution corresponding each class in 22 with a mixture of two Gaussian distributions since the representation power of Gaussian mixture can be quite powerful explore potential optimal distribution for ANNs. Skew Laplace distribution (SL) without based constraint is also included for comparison.
First, visualize the feature distributions of aforementioned baselines in Figure 3 Consistent with the analysis in Sec 3.c the direction of distribution peavy-tailednees cannot be optimized with only lower-order moments μ and Σ, shown in Figure (a). We further visualize the feature distributions of MoM and SL ANNs, as shown in Figure (b) and (c). MoM, we replace the Gaussian distribution for each class with mixture of Gaussian explore larger space of distributions. The feature distribution tails of both MoM-ANN and SL.-ANN form minor and irregular directions, which demonstrates that pre-defining the feature distributions alone cannot achieve desired properties that the heavy tails of different feature distributions form angle discrepancy. A visualization of proposed SL loss with angle-based constraint is shown in Figure 3 (d) where every class successfully forms cluster with its distribution tail separated in various angle areas.
We further compare the performance of these ANN variants. All the variants trained under the same training setting on CIFAR- 1FEAR-10/100 10/100 with ResNet-20 mentioned in Sec [.1] Note that all the baselines are trained with 800 epochs for a fair comparison. For the hyper- parameters of LGM and MoM, we adopt the same setting in [22] The results are reported Table 3 L-SoftMax can be treated naive approach for addressing the distribution overlapping issue through enlarging classification margin. However, L-SSftMax achieves the similar performance as vanilla ANN, which demonstrates that directly including margin optimization SoftMax loss function could be ineffective in ANNs due to the heavy-tailedness. Comparing different distributions on CIFAR-I0 10 dataset, ANN with LGM, MoM, and SL distribution achieve similar performance while SLAC-ANN achieves the best accuracy with 0.29% improvement compared to LGM. On CIFAR-100, the gaps become more obvious. Both LGM and SL cannot achieve competitive results. For LGM, we attribute this gap the natural limitation that single Gaussian distribution can hardly fit the long-tailed tailed one in ANNs. For SL, although the heavy tails in ANNs can be easily fitted, the overlapped areas of them cannot be handled, which leads to misclassification MoM achieves similar performance compared to vanilla ANN, which indicates the enlarged distribution space is not the major contributor to the improvement. SLAAC achieves the best performance among all the baselines, which demonstrates its effectiveness.
4.4 Ablation Studies
Number of Epochs. As mentioned in Sec. 3.3 we enlarge the number of epochs due to the higher training complexity of SLAC-ANN. To demonstrate the effectiveness of proposed algorithm, we enlarge the training epochs to 800 for vanilla ANN and conduct experiments under different training epochs. We first compare the training and testing curves of SLAC-ANN and vanilla ANN, As

---

6.V0
5.00
Aay
CP10-10
D00
130
64 CIEAR 10 Cuave
03 CIPARLOO Cuav
Trannsiin Epodis
Figure 4: Training and testing accuracy curves with 800 training epochs on CIFAR-10 abd CIFAR- 100 in (a) and (b) respectively. (c) denotes the testing accuracy of vanilla ANN and SLAC-ANN on CIFAR-10 and CIFAR-100 under different training epochs.
Table 4: Ablation Studies SLAC-ANN on CIFAR-ION 100.
Location
Covariance
Skewness
3
Accuracy
0.01
0.1
64.82%
SL Parameters
x
0.01
0.1
67.73%
X
0.01
0.1
62.04%
0.0
0.0
67.39%
Loss Terms
0.01
0.0
67.51%
0.0
0.1
67.73%
SLAC-ANN
0.01
0.1
68.31%
shown in Figure 4 (a), we train both SLAC-ANN and ANN with ResNet-20 on CIFAR-10 where SLAC-ANN consistently has higher training accuracy than vanilla ANN, which enables SLAC-ANN to achieve better classification performance on CIFAR-10. The superiority becomes more obvious on 100, as shown in Figure (b). There exist large gaps between SLAC-ANN and vanilla ANN in terms of both training and testing curves, which empirically verifies that SLAC-ANN successfully improves the classification performance besides the enlargement of epochs. To further demonstrate the influence of training epochs, we conduct experiments on CIFAR-10 and CIFAR-100 under different training epochs including 400, 800 and 1200 epochs, as shown in Figure (c). Through enlarging the training epochs CIFAR-IO, both SLAC-ANN and ANN achieve better performance, and SLAC-ANN consistently surpasses vanilla ANN with around 0. 0.1% to 0.2% improvement under different epochs. On CIFAR- 100, although SLAC-ANN has similar performance with ANN with 400 epochs, the superiority becomes more obvious with increasing epochs. We attribute this gap to the natural training complexity of SLAC which introduces clustering algorithm with angle-based based constraint on tails. After enlarge epochs to 800, SLAC-ANN achieves 0.52% accuracy improvements compared with ANN. When the epochs enlarged to 1200, both SLAC-ANN and vanilla ANN reach sub-optimal areas with accuracy reduction. Thus, we set the training epochs to 800.
Effectiveness of Different Components. We conduct ablation studies of the proposed SLAC-ANN to verify the effectiveness of the distribution parameters and loss terms. Since we assume ANN features follow mixture of Multivariate Skew Laplace distributions, all the distribution parameters including location, covariance, and skewness are studied. The influence of different loss terms is included in our studies. We make of ResNet-20 with ANN the baseline model and evaluate all the variants on CIFAR-100. The results are reported in Table For SL parameters, X denotes fixed after initialization while / denotes involved in optimization.
corresponding parameters As shown in rows of SL Parameters, all the distribution parameters contribute to the classification accuracy. Location and skewness play more importance roles since they the key factors in clustering and proposed constraint while covariance models the variance of each feature dimension for different classes. Without the involvement of regularization term angle-based constraint, all the variants have relatively large accuracy drop, as shown in the rows of Loss Terms. Through incorporating the optimization of all distribution parameters and including proposed loss terms, SLAC-ANN achieves the best performance.

---

5 Conclusion
Adder Neural Networks are more efficient for hardware designs and achieve a satisfactory energy reduction, which has wide application potential for computer vision tasks. To alleviate the existing performance gaps between ANNs and CNNs in classification task, we propose to investigate the potential natural weakness due to the replacement of operations. We argue that the heavy-tailed feature distributions in ANNs could lead to worse classification and propose to pre-define ANN features to follow a mixture of Multivariate Skew Laplace distributions, with which the heavy tails in ANNs can be better controlled with high order moment skewness. We introduce an angle-based constraint distribution parameters incorporate high diversity of distribution tails in angle space so that the overlapping can be eliminated. We conduct experiments on various models and datasets where the proposed SLAC-ANN consistently achieves superior performance.
Acknowledgment
The authors would like to thank the Area Chair and the reviewers for their constructive comments. This work was supported in part by the Australian Research Council under Projects DE180101438 and DP210101859.
References
[1] O. Arslan. An alternative multivariate skew laplace distribution: properties and estimation. Statistical Papers, 51(4):865-887, 2010.
2010:1:4: :::0::::p::66--8:: 887
[2] Z. Cai, X. He, J. Sun, and N. Vasconcelos. Deep learning with low precision by half-wave gaussian quantization. CoRR, abs/1702.00953. 2017. /ttp://:://aaxxx org/
[3] H. Chen, Y. Wang. C. Xu, B. Shi, C. Xu, Q. Tian, and C. Xu. AdderNet: Do really need multiplications in deep learning? In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition (CVPR), June 2020.
[4] Courbariaux and Y. Bengio. Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1. CoRR, abs/1602.02830, 2016. RtLL/http///axii orgo /aba/ /60//.60...0 1602.02830 02830
[5] C. Dong, C. C. Loy, He, and X. Tang. Leaming a deep convolutional network for image super-resolution. In Eurupean conference on computer vision, pages 184-199. Springer, 2014.
[6] G. Ghiasi, T.-Y. Lin, and Q. V. Le. Nas-fpn: Learning scalable feature pyramid architecture for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7036-7045 2019.
[7] J. Guo, K. Han, Y. Wang, C. Zhang, Z. Yang, H. Wu, X. Chen, and C. Xu. Hit-detector: Hicrarchical trinity architecture search for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11405-11444 11414, 2020.
[8] S. Han, J. Pool. J. Tran, and W. J. Dally. Learning both weights and connections for efficient neural networks. CoRR, 506./660.. 2015. URL /ttp:://ray arg/abss/106. 026266
[9] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual leamning for image recognition. CoRR, abs/ 151200333 2015. URL URL lttt:://axxxii www./ogg////.eeee.eeoo.o 1et./.11....00 03385
[10] G. Huang, Z. Liu, and K. Q. Weinberger. Densely connected convolutional networks. abs/1608.06993, 2016. URL htll/attp://axxvv.oom www..v..rg/bbbbaab wrwg/ba/////o0o httba///6..00999999
CoRR,
[11] Kim, J. Kwon Lee, and Mu Lee. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2016.
[12] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1 105, 2012.
[13] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg. Ssd: Single shot multibox detector. In European conference computer vision, pages 21-37. Springer, 2016.
10

---

[14] W. Liu, Y Wen, Z. Yu, and M. Yang. Large- softmax loss for convolutional neural networks, 2017.
[15] Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang. Learning efficient convolutional networks through network slimming. CoRR, abs/1708.06519, 2017.
[16] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks, 2016.
[17] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, You only look once: Unified, real-time object detection. In Proceedings of the IEFE conference computer vision and pottern recognition, pages 779-788, 2016.
[18] D. Song, Y. Wang, H. Chen, C. Xu, C. Xu, and D. Tao. Addersr: Towards energy efficient image super-resolution. arXiv preprint arXiv:2009.0889) 2020.
[19] V. Sze, Y.HH. Chen, T.-. Yang, and J. Emer. Efficient processing of deep neural networks: A tutorial and survey. Proceedings of the IEEE, 105(12):229--333) 2329, 2017.
[20] M. Tan and Q. V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. CoRR, abs/1905.1 1946, 2019. URL http:::::/aaiwi.org//aaaaaa htt///aaa/999/ 11946
[21] Y. Tang, Y. Wang, Y. Xu, Y. Deng, C. Xu, D. Tao, and C. Xu. Manifold regularized dynamic network pruning. CoRR, abs/2103.00 10..05561, 2021. https:s:: org/ att/a/bbb///... 05861
[22] W. Wan, Y. Zhong, T. Li, and J. Chen. Rethinking feature distribution for loss functions in image classification. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 9117-9126, 2018.
[23] M. Wang. W. Fu, X. He, S. Hao, and X. Wu. A survey large-scale machine learning. IEEE Transactions Knowledge and Data Engineering, 2020.
[24] Y. Wang. C. Xu, S. You, D. Tao, and C. Xu. Cnnpack: Packing convolutional neu- ral networks in the frequency domain. In D. D. Lee, M. Sugiyama, Guyon, and Garnett, editors, Advances in Neural Information Processing Systems 29, pages 253-261. Curran Associates, Inc.. 2016. URL http::/ppapers cctppprrccpppe .pdf
Luxburg,
6390 6390- ce000onnnacc-ppp
[25] Y. Wang, M. Huang, K. Han, H. Chen, W. Zhang, C. Xu, and D. Tao. Addernet and its minimalist hardware design for energy-efficient artificial intelligence, 2021.
[26] Y. Xu, C. Xu, X. Chen, W. Zhang, C. Xu, and Y. Wang. Kernel hased progressive distillation for adder neural networks. arXiv preprint arXiv:2009. 13044, 2020.
[27] Z. Yang, Y. Wang, X. Chen, B. Shi, C. Xu, C. Xu, Q. Tian, and C. Xu. Cars: Continuous evolution for efficient neural architecture search, 2020.
[28] H. You, X. Chen, Y. Zhang, C. Li, S. Li, Z. Liu, Z. Wang, and Y. Lin. Shiftaddnet: A hardware-inspired deep network. arXiv preprint arXiv:2010. 12785, 2020.
[29] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu. Image super-resolution using very deep residual channel attention networks. CoRR, abs/1807.000 00758, 2018. httpp/////aa org/ httgg/ab/1887... 1807 02758
[30] S. Zhou, Y. Wa, Z. Ni, X. Zhou, H. Wen, and Y. Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv: 2016.


Reviews:
[{"review_id": "SFhQ62jGiUj", "reviewer": "Reviewer_AHNP", "summary": "The paper studies the potential influence due to different operations in similarity measurement, which could result in the worse classification performance of AdderNets compared with CNNs. Through empirical observations and toy examples, the paper focuses on the long-tailed feature distributions. The authors propose to formulate features to follow distributions with trainable parameters including skewness. A constraint is introduced to utilize sknewness to encourage tail divergence in angle space. Through extensive experiments, proposed SLAC-ANN outperforms all other baselines and achieves competitive results compared with CNNs.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is organized with a clear motivation which prevents the existing long tails to be overlapped through pre-defining features to follow trainable skew Laplace mixture with skewness parameters for high-level control of feature tails. The proposed angle-based constraint seems reasonable, which establishes connection between cosine similarity and Mahalanobis distance. The authors verify the effectiveness of SLAC-ANN through sufficient evaluations under various scenarios. The improvement seems obvious.\n\nOverall, the contribution is solid. My main concern lies in other simple baselines which can eliminate the long-tails feature distributions. Since the major risk comes from overlapped tails, why not apply some constraint to directly remove the long-tails, such as center loss? If these simple baselines can also solve this problem, what are the advantages of SLAC-ANN compared with them?\n\nMinor Comments:\na fatter tails => fatter tails\nan naive => a naive\nbe easily applied on => be easily applied to\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The long-tailed feature distribution could be only one of the factors which constrains the performance of ANNs. Other potential factors based on empirical observations or theoretical analysis remain unexplored and deserve more discussion. \n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "Bl_CEYu18UV", "reviewer": "Reviewer_aWB2", "summary": "This paper investigates the performance gap between ANNs and CNNs in classification task and focuses on the heavy-tailed feature distributions in ANNs. To alleviate this problem, this paper proposes to pre-define ANN features to follow a mixture of Multivariate Skew Laplace distributions, and introduces an angle-based constraint on distribution parameters to incorporate high diversity of distribution tails in angle space. Extensive experiments and ablation studies are conducted on CIFAR and ImageNet, which properly demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well written and easy to understand. The motivation of dealing with long-tailed feature distribution of Adder neural network makes sense and is quite interesting. Based on the observations and analysis, this paper proposes to replace classifier head with appropriate trainable distribution parameters. Furthermore, an angle-based constraint is introduced for controlling feature distribution tails to avoid potential entanglement. The proposed method consistently improves the performance of AdderNets on CIFAR and ImageNet datasets. Extensive ablation studies are provided to justify the proposed method.\n\nI have several concerns:\n1. In Figure 1, could the feature distribution of CNN also be visualized to better demonstrate the long tails phenomenon of AdderNet. I wonder whether the long-tailed distribution also exists for CNN.\n2. The proposed method shows good performance for ResNet-18 on ImageNet. I would like to see more experiments on ResNet-50, which will better demonstrate the effectiveness of the proposed method.\n\nMinor comments:\n1. L324, a angle-based -> an angle-based\n\n---------------------------post rebuttal------------------------------\nMy concerns are addressed in the authors' rebuttal. I keep my score of 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of adder neural networks are discussed. This paper improves the performance of adder neural networks, which are beneficial for energy-efficient neural networks.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "3z6xe_ovku9", "reviewer": "Reviewer_rrae", "summary": "This paper presents a potential problem in AdderNets where the long-tailed features could be difficult for classification tasks owing to the entanglement of feature tails. To achieve disentanglement, the authors assume features of each class follow Skew Laplace (SL) distribution to obtain better control of feature tails. With the skewness parameters in SL distribution, the authors propose to eliminate overlapped areas through imposing constraints on the directions of feature tails. The entire algorithm replaces the classifier with distribution parameters and can be updated by SGD. Experiments are conducted on several datasets and models to demonstrate the advantage of the proposed algorithm.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The presentation of this paper is clear. The authors demonstrate a novel approach for addressing long-tailed feature distributions in AdderNets. The assumption seems reasonable and the reasons why SL distributions are needed in AdderNets are well-justified. The experimental results seem promising and sufficient enough. The comparison on various datasets shows strong evidence that the proposed algorithm can address the existing problem and improve the performance of AdderNets. The ablation studies show the influence of each loss term and distribution parameters.\n \nMy major concern is the selection of distance in Eq. 7, which uses Mahalanobis distance for measuring the location difference among class centers. However, there is no explanation for this choice. And there is no other simple baseline comparison in ablation studies, such as Euclidean distance.\n\nSome minor points:\n1. AdderNet and ANN are mixed.\n2. Multiple citations for one paper [11, 12], [29,30].\n3. Miss relevant citations of Skew Laplace distributions.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Well addressed in this paper and I have no particular concern", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concern", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "1egVWqXgKaa", "reviewer": "Reviewer_P2Ke", "summary": "The paper proposes to alleviate the gap between CNNs and ANNs through addressing the heavy-tailed features in ANNs. The authors argue that the observed heavy tails in feature distributions can be easily overlapped, which limits the classification performance of ANNs. To remove this risk, the skew Laplace distribution with higher-order moment is introduced to be embedded into loss function through involving likelihood regularization and conditional probability computation. With introduced distributions, the angle-based constraint is proposed for arranging for long tails of different classes to lie in their corresponding angle regions so that the overlapping of tails can be avoided.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ The paper is well-organized and easy to follow, with motivation well-explained.\n+ The proposed algorithm is interesting and novel. The involvement of higher-order moment seems natural for directly constraining the heavy tails. The introduced constraint is also explicit, which imposes high diversity of tails in angle space.\n+ Extensive empirical evaluations on different architectures and datasets. Superior performance compared to other baselines. The feature visualization demonstrates that proposed algorithm can successfully impose angle-based constraint. Ablation studies verifies the effectiveness of each components.\n\nThere exist several concerns:\n- The classifier in ANN is replaced by the distribution parameters, including location, covariance and skewness. The classifier forwarding is replaced by the distribution density function in Eq. 2. However, any increment of the number of parameters or FLOPs is not discussed in this paper.\n- In Eq. 8, the authors introduce a target distance \\hat{D_{gamma}} which makes use of the current Mahalanobis distance between center locations among classes to derive an adaptive target. However, the reason of it is not well-explained. I wonder how the angle-based constraint performs when the target distance is replaced by a fixed one with equal intervals in range [-1, 1]? It seems that a fixed target can also meet the requirements that avoids overlapped tails.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Proposed algorithm is designed for ANNs and might lack generalization to other networks, such as CNNs. The authors discuss this limitations in supplementary material.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "Three hours."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript has been reviewed by four experienced reviewers, all of whom acknowledged the contributions of the submission and recommend acceptance. Specifically, the reviewers agree that the manuscript is easy to follow and the proposed method is novel.

Since there is no attempt to reject the submission, there is no basis to overturn the consensus. The AC thus recommends an acceptance.
Decision: Accept (Poster)