Title: Handling Long-tailed Feature Distribution in AdderNets
Authors: Minjing Dong
Yunhe Wang
Xinghao Chen
Chang Xu
Venue: NeurIPS 2021
Keywords: AdderNet
Efficient Networks
Multivariate Skew Laplace

Abstract: Adder neural networks (ANNs) are designed for low energy cost which replace expensive multiplications in convolutional neural networks (CNNs) with cheaper additions to yield energy-efficient neural networks and hardware accelerations. Although ANNs achieve satisfactory efficiency, there exist gaps between ANNs and CNNs where the accuracy of ANNs can hardly be compared to CNNs without the assistance of other training tricks, such as knowledge distillation. The inherent discrepancy lies in the similarity measurement between filters and features, however how to alleviate this difference remains unexplored. To locate the potential problem of ANNs, we focus on the property difference due to similarity measurement. We demonstrate that unordered heavy tails in ANNs could be the key component which prevents ANNs from achieving superior classification performance since fatter tails tend to overlap in feature space. Through pre-defining Multivariate Skew Laplace distributions and embedding feature distributions into the loss function, ANN features can be fully controlled and designed for various properties. We further present a novel method for tackling existing heavy tails in ANNs with only a modification of classifier where ANN features are clustered with their tails well-formulated through proposed angle-based constraint on the distribution parameters to encourage high diversity of tails. Experiments conducted on several benchmarks and comparison with other distributions demonstrate the effectiveness of proposed approach for boosting the performance of ANNs.

TL;DR: Alleviate gaps between ANNs and CNNs through formulating feature distribution of ANNs.

Review discussion:
The long-tail problem also exists and has been investigated in ImageNet, and it is not a unique property for AdderNet. What about SLAC loss apply to full-precision CNNs. And the author should investigate the problem on other datasets [5].
The analysis is not convincing. Why does Fig1 use the CIFAR-10 dataset, but does Figure 3 use MNIST?
The training epoch with 300 epochs seems a little strange. The popular training setting is 100 epochs[1] or extends with 120 epochs[2]. The BWN[4] can achieve 60.8% with 90 epochs, and the author should report the result using 90/100 epochs for a fair comparison.
The addernet is similar to low-precision neural networks. The addernet should report latency or power consumption on GPUs, CPUs or FPGA compared with the Integer-arithmetic quantization method (e.g. 8 bits[1], 4 bits, binary).
The experiment should include other neural architecture on ImageNets (e.g., MobileNets).
[1]. Deep Residual Learning for Image Recognition
[2]. Bag of Tricks for Image Classification with Convolutional Neural Networks
[3]. Quantization and training of neural networks for efficient integer-arithmetic-only inference
[4]. XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
[5]. Large-scale long-tailed recognition in an open world
No
Thanks for your constructive comments.
**The long-tail problem also exists and has been investigated in ImageNet, and it is not a unique property for AdderNet. And the author should investigate the problem on other datasets [b].**
We argue that the long-tail problem investigated in [b] is different from the one in our work. The long-tail problem in [b] refers to the long-tailed datasets where the samples sizes of different categories follows a long-tailed distribution, which results in imbalanced classification.
However, the long tail issue discussed in our work refers to the observation that features of ANNs follows long-tailed distributions, which could lead to the overlapping problem of different class features. As shown in Figure 1 (a) and (b), we draw the histograms of weight parameters and features in ANNs and illustrate the overlapping issue with a toy example in Figure 1 (c), which motivates us to formulate the distribution of ANN features.
**What about SLAC loss apply to full-precision CNNs?**
We have included the experiments of applying SLAC to CNN baselines, as shown in Section B and Table 2 of our supplementary materials. SLAC loss slightly improves CNN accuracy with 0.1% to 0.2% while SLAC loss boosts the performance of ANN with 0.7% improvement. We attribute this observation to the natural difference between ANNs and CNNs.
It is true that the features of both ANNs and CNNs have long tails, however, ANN features are more likely to follow long-tailed distributions and these long tails tend to be more critical than those in CNNs. This is because the prior of $\ell_1$ norm is Laplace distribution and ANNs make use of $\ell_1$ norm to measure the similarity of feature and filters.
That is why ANN features tend to follow a long-tailed distribution instead of bell-curved one in CNNs.
Thus, our proposed SLAC loss improves ANNs with a relatively large margin while it improves CNNs slightly.
**The analysis is not convincing. Why does Fig1 use the CIFAR-10 dataset, but does Figure 3 use MNIST?**
Figure 1 draws the histogram of ANN features and weights on CIFAR-10 to reveal the existing long-tails. Through the analysis of long tails, we find it is possible for the features of different classes to be overlapped in ANNs, as illustrated in Figure 2. For a quick evaluation of our proposed algorithm, we conduct some toy experiments with LeNet on MNIST, which follows [a], to verify whether angle-based constraint influences ANN features to form diversity in space space, which is illustrated in Figure 3. Due to the low complexity of MNIST, both SLAC-ANN and vanilla ANN achieves high accuracy where SLAC-ANN achieves 99.32% and ANN achieves 99.21%. The long tails problem is not obvious on MNIST.
However, as shown in Table 1, the gap becomes more obvious on CIFAR-10/100 where SLAC improves ANN by 0.6-0.7% while SLAC improves ANN by 0.1% on MNIST, which verifies our analysis and concerns of long tails visualized in Figure 1 and demonstrates the effectiveness of SLAC-ANN. For a better visualization of the long tail issue, we will visualize LeNet features on CIFAR-10 in the final version.
**The training epoch with 300 epochs seems a little strange.**
We argue that the training of AdderNet naturally needs more epochs. In [a], vanilla ANN approximates the gradient through $\ell_2$-norm, which makes the optimization of ANN difficult to reach local minimum.
Thus, we conduct experiments in Section 4.4 to evaluate the influence of training epochs, which demonstrates the necessity of extending training epochs. As shown in Figure 4(c), vanilla ANN achieves slightly better performance with more training epochs while SLAC-ANN achieves better results with larger margin compared with vanilla ANN. For a fair comparison with BNNs, we also doubles the training epochs for Dorefa-net (BNN) [c] from 400 epochs to 800 epochs for comparison, as shown in the following table:
| Method | DataSet | Top-1 Acc |
| ------ | ------ | ------ |
| BNN | CIFAR10 | 85.13% |
| SLAC-ANN | CIFAR10 | 92.29% |
| BNN | CIFAR100 | 53.94% |
| SLAC-ANN | CIFAR100 | 68.31% |
As suggested, we include BWN as baseline for comparison.
Similarly, we train BWN [d] for 300 epochs on ImageNet. The experiment is still underway and the results will be updated in the rolling discussion.
**The addernet is similar to low-precision neural networks. The addernet should report latency or power consumption on GPUs, CPUs or FPGA compared with the Integer-arithmetic quantization method (e.g. 8 bits[1], 4 bits, binary).**
The analysis of AdderNet latency and power consumption is not the major objective of this work.
However, relevant analysis of AdderNet efficiency has been included in [e], which is evaluated on FPGA platform. Through deploying quantized AdderNet on FPGA, the high efficiency has been verified, such as 16% speed acceleration and 47.85% to 77.9% reduction in power consumption compared to quantized CNNs under the same circuit architecture. The detailed comparison can be found in Figure 5 of [e] where the number of LUT and energy are significantly reduced compared to 8bit and 16bit CNNs.
SLAC-ANN, on the other hand, focuses on alleviating the existing accuracy gap on classification task between ANNs and CNNs without a significant modification of vanilla ANNs. Note that SLAC-ANN only replaces the linear classifier of vanilla ANNs with SL distribution parameters, the influence of FLOPs and energy consumption can be ignored.
Thus, SLAC-ANN achieves better accuracy and maintains high efficiency.
**The experiment should include other neural architecture on ImageNets.**
We simply follow the experiment setting in [a], which evaluates ANN with VGG and ResNet on CIFAR-10/100 and ResNet on ImageNet. Thank you for your suggestion. We will apply SLAC-ANN to other neural architecture for further research.
[a]. AdderNet: Do we really need multiplications in deep learning?
[b]. Large-scale long-tailed recognition in an open world.
[c]. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients.
[d]. XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks.
[e]. Addernet and its minimalist hardware design for energy-efficient artificial intelligence.
**Experiments on BWN**
As mentioned in the previous response, we conduct more experiments on BWN with ResNet-18 as suggested. We train BWN for 200 epochs and 300 epochs on ImageNet. Both of them achieve similar performance. BWN with 200 epochs achieves 64.3% Top-1 Accuracy on ImageNet and BWN with 300 epochs achieves 64.4%.
Although we double or triple the training epochs for BWN, there exist gaps between SLAC-ANN and BWN, with 3.3% accuracy reduction [67.7% $\to$ 64.4%].
Thus, our proposed method outperforms BWN.
* Originality
High originality towards more accurate ANNs.
* Quality
This paper provides decent visualization to demonstrate and motivate readers to understand the long-tail distribution problem in ANNs. The discussion of potential distributions to fit the feature maps is also interesting. Results show the first time that ANNs perform better than CNNs.
* Clarity
Overall clarity is acceptable, but better to specify the new contributions of this paper in the introduction part and explain more clearly how the distribution-related loss is formulated into the objective functions.
* Significance
This paper is of great significance to an efficient network community. I expect to see it open-source to facilitate the growth of this research direction.
* Weakness
(1) This assumption of Multivariate Skew Laplace distribution (SL) in features needs to be justified.
Also, I was confused about the overlapped tails since it is not shown in your real trained networks as shown in Fig. 3 (not like your illustrations in Fig. 1 and 2).
(2) How many epochs do you use for CNN baselines? The training recipes of this paper are different from the original ANN work (e.g., 800 epochs instead of 400 epochs), is the slightly improved accuracy mainly attributed to unfair comparison?
(3) This paper fails to provide any results about the efficiency but highlights that in the introduction. How is the training/inference speed in modern GPUs and customized hardware platforms as compared to baselines?
No
Thanks for your constructive comments.
**Assumption of Multivariate Skew Laplace distribution.**
The assumption of Multivariate Skew Laplace distribution mainly comes from two aspects.
Firstly, the features of ANNs naturally follow long tail distributions instead of bell curves such as Gaussian distributions, as shown in Figure 1. Since the prior of $\ell_1$ -norm is Laplace distribution and ANNs use $\ell_1$-norm for similarity measurement, it is natural to assume that ANN features follow Laplace distributions. Note that we have also included Gaussian distribution as baseline and conducted the comparison in Table 3 where we found Gaussian distributions did not fit ANNs well.
Secondly, the skewness is necessary for controlling the directions of feature tails in order to tackle the potential overlapping issue. Note that SL becomes Laplace distribution when skewness becomes zero, which makes Laplace distribution become a subset of SL.
Thus, we include the skewness parameter in the prior distribution so that the angle-based constraint can be derived.
**Overlapped Tails Visualization in Figure 1, 2 and 3.**
Figure 1 draws the histogram of ANN features and weights on CIFAR-10 to reveal the existing long-tails. Through the analysis of long tails, we find it is possible for the features of different classes to be overlapped in ANNs, as illustrated in Figure 2. For a quick evaluation of our proposed algorithm, we conduct some toy experiments with LeNet on MNIST, which follows [2], to verify whether angle-based constraint influences ANN features to form diversity in space space, which is illustrated in Figure 3. Due to the low complexity of MNIST, both SLAC-ANN and vanilla ANN achieves high accuracy where SLAC-ANN achieves 99.32% and ANN achieves 99.21%. The long tails problem is not obvious on MNIST.
However, as shown in Table 1, the gap becomes more obvious on CIFAR-10/100 where SLAC improves ANN by 0.6-0.7% while SLAC improves ANN by 0.1% on MNIST, which verifies our analysis and concerns of long tails visualized in Figure 1. For a better visualization of the long tail issue, we will visualize LeNet features on CIFAR-10 in the final version.
**Training epochs.**
The CNN baseline results are cited from [2] which takes 400 epochs. Based on our experiments, vanilla ANNs tend to have slightly better performance when the training epochs are extended to 800 epochs (67.70 to 67.79 with ResNet 20 on CIFAR100).
Similarly, all other ANN baselines in Table 3 tend to converge with longer epochs so all of them are set to 800 training epochs for a fair comparison. We attribute this low converge speed to the approximated gradients in ANN optimization, which uses $\ell_2$ norm instead of $\ell_1$ norm for gradient computation as discussed in [2]. In order to form a fair comparison with vanilla ANN and explore the influence of training epochs, we have included the comparison of SLAC-ANN and vanilla ANN under the same training epochs on CIFAR10/100 in Figure 4(c) where 400, 800 and 1200 training epochs are included for comparison. We find both SLAC-ANN and ANN achieves best results with 800 epochs, however, our propose SLAC-ANN consistently outperforms vanilla ANN under different training epochs.
**AdderNet Efficiency**
The high efficiency of AdderNet has been well-studied in [23] which deploys AdderNet on FPGA platform. ANN achieves 16% speed acceleration, 67.6% to 71.4% reduction in logic resource utilization and 47.85% to 77.9% decrease in power consumption compared to CNN baselines. Note that the majority of SLAC-ANN remains the same as vanilla ANN and only difference lies in the classifier which replaces the linear layer with SL distribution parameters.
Thus, the efficiency remains the same as the one analyzed in [23].
To be honest, I am not an expert on AdderNet, so maybe I cannot give a precise review.
Pros:
+ well-organized structure.
+ clear motivation and technically sound solution.
Cons:
- No error bars are reported. Some improvements of results are within the randomness of seed.
- The authors said the kernel-based KD costs more training costs.
However, the SLAC-ANN also double the training time. I wonder how to compare the efficiency?
- Training 800 epochs for CIFAR10 seems too long. Will Binary NN get improved for longer training?
No
Thanks for your constructive comments.
**No error bars are reported.**
We report the error bars with multiple runs. The experiments are conducted on CIFAR-10/100 with ResNet20. We run 4 times for all the methods, and the best results with the standard error are as shown in the following table:
| Method | Dataset | Acc |
| ------ | ------ | ------ |
| Vanilla ANN | CIFAR-10 | 91.84% $\pm$ 0.034 |
| SLAC-ANN | CIFAR-10 | 92.29% $\pm$ 0.037 |
| Vanilla ANN | CIFAR-100 | 67.60% $\pm$ 0.077 |
| SLAC-ANN | CIFAR-100 | 68.31% $\pm$ 0.039|
Vanilla ANN and SLAC-ANN have similar standard error on CIFAR-10. SLAC-ANN has smaller standard error than vanilla ANN on CIFAR-100.
**How to compare the training efficiency?**
Although SLAC-ANN doubles the training time, the training costs of SLAC-ANN are lower than kernel-based KD in terms of GPU memory cost. Note that kernel-based KD relies on progressive KD which requires training both ANN and CNN from scratch.
On the contrary, SLAC-ANN only requires the training of a single ANN, which is nearly half of the entire GPU memory cost of kernel-based KD method.
Furthermore, we set the training epoch to 800 for all the baselines in Table 3 so that each baseline can obtain a relatively stable result for a fair comparison since different baselines have different convergence speeds. Actually, SLAC-ANN requires less epochs to achieve similar performance, as shown in the following Table. With 600 epochs, SLAC-ANN still achieves similar results, which demonstrates the low training costs of proposed method.
| Epoch | Dataset | Acc |
| ------ | ------ | ------ |
| 800 | CIFAR10 | 92.29% |
| 600 | CIFAR10 | 92.16% |
| 800 | CIFAR100| 68.31% |
| 600 | CIFAR100| 68.15% |
**Will Binary NN get improved for longer training?**
Similarly to SLAC_ANN, we extend the training epochs of BNN with ResNet-20 on cifar10/100 to 800 epochs and the results are reported in the following table. BNN achieves slightly better results on CIFAR10 after doubling training epochs, however, slightly worse results on CIFAR100. SLAC-ANN consistently outperforms BNN.
| Method | DataSet | Top-1 Acc |
| ------ | ------ | ------ |
| BNN | CIFAR10 | 85.13% |
| SLAC-ANN | CIFAR10 | 92.29% |
| BNN | CIFAR100 | 53.94% |
| SLAC-ANN | CIFAR100 | 68.31% |
The authors first analyze the limitations of ANNs and focus on the long feature tail problem. To tackle this problem, they introduce Skew Laplace Mixture as the pre-defined prior and embed it into loss function. To consider the dependency among different classes, they also introduce an angle-based constraint. An ablation study is given to clarify the effectiveness of each component.
However, the experimental results need crucial analysis to confirm the effectiveness of the proposed approach.
For example, the classification results on ImageNet show that SLCA-ANN still underperforms CNN. What's the reason? Is that still caused by the long-tail overlapping issue?
No
Thanks for your constructive comments.
**SLCA-ANN still underperforms CNN. What's the reason?**
We agree that there exists a gap between SLAC-ANN and CNN on ImageNet. CNNs can be treated as an upper boundary of ANNs since the number of multiplication operation is massive in CNNs while those expensive operations are replaced by cheap additions in ANNs.
For example, CNN on ResNet-18 needs 1.8G FLOPs of multiplications while ANN only needs 0.1G.
Thus, there exist trade-offs between accuracy and efficiency after applying ANNs. The single comparison of accuracy could be unfair for ANNs.
Our proposed SLAC-ANN aims at alleviating the existing gap between ANNs and CNNs without involving modifications with larger FLOPs to maintain high efficiency. With SLAN-ANN, the Top-1 accuracy gap between CNN and ANN is reduced by 0.7% with ResNet-18 on ImageNet, as shown in Table 2. The remaining gap between CNNs and ANNs is still an open problem for further research.

Meta-review:
role: PC
data: comment: AdderNets (ANNs) are a non-trivial variant of convolutional networks (CNNs) where each conv layers performs L1 comparisons of the input image/feature patches to its filters instead of multiplying the patches by its filters. These networks use essentially no multiplication operations and are thus potentially more efficient than CNNs. Surprisingly, ANNs perform almost as well as CNNs on CIFAR and Imagenet, and this paper aims at narrowing the remaining performance gap by proposing a new loss function which is tailored to overcome a specific problem they identify in ANNs.

The reviewers thought the contribution of this paper was novel and well explained. However, they found the experimental evidence to be pretty weak, and the long-tail hypothesis wasn't given proper empirical support (beyond the slightly higher test accuracies achieved by their method, which could be attributed to other factors). They recommend that the authors resubmit their manuscript after improving these aspects.

Decision: Accept (Poster)