Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes trainable distribution parameters for ANNs to tackle the overlapping between classes caused by long feature tails. The authors also add an angle-based constraint to further eliminate overlapping.
main_review: The authors first analyze the limitations of ANNs and focus on the long feature tail problem. To tackle this problem, they introduce Skew Laplace Mixture as the pre-defined prior and embed it into loss function. To consider the dependency among different classes, they also introduce an angle-based constraint. An ablation study is given to clarify the effectiveness of each component.

However, the experimental results need crucial analysis to confirm the effectiveness of the proposed approach. For example, the classification results on ImageNet show that SLCA-ANN still underperforms CNN. What's the reason? Is that still caused by the long-tail overlapping issue?
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: **Post Rebuttal**:

I have read the author's response and other reviewers' comments. I appreciate the authors' effort in the variance study. However, as reviewer 5BV4 pointed out, there are some issues with this paper. For example, the comparison between BWN and the ANN. I found the authors run an experiment of BWN and reach 64.4% accuracy. However, the result of IR-net [Qing et al., 2020] is 66.5% and it requires much fewer epochs. Based on this, I'll keep my original rating. 
  
------

This work presents two complement prior objectives for optimizing the adder neural networks (ANN). Start by analyzing the difference between the convolutional network (CNN) and ANN, the authors propose a new prior distribution for ANN, namely the Skew Laplace Mixture Loss. Then an Angle based constraint is provided. The authors verify the SLAC-ANN in imagenet and cifar classification tasks. 
main_review: To be honest, I am not an expert on AdderNet, so maybe I cannot give a precise review. 

Pros:

+ well-organized structure.

+ clear motivation and technically sound solution.

Cons: 

- No error bars are reported. Some improvements of results are within the randomness of seed. 

- The authors said the kernel-based KD costs more training costs. However, the SLAC-ANN also double the training time. I wonder how to compare the efficiency? 
- Training 800 epochs for CIFAR10 seems too long. Will Binary NN get improved for longer training? 
limitations_and_societal_impact: My concerns are listed above.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper identifies a unique long-tailed problem in Adder neural network (ANN) which measures the $l$-1 term between weight and activation and thus incurs Laplacian distributions. Further, this paper solves it by proposing to leverage Multivariate Skew Laplace distribution (SL) to fit feature distributions and to integrate it to the objective loss function for controlling the long-tails. Also, this paper considers adding angle-based constraints to the objective for promoting the disentanglement among features of all classes.
main_review: * Originality

High originality towards more accurate ANNs.

* Quality

This paper provides decent visualization to demonstrate and motivate readers to understand the long-tail distribution problem in ANNs. The discussion of potential distributions to fit the feature maps is also interesting. Results show the first time that ANNs perform better than CNNs.

* Clarity

Overall clarity is acceptable, but better to specify the new contributions of this paper in the introduction part and explain more clearly how the distribution-related loss is formulated into the objective functions.

* Significance

This paper is of great significance to an efficient network community. I expect to see it open-source to facilitate the growth of this research direction.

* Weakness

(1) This assumption of Multivariate Skew Laplace distribution (SL) in features needs to be justified. Also, I was confused about the overlapped tails since it is not shown in your real trained networks as shown in Fig. 3 (not like your illustrations in Fig. 1 and 2).

(2) How many epochs do you use for CNN baselines? The training recipes of this paper are different from the original ANN work (e.g., 800 epochs instead of 400 epochs), is the slightly improved accuracy mainly attributed to unfair comparison?

(3) This paper fails to provide any results about the efficiency but highlights that in the introduction. How is the training/inference speed in modern GPUs and customized hardware platforms as compared to baselines?
limitations_and_societal_impact: * Limitations

Although the authors claim they describe the limitations, I only find the limitation of Gaussian distribution to fit the features instead of the proposed techniques.

* Societal impact

The authors are apparently veterans in ANN designing. However, I do not find any promise of open-source. That may impede the development of ANNs and related research works since others may not be able to reproduce the reported accuracies.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper studies the feature distribution overlapping problem of AdderNets and proposes a Skew Laplace Mixture Loss to alleviate it.

The overall logic and organization are good. And the technique is somehow novel.
main_review: The long-tail problem also exists and has been investigated in ImageNet, and it is not a unique property for AdderNet. What about SLAC loss apply to full-precision CNNs. And the author should investigate the problem on other datasets [5].  

The analysis is not convincing. Why does Fig1 use the CIFAR-10 dataset, but does Figure 3 use MNIST?

The training epoch with 300 epochs seems a little strange. The popular training setting is 100 epochs[1] or extends with 120 epochs[2]. The BWN[4] can achieve 60.8% with 90 epochs, and the author should report the result using 90/100 epochs for a fair comparison.

The addernet is similar to low-precision neural networks. The addernet should report latency or power consumption on GPUs, CPUs or FPGA compared with the Integer-arithmetic quantization method (e.g. 8 bits[1], 4 bits, binary).

The experiment should include other neural architecture on ImageNets (e.g., MobileNets).

[1]. Deep Residual Learning for Image Recognition

[2]. Bag of Tricks for Image Classification with Convolutional Neural Networks

[3]. Quantization and training of neural networks for efficient integer-arithmetic-only inference

[4]. XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks

[5]. Large-scale long-tailed recognition in an open world
limitations_and_societal_impact: Yes
review_title: null

Meta-review: AdderNets (ANNs) are a non-trivial variant of convolutional networks (CNNs) where each conv layers performs L1 comparisons of the input image/feature patches to its filters instead of multiplying the patches by its filters. These networks use essentially no multiplication operations and are thus potentially more efficient than CNNs. Surprisingly, ANNs perform almost as well as CNNs on CIFAR and Imagenet, and this paper aims at narrowing the remaining performance gap by proposing a new loss function which is tailored to overcome a specific problem they identify in ANNs.

The reviewers thought the contribution of this paper was novel and well explained. However, they found the experimental evidence to be pretty weak, and the long-tail hypothesis wasn't given proper empirical support (beyond the slightly higher test accuracies achieved by their method, which could be attributed to other factors). They recommend that the authors resubmit their manuscript after improving these aspects.
Decision: Accept (Poster)