Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This is an interesting paper to discuss the variance in the adder neural network. The authors have an in-depth comparison between ANN and CNN. Also, the adder operation in the neural network can enjoy some nature advantage over CNN in terms of the adversarial robustness. The authors have conducted comprehensive experiments to support these analyses.
main_review: In Line 86, the authors stated that "the adder weights follow Laplace distributions". But I did not get how the authors achieve such a conclusion. Why not the adder weights follow a gaussian distribution? I guess by default, the network is initialized with a gaussian distribution.

What do you mean by "activation xl follows the Rectified Gaussian distribution"?

In Eq. (15), how the authors get the conclusion that the second line has a smaller variance than that of the first line? I could agree that the denominator for ANN is larger than that of CNN, but the numerators are different.

The authors proposed using running mean to improve the adversarial robustness. Will this trick be applied over CNN and what the effect will be? 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper reveals the existing instability of Adder Neural Networks (ANNs) through the variance estimation of output features and proposes adaptive weight normalization to tackle this issue. Additionally, the adversarial robustness of ANNs is highlighted through further analysis of perturbation variance in both ANNs and CNNs, and the paper proposes to automatically eliminate the perturbations based on the statistics in the batch normalization layer. Experiments on several tasks validate the effectiveness of the proposed method in improving the stability and robustness of ANNs.
main_review: Pros:
- The entire paper is easy to follow. The observations in ANNs seem interesting and the motivation is straightforward, which aims at eliminating instability of batch normalization layers according to the lower boundary of feature variance in Eq. 8. The variance study in Section 3 seems well-organized and reasonable. The proposed adaptive weight normalization then comes naturally. Through the discussion of perturbation influence in ANNs compared with CNNs, the paper explains that batch normalization layers can form automatic perturbation eliminations and introduced a simple yet effective inference strategy in Eq. 17.
- Sufficient experimental results are provided to verify the effectiveness of proposed adaptive weight normalization for improving ANN stability on object detection tasks and stability studies in Sections 5.2 and 5.3. Adversarial robustness is also evaluated through white-box attacks in Table 1. All the experiments show obvious superiority in both network stability and robustness over other baselines.
 

Cons:

- My main concern is about the normalization of ANNs. The analysis in Section 3 indicates that the instability mainly comes from the statistics of batch normalization layers in Eq. 9. However, there exist various normalization layers without the statistics involving batch, such as layer normalization. Although the analysis shows that a large variance of feature comes from the weight parameters, I wonder why not directly apply the aforementioned normalization variants to ANNs for improving stability? This discussion or comparison is missing in the paper.

Additionally, some texts in figures are too small to read, such as Figures 1 and 2.

limitations_and_societal_impact: The proposed ANN-R-AWN seems sensitive to the batch size since the inference strategy involves running mean of batch normalization layer, which has been discussed by the authors. 

There is no immediate social impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper aims at enhancing the stability of AdderNets for performance improvement in downstream tasks. For an adder layer, approximations of feature variance and expectation are derived, with which the variance of weight is found to be the major cause of instability. Adaptive weight normalization (AWN) is thus introduced for addressing this problem. Together with the feature variance approximation, the potential adversarial robustness of AdderNets is also analyzed through comparing the perturbation variance in both AdderNets and CNNs. A simple robust inference strategy with BN layer is thus designed for disturbance elimination.
main_review: I think this paper provides a good analysis on the variance and exposes an important problem in AdderNets that the instability could constrain the performance of other downstream tasks, such as object detection which requires pre-trained AdderNets. Proposed AWN normalizes weights and involves affine transformation to explore the trade-offs between representation power and stability, which can be treated as a modified version of weight standardization. Extensive experiments on stability evaluation and object detection task demonstrates the necessity of proposed method. 

The further discussion of adversarial robustness seems interesting. Due to the difference of addition and multiplication, the paper provides a contrastive analysis of disturbance in AdderNets and CNNs to highlight the potential robustness in AdderNets. The BN statistics in Eq. 9 is further extended to the one under attack in Eq. 16, where the disturbance can be automatically eliminated by the running mean in BN layer, which seems efficient and effective.

Overall, the paper is a good contribution to Adder Neural Networks and adversarial robustness. My major concerns come from the comparison with WS. Proposed AWN is apparently a modified version of existing WS, however, the reason why such a modification is necessary is not fully discussed. Based on the analysis in Eq. 8 and Eq. 9, the instability is highly correlated with the variance of weight, which can be solved through WS. Similarly, WS also holds true for the potential adversarial robustness discussed in Eq. 16. Although I notice that all the empirical evaluations demonstrate that AWN performs better than WS in Table 1-3 and Fig 2, the key factor which brings this improvement is not well-explained in the paper.

limitations_and_societal_impact: The limitations of AdderNets have been discussed by the authors in the introduction. For AWN, there exist trade-offs between stability and classification performance, which is discussed in the Sec E of supplementary material.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper investigates the mean and variance of features for AdderNets and reveals that the instability of AdderNets mainly comes from fluctuations of running mean and variance in batch normalization layers. To this end, this paper proposes the adaptive weight normalization (AWN) to optimize adder weight distributions adaptively. Extensive experiments demonstrate that the proposed method improves AdderNets on both stability and robustness.
main_review: Pros:
1.	This paper is technically sound and well written.
2.	The motivation of the proposed method is clear and the proposed idea is quite interesting. The theatrical analysis well justifies the proposed adaptive weight normalization.
3.	The analysis of adversarial robustness for AdderNets makes sense and the proposed method for improving robustness is simple yet effective.
4.	The proposed method shows strong performance for adversarial robustness and presents good stability for downstream tasks like object detection.

Cons:

1.	In Line 193, the learning rate of trainable parameters in AWN is rescaled by a hyper-parameter 1e-5. How is this parameter determined? Are there more ablation studies for the impact of this hyper-parameter?
2.	The concatenation operations in Eq. (10) and Eq. (11) seem a bit weird and I think the concatenation is not necessary. Please carefully check the formulations.
3.	Line 10: “weigh normalization” should be “weight normalization”. Please carefully proofread the manuscript.
limitations_and_societal_impact: Please discuss a bit how AdderNets affect the future design of neural networks and corresponding chips since they are tailored for energy-efficient networks.
review_title: null

Meta-review: This paper focuses on enhancing the stability of AdderNets for performance improvement in downstream tasks. The proposal is the adaptive weight normalization (AWN) to optimize adder weight distributions adaptively. The philosophy behind sounds quite interesting to me, namely, batch normalization layers can form automatic perturbation eliminations. This philosophy leads to a simple yet effective inference algorithm design I have never seen.

The clarity and novelty are clearly above the bar of NeurIPS. While the reviewers had some concerns on the significance, the authors did a particularly good job in their rebuttal. Thus, all of us have agreed to accept this paper for publication! Please include the additional experimental results and merge the reviewers' comments in the next version.
Decision: Accept (Poster)