Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary:  This paper designs the novel Adder paradigm for vision transformer. Specifically, the attention module is conducted with additions which saves energy consumption. This paper demonstrate the expressive power of addition computation paradigm both theoretically and experimentally on vision transformers. Experiments results show comparable accuracy on classification tasks on cifar10，cifar100 and ImageNet while saving the energy cost by 2x-3x . 


main_review: This is an interesting paper. The authors develop the visual transformer structures with only addition operations, where the structure and computation paradigm of attention module is carefully redesigned. This proposal can save the energy cost of transformer structures since the addition is computationally cheaper than multiplications according to previous literatures.

The author revisit the main visual transformer modules and adaptively design them in additive way. For attention module, the author discuss the possible way of realizing it with only additions and absolute operation and choose the way which both satisfy the computation constraint and the attention principle following transformers. The authors analyze the adder attention module both theoretically and experimentally. Specifically, the author conduct rank analysis on adder attention module and point out that the low rank phenomenon of the adder attention module could cause the performance decay. The decrease of attention map rank is proved theoretically and observed experimentally. To solve the issue, the author proposed to increase attention map rank by adding an identity mapping. This operation improve the map rank and the final performance.  

With all the above, the ViT with adder attention achieves comparable accuracy on classification tasks on CIFAR10, CIFAR100 and ImageNet while saving 2x-3x energy cost compared with conventional structures of DeiTs. 

limitations_and_societal_impact: 
 Overall I like this paper and think it made good contributions to both ViT and efficient computation fields. Some weaknesses and comments are:

1.Why the identity of the attention map can improve the expressive power of attention module? Is Identity mapping the best choice? More discussion should be conducted. 
2.How about this adder attention applied on natural language processing tasks?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a method that converts multiplication by addition operation during the deployment of vision transformers. 
**the papers main contribution**
1) The authors theoretically analyzed the mechanism of self-attention and the difficulty of applying adder operation into this module. Specifically, the feature diversity i.e., the rank of attention map using only additions cannot be well preserved.
2) Addressing previously raised difficulty, the authors develop an adder attention layer that includes an additional identity mapping, which allows vision transformers to be constructed only by additions operations.
3) The empirical results show that the proposed scheme achieves near baseline performance on several image classification datasets while achieving about 3$\times$ reduction on the total energy consumption 


main_review: This paper is mainly focused on reducing the power consumption of VIT (Vision Transformers). deployment. 
For that proposed the suggest converts multiplication operations by addition operations. \
**Strengths**
The paper is a very well-written and theoretical analysis that raised the difficulties of straightforward usage of addition operations looks convincing. Also, the proposed solution insertion of the extra identity mapping in the adder attention module is well motivated. In the end, the empirical results show that the proposed scheme achieves near baseline performance on several image classification datasets while achieving about 3$\times$ reduction on the total energy consumption and specifically about 2$\times$ reduction for high energy consumption ImageNet classification VIT.

**Weaknesses**
The authors didn't provide any experiments on NLP tasks which is well known as the common usage of transformers models. 

I encourage the authors to address this issue in the rebuttal phase

Post-rebuttal: 

I am satisfied with the provided answers by the authors to the reviewers' concerns and I am convinced that this manuscript presents important insight to the community, hence I vote for his acceptance and keep my pre-rebuttal score.


limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents AdderNet for Transformer. To this end, the feed-forward module and self-attention mechanism are replaced with adder operations (l1 distance). The details of forward and backward computations of AdderNet for Transformer are also provided for computer vision tasks. The proposed network is tested on CIFAR-10, CIFAR-100 and ImageNet. It's shown that the proposed network can successfully perform the image classification tasks with a small accuracy degradation while reducing the energy consumption by a factor of about 3.
main_review: In general, the paper is novel and the idea of replacing multiplications with additions in Transformer is interesting specially in energy-constrained deployment scenarios.The paper is well-written and well-motivated in my opinion. Also, the accuracy results on image classification task are vigorous. My only suggestion is to make a comparison with AdderNet for CNNs in order to show trade-offs or superiority of one over another.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper applies adder neural networks to vision transformers. Specifically, the authors propose adder attention to avoid multiplication operations and offer rank analysis and back-propagation on the adder attention.
main_review: ###  Originality
Although the concept of adder neural netowrk is not new, this paper applies this conect into transformers and propose adder attention to aovid multiplicaiton operations for conducting attention. The idea is okey. The author also offers analysis on adder attention.

### Quality
#### Strengths
+ The paper writes well and the proposed methods are easily to understand.
+ Beyond dot product of attention, this paper proposes new attention method under the concept of adder neural netowrk that can inspires more different methods to conduct attention.

#### Weaknesses
+ My major concern is the performance of adder attention. Taking Table 3 for example, it seems the ader neural network (ANN) is much worse than original multiplicative network (MNN), like for DeiT-S (ANN 10.5B energy 78.3% vs MNN 21.2B 79.8%). How about the performance of ANN and MNN with the same energy? DeiT-S has 384 channel dimension and 12 layers.  Maybe we can build a baseline of DeiT called DeiT-XS with 256 channel dimension and 12 layers so that DeiT-XS (MNN) has half of energy of DeiT-S (MNN), the around same as energy of DeiT-S (ANN). In this way, we can directly compare the performance of DeiT-XS (MNN) and DeiT-S (ANN). 

#### Clarity
The submission is clearly written.

### Significance
Under the concept of adder nerural nentowrk, this paper propose a new attention method adder attention, but the performance of the method is not good.
 
limitations_and_societal_impact: Yes
review_title: null

Meta-review: The reviewers are all in agreement that this paper provides novel and interesting results exploring a theoretical analysis of self-attention, and the introduction of an adder attention layer which helps reduce energy requirements. I would encourage the authors to incorporate some of the additional results on NLP tasks and CNN comparisons into their main text.
Decision: Accept (Poster)