Adder Attention for Vision Transformer
Authors: Han Shu
Jiahao Wang
Hanting Chen
Lin Li
Yujiu Yang
Yunhe Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Transformer
Addernet
Energy consumption

Abstract: Transformer is a new kind of calculation paradigm for deep learning which has shown strong performance on a large variety of computer vision tasks. However, compared with conventional deep models (e.g., convolutional neural networks), vision transformers require more computational resources which cannot be easily deployed on mobile devices. To this end, we present to reduce the energy consumptions using adder neural network (AdderNet). We first theoretically analyze the mechanism of self-attention and the difficulty for applying adder operation into this module. Specifically, the feature diversity, i.e., the rank of attention map using only additions cannot be well preserved. Thus, we develop an adder attention layer that includes an additional identity mapping. With the new operation, vision transformers constructed using additions can also provide powerful feature representations. Experimental results on several benchmarks demonstrate that the proposed approach can achieve highly competitive performance to that of the baselines while achieving an about 2~3× reduction on the energy consumption. 

Adder Attention for Vision Transformer
Han
Wang? HoinggHHatting Chen 1.3 Lin Li Yujiu Yang² Yunhe Wang"1
HHaawei Noah's Ark Lab TTsinghua Shenzhen International Graduate School Peking University "Huawei Technologies
(han.shu, yunhe. wang, 1ilin29) @huawei com, wang jh199hails......m.mmmm wtt.emmmll..ssnhuuuuee tsinghua. edu. cn htchen@pku. edu. en, yang. yugg.yyujuss... twi.uusss.tsinnaaa.... edu.cn
Abstract
Transformer is new kind of calculation paradigm for deep learning which has shown strong performance on large variety of computer vision tasks. However, compared with conventional deep models (e.g., convolutional neural networks), vision transformers require more computational resources which cannot be easily deployed on mobile devices. To this end, we present reduce the energy con- sumptions using adder neural network (AdderNet). We first theoretically analyze the mechanism self-attention and the difficulty for applying adder operation into this module. Specifically, the feature diversity, i.e., the rank of attention map using only additions cannot well preserved. Thus, we develop an adder atten- tion layer that includes an additional identity mapping. With the new operation, vision transformers constructed using additions can also provide powerful feature representations. Experimental results several benchmarks demonstrate that the proposed approach can achieve highly competitive performance to that of the baselines while achieving about 2" 3x reduction the energy consumption.
1 Introduction
Transformers 27 have prevailed in natural language processing (NLP) due to its superior capability in capturing long-distance dependencies based on self-attention mechanism. Some homologous large-scale models, e.g., BERT [10] and GPT-3 [2) provide a significant performance improvement on learning powerful language representations from unlabeled text. The great breakthrough of transformers in NL.P has sparked particular interest from the vision community in applying transformer computer vision (CV) tasks such as image recognition
11 35 25 26
object detection
38 33
and image generation 4 12 17 15)
The high-power consumption of transformer-base models has blocked them from being deployed on mobile devices, e,g., smart phone, camera, and micro-robots. Therefore,
necessary to study efficient transformers which can be embedded mobile devices with affordable computation resources. Wu et. (31] present an efficient Long-Shon Range Attention with two groups heads, respectively for local context and long- distacce relationship modeling. Wang et.al. (28) decompose the original dot not-product attention into smaller attentions through linear projections. Unfortunately, previous arts mainly focus compressing and accelerating transformers for language processing tasks, which motivates us for building efficient transformers for computer vision applications.
Various attempts have been made compressing CNN models, including quantization [13] [6]. pruning [9 30 23 and distillation 25). Beyond these methods, Chen et.al. [6] propose Adder Neural Network (AdderNet) for building efficient deep learning models, which avoids massive computational cost by replacing the convolutional operation by using fiddistance between input
"Equal contribution. 'Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(a) vanilla AdderNet
(b) vanilla Transformer
(c) Adder Transformer
Figure I: Visualization of features in different neural networks on MNIST dataset. From left to right
are vanilla AdderNet, vanilla Transformer and Adder Transformer, respectively.
signals and weights instead of correlation. Xu et.al. 32] further utilize knowledge distillation
technique make AdderNet achieve better performance than CNN. Song et.al. [24 successfully apply AdderNet on single image super-resolution (SISR) task by learning identity mapping with self-shortcuth and implementing tigh-pass filter with learnable power activation. Wang et.al. [29] conduct practical implementations on hardware platform (FPGA), and verify its superior performance in suppressing chip area and power consumption. Thus, are motivated to investigate the feasibility of replacing multiplications by additions transformer architectures.
However, it is very hard to directly transport the cxisting success of AdderNet CNNs to trans- formers, since their basic calculation paradigm is completely different. In particular, the existing works mainly focus the additiveization of convolutional kernel filters which do not involve the unique linear transformations and self-attention mechanisms transformers. To this end, we propose general adder linear transformation operation and adder self-attention mechanism to replace the tra- ditional multiplication version in both -eewwfd modules and self-attention layers transformer model. Specifically,
utilize ( diistance between query and key instead of scaled dot- dot-product to measure the distance between them. For the attention module, we find that the directly using additions could cause more information concentrating the attention map rank, i.e., the functionality of the attention mechanism cannot be well preserved. Thus, we propose insert an decay. This operation can also homogenize the information distribution of the adder attention map. Meanwhile, we provide the detailed feed-forward and pack-propagation for optimizing the adder vision transformer with stable training process. As result, we can obtain comparable performance using the proposed adder transformer models that of original baselines while reducing about 3 of the overall energy consumption on various benchmark models and datasets.
few largest singular values, leading to decay of
identity mapping the adder attention module to solve the
2 Preliminaries and Motivation
In this section, we briefly revisit the basic related components, including AdderNet and transformer.
Adder Neural Networks (AdderNet), Denote filters convolutional layer of neural networks W E www.kxxxcs XCout the input feature map X H Xcin, and the output feature map as E " where the kernel size, Cin and Cout are the number of input channels and output channels, respectively. The traditional convolutional operation is defined
Ym... A x ΣD (X (Xmtin+jjkr Fiskaa)
(1)
where -(.,)) pre-defined similarity measure (S(x, y) = x x y for the convolution multiplications operation). AdderNet [6] maximizes the use of additions by treating the e -iittance between the input feature map and filter as the similarity measure (i.e., S(x, v) = yl in Eq.
D Y Ym... 2 x
[Xm+4.44kkk F.g.k.gl-
However the pioneering work only focused additiveization of CNNs, while ignoring transformers. Vision Transformers. Vision transformer [[]] mainly consists of MHSA (Multi-head Self Attentinn)),, FFN (Feed-Forwart Network) and LN (Layer Normalization).

---

For multi-head head self attentttnnion, the inputs X E RND formations and output queries Q Xxd,, keys K E RMMNXxx xd, and values V R M xd, where M is total number of heads, dg. dy. dp is the dimension of queries, keys and values for each head, respectively. Generally, is performed as follows:
are applied with three different linear trans-
Mdg Mds- = Md, = Md,, and the multi-head self-attention
Qu. K mitil
exp
Vdle
Nuv
Qm.sk
Am.g.
where Z3 = Σ exp
Om.i
Am.ii Vinger
Zy
dde
Of Concat (O1, Oz
OMMR)) Wo.
(3)
where d, - dr. Wo E R denotes the weight matrix of a linear projection layer used to produce the output values O' E RNND
From the above statements, transformer has a different structure and computation paradigm compared to CNNs. Thus, necessary to study how to use additions to fulfill the modules.
3 The Proposed Model: Adder Transformer
In this section, we present the Adder Transformer- which implement the multi-heat attention module and FFN module using adder operations. We will present the details in the following subsections and an illustration of the proposed method can be found in Section A. of the supplementary material.
3.1 Adder Linear Transformation
We first consider the linear transformation and FFN layers in transformers. The calculation paradigm of these two modules based on fully connected layer, which is equivalent a special 1x1 convolu- tion. Therefore, we can use the original computation mechanism of adder convolutional kernel in this module. Considering weight matrices Wq xd, Wk MM-xxxxD Wv xxxxx in projection layer of the multi-head self-attention, we maximize the use of additions by taking L distance to measure the distance between the weight matrices and input embedding as:
D
D Σ
D
Qmija Σ d=1
1Xj.a Wam.gall Kmiik
|xa. Wkm.k.al Vinia
Σ d=1
Xu,. Wvm.eb
(4)
wherein, m E [1. ML E [1, [1, E dyl, 1 E de) v [1, d,) and generally set N N₂ Ne the resulting number of patches and D Md,, Md, Md, Md as the model embedding dimension. The feed-forward network can also be modified directly:
FFN. W2 + (W1 @ X +b) b2.
(5)
As mentioned in AdderNet (6), the adder layers output values with large magnitude and should be followed by batch normalization layer. Similarly, we apply layer normalization (LN) to scale the output values of each layer, since LN is widely used in the transformer architecture. In fact, although LN introduces multiplications, the magnitude of the multiplication can be omitted compared to the total computation of a layer. The FLOPs of adder MHSA is DM + Nkw) + d,) N2, M (ds. + d,),, and the FLOPs of adder FFN is 2N, D. where is the dimension expansion ratio. The overall FLOPs of a standard adder transformer block is O (2N D(6D + N)), for regular setting r N N₂ Ng.. The overall FLOPS of multiplication in Layer Normalization D). In practice, given D = 768 and N = 197, the ratio of standard adder transformer block with the LN could be estimated as 2N D() 12ND D
800, which shows that LN can be easily applied in
adder transformer with tiny extra energy cost.
For pac-prropggttion, directly calculating the partial derivative of output w.r.t input and weight will lead to sign update of (- +1) value, respectively. Back-propagation using the sign gradient detrimental to the efficient updating of the parameters because the direction of the gradient can never
3

---

reach the steepest. Thus, we follow AdderNet [6] to compute the derivative of the f2-norm with HardTanh function to prevent gradients from exploding. Back-propagation process formulated as:
iQmia Qm... Xz,ā), ,x
m.ik HWWWK,,, K,, Xx,d), 8X,,a
VVm.s HTWWV Vmngoi Xx,a) 0X.aa
where HT(-) denotes the HardTanh function, and adder linear transformation in FFN modules follows the same inference and optimization process.
3.2 Adder Multi-Head Self-attention
3.2.1 Inference process in adder multi-head self-attention
Self-attention mechanism can be viewd as measument the similarity of input query and key matrices. The output is usually positively correlated with the value and the correlation is determined by the magnitude of the attention score after normalization. We design the adder multi-head self- attention strict compliance with the above principles which measure the similarity between vectors with the help of the 1 norm, an effective mcasure to avoid multiplication operations. Hence, by calculating di-dittance between each query and key vector, adder multi-head self-attention can be formulated as:
exp
Qm, Km
Vda
NEo
-110m. - Kmmilll }. da = 2dg (1
2 π
Am..i
where Z3
=
exp
Vdla
(7)
wherein, Q R.MM,, K R.M.N...dd A E IR MRMM m E M] indicates head index, j E [L, [ke] and generally we set = Ng = New as number of patches, de dp indicates embedding dimension for each head. Ja. and Vd. represent the scaling factor of the dot-product attention function and adder attention function, respectively.
Theorem 1. Assuming that the components of Qm. and Km.,; i, are independent random variables following normal distribution, the variance of dot-product and Eg Eidiitaacee between Qu.j. and K Km can be formulated respectively as follows:
Var(Qmm.: Km,..) de, Var (- 1Qm.j Km.), = 2d, (1-2 2
(8)
where the independent assumption is followed by 27 and the proof is provided in Section A.2 of the supplementary material. According to Theorem. scaling factor -
v., in dot-product attention is to counteract the variance explosion, and we adjust the scaling factor to match the adder circumstances.
For the process of yielding output values, the model jointly attend to information different positions according to the attention map. Specifically, transformer model multiply the attention map with the value matrix directly to allocate each value with corresponding attention weights as Eq. 2 For the projection to the output, we still use additions descibed in Sec. 3.1 where Wo E R D IS the weight matrix of the projection adder linear layer.
New Om.j.v Am.j. Von..
Ómij.v LayerNorm (...i))
(9)
Σ
Concat (Orars O2ses Ohiess
,.M.:) j.x -Wond
3.2.2 Rank Analysis on Adder Attention
In this section, we analyze the feature representation capability of adder attention by the rank of attention map. We first perform spectral analysis of the normalized self-attention map matrix H, (Here we use H E R. N instead of A to represent 2-dimensional attention matrix) and adder attention map matrix H..

---

Specifically, we use Diny-Tiny model and adder DeiT-Tiny model on CIFAR-I0 dataset. order to explore the relationship between the rank of the two matrices, we apply singular value de-
composition on H, and H, over different layer
Normalized sinquar
and heads of the two models, and plot the nor-
malized cumulative singular value over 10000 images. As shown Figure. 2 the cumulative singular values of both attention matrices show a clear long- tail distribution, which indicates that few singular values dominate the matrix. Therefore, the original matrix can be replaced by low-rank matrix with small error. In fact, the
0.6
0.-
0.-
Transformer
Aditee Thansorreer
long-tail distribution effect of the adder atten- tion matrix is more skewed than that in common self-attention matrix, indicating that the rank of the additive matrix is much lower. For example, the normalized cumulative singular value of at- tention matrix reached 0.9 the 65-th largest singular value for common self celf-attention while the 25-th largest singular value for adder at- tention, ie., main information is concentrated in the less large singular values for adder attention and the rank of Ha is lower than H, Here we make theoretical analysis of the above results.
Adder Transformet Iscncitt
160
Singular valuee index
Figure 2: Spectrum analysis of the attention map in DeiT--Tiny and Adder DeiT-TTny with N = 197. The Y-axis represents the cumulative normalized singular value of the attention map H, and H and the X-axis represents the index the eigenvalue.
Theorem 2.
The attention matrix of adder self-attention Ha can be approximated by lower rank matrix with a certain degree of confidence than that of common self-attention Hy, i.c., when:
LB Pr (1H H,o1l =.))
(I(Ht, @w) (H, + w) = co))
(10)
where LB(-) denotes the lower bound, denotes the adder operation in Eq Ee: Ea for any column vector w, v E IRN of the value matrix, we have:
Rank (1)) < Hank(fi.)
(11)
Proof. We base on the distributional Johnson Linnsn--nLnddnssaruus lemma 18 finish the proof.
Lemma 1. Let R be a random N matrix, S k S N, with i.i.d. entries from N (0, 1/k). For any vector u, U R.N. have:
Pr Rull S (1+ > - e ( (22-2)))
(12)
Pr (ll=" R Ru u''vll S cl Tf bl) 2e (62 2//44
(13)
Given an approximation error > O. we define matrices A, H. to be the approximate low-rank matrices of He, H, respectively.
A. H R, 1. = 5 SHa,
(14)
S be where R be a random h X N matrix, 1 < hi < N, with ii.d entries from
random la N matrix, 1 S kj I N, with i.ddd entries from 1/k22) According to the
Sylvester's Inequality [1], we have:
rank (f)) I rank(H.) H.) kn (H.) S raak k2,
(15)
In common self-attention, for row vector I of matrix H, and any column vector v e IR of the value matrix, according to the union bound (7). we have:
Pr (1 Hewll I cl| ((HHe))) Pr (H Rv Hawl s (||
(16)
>1
Σ seH.
Pr (1 R Re rell c([zo|)),
5

---

Then according to the second item of the Johnson Linionnttnannssauuss. lemma, we reformulate Eq. 16as: as:
Pr (1 H I e||Hee/l
2Ne -
(17)
Note that Eq. 17 can be viewed as lower bound of the reliability when using low-rank matrix in common self-attention to approximate the original attention matrix within small error range e, and the lower bound is I.B,. 1 2N 2Ne-(t'-2 -("--e"ka/ which is determined by kj.
In adder self-attention, for any column vector E TRN of the value matrix, have:
Pr (I1(Ha @ @w) (Ha @ will - c|l Pr (II(S SH. Ha)ell S d|eeal)
= Pr (H, H )) - cl|eall)
(18)
Pr (lle 5|1 || <lle,ll)
where all elements in vector e E N are fixed to 1, and we define ea = Hac. Then according to the first item of the Johnson- Lindenssnnnsssausss lemma, reformulate Eq. 18
Pr (I @ w) (Ha @ w) s e||Haell)
Ne -e
(19)
Note that Eq. 19 can also be viewed as a lower bound of the reliability when using low-rank matrix in adder self- attentonnn to approximate the original attention matrix within small error range and the lower bound is LB, 1 Ne- (c2-<e))) which is determined by kg- When LB,, LB, have l22 ha = which means that the approximate matrix in adder self-attention is much lower rank than that of common self-attention, then the theorem follows.
2
According Theorem 2 and the above analysis, the attention matrix of the adder self-attention can be approximated by lower rank matrix, resulting in skewed distribution of information. To address this problem, we propose to increase the rank of the attention matrix by more balanced distribution of singular values to attenuate the information bias in the attention map. Taking the singular value index when the cumulative normalized singular value reaches 0.9 as the equivalent rank of the attention matrix, in practice, we add an Identity matrix each attention matrix, i.e..
H. Ha +I.
(20)
where Identity matrix I E RNNN, We further investigate the effect of adding an identity mapping.
Proposition 1. Denote the input attention matrix as H E NxN and an Identity mapping matrix as E RRNN The cumulative normalized singular value function f()), linear function g(-) and their difference D(-) are respectively defined as:
[(r)
Ei EmTos
g()) =
T. N - D())=
(21)
where 1,2,.... N, and [01, 02.-.,, Tr 0r:0011:111-110. 0r-1:--.... NN) (0) 02 ... σ represents the singular value of H, and D(r) 2 O. Denote the difference function after performing the operation H = H + as D(r). and we have the following assertions:
......., N) is non-negative and the second-order difference is non-positive, which is consistent with the concave function.
The first-order difference of fr) (r 1,2,
f(r) is closer to g(r) after performing the operation: D(r) S D(r).
The equivalent rank of il is higher than that of H: Rank(ft) 2 Rank(H).
According to Proposition. (The proof is provided in Section A.3 of the supplementary material), the distribution of singular values is more homogenized through adding identity mapping. Therefore, self-attention of each patch is reinforced to enable more balanced distribution of information in attention map, which increases the rank of the attention matrix. As result, more information recovered in the attention matrix, and is correspondingly beneficial for the model to capture.

---

3.2.3 Back-propagation in adder multi-head self-attention
Back-propagation of the attention layer needs to be done with two parts of gradients, namely the partial derivative of the output w.r.t value and normalized attention score and the partial derivative of attention w.r.t query and key, respectively. Denote A E RMx N2 derived from query and key and Á E x as the attention map after softmax function. In self-attention layers, the above partial derivatives are calculated as:
X XNas as the attention map
0Am.s. 3Qm.j vdn Km,i.g
@Am.j.
1 Qmiit
00m... Vv
00m.5.
Am.ie
Vmniee (22)
akm,s,t ddn
AAm.j,
For adder attention layer, we directly calculate the partial derivative of attention w.r. query and key to conduct to sign update. We also tried the HardTann gradient described in Sec. 3.1, but
the sign gradient showed better performance which is described in experiment part. Thus, the back- back- sockpropaggatin process is formulated as:
0.m..
1
AAmod 1 sign(Qm..j. Kmiis)s
sign(Km.5. Qmiiaah,
mQm.j.
DKm,ik
vdaa
(23)
80 = Amaid
80m.j.u = Vmndi.
A.m.j.
4 Experiments
4.1 Experiments on MNIST
To illustrate the effectiveness of the proposed Adder Transformer, we first train 6 block DeiT- Tiny (25) on the MNIST dataset. The detailed network structure is shown in the supplemental material. We use Adam optimizer [21] with. The batch size is set as 256. The original Deit- Tiny achieves a 99.46% accuracy with 0.62B multiplications and 0.62B additions. By replacing the multiplications in linear transformation and self-attention layers with additions, the proposed model achieves a 99.39% accuracy with 1. I.24B additions and minor multiplications. We visualize features in different networks as shown in Figure. [] In CNNs and vanilla AdderNet, the features are classified according to their angles [6] or gathered into clusters, since the convolution operation can be regarded as the cosine distance and adder operation takes ε fidisstnce as the similarity measurement. The features of vanilla transformer are distributed in an oval shape based on angles due to the unique computational paradigm. Adder Transformer combines the advantages of both AdderNet and transformer, and the features are gathered together while at the same time distributed in an oval shape.
4.2 Experiments on CIFAR
We then validate our method through the representative DeiT baselines 25 on CIFAR-10 and CIFAR- 100 dataset. CIFAR-1O 10 (CIFAR-I00 --00) dataset is composed of 50k different 32 32 training images and 10k test images from 10 (100) categories. We adopt the same data augmentation strategy as that in DeiT including random crop, random clip, Rand-Augment (8). Mixup (37) and CutMix (36) to boost the performance of baseline models following, For both model we use AdamW optimizer 21 and cosine learning rate decay policy with an initial learning rate of 0.000125. We use epochs for learning rate warm-up up 20 with 0.05 weight decay rate. For all experiments, the image size is set to be 224x224. We use NVIDIA Telsa-V V100 GPUs and train baseline model and corresponding adder model for same epochs using PyTorch (2 library for fair comparison. Note that the models are directly trained on CIFAR dataset from scratch instead of finetuning from a pretrained model from larger dataset like ImageNet. The experimental results are shown in Table. [ the original multiplicative network, while ANN denotes replacing the multiplications in lincar transformation and self-attention layers with additions by the proposed method.
MNN denotes
For instance, the DeiT-Base model, the Adder Transformer achieve nearly the same results (94.47% in CIFAR-10 and 74.49% in CIFAR-100) with MNN (94.83% in CIFAR- 10 and 74.75% in CIFAR- 100) with little multiplications. We further calculate the energy consumptions of different models. Values in both models are 32-bit floating numbers, and the energy consumptions for a 32-bit addition and multiplication are 0.9p/ and 3.7pJ, respectively (9. Adder Transformer can obtain an about 2.5x reduction on energy consumption of the Deit-B model from 80.7 to 32.9Bp/ at the cost of little performance loss. The results in DeiT-S and D- also suggest the proposed adder transformer can also achieve comparable performance to those of their basclines with massive multiplications.
7

---

Table 1: Classification results CIFAR-I0 -10 and CIFAR- 100
datasets (Training from scratch).
Model
Method
#Mul.
#Add.
Energy (pJ)
CIFAR-10
CIFAR-100
MNN
17.56B
17.56B
80.7B
94.83%
74.75%
DeiT-B
ANN
0,48B
34.64B
32.9B
94.47%
74.49%
MNN
4.60B
4.60B
21.2B
93.22%
73.06%
DeiT-S
ANN
0.24B
8.96B
8.9B
92.91%
72.74%
MNN
1,25B
1.25B
5.8B
92.61%
72.58%
DeiT-T
ANN
0.12B
2.38B
2.6B
92.38%
72.23%
Table 2: Classification results on ImageNet datasets (Training from scratch).
Model
Method
#Mul.
#Add.
Energy (pJ)
Top-1 Acc
Top-5 Acc
MNN
17.56B
17.56B
80.7B
81.8%
95.6%
DeîT-B
ANN
3.32B
31.80B
40.9B
80.4%
94.3%
MNN
4.60B
4.60B
21.2B
79.9%
95.0%
DeïT-S
ANN
0.96B
8.24B
11.0B
78.3%
93.6%
MNN
1.25B
1.25B
5.8B
72.2%
91.1%
DeiT-T
ANN
0.30B
2.20B
3.1B
70.5%
89.9%
4.3 Experiments on ImageNet
We also conduct experiments on ImageNet dataset. ImageNet is large scale vision dataset which consists of 1.2M different 224 224 pixel training images and 50k test images from 1000 different categories. We use the same data augmentation and data pre-processing method that of DeiT. Networks are trained for 600 epochs with an initial learning rate of 0.0005 and cosine learning rate decay. We use 5 epochs for learning rate warm-up [2) with 0.05 weight decay rate, and experiments are conducted on NVIDIA Tesla- V 100 GPUs. The first and last block remain multiplication. Experimental results shown in Table. 2 Adder Transformer DeiT-S achieves a 78.3% top-1 accurrcyy and 93.6% top-5 accuracy with 0.96B multiplications and 8.24B additions, which is slightly lower than the DeiT-S baseline in terms of accuracy but substantially reduce the energy-ined merry-ineeffcciett multiplications. The detailed energy costs are also reported Table. 2 All models trained ImageNet can reduce the energy cost for processing 224 x 224 image by factor of about 2x, which demonstrates that our models using additions intermediate blocks can also achieve comparable accuracy those of their baselines with massive multiplications.
4.4 Ablation Study
In this part, ablation studies are conducted to evaluate the effectiveness of the proposed adder linear transformation and adder self-attention. All experiments in this part is done with Deiy-Tiny network on the CIFAR dataset.
First we explore the effect of the different gradient calculation methods in back- propagation of different parts and the identity mapping, as shown in Table. B A total of five different settings were evaluated in this ablation study. The results shows that the optimal gradient calculation method is to use Hardtanh gradient linear transformation layer and sign gradient calculating the attention map. Besides, adding the identity matrix to the attention map brings more benefit to the performance, indicating the low rank property of the adder self-attention matrix and the positive effect identity matrix homogenizing the distribution of singular values, shown in Figure. The attention map visulization of different models is shown in Figure. 3 It can observed that for the original adder transformer the rank of attention matrix is low. After adding the identity matrix, the rank can be well preserved and is more close to the baseline.
To evaluate the fitness of the proposed adder linear transformation and adder self-attention with corresponding multiplicative versions, we take a reference to the mixed model which combines different types of linear transformation and self-attention. Specifically, we study four combinations: i) I Linear Transformation + Self-attention (i.e., original DeiT-Tiny); ii) Linear Transformation Adder Sclf-attention; iii) Adder Linear Transformation Self-attention; and iv) Adder linear Transformation

---

Table 3: Impact of different components of the proposed adder transformer CIFAR-10.
Sign update on LT
HT update on LT
Sign update on w.r.t Q and K HT update on A w.r.t Q and K Identity mapping Accuracy(%, (%)
90.17
88.63
89.98
90.31
92.38
(a) MNN, rank=65
(b) ANN, rank=25
(c) ANN with IM, rank=70
Figure 3: Visualization of attention map in different networks on CIFAR-10. 10. From left to right are Transformer, Adder Transformer w/o with identity mapping, respectively. The equivalent rank denoted as the singular value index when the cumulative normalized singular value reaches 0.9.
+ Adder Self-attention (i.e., Adder Transformer). For all models train them CIFAR-10 and CIFAR- 100 to evaluate the performance and the energy cost, as shown in Table. 4 The results reveal that after either replacing adder linear transformation with the corresponding multiplication version, the performance on both datasets can have a slight improvement of 0.17% and 0.25% with additional 2.9Bp/ of energy consumption. For the case in adder self-attention, the performance both datasets can also have slight improvement of 0.07% and 0.08% with an additional 0.2Bp/ of energy consumption. The mixed model results illustrate good adaptability of the proposed method.
Table 4: Mixed model results on CIFAR-10
10 and CIFAR-100.
Linear Transformation
Self-attention
C1FAR-10
CIFAR-I00
Energy (pJ)
M
92.61%
72.58%
5.8B
M A
92.55%
72.48%
5.5B
M 92.45%
72.31%
2.8B
A
A 92.38%
72.23%
2.6B
5
Conclusion
This paper investigates implementing transformers using cheap addition operations. We first theoret- ically analyze the mechanism of self-attention and the difficulty for applying adder operation into this module. Specifically, we demonstrate the low-rank property of the attention matrix the adder transformer models and propose homogenize the distribution of singular values through adding an identity mapping. With the new operation, vision transformers constructed using additions can also provide powerful feature representations, which shows the potential of the addernet.
Funding Disclosure

---

References
[1] R. L. Arriaga and S. Vempala. An algorithmic theory of learning: Robust concepts and random projection. Machine Learning, 2006.
[2] Tom B Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurlPS, 2020.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers, ECCV, 2020.
[4] Hanting Chen, Yunhe Wang. Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwci Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In CVPR, 2021.
[5] Hanting Chen, Yunhe Wang, Han Shu, Changyuan Wen, Chunjing Xu, Boxin Shi, Chao Xu, and Chang Xu. Distilling portable generative adversarial networks for image translation. In AAAI, 2020.
[6] Hanting Chen, Yunhe Wang, Chunjing Xu, Boxin Shi, Chao Xu, Qi Tian, and Chang Xu. Addernet: Do really need multiplications in deep learning? In CVPR, 2020.
[7] Wikipodia contributors. Boolc's incquality Wikipedia, the free encyclopedia, 2021.
[8] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V.. Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshops, 2020.
[9] B. Dally. Tutorial: highh-perforance hardware for machine learning.
[10] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. NAACL, 2019.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2020.
[12] Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high resolution image synthesis. arXiv preprint arXiv:2012.00 00041, 2020.
[13] Song Han, Huizi Mao, and William Dally: Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. ICLR, 2016.
[14] Ben Ward (https://math.. (ttps://math.stcceechaag www.sccecccggg..ooeeseee com/users/61 189/ben ward). Sylvester rank incquality: rank a + rank b rank ab Mathematics Stack Exchange.
[15] Drew A Hudson and Lawrence Zitnick. Generative adversarial transformers. arXiv preprint vrXiv:2103.01209222 2021.
[16] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer- arithmetic-only inference. In CVPR, 2018.
[17] Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two transformers can make one strong gan. arXiv preprint 2rXiv:2102.07074. 2021.
[18] W. B. Johnson and J. Lindenstrauss. Extensions of lipschitz mappings into hilbert space. Contemp Math, 1982.
[19] Hao Li, Asim Kadav, Igor Durdanovie, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. ICLR, 2016.
[20] I. Loshchilov and F Hutter. Sgdr: Stochastic gradient descent with restarts. 2016.
[21] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv:171100510,, 2017.
arXiv preprint
[22] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, perh-pperfrmancc deep lcarning library. 2019.
[23] Han Shu, Yunhe Wang, Xu Jia, Kai Han, Hanting Chen, Chunjing Xu, Qi Tian, and Chang Xu. Co- evolutionary compression for unpaired image translation. In ICCV, 2019.
[24] Dehua Song, Yunhe Wang, Hanting Chen, Chang Xu, Chunjing Xu, and DaCheng Tao. Addersr: Towards energy efficient image super-resolution. In CVPR, 2021.
[25] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint 1rX1v:2012.12877. 2020.
[26] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles. Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXiv preprint arXiv:2103. 2021.
10

---

[27]
Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint vrXiv:/706.03762. 2017.
[28] Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006. 04768, 2020.
[29] Yunhe Wang. Mingqiang Huang. Kai Han, Hanting Chen, Wei Zhang, Chunjing Xu, Dacheng Tao. Addernet and its minimalist hardware design for energy-efficient artificial intelligence. arXiv preprint arXiv:2101. 10015,015000 2021.
[30] Yunhe Wang. Chang Xu, Shan You, Dacheng Tao, and Chao Xu. Canpack: Packing convolutional neural networks in the frequency domain. In NIPS, 2016.
[31] Zhanghao Wu, Zhijian Liu, Lin, Yujun Lin, and Song Han. Lite transformer with long- range attention. In ICLR, 2020.
[32] Yixing Xu, Chang Xu, Xinghao Chen, Wei Zhang, Chunjing Xu, and Yunhe Wang. Keree based progressive distillation for adder neural networks. In NeurlPS, 2020.
[33] Zhuyu Yao, Jiangho Ai, Boxun Li, and Chi Zhang. Efficient detr: Improving end-to-end object detector with dense prior. arXiv preprint orXiv:2104.01316 2021.
[34] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.1196. 2021.
[35] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. ICCV, 2019.
[36] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint PrXiv:171000PM
2017.
[37] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable transformers for end-te co-d object detection. arXiv preprint 49X0: 2020.


Reviews:
[{"review_id": "wzgwoFNT621", "reviewer": "Reviewer_KnYJ", "summary": "This paper presents a method that converts multiplication by addition operation during the deployment of vision transformers. \n**the papers main contribution**\n1) The authors theoretically analyzed the mechanism of self-attention and the difficulty of applying adder operation into this module. Specifically, the feature diversity i.e., the rank of attention map using only additions cannot be well preserved.\n2) Addressing previously raised difficulty, the authors develop an adder attention layer that includes an additional identity mapping, which allows vision transformers to be constructed only by additions operations.\n3) The empirical results show that the proposed scheme achieves near baseline performance on several image classification datasets while achieving about 3$\\times$ reduction on the total energy consumption \n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is mainly focused on reducing the power consumption of VIT (Vision Transformers). deployment. \nFor that proposed the suggest converts multiplication operations by addition operations. \\\n**Strengths**\nThe paper is a very well-written and theoretical analysis that raised the difficulties of straightforward usage of addition operations looks convincing. Also, the proposed solution insertion of the extra identity mapping in the adder attention module is well motivated. In the end, the empirical results show that the proposed scheme achieves near baseline performance on several image classification datasets while achieving about 3$\\times$ reduction on the total energy consumption and specifically about 2$\\times$ reduction for high energy consumption ImageNet classification VIT.\n\n**Weaknesses**\nThe authors didn't provide any experiments on NLP tasks which is well known as the common usage of transformers models. \n\nI encourage the authors to address this issue in the rebuttal phase\n\nPost-rebuttal: \n\nI am satisfied with the provided answers by the authors to the reviewers' concerns and I am convinced that this manuscript presents important insight to the community, hence I vote for his acceptance and keep my pre-rebuttal score.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "I didn't find any ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3-4h"}, "extra_text": {}}, {"review_id": "vd_VCSJieRm", "reviewer": "Reviewer_SWyG", "summary": " This paper designs the novel Adder paradigm for vision transformer. Specifically, the attention module is conducted with additions which saves energy consumption. This paper demonstrate the expressive power of addition computation paradigm both theoretically and experimentally on vision transformers. Experiments results show comparable accuracy on classification tasks on cifar10，cifar100 and ImageNet while saving the energy cost by 2x-3x . \n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an interesting paper. The authors develop the visual transformer structures with only addition operations, where the structure and computation paradigm of attention module is carefully redesigned. This proposal can save the energy cost of transformer structures since the addition is computationally cheaper than multiplications according to previous literatures.\n\nThe author revisit the main visual transformer modules and adaptively design them in additive way. For attention module, the author discuss the possible way of realizing it with only additions and absolute operation and choose the way which both satisfy the computation constraint and the attention principle following transformers. The authors analyze the adder attention module both theoretically and experimentally. Specifically, the author conduct rank analysis on adder attention module and point out that the low rank phenomenon of the adder attention module could cause the performance decay. The decrease of attention map rank is proved theoretically and observed experimentally. To solve the issue, the author proposed to increase attention map rank by adding an identity mapping. This operation improve the map rank and the final performance.  \n\nWith all the above, the ViT with adder attention achieves comparable accuracy on classification tasks on CIFAR10, CIFAR100 and ImageNet while saving 2x-3x energy cost compared with conventional structures of DeiTs. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\n Overall I like this paper and think it made good contributions to both ViT and efficient computation fields. Some weaknesses and comments are:\n\n1.Why the identity of the attention map can improve the expressive power of attention module? Is Identity mapping the best choice? More discussion should be conducted. \n2.How about this adder attention applied on natural language processing tasks? \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "11"}, "extra_text": {}}, {"review_id": "dBppNprxg-S", "reviewer": "Reviewer_dTzD", "summary": "This paper presents AdderNet for Transformer. To this end, the feed-forward module and self-attention mechanism are replaced with adder operations (l1 distance). The details of forward and backward computations of AdderNet for Transformer are also provided for computer vision tasks. The proposed network is tested on CIFAR-10, CIFAR-100 and ImageNet. It's shown that the proposed network can successfully perform the image classification tasks with a small accuracy degradation while reducing the energy consumption by a factor of about 3.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, the paper is novel and the idea of replacing multiplications with additions in Transformer is interesting specially in energy-constrained deployment scenarios.The paper is well-written and well-motivated in my opinion. Also, the accuracy results on image classification task are vigorous. My only suggestion is to make a comparison with AdderNet for CNNs in order to show trade-offs or superiority of one over another.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "I have no concern.", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "OkNi04rO95", "reviewer": "Reviewer_2Zxc", "summary": "This paper applies adder neural networks to vision transformers. Specifically, the authors propose adder attention to avoid multiplication operations and offer rank analysis and back-propagation on the adder attention.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "###  Originality\nAlthough the concept of adder neural netowrk is not new, this paper applies this conect into transformers and propose adder attention to aovid multiplicaiton operations for conducting attention. The idea is okey. The author also offers analysis on adder attention.\n\n### Quality\n#### Strengths\n+ The paper writes well and the proposed methods are easily to understand.\n+ Beyond dot product of attention, this paper proposes new attention method under the concept of adder neural netowrk that can inspires more different methods to conduct attention.\n\n#### Weaknesses\n+ My major concern is the performance of adder attention. Taking Table 3 for example, it seems the ader neural network (ANN) is much worse than original multiplicative network (MNN), like for DeiT-S (ANN 10.5B energy 78.3% vs MNN 21.2B 79.8%). How about the performance of ANN and MNN with the same energy? DeiT-S has 384 channel dimension and 12 layers.  Maybe we can build a baseline of DeiT called DeiT-XS with 256 channel dimension and 12 layers so that DeiT-XS (MNN) has half of energy of DeiT-S (MNN), the around same as energy of DeiT-S (ANN). In this way, we can directly compare the performance of DeiT-XS (MNN) and DeiT-S (ANN). \n\n#### Clarity\nThe submission is clearly written.\n\n### Significance\nUnder the concept of adder nerural nentowrk, this paper propose a new attention method adder attention, but the performance of the method is not good.\n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are all in agreement that this paper provides novel and interesting results exploring a theoretical analysis of self-attention, and the introduction of an adder attention layer which helps reduce energy requirements. I would encourage the authors to incorporate some of the additional results on NLP tasks and CNN comparisons into their main text.
Decision: Accept (Poster)