Adder Attention for Vision Transformer
Year: 2021

Abstract:
Transformer is a new kind of calculation paradigm for deep learning which has shown strong performance on a large variety of computer vision tasks. However, compared with conventional deep models (e.g., convolutional neural networks), vision transformers require more computational resources which cannot be easily deployed on mobile devices. To this end, we present to reduce the energy consumptions using adder neural network (AdderNet). We first theoretically analyze the mechanism of self-attention and the difficulty for applying adder operation into this module. Specifically, the feature diversity, i.e., the rank of attention map using only additions cannot be well preserved. Thus, we develop an adder attention layer that includes an additional identity mapping. With the new operation, vision transformers constructed using additions can also provide powerful feature representations. Experimental results on several benchmarks demonstrate that the proposed approach can achieve highly competitive performance to that of the baselines while achieving an about 2 ̃3× reduction on the energy consumption.

heading: 1 Introduction
text: Transformers [27] have prevailed in natural language processing (NLP) due to its superior capability in capturing long-distance dependencies based on self-attention mechanism. Some homologous large-scale models, e.g., BERT [10] and GPT-3 [2] provide a significant performance improvement on learning powerful language representations from unlabeled text. The great breakthrough of transformers in NLP has sparked particular interest from the vision community in applying transformer to computer vision (CV) tasks such as image recognition [11, 35, 25, 26], object detection [3, 38, 33], and image generation [4, 12, 17, 15].
The high-power consumption of transformer-based models has blocked them from being deployed on mobile devices, e.g., smart phone, camera, and micro-robots. Therefore, it is necessary to study efficient transformers which can be embedded on mobile devices with affordable computation resources. Wu et.al. [31] present an efficient Long-Short Range Attention with two groups of heads, respectively for local context and long-distance relationship modeling. Wang et.al. [28] decompose the original dot-product attention into smaller attentions through linear projections. Unfortunately, previous arts mainly focus on compressing and accelerating transformers for language processing tasks, which motivates us for building efficient transformers for computer vision applications.
Various attempts have been made on compressing CNN models, including quantization [13, 16], pruning [19, 30, 23] and distillation [5, 25]. Beyond these methods, Chen et.al. [6] propose Adder Neural Network (AdderNet) for building efficient deep learning models, which avoids massive computational cost by replacing the convolutional operation by using `1-distance between input
*Equal contribution. †Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
signals and weights instead of correlation. Xu et.al. [32] further utilize knowledge distillation technique to make AdderNet achieve better performance than CNN. Song et.al. [24] successfully apply AdderNet on single image super-resolution (SISR) task by learning identity mapping with self-shortcuts and implementing high-pass filter with a learnable power activation. Wang et.al. [29] conduct practical implementations on hardware platform (FPGA), and verify its superior performance in suppressing chip area and power consumption. Thus, we are motivated to investigate the feasibility of replacing multiplications by additions in transformer architectures.
However, it is very hard to directly transport the existing success of AdderNet on CNNs to transformers, since their basic calculation paradigm is completely different. In particular, the existing works mainly focus on the additiveization of convolutional kernel filters which do not involve the unique linear transformations and self-attention mechanisms in transformers. To this end, we propose a general adder linear transformation operation and adder self-attention mechanism to replace the traditional multiplication version in both feed-forward modules and self-attention layers in transformer model. Specifically, we utilize `1-distance between query and key instead of scaled dot-product to measure the distance between them. For the attention module, we find that the directly using additions could cause more information concentrating on a few largest singular values, leading to a decay of the attention map rank, i.e., the functionality of the attention mechanism cannot be well preserved. Thus, we propose to insert an extra identity mapping in the adder attention module to solve the decay. This operation can also homogenize the information distribution of the adder attention map. Meanwhile, we provide the detailed feed-forward and back-propagation for optimizing the adder vision transformer with stable training process. As a result, we can obtain comparable performance using the proposed adder transformer models to that of original baselines while reducing about 3× of the overall energy consumption on various benchmark models and datasets.

heading: 2 Preliminaries and Motivation
text: In this section, we briefly revisit the basic related components, including AdderNet and transformer.
Adder Neural Networks (AdderNet). Denote filters in a convolutional layer of neural networks as W ∈ Rk×k×cin×cout , the input feature map as X ∈ RH×W×cin , and the output feature map as Y ∈ RH×W×cout , where k is the kernel size, cin and cout are the number of input channels and output channels, respectively. The traditional convolutional operation is defined as:
Ym,n,q , X ∗ F = D∑ i=1 D∑ j=1 cin∑ k=1 S (Xm+i,n+j,k, Fi,j,k,q) . (1)
where S(·, ·) is a pre-defined similarity measure (S(x, y) = x× y for the convolution multiplications operation). AdderNet [6] maximizes the use of additions by treating the `1-distance between the input feature map and filter as the similarity measure (i.e., S(x, y) = |x− y| in Eq. 1):
Ỹm,n,q , X ⊕ F = − D∑ i=1 D∑ j=1 cin∑ k=1 |Xm+i,n+j,k − Fi,j,k,q|. (2)
However the pioneering work only focused on additiveization of CNNs, while ignoring transformers.
Vision Transformers. Vision transformer [11] mainly consists of MHSA (Multi-head Self-Attention), FFN (Feed-Forward Network) and LN (Layer Normalization).
For multi-head self-attention, the inputs X ∈ RN×D are applied with three different linear transformations and output queries Q ∈ RM×N×dq , keys K ∈ RM×N×dk and values V ∈ RM×N×dv , where M is total number of heads, dq, dq, dv is the dimension of queries, keys and values for each head, respectively. Generally, D =Mdq =Mdk =Mdv =Mdh and the multi-head self-attention is performed as follows:
Am,j,i =
exp
{ Qm,j,:K
T m,i,:√
dt } Zj , where Zj = Nkv∑ i=1 exp { Qm,j,:K T m,i,:√ dt } , Om,j,v = Nkv∑ i=1 Am,j,i · Vm,i,v,
O ′ j,d = Concat (O1,:,:, O2,:,:, · · · , OM,:,:)WO.
(3) where dt = dh, WO ∈ RD×D denotes the weight matrix of a linear projection layer used to produce the output values O
′ ∈ RN×D. From the above statements, transformer has a different structure and computation paradigm compared to CNNs. Thus, it is necessary to study how to use additions to fulfill the modules.

heading: 3 The Proposed Model: Adder Transformer
text: In this section, we present the Adder Transformer— which implement the multi-head attention module and FFN module using adder operations. We will present the details in the following subsections and an illustration of the proposed method can be found in Section A.1 of the supplementary material.

heading: 3.1 Adder Linear Transformation
text: We first consider the linear transformation and FFN layers in transformers. The calculation paradigm of these two modules is based on fully connected layer, which is equivalent to a special 1x1 convolution. Therefore, we can use the original computation mechanism of adder convolutional kernel in this module. Considering weight matrices WQ ∈ RM×dq×D, WK ∈ RM×dk×D, WV ∈ RM×dv×D in a projection layer of the multi-head self-attention, we maximize the use of additions by taking `1 distance to measure the distance between the weight matrices and input embedding as:
Qm,j,q = D∑ d=1 −|Xj,d −WQm,q,d |, Km,i,k = D∑ d=1 −|Xi,d −WKm,k,d |, Vm,i,v = D∑ d=1 −|Xi,d −WVm,v,d |, (4) wherein, m ∈ [1,M ] , j ∈ [1, Nq] , i ∈ [1, Nkv] , q ∈ [1, dq] , k ∈ [1, dk] , v ∈ [1, dv] and generally we set N = Nq = Nkv as the resulting number of patches and D =Mdq =Mdk =Mdv =Mdh as the model embedding dimension. The feed-forward network can also be modified directly:
FFNa(X) =W2 ⊕ (W1 ⊕X + b1) + b2, (5)
As mentioned in AdderNet [6], the adder layers output values with large magnitude and should be followed by a batch normalization layer. Similarly, we apply layer normalization (LN) to scale the output values of each layer, since LN is widely used in the transformer architecture. In fact, although LN introduces multiplications, the magnitude of the multiplication can be omitted compared to the total computation of a layer. The FLOPs of adder MHSA is DM(Nq + Nkv)(dk + dv) + NqNkvM(dk + dv), and the FLOPs of adder FFN is 2NqDrD, where r is the dimension expansion ratio. The overall FLOPs of a standard adder transformer block is O (2ND(6D +N)), for regular setting r = 4, N = Nq = Nkv. The overall FLOPs of multiplication in Layer Normalization are O(12ND). In practice, given D = 768 and N = 197, the ratio of a standard adder transformer block with the LN could be estimated as 2ND(6D+N)12ND ≈ 800, which shows that LN can be easily applied in adder transformer with tiny extra energy cost.
For back-propagation, directly calculating the partial derivative of output w.r.t input and weight will lead to a sign update of {−1, 0,+1} value, respectively. Back-propagation using the sign gradient is detrimental to the efficient updating of the parameters because the direction of the gradient can never
reach the steepest. Thus, we follow AdderNet [6] to compute the derivative of the `2-norm with a HardTanh function to prevent gradients from exploding. Back-propagation process is formulated as:
∂Qm,j,q ∂Xj,d = HT (WQm,q,d −Xj,d), ∂Km,i,k ∂Xi,d = HT (WKm,k,d −Xi,d), ∂Vm,i,v ∂Xi,d = HT (WVm,v,d −Xi,d) (6) where HT(·) denotes the HardTanh function, and adder linear transformation in FFN modules follows the same inference and optimization process.

heading: 3.2 Adder Multi-Head Self-attention

heading: 3.2.1 Inference process in adder multi-head self-attention
text: Self-attention mechanism can be viewd as the measument of the similarity of input query and key matrices. The output is usually positively correlated with the value and the correlation is determined by the magnitude of the attention score after normalization. We design the adder multi-head selfattention in strict compliance with the above principles which measure the similarity between vectors with the help of the `1-norm, an effective measure to avoid multiplication operations. Hence, by calculating `1-distance between each query and key vector, adder multi-head self-attention can be formulated as:
Am,j,i = exp
{ −‖Qm,j,:−Km,i,:‖1√
da } Zj where Zj = Nkv∑ i=1 exp { −‖Qm,j,: −Km,i,:‖1√ da } , da = 2dt ( 1− 2 π ) ,
(7)
wherein, Q ∈ RM×Nq×dq , K ∈ RM×Nkv×dk , A ∈ RM×Nq×Nkv , m ∈ [1,M ] indicates head index, j ∈ [1, Nq] , i ∈ [1, Nkv] and generally we set N = Nq = Nkv as number of patches, dt = dh indicates embedding dimension for each head. 1√
dt and 1√ da represent the scaling factor of the
dot-product attention function and adder attention function, respectively. Theorem 1. Assuming that the components of Qm,j,: and Km,i,: are independent random variables following normal distribution, the variance of dot-product and `1-distance between Qm,j,: and Km,i,: can be formulated respectively as follows:
V ar(QTm,j,:Km,i,:) = dt, V ar (−‖Qm,j,: −Km,i,:‖1) = 2dt ( 1− 2
π
) , (8)
where the independent assumption is followed by [27] and the proof is provided in Section A.2 of the supplementary material. According to Theorem. 1, scaling factor 1√
dt in dot-product attention is to
counteract the variance explosion, and we adjust the scaling factor to match the adder circumstances.
For the process of yielding output values, the model jointly attend to information at different positions according to the attention map. Specifically, transformer model multiply the attention map with the value matrix directly to allocate each value with corresponding attention weights as in Eq. 9. For the projection to the output, we still use additions descibed in Sec. 3.1, where WO ∈ RD×D is the weight matrix of the projection adder linear layer.
Om,j,v = Nkv∑ i=1 Am,j,i · Vm,i,v,
Ôm,j,v = LayerNorm(Om,j,v),
O ′ j,d = − D∑ s=1 ∣∣∣∣Concat(Ô1,:,:, Ô2,:,:, · · · , ÔM,:,:) j,s −WOs,d ∣∣∣∣ . (9)

heading: 3.2.2 Rank Analysis on Adder Attention
text: In this section, we analyze the feature representation capability of adder attention by the rank of attention map. We first perform a spectral analysis of the normalized self-attention map matrix Ht (Here we use H ∈ RN×N instead of A to represent 2-dimensional attention matrix) and adder attention map matrix Ha.
Specifically, we use DeiT-Tiny model and adder DeiT-Tiny model on CIFAR-10 dataset. In order to explore the relationship between the rank of the two matrices, we apply singular value decomposition on Ht and Ha over different layer and heads of the two models, and plot the normalized cumulative singular value over 10000 images. As shown in Figure. 2, the cumulative singular values of both attention matrices show a clear long-tail distribution, which indicates that a few singular values dominate the matrix. Therefore, the original matrix can be replaced by a low-rank matrix with small error. In fact, the long-tail distribution effect of the adder attention matrix is more skewed than that in common self-attention matrix, indicating that the rank of the additive matrix is much lower. For example, the normalized cumulative singular value of attention matrix reached 0.9 at the 65-th largest singular value for common self-attention while at the 25-th largest singular value for adder attention, i.e., main information is concentrated in the less large singular values for adder attention
and the rank of Ha is lower than Ht. Here we make a theoretical analysis of the above results. Theorem 2. The attention matrix of adder self-attention Ha can be approximated by a lower rank matrix with a certain degree of confidence than that of common self-attention Ht, i.e., when:
LB ( Pr ( ‖Ĥtv −Htv‖ ≤ t )) = LB ( Pr ( ‖(Ĥa ⊕ w)− (Ha ⊕ w)‖ ≤ a )) (10)
where LB(·) denotes the lower bound, ⊕ denotes the adder operation in Eq.9, t, a > 0, for any column vector w, v ∈ RN of the value matrix, we have:
Rank(H̃a) < Rank(H̃t) (11)
Proof. We base on the distributional Johnson–Lindenstrauss lemma [18, 1] to finish the proof.
Lemma 1. Let R be a random k ×N matrix, 1 ≤ k ≤ N , with i.i.d. entries from N(0, 1/k). For any vector u, v ∈ RN , we have:
Pr (‖Ru‖ ≤ (1 + )‖u‖) > 1− e−( 2− 3)k/4, (12) Pr ( ‖uTRTRv − uT v‖ ≤ ‖uT v‖ ) > 1− 2e−( 2− 3)k/4. (13)
Given an approximation error > 0, we define matrices Ĥt, Ĥa to be the approximate low-rank matrices of Ht, Ha respectively.
Ĥt = HtR TR, Ĥa = S TSHa, (14)
where R be a random k1 × N matrix, 1 ≤ k1 ≤ N , with i.i.d. entries from N(0, 1/k1), S be a random k2 × N matrix, 1 ≤ k1 ≤ N , with i.i.d. entries from N(0, 1/k2). According to the Sylvester’s Inequality [14], we have:
rank(Ĥt) ≤ rank(Ht) = k1, rank(Ĥa) ≤ rank(Ha) = k2, (15)
In common self-attention, for row vector x ∈ RN of matrix Ĥt and any column vector v ∈ RN of the value matrix, according to the union bound [7], we have:
Pr ( ‖Ĥtv −Htv‖ ≤ ‖Htv‖ ) =Pr ( ‖HtRTRv −Htv‖ ≤ ‖HtvT ‖ ) ≥1−
∑ x∈Ht Pr ( ‖xRTRv − xv‖ > ‖xv‖ ) ,
(16)
Then according to the second item of the Johnson–Lindenstrauss lemma, we reformulate Eq. 16 as: Pr ( ‖Ĥtv −Htv‖ ≤ ‖Htv‖ ) > 1− 2Ne−( 2− 3)k1/4. (17)
Note that Eq. 17 can be viewed as a lower bound of the reliability when using low-rank matrix in common self-attention to approximate the original attention matrix within a small error range , and the lower bound is LBm = 1− 2Ne−( 2− 3)k1/4 which is determined by k1.
In adder self-attention, for any column vector w ∈ RN of the value matrix, we have: Pr ( ‖(Ĥa ⊕ w)− (Ha ⊕ w)‖ ≤ ‖Hae‖ ) =Pr ( ‖(STSHa −Ha)e‖ ≤ ‖ea‖ ) =Pr ( ‖eT (HTa STS −HTa )‖ ≤ ‖ea‖
) >Pr ( ‖eTa STS‖ − ‖eTa ‖ ≤ ‖ea‖ ) .
(18)
where all elements in vector e ∈ RN are fixed to 1, and we define ea = Hae. Then according to the first item of the Johnson–Lindenstrauss lemma, we reformulate Eq. 18 as:
Pr ( ‖(Ĥa ⊕ w)− (Ha ⊕ w)‖ ≤ ‖Hae‖ ) > 1−Ne−( 2− 3)k2/4. (19)
Note that Eq. 19 can also be viewed as a lower bound of the reliability when using low-rank matrix in adder self-attention to approximate the original attention matrix within a small error range , and the lower bound is LBa = 1−Ne−(
2− 3)k2/4 which is determined by k2. When LBm = LBa we have k2 = k1 − 4 log 2 2− 3 , which means that the approximate matrix in adder self-attention is much lower rank than that of common self-attention, then the theorem follows.
According to Theorem 2 and the above analysis, the attention matrix of the adder self-attention can be approximated by a lower rank matrix, resulting in a skewed distribution of information. To address this problem, we propose to increase the rank of the attention matrix by a more balanced distribution of singular values to attenuate the information bias in the attention map. Taking the singular value index when the cumulative normalized singular value reaches 0.9 as the equivalent rank of the attention matrix, in practice, we add an Identity matrix to each attention matrix, i.e.,
H̃a = Ha + I. (20)
where Identity matrix I ∈ RN×N . We further investigate the effect of adding an identity mapping. Proposition 1. Denote the input attention matrix as H ∈ RN×N and an Identity mapping matrix as I ∈ RN×N . The cumulative normalized singular value function f(·), linear function g(·) and their difference D(·) are respectively defined as:
f(r) = ∑r t=1 σt∑N s=1 σs , g(r) = r N , D(r) = f(r)− g(r), (21)
where r = 1, 2, ..., N , and σ = [σ1, σ2, ..., σr, σr+1, ..., σN ] (σ1 ≥ σ2 ≥ ... ≥ σN ) represents the singular value of H , and D(r) ≥ 0. Denote the difference function after performing the operation H̃ = H + I as D̃(r), and we have the following assertions:
• The first-order difference of f(r) (r = 1, 2, ..., N) is non-negative and the second-order difference is non-positive, which is consistent with the concave function.
• f(r) is closer to g(r) after performing the operation: D̃(r) ≤ D(r).
• The equivalent rank of H̃ is higher than that of H: Rank(H̃) ≥ Rank(H).
According to Proposition. 1 (The proof is provided in Section A.3 of the supplementary material), the distribution of singular values is more homogenized through adding identity mapping, Therefore, self-attention of each patch is reinforced to enable a more balanced distribution of information in attention map, which increases the rank of the attention matrix. As a result, more information is recovered in the attention matrix, and is correspondingly beneficial for the model to capture.

heading: 3.2.3 Back-propagation in adder multi-head self-attention
text: Back-propagation of the attention layer needs to be done with two parts of gradients, namely the partial derivative of the output w.r.t value and normalized attention score and the partial derivative of attention w.r.t query and key, respectively. Denote A ∈ RM×Nq×Nkv as the attention map derived from query and key and Â ∈ RM×Nq×Nkv as the attention map after softmax function. In self-attention layers, the above partial derivatives are calculated as: ∂Am,j,i ∂Qm,j,q = 1√ dh Km,i,q, ∂Am,j,i ∂Km,i,k = 1√ dh Qm,j,k, ∂Om,j,v ∂Vm,i,v = Âm,j,i, ∂Om,j,v ∂Âm,j,i = Vm,i,v, (22) For adder attention layer, we directly calculate the partial derivative of attention w.r.t query and key to conduct to a sign update. We also tried the HardTanh gradient described in Sec. 3.1, but the sign gradient showed better performance which is described in experiment part. Thus, the back-propagation process is formulated as:
∂Am,j,i ∂Qm,j,q = 1√ da sign(Km,i,q −Qm,j,q), ∂Am,j,i ∂Km,i,k = 1√ da sign(Qm,j,k −Km,i,k),
∂Om,j,v ∂Vm,i,v = Âm,j,i, ∂Om,j,v ∂Âm,j,i = Vm,i,v,
(23)

heading: 4 Experiments

heading: 4.1 Experiments on MNIST
text: To illustrate the effectiveness of the proposed Adder Transformer, we first train a 6 block DeiTTiny [25] on the MNIST dataset. The detailed network structure is shown in the supplemental material. We use AdamW optimizer [21] with. The batch size is set as 256. The original DeitTiny achieves a 99.46% accuracy with 0.62B multiplications and 0.62B additions. By replacing the multiplications in linear transformation and self-attention layers with additions, the proposed model achieves a 99.39% accuracy with 1.24B additions and minor multiplications. We visualize features in different networks as shown in Figure. 1. In CNNs and vanilla AdderNet, the features are classified according to their angles [6] or gathered into clusters, since the convolution operation can be regarded as the cosine distance and adder operation takes `1-distance as the similarity measurement. The features of vanilla transformer are distributed in an oval shape based on angles due to the unique computational paradigm. Adder Transformer combines the advantages of both AdderNet and transformer, and the features are gathered together while at the same time distributed in an oval shape.

heading: 4.2 Experiments on CIFAR
text: We then validate our method through the representative DeiT baselines [25] on CIFAR-10 and CIFAR100 dataset. CIFAR-10 (CIFAR-100) dataset is composed of 50k different 32× 32 training images and 10k test images from 10 (100) categories. We adopt the same data augmentation strategy as that in DeiT including random crop, random clip, Rand-Augment [8], Mixup [37] and CutMix [36] to boost the performance of baseline models following. For both model we use AdamW optimizer [21] and cosine learning rate decay policy with an initial learning rate of 0.000125. We use 5 epochs for learning rate warm-up [20] with a 0.05 weight decay rate. For all experiments, the image size is set to be 224×224. We use NVIDIA Telsa-V100 GPUs and train baseline model and corresponding adder model for same epochs using PyTorch [22] library for fair comparison. Note that the models are directly trained on CIFAR dataset from scratch instead of finetuning from a pretrained model from a larger dataset like ImageNet. The experimental results are shown in Table. 1, MNN denotes the original multiplicative network, while ANN denotes replacing the multiplications in linear transformation and self-attention layers with additions by the proposed method.
For instance, the DeiT-Base model, the Adder Transformer achieve nearly the same results (94.47% in CIFAR-10 and 74.49% in CIFAR-100) with MNN (94.83% in CIFAR-10 and 74.75% in CIFAR-100) with little multiplications. We further calculate the energy consumptions of different models. Values in both models are 32-bit floating numbers, and the energy consumptions for a 32-bit addition and multiplication are 0.9pJ and 3.7pJ, respectively [9]. Adder Transformer can obtain an about 2.5× reduction on energy consumption of the Deit-B model from 80.7BpJ to 32.9BpJ at the cost of little performance loss. The results in DeiT-S and DeiT-T also suggest the proposed adder transformer can also achieve comparable performance to those of their baselines with massive multiplications.

heading: 4.3 Experiments on ImageNet
text: We also conduct experiments on ImageNet dataset. ImageNet is a large scale vision dataset which consists of 1.2M different 224× 224 pixel training images and 50k test images from 1000 different categories. We use the same data augmentation and data pre-processing method as that of DeiT. Networks are trained for 600 epochs with an initial learning rate of 0.0005 and a cosine learning rate decay. We use 5 epochs for learning rate warm-up [20] with a 0.05 weight decay rate, and the experiments are conducted on NVIDIA Tesla-V100 GPUs. The first and last block remain the multiplication. Experimental results are shown in Table. 2. Adder Transformer on DeiT-S achieves a 78.3% top-1 accuracy and 93.6% top-5 accuracy with 0.96B multiplications and 8.24B additions, which is slightly lower than the DeiT-S baseline in terms of accuracy but substantially reduce the energy-inefficient multiplications. The detailed energy costs are also reported in Table. 2. All models trained on ImageNet can reduce the energy cost for processing a 224×224 image by a factor of about 2×, which demonstrates that our models using additions in intermediate blocks can also achieve comparable accuracy to those of their baselines with massive multiplications.

heading: 4.4 Ablation Study
text: In this part, ablation studies are conducted to evaluate the effectiveness of the proposed adder linear transformation and adder self-attention. All experiments in this part is done with DeiT-Tiny network on the CIFAR dataset.
First we explore the effect of the different gradient calculation methods in back-propagation of different parts and the identity mapping, as shown in Table. 3. A total of five different settings were evaluated in this ablation study. The results shows that the optimal gradient calculation method is to use Hardtanh gradient in linear transformation layer and sign gradient in calculating the attention map. Besides, adding the identity matrix to the attention map brings more benefit to the performance, indicating the low rank property of the adder self-attention matrix and the positive effect of identity matrix on homogenizing the distribution of singular values, as shown in Figure. 2.The attention map visulization of different models is shown in Figure. 3. It can be observed that for the original adder transformer the rank of attention matrix is low. After adding the identity matrix, the rank can be well preserved and is more close to the baseline.
To evaluate the fitness of the proposed adder linear transformation and adder self-attention with the corresponding multiplicative versions, we take a reference to the mixed model which combines different types of linear transformation and self-attention. Specifically, we study four combinations: i) Linear Transformation + Self-attention (i.e., original DeiT-Tiny); ii) Linear Transformation + Adder Self-attention; iii) Adder Linear Transformation + Self-attention; and iv) Adder linear Transformation

heading: 5 Conclusion
text: This paper investigates implementing transformers using cheap addition operations. We first theoretically analyze the mechanism of self-attention and the difficulty for applying adder operation into this module. Specifically, we demonstrate the low-rank property of the attention matrix of the adder transformer models and propose to homogenize the distribution of singular values through adding an identity mapping. With the new operation, vision transformers constructed using additions can also provide powerful feature representations, which shows the potential of the addernet.
Funding Disclosure

Review summaries:
 This paper designs the novel Adder paradigm for vision transformer. Specifically, the attention module is conducted with additions which saves energy consumption. This paper demonstrate the expressive power of addition computation paradigm both theoretically and experimentally on vision transformers. Experiments results show comparable accuracy on classification tasks on cifar10，cifar100 and ImageNet while saving the energy cost by 2x-3x .   

This paper presents a method that converts multiplication by addition operation during the deployment of vision transformers.  **the papers main contribution** 1) The authors theoretically analyzed the mechanism of self-attention and the difficulty of applying adder operation into this module. Specifically, the feature diversity i.e., the rank of attention map using only additions cannot be well preserved. 2) Addressing previously raised difficulty, the authors develop an adder attention layer that includes an additional identity mapping, which allows vision transformers to be constructed only by additions operations. 3) The empirical results show that the proposed scheme achieves near baseline performance on several image classification datasets while achieving about 3$\times$ reduction on the total energy consumption   

This paper presents AdderNet for Transformer. To this end, the feed-forward module and self-attention mechanism are replaced with adder operations (l1 distance). The details of forward and backward computations of AdderNet for Transformer are also provided for computer vision tasks. The proposed network is tested on CIFAR-10, CIFAR-100 and ImageNet. It's shown that the proposed network can successfully perform the image classification tasks with a small accuracy degradation while reducing the energy consumption by a factor of about 3.