LMSA: Low-relation Mutil-head Self-Attention Mechanism in Visual Transformer
Authors: JingJie Wang
Xiang Wei
Xiaoyu Liu
Venue: ICLR 2022 Submitted (2022)

Abstract: The Transformer backbone network with the self-attention mechanism as the core has achieved great success in the field of natural language processing and computer vision. However, through the self-attention mechanism brings high performance, it also brings higher computational complexity compared to the classic visual feature extraction methods. To further reduce the complexity of self-attention mechanism and explore its lighter version in computer vision, in this paper, we design a novel lightweighted self-attention mechanism: Low-relation Mutil-head Self-Attention (LMSA), which is superior than the recent self-attention. Specifically, the proposed self-attention mechanism breaks the barrier of the dimensional consistency of the traditional self-attention mechanism, resulting in lower computational complexity and occupies less storage space. In addition, employing the new mechanism can release part of the computing consumption of the Transformer network and  make the best use of it. Experimental results show that the dimensional consistency inside the traditional self-attention mechanism is unnecessary. In particular, using Swin as the backbone model for training, the accuracy in CIFAR-10 image classification task is improved by 0.43$\%$, in the meanwhile, the consumption of a single self-attention resource is reduced by 64.58$\%$, and the number of model parameters and model size are reduced by more than 15$\%$. By appropriately compressing the dimensions of the self-attention relationship variables, the Transformer network can be more efficient and even perform better. The results prompt us to rethink the reason why the self-attention mechanism works.

Under review as a conference paper at ICLR 2022
LMSA: LOW-RELATION MUTIL-HEAD
SELF- ATTENTION MECHANISM IN VISUAL TRANSFORMER
Anonymous authors Paper under double-blind review
ABSTRACT
The Transformer backbone network with the self-attention mechanism as the core has achieved great success in the field of natural language processing and com- puter vision. However, through the self-attention mechanism brings high perfor- mance, it also brings higher computational complexity compared to the classic vi- sual feature extraction methods. To further reduce the complexity of self-attention mechanism and explore its lighter version computer vision, in this paper, we design a novel lightweighted self-attention mechanism: Low-relation Mutil-head Self-Attention (LMSA), which is superior than the recent self-attention. Specifi- cally, the proposed self-attention mechanism breaks the barrier of the dimensional consistency of the traditional self-attention mechanism, resulting in lower com- putational complexity and occupies less storage space. In addition, employing the new mechanism can release part of the computing consumption of the Trans- former network and make the best use of it. Experimental results show that the dimensional consistency inside the traditional self-attention mechanism is unnec- essary. In particular, using Swin the backbone model for training, the accuracy in CIFAR-10 10 image classification task is improved by 0.43%, in the meanwhile, the consumption
single self-attention resource is reduced by 64.58%, and the number of model parameters and model size are reduced by more than 15%. By appropriately compressing the dimensions of the self-attention relationship vari- ables, the Transformer network can be more efficient and even perform better. The results prompt us to rethink the reason why the self-attention mechanism works.
INTRODUCTION
The self-attention mechanism proposed by Vaswani et al. (2017) is originated from the natural lan- guage processing (NLP) Transformer network. Due its superior performance, the self-attention mechanism has been widely used the NLP field. The standard self-attention formula is defined
QKT V dim
Altention[(, = softmar
(1)
The standard self-attention mechanism converts the input features linearly to obtain the variables Query (Q), Key (K), and Value (V) with the same dimension(dim), then normalizes the result from the vector dot product of the Q and K, and then obtains the potential relationship between every two features, finally uses this potential relationship to re-combine the features get the output features. Compared to recurrent neural network (RNN) (Lipton et al. 2015) series, self-attention is characterized by directly calculating dependencies regardless of distance between features and can be calculated in parallel. (Tang al.) 2018) demonstrate that the self-Attention mechanism better at Word Sense Disambiguation (WSD) (aakanRa et al./ 2017). The authors believe that the powerful semantic feature extraction ability is the key reason why self-Attention has better performance than RNN.
Most recent studies have focused on improving the efficiency of the self-attention mechanism it brings the computational complexity of O to reach a considerable performance gain. There- fore, batch of variants that pay more attention computational efficiency are proposed the

---

Under review as a conference paper at ICLR 2022
basis of original transformer backbone, such LongFormer(Beltagy et al. 2020), Reformer(Kitaev al. 2020). Transforran,,rrer, Trabsforraofo et al. (020) etc.. Through changing the feature sequence, partial self-attention, hardware acceleration, and other methods, the Transformer model can be hetter used the NLP field.
In the meanwhile, the excellent performance of the self-attention mechanism in the field of NLP has attracted great attention in computer vision. The Vision Transformer (ViT) backbone model proposed by(Dosovitskiy et al. 2020) divides the image information into patches for parallel input into the model, discards the decoder in the original transformer model, and retains its encoder as the feature extractor connected to the MLP layer to get output features. The results show that ViT is potentially competitive to the CNN network series He et al. 2016). which increasingly makes the Self-Attention mechanism popular for visual tasks.
In addition to processing input and output features, VIT obeys the conventional Transformer Block, which makes ViT have high computational complexity, and coupled with the diversity of image feature information, it makes VIT training difficult. In essence, Transformer is Sata-hungry(Khan jet al., (021), which makes it consensus to establish more efficient and extensive attention fusion image features. (Liu et al., 202T) proposes Swin Transformer, which further divides the features into several windows based on patch division. Swin introduces Non-Local neural network Wang et al., 2018) to model the relationship of each patch in windows and realizes the cross-window relationship modeling through the shift window operation. Recently, Cswin(Dong et al., 2021 abandons the shift windows method, set windows stripe, and models the horizontal and vertical relationships of features at the same time.
Even there many similar applications of Transformer the field of vision, but the works aim at optimizing the visual self-attention mechanism is relatively rare. This paper proposes novel self-attention mechanism Low-relation Mutil-head (Llf-Attention((MMA)) reduce the number of feature mapping Query and Key dimensions while maintaining the Value dimension. This ex- ploration breaks the information consistency of the traditional self-attention mechanism and greatly saves the computational complexity of single self-Attention block. The diagram of LMSA is shown in Figure which also illustrates the difference between the proposed LMSA and the newly proposed multi-head self-attention mechanism.
Specifically, we modified the self-attention mechanism based on recently proposed models such as Swin and Cswin, and we conducted a controlled experiment on the two attention mechanisms. In the Swin model, by appropriately compressing the number of dimensions, the image classification performance of the model has been further improved. In the Cswin model, we maximize the di- mensionality compression of Query and Key, while its performance on different datasets remains competitive the traditional self-attention mechanism.
More importantly, through experiments, we demonstrate that the self-attention mechanism doesn't 't have align the dimensions of Query, Key, and Value to be same. The excessively high dimen- sions of Query and Key not only cause model data redundancy but even have a negative impact on the final model performance.
QKT Vddam
Sof (Max
Mutil-
X-
Head
X
QK" Vakdim
0
SoftMax
Mutil-
X-
Head
X
-
Figure/ Comparison of LMSA and MSA operation process. Up: Standard multi-head self- attention mechanism, Down: Low-relation multi-head self-attention mechanism. qkdim represents the dimension of the Q or variable after being compressed.
2

---

Under review as a conference paper at ICLR 2022
2
RELATED WORK
2.1 MUTIL-HEAM
MACHINISM
For long time, CNN and RNN the first choice for sequence task models have the disadvantage that they cannot be calculated in parallel. The input of each subsequence depends on the output of the previous subsequence, and as the sequence length increases, it will be accompanied by long-term memory failure. The Transformer was proposed by (Vaswani et al. 2017). Transformer abandons the recurrent network structure and uses scaled dot-Product attention (see Equation 1) as its com- puting core to obtain the correlation of features at any two positions, and perform feature fusion based on this. In order to enable the model to pay more attention to the information of different representation subspaces at different positions, the author embeds the multi-head mechanism into self-attention. The formula of the multi-head self-attention mechanism is as follows:
MutilH
Concat head,, W
Where head, = Attention(QW, ,WWK, VW.") Where w.P R.dmude .. E R.amod x du WWV E Rommede xds , E I how Xdmodell
Each head uses different weight transformation, all heads are calculated in parallel, and the calcu- lation results are concatenated to obtain the final output. Adjust the output scale by controlling the dimension of Value. Generally, the dimension of V Value set to be the same as the input feature. After complete multi-head attention calculation, the input and output sizes can be guaranteed to be consistent. It is convenient to embed various networks for multi- attention.
2.2 EFFICIENT ATTENTION
The multi-head self-attention mechanism has achieved great success in the field of NLP, but it has also brought excessive complexity, especially in long text sequence tasks. Memory Compressed Transformer(Liu et al. 2018) tries to divide long text sequence into several modules of similar size, and perform self-attention within the modules, and reduce the size of the self-attention matrix through stride convolution, as to achieve the goal of improving efficiency. (Child et al.) 2019)
proposed the Sparse Transformer, which simplifies the dense attention to sparse attention through explicit selection, and improves the concentration of the model. Longformer (Beltagy et al. 2020)
which based on Sparse Transformer combines sliding window and dilated sliding window to achieve local and global Balance of attention. Axial Transformer (Ho al., (019) applies multiple atten- tion along a single axis of the input tensor. Since the length of any single axis is usually much smaller than the total number of elements, this model can significantly save computation and mem- ory. Unlike the previously mentioned Efficient Transformer, Refomer(Kitaey et al., (020) has more in-depth discussion of the self-attention mechanism itself. By designing a reversible Transformer Block, Reformer greatly saves the space used to store intermediate results and proposed that locality sensitive hashing only calculates similar feature relationships, which effectively reduces the com- putational complexity. Synthesizer (Tay et al.) changes the self-attention mechanism more boldly. directly abandoned the generation and calculation of Query and Key, and instead used random initialization or linear projection of input features directly replace the calculation results of QKT. The author concludes through experiments that this method has advantages and disadvantages with the traditional self-attention mechanism, and has affiliation with it, which makes people interested in the effective principle of the self- eelf-attention mechanism.
2.3 VISUAL TRANSFORMER
With the continuous development of Transformer the NLP field, people have also begun try to introduce the self-attention mechanism into visual tasks. After Vosovosvvis et al.) 2020) was proposed, more efficient visual Transformers have also been proposed. Among them, the most concerned are Swin Tranformer(Liu et al. 2021) and Cswin Transformer(Dong et al. 202 I). Swin Transformer realizes a more flexible local fusion of information through multi-level division of image features and then realizes information interaction between windows through shift operations. Cswin sets the windows in Swin as a bidirectional stripe with cross coas-feeture capability, which is a successful application of Axial Attention in the computer vision field. Both Swin and CSwin adopted
3

---

Under review as a conference paper at ICLR 2022
the strategy proposed by PiT (Heo al. 2021 dividing the entire model into several Layers, and down-sampling between the two layers. As the features move forward in the model, their scale decreases while their dimensions increase until the final High-Level feature is obtained to continue downstream tasks. The network structure of the Swin and Cswin model is shown in Figure 2.
H W x 3
H W x X2C
W X
H W B g
Patches Embeding
Sell-A ttention Block
Deptb-wrke
Solf-Attentionn Block
Depth- Cancolution
Solf-A ttrntion Block
Depthww/w Convolution
Self-A ttration Block
N2
N3
N4
Layer
Layer 2
Layer 3
Layer
Figure Overall structure of Swin and Cswin Transformer.
2.4 CONVOLUTIONAL VISION TRANSFORMER
Visual Tranformer's extensive research makes the Self-Attention mechanism often used to compare with convolution, Convolutional Vision Transformer(W) Wu et al., (021) groundbreakingly combines convolution and self-attention mechanism. CvT discards position embeding and uses convolutional projection instead of linear projection to generate the Q. K, and required for self-attention cal- culations, shown in Figure 3. These changes enable CvT to well inherit the advantages of CNN while maintaining the dynamic self-encoding characteristics of Transformer: translation, scaling, and distortion invariance.
Flatlen
Comolltion
Quary
Quary
Reshape
Key
Key
Tokens
Tokess
Value
Value
(a)
Figure 3: Comparison of Q, K. V generated by different self attention mechanisms. Q, K, generated by standard Self-Attention.(b) Q, K, V generated by CvT Self-Attention
METHODOLOGY
Unlike text sequence features, visual features often contain more nonobjective information. The development of Transfomer in the computer vision field shows that the self-attention mechanism can effectively extract image features, and there is still much room for improvement. However, the standard self-attention mechanism has computational complexity of O(N2). As the image size

---

Under review as a conference paper at ICLR 2022
increases, the self-attention mechanism consumes a lot more computing time and storage space, which limits the efficiency of the self-attention mechanism in visual tasks. To reduce the complexity of the self-attention mechanism and improve the computational efficiency, in this paper, propose Low-Relation Mutil-Head Self- Attentinnn,nn which aims to reduce the low-quality data that has a small effect on the calculation result in the self-attention.
3.1 LOW-RELATION MUTH.-HEAR
In order to solve the problem of the high complexity of the self-attention mechanism, a large number of Efficient Attentions have been proposed Tay et al. 2020). The most common way to increase calculation speed is control the number of Keys and Values. Associate the Query generated by the feature map with the Key whose number is much smaller than Query, and act on the scaled V Value achieve the purpose of improving efficiency by losing certain accuracy. In the visual task, the most concerned optimization method is to adjust the receptive field of the self-attention mechanism to achieve balance of accuracy and efficiency.
The above self-attention optimization methods mostly focus on adjusting the number of features used for calculation. From the perspective of feature dimensions, we try to compress the number of Query and Key dimensions and explore the impact of this compression on the accuracy of stlf-attention. the consistency of Query, K Key, and V Vluue dimensions is destroyed. We control the overall com- plexity by adjusting the dimensions of Query and K ey.. The core formula of LMSA expressed as follows:
much as possible while keeping the Value dimension unchanged
the proposed LMSA,
LMS K, V) MMiileea(((P()))C(X))
(3)
where CP(X) indicates the compress operation of X. In LMSA, still use the multi-head mech- anism to divide the feature into several heads, and each head is calculated in parallel. In the case of extreme compression, after Query and Ky are divided into several heads, each head can only retain one dimension, that is, dil In specific tasks, we need adjust the dimensions of Query and Key to meet the actual needs.
3.2 COMPLEXITY ANALYSIS
Minimizing the cost of Query and Key generation and the occupied by them is the LMSA's core to improve the overall computing power. We take dim 2 head=a as the calculation standard, and compare the optimization effects of self-attention complexity under different mapping methods, shown in Table 1. We use linear mapping or convolutional mapping to generate Query and Key, the dimensions of Query and Key are the same as heads number in the case of extreme com- pression, d and dim
which is much smaller than the standard self-attention mechanism dim and dim =64. Fewer dimensions make the calculation of the relationship matrix faster, and the complexity of the dot product between any two positions is reduced by 32 times, which significantly reduce the complexity of the self-attention mechanism. In a single self-attention calculation process, when the extreme compression mode is used, time and space consumption are reduced by more than 60 percent.
((LMSAINEAR) ((Line(()++ + + O(N2 dim,, + N dim.)
(4)
I(LMSAcONV) = D(Con((C + + dim & + N -dim,)
3.3 DATA EFFICIENCY ANALYSIS
In the standard self-attention mechanism, the dimensions of Query, Key, and Value are usually consistent, but we believe that this consistency is unnecessary. In self-attention calculations, the relationship between any two location features is derived from Query and Key. The peak of the dingle-dimensinna signal value of Query and Key has crucial impact on the final result. The number of peaks that can be extracted from an element is limited. As the dimension increases, the Query and Key not give more effective peak signals but will generate large number of
5

---

Under review as a conference paper at ICLR 2022
Table 1: Complexity Analysis Between MSA And LMSA
TOTAL
ATTENTION
PROJECTION
Q K DIM
SAVE(%)
PARAMS
SAVE(%)
OPERATION
stantard MSA
64
25354
12288
LMSA
Linear
32
16906
33.32
8192
33.33
16
12682
49.98
6144
50
LMSA(extreme mode)
2
8986
64.56
4352
64.58
stantard MSA
64
14.38M
110592
0
Convolution
32
9.58M
33.38
73728
33.33
LMSA
(kemel size:3x3)
16
7.19M
50
55296
50
LMSA(extreme mode)
2
5.09M
64.6
39168
64.58
low-quality signals. A single low-quality signal effect on the results is very limited, but the high proportion of low-quality signals will cause lot of wasted space, and make the training slow, even interfere with peak signals. Instead, we also found in experiments that too few dimensions of Query and Key will also cause the loss of key peaks, which will have a destructive effect. Properly adjusting the dimensions of Query and Key can reduce the overall complexity while maintaining the original accuracy or even improving accuracy. To explore the data efficiency of Query and Key at deeper level, we take the Swin model as example, use the standard self-attention mechanism and LMSA for training respectively, and count the data efficiency of Query and Key at different features from low-level to high-level. We use four thresholds of 0.1, 0.5, 1, and 2 to calculate the proportion of signals whose absolute value exceeds these thresholds in Query and Key to the total number of signals. As shown in Figure LMSA is more efficient than MSA.
.SAA
LMSA
Thresoki-011
MSA
0hresood:05
Data Eifieent Rale
0.55
0.
0.9
0.65
0.7
0.8
Dala Efficent
0.75
055
0.7
0.65
033
0.6
0.55
0.1
0.5
1
2
6
10
11
12
Model Depth
Model Depth
MSA
LMSA
Threshod=10
MSA LMSA
Thushold=20
Data Efficeent Rate
Eificent
0.3
022
0.2
0.1
0.1
4
5
10
11
12
5
6
10
11
12
Model Depth
Model Depth
Figure 4: Data effcient analysis of MSA and LMSA different thresholds

---

Under review as a conference paper at ICLR 2022
Table 2: Comparison of complexity between different efficient attention
EFFCIENT ATTENTION
Complexity
Memory Compressed Attention
O(N D)
Sparse Attention
O(N ONNN D)
Longformer
O(N (k+ g) D)
Axial Attention
O(N (H +W) ) D)
Refomer
O(N log(N). D)
Synthesizer
O(N2 D)
LMSA
O(N2 D..k)
3.4 COMPARISON WITH OTHER EFFICIENT ATTENTION
When using Transformer for image feature extraction tasks, in order to reintegrate feature infor- mation and reduce feature scales, is often used in conjunction with downsampling, which makes High-Level features have more dimensions, this also shows that it necessary to optimize the num- ber of dimensions for visual tasks. Table 2 lists the comparison of the computational complexity of the main Efficient Transformer, the work of dimensional optimization is very rare among them. From the perspective of the number of dimensions, LMSA can reduce the computational complexity more conveniently and effectively.
4 EXPERIMENTS
4.1 IMAGE CLASSIHICATION
In order to verify the effectiveness of the method, we conduct comparative experiments on the cur- rently common Visual Transformer to get the results. Taking into account the different mapping methods of the self-attention mechanism, we selected linear correlation self-attention models: Swin- Transformer and Cswin Transformer, and also selected convolutional self-attention models: CvT, And perform image classification training on CIFAR (Krizhekkyyyk et al. (2009) and Caltech(Griffin al., (007) datasets. Without quoting any external data for pre-training, we got the results and analyzed them. The experimental results are shown in Table 3.
Both Swin et al. 2021 and Cswin(Dong et al. 202T) adopt a similar overall architecture based on PiT et al. (021). Divide the picture into several patches for embedding firstly and then do subsequent calculations. The overall model divided into four layers. Down-sampling opera- tions assembled between layers. The layers are composed of multiple self- Blocks and MLPs. The overall architecture shown in Figure 2. We replaced all internal MSAs with LMSA and got the modified model.
We train on the CIFAR dataset firstly. CIFAR-10 10 and CIFAR-100 are subsets of the 80 million labeled tiny image datasets, and their image size is 32x32. Because the image size is small enough, the Patches Embedding stage of the model is replaced by Conv 1x1, which can preserve the image information to the maximum extent and adapt the model. We have followed the Cswin training strategy, using the MSA and LMSA methods to train 300 epochs on both the Swin and CSwin models. We set to 0.05. The warmup GGoyal et al., (017)) method is used to initialize the learning rate and the learning rate set to 0.0005, and the Cosine Annealing Loshchilov & Hutter, 2016) method used to adjust the learning rate. In order to prevent overfitting, we added Dropout with a probability of 0.1 inside each Transformer Block. In data processing, we combined MixUp(Zhang et al. (017)) and CutMix Yun et al. 2019) data augmentation methods, and set label smoothing (Szegedy et al. 2016) to O. Finally, we use soft target cross cross-entropy to calculate the loss between the label and the prediction result.
AdshwoLoshchilov & Hutter 2017) the optimizer, and the weight decay is
In order to further verify that LMSA can adapt well to datasets with different digital features, we also conducted experiments the Caltech dataset. The 1altech101 and Caltech-256 datasets have the characteristics of uneven distribution of categories, and their image sizes are not fixed, which can better test the performance of the model. In data processing, we fixed the pictures to 224x224 by

---

Under review as a conference paper at ICLR 2022
Table 3: Comparison of image classification results between MSA and LMSA
Model
Model
Dataset
Method
Q or K. Dim
Params(M)
Acc(%)
Size(M)
MSA
[64,128,256.51]
21.8
262.1
95,25
CIFARIO
LMSA(extreme)
[2,4,8,16]
18.65
224.4
95.24
MSA
(64,128,256.512)
21.85
262.6
80.28
CIFARIO0
LMSA(extreme)
[2,4,8.16]
18.69
224.5
80.09
CSwin
MSA (64,128,256.512]
21.85
262.7
73.62
Caltech101
LMSA(extreme)
12,4,8,16]
18.7
225
73.63
MSA (64,128,256.512)
21.86
263.7
70.3
Caltech256
LMSA(extreme)
[2,4.8,16]
18.7
226
71.71
MSA (96,192.384,7)) 7688
27.52
331.1
93.21
CIFAR10
LMtA(extreme)
[3,6,12.24]
23.33
280.9
93.64
Swin
MSA
196,192.384, 768]
27.59
331.9
77
CIFAR100
LMSA
148,96,122.388
25.43
306.1
77.38
LMSA(extreme)
[3,6.12,24]
23.40
281.7
75.46
MSA
[64,192,384]
61.4
234.5
85.36
CIFARIO
LMSA(extreme)
[1,3,6]
30.8
117.6
85,24
CvT
MSA
664,
164,192.344
61.4
246.1
63.83
CIFARI00
LMSA(extreme)
[1.3,6]
30.8
123.5
63.48
random cropping method, set the patch size for Patches Embedding, obtained 56x56 features, and input them into the network for training.
CvT changes the linear correlation in self-attention to 3x3 convolution, which makes it necessary to test LSMA on CvT. We initialize the learning rate to 0.02 and train 300 epoch with cosine learning rate decay scheduler.
On the CIFAR-10 10 dataset, using the Q, extreme compression strategy, the performance CSwin and CvT with LSMA is basically the same with MSA, and the number of parameters has been greatly reduced. Especially CvT, both the parameter number and the final model size are reduced by more than 50 percent. In the Swin model, using extreme compression LMSA even achieved higher accuracy than MSA, which shows that controlling the proportion of low-quality data is con- ducive model convergence. On the CIFAR-100 dataset, LMSA extreme compression brings slight accuracy loss. We also appropriately increase dimensions, for example, on Swin, we only perform dimension compression by times, which also exceeds MSA.
We are also training on the Caltech dataset, especially on the Caltech-255 saaansttsslf-atteniiie with extreme compression shows stronger performance than MSA, which further shows that self aelf-attention data redundancy is harmful to the model.
4.2 SEMANTIC SEGMENTATION
In addition to image classification tasks, Transformer can also be well applied other tasks, such as semantic segmentation. Segformer et al.) 2021) is simple, efficient, and powerful semantic segmentation framework. It combines Transtormer with MLP, uses Self-Attention mechanism as a feature extractor improve the effective receptive field(Luo et al. 2016), and uses MIP as a feature decoder. Segfomer also uses Efficient attention to control the complexity of the overall self-attention mechanism by establishing one-to- many relationship between Query and Key. We test the performance LSMA on ADE20K (Zhou et al. 2017) without pre-training by introducing to Segformer. The results are shown in Table 4. The Segformer using LMSA has the same effect as the original Segformer, which also shows that LSMA can be used in conjunction with other Efficient Attention achieve better results.
4.3 ABILITION STUDY
We design ablation experiments based on the Swin model, starting with Standard self-attention, gradually compressing the and K dimensions of self-attention, using the same hyperparameter
8

---

Under review as a conference paper at ICLR 2022
Table 4: Comparison of semantic segmentation results between MSA and LMSA
Model
Method
Q or K Dim
mloU(%)
mACC(%)
MSA
[62,64,100.256
22.97
31.74
SegFormer
224,48,120,199
22.95
31.82
LMSA
[16,32,80,128]
22.87
31.79
settings for CIFAR-100 image classification training, and observing the results. We compressed 2, 4. 8, 16, and 32 times (extreme) respectively. We found that the compression of the model is accompanied by an increase in data efficiency. After times and times compression, the final performance of the model has been improved. But after 16,32 times compression, the accuracy of the model is destroyed. This shows that it not that the lower the dimensionality. the more beneficial the model. Too low Query and Key dimensionality will cause high-quality signal loss and affect the overall performance of the model.
IxC (ooopression(MA))
2x MSA)
77.5
MxAL
8c MSA)
32x(LMSA extreme)
77
A0CP%)
76.5
Dota Efficient Rato(%)
0.8
76
0.6
75.5
75
0.2
74.5
1/2
1/4
1/8
1/16
1/32
3
9
12
Compression Ratio
Model Depth
Figure Effect of different compression ratio on results.Cifar-100 used to experiment on Swin transformer without pre-training Let.ing.Leffnnnfeeeneee. of different compression ratio on accu- racy.Right: effective rate under different compression rates
5 CONCLUSION
This paper proposes novel variant of Efficient Attention: LMSA, which limits the number of signals that generate relational matrices, improves the quality of Query and Key signals and greatly reduces the computational consumption of MSA. Experiments have verified that LMSA be equal to MSA in accuracy and even has advantages. This brings the confidence to LMSA instead of MSA embed various types of Transformers. We demonstrate that dimensional consistency is unnecessary in self-attention calculations, choosing an appropriate LMSA can achieve competitive results while saving computing resources.
REFERENCES
Iz Beltagy, Mathew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arX1v:200.055550, 2020.
Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint urXiv: 10509, 2019.
Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. arXiv preprint arXiv:2107.00652. 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An

---

Under review as a conference paper at ICLR 2022
image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11922 2020.
Jiarui Fang, Yang Yu, Chengduo Zhao, and Jie Zhou. Turbotransformers: An efficient gpu serving system for transformer models. urXiv preprint arXiv:2010.05680, 2020.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arViv preprint ar0iv:1706..0 02677, 2017.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Scong Joon Oh. Rethinking spatial dimensions of vision transformers. arXiv preprint arXiv:2103. 16302, 2021.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidi- mensional transformers. arXiv preprint arXiv: 12100 2019.
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and
Mubarak Shah. Transformers in vision: A survey, arXiv preprint arXiv:2101.01169, 2021.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint wrXiv:2001.0445/, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Zachary C Lipton, John Berkowitz, and Charles Elkan. A critical review of recurrent neural networks for sequence learning. arXiv preprint arXiv: 1506,000/9, 2015.
Peter Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. arXiv preprint arXiv:1801.10199, 2018.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint 1rX1v:2103.140300 2021.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.0399 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 171 2017.
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 4905-4913, 2016.
Alessandro Raganato, Jose Saimacho-Collados and Roberto Navigli. Word sense disambiguation: A unified evaluation framework and empirical comparison. In Proceedings the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume Long Papers, pp. 99-110, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826. 2016.
Gongbo Tang. Mathias Müller, Annette Rios, and Rico Sennrich. Why self-attention? a targeted evaluation of neural machine translation architectures. arXiv preprint arxiv: / 1808 08946, 2018.
10

---

Under review as a conference paper at ICLR 2022
Y Tay, D Bahri, D Metzler, D Juan, Z Zhao, and C Zheng. Synthesizer: Rethinking self-attention in transformer models. arxiv 2020. arXiv preprint arXiv:2005.00743.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey, arXiv preprint wiX::i000000673: 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 1999-60088 2017.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7794-7803 2018.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103. 15808, 2021.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. arXiv preprint arXiv:2105. 15203, 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 2017.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 633-641, 2017.
11


Reviews:
[{"review_id": "VpMjMroGFK", "reviewer": "Reviewer_S5RZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe paper is simple, easy to understand, and applicable to most self-attention operators.\n\nWeaknesses:\n1. Technical novelty is very limited. Using fewer channels somewhere in a neural network can hardly be a contribution to the community. Furthermore, [1] searches for the best number of channels for query and key.\n2. Experiments are done mostly on very small datasets and the performance is usually worse with the proposed LMSA. The conclusion may not be scalable.\n3. Data efficiency usually refers to training with fewer data points/samples, but in section 3.3, the authors measure something like activation norm/peak/temperature (no equation is given) and refer to it as data efficiency. However, it is expected that two models with different query/key channels can have different norm/peak/temperature, even at random initialization.\n\n[1] Neural Architecture Search for Lightweight Non-Local Networks. CVPR 2020.", "summary_of_the_paper": "The authors propose to use fewer channels for query and key in multi-head self-attention and claim that the choice is better for CIFAR, Caltech, etc.", "summary_of_the_review": "I think this paper is not good enough mainly due to the very limited novelty and results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Rd0n9nSicMh", "reviewer": "Reviewer_kY4w", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\nThe proposed idea is simple and technically soundable. The writing is good and the organization is easy to follow. They also integrated  LMSA with various existing transformer-based networks and evaluated on several benchmarks. \n\nCons:\n1. Novelty & Significance are limited. The proposed idea is too natural and straightforward, while similar techniques (or tricks) has already been used in many existing approaches/practices. So I did not think the novelty/significance of this work meets the accepted criteria\n2. The experimental validation was only conducted on the weak baseline or small datasets. For image classification, the validation on ImageNet is missed, and the reported performance on semantic segmentation is too weak. These weaknesses make it is hard to justify the effectiveness of LMSA.\n3. The self-attention itself may not be the bottleneck of many existing transformer-based self-attention, such as Swin & CSwin. In fact, the FFN is the bottleneck. Therefore, I really wonder if the proposed method could accelerate the inference speed, and it is not reported in the paper.", "summary_of_the_paper": "This paper presents a mechanism to reduce the computation costs of a standard self-attention module, named LMSA. \nThe basic idea of LMSA is to reduce the dimension of key&query of self-attention(SA) while keeping the dimension of value unchanged. Therefore, the computational complexity of SA will be reduced from O(N^2D) to O(N^2D_q,k). \nThey study the proposed mechanism on image classification and semantic segmentation.\n", "summary_of_the_review": "This work lacks novelty and significance. The experiments also have many drawbacks, such as only validating on the weak baseline or small dataset.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "9vNwH-Ijrzc", "reviewer": "Reviewer_ZniF", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This paper is well-motivated since self-attention computation is the critical issue for Transformer.\n2. The experimental results demonstrate the effectiveness of the proposed method.\n\nWeaknesses:\n1. novelty concerns:\n\n(a) The main issue is the novelty. This paper mainly proposes to reduce the dimensions of Query and Key. However, this looks like a hyperparameter tuning for the dimension numbers unless the paper proposes a novel way to compress the dimension, which is missing in this paper.\n\n(b) Moreover, the spatial-wise computation is the main bottleneck for self-attention. This paper does not address this issue. Table 2 shows that the complexity is still O(N^2). It would be much better to explain this or provide proof showing that spatial-wise computation is not an issue for the problem that the paper is addressing.\n\n\n2. claim concerns:\n\n(a) The paper claims that “the works aim at optimizing the visual self-attention mechanism is relatively rare”, but it is not true. For example, there are papers working on changing computing orders of Queries, Keys, and Values, inside self-attention, like ET [1], LambdaNetworks [2]. Some papers work on reducing spatial computation of self-attention, such as HaloNet [4]. Moreover, LRA [3] proposed a new benchmark including vision tasks to investigate several state-of-the-art self-attention mechanisms.\n\nReferences:\n\n[1] Zhuoran Shen, et al. \"Efficient Attention: Attention with Linear Complexities\", WACV, 2021.\n\n[2] Irwan Bello. \"LambdaNetworks: Modeling long-range Interactions without Attention\", ICLR, 2021. \n\n[3] Yi Tay, et al. \"Long Range Arena: A Benchmark for Efficient Transformers\", ICLR, 2021.\n\n[4] Ashish Vaswani, et al. \"Scaling Local Self-Attention for Parameter Efficient Visual Backbones\", CVPR, 2021.\n\n(b) The paper claims that “we demonstrate that the self-attention mechanism doesn’t have to align the dimensions of Query, Key, and Value to be same”. However, this result is not surprising. Adjusting dimension numbers for Query, Key, and Value is the way to find the trade-off between computation and accuracy.\n\n\n3. technical detail concerns:\n\n(a) It is not clear why the paper chooses to reduce the dimensions of Query and Key instead of Key and Value like most previous work. It would be great to provide more analyses to support this decision.\n\n(b) The paper only mentions CP(X) is the compress operation of X, but does not describe technical details of CP(X). \n\n\n4. experiment concerns:\n\n(a) The experimental results shown in Tables 3 and 4 do not show significant improvement of LMSA over MSA. Moreover, there are no experiments on ImageNet like most previous work does, which makes the experimental results not convincing.\n", "summary_of_the_paper": "This paper proposes an efficient self-attention mechanism which compresses the channel numbers of features, to reduce the computation for the Transformer.", "summary_of_the_review": "The proposed method lacks novelty, and the technical details of the proposed method are missing. Moreover, the proposed method is not evaluated on a common benchmark dataset (i.e., ImageNet) and the results are not impressive. Therefore, I recommend rejecting this paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a simple change to Transformer architecture to improve efficiency. While the reviewers appreciate the writing, all the reviewers agree that the novelty and contributions of the paper are limited both in the problem being solved by the paper and the level of experiments in it. Authors did not respond to reviewer's comments. Hence I recommend rejection.
Decision: Reject