## 2 Methods
### Methodology
#### 2.2.3 Improved Vision Transformer Module  
1. Classical Transformer model  
The transformer model was initially proposed for natural language processing by the team at Google [34]. Recently, the Transformer model was successfully adapted for visual tasks. The key innovation of the transformer model is a self-attention mechanism, which enables parallel training and facilitates the capture of global information. When applied to visual data, this architecture can effectively process image information in a manner analogous to its success in NLP tasks. In the transformer model, there are two main components: the encoding component consists of multiple encoder layers, while the decoding component consists of an equivalent number of decoder layers. In order to address the sequence-related challenges at both the input of the encoder and decoder, locational encoding information is added. The position is encoded with sine and cosine functions as follows:  
\[P(p,2j)=\sin\left(\frac{p}{10000^{\frac{2j}{2}}}\right),\ P(p,2j+1)=\cos\left( \frac{p}{10000^{\frac{2j}{2}}}\right) \tag{3}\]  
where \(P\) represents the position of the sequence of words in the sentence, \(j\) represents the dimensional position of the vector, and \(d_{\text{model}}\) represents the size of the vector dimension. The self-attention mechanism plays a pivotal role by simulating a human in the transformer model cognitive behavior, dynamically focusing on specific regions. The attention  
Figure 2: Complemented attention module. mechanism comprises two essential components: The first component is the transformation layer, which performs linear transformations on the input sequence information \(\mathbf{X}\in\mathbb{R}^{n_{x}\times d_{x}}\), \(\mathbf{Y}\in\mathbb{R}^{n_{y}\times d_{y}}\) to three distinct sequence vectors, respectively, with linear transformation. These three vectors are called query vector Q, key vector K, and value vector V. Here, \(n\) and \(d\) are the input sequence length and dimension, respectively. The self-attention mechanism is manifested with the following:  
\[\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{Softmax}\bigg{(} \frac{\mathbf{Q}\cdot\mathbf{K}^{T}}{\sqrt{d_{k}}}\bigg{)}\cdot\mathbf{V} \tag{4}\]  
where \(\mathbf{Q},\mathbf{K},\mathbf{V}\) are the three input matrices, and their dimensions are \(d_{q},d_{k},d_{p}\). The limitation of the single-headed attention layer is that it can only focus on a specific location and cannot focus on other important locations at the same time. To solve this problem, the multi-head attention mechanism uses \(\mathbf{Q},\mathbf{K},\mathbf{V}\) to calculate multiple information selected in parallel from the input information for linear projection, and each attention is focused on a different part of the input information and output the content. Finally, the output values are again concatenated and projected to generate the output value. The multi-head attention mechanism is presented as follows:  
\[ \begin{split}\text{MultiHead}(\mathbf{Q},\mathbf{K},\mathbf{V})= Concat(head_{1},\dots,head_{n})\mathbf{W}^{0}\\ head_{i}=\text{Attention}(\mathbf{Q}\mathbf{W}_{i}^{\text{d}}, \mathbf{K}\mathbf{W}_{i}^{\text{e}},\mathbf{V}\mathbf{W}_{i}^{\text{e}}) \end{split}  \tag{5}\]  
where \(\mathbf{W}^{\text{e}}\), \(\mathbf{W}^{\text{e}}\), and \(\mathbf{W}^{\text{e}}\) are parameter matrices for linear projection, n is the head number, and \(\mathbf{W}^{0}\) is the projection weight. (2) Improved vision Transformer module  
Vision transformers (ViT), which is a transformer model for computer vision, substitute transformer structures for convolutional structures. Suppose the input feature map is \(\mathbf{X}\in\mathbb{R}^{H\times W\times C}\), and it is subdivided into \(N\) two-dimensional patches with a size of \(P\times P\). Then the length of the transformer input sequence is \(N=HW/P^{2}\). \(N\) patches are projected onto a \(D\)-dimensional space with a projection matrix. Its output is called patch embedding.