## 2 Related Work
### Deep Learning Visual Model Architecture
#### 2.4.2 Transformer  
The Transformer was proposed by [PERSON] et al. [33] and has achieved great success in the field of natural language processing. Especially in machine translation tasks, it has become an important benchmark model. The model is based on the self-attention mechanism. By weighted aggregation of information at different positions in the input sequence, it realizes the modeling of long-distance dependencies, thus better capturing the semantic information in the sequence. This model has excellent global feature extraction ability. It can extract features for upstream tasks through pre-training and be applied to different transfer learning tasks. In the field of computer vision, research has found that the Transformer can achieve performance comparable to that of CNN. At the ICLR 2021 conference, Vision Transformer (ViT) proposed by [PERSON] et al. [34] successfully applied the Transformer model to image classification tasks. The ViT model adopts a brand-new image representation method. It divides the input image into small image patches, converts each image patch into a sequence form, and then inputs these sequences into the Transformer encoder for processing. Currently, the Transformer has achieved remarkable results in both natural language processing and visual classification [35; 36], detection [37; 38], and segmentation [39; 40]. As shown in Figure 6, on the left is the encoder of the Transformer. The encoder is stacked by multiple encoder layers with the same structure. Each encoder layer includes a multi-head attention mechanism and a feed-forward neural network. The encoder is responsible for mapping the input sequence to a high-dimensional feature representation space. On the right is the decoder architecture. The decoder is stacked by multiple decoder layers with the same structure. Compared with the encoder, in addition to the multihead attention mechanism and the feed-forward neural network, each decoder layer also has an additional encoder-decoder attention mechanism. The decoder is responsible for mapping the feature representation output by the encoder to the target sequence. The main components of the Transformer model include the following:  
1. Attention mechanism. The Transformer model mainly adopts the multi-head self-attention mechanism. The calculation of self-attention requires Query, Key, and Value. After obtaining the values of QKV, the output can be calculated: \[Attention(Q,K,V)=Softmax\Big{(}\frac{QK^{T}}{\sqrt{d_{k}}}\Big{)}V,\] (2) where \(d_{k}\) represents the vector dimension, that is, the number of columns of \(Q\) and \(K\). To prevent issues such as gradient vanishing or gradient explosion caused by overly large inner products, the result of the dot-product between the query and the key is divided by \(\sqrt{d_{k}}\). The multi-head attention mechanism is composed of multiple self-attention mechanisms. The input sequence is first mapped to the representations of query, key, and value through linear transformation, and then the calculations of multiple attention heads are performed in parallel, respectively. 2. Feed-forward neural network. It usually consists of two linear transformation layers and an activation function. These two linear transformation layers are called fully connected layers or hidden layers, and there is no limit to the number of hidden layers. An activation function is usually added between each hidden layer, and the most commonly used activation function is the ReLU function. The role of the feed-forward neural network is to perform non-linear transformation and feature extraction on the representation of each position, thereby helping the model better understand  
Figure 6: Transformer structure [33]. and process the information of the input sequence and improving the generalization ability of the model. The calculation process of the feed-forward neural network [33] is as follows.