## 2 Related Works
### Transformer Models in Computer Vision  
A paradigm change has occurred as a result of the introduction of transformer models--originally developed for natural language processing--to the field of computer vision. The principles of attention and multi-head attention described in the original transformer paper (presented in Figure 1) must be understood before delving deeply into how vision transformers function. In [2], the transformer model is put forth. Three variables are used in the transformer's attention mechanism: K (Key), Q (Query), and V (Value). As inputs, the sets of key-value pairs with query vectors are used. The softmax operator is used to calculate the output vector after a weighted sum of the values is determined; the weights are determined using a scoring function (Equation (1)). \[\text{Attention}(\text{Q},\text{K},\text{V})=\text{softmax}\left(\frac{\text {Q}\cdot\text{K}^{\text{T}}}{\sqrt{\text{d}_{\text{k}}}}\right)\cdot\text{V} \tag{1}\]  
where V, Q, and K\({}^{\text{T}}\) are value, query, and transposed key matrix, respectively. The scaling factor is \(1/\sqrt{\text{d}_{\text{k}}}\), and d\({}_{\text{k}}\) represents the key matrix's dimensions. The multi-head attention mechanism is defined using the Q, K, and V calculation as a single head. Q and K are used by the (single head) attention mechanism in the preceding diagram. Figure 1: The attention layer’s structure. Scaled dot-product attention is on the left, multi-head attention mechanism on the right. However, in the multi-head attention mechanism, each head has its own projection matrix, and they use the feature values projected using these matrices to determine the attention weights. \[ \begin{split}\text{Multihead}\left(\text{Q}^{\prime},\text{K}^{ \prime},\text{V}^{\prime}\right)=\text{Concat}\left(\text{head}_{1},...,\text{ head}_{\text{h}}\right)\text{W}^{\text{O}}\\ \text{where}\;\text{head}_{\text{i}}=\text{Attention}\Big{(} \text{Q}\text{W}_{\text{i}}^{\text{Q}},\text{K}\text{W}_{\text{i}}^{\text{K}},\text{V}\text{W}_{\text{i}}^{\text{V}}\Big{)}\end{split}  \tag{2}\]  
where, \(\text{Q}^{\prime},\text{K}^{\prime},\text{V}^{\prime}\) are the projection matrix; \(\text{W}_{\text{l}\_h}^{\text{O}}\in\mathbb{R}^{\text{D}\times\text{d}_{\text {k}}}\), \(\text{W}_{\text{l}\_h}^{\text{K}}\in\mathbb{R}^{\text{D}\times\text{d}_{\text {k}}}\), \(\text{W}_{\text{l}\_h}^{\text{V}}\in\mathbb{R}^{\text{D}\times\text{d}_{\text {v}}}\) and \(\text{W}^{\text{O}}\in\mathbb{R}^{\text{h}\cdot\text{d}_{\text{v}}\times\text {d}_{\text{out}}}\) are trainable weight matrices; D is the size of the embedding vector of each input element from our sequence; \(\text{d}_{\text{k}}\) and \(\text{d}_{\text{v}}\) are the inner dimensions of each self-attention layer, and h is the number of heads. Multi-head attention is theoretically based on the ability to consistently pay attention to different sequence pieces in diverse ways. The model can better collect location information as a result because, in practice, each head will focus on a separate input segment. We will have a stronger representation as a result of their combination. Each head will also uniquely correlate words to gather various pieces of contextual information. A turning point was reached with the development of the vision transformer (ViT) [3], which highlighted the potential of transformer designs in image categorization. With this method, each patch of an image is processed by a layer of a transformer. This novel viewpoint cleared the way for the successful representation of global contextual information in images. The following are some significant application areas for vision transformers, which are widely used in common computer vision tasks: image classification, segmentation, object detection, and cluster analysis. CrossViT was suggested by the authors in [4] for image classification. Excellent teacher-guiding small networks (ES-GSNet) were suggested by the authors in [5] for the classification of remote sensing image scenes.