## 3 Analysis of Vision Transformer Models
### Vision Transformer Models  
#### 3.1.1 Base Vision Transformer (ViT)  
An innovative model that expands the transformer architecture to handle image classification problems was introduced by [PERSON] et al. [3] in 2020. ViT takes a distinctive approach to image processing, much like its equivalent in natural-language processing. The typical transformer model used a one-dimensional sequence of word embeddings as input due to the model design, which was based on NLP. On the other hand, when the transformer model is used for the problem of image classification in CV, the input data is given to it as two-dimensional images. Figure 2 presents an illustration of the main ViTs. Figure 2: Architecture of the base vision-transformer model. Divided into smaller two-dimensional patches, the input image, with height, width, and number of channels, is used to structure the input image data in a manner consistent with the input's structure in the NLP domain. As a result, there are \(N=\frac{HW}{P^{2}}\) patches, with each patch having a resolution of (P, P) pixels. The following operations are carried out prior to feeding the data into the transformer encoder:  
* Every picture patch is flattened into an \(x_{p}^{n}\) vector of length \(P^{2}\times C\), where \(n=1,\ldots N\). * Mapping the flattened patches to D dimensions using a trainable linear projection, E, produces a series of embedded picture patches. * The classification output, y, is represented by the learnable class embedding, \(x_{\text{class}}\), which prefixes the list of embedded image patches. * Finally, positioning information, which is also learned during training, is added to the input by augmenting the patch embeddings with one-dimensional positional embeddings \(\text{E}_{\text{pos}}\). The embedding vectors that arise from the aforementioned procedures are as follows:  
\[\text{z}_{0}=\left[x_{\text{class}};x_{p}^{1}\text{E};\ldots;x_{p}^{N}\text{E} \right]+\text{E}_{\text{pos}} \tag{3}\]  
where, \(\text{E}_{\text{pos}}\) is the positional embedding matrix, \(x_{p}^{n}\) is a flattened vector of image patches, and \(x_{\text{class}}\) is the class embedding vector. At the input of the transformer encoder, a stack of L identical layers \(z_{0}\) is fed for classification. They then proceed to feed a classification head with the value of \(x_{\text{class}}\) at the \(L^{\text{th}}\) layer of the encoder output. It is crucial to understand the transformer encoder and its essential components. These are the parts of the transformer encoder:  
* Multi-head self-attention (MHSA) layer: By using multiple \"heads\", the ViT model can simultaneously focus on different segments of an image. Each head calculates attention independently, allowing for a variety of image representations to be generated. These representations are then combined to create a final image representation. This approach allows the model to capture more nuanced interactions between input components. However, this also makes the model more complex and computationally expensive due to the need to aggregate the outputs from all the heads. Within the images in Figure 3, we offer insights into the visualization of the attention mechanism in a vision transformer. We explore different numbers of heads in the multi-head self-attention layer. The upper-left corner displays the original image and its attention visualization using the mean value of the heads. Meanwhile, the lower part of the image showcases visualizations with varying numbers of heads in the multi-head self-attention layer. Based on Figure 3, it is evident that as we increase the number of heads in the MHSA layer, the model's ability to identify interrelated objects in our dataset improves. However, it is worth noting that using a large number of heads can also negatively impact the accuracy of the model. Therefore, in most cases, it is crucial to determine the optimal number of heads for each model. In Figure 3, the ideal choice would be to set the number of heads to 4.