## 4 Experimental Results
### Results  
In this section, we will discuss the results of our evaluation and comparison of 13 different vision transformer (ViT) models on the facial expression recognition RAF-DB and FER2013 datasets, as well as our Balanced FER2013 dataset. Our process involved training the model on a portion of the dataset, testing it on a validation set, and then reporting its accuracy on the test set. Before we describe the specific aspects of the model's performance on various datasets, we will briefly outline our training process. We ensured consistency in the architecture and hyperparameters of each model and trained a single model for each dataset. For all experiments, we utilized a workstation featuring an Intel i7 CPU with 32 GB of RAM and an Nvidia GTX 3090 GPU with 24 GB of memory. In this study, each vision transformer model was trained for 25 epochs from scratch to develop the models. To optimize the model, we applied an Adam optimizer with a learning rate of \(3\times 10^{-5}\). In order to ensure that the model was trained on a diverse range of images and could handle minor variations, we applied data augmentation to the images in the training sets. This involved flipping, zooming, rotating, and distorting the images to enhance the quality of the data. We measured each model's performance in terms of recognition accuracy. The following Table 1 provides an overview of the accuracy results obtained:  
The Tokens-to-Token ViT model outperforms all other models on the RAF-DB dataset, achieving accuracy values of 76.40%. With high accuracy rates across all three datasets, Mobile ViT, PiT and Cross Former models also produce great resu \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **RAF-DB** & **FER2013** & **Balanced FER2013** \\ \hline Base ViT [3] & 68.34 & 49.64 & 60.25 \\ CrossViT [4] & 69.74 & 50.27 & 62.43 \\ CaiT [24] & 70.45 & 45.68 & 60.15 \\ ViT for Small-Size Datasets [25] & 72.37 & 55.35 & 67.88 \\ Deep ViT [26] & 63.25 & 43.45 & 50.37 \\ Cross Former [27] & 72.47 & 59.95 & 75.12 \\ Tokens-to-Token ViT [28] & 76.40 & 61.28 & 74.20 \\ Parallel ViT [29] & 67.16 & 50.94 & 64.40 \\ LeViT [30] & 65.71 & 47.22 & 60.85 \\ Early ConViT [31] & 68.24 & 51.02 & 66.70 \\ Mobile ViT [32] & 74.28 & 62.73 & 77.33 \\ Region ViT [33] & 69.62 & 56.03 & 73.79 \\ PiT [34] & 72.84 & 58.67 & 76.09 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Classification accuracy (%) of different vision-transformer models on facial expression recognition datasets. Figure 9: Class-wise image distribution in the FER2013 dataset: (**a**) the image distribution per class in the original FER2013 dataset; (**b**) the image distribution for the FER2013 balanced dataset. Parallel ViT models underperform on all three datasets. They may not be suitable for FER, as they were developed for other tasks and may not work efficiently with small databases. Further observations indicate that the models perform insufficiently on the FER2013 dataset compared to the RAF-DB dataset. The reason for this could be the challenging nature of the images in the FER2013 dataset, which exhibit higher levels of noise and occlusion. However, on the Balanced FER2013 dataset, the models perform better than on the FER2013 dataset. This is because the Balanced FER2013 dataset has an equal number of images for each expression class, while the FER2013 dataset is imbalanced. In Figure 10, one can compare each emotion category's accuracy with the Tokens-to-Token ViT model's application to the FER2013 and FER2013_Balanced datasets. As can be seen from the following categories, the model achieves high accuracy when using the FER2013_balanced dataset. The analysis covers a broad range of emotions including anger which increased from 47.3% to 73.4%, disgust which improved from 43.5% to 91.2%, fear which rose from 33.5% to 79.1%, and neutral which advanced from 60% to 79%. However, it is also important to acknowledge a few negative outcomes in the categories of surprise, sadness, and happiness.