# Facial Expression Recognition Based on Vision Transformer with Hybrid Local Attention
## 4 Experimental Results
### Visual Analysis  
Visualization techniques in experiments are also essential, and attention visualization enables a clearer intuitive view of the effectiveness of the local attention module and the global self-attention in ViT. Figure 4 shows the attention maps under occlusion conditions. The first row is the original input images, the second row is the visualized images of the baseline method, and the third row is the visualized images of the proposed method HLA-ViT in this paper. The input images are the facial images selected from the dataset with occlusion. It can be observed that the utilization of the local attention module guides the network to focus attention on locally significant areas. These regions are crucial for the accurate recognition of facial expression categories. Compared with the baseline, the method proposed in this article can better focus on areas useful for facial expression recognition, such as eyes, mouth, and nose, under occlusion. For example, in the first column, where the mouth of the face in the original image is occluded, the baseline method concentrates attention on the nose and cheek regions while ignoring the eye regions. In contrast, our method directs attention to the eyes and nose, which are crucial for accurate expression recognition. Similarly, in the fourth column, where a person's hand covers a significant portion of the right side of the face in the original image, the baseline method fails to capture effective attention regions. However, our method adeptly focuses on the uncovered side of the face, specifically the eyes, nose, and mouth regions. The attention maps in the fifth and seventh columns also illustrate this issue. Results from the second, third, and sixth columns reveal that the baseline method tends to treat the occluded regions as interesting regions in cases of partial facial occlusion, leading to the introduction of useless information and, ultimately, obtaining incorrect recognition results. The combination of the hybrid local attention module and global attention in HLA-ViT enables the network to focus on unoccluded areas, \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Method** & **HLA** & **ViT** & **RAF-DB** & **FERPlus** & **AffectNet** \\ \hline Baseline & \(\times\) & \(\times\) & 87.22\% & 87.14\% & 61.02\% \\ b & \(\times\) & \(\surd\) & 90.16\% & 88.14\% & 63.84\% \\ c & \(\surd\) & \(\times\) & 89.93\% & 88.23\% & 62.34\% \\ HLA-ViT & \(\surd\) & \(\surd\) & 90.45\% & 90.13\% & 65.07\% \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Ablation study. aligning with human perception. This further demonstrates the robustness of HLA-ViT in the presence of facial occlusion. Figure 5 focuses on the face images with variations in head pose, specifically selecting images with a lateral tilt of approximately 45\({}^{\circ}\). The first row presents the original images, the second row displays results from the baseline method, and the third row shows the attention maps obtained through visualization using the proposed HLA-ViT method. From the images, it can be observed that, in the presence of variations in head pose, the baseline method sometimes misses the regions of interest that require attention, as shown in the first, third, and fourth columns. It may also overlook crucial areas for facial expression recognition, as seen in the sixth column where the mouth and eyebrows are essential for recognizing expressions. The baseline method neglects these areas, whereas our proposed approach utilizes the hybrid local attention module to concentrate more attention on the mouth and eyebrow regions. By leveraging the information conveyed by these focused areas, the network is better equipped to recognize and classify facial expressions. Compared with the original image, it can be seen that the use of the local attention module can guide the network to focus on locally significant regions that are crucial for the correct recognition of expression categories.