# Facial Expression Recognition Based on Vision Transformer with Hybrid Local Attention
## 4 Experimental Results
### Ablation Study  
Since HLA-ViT is composed of a hybrid local attention module (HLA) and ViT module after extracting features from the backbone, to verify the effectiveness of the modules, abla  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Methods** & **Year** & **RAF-DB** & **FERPlus** & **AffectNet** \\ \hline SCN [27] & CVPR 2020 & 87.03 & 89.39 & - \\ PSR [28] & CVPR 2020 & 88.98 & - & 63.77 \\ RAN [29] & TIP 2020 & 86.90 & 89.16 & - \\ DMUE [30] & CVPR 2021 & 89.42 & - & 63.11 \\ FDRL [31] & CVPR 2021 & 89.47 & - & - \\ FER-VT [32] & 2021 & & 90.04 & \\ Meta-Face2 Exp [33] & CVPR 2022 & 88.54 & 88.54 & 64.23 \\ SPWFA-SE [34] & IEEE Trans 2023 & 86.31 & - & 59.23 \\ RCL-Net [35] & 2023 & 88.2 & 89.53 & - \\ \hline Ours & - & 90.45 & 90.13 & 65.07 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison results with SOTA methods on RAF-DB, FERPLUS, and AffectNet.  
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline
**Dataset** & **Method** & **Angry** & **Disgust** & **Fear** & **Happy** & **Sad** & **Surprise** & **Neutral** &
\begin{tabular}{c} **Mean** \\ **Class Accuracy** \\ \end{tabular} \\ \hline RAF-DB & SPWFA-SE [34] & 80 & 59 & 59 & 93 & 84 & 88 & 86 & 78 \\ RCL-Net [35] & 86 & 57 & 57 & 95 & 86 & 88 & 89 & 79 \\  & Ours & 89 & 79 & 73 & 96 & 82 & 87 & 93 & 86 \\ \hline FERPlus & RCL-Net [35] & 84 & 46 & 61 & 97 & 76 & 93 & 91 & 78 \\  & Ours & 86 & 56 & 62 & 96 & 81 & 89 & 91 & 80 \\ \hline AffectNet & Face2 Exp [33] & 57 & 56 & 64 & 87 & 66 & 56 & 62 & 64 \\  & SPWFA-SE [34] & 35 & 59 & 49 & 95 & 59 & 45 & 73 & 59 \\  & Ours & 57 & 61 & 60 & 88 & 61 & 61 & 67 & 65 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Class-wise accuracy on RAF-DB, FERPLUS, and AffectNet datasets. Green, blue, and red mark the highest value of single category in RAF-DB, FERPLUS, and AffectNet, respectively.  
tion experiments are compared for each module on the RAF-DB, FERPlus, and AffectNet datasets, as shown in Table 4.  
IR50 is used as the baseline directly for the expression recognition task after pre-training. We individually investigated the impact of introducing the ViT module, the HLA module, and both on the model. These three methods correspond to method b, method c, and HLA-ViT in Table 4, respectively. The accuracy of the model on RAF-DB, FERPlus, and AffectNet is improved by 2.94%, 1.0%, and 2.82%, respectively, when method b is compared with Baseline, which indicates that, after applying ViT, the model obtains the global self-attention provided by ViT and enhances the network's ability to extract image features. The ViT has a strong robustness to the interference factors present in expression images in natural scenes. Method c is compared with Baseline, and the accuracy of the model on RAF-DB, FERPlus, and AffectNet is improved by 2.71%, 1.09%, and 1.32%, respectively, indicating the effectiveness of the HLA module, which also indicates that extracting local attention features is beneficial to improving the expression recognition performance. This can be explained by the fact that the local attention feature module enhances the robustness of the network to local occlusions and head pose changes, thus improving the performance of expression recognition in natural scenes. HLA-ViT combining hybrid local attention and global attention from ViT has improved the accuracy on RAF-DB, FERPlus, and AffectNet compared with other methods, which shows the effectiveness of both modules in this method.