# Facial Expression Recognition Based on Vision Transformer with Hybrid Local Attention
###### Abstract  
Facial expression recognition has wide application prospects in many occasions. Due to the complexity and variability of facial expressions, facial expression recognition has become a very challenging research topic. This paper proposes a Vision Transformer expression recognition method based on hybrid local attention (HLA-ViT). The network adopts a dual-stream structure. One stream extracts the hybrid local features and the other stream extracts the global contextual features. These two streams constitute a global-local fusion attention. The hybrid local attention module is proposed to enhance the network's robustness to face occlusion and head pose variations. The convolutional neural network is combined with the hybrid local attention module to obtain feature maps with local prominent information. Robust features are then captured by the ViT from the global perspective of the visual sequence context. Finally, the decision-level fusion mechanism fuses the expression features with local prominent information, adding complementary information to enhance the network's recognition performance and robustness against interference factors such as occlusion and head posture changes in natural scenes. Extensive experiments demonstrate that our HLA-ViT network achieves an excellent performance with 90.45% on RAF-DB, 90.13% on FERPlus, and 65.07% on AffectNet. facial expression recognition; attention; vision transformer +
Footnote †: journal:  
in a timely manner to assist in teaching. From the above application cases, it is clear that facial expression recognition has a wide application prospect and has important research significance and application value. With the rapid development of related computing devices and algorithms, significant progress has been made in the field of face expression recognition. The feature extraction methods have gradually evolved from the early manual feature extraction to adaptive feature extraction using deep-learning methods. The performance of face expression recognition has improved significantly in various published datasets, and the performance has reached saturation on some datasets under traditional controlled conditions like CK+ [2]. However, facial expression recognition in natural scenes faces many challenges. The dataset in the natural scene is different from the traditional control condition dataset, such as the source of the face image being different, and the face expression image in the dataset is obtained through the Internet, TV series, movies, etc. Under controlled conditions, expressions are generated through guidance, while expressions in natural scenes are spontaneous. In addition, the main challenges of facial recognition tasks in natural scenes include face occlusions and head pose variation. Face occlusion results in the loss of face information in the occluded part, which is also a common problem affecting the accuracy of expression recognition in natural scenes. Head pose variation will change the shape of the face, blur the facial expression information, make the expression information distributed in different positions and angles, and affect the correct recognition. With the advancement in deep learning, most expression recognition methods use convolutional neural networks (CNNs) for extracting image features. These methods have demonstrated exceptional performance in the field of expression recognition. However, CNNs are implemented by continuously overlapping convolutional layers to perform image extraction from local to global, which not only leads to a significant increase in computation, but is also accompanied by the problem of gradient disappearance, making the network unable to converge. In addition, these CNN-based expression recognition methods also lack robustness and can be affected by occlusion and head pose variation, failing to focus on more important facial regions. To address the above problems, this paper proposes a Vision Transformer expression recognition method based on hybrid local attention.