# Facial Expression Recognition Based on Vision Transformer with Hybrid Local Attention
## 3 Methodology
### Vision Transformer with Hybrid Local Attention  
Vision Transformer learns the features of images through a global self-attention mechanism. The application of Vision Transformer to facial expression recognition can improve the network's robustness to occlusion and head pose variations in natural scenes. We combine Vision Transformer with the proposed hybrid local attention module for facial expression recognition, which can effectively extract local and global features of the image. Fusion strategies is adopted to further improve the model's expression ability. The specific operation is to expand the feature map sequence x extracted from the backbone IR50 into a linear sequence by flattening operation and inputting it into the multilayer Transformer encoder. The dimension of the feature map sequence x is 256 \(\times\) 14 \(\times\) 14, where 14 \(\times\) 14 is the feature map space size and 256 is the channel size. The 2D feature map needs to be flattened into a 1D visual embedding sequence, so the model reshapes the feature map x into a flat sequence and feeds it into a linear projection to obtain a 196 \(\times\) 768 feature vector. In ViT, it is also necessary to append classification tokens (cls) to the beginning of the input sequence of the final prediction. In addition, a one-dimensional learnable position embedding is added to the feature embedding to obtain the position information of the image blocks. The multi-layer Transformer encoder consists of a sequence of L attention modules, where the size of L in this paper is 8. Each sequence consists of a multi-headed self-attention (MHSA) module and a multilayer perceptron (MLP), where the number of self-attention heads is set to 12. The MLP consists of two fully connected feature projection layers and a GELU for nonlinearity. Finally, a classification head implemented by a single-layer MLP is used for the classification output. Two traditional fusion methods are feature-level fusion and decision-level fusion when choosing a fusion strategy. Feature-level fusion directly combines the feature maps of two streams into a joint feature vector, and then trains the classifier, while decision-level fusion combines the classification results of these two streams. [PERSON] et al. [24] proposed a separated score integration method based on alpha integrals, and experimental results show that this method performs better than the considered single classifier and classical fusion techniques. [PERSON] et al. [25] studied the decision-level fusion method and proposed two new graph-based regularization methods, and the experimental results proved the superiority of the method. In HLA-ViT, the hybrid local features extracted by the network and the global contextual features extracted by the multi-layer Transformer encoder are complementary at the feature level, constituting global-local fusion attention. Therefore, the network model in this paper adopts a decision-level fusion strategy. Decision-level fusion is the highest level of fusion, which requires modeling emotional information separately for each channel, and then merging the recognition results of all channels. The advantage of decision-level fusion is its ability to parallelly combine multiple classifiers, independently integrating each classifier. This allows them to work autonomously, enabling each sub-classifier result to contribute to the final classification decision through a rational approach. When combining multiple classifiers in a parallel manner, the results of each sub-classifier can be in the form of classification probabilities, classification distances, actual classification results, or relevance metrics for different information classes. Typically, in practical expression recognition systems, they are designed as classification results. Because each sub-classifier works independently in a parallel combination, the output information of each classifier does not affect others.