# Method of Multi-Label Visual Emotion Recognition Fusing Fore-Background Features
## 2 Related Work  
### Visual Emotion Recognition  
Significant progress has been made in the study of facial emotions in recent years. These methods have achieved superior performance by developing effective facial feature extraction networks. [PERSON] et al. [8] and [PERSON] et al. [10] optimized their CNN models for computational efficiency, which resulted in models with both high computational efficiency and accuracy. [PERSON] et al. [9] enhanced emotion recognition by incorporating facial optical flow information, while [PERSON] et al. [17] constructed a facial key point grid and utilized GCN for emotion classification. [PERSON] et al. [18] adopted the designs of VGG, Inception-v1, ResNet, and Xception to propose the CVGG-19 architecture for classifying driver emotions, which offers the advantages of high accuracy and low cost. [PERSON] et al. [19] investigated the impact of noise in image data on classification accuracy and proposed a deep learning model to enhance robustness against noise. [PERSON] et al. [20] proposed a novel temporal shifting approach for a frame-wise transformer-based model using multi-head self/cross-attention (MSCA) to reduce the computational cost of emotion recognition. However, these methods focus solely on the facial region and do not consider other visual information beyond the face.  
Several studies [11; 21; 22; 23; 24] explored the use of visual information beyond facial features. For instance, [PERSON] et al. [21] employed a dual-stream network, where one branch extracted body features and the other extracted environmental features using a k-NN classifier to independently identify arousal and valence from these feature sets. However, due to the simplicity of the feature extraction network, it could not fully capture the necessary features. Additionally, the fusion of the two branches' outputs occurred at the decision level, without accounting for the influence of environmental features on emotion recognition. Building on this work, [PERSON] et al. [11], [PERSON] et al. [22], and [PERSON] et al. [23] concatenated  
Figure 1: Fore\(-\)background features diagram.  
Figure 2: Directed graph between the first five emotions (threshold: 0.2). Dashed lines represent weak correlations, solid lines represent strong correlations.  
different visual features for emotion classification. Nonetheless, the recognition accuracy varied greatly across different emotions, particularly those that are more difficult to classify. [PERSON] et al. [24] proposed a multi-label focal loss function to address the issue of imbalanced emotion categories, which led to improved emotion recognition performance. However, the networks used in these studies exhibited limited feature extraction capabilities, under-utilized background features, and relied on the simple concatenation of features without considering their complementarity. Furthermore, foreground information, such as social interactions between individuals, which can provide valuable insights into behavioral characteristics and enhance emotion recognition accuracy, remains underexplored in the current literature.