# Method of Multi-Label Visual Emotion Recognition Fusing Fore-Background Features
###### Abstract  
This paper proposes a method for multi-label visual emotion recognition that fuses foreground features to address the following issues that visual-based multi-label emotion recognition often overlooks: the impacts of the background that the person is placed in and the foreground, such as social interactions between different individuals on emotion recognition; the simplification of multi-label recognition tasks into multiple binary classification tasks; and it ignores the global correlations between different emotion labels. First, a fore-background-aware emotion recognition model (FB-ER) is proposed, which is a three-branch multi-feature hybrid fusion network. It efficiently extracts body features by designing a core region unit (CR-Unit) that represents background features as background keywords and extracts depth map information to model social interactions between different individuals as foreground features. These three features are fused at both the feature and decision levels. Second, a multi-label emotion recognition classifier (ML-ERC) is proposed, which captures the relationship between different emotion labels by designing a label co-occurrence probability matrix and cosine similarity matrix, and uses graph convolutional networks to learn correlations between different emotion labels to generate a classifier that considers emotion correlations. Finally, the visual features are combined with the object classifier to enable the multi-label recognition of 26 different emotions. The proposed method was evaluated on the Emotic dataset, and the results show an improvement of 0.732% in the mAP and 0.007 in the Jaccard's coefficient compared with the state-of-the-art method. multi-label; emotion recognition; fore-background features; graph convolutional networks; correlation +
Footnote †: journal: Medical  
Most current research on emotion recognition [8; 9; 10] focuses on the single-label recognition of a few common emotions, such as happiness, sadness, surprise, neutrality, and anger. However, human emotions are inherently complex and multidimensional, and single-label classification falls short of capturing the full spectrum of emotional expression. Emotion datasets [11] encompass 26 fine-grained emotion categories, and multi-label recognition of these categories can provide a more nuanced and comprehensive description of an individual's emotional state. In multi-label emotion recognition, some studies [12] overlook the relationships between different emotions, often decomposing the task into independent binary classification tasks, which fails to account for the global correlations between all emotion labels and leads to reduced recognition accuracy. Traditional multi-label classification methods, such as ML-KNN [13] and ML-RBF [14], attempt to learn label correlations through kernel functions, loss functions, and association rules. However, these methods are often inefficient and primarily capture local correlations. The self-attention mechanism in Transformer architectures was also employed for modeling label correlations [15; 16], but it is more suited for long sequence data and demands substantial computational resources. Graph-based approaches can model label correlations as well, but the construction of the adjacency matrix is crucial. Consequently, developing a multi-label classifier that effectively captures the global correlations of emotions remains a significant challenge. To address the above issues, this paper proposes a method of multi-label visual emotion recognition that fuses fore-background features. The contributions of this approach are highlighted in three key aspects as follows:  
* A fore-background-aware emotion recognition model (FB-ER) is proposed. The background in which the person is placed and the foreground, such as interactions between different individuals, provide beneficial visual cues for emotion recognition, as shown in Figure 1. FB-ER is a three-branch multi-feature hybrid fusion network.