# Method of Multi-Label Visual Emotion Recognition Fusing Fore-Background Features
## 3 Approach
### Fore-Background-Aware Emotion Recognition (FB-ER)
#### 3.1.2 Emotion Recognition Fusing Fore-Background Features  
FB-ER consists of three sub-network branches, as depicted in Figure 3. The first branch is dedicated to the human body, where it extracts essential cues, such as facial expressions and body postures. To achieve this, ResNet18 [32] is utilized as the backbone network for feature extraction. Transfer learning is employed to fine-tune the pre-trained model developed by [PERSON] et al. [33]. Following the extraction of the body features, a CR-Unit is introduced to emphasize regions that contribute the most to emotion recognition. The structure of the CR-Unit is presented in Figure 4, with its mathematical formulation detailed in Equations (1)-(3):  
\[f\Big{(}F_{body},num_{1}\Big{)}=BatchNorm\Big{(}Linear_{1}\Big{(}F_{body},num_{1} \Big{)}\Big{)} \tag{1}\]  
\[Core\_weights=Adaptive[\sigma(Linear_{2}(num_{1},num_{2}))] \tag{2}\]  
\[F_{core}=\text{diag}\Big{[}dot\Big{(}F_{body},Attention\_weights\Big{)}\Big{]} \tag{3}\]  
In Equation (1), \(F_{body}\) represents the body features, which are mapped to length \(num_{1}\) through \(Linear_{1}\), followed by \(BatchNorm\). In Equation (2), the features are mapped from length \(num_{1}\) to length \(num_{2}\) by \(Linear_{2}\), and then \(Core\_weights\) of the same length as \(F_{body}\) is obtained using \(Sigmoid\) and \(Adaptive\). In Equation (3), \(dot\) denotes the dot product operation, and \(diag\) represents the extraction of diagonal elements, thus ultimately yielding \(F_{core}\). \(F_{core}\) focuses more on the regions of \(F_{body}\) that are beneficial for emotion classification, thus effectively enhancing the model's ability to recognize the important parts of the \(F_{body}\) \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Index** & **Description** & **Approximate Proportion in the Dataset** & **Original Figure** & **Depth Map** & **Emotions of Character A** & **Emotions of Character B** \\ \hline No. 1 & Interaction occurs when individuals share a common identity or mutual familiarity. & 10.4\% & & [”Affection”, “Happiness”] & [”Affection”, “Happiness”] \\ No. 2 & Having different identities or being unfamiliar with one another. & 19.6\% & & [”Confidence”, “Engagement”] & [”Confidence”, “Engagement”] \\ \hline No. 2 & Having different identities or being unfamiliar with one another. & 19.6\% & & [”Anticipation”, “Confidence”, “Engagement”] \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Example table of correlations between foreground and emotions. The second sub-network branch was designed for background feature extraction, which provides useful information for understanding the emotions of individuals in the image. The background can be considered as a set of keywords associated with objects and ongoing activities within a scene. To evaluate the scene comprehension, the Places365 model [34] was utilized. The results presented in Table 3 exhibit two different scenes: outdoors and indoors. For the outdoor scene, the recognition keywords include \"outdoor\", \"cliff\", \"climbing\", and \"sunny\". For the indoor scene, the keywords include \"indoor\", \"office\", \"working\", and \"closed area\". ResNet18 was also employed for extracting background features. Transfer learning techniques were used to fine-tune the pre-trained weights of the Places365 model in this study. Figure 4: CR-Unit diagram. Figure 3: Overall architecture diagram. The third sub-network branch was designed for foreground feature extraction, which primarily includes the interactions between different individuals. These interactions are valuable for emotion evaluation. This study adopted a method based on depth map extraction to simulate the interaction and proximity between different individuals. The steps are organized as follows:  
Step 1--Data preprocessing: all images in the dataset are normalized to RGB three channels.