# Method of Multi-Label Visual Emotion Recognition Fusing Fore-Background Features
## 3 Approach
### Multi-Label Emotion Recognition Classifier (ML-ERC)  
#### 3.2.1 The Design of the Label Co-Occurrence Probability Matrix  
\(G=\{N,E\}\) represents a global label correlation graph structure, where \(N\) represents all nodes and \(E\) represents all edges within the graph, as illustrated in Equations (7) and (8):  
\[\left\{\begin{array}{l}N=\{n_{1},n_{2},n_{3},\ldots\ldots,n_{m}\}\\ n_{k\in[1,m]}=R_{k}^{\omega}\Rightarrow N=\mathbb{R}^{m\times\omega}\\ N=\mathbb{R}^{m\times\omega}=\{R_{1}^{\omega},R_{2}^{\omega},R_{3}^{\omega}, \ldots\ldots,R_{m}^{\omega}\}\end{array}\right. \tag{7}\]  
\[\left\{\begin{array}{l}E=\{e_{ij}|_{i,j\in\Omega}\}\\ \left(e_{ij}\right)_{i,j\in\Omega}=\left(\theta_{ij}\right)_{i,j\in\Omega} \end{array}\right. \tag{8}\]  
In Equation (7), each graph node, as denoted by \(n_{k}\), corresponds to an emotion type, where \(k\in[1,m]\), with \(m\) representing the number of emotion categories. Each node is represented by a word-embedding vector \(R_{k}^{\omega}\) with a length of \(\omega\) using Word2 Vec and GloVe. Consequently, all the nodes in the graph structure can be represented as \(\mathbb{R}^{m\times\omega}\). In Equation (8), \(e_{ij}\) represents the edge between node \(i\) and node \(j\), \(\Omega\) represents the set of all edges in the graph structure, and \(\theta_{ij}\) denotes the weight of the edge between node \(i\) and node \(j\). A larger value of \(\theta_{ij}\) indicates a stronger association between nodes \(i\) and \(j\).  
In this study, the correlation between different emotions was determined by analyzing the co-occurrence patterns of labels. \(A\) represents the label co-occurrence probability matrix, which can be constructed as follows:  
Step 1: Compute the co-occurrence count \(C(i,j)_{i,j\in[1,m]}\) of each pair of emotion labels \(i\) and \(j\) statistically from the dataset.  
Step 2: Compute the conditional probability \(P(i|j)\) for each pair of emotion labels in Equation (9):  
\[P(i|j)=\frac{P(ij)}{P(j)}=\frac{C(i,j)_{i,j\in[1,m]}}{C(j)_{j\in[1,m]}} \tag{9}\]  
\(C(j)_{j\in[1,m]}\) represents the frequency of occurrence of emotion label \(j\), while \(C(i,j)_{i,j\in[1,m]}\) represents the frequency of co-occurrence of emotion labels \(i\) and \(j\).  
Step 3: Construct the conditional probability matrix \(P\):  
\[P=\begin{bmatrix}1&P(2|1)&\ldots&P(m|1)\\ P(1|2)&1&\ldots&P(m|2)\\ \ldots&\ldots&\ldots&\ldots\\ P(1|m)&P(2|m)&\ldots&1\end{bmatrix} \tag{10}\]Step 4: According to Equation (11), a judgment is made on whether there is a correlation between different emotional labels. \(\tau\) represents the empirical threshold, which indicates the degree of correlation between \(i\) and \(j\). Values lower than \(\tau\) indicate a weaker correlation and should be ignored. After many experimental verifications, \(\tau\) was set to 0.2.  
\[\hat{P}=\left\{\begin{array}{l}0,if\ P(j|i)<\tau\\ 1,if\ P(j|i)\geq\tau\end{array}\right. \tag{11}\]  
Step 5: The normalization operation in Equation (12) is used to balance the connection weights between different nodes. The parameter \(\lambda\) was set to \(1\times 10^{-6}\), which was used to prevent the denominator from being zero.  
\[\hat{P}_{ij}=\left\{\begin{array}{l}\frac{\hat{P}_{ij}}{\lambda+\sum_{i=1}^ {m}\hat{P}_{ij}},i\
eq j\\ 1,i=j\end{array}\right. \tag{12}\]  
Step 6: To ensure more balanced and stable feature updates in the GCN and improve the model's performance, further normalization of \(\hat{P}_{ij}\) is applied using Equations (13)-(15) based on the normalized Laplacian matrix principle. \(D_{ij}\) represents the degree matrix, and \(A\) represents the label co-occurrence probability matrix.  
\[D_{ij}=\left\{\begin{array}{l}\sum_{j=1}^{m}\hat{P}_{ij},i=j\\ 0,i\
eq j\end{array}\right. \tag{13}\]  
\[\tilde{D}_{ij}=D_{ij}^{-\frac{1}{2}} \tag{14}\]  
\[A=\left\{\begin{array}{l}\tilde{D}_{ij}\tilde{P}\tilde{D}_{ij},i\
eq j\\ 1,i=j\end{array}\right. \tag{15}\]