# Learning Better Representations for Audio-Visual Emotion Recognition with Common Information
## 4 Methodology
### Model Learning  
To jointly extract emotional features from audio and visual data, we propose the full loss function of the whole network, which is a linear combination of classification loss and correlation loss, defined as Equation (3):  
\[L=L_{clf}+\alpha L_{corr} \tag{3}\]  
The classification loss, \(L_{clf}\), measures the classification performance. The correlation loss, \(L_{corr}\), measures the dependencies between audio data, visual data, and the corresponding emotion labels. The parameter \(\alpha\) is the weight coefficient. By considering the correlation loss in the training process, our model can extract the non-linear correlated feature representations with common information in order to improve the performance of audio-visual emotion recognition.  
When designing the correlation loss, most of previous works [29; 52] only compute the correlation between audio data and visual data, but ignore the effect of label information, which may lead to the learned features not directly related to emotion prediction. It is significant to incorporate emotional label information into the correlation loss in order to enhance the discrimination ability of the learned feature representations. Therefore, we introduce a new form of correlation loss based on HGR maximal correlation, \(L_{corr}\), as shown in Equation (4).  
\[\begin{split} L_{corr}(X,Y,Z)&=-\mathbb{E}[\mathbf{ f}^{\text{T}}(X)\mathbf{g}(Y)]+\frac{1}{2}\text{tr}(\text{  
Following [29], we use cross-entropy loss as the classification loss for emotion classification, as shown in Equation (6):  
\[L_{clf}(X,Y,Z)=-\mathbb{E}[\log P_{Z|XY}] \tag{6}\]  
where  
\[P_{Z=j|XY}(X,Y,Z)=\frac{\exp\left(\mathbf{\Phi}^{\mathrm{T}}\left(\mathbf{f}(X), \mathbf{g}(Y)\right)\mathbf{\theta}_{j}\right)}{\sum_{i=1}^{|\mathcal{Z}|}\exp \left(\mathbf{\Phi}^{\mathrm{T}}\left(\mathbf{f}(X),\mathbf{g}(Y)\right)\mathbf{ \theta}_{i}\right)} \tag{7}\]  
where \(j=1,\cdots,|\mathcal{Z}|\), \(\mathbf{\theta}_{j}\) represents the \(j\)-th term of the weights in the last layer of the fusion network, \(\mathbf{\Phi}\) represents the feature function of fully connected layers in the fusion network. Additionally, \(\mathbf{\Phi}^{\mathrm{T}}\left(\mathbf{f}(X),\mathbf{g}(Y)\right)\) represents the function, \(\mathbf{\Phi}^{\mathrm{T}}\), of the concatenation of \(\mathbf{f}(X)\) and \(\mathbf{g}(Y)\). In Section 5.3.3, we will conduct experiments to consider different forms of \(\mathbf{\Phi}\) to test the stability of common information in our framework.  
We can further consider the semi-supervised learning scenario [83; 84], where labeled data may be expensive or time-consuming to obtain. Semi-supervised learning can help us use labeled data and unlabeled data for better emotion recognition. Suppose that, in the training process, we have labeled data \((X_{t},Y_{t},Z_{l})\) and unlabeled data \((X_{u},Y_{u})\), where \(X_{t}\), \(Y_{t}\), and \(Z_{l}\), respectively, represent the labeled audio modality, labeled visual modality, and the corresponding emotional labels, and \(X_{u}\) and \(Y_{u}\) represent the unlabeled audio and visual modalities. We use Equation (4) in order to compute the correlation loss \(L_{corr}\) with the labeled data and Equation (5) to compute the correlation loss \(L_{corr\_simple}\) with the unlabeled data. The sum of correlation losses on these two parts is defined as \(L_{corr\_semi}\), as shown in Equation (8):  
\[L_{corr\_semi}(X_{l},Y_{l},Z_{l},X_{u},Y_{u})=L_{corr}(X_{l},Y_{l},Z_{l})+L_{ corr\_simple}(X_{u},Y_{u}) \tag{8}\]  
When the training data are 100% labeled, \(L_{corr\_semi}\) will become \(L_{corr}\). Besides, we use the labeled data \((X_{l},Y_{l},Z_{l})\) to compute the classification loss \(L_{clf}\). Subsequently, we combine \(L_{corr\_semi}\) and \(L_{clf}\) in the form of Equation (3) to jointly train our network for semi-supervised learning. In this way, our model can make full use of labeled data and unlabeled data for emotion recognition.