# Learning Better Representations for Audio-Visual Emotion Recognition with Common Information
## 5 Experiments
### Experimental Results
#### 5.3.3 Stability Analysis of Common Information  
From Figures 1 and 4, we can see that our proposed correlation loss can make the input features of the fusion network, \(\mathbf{f}\) and \(\mathbf{g}\), maximally correlated. The more fully connected layers in the fusion network, the farther common information is from the softmax output, which may lead to different emotion recognition performance. Besides, we know that \(\alpha\) determines how the correlation loss works during the training process. Different \(\alpha\) values will lead to different operating mechanisms of common information. Motivated by these two factors, we investigate the stability of common information in our deep learning model. We do the following study with different settings on the eNTERFACE'05, BAUM-1s, and RAVDESS datasets, as shown in Tables 4-6. We, respectively, make the fusion network has one, two and four fully connected layers before the softmax layer, which is shown in the setting column. The baseline in the method column means that we only use classification loss for training. Figure 5: The confusion matrices of audio-visual emotion recognition using our method on the eNTERFACE’05, BAUM-1s, and RAVDESS datasets. Figure 6: Some misclassification samples on the eNTERFACE’05, BAUM-1s, and RAVDESS datasets. For instance, the image on the left of the first row is marked with “disgust \(\rightarrow\) anger”, which indicates that the true label of the input data is “disgust”, but our model predicts that its label is “anger”. We compare the performance of two forms of correlation loss, one is \(L_{corr}\), the other is \(L_{corr\_simple}\), which only computes the HGR maximal correlation between the features \(\mathbf{f}\) and \(\mathbf{g}\). We set \(\alpha\) to 0.01, 0.1, 1, and 10, respectively, to implement audio-visual emotion recognition to find how the common information works. For each setting, we report the performance of \(L_{corr\_simple}\) method and \(L_{corr}\) method, respectively. We have the following observations on the three datasets: (1) the accuracy of baseline methods is higher than that of audio network and visual network, which indicates that baseline methods can combine the information of audio data and visual data to some extent to improve the emotion recognition performance. (2) When compared with the baseline method, \(L_{corr}\) can significantly improve the emotion recognition performance for different weight coefficients and different fully connected layers in the fusion network. This shows that the correlation loss we proposed, \(L_{corr}\), is very stable in ing audio-visual emotion recognition. (3) It should be noted that the \(L_{corr\_simple}\) method can also improve emotion recognition performance, but it is weaker than \(L_{corr}\) method. For example, on the eNTERFACE'05 dataset, when the fusion network has one fully connected layer, \(L_{corr\_simple}\) methods with the weight coefficient of 0.1 perform worse than the baseline method. On the RAVDESS dataset, when the fusion network has four fully connected layers, \(L_{corr\_simple}\) method with the weight coefficient of 0.01 also performs worse than the baseline method. (4) On the eNTERFACE'05 dataset, the highest accuracy of 88.66% is achieved when we use \(L_{corr}\) and set \(\alpha=1\) with two fully connected layers in the fusion network. On the BAUM-1s dataset, the highest accuracy 67.59% is achieved when we use \(L_{corr}\) and set \(\alpha=1\) with one fully connected layer in the fusion network. On the RAVDESS dataset, the highest accuracy 97.57% is achieved when we use \(L_{corr}\) and set \(\alpha\) as 0.1 with one fully connected layer in the fusion network. These show that our method can benefit from common information with appropriate network settings. (5) When we set the weight coefficient to 0.1 or 1 and make the fusion network has one or two fully connected layers, our model performs better on all three datasets than models with other settings.