# Learning Better Representations for Audio-Visual Emotion Recognition with Common Information
## 5 Experiments
### Experimental Results
#### 5.3.5 Semi-Supervised Audio-Visual Emotion Recognition  
Semi-supervised learning has been used in many tasks when labeled data are scarce or difficult to obtain. It assumes that, during the training process, we have some labeled data and some unlabeled data. By using these two parts of data, semi-supervised learning can help us to perform better classification than supervised learning using only labeled data. Here, we show that our method can be easily adapted to semi-supervised audio-visual emotion recognition task. We conduct experiments on the RAVDESS dataset. Specifically, we mask the labels of some training data to indicate that they are unlabeled data. For the labeled data, audio modality, visual modality, and the corresponding emotion labels are available. For the unlabeled data, only audio modality and visual modality are available. We, respectively, set 20%, 50%, 80%, and 100% of the training data as labeled data. For each semi-supervised scenario, we compare the performance of different methods. The average accuracy of each method is reported in Table 7.  
In the method column of Table 7, audio means that only the audio modality in the labeled data is trained for audio emotion recognition. Visual means that only the visual modality in the labeled data is trained for visual emotion recognition. Baseline means that audio and visual modalities in the labeled data are trained by our whole network with only classification loss for audio-visual emotion recognition. \(L_{corr\_simple}\) means that whether labeled data or unlabeled data, we use the correlation loss \(L_{corr\_simple}\) to compute the HGR maximal correlation between audio and visual modalities. In addition, for labeled data, we use Equation (6) in order to compute the classification loss \(L_{clf}\). \(L_{corr\_semi}\) means that we use Equation (8) to compute the correlation loss \(L_{corr\_semi}\). Additionally, similar to \(L_{corr\_simple}\) method, the classification loss \(L_{clf}\) is also computed using labeled data. It is worth noting that, for the \(L_{corr\_simple}\) method and \(L_{corr\_semi}\) method, classification loss and correlation loss are combined in the form of Equation (3) with the weight coefficient \(\alpha\) to train our network together. Here, we make the fusion network have two fully connected layers to implement the baseline, \(L_{corr\_simple}\) and \(L_{corr\_semi}\) methods for audio-visual emotion recognition. By comparing with the baseline method that only uses  
Figure 7: Three scenarios with missing modality during the testing process: only audio data are missing, only visual data are missing, and audio and visual data are missing. The performance of \(L_{corr}\) method, \(L_{corr\_simple}\) method and baseline method are compared. The missing rate of 0% means that there are no missing data.  
labeled data, we can find that our proposed method \(L_{corr\_semi}\) can effectively combine unlabeled and labeled data for audio-visual emotion recognition.  
From Table 7, we have the following observations: (1) the performance of baseline method is better than that of audio method and visual method, which shows the importance of combining information of different modalities to improve the emotion recognition accuracy. (2) \(L_{corr\_semi}\) method achieves the highest accuracy among all methods, and it shows that it can significantly improve the performance of audio-visual emotion recognition for different weight coefficients and different percentages of labels. (3) \(L_{corr\_simple}\) method can also contribute to improving the performance of audio-visual emotion recognition, but it is weaker than \(L_{corr\_semi}\) method, especially when the labels are insufficient. To sum up, our method can effectively improve the performance of audio-visual emotion recognition, showing its potential for semi-supervised learning.