# Learning Better Representations for Audio-Visual Emotion Recognition with Common Information
## 6 Conclusions  
In this paper, we propose an efficient deep learning approach to exploit common information between audio data, visual data, and the corresponding emotion labels for emotion recognition on the eNTERFACE'05, BAUM-1s, and RAVDESS datasets. To be specific, we design an audio network and a visual network to learn the feature representations from audio data and visual data, respectively, and then use a fusion network to combine the audio and visual features for emotional recognition. The full loss function of our whole neural network is a linear combination of correlation loss and classification loss. The former is used to extract common information between audio data, visual data, and the corresponding emotion labels with HGR maximal correlation. The latter is used to extract discriminative information from different modalities. We further generalize our framework to the semi-supervised learning scenario. The experimental results demonstrate that by combining the common information with HGR maximal correlation, our deep learning approach can significantly enhance the stability of features that are learned from different modalities, and improve the emotion recognition performance.  
In the future, we will investigate the performance of our method for emotion recognition in more datasets. It is worth noting that, in the real world, both audio data and visual data may be noisy. We will further consider audio-visual emotion recognition in the noisy environment. Besides, in addition to audio and visual data, physiological signals [101; 102; 103] and text data [104] are important modalities for characterizing human emotions. Therefore, we will consider combining the information of these modalities for multimodal emotion recognition.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multicolumn{2}{c}{**Method**} & \multicolumn{5}{c}{**Percentage of Labels**} \\  & & **20\%** & **50\%** & **80\%** & **100\%** \\ \hline \multirow{3}{*}{\(\alpha=0.01\)} & Audio & 42.54 & 63.01 & 71.56 & 75.61 \\  & Visual & 65.43 & 84.16 & 91.79 & 95.49 \\  & Baseline & 67.40 & 88.44 & 93.29 & 96.07 \\ \hline \multirow{2}{*}{\(\alpha=0.01\)} & \(L_{corr\_simple}\) & 68.90 & 88.79 & 94.22 & 97.11 \\  & \(L_{corr\_semi}\) & 69.94 & 89.36 & 94.68 & **97.23** \\ \hline \multirow{2}{*}{\(\alpha=0.1\)} & \(L_{corr\_simple}\) & 69.60 & 88.79 & 94.80 & 97.11 \\  & \(L_{corr\_semi}\) & **70.64** & 89.71 & 94.68 & **97.23** \\ \hline \multirow{2}{*}{\(\alpha=1\)} & \(L_{corr\_simple}\) & 69.02 & 87.63 & 94.34 & 96.88 \\  & \(L_{corr\_semi}\) & 69.13 & **90.29** & 95.26 & 96.99 \\ \hline \multirow{2}{*}{\(\alpha=10\)} & \(L_{corr\_simple}\) & 67.28 & 88.09 & **95.72** & 96.42 \\  & \(L_{corr\_semi}\) & 69.60 & 89.48 & **95.72** & 96.65 \\ \hline \hline \end{tabular}
\end{table}
Table 7: The performance of semi-supervised audio-visual emotion recognition on the RAVDESS dataset.  
Author Contributions:Conceptualization, F.M.; methodology, F.M.; software, F.M. and W.Z.; validation, F.M., W.Z., and Y.L.; formal analysis, F.M., W.Z., and Y.L.; investigation, F.M.; resources, S.-L.H. and L.Z.; data curation, F.M.; writing-original draft preparation, F.M.; writing-review and editing, F.M., W.Z., Y.L., and S.-L.H.; visualization, F.M. and W.Z.; supervision, Y.L., S.-L.H., and L.Z.; project administration, S.-L.H.; funding acquisition, S.-L.H. All authors have read and agreed to the published version of the manuscript. Funding: The research of [PERSON] was supported in part by the Natural Science Foundation of China under Grant 61807021, in part by the Shenzhen Science and Technology Program under Grant KQTD20170810150821146, and in part by the Innovation and Entrepreneurship Project for Overseas High-Level Talents of Shenzhen under Grant KQJSCX20180327144037831. Conflicts of Interest: The authors declare no conflict of interest.