# Learning Better Representations for Audio-Visual Emotion Recognition with Common Information
## 2 Related Works
### Multimodal Learning  
In the real world, human emotion expression has the inherent multimodality characteristic [38, 39]. Multimodal learning [40, 41], also referred to as multiview learning [42], is proposed to build models  
Figure 1: The structure of our proposed system for audio-visual emotion recognition. The full loss function of our framework is a linear combination of correlation loss and classification loss. Audio network and visual network use ResNet-50 [37] as the backbone architectures. Fusion network has several fully connected layers. Different settings of fusion network are considered in Section 5.2.2. The correlation loss is used to extract common information between different modalities. Additionally, the classification loss is used to capture discriminative information from each modality for emotion prediction. During the training process, emotion labels are used twice, once to compute the classification loss, and the another as the third modality to compute the correlation loss with audio and visual modalities. In this way, label information can be fully used in order to improve the discrimination ability of the feature representations. In the testing process, audio and visual data are used to predict the corresponding emotion labels.  
that can better process and relate information from multiple modalities in order to efficiently use the multimodal data similar to the emotional data [43]. One important research topic is to extract the information between different modalities to achieve better predictions, which is consistent with the common information in our paper.  
The straightforward way is to concatenate the feature representations that were learned from each modality into a feature vector. Although this strategy is widely used, it often ignores the complex dependencies and interactions between different features of each modality and they may result in unsatisfactory performance [44]. Some CCA-based approaches are further proposed in order to capture the complex correlation between different modalities. It can extract the linearly maximally correlated feature mappings of two random variables. Kernel CCA [45; 46] and deep CCA [47] are proposed in order to generalize CCA to non-linear setting using the kernel method and the deep neural network (DNN) respectively. Subsequently, [PERSON] et al. [48] proposed the deep canonically correlated autoencoder (DCCAE) by combining the canonical correlation loss and the reconstruction loss of autoencoders.  
HGR maximal correlation is another important measure of dependence [29]. It can extract the maximally non-linear correlated features of different modalities, showing higher efficiency than CCA. Recently, ref [28; 49] implemented the HGR maximal correlation with deep learning. HGR maximal correlation and the deep learning version have become widely used [29; 50; 51; 52]. For example, ref [52] adopted HGR maximal correlation to extract the correlation between different modalities and uses the hand-crafted features as the input for audio-visual emotion recognition while [29] considered the HGR maximal correlation in autoencoder architectures for multimodal emotion recognition.  
Although HGR maximal correlation is used in the DNN methods [29; 52] to compute the common information between the input audio and visual data, the emotional label information is not considered in the correlation loss, so it cannot guarantee that the learned features have sufficient discrimination ability for the emotion recognition task. Besides, the stability of common information in the deep learning models is not investigated in these methods. Here, we will sufficiently study the effectiveness of common information for emotion recognition.