# Learning Better Representations for Audio-Visual Emotion Recognition with Common Information
## 1 Introduction  
Emotion recognition is an important component in affect-related human-machine interaction systems [1, 2], as emotion can provide implicit feedback about human experience and conditions that are not easily captured by the explicit input. Audio-visual emotion recognition is a common type of emotion recognition [3, 4]. The comprehensive overview can be found in the surveys [5, 6, 7, 8, 9, 10]. Recent works have successfully applied it for many areas, such as disease diagnosis [11, 12], affective tutoring system [13, 14], marketing [15, 16], and entertainment [17, 18]. One challenge of audio-visual emotion recognition is how to extract feature representations with an acceptable size from audio and visual data that are effective for emotion recognition. A number of previous works [19, 20, 21, 22, 23, 24, 25, 26, 27] have been proposed to tackle this challenge. Although progresses have been made by previous works, they usually suffer from the following two limitations. First and foremost, these conventional strategies usually cannot efficiently utilize common information between different modalities by correlation analysis. For example, in [19; 20; 21], the common information is captured by combining the features that are learned from each modality into a feature vector. This technique often fails to exploit the complex dependencies and interactions between different modalities. Motivated by this concern, approaches [22; 23; 24] that are based on canonical correlation analysis (CCA) and some variant methods, such as kernel probabilistic CCA [25], sparse local discriminative CCA [26], and low-rank representation [27], are proposed. Although these methods have made some performance improvements, they may suffer from numerical issues [28; 29]. This instability arises in that they need to use the inverse of the empirical covariance matrix, which easily become singular over some mini-batches. As a workaround, such methods often limit the feature dimensionality to be relatively small in order to ensure stability, which is undesirable for the complex audio-visual emotion recognition task. Besides, some previous works use heuristic features [22; 23; 24; 25; 27] or the features learned from shallow neural network structures [19; 20; 21; 26] for emotion recognition. For example, the widely used audio heuristic features include prosody features and voice quality features [9], while typical visual heuristic features include Gabor features and HOG-TOP features [30]. Recently, convolutional neural networks (CNNs) have become popular for extracting audio and visual features for emotion recognition [19; 20; 21]. However, due to the high-dimensional emotional data, the learned representations using such methods with shallow structures are not expressive enough to predict emotions, which may lead to limited performance. To address the above two problems, we propose an deep learning framework for audio-visual emotion recognition by efficiently utilizing common information between audio data, visual data, and the corresponding emotion labels. Figure 1 presents the structure of our system, which satisfies: (i) the highly non-linear correlation of the feature representations among audio data, visual data, and the corresponding labels should be fully analyzed in order to capture the common information, and (ii) the learned audio and visual features should have enough expressiveness to classify the emotions. By considering these two goals together, we can learn the feature representations that are fully discriminative for the emotion recognition task. Specifically, we design an audio network and a visual network to learn the feature representations from audio and visual data, respectively, and then adopt a fusion network to combine the extracted audio and visual features for emotion prediction. Our neural network is trained by a joint loss function which is the linear combination of correlation loss and classification loss.