# Learning Better Representations for Audio-Visual Emotion Recognition with Common Information
## 2 Related Works
### Feature Extraction with Deep Learning  
One key challenge in audio-visual emotion recognition is feature extraction [4; 53], which decides what types of features with acceptable sizes are learned for emotion recognition. Since the success of deep learning, DNN has been gradually used to extract features and achieves better performance than traditional heuristic methods [54; 55]. Here, we focus on DNN approaches for emotion recognition, which can be divided into two categories according to the type of input data: raw data-based and hand-crafted features-based. For audio emotion recognition, some works use hand-crafted features as the input to DNN models. For example, [PERSON] et al. [56] proposed a multi-task attention-based DNN model and feed the hand-crafted features into the model. Some works directly feed the raw data to the DNN models. For example, [PERSON] et al. [20] designed a network with several convolutional layers to extract audio features. [PERSON] et al. [57] proposed a new CNN with Long Short-Term Memory (LSTM) for end-to-end audio emotion recognition. [PERSON] et al. [26] used a sparse autoencoder in order to obtain the hidden features of audio data. [PERSON] et al. [58] presented an approach to learn discriminative features from the audio data by integrating center loss in the deep learning model for audio emotion recognition. For visual emotion recognition, most works directly send visual data to the DNN models. For example, [PERSON] et al. [59] proposed a network that consisted of two convolutional layers, each of which was followed by max-pooling and four Inception layers. The Inception layers increased the depth and width of the network while keeping the computational budget constant. [PERSON] et al. [60] proposed a hybrid convolution-recurrent neural network. It consists of convolution layers followed by a recurrent neural network, which can consider the temporal dependencies that exist in the facial images. [PERSON] et al. [61] presented an algorithm to automatically infer facial expressions by analyzing only a partially occluded face while the user is engaged in the virtual reality experience. [PERSON] et al. [4] employed a 3D CNN pre-trained on large-scale video classification tasks to capture the feature representations in visual data. Besides, the representation of the input data also has a significant impact on the feature extraction process. For example, [PERSON] et al. [62] divided the visual emotion recognition methods into two categories according to the representation: static-based and dynamic-based. In static-based methods [59; 63; 64], two-dimensional (2D) networks are used to extract spatial information from the single facial expression image, whereas dynamic-based methods [65; 66; 67] use 3D networks to capture the spatial and temporal information from the facial expression sequence. Although 3D networks contain more information than 2D networks, it is difficult to choose the appropriate 3D networks that can effectively improve the emotion recognition performance. Besides, the large scale of 3D networks may lead to cumbersome training processes. Analogously, in terms of audio emotion recognition, some works [29; 54] directly send the time-domain audio data into the DNN models, while other works [58; 68; 69] first convert the audio data into spectrum representations that are similar to the RGB images, and then feed the spectrums into deep learning models. The latter approach is considered to be more effective [69]. Recently, some popular deep learning models have been widely used, such as VGGNet [70], GoogLeNet [71], and ResNet [37]. When compared with VGGNet and GoogLeNet, ResNet has some advantages. For example, ResNet is shown to have better performance than VGGNet and GoogLeNet using a residual learning framework to ease the training of networks that are substantially deeper than those used previously [37]. Besides, ResNet has been successfully applied to audio emotion recognition [72; 73; 74; 75] and visual emotion recognition [76; 77; 78; 79; 80].