# Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN
###### Abstract  
Audio-visual emotion recognition is the research of identifying human emotional states by combining the audio modality and the visual modality simultaneously, which plays an important role in intelligent human-machine interactions. With the help of deep learning, previous works have made great progress for audio-visual emotion recognition. However, these deep learning methods often require a large amount of data for training. In reality, data acquisition is difficult and expensive, especially for the multimodal data with different modalities. As a result, the training data may be in the low-data regime, which cannot be effectively used for deep learning. In addition, class imbalance may occur in the emotional data, which can further degrade the performance of audio-visual emotion recognition. To address these problems, we propose an efficient data augmentation framework by designing a multimodal conditional generative adversarial network (GAN) for audio-visual emotion recognition. Specifically, we design generators and discriminators for audio and visual modalities. The category information is used as their shared input to make sure our GAN can generate fake data of different categories. In addition, the high dependence between the audio modality and the visual modality in the generated multimodal data is modeled based on Hirschfeld-Gebelein-Renyi (HGR) maximal correlation. In this way, we relate different modalities in the generated data to approximate the real data. Then, the generated data are used to augment our data manifold. We further apply our approach to deal with the problem of class imbalance. To the best of our knowledge, this is the first work to propose a data augmentation strategy with a multimodal conditional GAN for audio-visual emotion recognition. We conduct a series of experiments on three public multimodal datasets, including NTERACE'05, RAVDESS, and CMEW. The results indicate that our multimodal conditional GAN has high effectiveness for data augmentation of audio-visual emotion recognition. audio-visual emotion recognition; data augmentation; multimodal conditional generative adversarial network (GAN); Hirschfeld-Gebelein-Renyi (HGR) maximal correlation +
Footnote †: journal:  
deep learning may not be sufficient, which leads to overfitting of deep learning models. Secondly, the emotional data may be class-imbalanced, which means that the minority class contains significantly fewer samples than the majority class. The problem of class imbalance may result in deep learning models showing bias towards the majority class and completely ignoring the minority class in extreme cases [22; 23], which can further deteriorate the effectiveness of deep learning for audio-visual emotion recognition. To address the aforementioned issues, we propose an efficient data augmentation approach with a multimodal conditional generative adversarial network (GAN) for audio-visual emotion recognition. Data augmentation is shown to be an effective way to tackle problems of data insufficiency and class imbalance in many fields, such as speech [24; 25; 26; 27; 28], vision [29; 30; 31; 32; 33; 34], and time series [35; 36; 37; 38]. Different data augmentation techniques are proposed to increase the size of the training data, such as geometric transformations, random occlusion, and photometric transformations [35; 39]. GAN was first proposed in [40] to generate new data and is used for data augmentation in many applications [28; 30; 31; 41; 42; 43; 44]. Compared to other data augmentation techniques, GAN has received increasing attention by bringing more variations to the training data through deep learning [45; 46]. Our proposed multimodal conditional GAN is a generalization of existing GANs for data augmentation to improve the performance of audio-visual emotion recognition. Figure 1 shows the architecture of our data augmentation approach.