# Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN
## 5 Conclusions  
In this paper, we propose an efficient data augmentation framework with a multimodal conditional GAN for audio-visual emotion recognition. To be specific, our framework builds generators and discriminators for audio and visual modalities. Then, we use the category information as their shared input to make our multimodal conditional GAN generate new multimodal data of different categories. In addition, the correlation between the audio modality and the visual modality in the generated multimodal data is fully considered in our framework. Finally, we use the generated data to supplement the original training set for the emotion classification task. To the best of our knowledge, we are the first to propose an efficient multimodal condition GAN to augment the multimodal data for audio-visual emotion recognition. A series of experiments on the datasets of eNTERFACE'05, RAVDESS, and CMEW show that compared with different baselines, our framework achieves the highest classifica \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **Class Balance** & \multicolumn{2}{c}{**Class Imbalance**} \\  & & **Case1** & **Case2** \\ \hline Baseline (A) & 31.95 & 31.44 & 30.41 \\ Baseline (V) & 44.85 & 38.14 & 34.54 \\ Baseline (A+V) & 45.88 & 39.69 & 36.60 \\ \hline Attention Mechanism [97] & 46.39 & 41.24 & 35.57 \\ CCA [98] & 45.88 & 39.18 & 36.08 \\ Tensor Fusion [99] & 46.91 & 40.72 & 37.11 \\ \hline Ours (wo corr) & 48.45 & 43.30 & 41.75 \\ Ours & **49.48** & **44.33** & **42.78** \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Classification accuracy (%) of different methods when the training set is class-imbalanced on the eNTERFACE’05 dataset. Best results are shown in bold. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **Class Balance** & \multicolumn{2}{c}{**Class Imbalance**} \\  & & **Case1** & **Case2** \\ \hline Baseline (A) & 39.88 & 37.57 & 36.99 \\ Baseline (V) & 54.35 & 51.45 & 50.29 \\ Baseline (A+V) & 59.53 & 54.34 & 53.75 \\ \hline Attention Mechanism [97] & 61.27 & 55.49 & 52.60 \\ CCA [98] & 60.69 & 56.07 & 53.18 \\ Tensor Fusion [99] & 60.12 & 52.60 & 51.45 \\ \hline Ours (wo corr) & 64.16 & 59.54 & 58.95 \\ Ours & **65.90** & **61.85** & **60.11** \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Classification accuracy (%) of different methods when the training set is class-imbalanced on the RAVDESS dataset. Best results are shown in bold. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **Class Balance** & \multicolumn{2}{c}{**Class Imbalance**} \\  & & **Case1** & **Case2** \\ \hline Baseline (A) & 36.19 & 33.57 & 29.76 \\ Baseline (V) & 38.10 & 34.76 & 31.42 \\ Baseline (A+V) & 40.71 & 35.48 & 34.76 \\ \hline Attention Mechanism [97] & 41.19 & 36.19 & 33.81 \\ CCA [98] & 41.43 & 36.10 & 33.10 \\ Tensor Fusion [99] & 40.00 & 35.24 & 32.86 \\ \hline Ours (wo corr) & 44.05 & 39.05 & 37.62 \\ Ours & **46.19** & **40.95** & **39.04** \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Classification accuracy (%) of different methods when the training set is class-imbalanced on the CMEW dataset. Best results are shown in bold. tion accuracy for audio-visual emotion recognition, even when the training data is scarce. In addition, experiments show that the data generated by our multimodal conditional GAN can improve the training speed of DNN-based emotion classifiers. Moreover, the results further point out that the size of the data generated using our multimodal conditional GAN should be set appropriately for data augmentation. Finally, we exploit our approach to cope with the problem of class imbalance. Experiments show that in this scenario, the classification performance of our approach is still significantly higher than that of different baselines. Despite the progress of our framework, it may have the following limitations: (1) Our approach is used for audio-visual emotion recognition.