# Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN
## 4 Experiments
### Experiment Results  
To show the effectiveness of our approach, we first compare the classification performance of our method with other methods on different datasets, including eNTERFACE'05, RAVDESS, and CMEW. On each dataset, three settings are used for experiments. Note that to take full advantage of the methods of Ours (wo corr) and Ours, false multimodal data with appropriate sizes should be generated for data augmentation. The classification accuracies of different methods on the test set are reported, as shown in Tables 1-3. We find the following summarizations: (1) Our approach achieves the highest performance compared to other methods, which shows that the data generated using our multimodal conditional GAN can significantly benefit audio-visual emotion recognition. (2) Although the classification performance of Ours (wo corr) is significantly higher than that of baseline methods, it is still lower than that of Ours, which shows that the correlation between the audio modality and the visual modality is helpful for data augmentation to improve the performance of emotion recognition. (3) In most cases, the classification accuracies of Attention Mechanism, CCA, and Tensor Fusion are higher than those of Baseline (A), Baseline (V), and Baseline (A+V), but still obviously lower than those of Ours (wo corr) and Ours. In addition, when the scale of training data is relatively small, especially in the case of Setting1, the performance of Attention Mechanism, CCA, and Tensor Fusion may be lower than that of Baseline (A+V). These findings show that the above previous works are not as effective as our method to utilize the information from the size-limited training set for classification. (4) We can further find that as the size of the original training set becomes smaller, the classification performance gap between Ours and Baseline (A+V) becomes larger. For example, on the RAVDESS dataset, the accuracy gap between Ours and Baseline (A+V) is 8.10% and 6.37% in the case of Setting1 and Setting3, respectively. This indicates that the fewer the real data, the more obvious the advantages of our approach. (5) In most cases, classification accuracies on the eNTERFACE'05 dataset and the RAVDESS dataset are higher than those on the CMEW dataset, which is consistent with the experimental results of previous works [10,17]. Next, we show the confusion matrices using Baseline (A+V), Ours (wo corr), and Ours in the case of Setting3 on the datasets of eNTERFACE'05, RAVDESS, and CMEW, as shown in Figures 3-5. It can be found that on the eNTERFACE'05 dataset, \"disgust\" is the easiest to be recognized. On the RAVDESS dataset, \"happiness\" is the easiest to be recognized. On the CMEW dataset, \"anger\" is the easiest to be recognized. This shows that different datasets have different emotional cues. In addition, we can further find that for a few emotions, the classification accuracy using Baseline (A+V) may be higher than that using Ours (wo corr) or Ours, such as \"disgust\" on the RAVDESS dataset and \"anger\" on the CMEW dataset. However, for most emotions, the classification accuracy using Ours (wo corr) or Ours is higher. This shows that Ours (wo corr) or Ours can more effectively use the information of most emotions for the classification task. \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{3}{c}{**Sample Size**} \\  & **Setting1** & **Setting2** & **Setting3** \\ \hline Baseline (A) & 25.26 & 31.95 & 39.18 \\ Baseline (V) & 25.77 & 44.85 & 61.34 \\ Baseline (A+V) & 26.29 & 45.88 & 65.46 \\ \hline Attention & 26.29 & 46.39 & 65.98 \\ Mechanism [97] & 26.80 & 45.88 & 66.49 \\ Tensor Fusion [99] & 27.32 & 46.91 & 67.53 \\ \hline Ours (wo corr) & 29.38 & 48.97 & 69.07 \\ Ours & **31.95** & **50.52** & **71.13** \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Classification accuracy (%) of different methods on the eNTERFACE’05 dataset. Best results are shown in bold.