# Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN
## 4 Experiments
### Experiment Results
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Sample Size**} & \multirow{2}{*}{**Method**} & \multicolumn{5}{c}{**Augmentation Ratio**} \\  & & **0.5** & **1** & **2** & **3** & **4** & **5** \\ \hline \multirow{2}{*}{Setting1} & Ours (wo corr) & 27.32 & 28.35 & 27.84 & **29.38** & 28.87 & 27.84 \\  & Ours & 28.35 & 29.38 & **31.96** & 31.44 & 30.93 & 29.38 \\ \hline \multirow{2}{*}{Setting2} & Ours (wo corr) & 46.39 & 48.45 & 48.45 & **48.97** & 47.94 & 46.91 \\  & Ours & 48.45 & 49.48 & 50.00 & **50.52** & 50.00 & 48.97 \\ \hline \multirow{2}{*}{Setting3} & Ours (wo corr) & 67.53 & **69.07** & 68.56 & 68.04 & 68.56 & 67.01 \\  & Ours & 70.10 & **71.13** & 70.62 & 69.59 & 70.10 & 69.07 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Classification accuracy (%) with different augmentation ratios on the eNTERFACE’05 dataset. Best results are shown in bold. Figure 6: _Cont._We can find that on each setting, there is an augmentation ratio with the highest classification accuracy for Ours or Ours (wo corr). We can call this augmentation ratio the most appropriate augmentation ratio. In most cases, when the augmentation ratio is lower than the most appropriate augmentation ratio and increases, the corresponding classification accuracy also increases, while when the augmentation ratio is higher than the most appropriate augmentation ratio and increases, the corresponding classification accuracy decreases. For example, on Setting2 of the eNTERFACE'05 dataset, the most appropriate augmentation ratio is higher than the most appropriate augmentation ratio and increases, while the corresponding classification accuracy decreases. augmentation ratio is 3 for Ours. When the augmentation ratio increases from 0.5 to 1, 2, and then to 3, the corresponding classification accuracy also increases. However, when the augmentation ratio increases from 3 to 4 and then to 5, the corresponding classification accuracy becomes smaller. The reason behind this phenomenon may be that when the augmentation ratio is relatively small, the generated data is not enough for data augmentation effectively. Therefore, with the increase of augmentation ratio, the corresponding classification performance is improved. However, when the augmentation ratio is too large, the generated data may obscure the information of the original training set, which weakens the effect of data augmentation. Driven by these factors, we emphasize that the augmentation ratio should be selected appropriately for data augmentation. From Tables 4-6, it can be further found that when the scale of the original training set is relatively small, the augmentation ratio should be set larger, i.e., more generated data can be used for data augmentation, while when the scale of the original data is relatively large, the augmentation ratio should be set smaller. For example, on the eNTERFACE'05 dataset, the sizes of Setting1 and Setting2 are relatively small, and it is more appropriate to set the augmentation ratio to 2 or 3, while Setting3 has more samples, and it is better to set the augmentation ratio to 1. The RAVDESS dataset and the CMEW dataset are similar to the eNTERFACE'05 dataset. In addition, we can see that the classification performances of Ours and Ours (wo corr) have a similar trend with the augmentation ratio, but the classification performance of Ours is higher than that of Ours (wo corr) under each setting. For example, on the eNTERFACE'05 dataset, the most appropriate augmentation ratios of Ours or Ours (wo corr) are 3 and 2, respectively, for Setting2 and Setting3. However, under these two settings, the classification accuracy of Ours is 1.55% and 2.06% higher than that of Ours (wo corr), respectively, which again shows the importance of capturing the correlation between the audio modality and the visual modality in our framework. Furthermore, we show that our multimodal conditional GAN can deal with the problem of class imbalance of the training data. We conduct experiments on the datasets of eNTERFACE'05, RAVDESS, and CMEW.