# Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN
## 4 Experiments
### Experiment Results
\begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{3}{c}{**Sample Size**} \\  & **Setting1** & **Setting2** & **Setting3** \\ \hline Baseline (A) & 24.86 & 34.10 & 39.88 \\ Baseline (V) & 28.90 & 37.57 & 54.34 \\ Baseline (A+V) & 34.10 & 45.09 & 59.53 \\ \hline Attention & 32.37 & 45.66 & 61.27 \\ CCA [98] & 33.06 & 44.51 & 60.69 \\ Tensor Fusion [99] & 33.53 & 43.35 & 60.12 \\ \hline Ours (wo corr) & 39.88 & 50.28 & 64.16 \\ Ours & **42.20** & **52.02** & **65.90** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Classification accuracy (%) of different methods on the RAVDESS dataset. Best results are shown in bold. \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{3}{c}{**Sample Size**} \\  & **Setting1** & **Setting2** & **Setting3** \\ \hline Baseline (A) & 30.71 & 36.19 & 40.24 \\ Baseline (V) & 31.43 & 38.10 & 41.67 \\ Baseline (A+V) & 33.10 & 40.71 & 47.38 \\ \hline Attention & 33.57 & 41.19 & 47.86 \\ CCA [98] & 34.05 & 41.43 & 48.10 \\ Tensor Fusion [99] & 32.14 & 40.00 & 47.14 \\ \hline Ours (wo corr) & 36.90 & 44.76 & 50.47 \\ Ours & **39.29** & **46.43** & **52.14** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Classification accuracy (%) of different methods on the CMEW dataset. Best results are shown in bold. Figure 3: _Cont._  
Figure 4: _Cont._  
Figure 3: The confusion matrices of different methods on the eNTERFACE’05 dataset. (**a**) Baseline (A+V), (**b**) Ours (wo corr), (**c**) Ours. Figure 4: The confusion matrices of different methods on the RAVDESS dataset. (**a**) Baseline (A+V), (**b**) Ours (wo corr), (**c**) Ours. Figure 5: _Cont._  
In the above discussion, we show that our approach can efficiently help improve the classification accuracy on the test set. Inspired by [105], we next study the impact of the data generated by our multimodal conditional GAN on the training speed. On the datasets of eNTERFACE'05, RAVDESS, and CMEW, we plot the training loss with epochs of the DNN classifier in different methods, as shown in Figure 6. It is worth pointing out that the DNN classifier in Baseline (A+V) only uses the original training set for training, while the DNN classifier in the method of Ours (wo corr) or Ours uses the real data and generated data together for training. It can be found that the training loss of our approach decreases the fastest with epochs, which means that its convergence is the fastest, the convergence of Ours (wo corr) is the second fastest, and the convergence of Baseline (A+V) is the slowest. This shows that in addition to helping improve the accuracy of emotion classification, our framework can effectively speed up the training of the emotion classifier. When analyzing the experimental results in Tables 1-3, we mention that when using Ours or Ours (wo corr), it is necessary to generate new data of appropriate size. Here, we discuss what scale of generated data is appropriate under different settings. We name the ratio of the size of the generated data to the size of the original training set as the augmentation ratio. It is used to measure the scale of the generated data compared to the original training set. Then, we analyze the classification performance of the methods of Ours and Ours (wo corr) with six augmentation ratios, including 0.5, 1, 2, 3, 4, and 5. The augmentation ratio of 2 means that the size of the generated data is twice the size of the original training set. Other ratios are represented in the same way. The experimental results on the datasets of eNTERFACE'05, RAVDESS, and CMEW are shown in Tables 4-6, respectively. Figure 5: The confusion matrices of different methods on the CMEW dataset. (**a**) Baseline (A+V), (**b**) Ours (wo corr), (**c**) Ours.