# Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN
## 5 Conclusions
It does not consider combining information from other modalities, such as text [107] and physiological signals [108; 109], which also contribute to emotion recognition. (2) Our architecture focuses on designing a multimodal condition GAN for data augmentation. Previous works such as Attention Mechanism, CCA, and Tensor Fusion are not considered to be integrated in our framework. As shown in Section 4.4, these previous works can also perform emotion recognition to a certain extent. (3) The labels of all samples are assumed to be correctly annotated to train our multimodal architecture. However, in reality, the data collection may suffer from label noise [110; 111; 112]. To further validate the effectiveness of our approach, we will tackle the above limitations in the future. Firstly, we will generalize our architecture to utilize more modalities. Secondly, we will combine our approach with the above previous works to further improve the performance of our framework. Finally, we will consider coping with label noise in our architecture. Conceptualization, F.M. and Y.L. ; methodology, F.M. and Y.L. ; software, F.M. ; validation, F.M. and Y.L. ; formal analysis, F.M., Y.L. and S.N. ; investigation, F.M. ; resources, Y.L., S.N., S.-L.H. and L.Z. ; data curation, F.M. ; writing--original draft preparation, F.M. ; writing--review and editing, F.M., Y.L., S.N. and S.-L.H. ; visualization, F.M. and Y.L. ; supervision, L.Z. ; project administration, Y.L., S.N. and S.-L.H. ; funding acquisition, S.N., S.-L.H. and L.Z. All authors have read and agreed to the published version of the manuscript. The work of [PERSON] was supported by the National Natural Science Foundation of China under Grant 62001266. The work of [PERSON] was supported by the National Key R&D Program of China under Grant 2020 YFC0833402. The work of [PERSON] was supported in part by the National Natural Science Foundation of China under Grant 61807021, in part by the Shenzhen Science and Technology Program under Grant KQTD20170810150821146. The work of [PERSON] was supported by the Shenzhen Science and Technology Program under Grant KQTD20170810150821146. We conducted experiments on three datasets, including eNTERFACE'05, RAVDESS, and CMEW. eNTERFACE'05 can be found in [[http://www.enteface.net/results/](http://www.enteface.net/results/)]([http://www.enteface.net/results/](http://www.enteface.net/results/)), accessed on 4 December 2021; RAVDESS can be found in [[https://smartlaboratory.org/ravdess/](https://smartlaboratory.org/ravdess/)]([https://smartlaboratory.org/ravdess/](https://smartlaboratory.org/ravdess/)), accessed on 4 December 2021; CMEW can be found in [[https://mace-cream.github.io/en/](https://mace-cream.github.io/en/)]([https://mace-cream.github.io/en/](https://mace-cream.github.io/en/)), accessed on 4 December 2021. The authors declare no conflict of interest. The following abbreviations are used in this manuscript:  
GANGenerative Adversarial Network  
HGRHirschfeld-Gebelein-Renyi  
DNNDeep Neural Network  
CNNConvolutional Neural Network  
MTCNNMulti-Task Cascaded Convolutional Neural Network  
CCACanonical Correlation Analysis