# Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN
## 3 Methodology
### Proposed Multimodal Conditional GAN  
To train our multimodal condition GAN, we design loss functions from the following two aspects: (1) The high-dimensional multimodal data with audio and visual modalities should be generated appropriately; (2) The relationship between the audio modality and the visual modality in the generated multimodal data should be fully considered. In order to achieve these goals, we generate the audio modality and the visual modality simultaneously, and model their correlation in our framework. Specifically, to generate the visual modality in the multimodal data, the following loss function \(L_{V}(D_{V},G_{V})\) is considered:  
\[L_{V}(D_{V},G_{V})=\mathbb{E}[\log D_{V}(\mathbf{x}_{V}|y)]+\mathbb{E}[\log(1 -D_{V}(G_{V}(\mathbf{z}_{V}|y)))] \tag{1}\]  
where \(\mathbb{E}\) represents the expectation. The generator \(G_{V}\) takes the noise vector \(\mathbf{z}_{V}\) and the category label \(y\) as input, and outputs the fake visual modality. The discriminator \(D_{V}\) can be regarded as a binary classifier. It takes the real or fake visual modality as the input together with the corresponding category label, and outputs a single scalar to distinguish whether the input visual modality is real or fake. \(G_{V}\) and \(D_{V}\) are optimized adversarially. Similarly, to generate the audio modality, we consider the following loss function \(L_{A}(D_{A},G_{A})\):  
\[L_{A}(D_{A},G_{A})=\mathbb{E}[\log D_{A}(\mathbf{x}_{A}|y)]+\mathbb{E}[\log(1 -D_{A}(G_{A}(\mathbf{z}_{A}|y)))] \tag{2}\]It is worth pointing out that the category label is sent to \(G_{V}\) and \(G_{A}\) as the shared input. In this way, audio and visual modalities in the multimodal data are generated simultaneously, where they share the same emotional information.  
In addition, it is shown in [17; 47] that the audio modality and the visual modality in the real multimodal data are correlated, which contributes to emotional recognition. This factor inspires us to consider the relationship between different modalities in the generated multimodal data. Here, we adopt a novel correlation loss function based on the Hirschfeld-Gebelein-Renyi (HGR) maximal correlation [50; 51; 52] to capture the correlation between the audio modality and the visual modality:  
\[\begin{split} L_{corr}(G_{A},G_{V})&=-\mathbb{E}[G _{A}(\mathbf{z}_{A}|y)^{\text{T}}(G_{V}(\mathbf{z}_{V}|y)]\\ &+\frac{1}{2}\text{tr}(\text{cov}(G_{A}(\mathbf{z}_{A}|y))\text{ cov}(G_{V}(\mathbf{z}_{V}|y)))\end{split} \tag{3}\]  
where the covariance matrix \(\text{cov}(G_{A}(\mathbf{z}_{A}|y))=\mathbb{E}[G_{A}(\mathbf{z}_{A}|y)G_{A}( \mathbf{z}_{A}|y)^{T}]-\mathbb{E}[G_{A}(\mathbf{z}_{A}|y)]\cdot\mathbb{E}[G_{ A}(\mathbf{z}_{A}|y)]^{T}\), \(\text{cov}(G_{V}(\mathbf{z}_{V}|y))\) is formulated in the same way, and \(\text{tr}(\cdot)\) represents the trace operator. By optimizing the correlation function \(L_{corr}\) in our framework, we relate the audio modality and the visual modality in the generated multimodal data with high dependence, which makes the generated multimodal data as realistic as possible to help improve the performance of emotion recognition.  
Finally, we use the following loss function to train our multimodal conditional GAN:  
\[G_{A}^{*},G_{V}^{*}=\arg\min_{G_{A}G_{V}}\max_{D_{A}D_{V}}L_{A}+L_{V}+\lambda \cdot L_{corr} \tag{4}\]  
where \(\lambda\) is a coefficient to measure the importance of the correlation loss in our framework. Different modules are trained simultaneously. After training, optimized generators \(G_{A}^{*}\) and \(G_{V}^{*}\) are used to generate new multimodal data with audio and visual modalities for data augmentation. The procedure to update the parameters of our multimodal conditional GAN is summarized in Algorithm 1.