# Learning Better Representations for Audio-Visual Emotion Recognition with Common Information
## 5 Experiments
### Experimental Results
#### 5.3.4 Robustness Analysis on Missing Modality  
In our analysis above, we assume that audio and visual data are available during the testing stage. However, when generalizing the trained models to the real-world, we may encounter modality missing scenarios, which requires a good fusion model should perform well, even if missing modality occurs during the testing process. Motivated by this, we conduct the following study on the eNTERFACE'05 dataset in order to verify the robustness of our model in the testing process. In our architecture, we concatenate the feature \(\mathbf{f}\) of visual data and the feature \(\mathbf{g}\) of audio data into the fusion network for emotion classification. In order to represent the modality missing problem during testing stage, we set the feature of one modality to \(\mathbf{0}\) in order to indicate that this modality is missing, and then concatenate it with the feature of another modality to predict emotion. It should be noted that features of another modality is obtained by using the model we have trained. We consider three scenarios: only audio data are missing, only visual data are missing, and audio and visual data are missing. In each scenario, we, respectively, set 20%, 50%, and 80% of test data as missing. For the third scenario, the audio and visual data are missing at half of the missing rate, respectively. For example, the missing rate of 20% means that audio and visual data are missing with 10%, respectively. We compare the performance of three methods, \(L_{corr}\), \(L_{corr\_simple}\) and baseline. They have the same network structure, with two fully connected layers in the fusion network. For the \(L_{corr}\) and \(L_{corr\_simple}\) methods, we set \(\alpha=1\). The baseline method means that we only use classification loss for training. For each setting, we report the emotion recognition accuracy, which is shown in Figure 7. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multicolumn{2}{c}{**Method**} & \multicolumn{3}{c}{**Setting**} \\  & **One** & **Two** & **Four** \\ \hline \multicolumn{2}{c}{Baseline} & 95.83 & 96.07 & 96.53 \\ \hline \(\alpha=0.01\) & \(L_{corr\_simple}\) & 97.46 & 97.11 & 96.30 \\ \(L_{corr}\) & 96.53 & 97.23 & 96.76 \\ \hline \(\alpha=0.1\) & \(L_{corr\_simple}\) & 97.23 & 97.11 & 97.46 \\ \(L_{corr}\) & **97.57** & 97.23 & 96.88 \\ \hline \(\alpha=1\) & \(L_{corr\_simple}\) & 97.34 & 96.88 & 96.88 \\ \(L_{corr}\) & 97.23 & 96.99 & 97.11 \\ \hline \(\alpha=10\) & \(L_{corr\_simple}\) & 96.30 & 96.42 & 97.46 \\ \(L_{corr}\) & 97.11 & 96.65 & 96.99 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: The performance of audio-visual emotion recognition with different settings on the RAVDESS dataset. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multicolumn{2}{c}{**Method**} & \multicolumn{3}{c}{**Setting**} \\  & **One** & **Two** & **Four** \\ \hline \multicolumn{2}{c}{Baseline} & 64.56 & 64.05 & 64.30 \\ \hline \(\alpha=0.01\) & \(L_{corr\_simple}\) & 65.32 & 66.08 & 65.57 \\ \(L_{corr}\) & 66.58 & 64.30 & 65.32 \\ \hline \(\alpha=0.1\) & \(L_{corr\_simple}\) & 63.54 & 65.06 & 64.05 \\ \(L_{corr}\) & 65.06 & 65.57 & 66.58 \\ \hline \(\alpha=1\) & \(L_{corr\_simple}\) & 66.33 & 63.80 & 64.30 \\ \(L_{corr}\) & **67.59** & 65.32 & 64.56 \\ \hline \(\alpha=10\) & \(L_{corr\_simple}\) & 64.81 & 65.82 & 66.33 \\ \(L_{corr}\) & 67.34 & 65.82 & 65.32 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: The performance of audio-visual emotion recognition with different settings on the BAUM-1s dataset. From Figure 7, we have the following observations: (1) as the missing rate increases, the emotion recognition accuracies of \(L_{corr}\), \(L_{corr\_simple}\) and baseline methods decrease. (2) In the three data missing scenarios, the downward tendency of \(L_{corr\_simple}\) method with the increase of missing rate is basically consistent with that of \(L_{corr}\) method, but the performance of the \(L_{corr\_simple}\) method is always lower than that of the \(L_{corr}\) method.