# Causal Inference for Modality Debiasing in Multimodal Emotion Recognition
## 5 Experiments
### Main Results  
We conducted two main experiments: a standard benchmark in MER and an evaluation of missing modality scenarios on the IEMOCAP [31] and CMU-MOSEI [32] datasets. We aim to assess the efficiency of our CausalMER in a standard benchmark setting and evaluate its robustness in scenarios where a modality is missing. In real-world situations, \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & \multicolumn{3}{c}{**Samples**} & \multirow{2}{*}{**Classes**} & \multirow{2}{*}{**Metric**} \\ \cline{2-2} \cline{4-6}  & **Train** & & & **Val** & \\ \hline IEMOCAP [31] & 2717 & 798 & 938 & 4 & \\ CMU-MOSEI [32] & 16,265 & 1869 & 4643 & 7 & Accuracy (Acc), F1 score (F1) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Statistics and evaluation metrics of experimental datasets. the absence of certain modalities can occur, and our CausalMER is designed to address this issue by preventing reliance on any single modality. CausalMER is evaluated against recent state-of-the-art (SOTA) approaches in MER, with the methods grouped based on the dataset. On the IEMOCAP [31] dataset, we compare CausalMER with EF-LSTM [20], LF-LSTM [20], RMFN [38], MFM [39], RAVEN [33], MCTN [40], MulT [20], PMR [23], FDMER [22], and DMD [24]. For the CMU-MOSEI dataset, we compared our method with DCCA [41] and DCCAE [42], which address missing modalities in a non-recovery manner, and MCTN [40], MMIN [43], GCNet [44], DiCMoR [45], and IMDer [46]. Similar to previous studies [47; 48] that have re-run MulT, PMR, and DMD, we reimplemented it following the settings described in the original paper. We report the best performance achieved in our experiments with the configurations provided by the original authors. Table 2 illustrates the comparison on the IEMOCAP [31] dataset. We integrated the CausalMER framework into MulT [20], one of the most well-recognized models in the MER task. As a result, MulT [20] integrated with CausalMER outperforms the vanilla model in terms of both accuracy and F1 score across all emotions. In particular, the model with CausalMER achieves better results, with improvements of 1.6% in average emotion accuracy and 1.9% in F1 score. This improvement may be attributed to the fact that the original MulT [20] tends to rely on certain single modalities, which reduces the contributions of other less dominant modalities. CausalMER helps address this imbalance. In contrast, CausalMER shows performance improvements, effectively addressing modality bias and balancing modality contributions. Table 3 presents the standard benchmarks for the IEMOCAP dataset and the performance drop observed when the available modalities are limited, i.e., when certain modalities are missing. We use MulT [20] as the backbone for CausalMER. \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{**Happy**} & \multicolumn{2}{c}{**Sad**} & \multicolumn{2}{c}{**Angry**} & \multicolumn{2}{c}{**Neutral**} & \multicolumn{2}{c}{**Average**} \\  & **Acc (\%)** & **F1 (\%)** & **Acc (\%)** & **F1 (\%)** & **Acc (\%)** & **F1 (\%)** & **Acc (\%)** & **F1 (\%)** & **Acc (\%)** & **F1 (\%)** \\ \hline EF-LSTM [20] & 86.0 & 84.2 & 80.2 & 80.5 & 85.2 & 84.5 & 67.8 & 67.1 & 79.8 & 79.1 \\ LF-LSTM [20] & 85.1 & 86.3 & 78.9 & 81.7 & 84.7 & 83.0 & 67.1 & 67.6 & 79.0 & 79.7 \\ RMFN [38] & 87.5 & 85.8 & 83.8 & 82.9 & 85.1 & 84.6 & 69.5 & 69.1 & 81.5 & 80.6 \\ MFM [39] & 90.2 & 85.8 & 88.4 & 86.1 & 87.5 & 86.7 & 72.1 & 68.1 & 84.6 & 81.7 \\ RAVEN [33] & 87.3 & 85.8 & 83.4 & 83.1 & 87.3 & 86.7 & 69.7 & 69.3 & 81.9 & 81.2 \\ MCTN [40] & 84.9 & 83.1 & 80.5 & 79.6 & 79.7 & 80.4 & 62.3 & 57.0 & 76.9 & 75.0 \\ MulT [20] & 90.7 & 88.6 & 86.7 & 86.0 & 87.4 & 87.0 & 72.4 & 70.7 & 84.3 & 83.1 \\ \hline MulT + [20] & 85.6 & 82.3 & 84.0 & 84.4 & 88.1 & 87.8 & 69.5 & 69.5 & 81.8 & 81.2 \\ + CausalMER & 87.3 & 86.0 & 87.3 & 87.1 & 88.3 & 88.2 & 70.9 & 71.1 & 83.4 & 83.1 \\ (Difference) & +1.7 & +3.7 & +3.3 & +2.7 & +0.2 & +0.4 & +1.4 & +1.6 & +1.6 & +1.9 \\ \hline \hline \end{tabular}
\end{table}