# Causal Inference for Modality Debiasing in Multimodal Emotion Recognition
###### Abstract  
Multimodal emotion recognition (MER) aims to enhance the understanding of human emotions by integrating visual, auditory, and textual modalities. However, previous MER approaches often depend on a dominant modality rather than considering all modalities, leading to poor generalization. To address this, we propose Causal Inference in Multimodal Emotion Recognition (CausalMER), which leverages counterfactual reasoning and causal graphs to capture relationships between modalities and reduce direct modality effects contributing to bias. This allows CausalMER to make unbiased predictions while being easily applied to existing MER methods in a model-agnostic manner, without requiring any architectural modifications. We evaluate CausalMER on the IEMOCAP and CMU-MOSEI datasets, widely used benchmarks in MER, and compare it with existing methods. On the IEMOCAP dataset with the MulT backbone, CausalMER achieves an average accuracy of 83.4%. On the CMU-MOSEI dataset, the average accuracies with MulT, PMR, and DMD backbones are 50.1%, 48.8%, and 48.8%, respectively. Experimental results demonstrate that CausalMER is robust in missing modality scenarios, as shown by its low standard deviation in performance drop gaps. Additionally, we evaluate modality contributions and show that CausalMER achieves balanced contributions from each modality, effectively mitigating direct biases from individual modalities. emotion recognition; multimodal learning; causal inference +
Footnote †: journal: _Applied_  
0000-0001-5000-0000-0000-0000-0000-0000-0000-0000-0000-00000-0000-0000  
heavily rely on a dominant modality struggle to adapt, limiting their ability to handle incomplete or missing data effectively. As shown in the first column of Figure 1, this often leads to poor generalization, as the model becomes less capable of adapting to variations in input or missing modalities [6]. Many multimodal models tend to rely heavily on the language modality [6]. Previous works [9; 10; 11] have addressed the issue of modality bias in multimodal learning, particularly focusing on the role of the language modality. While these methods propose to mitigate language modality bias, they often overlook biases present in other modalities, such as video and audio data. In many multimodal datasets, the importance of a particular modality can vary at the sample level and is not always dominant in every situation [12]. Existing methods effectively handle language bias but struggle to address biases in audio or video modalities. Additionally, these methods may encounter difficulties when certain modalities, like video or audio, are missing or imbalanced in the input data. Therefore, it is crucial to account for biases in all modalities--language, video, and audio--when designing multimodal systems to ensure robust performance across various scenarios. By leveraging counterfactual reasoning, we can disentangle various effects in multimodal learning, enabling effective modality debiasing. Motivated by causal inference and counterfactual reasoning [13; 14], we propose a novel approach called **Causal** Inference in **M**ultimodal **E**motion **R**ecognition (CausalMER) to reduce bias in text, audio, and video modalities. CausalMER can be easily applied to existing MER methods in a model-agnostic manner without requiring any architectural modifications. We construct a causal graph to capture the relationships between modalities, allowing us to understand how different modalities influence the training process. Using this causal graph, CausalMER estimates the direct modality effects caused by modality bias. During inference, CausalMER calculates the total indirect effect by subtracting the direct modality effects from the total causal effect,  
Figure 1: **Problem Definition. ** If the model is debiased considering only language, it cannot effectively handle modality-biased samples where audio and video data are missing, as it fails to address biases in the other modalities. effectively eliminating direct modality bias.