# Causal Inference for Modality Debiasing in Multimodal Emotion Recognition
## 4 Proposed Method: CausalMER
### Counterfactual Inference at CausalMER  
Our proposed method aims to mitigate the impact of harmful bias on model predictions when a modality is missing. We represent an emotion \(Y\) (e.g., \"happy\") as the score obtained when \(T\) is set to \(t\) (e.g., the text \"I am so happy\"), \(A\) is set to \(a\) (e.g., a voice saying \"I am so happy\"), and \(V\) is set to \(v\) (e.g., a video showing people saying \"I am so happy\"). Using the counterfactual notations in Equation (1), the causality in the factual scenarios is formulated as follows:  
\[Y_{t,a,v,m}=Y(T=t,A=a,V=v,M_{t,a,v}=(T=t,A=a,V=v)). \tag{5}\]  
The total effect (\(TE\)) of _CausalMER_ is illustrated in Figure 3. Following the causal effect defined in Equation (2), the TE of \(T=t\), \(A=a\), and \(V=v\) on \(Y=y\) is defined as:  
\[TE=Y_{t,a,v,m}-Y_{t^{*},a^{*},v^{*},m^{*}}. \tag{6}\]  
where \(m^{*}=M_{t^{*},a^{*},v^{*}}\). Here, \(t^{*}\), \(a^{*}\), and \(v^{*}\) correspond to the no-treatment condition, representing a counterfactual scenario where the input from specific modalities \(t\), \(a\), and \(v\) is blocked. This allows the measurement of the remaining effects, excluding the direct impact. Further details are provided in Section 4.3, Counterfactual Scenario. In real-world scenarios, it is impossible to predict which modality might be missing. If the model is heavily biased toward a particular modality that is absent, its performance and generalization ability will degrade. Additionally, MER models are prone to spurious correlations between individual modalities and the target emotional labels, preventing the effective use of multimodal knowledge. To address this, we aim to eliminate the direct effect of individual modalities on the emotions. Specifically, we estimate the causal effect of \(T=t\), \(A=a\), and \(V=v\) on \(Y=y\) by blocking the impact of \(M\). Figure 3b illustrates the natural direct effects (_NDEs_) of each modality--text, audio, and video--as proposed in our method. To calculate the direct effect of the text modality \(T\), we apply an intervention [27] that isolates and measures the _NDE_ of \(T\) on \(Y\). This process involves assessing the impact of the text modality in the direct effect while excluding its contribution to the indirect effect. When \(T=t\), \(A=a\), and \(V=v\), the intervention sets the multimodal representation \(M\) to a  
Figure 3: **Proposed Method: CausalMER. ** (**a**) Causal graph for MER. \(T\) represents text, \(A\) represents audio, \(V\) represents video, \(M\) denotes multimodal knowledge, and \(Y\) is the predicted emotion. (**b**) Counterfactual notations for CausalMER. White nodes represent the values \(T=t\), \(A=a\), and \(V=v\), while gray nodes correspond to the values \(T=t^{*}\), \(A=a^{*}\), and \(V=v^{*}\). Here, \({}^{*}\) indicates a counterfactual scenario. specific value \(m\), effectively blocking the mediator \(M\) from responding to its inputs. As a result, when the model predicts \(Y\), it relies solely on the text modality \(T\). For the indirect effect, a no-treatment condition \(t^{*}\) is applied. Since the effect of \(T\) is blocked for the mediator \(M\) with \(M\) taking the value \(m^{*}=M_{t^{*},a^{*},a^{*}}\), the model explicitly captures the text bias. \[NDE_{t}=Y_{t,a^{*},a^{*},m^{*}}-Y_{t^{*},a^{*},a^{*},m^{*}}. \tag{7}\]  
Similarly, the \(NDE\) of \(A\) on \(Y\) is calculated as follows:  
\[NDE_{a}=Y_{t^{*},a,a^{*},m^{*}}-Y_{t^{*},a^{*},a^{*},m^{*}}. \tag{8}\]  
where \(NDE_{a}\) explicitly captures audio modality bias by isolating the direct effect of \(A\) while preventing its impact on the intermediate variable \(M\). The \(NDE\) of \(V\) on \(Y\) is calculated as:  
\[NDE_{v}=Y_{t^{*},a^{*},m^{*}}-Y_{t^{*},a^{*},a^{*},a^{*},m^{*}}. \tag{9}\]  
where \(NDE_{v}\) captures video modality bias in a similar manner to \(NDE_{a}\) by isolating the direct effect of \(V\) while preventing its influence on \(M\).