# Causal Inference for Modality Debiasing in Multimodal Emotion Recognition
## 4 Proposed Method: CausalMER
### Implementation of CausalMER  
As illustrated in Figure 4, we implement CausalMER framework based on the causal graph depicted in Figure 3b. Figure 4: **The framework of CausalMER. ** CausalMER aims to eliminate individual modality bias through counterfactual reasoning. By subtracting the bias from text, audio, and video modalities, CausalMER effectively generates unbiased predictions. **Model Architecture. ** To determine the effects of text \(T\), audio \(A\), video \(V\), and multimodal \(M\) on \(Y\), we utilize four neural models: the text-only branch \(\mathcal{F}_{T}\) (e.g., ALBERT [29]), the audio-only branch \(\mathcal{F}_{A}\) (e.g., ResNet18 [30]), the video-only branch \(\mathcal{F}_{V}\) (e.g., ResNet18 [30]), and multimodal branch \(\mathcal{F}_{M}\). Since the proposed method is model-agnostic, various existing MER approaches can be utilized in the multimodal branch \(\mathcal{F}_{M}\). As detailed in Section 5.3, we use MulT [20], PMR [23], and DMD [24] as the MER backbones for \(\mathcal{F}_{M}\). Our methods are defined as follows:  
\[Y_{t}=\mathcal{F}_{T}(t),Y_{a}=\mathcal{F}_{A}(a),Y_{v}= \mathcal{F}_{V}(v),Y_{m}=\mathcal{F}_{M}(t,a,v),\] \[Y_{t,a,v,m}=h(Y_{t},Y_{a},Y_{v},Y_{m}). \tag{12}\]  
where \(Y_{t}\), \(Y_{a}\), \(Y_{v}\), and \(Y_{m}\) represent the text-only effect (\(T\to Y\)), audio-only effect (\(A\to Y\)), video-only effect (\(V\to Y\)), and the multimodal effect (\(T/A/V\to M\to Y\)), respectively. \(h\) is the fusion function to obtain the final score \(Y_{t,a,v,m}\). **Counterfactual Scenario. ** The no-treatment condition is defined as blocking the direct effect of the single modality--text, audio, or video--i.e., when \(t\), \(a\), or \(v\) is not provided. These no-treatment conditions are expressed as \(T=t^{*}=\emptyset\), \(A=a^{*}=\emptyset\), and \(V=v^{*}=\emptyset\). However, neural models are unable to handle scenarios where inputs are absent. Therefore, we established the no-treatment condition with the approach used in this study [13]. The terms \(Y_{t}\), \(Y_{a}\), \(Y_{v}\), and \(Y_{m}\) in Equation (12) are expressed as follows:  
\[Y_{t}= \begin{cases}y_{t}=\mathcal{F}_{T}(t)&\text{if }T=t\\ y_{t}^{*}=c_{t}&\text{if }T=\emptyset\end{cases} ^{\prime}\] \[Y_{a}= \begin{cases}y_{a}=\mathcal{F}_{A}(a)&\text{if }A=a\\ y_{a}^{*}=c_{a}&\text{if }A=\emptyset\end{cases} ^{\prime}\] \[Y_{v}= \begin{cases}y_{v}=\mathcal{F}_{V}(v)&\text{if }V=vs.\\ y_{v}^{*}=c_{v}&\text{if }V=\emptyset\end{cases} ^{\prime},\]  
\[Y_{m}= \begin{cases}y_{m}=\mathcal{F}_{M}(t,a,v)&\text{if }T=t\text{ and }A=a\text{ and }V=vs.\\ y_{m}^{*}=\text{sum}(c_{t},c_{a},c_{v})&\text{if }T=\emptyset\text{ or }A=\emptyset\text{ or }V=\emptyset\end{cases} . \tag{13}\]  
where \(c_{t}\), \(c_{a}\), and \(c_{v}\) are learnable parameters. **Fusion Strategies. ** We combined \(Y_{t},Y_{a},Y_{v}\), and \(Y_{m}\) to derive a fused score \(Y_{t,a,v,m}\), which can be applied to various fusion strategies. The differences between these strategies are compared in Section 5.5. * SUM [13] calculates the final score by summing the four probabilities, where \(\sigma\) is the sigmoid function. \[h(Y_{t},Y_{a},Y_{v},Y_{m})=\log\sigma(Y_{t}+Y_{a}+Y_{v}+Y_{m}).\] (14)
* HM (Harmonic) [13] utilizes the product of sigmoid functions as the fusion method. \[h(Y_{t},Y_{a},Y_{v},Y_{m})=\log\frac{Y_{\text{HM}}}{1+Y_{\text{HM}}},\] (15) \[\text{where }Y_{\text{HM}}=\sigma(Y_{t})\cdot\sigma(Y_{a})\cdot \sigma(Y_{v})\cdot\sigma(Y_{m}).\]
* SUM-tanh [9] uses the sum of tanh function, combined with \(Y_{m}\) as the fusion function. \[h(Y_{t},Y_{a},Y_{v},Y_{m})=Y_{m}+tanh(Y_{t})+tanh(Y_{a})+tanh(Y_{v}).\] (16)MASK [14] uses the sigmoid function of the sum of \(Y_{t},Y_{a}\), and \(Y_{v}\) as the mask. \[h(Y_{t},Y_{a},Y_{v},Y_{m})=Y_{m}\cdot\sigma(Y_{t}+Y_{a}+Y_{v}) \tag{17}\]  
**Training.