# Causal Inference for Modality Debiasing in Multimodal Emotion Recognition
## 3 Preliminaries  
In this section, we introduce the key concepts and notations related to causal inference, referencing the work [13]. We denote random variables with capital letters and their observed values with lowercase letters.  
**Causal graph.** A causal graph is a probabilistic graphical model that reflects the causal relationships among variables. It is represented as a directed acyclic graph \(\mathcal{G}=\{\mathcal{N},\mathcal{E}\}\), where \(\mathcal{N}\) denotes the set of variables and \(\mathcal{E}\) represents the corresponding causal relationships. Figure 2a illustrates an example of a causal graph with three variables. The causality from cause variable \(X\) to effect variable \(Y\) consists of two parts: the direct effect (i.e., \(X\to Y\)) and the indirect effect, which occurs through the mediator variable \(M\) (i.e., \(X\to M\to Y\)). For example, a causal graph illustrating the effect of medication on disease treatment can be represented as follows: \(X=x\) refers to the medication, \(Y\) represents the disease treatment outcome, and \(M\) includes other factors like the patient's age, physical condition, and other relevant variables.  
**Counterfactual inference.** Counterfactual inference identifies the causal effect of a variable by answering hypothetical \"what if\" questions based on observed data [25]. In the factual scenario, the value of \(Y\) is formalized as:  
\[Y_{x,m} =Y_{x,M_{x}}=Y(X=x,M=m), \tag{1}\] \[m =M_{x}=M(X=x).\]  
where \(X\) is set to \(x\) and \(M\) is set to \(m\). In the counterfactual scenario, \(X\) is set to different values to assess how this affects both \(M\) and \(Y\). As shown in Figure 2b, when \(X\) is set to \(x^{*}\), the mediator \(M\) changes, and we obtain \(Y_{x^{*},m^{*}}=Y_{x^{*},M_{x^{*}}}=Y(X=x^{*},M=M(X=x^{*}))\). Similarly, \(Y_{x,m^{*}}\) describes the counterfactual scenario where \(X=x\) and \(m^{*}\) is set to the value when \(X=x^{*}\).  
**Causal effects.** Causal effects reveal the comparisons between two corresponding outcomes when the value of the reference variable changes [27]. For example, let \(X=x\) denote the \"taken medication\" scenario and \(X=x^{*}\) represent the \"not taken medication\" scenario. According to causal theory [27], the total effect (\(TE\)) of \(X=x\) on \(Y\) compares two hypothetical situations \(X=x\) and \(X=x^{*}\) and is formulated as:  
\[TE=Y_{x,M_{x}}-Y_{x^{*},m^{*}}. \tag{2}\]  
Figure 2: **Causal graph. (a) Example of a causal graph. (b) Examples of counterfactual notations. White nodes represent the value \(X=x\), while gray nodes correspond to the value \(X=x^{*}\). Here, \(x^{*}\) indicates a counterfactual scenario.**  
The total effect can be divided into two components [28]: the natural direct effect (_NDE_) and the total indirect effect (_TIE_). _NDE_ represents the effect of \(X\) on \(Y\) while the mediator \(M\) is held blocked. Therefore, the _NDE_ reflects the direct impact of \(X\) on \(Y\), excluding any indirect effects mediated by \(M\), with \(M\) set to the value it would have when \(X=x^{*}\):  
\[NDE=Y_{x,M_{x^{*}}}-Y_{x^{*},M_{x^{*}}}. \tag{3}\]  
_TIE_ is calculated by subtracting the _NDE_ from _TE_[28]:  
\[TIE=TE-NDE=Y_{x,M_{x}}-Y_{x,M_{x^{*}}}. \tag{4}\]  
Finally, we obtain \(TIE\) as the debiased result by eliminating the direct bias (_NDE_) and use it as the model's final output. Unlike conventional MER methods that derive outputs solely from \(t,a\), and \(v\), CausalMER leverages \(TIE\) to effectively eliminate direct biases.