SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\widehat {M} _ {s} ^ {\mathbf {w}} (\phi , \xi) := \mathbb {E} _ {\mathcal {S} ^ {n}} \mathbb {E} _ {\mathbf {c} \sim \hat {P} _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x})} \mathbb {E} _ {\bar {\mathbf {c}} \sim \hat {P} _ {s} ^ {\xi} (\bar {\mathbf {C}} | \mathbf {X} = \mathbf {x})} \mathrm {I} [ \operatorname {s i g n} (\mathbf {w} ^ {\top} \mathbf {c}) = \operatorname {s i g n} (\mathbf {w} ^ {\top} \bar {\mathbf {c}}) ],
$$
where  $\hat{P}_s^\phi (\mathbf{C}|\mathbf{X} = \mathbf{x})$  and  $\hat{P}_s^\xi (\bar{\mathbf{C}} |\mathbf{X} = \mathbf{x})$  describe the estimated distribution on dataset  $S^n$ .
Then, we use PAC-learning (Shalev-Shwartz & Ben-David, 2014) tools to formulate the upper bound of gap between empirical risk and expected risk as a theorem below.
Theorem 3.3. Given parameters  $\phi, \xi$ , for any  $\mathbf{w}: \mathbb{R}^d \to \mathcal{Y}$ , prior distribution  $\pi_{\mathbf{C}} := P_s(\mathbf{C})$  and  $\pi_{\bar{\mathbf{C}}} := P_s(\bar{\mathbf{C}})$  which make  $\mathbb{E}_{\mathcal{S}}\mathrm{KL}(P_s^\phi (\mathbf{C}|\mathbf{X} = \mathbf{x})\| \pi_{\mathbf{C}})$  and  $\mathbb{E}_{\mathcal{S}}\mathrm{KL}(P_s^\xi (\bar{\mathbf{C}} |\bar{\mathbf{X}} = \bar{\mathbf{x}})\| \pi_{\bar{\mathbf{C}}})$  both lower than a positive constant  $C$ , then with a probability at least  $1 - \epsilon$  over source domain data  $S_n$ ,
(1)  $|SF_s(\mathbf{w},\phi) - \widehat{SF}_s(\mathbf{w},\phi)|$  is upper bounded by
$$
\mathbb {E} _ {\mathcal {S} ^ {n}} \mathrm {K L} \left(\hat {P} _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x}) \| \pi_ {\mathbf {C}}\right) + \frac {\ln (n / \epsilon)}{4 (n - 1)} + C.
$$
(2)  $\left| M_s^{\mathbf{w}}(\phi, \xi) - \widehat{M_s^{\mathbf{w}}}(\phi, \xi) \right|$  is upper bounded by
$$
\mathbb {E} _ {\mathcal {S} ^ {n}} \mathrm {K L} (\hat {P} _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x}) \| \pi_ {\mathbf {C}}) + \mathbb {E} _ {\mathcal {S} ^ {n}} \mathrm {K L} (\hat {P} _ {s} ^ {\xi} (\bar {\mathbf {C}} | \mathbf {X} = \mathbf {x}) \| \pi_ {\bar {\mathbf {C}}}) + \frac {\ln (n / \epsilon)}{4 (n - 1)} + 2 C.
$$
Theorem 3.3 demonstrates that as the sample size increases and the terms with KL divergence decrease, the empirical risk on the source domain dataset becomes closer to the expected risk. Combining Theorems 3.2 and 3.3, we can evaluate the expected PNS risk on the test distribution using the empirical risk on the source dataset. In the next section, we present a representation learning objective based on the results of Theorems 3.2 and 3.3 and introduce the satisfaction of Exogeneity.
4 Learning to Minimizing PNS Risk
In this section, we propose a learning objective built upon the PNS risk that is used to capture the essential representation having a high PNS value from observational data.
4.1 The Semantic Separability of PNS
In Section 3, we present PNS risk and Monotonicity measurement. Furthermore, to ensure that finding interpretable representations is feasible, we need to make certain assumptions that the representation of the data retains its semantic meaning under minor perturbations. Specifically, we define the variable  $\mathbf{C}$  as Semantic Separability relative to  $Y$  if and only if the following assumption is satisfied:
Assumption 4.1 ( $\delta$ -Semantic Separability). For any domain index  $d \in \{s, t\}$ , the variable  $\mathbf{C}$  is  $\delta$ -semantic separable, if for any  $\mathbf{c} \sim P_d(\mathbf{C}|Y = y)$  and  $\bar{\mathbf{c}} \sim P_d(\mathbf{C}|Y \neq y)$ , the following inequality holds almost surely:  $\|\bar{\mathbf{c}} - \mathbf{c}\|_2 > \delta$ .
$\delta$ -Semantic Separability refers to the semantic meaning being distinguishable between  $\mathbf{c}$  and  $\overline{\mathbf{c}}$  when the distance between them is large enough, i.e.,  $\| \overline{\mathbf{c}} - \mathbf{c} \|_2 > \delta$ . This assumption is widely accepted because, without it, nearly identical values would correspond to entirely different semantic information, leading to inherently unstable and chaotic data. If  $\mathbf{C}$  satisfies Assumption 4.1, then considering the PNS value in a small intervention, such as  $\| \mathbf{c} - \overline{\mathbf{c}} \|_2 < \delta$ , will lead failure in representation learning. Therefore, during the learning process, we add the penalty of  $\| \mathbf{c} - \overline{\mathbf{c}} \|_2 > \delta$ .
4.2 Overall Objective
Depending on the diverse selections of  $P^{\xi}(\bar{\mathbf{C}} |\mathbf{X} = \mathbf{x})$ , there are multiple potential PNS risks. In the learning process, we consider minimizing the risk in the worst-case scenario lead by  $\bar{\mathbf{C}}$ , i.e., the maximal PNS risk lead by the selection of  $P^{\xi}(\bar{\mathbf{C}} |\mathbf{X} = \mathbf{x})$ . Minimizing the upper bounds in Theorems 3.2 and 3.3 can be simulated by the following optimization process
$$
\min  _ {\phi , \mathbf {w}} \max  _ {\xi} \widehat {M} _ {s} ^ {\mathbf {w}} (\phi , \xi) + \widehat {S F} _ {s} (\mathbf {w}, \phi) + \lambda L _ {\mathrm {K L}}, \quad \text {s u b j e c t t o} \| \mathbf {c} - \overline {{\mathbf {c}}} \| _ {2} > \delta , \tag {8}
$$
where  $L_{\mathrm{KL}} \coloneqq \mathbb{E}_{\mathcal{S}^n}\mathrm{KL}(\hat{P}_s^\phi (\mathbf{C}|\mathbf{X} = \mathbf{x})\| \pi_{\mathbf{C}})) + \mathbb{E}_{\mathcal{S}^n}\mathrm{KL}(\hat{P}_s^\xi (\bar{\mathbf{C}} |\mathbf{X} = \mathbf{x})\| \pi_{\bar{\mathbf{C}}}))$ . The constraint  $\| \mathbf{c} - \bar{\mathbf{c}} \|_2 > \delta$  is set because of the Semantic Separability assumption. We name the algorithm of optimizing Eq. (8) as CaSN (Causal Representation of Sufficiency and Necessity).
4.3 Satisfaction of Exogeneity
In the previous sections, we introduced an objective to satisfy monotonicity. Identifying PNS values not only needs to satisfy monotonicity but also exogeneity. In this part, we discuss the satisfaction of Exogeneity and provide the solution to find the representation under three causal assumptions below.
Assumption 4.2. The Exogeneity of  $\mathbf{C}$  holds, if and only if the following invariant conditions are satisfied separately under three causal assumptions in Figure 1(b): (1)  $\mathbf{X} \perp Y|\mathbf{C}$  (2)  $\mathbf{C} \perp V$  (3)  $V \perp Y|\mathbf{C}$  (for assumption in Figure 1(b).1, 2 and 3, respectively).
The above three assumptions are commonly accepted by the OOD generalization (Lu et al., 2021; Liu et al., 2021a; Ahuja et al., 2021). To satisfy the Exogeneity, we use different objective functions to identify C over three invariant causal assumptions. For Assumption 4.2 (1), we provide the following theorem showing the equivalence between optimizing Eq.8 and identifying invariant representation.
Theorem 4.3. The optimal solution of learned  $\mathbf{C}$  is obtained by optimizing the following objective (the key part of the objective in Eq. (8))
$$
\min _ {\boldsymbol {\phi}, \mathbf {w}} \widehat {S F} _ {s} (\mathbf {w}, \boldsymbol {\phi}) + \lambda \mathbb {E} _ {\mathcal {S} ^ {n}} \mathrm {K L} (\hat {P} _ {s} ^ {\boldsymbol {\phi}} (\mathbf {C} | \mathbf {X} = \mathbf {x}) \| \pi_ {\mathbf {C}})
$$
satisfies the conditional independence  $\mathbf{X} \perp Y|\mathbf{C}$ .
Theorem 4.3, details of the proof are shown in Appendix E, indicates optimizing overall objective Eq. (8) implicitly makes  $\mathbf{C}$  satisfy the property of Exogeneity under causal assumption  $\mathbf{X} \perp Y|\mathbf{C}$ . For Assumption 4.2 (2), to identify the invariant assumption  $\mathbf{C} \perp V$  (Li et al., 2018b), we introduce the following Maximum Mean Discrepancy (MMD) penalty to the minimization process in Eq. (8),
$$
L _ {\mathrm {m m d}} = \sum_ {v _ {i}} \sum_ {v _ {j}} \mathbb {E} _ {\mathbf {x _ {i}} \sim P (\mathbf {X} | V = v _ {i})} \mathbb {E} _ {\mathbf {c _ {i}} \sim \hat {P} _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x _ {i}})} \mathbb {E} _ {\mathbf {x _ {j}} \sim P (\mathbf {X} | V = v _ {j})} \mathbb {E} _ {\mathbf {c _ {j}} \sim \hat {P} _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x _ {j}})} \| \mathbf {c _ {i}} - \mathbf {c _ {j}} \| _ {2}.
$$
For Assumption 4.2 (3), to specify the representation of  $\mathbf{C}$  and allows the Exogeneity when the assumption  $V\perp Y|\mathbf{C}$  holds, we introduce the IRM-based (Arjovsky et al., 2019) penalty into Eq.(8).
$$
L _ {\mathrm {i r m}} = \sum_ {v} E _ {(\mathbf {x}, y) \sim P _ {s} (\mathbf {X}, Y | V = v)} \left\| \nabla_ {w | w = 1. 0} \mathbb {E} _ {\mathbf {c} \sim \hat {P} _ {s} ^ {\phi} (\mathbf {C} | \mathbf {X} = \mathbf {x})} \mathrm {I} [ \mathrm {s i g n} (\mathbf {w} ^ {\top} \mathbf {c}) \neq y ] \right\| ^ {2}
$$
Noticeably, to address the issue of invariant learning and satisfy the Exogeneity under Assumptions 4.2 (2) and (3), it is necessary to introduce additional domain information, such as domain index.
In this section, we review the progress of OOD prediction tasks. A research perspective for OOD prediction is from a causality viewpoint (Zhou et al., 2021; Shen et al., 2021). Based on the postulate that the causal variables are invariant and less vulnerable to distribution shifts, a bunch of methods identify the invariant causal features behind the observation data by enforcing invariance in the learning process. Different works consider causality across multiple domains in different ways. One series of research called the causal inference-based methods model the invariance across domains by causality explanation, which builds a causal graph of data generative process (Pfister et al., 2019; Rothenhausler et al., 2018; Heinze-Deml et al., 2018; Gamella & Heinze-Deml, 2020; Oberst et al., 2021; Zhang et al., 2015). The other series of methods consider invariant learning from a causality aspect. They formulate the invariant causal mechanisms by representation rather than causal variables. Invariant risk minimization (IRM) methods (Arjovsky et al., 2019) provide a solution for learning invariant variables and functions. Under this viewpoint, some pioneering work (Ahuja et al., 2020a; Chen et al., 2022b; Krueger et al., 2021; Lu et al., 2021; Ahuja et al., 2021; Lin et al., 2022) further extend the IRM framework by considering game theory, variance penalization, information theory, nonlinear prediction functions, and some recent works apply the IRM framework to large neural networks (Jin et al., 2020; Gulrajani & Lopez-Paz, 2020). In this paper, different from the aforementioned works which consider to learn the invariant information, we think that information satisfying invariance is not enough to be most appropriate for the generalization task. We thus focus on learning to extract the most essential information from observations with a ground on the sufficient and necessary causal theorem. In the main text, we only provide a review of OOD prediction. We further elaborate on the correlation with other lines of work, such as domain adaptation, causal discovery, representation learning, causal disentanglement, and contrastive learning, in Appendix F.
In this section, we verify the effectiveness of CaSN using synthetic and real-world OOD datasets.
6.1 Setups
Synthetic data. The effectiveness of the proposed method is demonstrated by examining whether it can learn the essential information (i.e., sufficient and necessary causes) from source data. To this
(a) Spurious degree  $s = 0.1$
(c) Results of CaSN and the CaSN(-m)
Figure 2: The synthetic results for validating the property of learned representation under different spurious degrees in data,  $s = 0.1$  for (a) and  $s = 0.7$  for (b), the x-axis shows different causal information y-axis shows the choice of  $\delta$ . (c) The results of the feature identification when  $s = 0.7$ .
(b) Spurious degree  $s = 0.7$
end, based on the causal graph in Figure 1(b).1 we designed a synthetic data generator that produced a sample set  $\{\mathbf{x}_i\}_{i=1}^n$  with corresponding labels  $\{y_i\}_{i=1}^n$ . Four types of information were considered, including: (i) SN: Sufficient and Necessary Cause  $\mathbf{sn}_i$  of  $y_i$ . The value of  $y_i$  is directly calculated as  $y_i = \mathbf{sn}_i \oplus B(0.15)$ , where  $\oplus$  represents the XOR operation and  $B(0.15)$  is a Bernoulli distribution with a probability of 0.15 to generate 1. (ii) SF: sufficient and unnecessary cause  $\mathbf{sf}_i$  of  $y_i$ .  $\mathbf{sf}_i$  is a transformation of  $\mathbf{sn}_i$ . We set  $\mathbf{sf}_i = B(0.1)$  when  $\mathbf{sn}_i = 0$ , and  $\mathbf{sf}_i = \mathbf{sn}_i$  when  $\mathbf{sn}_i = 1$ . SF is designed to decrease the probability of necessity (i.e.  $P(Y = 0|\mathrm{SN} = 0)$ ). (iii) NC: insufficient and necessary cause  $\mathbf{nc}_i$  of  $y_i$ . We set  $\mathbf{nc}_i = \mathrm{I}(\mathbf{sn}_i = 1) \cdot B(0.9)$ . NC is designed to decrease the probability of sufficiency (i.e.  $P(Y = 1|\mathrm{SN} = 1)$ ). (iv) Spurious: spurious correlation information  $\mathbf{sp}_i$ . Spurious correlated information is generated by  $s * \mathbf{sn}_i * \mathbf{1}_d + (1 - s)\mathcal{N}(0,1)$ , where  $d$  denotes dimension and  $s$  denotes the spurious degree. When  $s$  gets higher, the spurious correlation becomes stronger in data  $\mathbf{x}$ . We select  $d = 5$  and  $s \in \{0.1, 0.7\}$ . In the synthetic generative process and develop a non-linear function to generate  $\mathbf{x}$  from  $[\mathbf{sn}_i, \mathbf{sf}_i, \mathbf{nc}_i, \mathbf{sp}_i]$ . We use Distance Correlation (Jones et al., 1995) as evaluation metrics to measure the correlation between the learned representation  $\mathbf{C}$  and ground information (i.e. SN, SF, NC, SP). We provide an ablation study of CaSN without the Monotonicity evaluator CaSN(-m) in comparison results, which evaluates the effectiveness of CaSN.
Performance on OOD prediction task. The proposed method CaSN is implemented based on codebase DomainBed (Gulrajani & Lopez-Paz, 2020). We provide three implementations of our method, which are CaSN, CaSN(irm) and CaSN(mmd) with the same architecture but using Eq.(8), Eq.(8)  $+L_{\mathrm{irm}}$  and Eq.(8)  $+L_{\mathrm{mmd}}$  as their final objective, respectively. We compare CaSN with several common baselines, including ERM(Vapnik, 1999), IRM (Arjovsky et al., 2019), GroupDRO (Sagawa et al., 2019), Mixup (Xu et al., 2020), MLDG (Li et al., 2018a), MMD (Li et al., 2018b), DANN (Ganin et al., 2016) and CDANN (Li et al., 2018c), where the best accuracy scores are directly given by training-domain-validation in Gulrajani & Lopez-Paz (2020). We test the performance on commonly used ColoredMnist (Ahuja et al., 2020a), PACS (Li et al., 2017), and VLCS (Fang et al., 2013) datasets. During the experiment process, we adjust the hyperparameters provided by DomainBed and extra hyperparameters  $\delta$  and  $\lambda$  in CaSN. The results show the mean and standard error of accuracy by executing the experiments randomly 2 times on 40 randomly selected hyperparameters. We also provide the extra experiments on large-scale spurious correlation dataset SpuCo (Joshi et al., 2023). Due to the page limitation, more experiment setups and results are provided in Appendix B.
6.2 Learning Sufficient and Necessary Causal Representations
We conducted experiments on synthetic data to verify the effectiveness of the learned representation. In experiments, we use single domain with different degrees of spurious correlation. The experiments aimed to demonstrate the properties of the learned representation and answer the following question:
Does CaSN capture the sufficient and necessary causes? We present the results in Figure 2 (a) and (b), which show the distance correlation between the learned representation and four ground truths: Sufficient and Necessary cause (SN, SF, NC and Spurious). A higher distance correlation indicates a better representation. From both Figure 2 (a) (b), we found that CaSN achieves higher distance correlations with the ground truths (e.g., SN, SF, and NC) and lower correlations with spurious factors compared to other methods. As an example, we consider Figure 2 (a) with  $\delta = 1.1$ . We obtain distance correlations of  $\{0.90, 0.65, 0.67, 0.13\}$  for SN, SF, NC, and spurious factors, respectively. We found that when we set  $\delta$  as a large value 1.1, CaSN captures more essential information SN.
Table 1: Results on PACS and VLCS dataset
Dataset	PACS	VLCS
Algorithm	A	C	P	S	Avg	Min	C	L	S	V	Avg	Min
ERM	84.7 ± 0.4	80.8 ± 0.6	97.2 ± 0.3	79.3 ± 1.0	85.5	79.3	97.7 ± 0.4	64.3 ± 0.9	73.4 ± 0.5	74.6 ± 1.3	77.5	64.3
IRM	84.8 ± 1.3	76.4 ± 1.1	96.7 ± 0.6	76.1 ± 1.0	83.5	76.4	98.6 ± 0.1	64.9 ± 0.9	73.4 ± 0.6	77.3 ± 0.9	78.5	64.9
GroupDRO	83.5 ± 0.9	79.1 ± 0.6	96.7 ± 0.3	78.3 ± 2.0	84.4	79.1	97.3 ± 0.3	63.4 ± 0.9	69.5 ± 0.8	76.7 ± 0.7	76.7	63.4
Mixup	86.1 ± 0.5	78.9 ± 0.8	97.6 ± 0.1	75.8 ± 1.8	84.6	78.9	98.3 ± 0.6	64.8 ± 1.0	72.1 ± 0.5	74.3 ± 0.8	77.4	64.8
MLDG	86.4 ± 0.8	77.4 ± 0.8	97.3 ± 0.4	73.5 ± 2.3	83.6	77.4	97.4 ± 0.2	65.2 ± 0.7	71.0 ± 1.4	75.3 ± 1.0	77.2	65.2
MMD	86.1 ± 1.4	79.4 ± 0.9	96.6 ± 0.2	76.5 ± 0.5	84.6	79.4	97.7 ± 0.1	64.0 ± 1.1	72.8 ± 0.2	75.3 ± 3.3	77.5	64.0
DANN	86.4 ± 0.8	77.4 ± 0.8	97.3 ± 0.4	73.5 ± 2.3	83.6	77.4	99.0 ± 0.3	65.1 ± 1.4	73.1 ± 0.3	77.2 ± 0.6	78.6	65.1
CDANN	84.6 ± 1.8	75.5 ± 0.9	96.8 ± 0.3	73.5 ± 0.6	82.6	75.5	97.1 ± 0.3	65.1 ± 1.2	70.7 ± 0.8	77.1 ± 1.5	77.5	65.1
CaSN (base)	87.1 ± 0.6	80.2 ± 0.6	96.2 ± 0.8	80.4 ± 0.2	86.0	80.2	97.5 ± 0.6	64.8 ± 1.9	70.2 ± 0.5	76.4 ± 1.7	77.2	64.8
CaSN (irm)	82.1 ± 0.3	77.9 ± 1.8	93.3 ± 0.8	80.6 ± 1.0	83.5	77.9	97.8 ± 0.3	65.7 ± 0.8	72.3 ± 0.4	77.0 ± 1.4	78.2	65.7
CaSN (mmd)	84.7 ± 0.1	81.4 ± 1.2	95.7 ± 0.2	80.2 ± 0.6	85.5	81.4	98.2 ± 0.7	65.9 ± 0.6	71.2 ± 0.3	76.9 ± 0.7	78.1	65.9
However, the result of CaSN decreases when  $\delta = 0.1$ , which suggests that CaSN tends to capture the most essential information when  $\delta$  is set to a larger value. This phenomenon aligns with Semantic Separability. We then compare Figure 2 (a) and (b). As an example, when  $\delta = 1.1$ , CaSN achieves distance correlations of 0.9 and 0.91 for SN on  $s = 0.1$  and  $s = 0.7$ , respectively. The distance correlation with spurious information is 0.13 and 0.37 for  $s = 0.1$  and  $s = 0.7$ , respectively. The results show that when more spurious correlations are in data, CaSN tends to capture information from those spurious correlations, but the algorithm is still able to get sufficient and necessary causes.
Ablation study. In Figure 2(c), we provide the comparison results between CaSN and the  $\mathrm{CaSN}(-\mathrm{m})$  that removes the Monotonicity measurement on synthetic data. The figure demonstrates the distance correlation recorded over 5 experiments. The green bars indicate the distance correlation between learned representation and ground truth by CaSN. CaSN can capture the desired information SN compared to others. As the blue bars show, the  $\mathrm{CaSN}(-\mathrm{m})$  can better capture the causal information (e.g. SN, SF and NC) rather than spurious correlation. It can not stably identify SN, compared to SF.  $\mathrm{CaSN}(-\mathrm{m})$  can be regarded as the method that only cares Exogeneity. The results support the theoretical results in Theorem 4.3, which show the effectiveness of introducing Monotonicity term.
6.3 Generalization to Unseen Domain
The results of the OOD generalization experiments on PACS and VLCS datasets are presented in Tables 1. Due to page limitation, we provide the results on ColoredMNIST in Table 2. The baseline method results are from Kilbertus et al. (2018). The proposed CaSN method exhibits good OOD generalization capability on both PACS and VLCS datasets. In Table 1, CaSN achieves the best average performance over 4 domains by 86.0 on PACS. On the VLCS, CaSN(irm) achieves a good average performance of 78.2, which is close to the best state-of-the-art performance achieved by DANN. For worst-domain test accuracies, the proposed method CaSN outperforms all the baseline methods. An intuitive explanation for the good performance of CaSN is that it aims to identify and extract the most essential information from observation data, excluding unnecessary or insufficient information from the optimal solution. This enables CaSN to better generalize on the worst domain.
In this paper, we consider the problem of learning causal representation from observation data for generalization on OOD prediction tasks. We propose a risk based on the probability of sufficient and necessary causes (Pearl, 2009), which is applicable OOD generalization tasks. The learning principle leads to practical learning algorithms for causal representation learning. Theoretical results on the computability of PNS from the source data and the generalization ability of the learned representation are presented. Experimental results demonstrate its effectiveness on OOD generalization.
We are thankful to Juzheng Miao, Xidong Feng, Kun Lin and Jingsen Zhang for their constructive suggestions and efforts on OOD generalization experiments and for offering computation resources. We also thank Congmin Ji for checking the correctness of the mathematical proof, Pengfei Zheng for his helpful discussions and the anonymous reviewers for their constructive comments on an earlier version of this paper. Furui Liu is supported by the National Key R&D Program of China (2022YFB4501500, 2022YFB4501504). Jianhong Wang is fully supported by UKRI Turing AI World-Leading Researcher Fellowship,  $EP / W002973 / 1$ .
Vapnik, V. N. An overview of statistical learning theory. IEEE transactions on neural networks, 10 (5):988-999, 1999.
Wang, Y. and Jordan, M. I. Desiderata for representation learning: A causal perspective. arXiv preprint arXiv:2109.03795, 2021.
Wang, Y., Li, H., and Kot, A. C. Heterogeneous domain generalization via domain mixup. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3622-3626. IEEE, 2020.
Xiao, T., Wang, X., Efros, A. A., and Darrell, T. What should not be contrastive in contrastive learning. arXiv preprint arXiv:2008.05659, 2020.
Xu, M., Zhang, J., Ni, B., Li, T., Wang, C., Tian, Q., and Zhang, W. Adversarial domain adaptation with domain mixup. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 6502-6509, 2020.
Yan, S., Song, H., Li, N., Zou, L., and Ren, L. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001.00677, 2020.
Yang, M., Liu, F., Chen, Z., Shen, X., Hao, J., and Wang, J. Causalvae: disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9593-9602, 2021.
Yang, M., Cai, X., Liu, F., Zhang, W., and Wang, J. Specify robust causal representation from mixed observations. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2978-2987, 2023.
You, K., Wang, X., Long, M., and Jordan, M. Towards accurate model selection in deep unsupervised domain adaptation. In International Conference on Machine Learning, pp. 7124-7133. PMLR, 2019.
Zhang, K. and Hyvarinen, A. On the identifiability of the post-nonlinear causal model. arXiv preprint arXiv:1205.2599, 2012.
Zhang, K., Scholkopf, B., Muandet, K., and Wang, Z. Domain adaptation under target and conditional shift. In International conference on machine learning, pp. 819-827. PMLR, 2013.
Zhang, K., Gong, M., and Schölkopf, B. Multi-source domain adaptation: A causal view. In Twenty-ninth AAAI conference on artificial intelligence, 2015.
Zhang, Y., Liu, T., Long, M., and Jordan, M. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning, pp. 7404-7413. PMLR, 2019.
Zhang, Y., Gong, M., Liu, T., Niu, G., Tian, X., Han, B., Schölkopf, B., and Zhang, K. Adversarial robustness through the lens of causality. In International Conference on Learning Representations, 2022.
Zhao, H., Des Combes, R. T., Zhang, K., and Gordon, G. On learning invariant representations for domain adaptation. In International Conference on Machine Learning, pp. 7523-7532. PMLR, 2019.
Zhou, K., Liu, Z., Qiao, Y., Xiang, T., and Loy, C. C. Domain generalization: A survey. arXiv preprint arXiv:2103.02503, 2021.