SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For step (ii): The proof refers to the theoretical results in Shamir et al. (2010a), they correspond the optimal solution of information bottleneck with minimal sufficient statistics defined below. We first demonstrate the definition of Minimal Sufficient Statistic in Fisher (1922); Shamir et al. (2010a) below.
Definition E.1 (Sufficient Statistic Fisher (1922); Shamir et al. (2010a)). Let  $Y$  be a parameter of probability distributions and  $\mathbf{X}$  is random variable drawn from a probability distribution determined by  $Y$ . of  $\mathbf{C}$  is a function of  $\mathbf{X}$ , then  $\mathbf{C}$  is sufficient for  $Y$  if
$$
\forall \mathbf {x} \in \mathcal {X}, \mathbf {c} \in \mathbb {R} ^ {d}, y \in \mathcal {Y} \quad P (\mathbf {X} = \mathbf {x} \mid \mathbf {C} = \mathbf {c}, Y = y) = P (\mathbf {X} = \mathbf {x} \mid \mathbf {C} = \mathbf {c})
$$
The definition indicates that the sufficient statistic  $\mathbf{C}$  satisfy the conditional independency  $\mathbf{X} \perp Y|\mathbf{C}$ .
Definition E.2 (Minimal Sufficient Statistic Fisher (1922); Shamir et al. (2010a)). A sufficient statistic  $\mathbf{C}'$  is minimal if and only if for any sufficient statistic  $\mathbf{C}$ , there exists a deterministic function  $f$  such that  $\mathbf{C}' = f(\mathbf{C})$  almost everywhere w.r.t  $\mathbf{X}$ .
From Theorem 7 in Shamir et al. (2010a), the optimization of IB is correspond to finding the minimal sufficient statistic. We show Theorem 7 as a lemm below.
Lemma E.3 (Shamir et al. (2010a)). Let  $\mathbf{X}$  be a sample drawn according to a distribution determined by the random variable  $Y$ . The set of solutions to
$$
\min  _ {\mathbf {C}} I (\mathbf {X}, \mathbf {C}) \text {s . t .} I (Y, \mathbf {C}) = \max  _ {\mathbf {C} ^ {\prime}} I \left(Y; \mathbf {C} ^ {\prime}\right)
$$
is exactly the set of minimal sufficient statistics for  $Y$  based on the sample  $\mathbf{X}$ .
From Lemma E.3, the process of optimizing the IB objective  $I(\mathbf{C}, Y) - \lambda I(\mathbf{C}, \mathbf{X})$  is equivalent to finding the minimal sufficient statistic. Then, the process of optimizing the IB objective is the process to find a  $\mathbf{C}$  satisfying conditional independency  $\mathbf{X} \perp Y|\mathbf{C}$ .
From Step (i) and (ii), we get the result in Theorem 4.3 that optimizing the objective  $\min_{\phi, \mathbf{w}, \lambda} L_{\mathrm{exo}}$  is equivalent to finding a  $\mathbf{C}$  satisfying conditional independency  $\mathbf{X} \perp Y|\mathbf{C}$ . Considering Assumption 4.2 in our paper, we can draw the conclusion that optimizing the objective Eq.(9) is equivalent to satisfying the exogeneity.
F Additional Related Works
OD generalization. In addition to the related works in main text, some works (Gong et al., 2016; Li et al., 2018b; Magliacane et al., 2017; Lu et al., 2021; Rojas-Carulla et al., 2018; Meinshausen, 2018; Peters et al., 2016) solve the OOD generalization problem by learning causal representations from source data by introducing invariance-based regularization into cross-domain representation learning and consider the task labels across multi-domains.
Domain adaptation. Domain Adaptation aims at learning to generalize to the test domain where the distribution is not the same as source domain, while the data from the test data is available during training process. Existing works on domain adaptation use a different way to achieve the goal (Pan et al., 2010; You et al., 2019; Zhang et al., 2013). Magliacane et al. (2018) considers solving the generalization problem from the causality perspective and Zhao et al. (2019) propose a framework of invariant representation learning in domain adaptation. Furthermore, Zhang et al. (2019) provides theoretical analysis of domain adaptation.
Causal discovery. In order to investigate the relationship between causally related variables in real-world systems, traditional methods often employ principles such as the Markov condition and the principle of conditional independence between cause and mechanism (ICM) to discover causal structures or differentiate causes from effects (Mooij et al., 2016). Additionally, several studies have focused on the inherent asymmetry between cause and effect, as seen in the works of Sontakke et al. (2021); Steudel et al. (2010); Janzing & Scholkopf (2010), and Cover (1999). These ideas have also been utilized by Parascandolo et al. (2018) and Steudel et al. (2010). Other research lines focus on modeling structural causal models and discussing the identifiability of causal structures and functions (Hoyer et al., 2008; Zhang & Hyvarinen, 2012). The task of causal discovery typically assumes that
all variables are observable, aiming to identify the unknown causal graph. In contrast, our work deals with causal factors that are hidden within observational data, given a known causal graph. Our objective is to extract causal information from the observational data.
Causal Disentangled Representation Learning: Causal representation learning methods, such as those discussed in Scholkopf et al. (2021), aim to find the disentanglement representation from observational data with causal semantic meaning. Previous works have utilized structural causal models to capture causal relationships within complex observational data, as seen in the works of Yang et al. (2021); Shen et al. (2020); Lu et al. (2021); Wang et al. (2020); Yang et al. (2023), which focus on disentangling causal concepts from the original inputs. While our work also aims to specify causal representations from observational data, we distinguish ourselves by focusing specifically on identifying sufficient and necessary causes. Wang & Jordan (2021) concentrate on learning to disentangle causal representations using non-spurious and efficient information from observational data. Their framework is inspired by the concept of PNS and extends it to a continuous representation space. In contrast, our learning framework is based on the task of OOD generalization, and our algorithms consider both monotonicity and exogeneity conditions. Additionally, we provide a theoretical analysis of the generalization ability of our approach based on PAC learning frameworks, as explored in the works of Shalev-Shwartz & Ben-David (2014) and Shamir et al. (2010b).
Contrastive Learning: The proposed method is also related to another line of research called contrastive learning. These works often deal with the OOD generalization task by designing an objective function that contrasts certain properties of the representation. Saunshi et al. (2022) argue that the framework learns invariant information by introducing certain biases, and the core problem of contrastive learning is the objective function (Xiao et al., 2020). Different from their approaches that perform learning by contrastivity, we use counterfactual PNS reasoning as the function to be optimized, which learns invariant causal information with an explicit stability guarantee for the model's performance.
In this paper, we propose a method to specify the more critical representations based on invariant learning, where we extract the representations from the invariant information that possess sufficient and necessary causalities for the prediction task. This effectively helps improve the performance and robustness of machine learning models in unknown domains. The application of this method can promote reliability and usability in various real-world scenarios. Furthermore, through experiments on real-world datasets, our method demonstrates the ability to enhance model performance even in the worst-case scenarios. This highlights the potential of our method to enhance reliability and usability in various out-of-distribution (OOD) settings. By extracting the essential causal representations from the invariant information, our method provides valuable insights into the underlying causal relationships in the data, leading to improved generalization and robustness in OOD scenarios. These findings support the broader impact of our approach in enhancing the reliability and usability of machine learning models across different domains and applications.